locked
Unexpected behavior step load test pattern in releation with active asp.net sessions RRS feed

  • Question

  • Hi,

    when I choose for the step load pattern within a load test I see unexpected behavior. If I start with 0 users and add a user each minute and monitor active sessions (asp.net application counter). A new session is created for each test until the maximum amount of users is reached (property within the step load pattern).

    I would expect to see that the number of sessions equals the number of loaded users and not the number of tests.

    When I use the constant pattern the number of active sessions does equal the number of loaded users (and there's no relation with the number of tests).

    Is this by design? If yes why?


    rt
    Wednesday, March 18, 2009 9:29 PM

Answers

  • Hi Rene,

    The behavior you are are describing is correct, I validated this today on my own test rig to make sure(I am using Visual Studio 2008 - SP1).

    The key to understanding this stems from your statement here:

    "I would expect to see that the number of sessions equals the number of loaded users and not the number of tests." Rene

    That is not actually how your LoadTest works... I shall elaborate:

    So, in Visual Studio LoadTests, you have a ThreadPool that executes the tests. Think of these Threads executing that run your test as having the ability to assume the identity (the context, asp.net session id's, other data) of a User, but they themselves are not a user. This is very different from other testing tools out there(i.e HP LoadRunner or Borland SilkPerformer) where a User is locked to the thread that created it.

    So, lets dissect your scenario and see if we can make sense of it, again assuming that we have Percentage of New Users = 0.

    When you set the Step Ramp to a Max User count, you can think of this as saying this is the maximum number of distinct users(context, session'id) that I would like to have, and when a thread picks one up from the pool, it assumes this identity (context, sessionid's, etc.) when it runs the tests.

    Your intial count is the number of active users(context,sessionid's) that are eligible to be assigned to a thread in the thread pool. So after a test iteration completes, the context for that user is stored for future runs, but the thread in the thread pool now picks up a 'new user' - again think Context/sessionid's , and then runs the test taking on this new users identity. After that thread completes the test, the user context is stored, and another 'new user - context is created. This cycle continues until you have you Max User Count == to users created (contexts).

    So, this is how you can setup a test that step ramps, but during the steps, the number of unique ASP.NET sessions that you see is actually a higher number, again becaue the threadpool is assuming new identies each iteration until the max users is created. You still get the desired number of concurrent users - you never have more active users submitting requests in-flight than what you specified. So your load test should be highly accurate, but understanding the thread pool model will allow you to confirm the behavior.

    Here is another way to picture this:

    1.       Step load pattern: 

    a.       Initial count: 1 user

    b.      Max count: 10 users

    c.       Step duration: 10 seconds

    d.      Step count: 1 user

    2.       Load test has a single web test that has a single request (just for brevity’s sake).

    3.       Percentage of new users is set to 0


    **note the following also assumes that these tests run and complete very fast - notice how 3 tests complete before 10 seconds, executed serially on one thread.

    LoadTest Begins:
    Thread1-> I am becoming User1 (execute 1 iteration) then return (I just got an ASP.NET Session now - store in my context)
    Thread1-> I am becoming User2 (execute 1 iteration) then return(I have an ASP.NET Session now - store in my context)
    Thread1-> I am becoming User3 (execute 1 iteration) then return(I have an ASP.NET Sesseion now -store in my context)
    **EVENT** 10 Seconds Pased time to Step Ramp 1 user - I am adding 1 more concurrent (Thread1 and now Thread2 pick up users(contexts)
    Thread1-> I am becoming User4 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread2-> I am becoming User5 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread1-> I am becoming User6 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread2-> I am becoming User7 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    **EVENT** 10 Seconds Passed time to Step Ramp 1 user - I am adding 1 more concurrent user(Thread1+Thread2+Thread3 now)
    Thread1-> I am becoming User8 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread2-> I am becoming User9 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread3-> I am becoming User10 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    **EVENT** Max User Count Reached - threads will now begin to re-use the earlier created contexts
    Thread1-> I am becoming User1 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread2-> I am becoming User2 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread3-> I am becoming User3 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    **EVENT** 10 Seconds Passed time to Step Ramp 1 user - I am adding 1 more concurrent user(Thread1+Thread2+Thread3+Thread4 now)
    Thread1-> I am becoming User4 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread2-> I am becoming User5 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread3-> I am becoming User6 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread4-> I am becoming User7 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)

    <<THIS CYCLE CONTINUES UNTIL Threads = Max User Count, Thread1 - Thread10 >>
    **EVENT** LoadTest Duration has been reached, we now stop the test
    LoadTest Completed


    I think the names of the properties in the Step Ramp pattern can be a bit misleading, but its hard to explain a thread pool conceptually in a small space.

    Please let me know if I have helped you or caused you more confusion.

    Robert
    Thursday, March 19, 2009 9:28 PM
  • Thanks for your explanation it's clear.

    On the other I have to say it's not very intuitive. Especially because the behavior is very different from a constant load (where the number of sessions = number of loaded users).

    The reason it confused me was the I first started a load for 6 hours with 100 users with the constant pattern. The number of active sessions was 100 continuously.

    Next I wanted to see the impact of the number of users on performance so I started a load test with the step load test pattern (from 0 to 200, every half an hour 20 users were added). My goal was to compare these results. But because the steps load patterns behaves differently it’s not fair to compare them. When the step load pattern test has reached 100 users the situation is very different from the situation when I used the test with the constant load pattern because the number of active sessions with 100 users in the step load pattern is 200. Because the number of active sessions are different I can't compare the tests. For example when there are more sessions it's very likely that the memory consumption is bigger since every session stores data in memory.

    That's why it confused me. I do know the explanation now but I thinks it's strange still. I don't see a reason why it's implemented this way.

    Regards,
    René

        
    rt
    Friday, March 20, 2009 7:40 AM
  • Rene, There's a very easy explanation for the behavior you are seeing. The step load and the constant patterns should generate the same number of sessions, and I suspect they did. However, think about this.  If you ran the constant pattern and generated 100 sessions, and then when the test was done and you changed the pattern and reran the test within 20 minutes and without resetting IIS, then you would spin up 100 more sessions while the first 100 sat idle.

    I suspect that if you reset IIS before each test that the count you see would be identical.

    :-)
    Friday, March 20, 2009 2:17 PM

All replies

  • Hi Rene,

    Have you made any changes to the Scenario between tests, particularly to the Percentage of New Users value? I believe you should see the number of active sessions = number of users if Percentage of New Users = 0, and you would see the sessions increase to the number of tests completed if Percentage of New USers = 100. I cannot verify this behavior right now, so maybe others can chime in if I have this wrong. I will be able to test tomorrow, if nobody confirms for you.

    The other thing I would ask is what is your timeout for the sessions on the server, and this could affect the values if the tests are running different time periods, since you are monitoring for Active Sessions. If you monitor total sessions and active sessions together, I think you could easily rule out this variable.

    Also, I think these default values were changed between VS2005 and VS2008 for the Percnetage of New User (100, 0) respectively.

    -Robert
    Wednesday, March 18, 2009 11:59 PM
  • Robert,

    thanks for your reply. I checked the value of percentage of new users which is 0. I would expect that with the step load pattern new sessions only will be created when a new user is added (if of course percentage of new users is 0).

    My test is very short (only 5 minutes) because I'm investigating this behavior.

    I'm looking forward to your test results.

    René
    rt
    Thursday, March 19, 2009 11:52 AM
  • Hi Rene,

    The behavior you are are describing is correct, I validated this today on my own test rig to make sure(I am using Visual Studio 2008 - SP1).

    The key to understanding this stems from your statement here:

    "I would expect to see that the number of sessions equals the number of loaded users and not the number of tests." Rene

    That is not actually how your LoadTest works... I shall elaborate:

    So, in Visual Studio LoadTests, you have a ThreadPool that executes the tests. Think of these Threads executing that run your test as having the ability to assume the identity (the context, asp.net session id's, other data) of a User, but they themselves are not a user. This is very different from other testing tools out there(i.e HP LoadRunner or Borland SilkPerformer) where a User is locked to the thread that created it.

    So, lets dissect your scenario and see if we can make sense of it, again assuming that we have Percentage of New Users = 0.

    When you set the Step Ramp to a Max User count, you can think of this as saying this is the maximum number of distinct users(context, session'id) that I would like to have, and when a thread picks one up from the pool, it assumes this identity (context, sessionid's, etc.) when it runs the tests.

    Your intial count is the number of active users(context,sessionid's) that are eligible to be assigned to a thread in the thread pool. So after a test iteration completes, the context for that user is stored for future runs, but the thread in the thread pool now picks up a 'new user' - again think Context/sessionid's , and then runs the test taking on this new users identity. After that thread completes the test, the user context is stored, and another 'new user - context is created. This cycle continues until you have you Max User Count == to users created (contexts).

    So, this is how you can setup a test that step ramps, but during the steps, the number of unique ASP.NET sessions that you see is actually a higher number, again becaue the threadpool is assuming new identies each iteration until the max users is created. You still get the desired number of concurrent users - you never have more active users submitting requests in-flight than what you specified. So your load test should be highly accurate, but understanding the thread pool model will allow you to confirm the behavior.

    Here is another way to picture this:

    1.       Step load pattern: 

    a.       Initial count: 1 user

    b.      Max count: 10 users

    c.       Step duration: 10 seconds

    d.      Step count: 1 user

    2.       Load test has a single web test that has a single request (just for brevity’s sake).

    3.       Percentage of new users is set to 0


    **note the following also assumes that these tests run and complete very fast - notice how 3 tests complete before 10 seconds, executed serially on one thread.

    LoadTest Begins:
    Thread1-> I am becoming User1 (execute 1 iteration) then return (I just got an ASP.NET Session now - store in my context)
    Thread1-> I am becoming User2 (execute 1 iteration) then return(I have an ASP.NET Session now - store in my context)
    Thread1-> I am becoming User3 (execute 1 iteration) then return(I have an ASP.NET Sesseion now -store in my context)
    **EVENT** 10 Seconds Pased time to Step Ramp 1 user - I am adding 1 more concurrent (Thread1 and now Thread2 pick up users(contexts)
    Thread1-> I am becoming User4 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread2-> I am becoming User5 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread1-> I am becoming User6 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread2-> I am becoming User7 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    **EVENT** 10 Seconds Passed time to Step Ramp 1 user - I am adding 1 more concurrent user(Thread1+Thread2+Thread3 now)
    Thread1-> I am becoming User8 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread2-> I am becoming User9 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    Thread3-> I am becoming User10 (execute 1 iteration) then return(I have an ASP.NET Session now -store in my context)
    **EVENT** Max User Count Reached - threads will now begin to re-use the earlier created contexts
    Thread1-> I am becoming User1 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread2-> I am becoming User2 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread3-> I am becoming User3 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    **EVENT** 10 Seconds Passed time to Step Ramp 1 user - I am adding 1 more concurrent user(Thread1+Thread2+Thread3+Thread4 now)
    Thread1-> I am becoming User4 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread2-> I am becoming User5 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread3-> I am becoming User6 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)
    Thread4-> I am becoming User7 (execute 1 iteration) then return(I REUSE my earlier ASP.NET Session now From my context)

    <<THIS CYCLE CONTINUES UNTIL Threads = Max User Count, Thread1 - Thread10 >>
    **EVENT** LoadTest Duration has been reached, we now stop the test
    LoadTest Completed


    I think the names of the properties in the Step Ramp pattern can be a bit misleading, but its hard to explain a thread pool conceptually in a small space.

    Please let me know if I have helped you or caused you more confusion.

    Robert
    Thursday, March 19, 2009 9:28 PM
  • Thanks for your explanation it's clear.

    On the other I have to say it's not very intuitive. Especially because the behavior is very different from a constant load (where the number of sessions = number of loaded users).

    The reason it confused me was the I first started a load for 6 hours with 100 users with the constant pattern. The number of active sessions was 100 continuously.

    Next I wanted to see the impact of the number of users on performance so I started a load test with the step load test pattern (from 0 to 200, every half an hour 20 users were added). My goal was to compare these results. But because the steps load patterns behaves differently it’s not fair to compare them. When the step load pattern test has reached 100 users the situation is very different from the situation when I used the test with the constant load pattern because the number of active sessions with 100 users in the step load pattern is 200. Because the number of active sessions are different I can't compare the tests. For example when there are more sessions it's very likely that the memory consumption is bigger since every session stores data in memory.

    That's why it confused me. I do know the explanation now but I thinks it's strange still. I don't see a reason why it's implemented this way.

    Regards,
    René

        
    rt
    Friday, March 20, 2009 7:40 AM
  • Rene, There's a very easy explanation for the behavior you are seeing. The step load and the constant patterns should generate the same number of sessions, and I suspect they did. However, think about this.  If you ran the constant pattern and generated 100 sessions, and then when the test was done and you changed the pattern and reran the test within 20 minutes and without resetting IIS, then you would spin up 100 more sessions while the first 100 sat idle.

    I suspect that if you reset IIS before each test that the count you see would be identical.

    :-)
    Friday, March 20, 2009 2:17 PM