Load Testing using a Step Load Pattern

Question

Bill,

Could you please help me out with a couple of questions.  If this is not the appropriate place to post, let me know.

1)  I am running a Stepped Load Test with the following Paramater values: 

Initial User Count - 10
Maximum User Count - 200
Step Duration (seconds) - 10
Step Ramp Time (seconds) - 0
Step User Count - 10

Based on these values, the Initial User count will be 10 and will increase by 10 users every 10 seconds.

Initial User Count = 10
Step 1 = 20 new users logging in to the system
Step 2 = 30 new users logging in to the system
Step 4 = 40 new users logging in to the system
Step 5 = 50 new users logging in to the system
Step 6 = 60 new users logging in to the system

Is this correct as I am seeing different results?

2)  Using the same Step Load Test as described above, when I select the RunSettings1 [Active] tab, I am interested in 2 Properties

a)  Test Iterations
b)  Timing - Run Duration

Are these properties ignored when a Stepped Load Test is run?  When are these properties used?

Thanks Bill!

Answer 1

Hello AAMPSAA,

The Step User Count controls how many new users are added to the existing users(exsiting users + stepcount). Using your chart, Step 1 would be the initial users + your step count of 10 (10 existing plus 10 new users), Step 2 is now the 20 existing users + 10 new users = 30 users, and so on.


The settings for Test Iterations and Run Duraton are mutually exclusive. You set your test to use one or the other.


Run Settings -> Test Iterations + Run Settings -> Use Test Iterations = true,  is often misunderstood: Think of it as the global counter, or total of all the test iterations for all users, and when it reaches the count specified the test stops - so duration of the test is not relevant since you are saying it should complete the total number of iterations you specified.  See this post for more if not clear: http://social.msdn.microsoft.com/Forums/en-US/vstswebtest/thread/b384dc59-86d4-42cb-962f-ab61673d7ce1

Run Settings-> Run Duration, the duration of the load test. If you specify a value here, then you are saying that the test should run for this length of time and will attempt to complete as many iterations as possible. You must remember to set  your Use Test Iterations = false, otherwise even if you have a time entered here, the test will ignore duration, and stop at the number of total iterations completed.

Hope this helps,
Robert

Answer 2

Hi Robert,

Please let me make sure I understand you correctly.  I have a web test which simulates a user logging in to a web site and navigating to various pages and logging out.  I add a datasource (.csv file with 3000 users ids and passwords) to the web test.  I add the web test to a load test and set a constant sequential  load of 10.  I then set Test Iterations = 1000 and Use Test Iterations = Yes. 

When I run the load test I would expect the load test to iterate through the first 1000 users in my .csv file and the test would end ignoring any time specified in the Run Duration.  Is this correct?

Now I want to run the same Load Test but this time I only want the test to run for 5 minutes.  So I set the Run Duration to 00:05:00. 

Do I need to set the Use Test Iterations = False so the Load Tests ignores the Test Iterations and just runs as many of the users in my .csv file as it can in 5 minutes?

Now using the same Load Test I change it to a Stepped Load Test with the following Parameter values: 

Initial User Count - 10
Maximum User Count - 200
Step Duration (seconds) - 10
Step Ramp Time (seconds) - 0
Step User Count - 10

These values will result in 10 users initially loaded, then after 10 secs another 10 users are loaded and will repeat until the first 200 users in my .csv file are loaded. 

Is the Run Duration and Test Iterations ignored when running Stepped Load Test?

For this load test I added a controller for the web server.  When I run the Load Test and Review the Load Test summary, I see a % Processor Time = 75.6.

Does this mean that the processor is busy 75.6% of the time or 75.6% of the processor Time is available or neither?  

Sorry for lengthening this so much, but I have read so much on this that I have become thoroughly confused.  I probably should have got up and walked away from it.

Thank you Robert!

Answer 3

Hi AAMPSAA,


When I run the load test I would expect the load test to iterate through the first 1000 users in my .csv file and the test would end ignoring any time specified in the Run Duration.  Is this correct?
Correct - because you set "Use Test Iterations" = true


Do I need to set the Use Test Iterations = False so the Load Tests ignores the Test Iterations and just runs as many of the users in my .csv file as it can in 5 minutes?
Correct

Is the Run Duration and Test Iterations ignored when running Stepped Load Test?
No, you need to adjust your test iterations or your run duration, depending on which one you are using then to allow your step ramp to complete. You might need a little trial and error to calibrate your test settings.

Does this mean that the processor is busy 75.6% of the time or 75.6% of the processor Time is available or neither?  
This is the % Processor Time Performance Counter, so 75.6% is how busy. I dont know if this is helpful, but check the long description here:

% Processor Time is the percentage of elapsed time that the processor spends to execute a non-Idle thread. It is calculated by measuring the percentage of time that the processor spends executing the idle thread and then subtracting that value from 100%. (Each processor has an idle thread that consumes cycles when no other threads are ready to run). This counter is the primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval. It should be noted that the accounting calculation of whether the processor is idle is performed at an internal sampling interval of the system clock (10ms). On todays fast processors, % Processor Time can therefore underestimate the processor utilization as the processor may be spending a lot of time servicing threads between the system clock sampling interval. Workload based timer applications are one example  of applications  which are more likely to be measured inaccurately as timers are signaled just after the sample is taken.

Answer 4

Robert,

Very helpful!

Thanks

Answer 5

Hi AAMPSAA,


When I run the load test I would expect the load test to iterate through the first 1000 users in my .csv file and the test would end ignoring any time specified in the Run Duration.  Is this correct?
Correct - because you set "Use Test Iterations" = true


Do I need to set the Use Test Iterations = False so the Load Tests ignores the Test Iterations and just runs as many of the users in my .csv file as it can in 5 minutes?
Correct

Is the Run Duration and Test Iterations ignored when running Stepped Load Test?
No, you need to adjust your test iterations or your run duration, depending on which one you are using then to allow your step ramp to complete. You might need a little trial and error to calibrate your test settings.

Does this mean that the processor is busy 75.6% of the time or 75.6% of the processor Time is available or neither?  
This is the % Processor Time Performance Counter, so 75.6% is how busy. I dont know if this is helpful, but check the long description here:

% Processor Time is the percentage of elapsed time that the processor spends to execute a non-Idle thread. It is calculated by measuring the percentage of time that the processor spends executing the idle thread and then subtracting that value from 100%. (Each processor has an idle thread that consumes cycles when no other threads are ready to run). This counter is the primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval. It should be noted that the accounting calculation of whether the processor is idle is performed at an internal sampling interval of the system clock (10ms). On todays fast processors, % Processor Time can therefore underestimate the processor utilization as the processor may be spending a lot of time servicing threads between the system clock sampling interval. Workload based timer applications are one example  of applications  which are more likely to be measured inaccurately as timers are signaled just after the sample is taken.

Hi Robert,

thank you for great explanation.
However, i have one question. Consider

Recorded webtest(user login, thinking, browsing the pages, logout) duration: 10 minutes

Initial User Count - 0
Maximum User Count - 1000
Step Duration (seconds) - 10
Step Ramp Time (seconds) - 0
Step User Count - 10

Run duration: 60 minutes

Should this load test reach 1000 simultaneously users?

Best regards

Answer 6

Anyone?

Answer 7

Hi Gamb,

Given the ramp up paramters you specififed you should have all 1000 users ramped up in 17 minutes. So, yes you would have 1000 concurrent users for the remaining 43 minutes of your test run.

Robert

---

http://blogs.msdn.com/rogeorge

Answer 8

Hi Robert,

thank you for replying.

What i dont understand is that after 10 minutes(webtest duration), first webtest(user) will end the test(logout).
So, what is happening after 10 minutes?
I hope you understand me.


Best regards

Answer 9

Hi Gamb,

The virtual users will continue to start / complete new iterations for the duration of the test run. If you want to limit the tests completed to 1,000 in a given time duration, then set the Use Test Iterations = True, and set the Test Iterations to 1,000. Hope that helps,
-Robert

---

http://blogs.msdn.com/rogeorge