Tests/Sec vs Avg. Test Time (sec)

Question

I am comparing the results of two load tests. Both load tests ran for 2 minutes with a constant load of 25 users.

One load test shows 109 Tests/Sec with 0.011 Avg. test time.

The other load test shows 198 Tests/Sec with 0.033 Avg. test time.

How can that be? How can one load test show higher tests/sec number, but avg.test time is 3 times worse.

Thanks.

---

Evgueni Tsygankov

Answer 1

Do both load tests contain the same tests in the test mix?

Also, these numbers do not take into account tests which may have been in progress when the loadtest ended.  (When the loadtest ends, it kills any tests in progress and those numbers are not reported)

And lastly, think time.  The think time between test iterations will not affect your average test time, but will affect how many tests get run in your 2 minutes.

--Mike

Answer 2

It's very odd.

There is no think time. And I can see the "discrepancy" during the entire run of the tests -- from the start Load Test #1 has lower Tests/Second but better avg. test time.

Both load test #1 and load test #2 contain 4 unit tests with the same structure, but one load test goes against SQL Server 2008 and the other one goes against Oracle 11G. The four unit tests retrieve 1, 100, 1000 and 10000 records from the database.

For SQL Server 2008, I get Tests/Sec 198 and Avg. Test Time (sec) 0.033 with the following unit  test breakdown:
Name Scenario Total Tests Failed Tests (% of total) Avg. Test Time (sec)
SqlServer_10000 Scenario1 5,907 0 (0) 0.048
SqlServer_1 Scenario1 6,024 0 (0) 0.029
SqlServer_1000 Scenario1 5,886 0 (0) 0.027
SqlServer_100 Scenario1 5,940 0 (0) 0.027

For Oracle 11G, I get Tests/Sec 105 and Avg. Test Time (sec) 0.010 with the following unit test breakdown:
Name Scenario Total Tests Failed Tests (% of total) Avg. Test Time (sec)
OracleODPNET_10000 Scenario1 3,391 0 (0) 0.029
OracleODPNET_1000 Scenario1 3,452 0 (0) 0.0054
OracleODPNET_100 Scenario1 3,375 0 (0) 0.0054
OracleODPNET_1 Scenario1 3,493 0 (0) 0.0013

Both tests use 25 user constant load.

Your explanation would be greatly appreciated.

Thanks.

 

 

---

Evgueni Tsygankov

Answer 3

The numbers look correct, based on the data.  For Loadtest 1:  (5097 + 6024 + 5886 + 5940) / 120 seconds  = 198 tests / second

However, I suspect what you're for looking is "if Loadtest 2 has tests which are faster, why aren't more executing?"

To answer this, you'll need to figure out what is going on between the iterations.  Obviously something is causing the machine to be unable to start an iteration for loadtest2 immediately after one is completed.   Is your test holding onto any memory which is taking awhile to free up?

If you're using Visual Studio 2010, I recommend looking at the "User Details View" (it's the 4th tab in the load test analyzer, just to the right of the tables view.)  It shows this sort of data graphically (You'd see a large empty space between test iterations)

--Mike

Answer 4

Did this answer your question?

--Mike

Answer 5

I haven't heard back, so I'm going to go ahead and mark this as answered - please feel free to follow up if you are still having trouble.

--Mike

Answer 6

The Tests/Sec was off for me as well (VS 2008)

I had a load test which ran 100 tests in the duration of 00:01:54. The overall result was 0.87 Tests/Sec, which was good. However, on the "Tables" tab, it reported:

Test               Scenario              Total               Passed          Failed             Tests/Sec           Test Time

UnitTest1        Load                    26.0               26.0              0                    5.20                  2.51

UnitTest2        Load                    26.0               26.0              0                    5.20                  5.52

UnitTest3        Load                    24.0               24.0              0                    4.80                  3.00

UnitTest4        Load                    24.0               24.0              0                    4.80                  4.63

How could it be right???

Answer 7

I have something even more weired. I have a web test which takes about 4 seconds to run. When I used this webtest in a load test the average test time climbes upto 21 seconds. I have changed the setting of the loadtest with only this webtest running and with user load of 1 and again had nearly 20 seconds for the average test time.

 

Does anyone know why webtests are taking longer to run in a load test.

Thanks

Hal