Performance decreases when increasing number of virtual but enough resources available

Question

Hi,

I have a load test which runs one web test. This web test executes requests to a WCF service. These are some results:

Test A: LAN - 10 users - 20 tests -> 20 minutes per test per user

Test B: 1,5 Mb - 10 users -20 test -> 2,5 hour per test per user

Test C: 1,5 Mb - 30 users - 60 test -> almost 7 hours per test per user

The strange thing about these results is the difference between B and C. Performance decreases when the number of users increases. This sounds logical at first but when I look at the resources on the web server and the agents they are simular in test B and C (for example 0-25% processor time for the web server). The agents are also ok.

When I run my application locally when the load test runs, there's no difference in performance, it even looks like my application performs better when at te same time a load test runs with 30 users instead of no load test.

I don't really trust my results. It looks like the test configuration doesn't reflect my experience with a real life situation.

Hopefuly somebody can help me explaining the results.  

---

rt

Answer 1

Rene,

Dont fall into the trap of looking only at CPU/Memory as the only resource. From the description above, it looks like your Test B and Test C are both running with reduced bandwidth (1.5mbs), is that right? If so, then your seeing the effects of bandwidth saturation. The servers are not working harder, they are working less with more users - but the time to first byte and last byte of responses and requests is increasing due to bandwidth saturation.  When you run the test/app locally then your not bound by the bandwidth limitations.

If the names of the test above are just coincidental, then your still showing the effect of some other bottleneck on the system, and typically this pattern of reduced CPU/Memory will show up despite adding more users. The trick is to narrow down the scope of where the problem is, you will likely need help from developers on this one to isolate, and get to root cause before trying to fix what may be the problem.

---

http://blogs.msdn.com/rogeorge

Answer 2

Hi,

thanks for your reply.

 You're right test B and C are running with reduced bandwidth using Visual Studio network emulation. I don't understand that bandwidth saturation could be the problem. There's a Gigabit connection between my agents and the webservers. So those 30 users have to share that Gigabit connection: 30 * 1,5 Mb = 45 Mb so it only uses a fraction of the total connection available.

We are indeed in the phase now of searching for the bottleneck on the system and narrow down the scope of the problem. I am one of the developers of the team but I can't explain the results we are getting. The question for me now is: are the results application related or is it related to the test configuration it self, I don't trust that network emulation feature fully.

I would really appreciate if you could explain bandwidth saturation in more detail if you think this could really be the explenation. Or if you do have any other ideas please let me know.

Thanks,

René  

 

---

rt

Answer 3

If I recall correctly the setting for the bandwidth constraint is applied in your case as 1.5 MB for all 30 users to share, its not per virtual user. I believe that is right, so if you removed the bandwidth constraint, re-run test, and get a different result I think that would be a quick way to help narrow this down.

---

http://blogs.msdn.com/rogeorge

Answer 4

Within the network mix in the load test I did configure 100% to 1,5 Mb. This means 1,5 Mb per user in my opinion.

Are we talking about the same property? I don't know where you could configure a constraint for a bandwidth (for example 1,5 MB) for all users.

Please explain.

---

rt

Answer 5

I am not 100% sure, I might be wrong, I do not routinely use that feature since its software based emulation, in the large scale labs we run we have hardware emulation devices that are dedicated for this purpose,  thats why I said if I recall - but either way if you run the test without these bandwidth constraint turned on it would help to narrow this down. 

To your point, I think the feature intent would be based on per user sounds right. Since you can specify a % of users and multiple network mix then it does seem to lean more towards a per user model.

---

http://blogs.msdn.com/rogeorge

Answer 6

Any help would be appreciated.

Cany anyone explain my results. Could they be related to network emulation?

---

rt

Answer 7

Hello René,

I am trying to involve someone familiar with this topic to further look at this issue.

Thanks.

---

Vicky Song [MSFT]
MSDN Community Support | Feedback to us

Answer 8

I would really appreciate that, it stops us from analyzing the performance of our system.

---

rt

Answer 9

Hi Vicky,

do you have any news?

I did do some further analyzing.

I inspected the performance counters of the network interface (the one which is bound to the VSTS network emulation driver).

I see a lot of treshold violations against output queue length (max is 13 with 30 users and 6 with 10 users).

These are the results for bytes received\sent (1,5 Mb emulation):

                            Bytes received\s                    Bytes send\s

10 users:             184.097                                11.293

30 users:             186.205                                14.573

You would expect that the 30 users send and receive 3 times as much bytes as 10 users (when the webservers are not busy)

When I run it without newtwork emulation the results are

                            Bytes received\s                    Bytes send\s

10 users:             1.184.273                              87.321

At that time the system under load becomes the bottleneck (100% cpu) but as you can see much more bytes can be transfered with the same network interface (but now without emulation). I also get treshold violations then with the output queue length, the maximum is 13.

It looks like the network emulation driver doesn't really emulate as expected.

The way it looks now is that when I choose for 1,5 Mbps emulation, that all users within that test share this 1,5 Mbps.

1,5 Mbps = 187.500 bytes\sec (1,5 * 1000 * 1000 \ 8), in my opinion and this is approximately the bytes received above.

So when I choose for 1,5 Mbps emulation, the total number of virtual users (10 or 30) have to share this bandwidth. This would explain my results. But it's weird in my opinion that emulation works this way. Because I would expect that when I choose for 1,5 Mbps and 30 users, that each user will have 1,5 Mbps.

Do you agree with my conclusion?  

Thanks.

---

rt

Answer 10

Hello Rene,

The connection should be per user, not all users share this 1.5Mbps. Is the WebTest Connection Model set to Connection Per User?

Thanks & Regards,

Jian-Wei Yu [MSFT]

Microsoft Online Community Support

--------------------------------------------------------------------------------

Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

Answer 11

Yes, connection per user.

I have a workaround now to do load tests. I created new custom emulations configurations. When I want to emulate 10 users with 1,5 Mbps, I create an emulation with 15Mbps and then everything works as expected (10 users share 15 Mbps, which 1,5 Mbps per user). But that is not the way it should work.

Since this is a workaround, I hope you can help to figure out why it doesn't work as it should.

---

rt

Answer 12

Hello Rene,

Do you encounter this issue if you are using another test machine?

If you are using Windows 2003, you may check the article:

http://blogs.technet.com/b/yongrhee/archive/2010/07/07/network-interface-output-queue-length-values-extremely-high.aspx

Thanks & Regards,

Jian-Wei Yu [MSFT]

Microsoft Online Community Support

--------------------------------------------------------------------------------

Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

Answer 13

Yes, we use w2k3, didn't try any other versions.

The article does explain the high values of output queue length but not why all users share the same bandwidth.

Regards,

René

---

rt