Decrease perfomance of Windows Azure web-site

Question

Hello, I have a web-site, which hosted in Windows Azure. One day ago I found perfomance issue: HTML-code generates on server 10 seconds instead of 0.7 secomds.

Solution structure: 2 WebRoles (ASP.NET), SQL Azure DB, Azure Storage for pictures, all stuff is located in North Europe.

Routing: there is CNAME-record "www" on Name Server, which follows on mysite.cloudapp.net

During the search solutions were made following steps:

1) Deploy solution with enabled profiler. It Was established that function which contatins request to DB is the most time-consuming.

2) It was created new hosted service and deploy on it the same solution (use the same DB and Storage). Web-site works perfect on this new hosted service.

3) It was created staging deployment 1234567890qweretyui.cloudapp.net, web-site works perfect on it, but after VIPs swaping mysite.cloudapp.net continue work slowly and 1234567890qweretyui.cloudapp.net continue work fine.

Is somebody have an idea what can I do.

PS: I wouldn't like change CNAME-records

Answer 1

Is this website being used in production already? I ask because a VIP swap doesn't do anything to the actual app, so for performance characteristics to change, I'm wondering if load changed (which would happen if people are hitting that URL).

Answer 2

Steve, thank you for answer.

Yes, this website is beeing used in production. I have 2 deployments: with enabled profiler and with disabled one. When I swap VIPs I can see that URL mysite.cloudapp.net refers to different different deployments.

Answer 3

humm saw a similar case very recently - was due to kind of network congestion on one host.
What if you delete the Production slot and redeploy? you should fall on another host and that'll help to see if it was machine-specific.

Answer 4

Well, certainly don't delete your production deployment, since this is an app being actively used, and that would cause downtime.

Alex, I assume by "host" you mean the servers on which his VMs are running? A VIP swap would cause different hosts to be involved, and he's already done that.

Oleg, I still believe my first guess is correct, that the app performance is suffering due to load. As soon as you put it in production, your users are hitting the site, and this is causing it to run slower.

Answer 5

Steve, unfortunately your assumption is incorrect, there were 8 requests per second maximum. Usually site copes with bigger load.

In a result, tonight I completely deleted compute instance and created it again, after that all has been working fine. Service was only 5 minutes in down.

Despite this I would like to understand what could be a reason of that issue.

Answer 6

When you swap something from staging to production, the deployment is still running on the same VMs on the same servers on the same rack behind the same load balancer. All that changes is that traffic is directed to it. Therefore, if the performance changes when you swap into production, it must be because of the traffic. (I'm making the assumption that your app logic doesn't make an attempt to determine which slot it's in and change behavior based on that.)

There are other possible explanations. Perhaps the app's performance wasn't actually changing. It could be that the measured performance difference is external to the app (such as DNS, networking, etc.). I don't see how you could determine that now, though, without a way to reproduce the observed difference.

Answer 7

Hi,

We actually ran into the same problems. Staging and production have the exact same code deployed. Putting load (4 times the current production load) on the staging environment works as expected (fast). The production environment also runs fine.

Hitting the swap VIP button will instantly kill the new production environment (timeouts at almost all the requests), although the load is less heavy than it was while the VMs where in staging.

(How) did you solve this problem?

Thanks in advance.

Edit:
Solution was to fully delete the staging environment and recreate it (simply redeploying did not do the trick).