Server Too Busy Exception on API call to Azure service

Question

I have an application that queries a bunch of Azure services to see how many instances they have.  These calls go to multiple services across multiple subscriptions.  After a while I start getting service too busy exceptions on all my calls and have to restart the app.  Is there a preferred way to query a Windows Azure service so that I do not keep getting server too busy exceptions?

Answer 1

I presume these are calls to the management API? How many TPS are you sending and how frequently are you checking? If you are sending a large number of requests, it could be that the WA security precautions are interpretting your requests as a DoS attack and throttling you dow.

Another option, and one I would consider if you have control over all the subscriptions is to have the services log this information as they scale up and down and write the results to a storage account. This way with a single query you can get the relative information and aggregate it. Taken to another level, you can have the services send your aggregator notification when they initiate a change in size so that doesn't even go look unless its notified there was a change.

Answer 2

Sorry, I didn't get an email saying that someone had responded to the thread.  Yes, these are calls to the management API.  The way my setup works is it goes to each service in each account and requests the server details for each.  I have about 10 accounts that I manage, and each account has about 20 services in each.  So the process queries about 200 services, and queries about 20 services at a time.  There are some sleeps in between these calls, but if I keep this going 24/7 eventually I get these errors.  I use this same strategy for the Amazon cloud and Rackspace cloud APIs and Azure is the only one that starts spitting out errors after a while.

How would I have the services log this information?  Is this a feature that comes with Azure, or are you saying something I should implement in my application?

Thanks

Answer 3

Self logging is something you'd have to implement. Its not overly complicated if you're just trying to get instance counts. If you can't hook into whatever auto-scaling agent you're working with, you can listen for the Role Environment Changing even and log things there. You could also have thread/process is a master elected master that crawls the service topology via the RoleEnvironment class to see how many roles/instances there are in the service. 

Answer 4

How would I hook into that event?  I am actually the VM Role for my servers so my applications don't talk directly to the Azure infrastructure.  I only use the Service Management API for server creation/termination.

---

KL