locked
Is anyone seeing Azure storage being slow right now (North Central US)? RRS feed

  • Question

  • Is anyone seeing Azure storage being slow right now (North Central US)?  We have a worker role that -

    - Reads a message from a queue.
    - Writes to Table storage
    - Deletes the message from the queue.

    We normally have 3-4 instances of this worker role that can handle our normal traffic.  This morning we saw our message queue growing, so we kept bumping the number of worker role instances up - we are currently running 10 instances and the message queue keeps growing.

    Our traffic has not increased and we have not deployed new code for the last three weeks, so we are thinking there's an issue with Azure storage.  I checked the dashboard http://www.microsoft.com/windowsazure/support/status/servicedashboard.aspx - but everything looks green.

    We have tried re-deploying to staging and swapping for new instances, but that did not help.  

    We have someone calling Microsoft support - but that always takes time as we have to go through whatever script they have - so I'm just wondering if anyone else is experiencing a similar issue?

    Thursday, July 7, 2011 7:27 PM

Answers

  • Hi Emmanuel,

    It seems like a service degradation. Have your figured out the cause by calling Microsoft support?

    Thanks.


    Wenchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    • Marked as answer by Brad Calder Monday, March 12, 2012 9:18 AM
    Monday, July 11, 2011 6:50 AM

All replies

  • What does your logging tell you? Is it the Queue operations or the Table storage operation that's taking up the time. I imagine you're going to need to provide some more detail as to where exactly the bottleneck is.

    Do you read from Table storage before you write? Are you possible reading a large number of records that is increasing over time? This can cause a slow-down for obvious reasons, especially when you're inadvertently retrieving more than 1000 rows.

    Really hard to know if Azure storage is at fault or whether there's something funky with your code without more info and some detailed log analysis.

    Not saying this will solve your problem however this article is certainly worth a read if you haven't seen it already http://windowsclient.net/blogs/anshulee/archive/2010/05/27/best-practices-for-ado-net-dataservices-when-using-azure-table-storage.aspx. The optimizations suggested are well worth it (and very easy to implement).

    Thursday, July 7, 2011 8:14 PM
  • Yeah, you're right, we'll need to get better numbers on each of the actions we're taking to really find the problem.  We do work with Azure diagnostics and we even write and track custom performance counters, so we can trace our actions a bit more in this worker role.

    My main goal was just to see if there was an issue with Azure storage and that does not seem to be the case.  Thanks for the tips!

    Thursday, July 7, 2011 9:46 PM
  • Hi Emmanuel,

    It seems like a service degradation. Have your figured out the cause by calling Microsoft support?

    Thanks.


    Wenchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    • Marked as answer by Brad Calder Monday, March 12, 2012 9:18 AM
    Monday, July 11, 2011 6:50 AM