none
Azure Storage Queue very slow from a worker role in the cloud, but not in my machine.

    Question

  • Hi,

    I'm doing a very simple test with queues pointing to the real Azure Storage and, I don't know why, executing the test from my computer is quite faster than deploy the worker role into azure and execute it there. I'm not using Dev Storage when I test locally, my .cscfg is has the connection string to the real storage.

    The test is a web role and a worker role. The page tells to the worker what test to do, the the worker do it and returns the time consumed. This specific test meassures how long takes get 1000 messages from an Azure Queue using batches of 32 messages. First, I test running debug with VS, after I deploy the app to Azure and run it from there.

    The results are:

    • From my computer: 34805.6495 ms.
    • From Azure role: 7956828.2851 ms.

    That could mean that is faster to access queues from outside Azure than inside, and that doesn't make sense.

    I'm testing like this:

      private TestResult InQueueScopeDo(String test, Guid id, Int64 itemCount)
      {
        CloudStorageAccount account = CloudStorageAccount.Parse(_connectionString);
        CloudQueueClient client = account.CreateCloudQueueClient();
        CloudQueue queue = client.GetQueueReference(Guid.NewGuid().ToString());
    
        try
        {
          queue.Create();
          PreTestExecute(itemCount, queue);
    
          List<Int64> times = new List<Int64>();
          Stopwatch sw = new Stopwatch();
          for (Int64 i = 0; i < itemCount; i++)
          {
            sw.Start();
            Boolean valid = ItemTest(i, itemCount, queue);
            sw.Stop();
            if (valid)
              times.Add(sw.ElapsedTicks);
            sw.Reset();
          }
    
          return new TestResult(id, test + " with " + itemCount.ToString() + " elements", TimeSpan.FromTicks(times.Min()).TotalMilliseconds,
                             TimeSpan.FromTicks(times.Max()).TotalMilliseconds,
                             TimeSpan.FromTicks((Int64)Math.Round(times.Average())).TotalMilliseconds);
        }
        finally
        {
          queue.Delete();
        }
    
        return null;
      }
    

    The PreTestExecute puts the 1000 items on the queue with 2048 bytes each.

    And this is what happens in the ItemTest method for this test:

      Boolean done = false;
      public override bool ItemTest(long itemCurrent, long itemCount, CloudQueue queue)
      {
        if (done)
          return false;
    
        CloudQueueMessage[] messages = null;
    
        while ((messages = queue.GetMessages((Int32)itemCount).ToArray()).Any())
        {
          foreach (var m in messages)
            queue.DeleteMessage(m);
        }
    
        done = true;
    
        return true;
      }
    

    I don't what I'm doing wrong, same code, same connection string and I got these resuts.

    Any idea?


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Wednesday, April 27, 2011 2:46 PM

Answers

All replies

  • Just an experiment, increase the size of the worker role vm.

    When you run locally, you're not bound to the hosted vm's size/resource constraints. So if you're being throttled due to CPU/bandwidth, it should be visible if you increase from the default small size to a medium size.

    You may also check to see if your application service and storage services are deployed into the same datacenter. Just in the off-chance they aren't. :)

    Wednesday, April 27, 2011 2:55 PM
    Moderator
  • Yes, both are deployed in the same affinity group.

    Well, I don't expect the VM size has something to do here, the test is very very simple, it is just sending HTTP calls to a server. Unless Azure works on a 386 :D Anyway, I'll give it a try if nothing else works.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Wednesday, April 27, 2011 3:04 PM
  • Yes, both are deployed in the same affinity group.

    Well, I don't expect the VM size has something to do here, the test is very very simple, it is just sending HTTP calls to a server. Unless Azure works on a 386 :D Anyway, I'll give it a try if nothing else works.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    ;Same affinity group doesnt mean that you have roles and storage in the same datacenter! Try to run storage and roles in exactly the same datacenter. A got no problem in Dublin datacenter.
    • Edited by cloudikka Wednesday, April 27, 2011 4:08 PM I was wrong
    Wednesday, April 27, 2011 3:23 PM
  • I see that they share the same affinity group and the same Region (West Europe), how can I chose the datacenter?


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Wednesday, April 27, 2011 3:39 PM
  • If both is in West Europe it is in Amsterdam datacenter. So problem will be anywhere else. 
    Wednesday, April 27, 2011 3:53 PM
  • -- Same affinity group doesnt mean that you have roles and storage in the same datacenter!

    This is incorrect. An affinity group is associated with a datacenter (i.e. region).

    Wednesday, April 27, 2011 4:03 PM
    Answerer
  • Yeah.....Neil is right. But I remember I was able to create my own affinity groups and select datacenters into it in past. Or I am totally mistaken, Neil?
    Wednesday, April 27, 2011 4:06 PM
  • It could be that your requests are failing initially--perhaps they are throttled if you generate too many requests--and then they succeed after a retry. However, the retries occur after a delay so that would slow things considerably. For troubleshooting, you could try disabling the retry policy in your CloudQueueClient and then check for any exceptions thrown by your requests.

    Regarding Brent's suggestion about VM size, the bandwidth available to your role is proportional to the VM size, so that could indeed make a difference if you are limited by bandwidth. However, I doubt that the very large difference you observe between your computer and the Azure role could be attributed to bandwidth, unless your network connection is that fast.

    Wednesday, April 27, 2011 4:10 PM
  • You should look at the AzureScope website. This contains sample code and results for performance testing with Azure Queue (as well as Table and Blob).

    Were I repeating this test I would refactor the code to simplify it and clarify precisely what is being measured. That being said, Windows Azure is a shared system for which there can be temporary performance anomalies. For example, all cloud providers overcommit network capacity (i.e. don't provide enough capacity for everyone to simultaneously get maximum capacity). This is particularly true of an extra-small instance which has a very limited bandwidth - and, moreover, has a performance characteristic below that of the typical laptop. You pay for what you get - as they say.

    Wednesday, April 27, 2011 4:11 PM
    Answerer
  • It has something to do with the pass from ticks to timestamp.

    If I save the milliseconds directly, instead save the ticks and try to convert it later on, it looks right. Any idea why this could happen?

    Cheers.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Friday, April 29, 2011 3:46 PM
  • It has something to do with the pass from ticks to timestamp.

    If I save the milliseconds directly, instead save the ticks and try to convert it later on, it looks right. Any idea why this could happen?

    Cheers.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Note Note

    Stopwatch ticks are different from DateTime.Ticks. Each tick in the DateTime.Ticks value represents one 100-nanosecond interval. Each tick in the ElapsedTicks value represents the time interval equal to 1 second divided by the Frequency.

    Quoted from http://msdn.microsoft.com/en-us/library/system.diagnostics.stopwatch.elapsedticks.aspx.


    • Marked as answer by vtortola Tuesday, May 03, 2011 10:49 AM
    Monday, May 02, 2011 8:59 AM
  • I didn't know that! Thanks a million :)

    I had another issue with Stopwatch and Hyper-V today, I will handle with care that class from now on.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Tuesday, May 03, 2011 10:50 AM