none
Azure queues slow in simple demo.

    Question

  • Hi,

    I'm doing a simple application that represents a execution flow:

    Basically it works like this:

    • I create a message with an ID and associate that ID with a StopWatch. Repeat this as many times as items.
    • I start the StopWatch and send the message to Q1. This is done in parallel in 32 batches.
    • A separate thread reads messages from Q1 every 150ms, waits 100ms simulating some processing per each message and send the message to Q2, this is done in parallel.
    • Another separate thread reads messages from Q2 every 150ms, gets message ID, find the timer and stops it. Increment the number of processed messages.
    • When all messages have been processed, the app shows the Minimun, Maximun and Average time that a message lasts to cross the full circuit.

    At this moment, using a real Azure Storage, the average is 60 seconds! I think that is too much, so I think I'm doing something wrong with the loops that get messages, with the ammount of messages or with the whole approach.

    This is the code in TXT, and I have upload a ZIP with the solution here

    Any idea what is the problem?

    Cheers.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Friday, April 29, 2011 4:03 PM

Answers

  • Here are a few suggestions to try (if you have not already):
    1> turn Nagle off (see http://blogs.msdn.com/b/windowsazurestorage/archive/2010/06/25/nagle-s-algorithm-is-not-friendly-towards-small-requests.aspx)
    2> without a warm up phase, set Min Threads for the ThreadPool using ThreadPool.SetMinThreads
    3>set the max connection to something larger than default of 2 (it looks like you have tried this but listing it here for completeness)
    4> turn off retries just for this test to ensure that storage client library is not silently retrying a failed request. You can handle all exceptions and make note of the number of failures if you are interested in tracking it

    In the near future, we will be posting our queue performance similar to this that we did for tables. However,  we do see that we get up to 500 single message requests/sec for a single queue.


    Thanks,
    Jai


    • Marked as answer by vtortola Thursday, May 26, 2011 10:40 AM
    Saturday, April 30, 2011 7:26 PM

All replies

  • Since you are doing a lot of multithreading with the queue you should look at item 1 in the performance tips sticky post on the forum - the one that advises that "Default HTTP Connections is set to 2."

    The scalability targets for queues are described here. For a single queue the target is "up to 500 messages per second."

    Friday, April 29, 2011 7:10 PM
    Answerer
  • Since you are doing a lot of multithreading with the queue you should look at item 1 in the performance tips sticky post on the forum - the one that advises that "Default HTTP Connections is set to 2."

    Fantastic. Now the average is around 30 seconds, but it's still too much.

     

    The scalability targets for queues are described here. For a single queue the target is "up to 500 messages per second."

    But, how? I've been doing tests for a while and not even close to 50 per second, actually close to 20 per second. I'm using one "reading thread" and one "writing thread" anyway, I guess that with more of them the performance will be better, but I'm trying to figure out which is the optimum delay between reads. 

    What more could cause the messages flow so slow in that circuit?

    Cheers.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Friday, April 29, 2011 8:45 PM
  • It is probably worth stripping out all the reads that simulate work to see how much time is spend doing "work" as opposed to processing messages. Indeed, I would be tempted to use synchronous calls to see if the multithreading is somehow getting in the way. You might also check out this String constructor.
    Friday, April 29, 2011 9:21 PM
    Answerer
  • Thanks for the String tip!

    I've created a version using ConcurrentQueue's ,  and removing all the Thread.Sleep the average time is 523ms.

    In the Azure queue version, removing all the Thread.Sleep the average time is 26 seconds.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Saturday, April 30, 2011 10:16 AM
  • You have not mentioned, whether you are running your application from on-premise or from Azure role. If you are currently doing this test from on-premise and have any plans to do this from Azure platform in future, you might want to deploy your test application to Azure by choosing same data center for both computing and storage. Since you have lots of reads and writes to Queue, this would elemenate network latency and give much better performance.

     

    HTH.


    Please mark it as answer by clicking on "Propose As Answer", if it helps. My Blog : http://dotnetizen.blogspot.com
    Saturday, April 30, 2011 12:30 PM
  • I suggest you try out the AzureScope sample code in your setup to see the results you get. Its benchmarking reports far higher throughput that you are getting.
    Saturday, April 30, 2011 5:22 PM
    Answerer
  • Here are a few suggestions to try (if you have not already):
    1> turn Nagle off (see http://blogs.msdn.com/b/windowsazurestorage/archive/2010/06/25/nagle-s-algorithm-is-not-friendly-towards-small-requests.aspx)
    2> without a warm up phase, set Min Threads for the ThreadPool using ThreadPool.SetMinThreads
    3>set the max connection to something larger than default of 2 (it looks like you have tried this but listing it here for completeness)
    4> turn off retries just for this test to ensure that storage client library is not silently retrying a failed request. You can handle all exceptions and make note of the number of failures if you are interested in tracking it

    In the near future, we will be posting our queue performance similar to this that we did for tables. However,  we do see that we get up to 500 single message requests/sec for a single queue.


    Thanks,
    Jai


    • Marked as answer by vtortola Thursday, May 26, 2011 10:40 AM
    Saturday, April 30, 2011 7:26 PM
  • Hi Jai,

    The first point didn't make to much effect, but the second point has improved the result a lot, now it gives and average of around 17 seconds. Third and fourth points were already done. Thanks!

    However, we do see that we get up to 500 single message requests/sec for a single queue.

    Could you provide a runnable code example that gets that throughput? Obviously I'm doing something wrong here, and I want to know what. With 500msg/s I would expect a message do this circuit in less than a couple of seconds. What is the delay between a message is inserted in the queue and HTTP call can read it?

    @Neil, actually the original form of this test is in a worker role :D I just put the code in a console application in order to enable anyone to test it, but when I be back at the office, I will add the new improvements and see how it does, although considering that right now it's still 16 seconds, I don't expect miracles :D

    Regards.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Sunday, May 01, 2011 7:37 PM
  • For the record, running on the cloud on a worker role the average time for this test is now around 10 seconds :)
    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Tuesday, May 03, 2011 10:48 AM
  • Hi vtortola,

    So the demo is fast now and the issue is resolved?

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Friday, May 06, 2011 5:18 AM
  • Hi Wenchao,

    Well, although the test now is almost 6 times faster thanks to Jai Haridas, still is far far far away from the 500msg/s that a queue is supposed to be able to process. It will be helpful know how do you peope get that throughput, could you point me to the code that demonstrate that performance?

    Thanks a milliion!


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Monday, May 09, 2011 8:56 AM
  • still is far far far away from the 500msg/s that a queue is supposed to be able to process.



    haven't checked all of this thread, but remember it's not sayint 500msg/s, but 500 storage transactions/s on a single partition (== a single queue). Given that you read and write and delete messages from the queue, thats are 3 transactions already so your throughput counted in messages is less.
    Monday, May 09, 2011 9:21 PM
  • Hi Valeriano,

    Queue Storage and ConcurrentQueue are different. At least using Queue Storage, a message needs to be serialized and transferred through the internal network of a data center. So I think for testing 1000 items, 10 seconds is really acceptable.

    Or do you have another thought?

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Tuesday, May 10, 2011 2:39 AM
  • Yes I think you're right. Sorry for the delay replying.

     

    Regards.


    .: Valeriano Tórtola MCTS WPF :.: http://www.vtortola.net :.
    Thursday, May 26, 2011 10:40 AM