locked
Azure Storage - Queue Pricing RRS feed

  • Question

  • I have a service that needs to have thousands of tiny messages per second available to a service with multiple threads reading and processing those messages. Both services are in the same Azure application, different worker roles. Does the $0.01 per 10K transactions apply here. Will the Queue handle that volume or should I seek another mechanism.

    Thanks

    Monday, October 11, 2010 1:00 PM

Answers

  • Welcome to the club.

     

    You might want to look into intra role communication using the internal ip endpoints, scaling out to supersized instances till velocity is available, and might also want to explore the use of sql azure as a queue, possibly fanned out.

    Martin

     


    -- My Azure Sites: www.eblizz.com , www.freshfugu.com Blog: martinatsunset.com Twitter: martin_sunset
    Monday, October 11, 2010 8:02 PM
  • As Martin notes SQL Azure may be an option here. It's certainly a cheaper option for session state storage for large scale sites thanks to the lack of a transaction cahrge.

    If you do not require a true async messaging approach with persistence then doing direct WCF over TCP between internal endpoints is a good approach.

    Finally you could use MemCached or even SharedCache (http://www.sharedcache.com/cms/) but that really only makes sense once you scale out to more than the two roles that you note above- i.e. with just two roles a distributed cache doesn't make any more sense than direct WCF comms.


    Chris Auld - Intergen Limited and MedRecruit Locum Jobs
    Microsoft Regional Director
    Microsoft MVP
    • Marked as answer by Yi-Lun Luo Friday, October 15, 2010 9:33 AM
    Thursday, October 14, 2010 1:31 AM

All replies

  • If you have your storage account in the same data center/region as your services(worker roles), then data transfer between your worker role and queue service is free ($0.10 in / $0.15 out / GB). But transactions pricing are applied as $0.01 / 10K as you mentioned.

    Also while using queues with multiple threads, keep in mind that you have to delete messages after consumption and make sure that message processing is idempotent in nature.

    HTH, 


    Please mark it as answer by clicking on "Propose As Answer", if it helps
    Monday, October 11, 2010 2:30 PM
  • I think to get thousands of messages per second, you'll need to shard across multiple queues.  The scalability target for a single queue is on the order of about five hundred operations per second: http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx.  (And that's "operations," not messages, so if you're doing an enqueue, dequeue, and delete, I believe that's three operations with respect to that scalability target.)

    Monday, October 11, 2010 3:18 PM
  • The queue is only used between two worker roles in the same data center, not used externally at all. If the $0.01/10K is applicable between the two roles then I'm looking at thousands of dollars per month to simply transfer info between two services. That won't work. I'll need some sort of shared memory cache or equivalent to move data between them.
    Monday, October 11, 2010 5:50 PM
  • Welcome to the club.

     

    You might want to look into intra role communication using the internal ip endpoints, scaling out to supersized instances till velocity is available, and might also want to explore the use of sql azure as a queue, possibly fanned out.

    Martin

     


    -- My Azure Sites: www.eblizz.com , www.freshfugu.com Blog: martinatsunset.com Twitter: martin_sunset
    Monday, October 11, 2010 8:02 PM
  • You might consider batching your messages... it's likely to be faster (and certainly less expensive) than creating many many tiny messages.

    Tuesday, October 12, 2010 1:21 AM
  • As Martin notes SQL Azure may be an option here. It's certainly a cheaper option for session state storage for large scale sites thanks to the lack of a transaction cahrge.

    If you do not require a true async messaging approach with persistence then doing direct WCF over TCP between internal endpoints is a good approach.

    Finally you could use MemCached or even SharedCache (http://www.sharedcache.com/cms/) but that really only makes sense once you scale out to more than the two roles that you note above- i.e. with just two roles a distributed cache doesn't make any more sense than direct WCF comms.


    Chris Auld - Intergen Limited and MedRecruit Locum Jobs
    Microsoft Regional Director
    Microsoft MVP
    • Marked as answer by Yi-Lun Luo Friday, October 15, 2010 9:33 AM
    Thursday, October 14, 2010 1:31 AM