locked
Azure Storage bus scale message retrieval count from queue RRS feed

  • Question

  • I have a web service hosted in Azure.The service receives data from multiple users(50,000 requests) and it stores data to a table in Azure. Now I am facing database concurrency issues as the table gets locked with huge incoming data.

    I am exploring on the possible solutions. One which I came across is the Azure Storage Queue.Wherein I can store the incoming data to the queue and then push the data to table.

    I need clarifications on few items which will help me to confirm on the approach which will help me.

    1. Can I use Web jobs to read the messages from queue and write to table. If yes, how can I scale the web jobs to read multiple messages from queue and write to table.

    2.I read that maximum size for retrieval of messages from queue is 32. This will result in a poor performance as I am restricted to 32 messages to be pushed to table. How can I scale to write more messages to the table from queue.

    Note:The table should be updated with the latest data so we want the queue processing to  be faster.


    Will Storage queue serve the purpose or is there any other approach which I should consider.

    Thanks in Advance!
    • Moved by VeeraGiri Babu Friday, August 10, 2018 11:21 AM Better suited here than Azure App Service - Web Apps
    Friday, August 10, 2018 5:52 AM

All replies

  • Azure supports two types of queue mechanisms:

    Storage queues, which are part of the Azure storage infrastructure, feature a simple REST-based GET/PUT/PEEK interface, providing reliable, persistent messaging within and between services.

    Service Bus queues, are part of a broader Azure messaging infrastructure that supports queuing as well as publish/subscribe, and more advanced integration patterns.
    While both queuing technologies exist concurrently, Storage queues were introduced first, as a dedicated queue storage mechanism built on top of Azure Storage services. Service Bus queues are built on top of the broader messaging infrastructure designed to integrate applications or application components that may span multiple communication protocols, data contracts, trust domains, and/or network environments.


    For Azure web jobs, suggest you to refer the suggestions outlined in the SO thread.


    For more information, suggest you to refer Storage queues and Service Bus queues - compared and contrasted


    Let me know if there are still any questions I can help with.

    -----------------------------------------------------------------------------------------------

    If this answer was helpful, click “Mark as Answer” or “Up-Vote”. To provide additional feedback on your forum experience, click here

    • Proposed as answer by VeeraGiri Babu Friday, August 10, 2018 12:06 PM
    Friday, August 10, 2018 12:06 PM
  • Checking in to see if the above answer helped. Let me know if there are still any additional issues I can help with.

    Tuesday, August 14, 2018 11:39 AM
  • Hi,

    Unfortunately I think I cannot use web job as my service is in Java and I cannot use the web job sdk .

    Is there any other alternate way for processing messages from queue for java services which will fetch incoming messages and store them in table.

    Thanks!

    Thursday, August 16, 2018 9:46 AM
  • I would suggest you to refer How to use Queue storage from Java.

    Friday, August 17, 2018 7:21 AM
  • Just checking in if you have had a chance to see the previous response.
    Sunday, August 19, 2018 2:59 PM