locked
How to check if Message already exists in the queue and if message is processing currently RRS feed

  • Question

  • Hi everyone

    I am new to Azure and worked on adding messages to the queue through workerrole1. Worker role 2 pulls them out from queue and processing them and de-queing them.

    Worker role1 runs method gets called after every 10 seconds and puts messages in queue

    CloudQueueMessage

    message = newCloudQueueMessage(oAzureWorker.WorkerInstanceOf + "_"+ oAzureWorker.AgentId.ToString() + "|"+ ExecutionId.ToString());

                                    queue.AddMessage(message);

    Worker role2 runs method gets called after every 10 seconds too and checks the queue like this

    foreach

    (CloudQueueMessagemessage inqueue.GetMessages(20, TimeSpan.FromMinutes(5)))

    {

    // Process all (20) messages in less than 5 minutes, deleting each message after processing.

    // Process message

    queue.DeleteMessage(message);

    }

    Following are my questions

    1) How do I check in worker role1 if the message is already in queue, Because I don't want to queue it back again if its not yet processed and is in the queue already

    2) How do I check in worker role1 if the message is currently processing. Because I don't want to queue it back again.

    3) How do I make sure that ALL the messages get processed in the order they are inserted. I know Queue is FIFO, but I know if the message gets delayed in processing another instance can pick it up, even if it gets picked up by another instance, I want to make sure that the order remains.

    Right now the instances of both these worker roles are 1, in the future when we increase them, I don't want them to queue the same messages multiple times or queue them if the message is already in process mode.

    Tuesday, December 30, 2014 8:04 PM

Answers

  • You may need to review the design pattern used here. To ask Worker Role #1 to monitor and check if a message has been processed or not puts too much burden on it. An alternative is for Worker Role #2, whatever instance it is, to lock and process the message. If successful, remove it; if not, do something about it (like putting aside for special processing) or leave it in the queue for later processing again.

    Frank

    • Marked as answer by Sarah2005 Monday, January 5, 2015 5:50 AM
    Tuesday, December 30, 2014 8:44 PM
  • Sarah,

    1 >> Shared entity meaning e.g. the database using which your worker role 1 instances can talk to each other. something like - worker role 1 instance 1 adds up a record to the table in the database after pushing message on queue, later worker role 1 instance 2 looks up to the same table first and tries to find out the message to be pushed on the queue - if it finds it - it simply won't push the message on queue, which will avoid the case of duplicates.

    2>> The way GetMessage works by design is - it hides the message from the queue for mentioned time to make sure that it is processed only once, if message doesn't get processed by that time (say due to some exception) it makes the message visible again on the queue, so worker role instances picks it up again and tries to process. Note again - it makes visible on the queue only if it doesn't get processed in mentioned time.


    Bhushan | http://www.passionatetechie.blogspot.com | http://twitter.com/BhushanGawale

    • Marked as answer by Sarah2005 Monday, January 5, 2015 5:49 AM
    Friday, January 2, 2015 5:04 AM

All replies

  • You may need to review the design pattern used here. To ask Worker Role #1 to monitor and check if a message has been processed or not puts too much burden on it. An alternative is for Worker Role #2, whatever instance it is, to lock and process the message. If successful, remove it; if not, do something about it (like putting aside for special processing) or leave it in the queue for later processing again.

    Frank

    • Marked as answer by Sarah2005 Monday, January 5, 2015 5:50 AM
    Tuesday, December 30, 2014 8:44 PM
  • Hi Frank

    So what your saying is in Worker role 2, if Instance 1 is running local the message say message1, process it and then unlock the message and delete it. When instance 2 is running at the same time it will local the message 2 and process it and delete it.

    What do you mean by lock, is it any different than getting messages and making them invisible for 5 minutes using the following code?

    queue.GetMessages(20,

    TimeSpan.FromMinutes(5)))

    And how can I lock a message?

    Thanks much

    -Sarah

    Wednesday, December 31, 2014 8:00 AM
  • Hi Sarah,

    I agree to the Frank's suggestion. Why you need to burden the worker role 1 to check if the message really sits on the queue or not? You can do this simply in your code before pushing it on queue instead querying queue.

    All you need to do on worker role 1 is - push the message on the queue and forget as the entire queue design in azure is designed from asynchronous processing.

    About worker role 2 - Use the GetMessage method which hides retrieved message's from other clients and hence makes sure that only one client is processing it at a time. If processing is successful - delete the message. if it is not - the message will be visible anyways after the mentioned time provided in the GetMessage method.

    I agree that when you will increase number of instances of your worker role 1 which might insert duplicates in the queue - in that case - you might need to introduce the shared entity (like database) and let all instances communicate through it to avoid the duplication of messages on queue. 


    Bhushan | http://www.passionatetechie.blogspot.com | http://twitter.com/BhushanGawale

    Wednesday, December 31, 2014 9:43 AM
  • Hi Bhushan

    I have two questions

    1)

    Can you please elaborate this a little bit more "- in that case - you might need to introduce the shared entity (like database) and let all instances communicate through it to avoid the duplication of messages on queue. "

    How can I go about doing this?

    2)

    When you say " if it is not - the message will be visible anyways after the mentioned time provided in the GetMessage method "

    This sounds like the FIFO attribute of a queue will not be followed correct? if the message does not get processed within the time period, that means next messages will get processed before that one appears again to be processed.

    Thanks very much for your help and clearification on this

    -Sarah

    Thursday, January 1, 2015 11:35 PM
  • Sarah,

    1 >> Shared entity meaning e.g. the database using which your worker role 1 instances can talk to each other. something like - worker role 1 instance 1 adds up a record to the table in the database after pushing message on queue, later worker role 1 instance 2 looks up to the same table first and tries to find out the message to be pushed on the queue - if it finds it - it simply won't push the message on queue, which will avoid the case of duplicates.

    2>> The way GetMessage works by design is - it hides the message from the queue for mentioned time to make sure that it is processed only once, if message doesn't get processed by that time (say due to some exception) it makes the message visible again on the queue, so worker role instances picks it up again and tries to process. Note again - it makes visible on the queue only if it doesn't get processed in mentioned time.


    Bhushan | http://www.passionatetechie.blogspot.com | http://twitter.com/BhushanGawale

    • Marked as answer by Sarah2005 Monday, January 5, 2015 5:49 AM
    Friday, January 2, 2015 5:04 AM
  • Thanks so much Bhushan, yes I am using database to do that. Based on your advice I made the changes and I think it looks good. Thanks so much

    Is there a way to find out which instance of the worker role picked up a certain message? I want to be able to save that information in my sql tables, or is that information saved in the log tables? which table would that be. This will help me with the concurrency issue.

    Thanks much

    -Sarah

    Monday, January 5, 2015 5:49 AM
  • Thanks Sarah, Glad it helped you!!

    Yes, there is a way to get the current role instance name / id. You can use following and log it

    RoleEnvironment.CurrentRoleInstance.Id
    RoleEnvironment.CurrentRoleInstance.Role.Name


    Bhushan | http://www.passionatetechie.blogspot.com | http://twitter.com/BhushanGawale

    Monday, January 5, 2015 6:46 AM
  • Thanks Bushan again for your help.

    Where do I need to add these lines of code. In the Onstart process of the role? and where can I see the results of it, in which log table?

    So far I am seeing random results only in this table "WADLogsTable" and that too I am confused, how to get the results there. I am using the diagnosticmonitor in the onstart process.

    Friday, January 9, 2015 4:27 AM