Queue getmessage fails - some messages only.

Question

I have a system that uses queues to process messages from IOT devices. It has worked fine for years. A few days ago I became aware that some messages were not being processed. I looked in the queue and found a few messages sitting there with high dequeue counts. Debugging the process reveals:

• The CloudQueue.Getmessage function returns a null result.
• The message at the top of the queue disappears for a while and reappears with its dequeue count incremented as if it has been read.
• Attempts to Peek the messages also return a null result.
• Some messages are processed first time with no problem; are dequeued and deleted normally.
• If I leave the process running, the same few messages rattle around, with their dequeue counts going into the thousands, while other messages are being processed normally.
• The messages that won't be read contain valid data. I can see no difference between them and the ones that are successfully read.

Any clues would be appreciated.

Thanks in advance,

Rob

Answer 1

Apologies for the delay!

Just for clarification: Have you referred you the suggestion mentioned in this article?

If possible, can you restart the service and check for the status again.

Additional information: Regarding putting the message back in the queue, you don't have to do anything special. It is a feature offered by Storage Queues. When you dequeue a message (GET Message in Azure Storage lingo), the message becomes invisible for certain amount of time and if not deleted by the process which dequeued it will become visible again and can be picked up by another process.

So when you dequeue the message, check the time and if it is not the right time you don't do anything. However please make sure that once the message is processed, you delete that message otherwise it will be picked up again.

One more thing you could do is when you dequeue the message and find that it is not the right time to process that message, you update that message and set its visibility timeout property to a value that will make the message visible again closer to the processing time. For example, you dequeued the message at 18:00 and found that this message needs to processed at 19:00. In this case you will update the message and set its visibility timeout to 50 minutes (or a value greater than 30 minutes as 30 minutes is the schedule for your web job). What this will do is ensure that when your webjob runs at 18:30, this message will not be picked up by the web job because the message will only become visible at 18:50.

You can read more about updating message here: https://docs.microsoft.com/en-us/rest/api/storageservices/update-message and about dequeuing messages here: https://docs.microsoft.com/en-us/rest/api/storageservices/get-messages.

If the issue persists, I would request you to share the code.
Kindly let us know if the above helps or you need further assistance on this issue.

------------------------------------------------------------------------------------------

Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

Answer 2

Thank you very much for the long reply. I woke up this morning and found the problem. The code was failing to delete some messages so they rattled around in the queue for a week. It had been happening for a while and was causing unexplained processor load in the worker role. One day I need to change it to a web job.

Answer 3

Thanks for the update. Kindly let us know if you need any further assistance.