none
Handling Cleanup jobs using Azure storage Queues

    Question

  • In order to process cleanup jobs, that will run for every 8 hours, currently we have implemented as :

    1. Create scheduled Job using Azure Scheduler that puts message in storage queue when it is triggered.
    2. Implement client in such a way that it will poll continuously and process whenever it receives message. Sample implementation of client is :

       while (!CancellationToken.Value.IsCancellationRequested)    
              {
      
               var message = await client.GetMessageAsync();
      
                  if (message != null)
                  {
                      // process the message
                  }
      }
      

    But the problem is we are waiting indefinetly even we know that we will get messages only after 8hours and also as per documentation, every attempt to read message from queue will incur cost.

    How to optimize this in such a way that listeners will be spawned up on the fly for every configurable time instead of continuous loop?

    Monday, July 20, 2015 4:48 PM

All replies

  • Hi,

     A well-known technique for solving this is using a self-adjusting delay in the queue polling algorithm. For example, one could poll the queue and if a message exists, proceed. If no message is present, it sleeps for one second. If the second time no message is present, it sleeps for 2 seconds, then 3 seconds, and so on until the sleep period reaches a maximum limit you have set. If messages are present at a given time, this number of seconds to sleep is reset. By doing this, the amount of transactions is drastically decreased while message delivery is still relatively fast. Of course this is a simple algorithm. A more advanced version of this is available by the name of truncated exponential back off, but we will not be using that for this scenario. If you are interested in this technique: it’s used by the TCP protocol and described on Wikipedia (http://en.wikipedia.org/wiki/Retransmission_(data_networks).
     Refer : http://www.developerfusion.com/article/120619/advanced-scenarios-with-windows-azure-queues/
     
     Let us know if this helps.

    Regards,
    Nithin Rathnakar

    Tuesday, July 21, 2015 1:30 PM
  • Thanks for the reply.

    I am thinking of this approach.

    1. Expose an REST service on one endpoint of existing service.

    2. Create scheduled job in such a way that it will call the hosted REST service in configured time.

    3. During handling of REST call, we will spawn up listener and continue reading for 'X' minutes.

    Reason for selecting this is we know that background jobs will be triggered every 8 hours . Also doing backoff mayn't be applicable.

    Any Thoughts??

    Thanks

    RaghuNandan


    Tuesday, July 28, 2015 3:22 PM
  • Any suggestions on this ?
    Friday, August 28, 2015 10:43 AM