locked
How to handle “The specified resource does not exist” exception while using entity group transactions to purge WADLogs table RRS feed

  • Question

  • Hi,

    We have a requirement to purge the Azure WADLogs table on a periodic basis. We are achieving this by using Entity group transactions to delete the records older than 15 days. The logic is like this.

    bool recordDoesNotExistExceptionOccured = false;
    CloudTable wadLogsTable = tableClient.GetTableReference(WADLogsTableName);
    
    partitionKey = "0" + DateTime.UtcNow.AddDays(noOfDays).Ticks;
    TableQuery<WadLogsEntity> buildQuery = new TableQuery<WadLogsEntity>().Where(
                                TableQuery.GenerateFilterCondition("PartitionKey",                
                                QueryComparisons.LessThanOrEqual, partitionKey));
    
    while (!recordDoesNotExistExceptionOccured)
    {
        IEnumerable<WadLogsEntity> result = wadLogsTable.ExecuteQuery(buildQuery).Take(1000);
    
        //// Batch entity delete.
        if (result != null && result.Count() > 0)
        {
             Dictionary<string, TableBatchOperation> batches = new Dictionary<string, TableBatchOperation>();
             foreach (var entity in result)
             {
                 TableOperation tableOperation = TableOperation.Delete(entity);
    
                 if (!batches.ContainsKey(entity.PartitionKey))
                 {
                     batches.Add(entity.PartitionKey, new TableBatchOperation());
                 }
    
                 // A Batch Operation allows a maximum 100 entities in the batch which must share the same PartitionKey.                        
                 if (batches[entity.PartitionKey].Count < 100)
                 {
                     batches[entity.PartitionKey].Add(tableOperation);
                 }
            }
    
            // Execute batches.
            foreach (var batch in batches.Values)
            {
                try
                {
                    await wadLogsTable.ExecuteBatchAsync(batch);
                }
                catch (Exception exception)
                {
                    // Log exception here.
    
                    // Set flag.
                    if (exception.Message.Contains(ResourceDoesNotExist))
                    {
                        recordDoesNotExistExceptionOccured = true;
                    }
                    break;
                }
            }
        }
        else
        {                            
            break;
        }
    }

    My questions are:

    1. Is this an efficient way to purge the WADLogs table? If not, what can make this better?
    2. Is this the correct way to handle the "Specified resource does not exist exception"? If not, how can I make this better?
    3. Would this logic fail in any particular case?
    4. How would this approach change if this code is in a worker which has multiple instances deployed?

    I have come up with this code by referencing the solution given here by Keith Murray.


    Wednesday, October 29, 2014 5:26 AM

Answers

  • Hi Nikhil,

    Thanks for your posting!

    I tested your and Keith's code on my side, every thing worked fine. And when result is null or "result.count()<0", the While() loop is break. I found you code had some logic to handle the error "ResourceDoesNotExist" . It seems that the code worked fine. If you always occurred this error, I suggest you could debug your code and find which line of code throw the exception.   

    >> Is this an efficient way to purge the WADLogs table? If not, what can make this better?

    Base on my experience, we could use code (like the above logic code) and using the third party tool to delete the entities manually. In my opinion, I think the code is every efficient, it could be auto-run and save our workload.

     >>Is this the correct way to handle the "Specified resource does not exist exception"? If not, how can I make this better?

    In you code, you used the "recordDoesNotExistExceptionOccured " as a flag to check whether the entity is null. It is a good choice. I had tried to deleted the log table entities, but I used the flag to check the result number. For example, I planed the query result count is 100, if the number is lower than 100, I will set the flag as false, and break the while loop. 

    >>Would this logic fail in any particular case?

    I think it shouldn't fail. But if the result is "0", your while loop will always run. It will never stop. I think you could add "recordDoesNotExistExceptionOccured = true;" into your "else" block.

    >>How would this approach change if this code is in a worker which has multiple instances deployed?

    You don't change anything expect the "else" block. It would work fine on the worker role.

    If any question about this issue, please let me know free.

    Regards,

    Will


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    • Marked as answer by Nikhil Bhawkar Wednesday, November 5, 2014 2:16 AM
    Thursday, October 30, 2014 5:36 AM

All replies

  • Hi Nikhil,

    I'm trying to involve some senior engineers into this topic and it might take some time. Your patience is greatly appreciated.

    Regards,

    Manu Rekhar

    Thursday, October 30, 2014 3:15 AM
  • Hi Nikhil,

    Thanks for your posting!

    I tested your and Keith's code on my side, every thing worked fine. And when result is null or "result.count()<0", the While() loop is break. I found you code had some logic to handle the error "ResourceDoesNotExist" . It seems that the code worked fine. If you always occurred this error, I suggest you could debug your code and find which line of code throw the exception.   

    >> Is this an efficient way to purge the WADLogs table? If not, what can make this better?

    Base on my experience, we could use code (like the above logic code) and using the third party tool to delete the entities manually. In my opinion, I think the code is every efficient, it could be auto-run and save our workload.

     >>Is this the correct way to handle the "Specified resource does not exist exception"? If not, how can I make this better?

    In you code, you used the "recordDoesNotExistExceptionOccured " as a flag to check whether the entity is null. It is a good choice. I had tried to deleted the log table entities, but I used the flag to check the result number. For example, I planed the query result count is 100, if the number is lower than 100, I will set the flag as false, and break the while loop. 

    >>Would this logic fail in any particular case?

    I think it shouldn't fail. But if the result is "0", your while loop will always run. It will never stop. I think you could add "recordDoesNotExistExceptionOccured = true;" into your "else" block.

    >>How would this approach change if this code is in a worker which has multiple instances deployed?

    You don't change anything expect the "else" block. It would work fine on the worker role.

    If any question about this issue, please let me know free.

    Regards,

    Will


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    • Marked as answer by Nikhil Bhawkar Wednesday, November 5, 2014 2:16 AM
    Thursday, October 30, 2014 5:36 AM
  • Hi Will,

    Thanks for the reply. I tried debugging my code, but it seems to work without an exception. I guess the exception occurs for some specific scenario. Will let you know if I can lay my finger on that.

    Also, regarding the suggestion to add recordDoesNotExistExceptionOccured = true; to my else block, I already have a "break;" in that block which will cause the while loop to be terminated in case the result.Count() is null or less than equal to zero. So I guess this can be avoided.

    In the multi-instance scenario, will I not be required to avoid overlap among the instances? I would not want both the worker instances to try to delete the same records. I this case, the role which starts deleting first will have records to delete, but the other instance will try to delete records which are already deleted, hence causing the "Specified resource does not exist" exception. Please correct me if my understanding is wrong.

    Thursday, October 30, 2014 6:58 AM
  • Hi Nikhil,

    If the multi-instance or concurrency request, you need use the azure table ETag.

    >>In the multi-instance scenario, will I not be required to avoid overlap among the instances?

    In this scenario, to use optimistic concurrency and to check if another process modified an entity since you retrieved it from the table storage service, you can use the ETag value you receive when the table service returns an entity.

    I recommend you refer to this document (http://azure.microsoft.com/blog/2014/09/08/managing-concurrency-in-microsoft-azure-storage-2/) and this docs(http://stackoverflow.com/questions/9488702/concurrent-operations-on-windows-azure-table-storage).

    Regards,

    Will


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, October 31, 2014 7:37 AM