none
Blob storage is failing with 409 and then retrying (without any retry code) and succeeding.

    Question

  • This started around 7th April 2017.
    Our web api receives some data and saves it to blob storage, this has been working for a while.
    I got a message from application insights saying that one of our dependencies was failing.

    The failure rate is 50%, it appears this is what is happening.

    1. A request comes in.
    2. The data is put to blob storage
    3. This fails with a 409
    4. The file is put again and succeeds.

    I have looked through our code and there is no retry logic and any failures should bubble up and fail the request.

    Monday, April 24, 2017 7:53 AM

Answers

  • A 409 response is expected in some cases. App insights is simply monitoring the request status without knowing whether the status is expected or not.

    For example, you could be using CloudBlobContainer.CreateIfNotExists. Since .NET client version 8.0, there would be a 409 response when the container already exists. The storage client correctly interprets the 409 response and does not surface an exception. If this is the case, you can cut down on the number of unnecessary calls to CreateIfNotExists to increase the success rate in app insights.

    • Marked as answer by RichRubie Monday, May 8, 2017 9:19 AM
    Friday, May 5, 2017 3:59 PM

All replies

  • Update - this started Thursday 6th April at about 11am (uk)
    Monday, April 24, 2017 7:57 AM
  • Hi @RichRubie,

    Thanks for reporting your issue. Would you please provide via email (ascl@microsoft.com) the following information so that we can track down the root-cause:

    - Storage Account Name

    - Request ID

    - Timestamp

    Thanks,

    Elham


    Tuesday, April 25, 2017 7:11 PM
  • I have sent the email.
    Wednesday, April 26, 2017 9:44 AM
  • Hi @Richie,

    Thanks for providing the information. Given the details you have shared (adjusting the timestamp), I was not able to find anything unusual in the logs. And this is mainly the reason we usually ask for request IDs. You can either use fiddler if the failure is repro-able to capture the traces or try enabling Storage Analytics logs(https://docs.microsoft.com/en-us/azure/storage/storage-analytics) and find the failed request IDs. Once you have some failed IDs along with exact timestamps(UTC), we will be able to look into the failures and provide more details/solutions.

    Thanks,

    Elham

    Friday, April 28, 2017 5:13 PM
  • A 409 response is expected in some cases. App insights is simply monitoring the request status without knowing whether the status is expected or not.

    For example, you could be using CloudBlobContainer.CreateIfNotExists. Since .NET client version 8.0, there would be a 409 response when the container already exists. The storage client correctly interprets the 409 response and does not surface an exception. If this is the case, you can cut down on the number of unnecessary calls to CreateIfNotExists to increase the success rate in app insights.

    • Marked as answer by RichRubie Monday, May 8, 2017 9:19 AM
    Friday, May 5, 2017 3:59 PM
  • Thanks, i do call CreateIfNotExists. 
    Monday, May 8, 2017 9:19 AM