none
Seeing Lots of 400 Bad Request Responses from North Central US

    Question

  • We have a service that's been running fine for a year and suddenly we are getting lots of

    Exception Thrown: The remote server returned an error: (400) Bad Request. at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
    at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Commit()
    at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromStreamHelper(Stream source, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
    at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromStream(Stream source, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)

    This is when uploading files that are around 50 mb. Not sure what is going on. We upgraded to the latest WindowsAzure.Storage DLL (version 3.0.2.0) but that hasn't helped. The behavior is intermittent. 

    Any ideas?

    Saturday, January 25, 2014 12:10 AM

Answers

  • Well, we resolved this issue. Turns out that we were getting the following error back as part of the 400:

    InvalidBlobOrBlock


    The specified blob or block content is invalid.

    Somehow some of our blobs had gotten corrupted, although the files themselves were fine. We think there were uncommitted blocks. The fix was to copy the blob contents to a backup blob name, and then call Delete() on the blob and then reupload the blob to its original name. That solved our problem. 

    We also realized that, under the hood, the SDK calls PutBlock if a file is of a certain size even when you call UploadFromStream, FWIW.

    • Marked as answer by atombomb2222 Tuesday, January 28, 2014 9:05 PM
    Tuesday, January 28, 2014 9:05 PM

All replies

  • Incidentally, we thought this might have to do with using UploadFromStream and so we changed our code to use PutBlock but are still seeing this intermittently. Our blocks are only 250 kb.
    Saturday, January 25, 2014 6:08 AM
  • Hi,

    May I know which version Azure SDK were you used? It was unchanged before encountered this error? Based on your descripted, this behavior is intermittent, this means sometimes it works well, sometimes give us error? It is hard to me to reproduce this issue, from my experience, you met 400 bad request, I suggest you to check the network's stability.

    Hope this helps

    Monday, January 27, 2014 5:52 AM
  • We started encountering the error with an earlier version of the Windows Azure Storage. We then upgraded to the latest.

    Network stability should not be an issue because this is a Worker Role hosted in North Central US talking to blob storage, which should be the fastest possible network connection.

    We have even implemented a retry policy that reattempts every 5 seconds for 20 tries and we still will see failures, but again they are intermittent.

    What else can we do? How else can we troubleshoot?

     
    Monday, January 27, 2014 9:26 PM
  • Well, we resolved this issue. Turns out that we were getting the following error back as part of the 400:

    InvalidBlobOrBlock


    The specified blob or block content is invalid.

    Somehow some of our blobs had gotten corrupted, although the files themselves were fine. We think there were uncommitted blocks. The fix was to copy the blob contents to a backup blob name, and then call Delete() on the blob and then reupload the blob to its original name. That solved our problem. 

    We also realized that, under the hood, the SDK calls PutBlock if a file is of a certain size even when you call UploadFromStream, FWIW.

    • Marked as answer by atombomb2222 Tuesday, January 28, 2014 9:05 PM
    Tuesday, January 28, 2014 9:05 PM