none
queue.CreateIfNotExistsAsync() has stopped working

    Question

  • Our worker role has worked - for months. Just stopped working. Something happened after 11/8/2015.  (All the _queueFailCount just added to stop log file from overflowing.)

    Our code:

    CloudQueue queue = _queueClient.GetQueueReference("taskqueue");

    if (_queueFailCount <= 100)
    {
    bool isOK = false;
    try
    {
    await queue.CreateIfNotExistsAsync();  // FAILS HERE
    isOK = true;
    }
    finally
    {
    if (!isOK)
    {
    _queueFailCount++;
    if (_queueFailCount >= 100) { _log.Warn("Worker role queue fail count hit"); }
    }

    }
    }

    The error log:

    <DSErrorTracking xmlns="http://www.usawinewest.net/DSErrorTracking.xsd">
      <Log4Net>
        <Log4NetID>2562</Log4NetID>
        <Application>WorkerRole</Application>
        <Exception>Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (400) Bad Request. ---&gt; System.Net.WebException: The remote server returned an error: (400) Bad Request.
       at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Shared\Protocol\HttpResponseParsers.Common.cs:line 55
       at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.&lt;ExistsImpl&gt;b__1d(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 2462
       at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 306
       --- End of inner exception stack trace ---
       at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs:line 72
       at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndCreateIfNotExists(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 323
       at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.&lt;&gt;c__DisplayClass1`1.&lt;CreateCallback&gt;b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 66
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
       at WineWest.WorkerBusLogic.Persistence.QueueManager.&lt;ProcessMessagesAsync&gt;d__0.MoveNext() in c:\Usr\Source\Repos\WineWestAzureServer\ClassLibraries\WineWest.WorkerBusLogic\Persistence\QueueManager.cs:line 65
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at WineWest.WorkerRole.WorkerRole.&lt;RunAsync&gt;d__0.MoveNext() in c:\Usr\Source\Repos\WineWestAzureServer\WineWest.WorkerRole\WorkerRole.cs:line 82</Message>
        <Thread>10</Thread>
        <TS>5609574</TS>
      </Log4Net>
    </DSErrorTracking>



    Jim Rand

    Monday, November 9, 2015 8:12 PM

All replies

  • Hi Jim,

    Thanks for posting here.

    400 bad request is usually because of a bad name, but can happen from a number of other possibilities. Please check for the storage account name or the container name which is mentioned.

    There's nothing wrong with the code. However, I suggest you to check StorageException's ExtendedErrorMessage hope that helps. It'll be in the StorageException.RequestInformation.ExtendedErrorInformation

    https://msdn.microsoft.com/en-us/library/azure/microsoft.windowsazure.storage.storageextendederrorinformation.aspx

    https://msdn.microsoft.com/en-us/library/azure/microsoft.windowsazure.storage.requestresult.aspx

    Hope this helps you.

    Girish Prajwal

    Tuesday, November 10, 2015 2:27 PM
    Moderator
  • Also, take a look at the Queue service error codes:

    https://msdn.microsoft.com/en-us/library/azure/dd179446.aspx

    General error codes for storage are here:

    https://msdn.microsoft.com/en-us/library/azure/dd179357.aspx

    Wednesday, November 11, 2015 1:37 AM
  • Here is what happened. 

    At the end of October, CloudConfigurationManager was moved from Microsoft.Azure to Microsoft.WindowsAzure.  This was a breaking change that caused the worker role to fail after running quite successfully with no problems.

    After NuGetting the Azure Storage update, the end point for development storage got clobbered. No idea why this would impact running the Worker Role in the Cloud as the real storage account was specified - not development storage.

    Also, the solution would not compile until after I added the using Microsoft.WindowsAzure line.

    Anyway, the Worker role is running correctly again in production.

    New problem. The Azure Storage Emulator on my development machine is causing the spinning icon. Kill it in Task Manger, and the icon stops spinning. While I'm typing here, the icon is not spinning. Move it out of this text box and it spins.

    Please fix this ASAP.

    Thanks.


    Jim Rand

    • Marked as answer by Jim Rand Thursday, November 12, 2015 5:18 PM
    • Unmarked as answer by Jim Rand Thursday, November 12, 2015 5:18 PM
    Wednesday, November 11, 2015 1:30 PM
  • The spinning cursor just stopped spinning. Don't know why, but I'm happy. Consider case closed.  Thanks.

    Jim Rand

    • Marked as answer by Jim Rand Thursday, November 12, 2015 5:20 PM
    • Unmarked as answer by Jim Rand Tuesday, November 17, 2015 3:55 PM
    Thursday, November 12, 2015 5:20 PM
  • Need to open this up again.  The spinning cursor is related to the worker role being unhealthy.  Don't know how to fix this.  See http://stackoverflow.com/questions/32750680/cannot-emulate-c-sharp-worker-role. Suggested fix is way past my ability.

    This is a problem introduced with the Azure SDK 2.7.1.  Tried deleting diagnostics.wadcfgx. That didn't fix it. Emulator still ran, but again with the spinning icon everywhere and reduced responsiveness for all desktop apps.

    Only way to stop the icon is to stop the Compute Emulator.  

    Is it possible to roll back to SDK 2.6 which worked? Are you going to introduce a fix for this?  Is there a simple step-by-step fix to this?

    Thanks.


    Jim Rand

    Tuesday, November 17, 2015 4:03 PM
  • Not an answer - but a search on a problem I was having led to this page.  This bug breaks running a Hive job on a hadoop cluster for Azure.  When I specify a query directly but tell it to run as a file Job, the internal Microsoft.Hadoop.Client.... stuff will return this 400 bad request storage exception - presumably because its using this function internally.  Rolling back to earlier Microsoft.WindowsAzure.Storage (before 6.0) fixes the problem...

    Tuesday, January 5, 2016 3:14 PM
  • The problem I was having was with the emulator / spinning unhealthy icon.  Making the worker role single threaded fixed that problem.  Not an issue in production as the worker role is just doing some nightly processing as a single task.

    queueManager.ProcessMessage() instead of 

    await queueManager.ProcessMessagesAsync(cancellationToken)

    I'll revisit this with version 2.8x at some point.


    Jim Rand

    Tuesday, January 5, 2016 3:58 PM