sticky
Troubleshooting deployments stuck in Initializing/Busy/Stopping

    General discussion

  • Some time ago we published a troubleshooting guide as part of the MSDN documentation (available here: http://msdn.microsoft.com/en-us/library/gg465402.aspx ). For your convenience, we’re also posting the article here. If you feel we missed something, please let us know. We hope that this post will help you debug common issues with deployments.

     

    If after you deploy a Windows Azure service your role instance fails to start, or it cycles between the initializing, busy, and stopping states, one of the following problems may be present. This topic lists some of the common underlying causes of deployment state problems, and offers troubleshooting tips to help you resolve the issue.

     

    Role Relies on Missing Runtime Dependencies

    If your role relies on any assembly that is not part of the .NET 3.5 Framework or the Windows Azure managed library, you must explicitly include that assembly in the deployment package. Services running in the Windows Azure environment have programmatic access to the classes of the .NET 3.5 Framework and the Windows Azure library, but not to other assemblies, including other assemblies provided by Microsoft. Some examples of third-party assemblies include MVC, WCF RIA Services, Unity, Microsoft Report Viewer, and so on.

     

    Before you build and package your service, verify that:

     

    ·          The Copy Local property is set to True for each referenced third-party assembly, if you are developing and building your service in Microsoft Visual Studio.

    ·          The web.config file does not reference any unused assemblies in the <compilation> element.

     

     

    Service Configuration Includes Misconfigured DiagnosticsConnectionString Setting

    If your role uses Windows Azure Diagnostics, then the DiagnosticsConnectionString configuration setting must be specified. This setting should specify an HTTPS connection to your storage account in the cloud. Verify that:

     

    ·          The DiagnosticsConnectionString configuration setting is specified in the service configuration file.

    ·          You have modified the value of the configuration setting to point to a valid storage account in the cloud. By default, this setting points to the development storage account, so you must explicitly change this setting before you deploy your service.

    ·          The connection string is specified in the following format (note that the protocol must be specified as HTTPS):

     

    DefaultEndpointsProtocol=https;AccountName=myaccountname;AccountKey=myaccountkey

     

    See Configuring Connection Strings for more information.

     

    If you are developing your service using the Windows Azure Tools for Microsoft Visual Studio, you can use the property pages to construct this setting. See Configuring the Cloud Service for more information.

     

    Role Returns from Run Method

    If your code overrides the RoleEntryPoint.Run method, it should sleep indefinitely. The Run method is never expected to return. If it does, the role will be restarted.

     

    Role References Incorrect Platform Version of an Assembly

    The Windows Azure Guest OS and hosting processes are 64-bit. You should not reference any 32-bit assembly from your role.

     

    Role Requires Admin Privileges

    Windows Azure currently does not support administrator access. Any method call that requires admin privileges will fail.

     

    Exported Certificate Does Not Include Private Key

    When you export a certificate to associate with a role, its private key must be exported along with the certificate. If you use the Windows Certificate Manager to export the certificate, be sure to select the Yes, export the private key option.

     

    Role Throws Unhandled Exceptions During Initialization

    Any exceptions thrown by the methods of the RoleEntryPoint class – OnStart , OnStop , and Run – are unhandled exceptions. If you have overridden these methods, and your role is cycling between the initializing, busy, and stopping states, your code may be throwing an exception that is unhandled.

     

    We will continue to update the troubleshooting guide on MSDN (updated) as well as this post if we discover new causes for the problem.

     

    Thank you and looking forward to your feedback!

    Windows Azure Developer Platform Team

    Tuesday, May 25, 2010 1:16 AM
    Moderator

All replies

  • With the June 2010 release of the Windows Azure Tools, you can now debug these issues through the historical debugging capabilities of IntelliTrace (requires .NET 4 and Visual Studio 2010 Ultimate).

    In the following blog post I go through many of the common problems listed above and show you how to debug them using the IntelliTrace for the cloud feature: http://blogs.msdn.com/b/jnak/archive/2010/06/07/using-intellitrace-to-debug-windows-azure-cloud-services.aspx


    Jim Nakashima, Microsoft. (This post is provided "as-is") Suggest and Vote for Windows Azure features: http://www.mygreatwindowsazureidea.com/
    Wednesday, June 09, 2010 7:12 AM
    Moderator
  •  

    I was also able to run the 4.0 application on Azure, the problem I am facing is that when I try to deploy anything which uses LIINQ to SQL or Entity Framework the deployment to the Azure fails. I can run the application fine on my development fabric.  I am creating a Silverlight Business Application in Visual Studio 2010.  I am not using any Third party control or assembly and I can add a simple web application hosting Silverlight but the deployment of the application wth RIA services always fails. I was wondering that do I have to create a WCF service role to host the Domain Services or a normal web role can have these services.

     

     

    Wednesday, July 14, 2010 8:41 AM
  • Hi

    I have attempted to deploy an Azure application many times but when I run it, it simply stays in the Busy state forever. I have a Worker role which starts and runs fine, however the web role does not. I have followed the instructions above but to no avail. One of the problems is that  I cannot really diagnose it is because no diagnostic tables (WAD*) are created or present and there are no clear directions on how to ensure diagnostic tables are present so I can view diagnostic information. I have completely deleted and recreated my Azure and SQL Azure instances and redeployed to no avail. I am also using the diagnostic tool from Cerebrata but it complains there are no diagnostic tables. Again, there is no provision for creating them anywhere.

     

    Any idea on why I dont have diagnostic tables and how to go about ensuring they are present?

    Thursday, July 29, 2010 10:38 PM
  • Hi,

    Try debugging your app using Intellitrace (http://blogs.msdn.com/b/jnak/archive/2010/06/07/using-intellitrace-to-debug-windows-azure-cloud-services.aspx ). You would need VS Ultimate to make use of Intellitrace.

    Hope this helps.

    Thanks

    Gaurav Mantri

    Cerebrata Software

    http://www.cerebrata.com

    Friday, July 30, 2010 9:49 AM
  • Hi,

     

    Is it possible to see the detail why our deployment stuck?

    Otherwise, we're quite blind about where the problem lies into.

    Friday, July 30, 2010 1:34 PM
  • Hi Gaurav,

     

    thanks for the response and I will be trying that technique. However, I would like to know how to get the diagnostic tables though as I would like the diagnostic information to support the application on an ongoing basis. Any help to enable this would be good. In addition, without diagnostic tables it makes tools like the one you guys make not very helpful as it requires the tables to function properly.

     

    - Glav

    Saturday, July 31, 2010 4:03 AM
  • Hi Glav,

    I think you'll be able to find out why your application is not working by using Intellitrace (It's basically a debugger in the cloud). One thing you would need to keep in mind is that Intellitrace feature comes with its own overhead and is not recommended for applications running in production environment.

    Once you have your roles running properly, the diagnostics table will be created automatically (You can also take this as a sign that diagnostics is working properly). Even if you manually create these diagnostics tables & containers, I don't think that's going to help as the diagnostics process will not be moving the data from your Azure VMs to these tables & containers.

    If you wish, you could share your role's OnStart() code here and somebody could take a look and possibly point you in the right direction.

    Hope this helps.

    Thanks

    Gaurav

    Saturday, July 31, 2010 8:09 AM
  • Yes it does. You have been most helpful Gaurav. Far more than Microsoft thats for sure.

     

    After using Intellitrace, I know what the issue is (the app was a large scale website, migrated to a web app, then the cloud, and it was having trouble with Azure storage - reliance on physical drives so I have to create a virtual drive for it).

    So, I know what the issue is now, I guess I just thought that Diagnostics would be available prior to it running properly. Seems kinda stupid that I have to figure out how to make it work properly before diagnostics is available. At any rate, many thanks for your assistance and I know how to proceed.

    Again, thanks for your help.

     

    - Glav

    Saturday, July 31, 2010 9:19 AM
  • If you are interested how to setup diagnostcs I have two posts that may be helpful:

    http://blog.toddysm.com/2010/04/windows-azure-diagnosticswhere-are-my-logs.html

    http://blog.toddysm.com/2010/05/collecting-event-logs-in-windows-azure.html


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Friday, August 20, 2010 5:29 PM
    Moderator
  • @wely_live: Right now the only option you have is to click on Get Support Info link in the portal that can eventually give you some details.

    We are working to improve this in the future versions.


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Friday, August 20, 2010 5:31 PM
    Moderator
  • What about if i don't have VS 2010 Ultimate and i want to use Intellitrace tool, Is there any way to get it on VS 2010 Premium Edition

     

    Any ideas ?

    Thursday, August 26, 2010 3:17 PM
  • Would using VS2010 Ulitmate as a trial work for you?  You can get a trial from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e2a1a098-995e-4468-816d-7fdbe0a64f38&displaylang=en.  It is good for 30 days by default, but you have the option to extend for an additional 60 days if you register the software.
    Monday, October 04, 2010 6:05 PM
  • I've gone through the steps suggested, but still am getting stuck at the Initializing state.  I'm not quite sure how to see the logs, at this point I've seen nothing transfered to storage, but perhaps it's an issue in my application.  The Intellitrace tool looks great, but I don't have VS2010 Ultimate.  Would love some suggestions...

     

    Here is my OnStart() method.  This role is really just a crude application to pull a large file out of blob storage, and create table entities from it.  

    Thanks in advance for any suggestions.

     

     

     public override bool OnStart() {
       // Set the maximum number of concurrent connections 
       ServicePointManager.DefaultConnectionLimit = 12;
    
       
       DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
       dmc.WindowsEventLog.DataSources.Add("Application!*");
       dmc.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
       dmc.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
       dmc.Logs.BufferQuotaInMB = 128;
       dmc.DiagnosticInfrastructureLogs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
       dmc.ConfigurationChangePollInterval = TimeSpan.FromSeconds(10.0);
    
       DiagnosticMonitor.Start("DiagnosticsConnectionString", dmc);
    
       Microsoft.WindowsAzure.Diagnostics.CrashDumps.EnableCollection(true);
       
    
       RoleEnvironment.Changing += RoleEnvironmentChanging;
    
       return base.OnStart();
      }
    

     

    Update:

    Ok, so it finally went into the "Ready" state.  According to the microsoft support technician I spoke with, this can take up to 20 minutes to initialize the application.  I was expecting this to be a minute or two, but this is not the case.  

     

     

    Friday, October 15, 2010 1:34 PM
  • Note that the new troubleshooting topic has moved with the 1.3 release. 

    Old  http://msdn.microsoft.com/en-us/library/ff431732.aspx . Will be removed eventually.

    New topic is at

    http://msdn.microsoft.com/en-us/library/gg465402.aspx

    Rob

    Monday, December 06, 2010 4:29 PM
  • I have updated the original post with the update link, thanks Rob.
    Brian Aurich
    Windows Azure Community PM
    Monday, January 17, 2011 6:46 PM
  • Role Relies on Missing Runtime Dependencies

    If your role relies on any assembly that is not part of the .NET 3.5 Framework or the Windows Azure managed library, you must explicitly include that assembly in the deployment package....

     

    I am really lost a point, I need open my 'ServiceConfiguration.cscfg' and add additional dll which I do use in my service bushiness logic ?

    If yes how can I do this, can you show some format ?

     

    I have some WCF service with config files and etc...

    I add reference to this WCF dll to my worker role.

    And properties of this reference 'Copy Local ' == 'True' is it enough ? 


    Service Configuration Includes Misconfigured DiagnosticsConnectionString Setting...

    I didn't want to use it. Can I delete it ?

    My file look like :

     

    <?xml version="1.0" encoding="utf-8"?>

    <ServiceConfiguration serviceName="THCloudService" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">

      <Role name="THChatWorkerRole">

        <Instances count="1" />

        <ConfigurationSettings>

        </ConfigurationSettings>

      </Role>

    </ServiceConfiguration>

     

    thanks for any help...

     


    Rost
    Thursday, August 18, 2011 11:23 PM
  • Any exception that happens before application starts, will keep the app from completing the startup process in azure cloud. So, one of the major reason I phased personally is referenced assemblies not included.

     

    So follow the below steps.

     

    1. In Solution Explorer, right click on your web and/or webservice projects and "Add Deployable Depedencies".

    2. And remove "WebMatrix*.dll" from added deployable dependencies.

     

    This process will add all dependecies and add reference to them in the configuration file. It will also add WebMatrix references which was causing runtime logical error in the app after successful startup when I checked last 2 months back so remove all webmatrix dll files to avoid this issueunless you need it in which case you are on your own. The issue can be read in http://forums.asp.net/p/1678471/4408864.aspx/1?Re+Wrongly+redirecting+to+Account+Login+ReturnUrl

    Cheers

     

     

     

    Sunday, September 04, 2011 12:43 PM
  • First time post, so apologies if I've put my question in the wrong place....as a newbie to Azure Development, I'm trying to deploy the Hello World example for Java (http://msdn.microsoft.com/en-us/library/windowsazure/hh690944(v=vs.103).aspx). I've got it running in the Compute Emulator function in Eclipse, but when I try deploying to the Azure portal, the Worker Role has the following message next to it:

    Waiting for role to start... Application startup task failed with exit code 1.

    I realise there could be quite a few root causes for this, so my initial question is: how do I go about troubleshooting? Are there log files I can get hold of, for example? I couldn't see anything inside the portal that would shed any light on this and my initial googling hasn't turned up anything either.

    Alternatively, are there any other Java developers who have tried this tutorial and got it working?

    Cheers

    Friday, April 27, 2012 7:55 AM
  • Is there any way to actually get a console log of any kind from Azure deployments?  The opaqueness of the deployment process is maddening.

    This guide is good, but it's not troubleshooting; It's just a checklist of frequent faults.

    IntelliTrace seemed like a possibility for troubleshooting, but it apparently requires the instance to start before it can function, rendering it useless for deployment issues in which the instance isn't starting properly:

    7:48:11 PM - Connecting to the IntelliTrace agent on the VM instance
    7:49:12 PM - IntelliTrace log request failed. Details: The IntelliTrace agent failed to respond for this instance.  This could be due to network issues, the role targets .NET 3.5 or an unexpected error in the agent.

    Azure Diagnostics is similarly lacking since it too starts too late in the process.

    There has GOT to be something more that offers useful insight into the deploy process?!  If I had a #1 feature request from Azure it would unquestionably be a, "View Startup Log" option for instances.  The lack of this most basic, absolutely critical feature is incredibly disheartening and a stunning oversight.

    Tuesday, November 20, 2012 6:51 PM
  • I too am no receiving the following message:

    Not all role instances are ready.

    2 Instances: 2 Waiting for status.

    This has been sitting in the same status for 20 minutes.  I am trying to push to my staging environment and have done this dozens of times with no problems.

    Is there a way I can see a more detail reason as to why this is happening ?

    Tuesday, November 27, 2012 4:03 AM
  • It's some way comforting that I'm the only one having these problems. I'm also going crazy with deploying a web application as a cloud service.

    This check list is useful, but doesn't help me. I have done near 20 deploys using two different subscriptions, and only 2 times did I manage to get a working deploy. After those two deployments i didn't manage to deploy for around 5 times.

    That whole deployment thingie is to "magical". There should be some kind of console/log, including exceptions/errors.

    What's the most frustrating is that a single deploy takes 30 minutes on average.

    Wednesday, December 19, 2012 4:04 PM