Worker Role stuck in Initializing state on Azure Fabric -- Works Fine in Dev Fabric

Question

I'm having a problem deploying a worker role to Azure. The Forums site suggests three similar issues. Two of them are answered but unrelated, and my problem seems to be the same as one that is still unanswered. Also note that the project I'm working on is for Microsoft, for the purposes of demonstrating how to port a .NET application (StockTrader, in this case) to Azure.

Specifically, I am hosting one of three WCF services in a worker role using a Service Bus endpoint. The other two WCF services are hosted in web roles, and they run fine both in the dev fabric and in the real cloud. The WCF service hosted in the worker role, however, runs just fine in the dev fabric but sits forever in the Initializing state when deploying to Azure. As a sanity check, I built a HelloWorld WCF app, and I'm experiencing exactly the same thing: Worker role runs and hosts the WCF service via the Service Bus just fine in dev fabric, but hangs forever in Initializing state when deploying to Azure.

Deployment ID: 65e4ce3012bf4e558ee77755478ee43f
Web Site URL: http://stocktrader2.cloudapp.net/

Here is the HelloWorld worker role code:

using System;
using System.Collections.Generic;
using System.Threading;
using System.Linq;
using System.Text;

using Microsoft.ServiceHosting.ServiceRuntime;

using System.ServiceModel;
using System.ServiceModel.Description;
using Microsoft.ServiceBus;
using WcfServiceImplementation;

namespace WcfWorkerRole
{
 public class WorkerRole : RoleEntryPoint
 {
  private ServiceHost _host;

  public override void Start()
  {
   RoleManager.WriteToLog("Information", "Host is starting");
   this._host = new ServiceHost(typeof(Service1));
   this._host.Open();

   ServiceRegistrySettings settings = new ServiceRegistrySettings();
   settings.DiscoveryMode = DiscoveryType.Public;
   foreach (ServiceEndpoint se in this._host.Description.Endpoints)
   {
    se.Behaviors.Add(settings);
   }

   RoleManager.WriteToLog("Information", "Host is running");
  }

  public override void Stop()
  {
   RoleManager.WriteToLog("Information", "Host is stopping");
   this._host.Close();
   base.Stop();
   RoleManager.WriteToLog("Information", "Host is stopped");
  }

  public override RoleStatus GetHealthStatus()
  {
   // This is a sample worker implementation. Replace with your logic.
   return RoleStatus.Healthy;
  }
 }
}

And here is the worker role's App.Config (Service Bus password replaced with *******)

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
 <system.serviceModel>
  <services>
   <service name="WcfServiceImplementation.Service1">
    <endpoint
     address="http://stocktraderservices.servicebus.windows.net/service1"
     binding="basicHttpRelayBinding"
     contract="WcfContracts.IService1"
     behaviorConfiguration="ServiceBusBehavior"
    />
   </service>
  </services>
  <behaviors>
   <endpointBehaviors>
    <behavior name="ServiceBusBehavior">
     <transportClientEndpointBehavior credentialType="UserNamePassword">
      <clientCredentials>
       <userNamePassword
          userName="StockTraderServices" password="********" />
      </clientCredentials>
     </transportClientEndpointBehavior>
    </behavior>
   </endpointBehaviors>
  </behaviors>
 </system.serviceModel>
</configuration>

The service contract is simple; just does an echo to prove that client and server are communicating properly. No need to include that code in the post, unless you request it. Again, as I said, it runs fine in the dev fabric.

I need to finish this project for Microsoft in the next day or two, and if necessary will need to document this problem with the deliverables, but I'd much rather get a resolution and have it working properly for you. Please help! Thanks in advance.

Answer 1

Make sure the Service Bus DLLs you're using are marked as "copy always" in their properties in VS.  My guess is they're not, and locally you have them in the GAC, but in the cloud you don't (so the assemblies can't be found).

Answer 2

Thanks for the quick response, Steve.

I'm not sure what you mean by "the Service Bus DLLs" I'm using. Do you mean Microsoft.ServiceBus.dll specifically, or something else? And by "copy always", do you mean that the "Copy Local" property should be set to True? I didn't see that mentioned in this MSDN forum post on hosting WCF services in worker roles using the Service Bus: http://social.msdn.microsoft.com/Forums/en/netservices/thread/7bf74134-673e-4139-8c0c-5dab7a726bc7

Anyway, I'll try your suggestion as soon as I can and post back with the result.

Answer 3

Well, I figured that by "copy always" you actually meant the "Copy To Output Directory" property, which has the possible settings of Do Not Copy, Copy If Newer, and Copy Always. Thing is, the "Copy To Output Directory" property is only available on items that are part of the project. The Microsoft.ServiceBus.dll is just a reference, and has no "Copy To Output Directory" property when the assembly is selected in the References node in Solution Explorer (just a "Copy Local" property).

So first I actually added a copy of Microsoft.ServiceBus.dll files as an actual item included in the project, and then set its "Copy To Output Directory" property to Copy Always.

Well that didn't work... same exact problem; worker role stuck Initializing when deployed to Azure.

Then I thought maybe you actually did mean the "Copy Local" property on the Microsoft.ServiceBus.dll assemby listed in the References node. I set Copy Local to True for Microsoft.ServiceBus.dll, but unfortunately, it's still a no-go after deploying again to Azure.

Deployment ID: de51e019844e4bbea4b3a702c9bb44fb

Worker role is just stuck in Initializing state. Any other suggestions? Help urgently needed, thanks again!

If it helps any, I did notice that the worker role switches to Busy status, then Stopping status, every so often, before it switches back to Initializing. So it appears to be failing during startup, and is in an endless loop trying it over and over again.

Answer 4

Hello, looks like you're setting the Service Bus bindings and behaviors in configuration. Please note the .NET Services SDK is not installed on the cloud machine, so the binding and behavior extensions do not exist in machine.config. You will have to manually configure the WCF extension in your app.config for it to work. However, if you do that, it will probablly stop working on your local machine, because WCF doesn't allow duplex extension configuration...

That's why it is recommended to take the code only approach if you want to use Service Bus in the cloud. Have a look at the third post in this thread: http://social.msdn.microsoft.com/Forums/en-US/netservices/thread/7bf74134-673e-4139-8c0c-5dab7a726bc7.

---

Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 5

Thanks for the info. I've modified the HelloWorld worker role service and web role client to configure the WCF settings using code-only, rather than app.config/web.config. I also made sure to package Microsoft.ServiceBuss.dll in the bin folder, as per Philip Richardson's post. With no more settings in app.config, I am also able to continue running successfully in the dev fabric.

Now the Worker Role does in fact start when deployed to Azure, rather than getting stuck in the Initializing state like it was before. So it is clear that switching to code-only WCF configuration corrected the problem I reported. Enter new problem!

New problem is, when the service method is invoked by the client on the channel, it fails with either "404 not found" (if I implement the infinite while-true loop) or "An existing connection was forcibly closed by the remote host" (if I don't). Philip Richardson's post uses the infinite while-true loop, but I'm desperately trying everything possible to get it working in the cloud. And again, it works fine in dev fabric -- both with and without the infinite while-true loop.

With the infinite loop version, the worker role starts normally, and seems to remain in a Started state indefinitely. But as soon as the first client request is issued, the worker role crashes and a 404 gets thrown on the client. After that, the worker restarts and then crashes continuously.

Without the infinite loop, the worker role never remains in a Started state indefinitely, even if a client request is never issued. So I'm pretty sure that the inifinite loop is correct and required, as shown in Philip Richardson's post.

[WebException: The remote server returned an error: (404) Not Found.]
   System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) +728
   System.ServiceModel.Channels.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result) +133

[EndpointNotFoundException: There was no endpoint listening at http://stocktraderservices.servicebus.windows.net/service1 that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.]
   System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) [phone]
   System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) +539
   WcfContracts.IService1.GetData(Int32 value) +0
   WcfWebRoleClient._Default.btnTest_Click(Object sender, EventArgs e) in C:\Projects\WcfWorkerRolePOC\WcfWebRoleClient\Default.aspx.cs:38
   System.Web.UI.WebControls.Button.OnClick(EventArgs e) +90
   System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +138
   System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +32
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3087

Why is this happening?!?

I'm at my wits end on this, and just can't get it to work. Here's the code (no more app.config)

public override void Start()
  {
   string endPoint = "http://stocktraderservices.servicebus.windows.net/service1";
   string userName = "StockTraderServices";
   string password = "paradox";

   Uri uri = new Uri(endPoint);

   var creds = new TransportClientEndpointBehavior();
   creds.CredentialType = TransportClientCredentialType.UserNamePassword;
   creds.Credentials.UserName.UserName = userName;
   creds.Credentials.UserName.Password = password;

   _host = new ServiceHost(typeof(Service1), uri);

   var contractDescription = ContractDescription.GetContract(typeof(IService1), typeof(Service1));

   var serviceEndPoint = new ServiceEndpoint(contractDescription);
   serviceEndPoint.Address = new EndpointAddress(uri);
   serviceEndPoint.Binding = new BasicHttpRelayBinding();
   serviceEndPoint.Behaviors.Add(creds);

   _host.Description.Endpoints.Add(serviceEndPoint);
   _host.Open();

   // Worker role starts and runs fine, but upon first client request, then crashes and client gets a 404 exception
   // Only in the cloud... not in dev fabric!
   while (true)
   {
   }
  }

And here's the client code (a button click event in a web role aspx page):

  protected void btnTest_Click(object sender, EventArgs e)
  {
   string endPoint = "http://stocktraderservices.servicebus.windows.net/service1";
   string userName = "StockTraderServices";
   string password = "paradox";

   Uri uri = new Uri(endPoint);

   var creds = new TransportClientEndpointBehavior();
   creds.CredentialType = TransportClientCredentialType.UserNamePassword;
   creds.Credentials.UserName.UserName = userName;
   creds.Credentials.UserName.Password = password;

   var cf = new ChannelFactory<IService1Channel>();
   cf.Endpoint.Address = new EndpointAddress(uri);
   cf.Endpoint.Binding = new BasicHttpRelayBinding();
   cf.Endpoint.Contract.ContractType = typeof(IService1Channel);
   cf.Endpoint.Behaviors.Add(creds);

   var channel = cf.CreateChannel();

   var result = channel.GetData(5);

   Response.Write(result + "<br />");
  }

Runs like a charm in dev fabric; crashes miserably when deployed to the cloud. Please help!