MSMQ Activation of workflow service

Question

Hi

I am hosting a workflow service in WAS and using AppFabric to manage.  The service is fairly simple, and only has one method.  My service has 2 endpoints; http for metadata and msmq for activating the method.

If I ask for the service wsdl via http and then send a msmq message to activate the method, the service works without issue.

However, if you 'restart' the IIS application then send an msmq message, the workflow fails.  Using the workflow tracing feature of appfabric, you can see what is happening, I get three events:

Information    MessageReceivedFromTransport       @16:08.54.627

Error             ServiceException                             @16:08.54.815

Information    ServiceHostStarted                          @16:08.55.237

In other words, the service fails fairly quickly after receiving the incoming message.  If I look at details of the error raised, I get the following exception:

System.NullReferenceException: Object reference not set to an instance of an object.

at System.ServiceModel.Activities.Dispatcher.DurableInstanceManager.GetInstanceAsyncResult.GetInstance()

at System.ServiceModel.Activities.Dispatcher.DurableInstanceManager.GetInstanceAsyncResult..ctor(DurableInstanceManager instanceManager, Guid instanceId, WorkflowGetInstanceContext parameters, TimeSpan timeout, AsyncCallback callback, Object state)

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.ControlOperationAsyncResult.Process()

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.ControlOperationAsyncResult..ctor(ControlOperationInvoker invoker, Object[] inputs, IInvokeReceivedNotification notification, TimeSpan timeout, AsyncCallback callback, Object state)

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.InvokeBegin(Object instance, Object[] inputs, IInvokeReceivedNotification notification, AsyncCallback callback, Object state)

at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)

at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

The reason I am raising this on the AppFabric forum as it looks (to me) as though the error is encountered by the DurableInstanceManager, which I assume is being used by AppFabric to persist the workflow or search for an previously persisted version that it can use?

I am using AppFabric release version on Win 2K8 SP2.  I have AppFabric and WCF traces of the error if needed.

Thanks

Nick

---

Nick C

Answer 1

I'm having the same problem. Any ideas?

Answer 2

No ideas at all :-(

I have come up with a work-around, which is to write a small wpf app with a web browser on it.  Point the web browser at the service xaml file, and get it to refresh every 2 mins.  This keeps the service alive, and avoids the problem.  However, its not a solution I can take to production!

I am now getting this error on two different computers, and if Chris is getting it as well it is looking like an AppFabric bug.  Does any one know how I can progress this with Microsoft?

Many thanks for your help

Nick

---

Nick C

Answer 3

We have some folks from the Product Group looking into this, but they have not found the answer yet.  We'll post something in a few days when we have figured out something more.

Answer 4

Hi Nick,

Another option that may work for you is to set the apps to autostart (Manage WCF and WF services->Configure->Auto-Start->Enabled).

Answer 5

The product team confirmed that this is a bug. You can use Chris' suggestion to work around this issue and enable autostart.

Regards, Ruppert

Answer 6

I have this exact issue, and I am using auto-start.  It seems to happen intermittently, and I'm not sure what to do about it, any other ideas?

Answer 7

Me too: I also have a Workflow Service with MSMQ endpoints and Auto-Start set to Enabled. After restarting the App-Pool Queue Messages are not processed until I browse to the services http address and refresh the website  (hit F5). The funny thing is that I tried to reproduce the issue using a simple workflow service by increasing complexity till it reaches the complexity (count of endpoints, behavior extensions, endpoint extensions, ...) of the workflow service with the issue. But this test workflow always works even with almost exact same configuration. Here is the error message we get:

<

 

 

E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">

<

 

 

System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">

<

 

 

EventID>131076</EventID>

<

 

 

Type>3</Type>

<

 

 

SubType Name="Error">0</SubType>

<

 

 

Level>2</Level>

<

 

 

TimeCreated SystemTime="2011-03-10T10:11:39.3466204Z" />

<

 

 

Source Name="System.ServiceModel" />

<

 

 

Correlation ActivityID="{71fe4e71-ec5c-43d3-8705-0185cab6d4b9}" />

<

 

 

Execution ProcessName="w3wp" ProcessID="5552" ThreadID="18" />

<

 

 

Channel />

<

 

 

Computer>DRSSA304</Computer>

</

 

 

System>

<

 

 

ApplicationData>

<

 

 

TraceData>

<

 

 

DataItem>

<

 

 

TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">

<

 

 

TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.TraceHandledException.aspx</TraceIdentifier>

<

 

 

Description>Handling an exception.</Description>

<

 

 

AppDomain>/LM/W3SVC/1/ROOT/DEV/WFS/MSMQService_CEP_Persistence_FileTrack-4-129442254938200153</AppDomain>

<

 

 

Exception>

<

 

 

ExceptionType>System.NullReferenceException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>

<

 

 

Message>Object reference not set to an instance of an object.</Message>

<

 

 

StackTrace>

at System.ServiceModel.Activities.Dispatcher.DurableInstanceManager.GetInstanceAsyncResult.GetInstance()

at System.ServiceModel.Activities.Dispatcher.DurableInstanceManager.GetInstanceAsyncResult..ctor(DurableInstanceManager instanceManager, Guid instanceId, WorkflowGetInstanceContext parameters, TimeSpan timeout, AsyncCallback callback, Object state)

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.ControlOperationAsyncResult.Process()

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.ControlOperationAsyncResult..ctor(ControlOperationInvoker invoker, Object[] inputs, IInvokeReceivedNotification notification, TimeSpan timeout, AsyncCallback callback, Object state)

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.InvokeBegin(Object instance, Object[] inputs, IInvokeReceivedNotification notification, AsyncCallback callback, Object state)

at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp; rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp; rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc&amp; rpc)

at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

</

 

 

StackTrace>

<

 

 

ExceptionString>System.NullReferenceException: Object reference not set to an instance of an object.

at System.ServiceModel.Activities.Dispatcher.DurableInstanceManager.GetInstanceAsyncResult.GetInstance()

at System.ServiceModel.Activities.Dispatcher.DurableInstanceManager.GetInstanceAsyncResult..ctor(DurableInstanceManager instanceManager, Guid instanceId, WorkflowGetInstanceContext parameters, TimeSpan timeout, AsyncCallback callback, Object state)

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.ControlOperationAsyncResult.Process()

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.ControlOperationAsyncResult..ctor(ControlOperationInvoker invoker, Object[] inputs, IInvokeReceivedNotification notification, TimeSpan timeout, AsyncCallback callback, Object state)

at System.ServiceModel.Activities.Dispatcher.ControlOperationInvoker.InvokeBegin(Object instance, Object[] inputs, IInvokeReceivedNotification notification, AsyncCallback callback, Object state)

at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp; rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp; rpc)

at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc&amp; rpc)

at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

 

</ExceptionString>

</

 

 

 

Exception>

</

 

 

TraceRecord>

</

 

 

DataItem>

</

 

 

TraceData>

</

 

 

ApplicationData>

</

 

 

E2ETraceEvent>

Answer 8

I found a solution for my case. We made an automatic deployment and it seems some required attributes had not been set in the applicationHost.config. Be sure to have the following attributes set for your AppPool and Application in applicationHost.config:

 

AppPool:

 

<add name="DevTestF5Problem" autoStart="true" enable32BitAppOnWin64="false" managedRuntimeVersion="v4.0" startMode="AlwaysRunning">

                <processModel identityType="SpecificUser" userName=".." password="..."/>

                <recycling>

                    <periodicRestart>

                        <schedule>

                            <clear />

                        </schedule>

                    </periodicRestart>

                </recycling>

                <cpu smpAffinitized="false" />

            </add>

 

Application:

 

<application path="/Dev/WorkflowHost/SendEMailTest/F5Problem" applicationPool="DevTestF5Problem" enabledProtocols="http,net.msmq,net.pipe" serviceAutoStartEnabled="true" serviceAutoStartProvider="Service" serviceAutoStartMode="All">

                    <virtualDirectory path="/" physicalPath="C:\inetpub\wwwroot\Dev\WorkflowHost\SendEMailTest\F5Problem" />

                </application>

 

Answer 9

Thanks, nabohi! I'll give that a try and see if it help in my case.

Answer 10

It's been a while since this was identified as a bug, currently I'm still experiencing it, any idea's on where or when a fix will be available? The workaround is OK but for memory usage this workaround has a big downside. I have multiple applications/appdomains which need to be autostarted due to this bug causing memory usage to skyrocket while this is not necessary because not all appdomains get used at the same time, leaving less available memory for the ones that do get used.

Answer 11

Likewise - getting the same problem here. Can anyone link to the ms connect bug?

---

Tech Lead