The execution of an InstancePersistenceCommand was interrupted because the instance '8f9769e0-4e5f-4899-84e2-74a808e2da89' has not yet been persisted to the instance store.

Question

Hi All,

I have a workflow service (I) which invoke some other WCF service(II) in four different activities(with some delay). I have hosted both them in IIS-7. I have configured Appfabric for persistence for workflow service(I). Now when I run the console client which is invoking workflow service (I) which internally calling WCF service (II). I am logging the information for my each activities of workflow runs in trace.log file. Meanwhile I intentionally stopped the WCF service (II) and completed the workflow execution. Exception is thrown that "ServiceTooBusyException" for the WCF service (II) which I stopped. Now I started my WCF service (II) service again and clicked on AppFabric Dashboard for the workflow service (I) in which I have one Persistence Instance as suspended. I resumed that instance, with a hope that workflow service (I) will completes its remaining activity which were not entertained because of WCF service(II) "ServiceTooBusyException". Now when I looked into database for table "InstanceControlCommandsTable" I found one entry with subsequent ExecutionAttempts as "3" and Exception as the Error-The execution of an InstancePersistenceCommand was interrupted because the instance '8f9769e0-4e5f-4899-84e2-74a808e2da89' has not yet been persisted to the instance store.

Now I am unable to understand why this happening? Though after some time its get removed from "InstanceControlCommandsTable" and later on entered into "AbandonedInstanceControlCommandsTable". That means entire flow is running fine, and we have entry in table "LockOwnersTable" also.

Can any one explain me how to resume all activities of my workflow even after it have exception thrown in it, to avail persistence benefits.

Thanks and Regards

Mohammed Haroon

(Technology Enthusiast, Active learner and decent developer.)

Answer 1

1) check if the windows services AppFabricEventCollectionService and AppFabricWorkflowManagementService are configured to run using accounts that are members of the AS_Admin APp fabric configured group

2) Additional error infomation would be available in the event log under Application and Services-->MIcrosoft--> Windows-->Appliction Server - System Services --> Admin

Answer 2

Hi Elan,

I am still getting same issue. I did as you recommended in Point "1)" and have seen in  event log under Application and Services-->Microsoft--> Windows-->Appliction Server - System Services --> Admin

This is the detail error description(XML View) I am getting:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

- <System>

  <Provider Name="Microsoft-Windows-Application Server-Workflow Management Service" Guid="{C821412B-1624-4ABE-A845-50B2C98C2405}" />

  <EventID>50910</EventID>

  <Version>0</Version>

  <Level>2</Level>

  <Task>0</Task>

  <Opcode>0</Opcode>

  <Keywords>0x8000000000000000</Keywords>

  <TimeCreated SystemTime="2012-05-15T07:07:18.662829500Z" />

  <EventRecordID>12468</EventRecordID>

  <Correlation />

  <Execution ProcessID="7152" ThreadID="6732" />

  <Channel>Microsoft-Windows-Application Server-System Services/Admin</Channel>

  <Computer>PC-7</Computer>

  <Security UserID="S-1-5-21-74034007-3002401088-2774838483-1000" />

  </System>

- <EventData>

  <Data Name="ControlAction">Resume</Data>

  <Data Name="InstanceId">6a52ace9-271d-430b-9012-108f1a2dd760</Data>

  <Data Name="EndpointAddress">net.pipe://pc-7/WorkFlowServicePersistenceTest/PersistenceChecker.xamlx/System.ServiceModel.Activities_IWorkflowInstanceManagement</Data>

  <Data Name="ExceptionMessage">System.ServiceModel.FaultException: The execution of an InstancePersistenceCommand was interrupted because the instance '6a52ace9-271d-430b-9012-108f1a2dd760' has not yet been persisted to the instance store.</Data>

  <Data Name="OccurrenceMap">0</Data>

  <Data Name="Occurrences">1</Data>

  <Data Name="InstanceName">Nameless_WMS_Instance</Data>

  </EventData>

  </Event>

I think I am getting same exception as I was getting earlier....!

Thanks and Regards
Mohammed Haroon

(Technology Enthusiast, Active learner and decent developer.)

Answer 3

1) I am assuming you are using sqlpersistancestore and unhandled exception behaviour configured via behaviour exttensions?? Similar to below

 <serviceBehaviors>

  <behavior>

    <sqlWorkflowInstanceStore connectionString="Data Source=servername;..." hostLockRenewalPeriod="00:00:30" runnableInstancesDetectionPeriod="00:00:30" />

    <workflowUnhandledException action="AbandonAndSuspend"/>

  </behavior>

</serviceBehaviors>

2) The error you have posted from the event vwr, is the error you see when you try and resume the service? Are there any errors when you try and forcefully stop the service in the first place?

Answer 4

Hi ELan,

This the Config file settings I am doing:

  <serviceBehaviors>
        <behavior>        
          <serviceMetadata httpGetEnabled="true" />         
          <serviceDebug includeExceptionDetailInFaults="true" />           
                <workflowUnhandledException action="AbandonAndSuspend" />
                <workflowIdle timeToPersist="00:00:10" timeToUnload="00:00:10" />
                    <etwTracking profileName="HealthMonitoring Tracking Profile" />
                    <sqlWorkflowInstanceStore instanceCompletionAction="DeleteNothing"
                                              instanceEncodingOption="GZip" 
                                              instanceLockedExceptionAction="BasicRetry" 
                                              connectionStringName="ApplicationServerWorkflowInstanceStoreConnectionString" 
                                              hostLockRenewalPeriod="00:00:30" 
                                              runnableInstancesDetectionPeriod="00:00:05" />       
        </behavior>
      </serviceBehaviors>

Where "ApplicationServerWorkflowInstanceStoreConnectionString" is the Connection string to my local database which is configured with AppFabric. as shown below:

The WorkFlowService (I) configuration for Workflow Persistence  in IIS-7(AppFabric):

Thanks and Regards

Mohammed Haroon

(Technology Enthusiast, Active learner and decent developer.)

Answer 5

And Following is the SQL database entries:


And about the error as I mention I have stopped the WCF Services(II) intentionally, so I have got error as :

The HTTP service located at http://localhost/WCFServiceForPersistenceCheck/WCFServicePersistenceChecker.svc is too busy. 

Apart from that there was no error except the one I mention earlier.

Thanks and Regards
Mohammed Haroon

(Technology Enthusiast, Active learner and decent developer.)

Answer 6

Thanks for posting the details.

Are you able to turn WCF tracing on for  your workflow service (I)  See config below? And then inspect the trace logs (wcftrace.svclog which will be located in you web app folder, ensure write access to this folder)  for additional information?

Also I noticed that you are using sql express, then the issue might be related to post http://social.msdn.microsoft.com/Forums/pl-PL/dublin/thread/c45afd21-646b-43f8-815f-e385fe49afe3

<configuration>
  <system.serviceModel>
    <diagnostics>
      <messageLogging logEntireMessage="true" maxMessagesToLog="50000" logMalformedMessages="true" logMessagesAtServiceLevel="true" logMessagesAtTransportLevel="true" maxSizeOfMessageToLog="2000000" />
  
    </diagnostics>
  </system.serviceModel>
  <system.diagnostics>
    <sources>
      <source name="System.ServiceModel.MessageLogging"  >
        <listeners>
          <add name="xml" />
        </listeners>
      </source>
      <source name="System.ServiceModel" switchValue="Information"  propagateActivity="false">
        <listeners>
        
          <add name="DefaultListener">
            <filter type="" />
          </add>
   
        </listeners>
      </source>
      <source name="CardSpace" switchValue="Warning" >
        <listeners>
          <add name="DefaultListener" />
        </listeners>
      </source>
      <source name="System.IO.Log" switchValue="Warning" >
        <listeners>
          <add name="DefaultListener"  />
        </listeners>
      </source>
      <source name="System.Runtime.Serialization" switchValue="Warning" >
        <listeners>
          <add name="DefaultListener" />
        </listeners>
      </source>
      <source name="System.IdentityModel" switchValue="Warning" >
        <listeners>
          <add name="DefaultListener" />
        </listeners>
      </source>
     
    </sources>
    <sharedListeners>
   
      <add name="DefaultListener"
        type="System.Diagnostics.XmlWriterTraceListener"
        initializeData="wcftrace.svclog" />
      <add name="xml"
        type="System.Diagnostics.XmlWriterTraceListener"
        initializeData="wcfmessage.svclog" />
    </sharedListeners>
    <trace autoflush="true" indentsize="4">
    </trace>
  </system.diagnostics>
</configuration>

Answer 7

Hi ELan

This are the snapshots of wcftrace.svclog:

Answer 8

Not able to understand what is

1.

<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent"><System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system"><EventID>655363</EventID><Type>3</Type><SubType Name="Information">0</SubType><Level>8</Level><TimeCreated SystemTime="2012-05-18T10:48:38.6876114Z" /><Source Name="System.ServiceModel" /><Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" /><Execution ProcessName="w3wp" ProcessID="2976" ThreadID="44" /><Channel /><Computer>PC-7</Computer></System><ApplicationData><TraceData><DataItem><TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Information"><TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.PortSharing.PortSharingDupHandleGranted.aspx</TraceIdentifier><Description>The PROCESS_DUP_HANDLE access right has been granted to the NetPipeActivator service's account with SID 'S-1-5-80-2943419899-937267781-4189664001-1229628381-3982115073'.</Description><AppDomain>/LM/W3SVC/1/ROOT/WorkFlowServicePersistenceTest-12-129818116660275994</AppDomain><Source>System.String/-35393209</Source></TraceRecord></DataItem></TraceData></ApplicationData></E2ETraceEvent>

Answer 9

2. 

<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>524312</EventID>
<Type>3</Type>
<SubType Name="Warning">0</SubType>
<Level>4</Level>
<TimeCreated SystemTime="2012-05-18T10:47:46.2896144Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="w3wp" ProcessID="2976" ThreadID="17" />
<Channel />
<Computer>PC-7</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Warning">
<TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.EvaluationContextNotFound.aspx</TraceIdentifier>
<Description>Configuration evaluation context not found.</Description>
<AppDomain>/LM/W3SVC/1/ROOT/WorkFlowServicePersistenceTest-12-129818116660275994</AppDomain>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>

3.

<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>524329</EventID>
<Type>3</Type>
<SubType Name="Warning">0</SubType>
<Level>4</Level>
<TimeCreated SystemTime="2012-05-18T10:47:46.2816140Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
<Execution ProcessName="w3wp" ProcessID="2976" ThreadID="17" />
<Channel />
<Computer>PC-7</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Warning">
<TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.OverridingDuplicateConfigurationKey.aspx</TraceIdentifier>
<Description>The configuration system has detected a duplicate key in a different configuration scope and is overriding with the more recent value.</Description>
<AppDomain>/LM/W3SVC/1/ROOT/WorkFlowServicePersistenceTest-12-129818116660275994</AppDomain>
<Source>System.ServiceModel.Configuration.ServiceBehaviorElementCollection/-700428053</Source>
<ExtendedData xmlns="http://schemas.microsoft.com/2006/08/ServiceModel/DictionaryTraceRecord">
<ElementName>behavior</ElementName>
<Name></Name>
<OldElementLocation></OldElementLocation>
<OldElementLineNumber>0</OldElementLineNumber>
<NewElementLocation></NewElementLocation>
<NewElementLineNumber>0</NewElementLineNumber>
</ExtendedData>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>

Thanks and Regards
Mohammed Haroon

(Technology Enthusiast, Active learner and decent developer.)

Answer 10

There is nothing out of the ordinary in the WCF service logs.

1) Have you checked out the other post  http://social.msdn.microsoft.com/Forums/pl-PL/dublin/thread/c45afd21-646b-43f8-815f-e385fe49afe3 , as you are using SQL express?

2) Also enable a similar WCF trace to service App fabcric workflow management windows service . Config File C:\Windows\System32\AppFabric\Workflowmanagementservice.exe.config. And see if there is any additional information in there.

Answer 11

Also,

The issue might also be related to the fact that an unhandled exception has occurred before the workflow service was persisted for the first time ( see http://msdn.microsoft.com/en-us/library/ee960214.aspx ).

If an instance is not persisted after the initial entry for the instance is created, the service instance is said to be in the non-persisted state. All the persisted service instances can be queried and controlled. Non-persisted service instances can neither be queried nor controlled. If a non-persisted instance is suspended due to an unhandled exception it can be queried but not controlled.

Durable service instances that are not yet persisted remain in a non-persisted state in the following scenarios:

• The service host crashes before the instance persisted for the first time. The initial entry for the instance remains in the instance store. The instance is not recoverable. If a correlated message arrives, the instance becomes active again.
• The instance experiences an unhandled exception before it persisted for the first time. The following scenarios arise
  
  
  • If the value of the UnhandledExceptionAction property is set to Abandon, the service deployment information is written to the instance store and the instance is unloaded from memory. The instance remains in non-persisted state in the persistence database.
  • If the value of the UnhandledExceptionAction property is set to AbandonAndSuspsend, the service deployment information is written to the persistence database and the instance state is set to Suspended. The instance cannot be resumed, canceled, or terminated. The service host cannot load the instance because the instance hasn't persisted yet and, hence the database entry for the instance is not complete.
  • If the value of the UnhandledExceptionAction property is set to Cancel or Terminate, the service deployment information is written to the instance store and the instance state is set to Completed.

Can you persist the workflow serivice either by using the persist activity or using perists before send option and then cause the exception to occur?