none
Workflow Manager stops processing workflows RRS feed

  • Question

  • Hello,

    I am having problems with the workflow manager at one of our farms.  It has been running fine for almost two years now, but stopped working a couple of weeks ago. 

    If I go to our app server (where workflow manager runs) and I restart the service Workflow Service Backend then workflows run fine for a while.  After some time (about 30 minutes, but not sure exact amount yet) it stops processing workflows, so the workflows get stuck at a "Not Started" state where the Initiator is Anonymous.  

    I don't see anything in the event viewer on the web server or the app server that would seem to point us in the right direction.

    I checked the web config file to see if anything has been changed or maybe installed on SharePoint recently to cause this problem.  The web config file hasn’t been changed since January 2017, over a year, so that doesn’t seem to be the problem.

    I also checked the profile services and search services to make sure those are running okay.  All of that seemed fine.  I even ran a full import on the profile services to be sure.

    I have also tried rebooting the app server, then I tried rebooting the web servers, and then I tried windows updates for everything other than SharePoint updates.  Most of the updates are installed but these servers can't connect to MS to check for the latest.  However, it has been running fine with this configuration for about 2 years now so I doubt updates are the problem.

    I tried re-registering the workflow on the web server with 
    Register-SPWorkflowService –SPSite "http://ourWFE" –WorkflowHostUri "http://ourAppSvr:12291" –AllowOAuthHttp -Force

    Even the ULS logs don't seem to have much info in them.  Here is what appears to be related from the logs.

    02/28/2018 02:10:20.75 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (GET:http://ourSite/ourList/_layouts/15/wrkstat.aspx?List=%7B31423651-8F26-4B8E-A5A8-93266A2D0243%7D&WorkflowInstanceName=3a5dfa21-bb8a-460f-b75d-02373ff200aa) 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.75 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation General ajji6 High Unable to write SPDistributedCache call usage entry. 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.80 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Logging Correlation Data xmnv Medium Site=/ 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.82 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Database ahjqp High [Forced due to logging gap, cached @ 02/28/2018 02:10:20.80, Original Level: Verbose] SQL connection time: 0.0625079444454533 for Data Source=SPDB;Initial Catalog=WSS_Content;Integrated Security=True;Persist Security Info=False;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=False;Connection Reset=True;MultipleActiveResultSets=False;Replication=False;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False;Load Balance Timeout=0;Packet Size=8000;Type System Version=Latest;Application Name=".Net SqlClient Data Provider";User Instance=False;Context Connection=False;Transaction Binding="Implicit Unbind";ApplicationIntent=ReadWrite;MultiSubnetFailover=False;ConnectRetryCount=1;ConnectRetryInterval=10 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.82 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Authentication Authorization ajmmt High [Forced due to logging gap, Original Level: VerboseEx] SPRequestParameters: AppPrincipal={0}, UserName={1}, UserKye={2}, RoleCount={3}, Roles={4} 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.90 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation General g3qj High [Forced due to logging gap, cached @ 02/28/2018 02:10:20.83, Original Level: Verbose] url is in site 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.90 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation General 6t8b High [Forced due to logging gap, Original Level: Verbose] Looking up {0} site {1} in the farm {2} 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.91 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Application Authentication aiv3l High Token request correlation id: '19aa4e9e-09f5-b03f-8e7c-65f772e21713'. 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:20.94 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Application Authentication aiv3l High Token request correlation id: '19aa4e9e-09f5-b03f-8e7c-65f772e21713'. 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.10 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Application Authentication ahi15 High [Forced due to logging gap, cached @ 02/28/2018 02:10:20.94, Original Level: Verbose] OAuth2 S2S Bearer Token: {0} 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.10 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Database 8acb High [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.22 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Database ahjqp High [Forced due to logging gap, cached @ 02/28/2018 02:10:21.14, Original Level: Verbose] SQL connection time: 0.0514730224092727 for Data Source=SPDB;Initial Catalog=WSS_Content;Integrated Security=True;Persist Security Info=False;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=False;Connection Reset=True;MultipleActiveResultSets=False;Replication=False;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False;Load Balance Timeout=0;Packet Size=8000;Type System Version=Latest;Application Name=".Net SqlClient Data Provider";User Instance=False;Context Connection=False;Transaction Binding="Implicit Unbind";ApplicationIntent=ReadWrite;MultiSubnetFailover=False;ConnectRetryCount=1;ConnectRetryInterval=10 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.22 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Database 8acb High [Forced due to logging gap, Original Level: VerboseEx] Reverting to process identity 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.29 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Database ahjqp High [Forced due to logging gap, cached @ 02/28/2018 02:10:21.22, Original Level: Verbose] SQL connection time: 0.0344317504040318 for Data Source=SPDB;Initial Catalog=WSS_Content;Integrated Security=True;Persist Security Info=False;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=False;Connection Reset=True;MultipleActiveResultSets=False;Replication=False;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False;Load Balance Timeout=0;Packet Size=8000;Type System Version=Latest;Application Name=".Net SqlClient Data Provider";User Instance=False;Context Connection=False;Transaction Binding="Implicit Unbind";ApplicationIntent=ReadWrite;MultiSubnetFailover=False;ConnectRetryCount=1;ConnectRetryInterval=10 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.29 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Web Controls ad9g2 High [Forced due to logging gap, Original Level: Verbose] QoS data point list: [] 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.32 w3wp.exe (0x0F24) 0x58C0 SharePoint Server Search Query dn4s High FetchDataFromURL start at(outside if): 1 param: start 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.38 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Database ahjqp High [Forced due to logging gap, cached @ 02/28/2018 02:10:21.36, Original Level: Verbose] SQL connection time: 0.0501460381137826 for Data Source=SPDB;Initial Catalog=WSS_Content;Integrated Security=True;Persist Security Info=False;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=False;Connection Reset=True;MultipleActiveResultSets=False;Replication=False;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False;Load Balance Timeout=0;Packet Size=8000;Type System Version=Latest;Application Name=".Net SqlClient Data Provider";User Instance=False;Context Connection=False;Transaction Binding="Implicit Unbind";ApplicationIntent=ReadWrite;MultiSubnetFailover=False;ConnectRetryCount=1;ConnectRetryInterval=10 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.38 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Asp Runtime aj1kp High [Forced due to logging gap, Original Level: Verbose] SPRequestModule.PreSendRequestHeaders 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:21.38 w3wp.exe (0x0F24) 0x58C0 SharePoint Foundation Micro Trace uls4 Medium Micro Trace Tags: 0 ajji6,3 agb9s,153 aiv3l,27 aiv3l,371 dn4s 22aa4e9e-791d-b03f-8e7c-6e80612baa8f
    02/28/2018 02:10:55.46 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (GET:http://ourSite/ourList/_layouts/15/wrkstat.aspx?List=%7B31423651-8F26-4B8E-A5A8-93266A2D0243%7D&WorkflowInstanceName=3a5dfa21-bb8a-460f-b75d-02373ff200aa) 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:55.46 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation General ajji6 High Unable to write SPDistributedCache call usage entry. 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:55.48 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Logging Correlation Data xmnv Medium Site=/ 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:56.42 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Database ahjqp High [Forced due to logging gap, cached @ 02/28/2018 02:10:55.48, Original Level: Verbose] SQL connection time: 0.0739619141538939 for Data Source=SPDB;Initial Catalog=WSS_Content;Integrated Security=True;Persist Security Info=False;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=False;Connection Reset=True;MultipleActiveResultSets=False;Replication=False;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False;Load Balance Timeout=0;Packet Size=8000;Type System Version=Latest;Application Name=".Net SqlClient Data Provider";User Instance=False;Context Connection=False;Transaction Binding="Implicit Unbind";ApplicationIntent=ReadWrite;MultiSubnetFailover=False;ConnectRetryCount=1;ConnectRetryInterval=10 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:56.42 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (SPSqlClient). Execution Time=938.223922314149 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:56.71 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Database ahjqp High [Forced due to logging gap, cached @ 02/28/2018 02:10:56.42, Original Level: Verbose] SQL connection time: 0.0430222276853622 for Data Source=SPDB;Initial Catalog=WSS_Content;Integrated Security=True;Persist Security Info=False;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Asynchronous Processing=False;Connection Reset=True;MultipleActiveResultSets=False;Replication=False;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False;Load Balance Timeout=0;Packet Size=8000;Type System Version=Latest;Application Name=".Net SqlClient Data Provider";User Instance=False;Context Connection=False;Transaction Binding="Implicit Unbind";ApplicationIntent=ReadWrite;MultiSubnetFailover=False;ConnectRetryCount=1;ConnectRetryInterval=10 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:56.71 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Files ak8dj High [Forced due to logging gap, Original Level: VerboseEx] UserAgent not available, file operations may not be optimized. 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:56.73 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (GetFileAndMetaInfo). Execution Time=313.310141372716 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:56.73 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (GetWebPartPageContent). Execution Time=313.504998540317 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:56.74 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Application Authentication aiv3l High Token request correlation id: '19aa4e9e-09f5-b03f-8e7c-65f772e21713'. 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:57.01 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Application Authentication ahi15 High [Forced due to logging gap, cached @ 02/28/2018 02:10:56.74, Original Level: Verbose] OAuth2 S2S Bearer Token: {0} 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:57.01 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Application Authentication age6f High [Forced due to logging gap, Original Level: Verbose] Pre-authenticating OAuth2 Bearer credentials. 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:57.01 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Application Authentication aiv3l High Token request correlation id: '19aa4e9e-09f5-b03f-8e7c-65f772e21713'. 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:57.07 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Web Controls 90i3 High [Forced due to logging gap, cached @ 02/28/2018 02:10:57.06, Original Level: Verbose] PersonalActions::CreateChildControls() - Was called. 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:57.07 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation General g3qe High [Forced due to logging gap, Original Level: Verbose] GetServerRelativeUrlFromUrl(_layouts/15/userdisp.aspx?Force=True,True,True) 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:57.07 w3wp.exe (0x0F24) 0x1A50 SharePoint Server Search Query dn4s High FetchDataFromURL start at(outside if): 1 param: start 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5
    02/28/2018 02:10:57.09 w3wp.exe (0x0F24) 0x1A50 SharePoint Foundation Micro Trace uls4 Medium Micro Trace Tags: 0 ajji6,2 agb9s,941 b4ly,314 b4ly,0 b4ly,9 aiv3l,262 aiv3l,73 dn4s 2aaa4e9e-d996-b03f-8e7c-655d5b935bd5

    Also, I assume that if it were a proxy problem or a trust problem then restarting the Workflow Manager Backend probably wouldn't resolve it for a while, so I don't think those are the problem (maybe the trust could, I'm not an expert on how that works)

    Any ideas on what is making our workflow service stop working after a while?

    Tuesday, February 27, 2018 6:30 PM

All replies

  • Hi,

     

    We are currently looking into this issue and will give you an update as soon as possible.

     

    Best Regards,

    Carl Zhou


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Thursday, March 1, 2018 1:45 AM
  • Hi Carl,

    I have some additional information for the problem I have with workflow manager.

    I have installed all of the updates now that come through windows updates.  I was able to get the servers to check updates from Microsoft so the app server and all of the SharePoint web servers are up to date on windows updates.  There are SharePoint updates available for SharePoint but I have not gone that far yet.  Basically I made sure that Service Bus 1.0 is on CU 3 and Workflow Manager 1.0 is on CU 4.

    Some other things seem very odd about this workflow deployment though.  The database SBMessageContainer01 is currently at about 59GB in size.  On other farms that I manage it is no where near that size, about 5GB at most.  In the event viewer I found that we get a lot of messages about "The maximum entity size has been reached or exceeded for Topic: 'WORKFLOWDEFAULTNAMESPACE:TOPIC:SHAREPOINT/DEFAULT/15606DEF-4EE8-449A-BED3-9663297EC201/D3AB6DCE-12F0-4B09-8333-32D27814DBFC/WFTOPIC'. Size of entity in bytes:27859065749, Max entity size in bytes: 6442450944"

    The full message is below:

    Service Bus exception swallowed at location RestoreWorkflowConsistencyRetryAsyncResult.HandleException. System.AggregateException: One or more errors occurred. ---> System.OperationCanceledException: A recoverable error occurred while interacting with Service Bus. Recreate the communication objects and retry the operation. For more details, see the inner exception. ---> Microsoft.ServiceBus.Messaging.QuotaExceededException: The maximum entity size has been reached or exceeded for Topic: 'WORKFLOWDEFAULTNAMESPACE:TOPIC:SHAREPOINT/DEFAULT/15606DEF-4EE8-449A-BED3-9663297EC201/D3AB6DCE-12F0-4B09-8333-32D27814DBFC/WFTOPIC'. Size of entity in bytes:27859065749, Max entity size in bytes: 6442450944..TrackingId:1a388e65-c701-4866-a489-5b827eb0beac_GSP13-APP01_BSP13-APP01,TimeStamp:3/13/2018 5:00:58 PM ---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: The maximum entity size has been reached or exceeded for Topic: 'WORKFLOWDEFAULTNAMESPACE:TOPIC:SHAREPOINT/DEFAULT/15606DEF-4EE8-449A-BED3-9663297EC201/D3AB6DCE-12F0-4B09-8333-32D27814DBFC/WFTOPIC'. Size of entity in bytes:27859065749, Max entity size in bytes: 6442450944..TrackingId:1a388e65-c701-4866-a489-5b827eb0beac_GSP13-APP01_BSP13-APP01,TimeStamp:3/13/2018 5:00:58 PM
    Server stack trace:
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)
    Exception rethrown at [0]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [1]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass17.<GetAsyncSteps>b__a(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [2]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [3]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpTransactionalAsyncResult`1.<GetAsyncSteps>b__36(TIteratorAsyncResult thisPtr, IAsyncResult a)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [4]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.EndSendCommand(IAsyncResult result)
       --- End of inner exception stack trace ---
    Server stack trace:
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.EndSendCommand(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.BatchManager`1.PerformFlushAsyncResult.OnSingleOperationCompleted(IAsyncResult result)
    Exception rethrown at [0]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.BatchManager`1.EndFlush(IAsyncResult result)
    Exception rethrown at [1]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.BatchManagerAsyncResult`1.OnBatchedCallback(IAsyncResult result)
       at Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(IAsyncResult result)
    Exception rethrown at [2]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.OnEndSend(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.OpenOnceManager.<>c__DisplayClass1.<Begin>b__0(IAsyncResult r, Boolean& output)
       at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult`1.OperationComplete(IAsyncResult result)
       at Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(IAsyncResult result)
    Exception rethrown at [3]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.Workflow.Management.ServiceBusNamespaceSender.SendInBatchAsyncResult.<>c__DisplayClassf.<OnMessageSent>b__e()
       at Microsoft.Activities.Hosting.ServiceBusUtilities.Isolate(Action callback, Boolean preserveExceptionMessage)
       --- End of inner exception stack trace ---
       at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.Workflow.Management.ServiceBusNamespaceSender.OnEndSendInBatch(IAsyncResult result)
       at Microsoft.Workflow.Management.BroadcastSendHelper.BroadcastSendAsyncResult`1.OnEndBranch(IAsyncResult branchResult)
       at Microsoft.Workflow.Common.ParallelAsyncResult.OnBranchCompleted(IAsyncResult result)
       --- End of inner exception stack trace ---
       at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.Workflow.Management.RestoreWorkflowConsistencyRetryAsyncResult.EndAsyncResult(IAsyncResult result)
       at Microsoft.Workflow.Common.BackoffRetryAsyncResult.IsolateWithRetry(IAsyncResult result)
    ---> (Inner Exception #0) System.OperationCanceledException: A recoverable error occurred while interacting with Service Bus. Recreate the communication objects and retry the operation. For more details, see the inner exception. ---> Microsoft.ServiceBus.Messaging.QuotaExceededException: The maximum entity size has been reached or exceeded for Topic: 'WORKFLOWDEFAULTNAMESPACE:TOPIC:SHAREPOINT/DEFAULT/15606DEF-4EE8-449A-BED3-9663297EC201/D3AB6DCE-12F0-4B09-8333-32D27814DBFC/WFTOPIC'. Size of entity in bytes:27859065749, Max entity size in bytes: 6442450944..TrackingId:1a388e65-c701-4866-a489-5b827eb0beac_GSP13-APP01_BSP13-APP01,TimeStamp:3/13/2018 5:00:58 PM ---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: The maximum entity size has been reached or exceeded for Topic: 'WORKFLOWDEFAULTNAMESPACE:TOPIC:SHAREPOINT/DEFAULT/15606DEF-4EE8-449A-BED3-9663297EC201/D3AB6DCE-12F0-4B09-8333-32D27814DBFC/WFTOPIC'. Size of entity in bytes:27859065749, Max entity size in bytes: 6442450944..TrackingId:1a388e65-c701-4866-a489-5b827eb0beac_GSP13-APP01_BSP13-APP01,TimeStamp:3/13/2018 5:00:58 PM
    Server stack trace:
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)
    Exception rethrown at [0]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [1]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass17.<GetAsyncSteps>b__a(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [2]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [3]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpTransactionalAsyncResult`1.<GetAsyncSteps>b__36(TIteratorAsyncResult thisPtr, IAsyncResult a)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
    Exception rethrown at [4]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.EndSendCommand(IAsyncResult result)
       --- End of inner exception stack trace ---
    Server stack trace:
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.EndSendCommand(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.BatchManager`1.PerformFlushAsyncResult.OnSingleOperationCompleted(IAsyncResult result)
    Exception rethrown at [0]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.BatchManager`1.EndFlush(IAsyncResult result)
    Exception rethrown at [1]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.BatchManagerAsyncResult`1.OnBatchedCallback(IAsyncResult result)
       at Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(IAsyncResult result)
    Exception rethrown at [2]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.OnEndSend(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.OpenOnceManager.<>c__DisplayClass1.<Begin>b__0(IAsyncResult r, Boolean& output)
       at Microsoft.ServiceBus.Messaging.OpenOnceManager.OpenOnceManagerAsyncResult`1.OperationComplete(IAsyncResult result)
       at Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(IAsyncResult result)
    Exception rethrown at [3]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.Workflow.Management.ServiceBusNamespaceSender.SendInBatchAsyncResult.<>c__DisplayClassf.<OnMessageSent>b__e()
       at Microsoft.Activities.Hosting.ServiceBusUtilities.Isolate(Action callback, Boolean preserveExceptionMessage)
       --- End of inner exception stack trace ---
       at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.Workflow.Management.ServiceBusNamespaceSender.OnEndSendInBatch(IAsyncResult result)
       at Microsoft.Workflow.Management.BroadcastSendHelper.BroadcastSendAsyncResult`1.OnEndBranch(IAsyncResult branchResult)
       at Microsoft.Workflow.Common.ParallelAsyncResult.OnBranchCompleted(IAsyncResult result)<---

    Does this information add any clues that I should check in to?  Is there a way I can see all of the SharePoint 2013 workflows and if there is one that is causing problems?

    Thanks,

    Eric

    Tuesday, March 13, 2018 5:26 PM