Service Bus Message Broker service fills up the CPU (100%)

Question

Hi,

yesterday we have a problem with our test environment with WFM and SB.

We found Service Bus Message Broker service fill up the CPU.

If we stop the Workflow backend service everything return "normal" (CPU 5% used), but when we started the Workflow backend service the CPU come back to 100%.

In workflow Event Viewer we find the following error:

Service Bus exception swallowed at location ServiceBusMessageSessionContext.OnLockRenewed.
System.OperationCanceledException: A recoverable error occurred while interacting with Service Bus.
Recreate the communication objects and retry the operation.
For more details, see the inner exception. --->
System.TimeoutException: The open operation did not complete within the allotted timeout of 00:00:08.8730368.
The time allotted to this operation may have been a portion of a longer timeout.
---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]:
The open operation did not complete within the allotted timeout of 00:00:08.8730368.
The time allotted to this operation may have been a portion of a longer timeout.

Server stack trace:
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)

Exception rethrown at [0]:
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

Exception rethrown at [1]:
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass17.<GetAsyncSteps>b__a(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

Exception rethrown at [2]:
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

Exception rethrown at [3]:
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpTransactionalAsyncResult`1.<GetAsyncSteps>b__36(TIteratorAsyncResult thisPtr, IAsyncResult a)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

Exception rethrown at [4]:
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSession.OnEndRenewLock(IAsyncResult result)
   --- End of inner exception stack trace ---
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSession.OnEndRenewLock(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.MessageSession.EndRenewLock(IAsyncResult result)
   at Microsoft.Activities.Hosting.ServiceBusUtilities.Isolate(Action callback, Boolean preserveExceptionMessage)
   --- End of inner exception stack trace ---
   at Microsoft.Activities.Hosting.ServiceBusUtilities.Isolate(Action callback, Boolean preserveExceptionMessage)
   at Microsoft.Workflow.Dispatcher.ServiceBusMessageSessionContext.OnLockRenewed(IAsyncResult result)

Someone has the same problem? What cause this problem?

How can we solve this problem?

Kind regards

Samuele

Answer 1

We remove the Server from the WFM farm, and then we join to the same WFM farm, but nothing change!

Answer 2

is that issue resolved. I am also facing the same issue

Answer 3

Hi Samuele, there were a lot of performance improvements on the WFM with the Cumulative Update 3. Do you have that update installed, if not I strongly recommend that you do that and then check the performance.

 Cumulative Update 3 for Workflow Manager 1.0 (KB3104066)

Must have the Cumulative Update for Service Bus 1.0 (KB2799752) installed on the entire computer farm before installing this update for Workflow Manager 1.0

https://www.microsoft.com/en-us/download/details.aspx?id=36794

 

This update can be applied to machines that have Workflow Manager 1.0 installed.

If the machine has Workflow Manger 1.0 installed, only Workflow Manager will be updated.

To update the Workflow Manager Client 1.0, you have to install Workflow Manager Client 1.0 CU3. Workflow Manager Client 1.0 CU3 and Workflow Manager Tools 1.0 CU3 are

MSIs that can be installed on a fresh computer or in addition to Workflow Manager Client 1.0 or Workflow Manager Tools 1.0 respectively

 https://www.microsoft.com/en-us/download/details.aspx?id=50043

Also, check my blog post that shows you how you can check if you have the latest CU3

Five Steps to check your wfm

Hope that helps

Cheers

Please mark as helpful if this was helpful.

---

Thanks & Regards, Jose