none
Service Bus Message Broker service fills up the CPU (100%) RRS feed

  • Question

  • Hi,

    yesterday we have a problem with our test environment with WFM and SB.

    We found Service Bus Message Broker service fill up the CPU.

    If we stop the Workflow backend service everything return "normal" (CPU 5% used), but when we started the Workflow backend service the CPU come back to 100%.

    In workflow Event Viewer we find the following error:

    Service Bus exception swallowed at location ServiceBusMessageSessionContext.OnLockRenewed.
    System.OperationCanceledException: A recoverable error occurred while interacting with Service Bus.
    Recreate the communication objects and retry the operation.
    For more details, see the inner exception. --->
    System.TimeoutException: The open operation did not complete within the allotted timeout of 00:00:08.8730368.
    The time allotted to this operation may have been a portion of a longer timeout.
    ---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]:
    The open operation did not complete within the allotted timeout of 00:00:08.8730368.
    The time allotted to this operation may have been a portion of a longer timeout.

    Server stack trace:
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.ThrowIfFaultMessage(Message wcfMessage)
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.HandleMessageReceived(IAsyncResult result)

    Exception rethrown at [0]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

    Exception rethrown at [1]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass17.<GetAsyncSteps>b__a(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

    Exception rethrown at [2]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

    Exception rethrown at [3]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpTransactionalAsyncResult`1.<GetAsyncSteps>b__36(TIteratorAsyncResult thisPtr, IAsyncResult a)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

    Exception rethrown at [4]:
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSession.OnEndRenewLock(IAsyncResult result)
       --- End of inner exception stack trace ---
       at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSession.OnEndRenewLock(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.MessageSession.EndRenewLock(IAsyncResult result)
       at Microsoft.Activities.Hosting.ServiceBusUtilities.Isolate(Action callback, Boolean preserveExceptionMessage)
       --- End of inner exception stack trace ---
       at Microsoft.Activities.Hosting.ServiceBusUtilities.Isolate(Action callback, Boolean preserveExceptionMessage)
       at Microsoft.Workflow.Dispatcher.ServiceBusMessageSessionContext.OnLockRenewed(IAsyncResult result)

    Someone has the same problem? What cause this problem?

    How can we solve this problem?

    Kind regards

    Samuele

    Thursday, December 19, 2013 9:51 AM

All replies

  • We remove the Server from the WFM farm, and then we join to the same WFM farm, but nothing change!

    Thursday, December 19, 2013 9:52 AM
  • is that issue resolved. I am also facing the same issue
    Tuesday, November 1, 2016 2:05 PM
  • Hi Samuele, there were a lot of performance improvements on the WFM with the Cumulative Update 3. Do you have that update installed, if not I strongly recommend that you do that and then check the performance.

     Cumulative Update 3 for Workflow Manager 1.0 (KB3104066)

    Must have the Cumulative Update for Service Bus 1.0 (KB2799752) installed on the entire computer farm before installing this update for Workflow Manager 1.0

    https://www.microsoft.com/en-us/download/details.aspx?id=36794

     

    This update can be applied to machines that have Workflow Manager 1.0 installed.

    If the machine has Workflow Manger 1.0 installed, only Workflow Manager will be updated.

    To update the Workflow Manager Client 1.0, you have to install Workflow Manager Client 1.0 CU3. Workflow Manager Client 1.0 CU3 and Workflow Manager Tools 1.0 CU3 are

    MSIs that can be installed on a fresh computer or in addition to Workflow Manager Client 1.0 or Workflow Manager Tools 1.0 respectively

     https://www.microsoft.com/en-us/download/details.aspx?id=50043

    Also, check my blog post that shows you how you can check if you have the latest CU3

    Five Steps to check your wfm

    Hope that helps

    Cheers

    Please mark as helpful if this was helpful.


    Thanks & Regards, Jose


    Tuesday, December 13, 2016 5:17 PM