none
Thread Starvation: 'Maximum engine threads' vs 'Maximum worker threads' vs 'Maximum IO threads'. What's the difference?

    Question

  • Hello all,   

        'Maximum engine threads' vs 'Maximum worker threads' vs 'Maximum IO threads'. What's the difference between these thread settings and what will most likely help my thread starvation problem described below? I noticed Max engine threads is set only on the host and the others are set on the instance.

    ----------------------------------------
    A SOAP Fault message was encountered by BizTalk.
    No port information was available in the SOAP Fault message properties.
    Part[1]: body
    --------------------------------------------------------------------------------
    <env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope">
    <env:Body>
    <env:Fault>
    <env:Code>
    <env:Value>MicrosoftBizTalkServerNegativeAcknowledgment</env:Value>
    </env:Code>
    <env:Reason>
    <env:Text>System.InvalidOperationException: There were not enough free threads in the ThreadPool to complete the operation.
    
    Server stack trace: 
    at System.Net.HttpWebRequest.BeginGetRequestStream(AsyncCallback callback, Object state)
    at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStreamAsyncResult..ctor(HttpWebRequest httpWebRequest, HttpOutput httpOutput, AsyncCallback callback, Object state)
    at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.BeginGetOutputStream(AsyncCallback callback, Object state)
    at System.ServiceModel.Channels.HttpOutput.SendAsyncResult.SendWithoutChannelBindingToken()
    at System.ServiceModel.Channels.HttpOutput.SendAsyncResult.Send()
    at System.ServiceModel.Channels.HttpOutput.SendAsyncResult..ctor(HttpOutput httpOutput, HttpResponseMessage httpResponseMessage, Boolean suppressEntityBody, TimeSpan timeout, AsyncCallback callback, Object state)
    at System.ServiceModel.Channels.HttpOutput.BeginSendCore(HttpResponseMessage httpResponseMessage, TimeSpan timeout, AsyncCallback callback, Object state)
    at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelAsyncRequest.SendWebRequest()
    at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelAsyncRequest.BeginSendRequest(Message message, TimeSpan timeout)
    at System.ServiceModel.Channels.RequestChannel.BeginRequest(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
    at System.ServiceModel.Dispatcher.RequestChannelBinder.BeginRequest(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
    at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.StartSend(Boolean completedSynchronously)
    at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.Begin()
    at System.ServiceModel.Channels.ServiceChannel.BeginCall(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, TimeSpan timeout, AsyncCallback callback, Object asyncState)
    at System.ServiceModel.Channels.ServiceChannel.BeginRequest(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
    at System.ServiceModel.Channels.ServiceChannel.BeginRequest(Message message, AsyncCallback callback, Object state)

    Thanks,

    JDN717


    JDN


    • Edited by jdn717 Thursday, November 3, 2016 9:08 PM
    Thursday, November 3, 2016 9:07 PM

Answers

  • Hi

    In the case of the SOAP adapter to avoid this problem you have to increase the MaxIOThreads and MaxWorkerThreads values to atleast 100 each and take it from there. You may have to increase these values further if you keep running into this error.

    Refer - https://msdn.microsoft.com/en-us/library/cc594552(v=bts.10).aspx

    You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host Instances -> Select the Host Instance -> Settings) . You have to restart the Host Instance afterwards-

    DWORD entry Default value Recommended value

    MaxIOThreads

    25

    100

    MaxWorkerThreads

    25

    100


    Thanks Arindam


    Friday, November 4, 2016 7:06 AM
    Moderator
  • Hi JDN,

    You can refer the article: Troubleshooting BizTalk Server SOAP Adapter

    Snippet from the blog:

    Within an instance of a BizTalk host where SOAP, HTTP, or WSE Adapters are running, worker threads handle queued work items, and I/O threads dedicated callback threads associated with I/O completion ports for completed asynchronous I/O requests. Performance problems occur when there are not enough free threads in the thread pools to handle the number of messages (SOAP, HTTP, or WSE requests). When a BizTalk host starts, the default thread pool sizes are small and cannot fulfill the requests in a floodgate scenario. As a result, the adapter must add more worker or I/O threads to the thread pool. This process can be time consuming. More worker or I/O threads are added until the requests can be fulfilled or the maximum thread limit is reached. If the maximum thread pool limit is not large enough to handle the sustained work load, messages must wait until a thread becomes available before they can be processed.

    Avoid Thread Starvation
    Set minimum worker and I/O thread pool size to a level appropriate to the initial and sustained work load. Maximal worker thread pool size should be set to accommodate possible peak burst load. Sustained and burst load should be defined by the business requirements and validated during your stress testing. We recommend that you do not define excessive thread pool sizes. While large thread pools may resolve thread starvation, they can increase context switching, a situation where Windows switch from one running thread to the other. Excessive context switching can offset the performance gained Add CLR Hosting key to the previous listed registry path if the key does not exist already.
    MaxIOThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 20
    • Max:
    • Min:
    • Key: DWORD
    MaxWorkerThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 25
    • Max:
    • Min:
    • Key: DWORD
    MinIOThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 1
    • Max:
    • Min:
    • Key: DWORD
    MinWorkerThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 1
    • Max:
    • Min:
    • Key: DWORD
    Calculate the value of the MinWorkerThreads DWORD entry, using the following formula:
    (Max number of messages adapter receives at initialization) + (10 percent)
    For example, if 50 messages are delivered to the SOAP adapter as soon as the adapter is initialized, set the value of MinWorkerThreads to 55.
    Note the following:
    • MaxIOThreads and MinIOThreads should have the same values as MaxWorkerThreads and MinWorkerThreads.
    • If MaxWorkerThreads or MinWorkerThreads are configured, but not MaxIOThreads or MinIOThreads, BizTalk Server sets MaxIOThreads and MinIOThreads to the values of MaxWorkerThreads or MinWorkerThreads.
    • The values specified for the thread pools are per CPU. For example, setting MaxWorkerThreads to 100 has an effective value of 400 on a 4 CPU computer.
    These numbers are recommended starting points. Perform appropriate testing to determine the optimal setting for your business requirements.

    Rachit Sikroria (Microsoft Azure MVP)

    Friday, November 4, 2016 6:46 AM
    Moderator

All replies

  • Hi JDN,

    You can refer the article: Troubleshooting BizTalk Server SOAP Adapter

    Snippet from the blog:

    Within an instance of a BizTalk host where SOAP, HTTP, or WSE Adapters are running, worker threads handle queued work items, and I/O threads dedicated callback threads associated with I/O completion ports for completed asynchronous I/O requests. Performance problems occur when there are not enough free threads in the thread pools to handle the number of messages (SOAP, HTTP, or WSE requests). When a BizTalk host starts, the default thread pool sizes are small and cannot fulfill the requests in a floodgate scenario. As a result, the adapter must add more worker or I/O threads to the thread pool. This process can be time consuming. More worker or I/O threads are added until the requests can be fulfilled or the maximum thread limit is reached. If the maximum thread pool limit is not large enough to handle the sustained work load, messages must wait until a thread becomes available before they can be processed.

    Avoid Thread Starvation
    Set minimum worker and I/O thread pool size to a level appropriate to the initial and sustained work load. Maximal worker thread pool size should be set to accommodate possible peak burst load. Sustained and burst load should be defined by the business requirements and validated during your stress testing. We recommend that you do not define excessive thread pool sizes. While large thread pools may resolve thread starvation, they can increase context switching, a situation where Windows switch from one running thread to the other. Excessive context switching can offset the performance gained Add CLR Hosting key to the previous listed registry path if the key does not exist already.
    MaxIOThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 20
    • Max:
    • Min:
    • Key: DWORD
    MaxWorkerThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 25
    • Max:
    • Min:
    • Key: DWORD
    MinIOThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 1
    • Max:
    • Min:
    • Key: DWORD
    MinWorkerThreads
    • Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$Hostname
    • Enumeration: count
    • Default: 1
    • Max:
    • Min:
    • Key: DWORD
    Calculate the value of the MinWorkerThreads DWORD entry, using the following formula:
    (Max number of messages adapter receives at initialization) + (10 percent)
    For example, if 50 messages are delivered to the SOAP adapter as soon as the adapter is initialized, set the value of MinWorkerThreads to 55.
    Note the following:
    • MaxIOThreads and MinIOThreads should have the same values as MaxWorkerThreads and MinWorkerThreads.
    • If MaxWorkerThreads or MinWorkerThreads are configured, but not MaxIOThreads or MinIOThreads, BizTalk Server sets MaxIOThreads and MinIOThreads to the values of MaxWorkerThreads or MinWorkerThreads.
    • The values specified for the thread pools are per CPU. For example, setting MaxWorkerThreads to 100 has an effective value of 400 on a 4 CPU computer.
    These numbers are recommended starting points. Perform appropriate testing to determine the optimal setting for your business requirements.

    Rachit Sikroria (Microsoft Azure MVP)

    Friday, November 4, 2016 6:46 AM
    Moderator
  • Hi

    In the case of the SOAP adapter to avoid this problem you have to increase the MaxIOThreads and MaxWorkerThreads values to atleast 100 each and take it from there. You may have to increase these values further if you keep running into this error.

    Refer - https://msdn.microsoft.com/en-us/library/cc594552(v=bts.10).aspx

    You can change the below values from Host Instance Settings in Admin Console(Platform Settings -> Host Instances -> Select the Host Instance -> Settings) . You have to restart the Host Instance afterwards-

    DWORD entry Default value Recommended value

    MaxIOThreads

    25

    100

    MaxWorkerThreads

    25

    100


    Thanks Arindam


    Friday, November 4, 2016 7:06 AM
    Moderator
  • Thank you for this. I will check and revert back to you if successful.

    JDN

    Friday, November 4, 2016 11:18 PM