none
ThreadPool deadlock? Throttling issue? RRS feed

  • Question

  • We are in the process of setting up a new project, using WCF. However, we found out that the problem we experience below is related to the CLR ThreadPool and most certainly not to WCF - that's why I'm posting it here.

     

    Our application has the following layout:

    [Clients (1-70)] calls [WCF Service (workflow and service orchestrations)] calls [WCF Service (basic hardware device controlling)]

     

    We are running custom load tests which are (request/response and one-way) sending random data to the services for now.

     

    If a certain amount of load is reached, the services stop responding. No more service calls are dispatched to service instances (InstanceContextMode = InstanceContextMode.PerSession, ConcurrencyMode = ConcurrencyMode.Multiple)). NetTcp / message security. Windows XP SP2 or Vista. Framework 3.5.

    Operation calls are hanging for up to 12 seconds (average 3-5). 0% CPU activity. Process thread count going up 2 / sec. WCF outstanding calls increasing.

    Then the CPU% will go up to 100% and after that the processing will continue normally.

     

    The problem is independent of the distribution of the services across machines.

    It'll happen on a single machine as well as on a distributed system with one machine per process.

    The problem will not happen if there is no call to the second service, i.e. when there is only one service. No garbage collection, WCF/TCP throttling or custom code deadlock issues.

     

    The problem is load dependent. It will happen when about 40 requests per second are running. Message sizes 50 - 500000 bytes.

     

    We replaced the CLR thread pool by IDesign's (Juval's) ThreadPoolSynchronizer (100) as the WCF service SynchronizationContext and the "blocking" is gone!

     

    Our ideas were that the problem arises with the CLR thread pool throttling (adding "only" 2 threads per second to pool, where the created threads are getting consumed by new pending operation calls immediately).

     

    -> Any ideas, what's wrong with the CLR ThreadPool, especially under WCF high performance situations?

    -> What is the recommende best practice?

     

    Thanks,

    Markus

    Tuesday, March 4, 2008 1:53 PM

Answers

  • This may have little to do with the thread pool.  In my services for the StockTrader app (sample on MSDN), which use just about every transport supported, I do the following as example for TCP, with good results for thousands of concurrent users hitting the WCF clients (ASP.NET app); talking to remote services hosted in IIS, WAS-hosted, or console/windows apps:

     

    1.  If using TCP as the transport, use a custom binding on host and set binaryEncoding (I have some values here jacked up too high for most services, so you can set things like maxReadPoolSize back down to defaults (I think 256), etc.:

     

    <binding name="Host_NodeSvc_CustomTcpBinding">

    <binaryMessageEncoding maxReadPoolSize="1024" maxWritePoolSize="1024" maxSessionSize="32768">

    <readerQuotas maxDepth="1024" maxStringContentLength="262144" maxArrayLength="262144" maxBytesPerRead="32768" maxNameTableCharCount="262144" />

    </binaryMessageEncoding>

    <tcpTransport manualAddressing="false" maxBufferPoolSize="524288" maxReceivedMessageSize="524288" connectionBufferSize="65536" hostNameComparisonMode="StrongWildcard" channelInitializationTimeout="00:01:00" maxBufferSize="524288" maxPendingConnections="200" maxOutputDelay="00:00:00.2000000" maxPendingAccepts="200" transferMode="Buffered" listenBacklog="500" portSharingEnabled="false" teredoEnabled="false">

    <connectionPoolSettings groupName="default" leaseTimeout="00:30:00" idleTimeout="00:30:00" maxOutboundConnectionsPerEndpoint="300" />

    </tcpTransport>

    </binding>

     

    2.  Make sure to assign a Service Behavior to the service (in config or in code, config below) that removes the default throttles on max instances, which is 10 by default, meaning only 10 active instances at a time (this is very important):

     

    <behavior name="NodeCommunicationBehaviors">

    <serviceDebug httpHelpPageEnabled="true" includeExceptionDetailInFaults="true" />

    <serviceThrottling maxConcurrentInstances="60" maxConcurrentCalls="60" />

    </behavior>

     

    3.  On clients, use a tcpBinding, with higher settings for max connections and listenBacklog, such as:

     

    <binding name="Client_TcpBinding" closeTimeout="00:00:45" openTimeout="00:00:45" receiveTimeout="00:02:00" sendTimeout="00:1:00" transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions" hostNameComparisonMode="StrongWildcard" listenBacklog="500" maxBufferPoolSize="524288" maxBufferSize="524288" maxConnections="300" maxReceivedMessageSize="524288">

    <readerQuotas maxDepth="1024" maxStringContentLength="262144" maxArrayLength="262144" maxBytesPerRead="32768" maxNameTableCharCount="262144" />

    <reliableSession ordered="true" inactivityTimeout="00:00:30" enabled="false" />

    <security mode="None">

    <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />

    <message clientCredentialType="Windows" />

    </security>

    </binding>

     

     

     

    Also, in the new StockTrader app (sample on MSDN) I am building, the new Config Service is multi-threaded itself (checking status of load balanced nodes, doing node notifications across clustered nodes for config changes, etc); AND for these operations I DO NOT USE THREADPOOL threads.  I simply create my own.  That is becuase some remote operations, if a service is not online or slow to respond, are simply too long running; and I want to keep the worker thread pool clean from these, and not use them lest I stack up .NET ThreadPool threads.  You can optionally implement your own thread pooling when creating your own threads not drawn from the thread pool, however, thread creation process is very fast and they get garbage collected properly if instancing correctly with the class that creates them.   Creating threads from .NET thread pool is one per .5 seconds with this delay built in, so if using thread pool need to ensure min free threads is high enough.  .NET 3.5 does change the thread pool behavior, as you note....also, in IIS values are set differently via machine.config, vs. the Thread.Pool API from windows/console/nt service apps...

     

     

    -Greg

     

    Thursday, March 6, 2008 2:43 AM

All replies

  • Hi Markus,

     

    Have you tried to tune ThreadPool using ThreadPool.SetMinThreads method?

     

    Tuesday, March 4, 2008 2:10 PM
  • Yes, also setting the min to 100 (1000 IOCP) did not help, which was also our first guess.

     

    Tuesday, March 4, 2008 2:23 PM
  •  

    I believe you can't set min higher than max which is default 25xcpu count . (you won't get an error, it just doesn't change min value).   I'm having similar problem, my test harness used to work fine with .net 3.0, I was able to warm up 500 threads but now after move to .net 3.5 my application is stack at 10 threads with awful 2 threads per second increase.  What's even worse changing to 3.0 in vs2008 doesn't fix it, I think the upgrade might have patched .net 3.0 thread pool.
    Tuesday, March 4, 2008 9:25 PM
  • This may have little to do with the thread pool.  In my services for the StockTrader app (sample on MSDN), which use just about every transport supported, I do the following as example for TCP, with good results for thousands of concurrent users hitting the WCF clients (ASP.NET app); talking to remote services hosted in IIS, WAS-hosted, or console/windows apps:

     

    1.  If using TCP as the transport, use a custom binding on host and set binaryEncoding (I have some values here jacked up too high for most services, so you can set things like maxReadPoolSize back down to defaults (I think 256), etc.:

     

    <binding name="Host_NodeSvc_CustomTcpBinding">

    <binaryMessageEncoding maxReadPoolSize="1024" maxWritePoolSize="1024" maxSessionSize="32768">

    <readerQuotas maxDepth="1024" maxStringContentLength="262144" maxArrayLength="262144" maxBytesPerRead="32768" maxNameTableCharCount="262144" />

    </binaryMessageEncoding>

    <tcpTransport manualAddressing="false" maxBufferPoolSize="524288" maxReceivedMessageSize="524288" connectionBufferSize="65536" hostNameComparisonMode="StrongWildcard" channelInitializationTimeout="00:01:00" maxBufferSize="524288" maxPendingConnections="200" maxOutputDelay="00:00:00.2000000" maxPendingAccepts="200" transferMode="Buffered" listenBacklog="500" portSharingEnabled="false" teredoEnabled="false">

    <connectionPoolSettings groupName="default" leaseTimeout="00:30:00" idleTimeout="00:30:00" maxOutboundConnectionsPerEndpoint="300" />

    </tcpTransport>

    </binding>

     

    2.  Make sure to assign a Service Behavior to the service (in config or in code, config below) that removes the default throttles on max instances, which is 10 by default, meaning only 10 active instances at a time (this is very important):

     

    <behavior name="NodeCommunicationBehaviors">

    <serviceDebug httpHelpPageEnabled="true" includeExceptionDetailInFaults="true" />

    <serviceThrottling maxConcurrentInstances="60" maxConcurrentCalls="60" />

    </behavior>

     

    3.  On clients, use a tcpBinding, with higher settings for max connections and listenBacklog, such as:

     

    <binding name="Client_TcpBinding" closeTimeout="00:00:45" openTimeout="00:00:45" receiveTimeout="00:02:00" sendTimeout="00:1:00" transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions" hostNameComparisonMode="StrongWildcard" listenBacklog="500" maxBufferPoolSize="524288" maxBufferSize="524288" maxConnections="300" maxReceivedMessageSize="524288">

    <readerQuotas maxDepth="1024" maxStringContentLength="262144" maxArrayLength="262144" maxBytesPerRead="32768" maxNameTableCharCount="262144" />

    <reliableSession ordered="true" inactivityTimeout="00:00:30" enabled="false" />

    <security mode="None">

    <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />

    <message clientCredentialType="Windows" />

    </security>

    </binding>

     

     

     

    Also, in the new StockTrader app (sample on MSDN) I am building, the new Config Service is multi-threaded itself (checking status of load balanced nodes, doing node notifications across clustered nodes for config changes, etc); AND for these operations I DO NOT USE THREADPOOL threads.  I simply create my own.  That is becuase some remote operations, if a service is not online or slow to respond, are simply too long running; and I want to keep the worker thread pool clean from these, and not use them lest I stack up .NET ThreadPool threads.  You can optionally implement your own thread pooling when creating your own threads not drawn from the thread pool, however, thread creation process is very fast and they get garbage collected properly if instancing correctly with the class that creates them.   Creating threads from .NET thread pool is one per .5 seconds with this delay built in, so if using thread pool need to ensure min free threads is high enough.  .NET 3.5 does change the thread pool behavior, as you note....also, in IIS values are set differently via machine.config, vs. the Thread.Pool API from windows/console/nt service apps...

     

     

    -Greg

     

    Thursday, March 6, 2008 2:43 AM
  • Thanks for reply Greg,

    I tried running my service and client with your bindings but didn't find any performance increase.  I'm using static channel factory to create client's channels and creating binding in-code so maybe I missed something in conversion.

    But it looks like in .net 3.5 the SetMinThreads doesn't work at all.

    http://www.michaelckennedy.net/blog/PermaLink,guid,708ee9c0-a1fd-46e5-8fa0-b1894ad6ce0f.aspx

    I'm just trying to figure out if I understand wcf multi threaded implementation right.

    For example if you have a service with :

    ConcurrencyMode = ConcurrencyMode.Multiple, InstanceContextMode = InstanceContextMode.PerCall

     there will be new instance created for each call and this new instance will be run on a thread from thread pool, so at the start of the service app you'll get 1 instance only, first 10 threads will be spun pretty quick but then it will slow down (not linear 2 per second anymore).  My service takes roughly 1second to process client request, so if I hit it with 100 simultaneous requests (clients) it runs out of threads really quick.  So unless you run multiple services I don't see a way to handle 1000 or even 100 simultaneous connections (within 1-3 sec delay on the client) . 

     

     

    P.S.  To further muddy the water, I think wcf uses i/o poolthreads so when setting SetMinThreads the i/o threads need to be increased as well.

    Thursday, March 6, 2008 11:44 PM
  • Greg, we factored out WCF bindings/throttling issues already - so as Gosha posts we also believe there is some quirk related to the thread pool. We experience a similar behavior as Michael posted.

     

    I read http://msdn2.microsoft.com/en-us/library/ms973903.aspx#threadpool_topic8 once again and think that just calling a WCF service (synch or asynchronously) from within another's WCF service operation (on an io completion port thread) brings us in this "deadlock" situation, because the thread pool throttling heuristics is (now, or was always) somewhat strange ...

     

    In the above link under "Deadlocks" see:

    ...

    If you want to avoid deadlocks in your applications, do not ever block a thread executed on the pool that is waiting for another function on the pool. This seems to be easy, but keep in mind that this rule implies two more:

    • Do not create any class whose synchronous methods wait for asynchronous functions, since this class could be called from a thread on the pool.
    • Do not use any class inside an asynchronous function if the class blocks waiting for asynchronous functions.

    If you want to detect a deadlock in your application, check the available number of threads on the thread pool when your system is hung. The lack of available threads and CPU utilization near 0% are clear symptoms of a deadlock.

    ...

     

    However, it looks as if we have still about 950 io threads available in the threadpool (up to the max), however the growing with 2/sec is too slow (consumed by other incoming requests immediately?).

     

    We have a Microsoft case running since February 7, they confirmed that they can reproduce our issue, but no solution so far.

     

     

    Friday, March 7, 2008 11:05 AM
  • I will continue to look into from my end; as I am doing some similar thingsand testing various blocks that can get encountered and how to avoid under high load. 

     

    I seem to be able to adjust thread pool sizes using the ThreadPool.SetMaxThreads/ThreadPool.SetMinThreads (both worker ann IOC)---querying during runs confirms this--I can see my settings take affect..

     

    You both seem very experienced; I may have missed it but I assume your app is in an .exe of some sort (NT Service, Win App, Console App); if in IIS/ASP.NET worker pool thread pool sizes are not set through this API, rather iin machine.config; with unknown results if you use the API, I believe...So if you can please tell me again:  what is client environment (ASP.NET?  .exe?); and what is HOST environment?

     

    Also, how WCF/.NET 3.5 respects my settings I need to do some investigation.  More to follow after some testing and a few emails...your test case is interesting, since I think you are saying every op takes 1 second (an eternity, really), and that is something you can't control; so I see why you quickly consume all worker threads especially if your test script is ONLY testing this single slow operation and no other work; and you would only be able to do as many req/sec as available worker threads, given the math....But I will investigate further with some tests/code...

     

    -Greg

    Monday, March 10, 2008 12:15 AM
  • Greg, in our case we self-host the WCF services in an one or two executables (both result, see me initial post above). Our operations return immediately, only the data chunk (raw 50B-500kB) transmission being relevant. We have similar binding settings as you showed.

     

    We may have brought you on a wrong track with WCF however, as NOT calling the second WCF service from within the first service gets rid of the problem (but is obviously no solution for us, only having a client and the intermediate service ;-)

     

    Furthermore, replacing the CLR ThreadPool with a custom built thread pool works great! However, we can hardly believe dumping the CLR thread pool is a good solution without other side effects.

     

    Monday, March 10, 2008 7:42 AM
  • Greg,

    Our client and service are console apps as well.  After adjusting ThreadPool.SetMaxThread (250, 5000) and ThreadPool.SetMinThreads (100, 2000)  (here I learned the hard way Smile to increase Max first or SetMinThread above 25XCpuCount will fail) we can see change take effect if we read GetMinThread.  Next we execute a loop in a client to call 1000 BeginMethods (Our singleton test service implements method which just sleeps for 5 second.), we expect to see service thread count to immediately jump to at least 100 but that doesn't happen.  We also made sure that our service behavior set up to properly.  We will try to test our implementation with your StockTrader boilerplate code and see if we can isolate the problem. 

    Gosha.

    Wednesday, March 12, 2008 12:52 AM
  • Seems to be exactly the same issue we experience:

     

    -> Setting ThreadPool Max/Min does not influence throttling (adding 2/sec) or Min not working!?

     

    Wednesday, March 12, 2008 9:21 AM
  • Hey guys, check out this blog entry for a follow up on the previous post quoted above:

    http://www.michaelckennedy.net/blog/PermaLink,guid,859d9d70-e822-418f-820c-ca91bed78dcf.aspx

    Regards,
    Michael
    Wednesday, April 23, 2008 3:20 PM
  • Markus,

    Did you ever figure out the solution here?  I'm hitting the same.  Also, can you expand on your statement: "replacing the CLR ThreadPool"?  How did you replace the CLR ThreadPool? 

    TIA
    Tuesday, June 10, 2008 7:33 PM
  • We were temporary (until Microsoft fixed the issue in 3.5 SP1) using a ThreadPool from IDesign Juval Lövy by just setting the SynchronizationContext of WCF

    // Set custom thread pool

    Context = new ThreadPoolSynchronizer(30);

    SynchronizationContext.SetSynchronizationContext(Context);

    After the fix, we go rid of it again - and seems to work fine.

    Monday, February 2, 2009 7:33 AM
  • We are still seeing this 2 new threads/sec behavior in our WCF service even after 3.5 SP1 and after setting ThreadPool.SetMax/MinThreads.
    We can see the same behavior with some of our old Remoting apps, a sudden burst of requests can only be handled 2/sec because of the time needed for new threads to start up.

    Could this ThreadPool bug have come back because of a hotfix or other issue?

    Right now our only alternative is to use Juval's custom ThreadPoolBehavior and avoid the .NET ThreadPool.
    Saturday, November 28, 2009 1:30 AM
  • we are having this exact same problem.  Hangs of about 5 to 10 seconds at a time randomly when we really push the WCF.  We solved a lot of the throttling problems with settings but we can't seem to get rid of this one.  We call wcf from the codebehind of a web page.

    This issue is definitely not fixed for us in .net 3.5 sp1.  Even with the 'trick' fix for keeping the IO thread hot.  The only thing that did for us is make the first call faster.

    We are going to try the custom threadpool and see if that gets rid of our problem.

     

    Monday, March 29, 2010 3:48 AM
  •  allen,

    I was in contact with MS about a QFE listed here: http://blogs.msdn.com/wenlong/archive/2010/02/11/why-are-wcf-responses-slow-and-setminthreads-does-not-work.aspx but that did not work for my situation.

    Is this the same IO thread hot trick you are talking about? http://blogs.msdn.com/wenlong/archive/2010/02/11/why-does-wcf-become-slow-after-being-idle-for-15-seconds.aspx

    I am still using Juval's custom thread pool in production and we have not had any issues with it so far ( 4 Months).

    Lets just hope this can get fixed in .NET 4.0 SP1

    • Edited by krisby Wednesday, March 31, 2010 3:49 PM formatting
    Wednesday, March 31, 2010 3:49 PM
  • Hello Devs

    Can Anyone tell me the latest news.

    I am suffering with almost lag 10ms between client and server where both are residing on same machine. but this increases with the passage of time like after 24hrs it become twice almost. 

    Some people said that this issue is still not fixed in 4.0 i havn't tried it yet but i am using 3.5sp1 i used SetMinThread(200,200) and this helped me to reduce this lag from 30ms to 10ms where there are only 50 concurrent users hitting the service with 1 avg sleep time. 

     

    What you guys suggest should i use the Default Threadpool or ThreadPool from IDesign Juval Lövy or should i shift to netfx 4.0.

     

    Please reply soon

    Friday, April 23, 2010 6:39 AM
  • I am using WinForms .NET 3.5Sp1. Would I be able to use that ThreadPool from IDesign Juval Lövy ?

    I am having maybe 10-25 ThreadPool threads in process, but am concern with 2 new thrads per second issue?

    Saturday, March 23, 2013 1:53 PM