none
Blocking behaviour on a long-running WCF Operation RRS feed

  • Question

  • Hello Fellow Developers,

    I have a minimal performance testing solution including following projects:

    • WcfPerf.Contract - the actual Service definition and implementation
    • WcfPerf.Host - service host as a console application
    • WcfPerf.Consumer

    The solution(~25KiB) can be downloaded from http://l3v.pl/WcfPerf.zip

    The Service Behaviour is set to:

    InstanceContextMode = InstanceContextMode.Single;
    ConcurrencyMode = ConcurrencyMode.Multiple;

    So it is a singleton with no synchronization, except the synchronization I provide.

    The one-and-only operation is:

            public BillResponse RequestBilling(BillRequest request)
            {
                Thread.Sleep(1000);
                return new BillResponse { CreditAfter = 1, CreditBefore = 2, Success = true };
            }
    

    And the consumer ,,nukes" this Service with 200 threads in a tight loop using net.pipe transport. (As to simulate a lot of simultaneous client requests).

    And those threads seem blocked by the previous requests.

    I would expect that the performance would be near <nuking-thread-count> / sec, due to the 1 sec sleep. However, it is far lower, it reaches around 5 calls/sec and slowly raises, until a timeout exception is thrown by the Consumer.

    This is unacceptable, and I cannot find the reason of this behavior. I've found some articles which I read and tried the solutions, yet, they do not work for me. Threads/Articles I tried(some of which are IIS-based):

    • http://social.msdn.microsoft.com/Forums/en/wcf/thread/7757619e-68ee-4ca0-9e19-513b91eae872
    • http://social.msdn.microsoft.com/forums/en-US/wcf/thread/95daa492-f4a2-474e-9772-7b10f9f5eda2/
    • http://social.msdn.microsoft.com/forums/en-US/wcf/thread/34b347df-37f3-42d7-9723-f676f3616597
    • http://social.msdn.microsoft.com/forums/en-US/wcf/thread/34b347df-37f3-42d7-9723-f676f3616597
    • and a couple more I cannot recall Urls to right now.
    (I think the problem is not resolved in any of the threads in social.msdn/forums/, yet these are marked as "Answered". Please correct me if I'm wrong)

    I tried using serviceThrottling, maxConnections in net.pipe bindings, tried using the basicHttpBinding, and wsHttpBinding as the transport layer. And it all comes down to the same result. Tried affecting the Reliable-Session and Reliable-Messaging where those apply. Yeah, still, performance I would rather expect from a PHP application ;-).

    What am I missing? Am I not understanding some ground-basics of WCF?

    Or, if I want to achieve the maximum performance, did I took the wrong path? Should I choose another transport/binding?

    Any help would be greatly appreciated, as if it proves to be slow, still, I would have to revert to Remoting, which is a thing I would like to avoid.


    -- Regards, Luke.
    • Edited by L3vi4than Wednesday, March 24, 2010 10:34 AM Source-Code formatting broken.
    Wednesday, March 24, 2010 10:29 AM

All replies

  • WCF uses the .NET Threadpool under the covers and as such is limited to how many threads it can handle in parallel. So it's not going to scale up to an unlimited number of threads. And as too many threads are added to the Threadpool it will start generating Exceptions, which in turn will smash your performance.

     

    Best way to handle this is to start your own Threads an do the ThreadMangement yourself. Or you could always loadbalance your WCF service and spread the load over multiple instances.

    Thursday, March 25, 2010 1:56 PM
  • WCF uses the .NET Threadpool under the covers and as such is limited to how many threads it can handle in parallel. So it's not going to scale up to an unlimited number of threads. And as too many threads are added to the Threadpool it will start generating Exceptions, which in turn will smash your performance.

    Yet the hosting process starts with just 20 threads, reaches around 40, very slowly increasing their numbers, and the threads seem to be blocking each following request.

    After:

    http://msdn.microsoft.com/en-us/library/system.threading.threadpool.aspx

    "The thread pool has a default size of 250 worker threads per available processor, and 1000 I/O completion threads." - it is being run on a Core i7, 4 physical processors, 8 logical. The threadpool can reach go way beyond ~40 threads(verified with Process Explorer). P.S. I tried calling the ThreadPool.SetMinThreads, needless to say, it does absolutely nothing towards the application's performance or the thread spawning strategy. It is still heavily below the expected worker thread count.

    Best way to handle this is to start your own Threads an do the ThreadMangement yourself. Or you could always loadbalance your WCF service and spread the load over multiple instances.

    How do I handle ThreadManagement myself if the ServiceHost does it under the hood?

    I am not eager to load-balance a "Hello World" solution over a server-farm, really, it should just work, it is as simple as the solution I provided. If the meaningless Sleep() is removed, the throughput of the Service reaches thousands of calls per second. This means, that a lengthy-operation is apparently blocking the WCF dispatcher/scheduler somehow.


    -- Regards, Luke.
    Thursday, March 25, 2010 2:37 PM
  • Hi Luke,

    How did you spawn 200 threads to call the service operation simultaneously? Are you using a single client proxy object or creating multiple client proxy to call the service operation concurrently? If you use single client proxy, it is possible that the bottleneck is at client-side, you can try creating multiple service proxies (and also make sure the service-side has turn off reliablesession or set max session limit to large value) to see whether it makes any difference.

     

     


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Friday, March 26, 2010 8:17 AM
    Moderator
  • Hi Luke,

    How did you spawn 200 threads to call the service operation simultaneously? Are you using a single client proxy object or creating multiple client proxy to call the service operation concurrently? If you use single client proxy, it is possible that the bottleneck is at client-side, you can try creating multiple service proxies (and also make sure the service-side has turn off reliablesession or set max session limit to large value) to see whether it makes any difference.

    I'm spawning those with:

                for (int i = 0; i < 200; i++)
                    new Thread(ThreadProc).Start(i);
    

    And a WCF proxy is created per-thread, i.e. it's not shared between threads. If I recall correctly sharing the proxy resulted indeed in a client-side blocking. Yet, this does not apply here.

    The complete code is available in the .zip package I linked.


    -- Regards, Luke.
    Friday, March 26, 2010 9:29 AM
  • After some research I think that the client is the blocking/slow side, rather than the host. Any ideas?
    -- Regards, Luke.
    Wednesday, March 31, 2010 11:49 AM
  • Dear Luke

    If you are using .Net client then you may not have set 

     

    //This says how many outgoing connection you can make to a single //endpoint. Default Value is 2
    System.Net.ServicePointManager.DefaultConnectionLimit = 200;

    But its doesn't end here please read the following as well 

     

    You are right problem is not at the server end but its on the client side i am also trying to build a test client for load test for my wcf service and i am facing the same problem, To check which side is doing bad I added performance counters in WCF Service functions and Client Proxy class. And i found there is a lag of 30-40ms between client to server and then back client. 

    I added this question on stackflow.com here is the link http://stackoverflow.com/questions/2637175/wcf-network-cost

     

    i have tried whatever you have done like checking different kind of bindings, throttling configs, per session, per call, reliable sessions and all that. And the bigger problem is that response time increases by the time. 

     

    Well i am not going to give you any solution but i am here to tell you that you are not alone who is suffering but if you found any thing please share and i ll do the same.

     

    Regards

    Mubashar

     

    Friday, April 16, 2010 10:41 AM