locked
.net concurrent server performance problems with quad core and up RRS feed

  • Question

  • Hello,

    So to start off with I just wanted to ask a general question to the community. I might try in the future to write some simplified code when I hone in on a possible problem area, but for now your thoughts/experiences are greatly appreciated!  I have a program that is a TCP/IP concurrent server that accepts messages from clients and adds data to sql server. The program operates lightning fast on single and dual core processors, but hits some significant "stuttering" when operating on a eight core system (and some program performance degradation on a quad core). This performance hurdle is seen with a single client and with the database interaction turned off. So it's something with how my managed code (or code in general) interacts with the OS.  I've kept great pains to keep everything threadsafe (and it doesn't have any issues on a dual core with multiple clients). I'm in the process of trying to get a comparable system and put a development environment to try to sort this out... but has anyone ever experienced this? Any ideas what would cause this scaling issue?

    The server is a dell poweredge r610.

     

    Thanks in advance for your time and for your help!

     

    Scott

    Friday, August 27, 2010 8:02 PM

Answers

All replies

  • Hi Scott

    please provide the .NET and Windows versions  (make sure to get all the updates if using .NET 2.0/3.x).

    Use the Windows Task Manager to set the Processor Affinity of your server process to only 1 core, then redo your tests.
    Friday, August 27, 2010 8:37 PM
  • Hi Thomas,

    Thanks for replying!

    Sorry about that... the app was compiled against .net 2.0 and the target machine has .net 3.5 sp1 with all the updates.. When I set affinity to 1 or 2 then everything is just dandy. I also rebooted to a single core through msconfig and I had the same results as setting the affinity.

    The windows version is Windows Server 2008 with all the latest updates.

     

    Thanks,

     

    Scott

    Friday, August 27, 2010 9:36 PM
  • When I set affinity to 1 or 2 then everything is just dandy.

    I would rate this as an indication for code side issues, not OS/Framework.
    Ok, how do you use sockets, which classes/methods?
    Using asynchrous operations (BeginX/EndX)?
    Are you accessing your socket instances from other/many threads?
    Are you modifying any socket options/timeouts/flags?

    Can you quantify  "significant stuttering", are this delays in the milliseconds or seconds?

    MS has significantly improved the scaling/performance of .net sockets for servers, read:
    http://msdn.microsoft.com/en-us/magazine/cc300760.aspx
    http://msdn.microsoft.com/en-us/magazine/cc163356.aspx

    thus I assume they also tested on many-core systems.
    Friday, August 27, 2010 10:33 PM
  • Hi Thomas,

     

    Oh no I don't want you get the wrong idea. I'm not sure where the problem is occurring... I was hoping to find some people that might have come across similar issues (with coding, framework, or the hardware I mentioned). Don't want to blame anyone! I will definitely read those articles though.

     

    As for the questions, I don't use asynch operations.

     

    The first thread is a data handler thread. This thread checks a queue for new messages and updates the database as appropriate. There is a lock on the queue prior to the handling of the data and the count check of the queue. There is a second thread that has a listener that spawns a client handler (new thread) when receiving a connection. This handler thread reads xml messages from the socket. The messages are then added to the queue of the first thread (once again locking it before adding). The queue being the common object between threads. So basically each thread is broken into a single task of work:

     

    The listener thread receives connections and creates threads to handle the connection/client.

    The client thread reads xml messages from the socket and puts them in a queue

    The data handler thread dequeues messages and updates the database.

     

    The actual database CRUD interaction was taken out to rule it out as a factor and only a single client connection is used to replicate the problem. The stuttering is seen on the number of messages processed. It has a variation as to how long it lasts and the frequency that it occurs. I'd say the max stutter time would be a second with it occuring every few seconds (maybe 4 or 5?)

    I hope this info helps out. I'm just not sure why this would happen on multicores and not single or dual cores. The single core/dual core have no problem even at insanely high speeds! I appreciate your help and hope you might be able to nudge me in the right direction as this issue is blocking us badly. Have you come across problems that only seem to occur on multicore systems?  My only option at this point is to programatically set the affinity to one or two cores.  Next week when I have access to the hardware I'm going to try to write a simple project that duplicates our problem and perhaps post it. You wouldn't happen to have access to a machine with more than 4 logical cpus?

     

    Thanks again,


    Scott

    Saturday, August 28, 2010 12:04 AM
  • The client thread reads xml messages from the socket and puts them in a queue
    are there any socket 'Send' (write) operations, for the response message?
    If yes, are they done by the same thread as the 'Receive'?

    Is your client process running on the same server machine, or over LAN?
      ( http://support.microsoft.com/kb/2020447 )

    > machine with more than 4 logical cpus?

    I have one with 4, another with 2.
    Saturday, August 28, 2010 1:29 AM
  • Hey Thomas,

     

    The client thread does not send any response upon receiving the message. The client is on the same machine but I don't believe it's connecting through the loopback address.  I'm not sure if the messages are >64kb but I'll check...That's a really good find Thomas! On Monday I'm going to see with another developer if we can set up a system to connect to this machine from the LAN (unfortunately the new version of the client has a breaking change) and see if that changes anything.

    I'll let you know what happens on Monday and if you find anything else on this let me know!!!

     

    Thanks,

     

    Scott

     

    Saturday, August 28, 2010 2:49 AM
  • I found a (unresolved?) problem report with some parallelism (but high client count?):
    http://stackoverflow.com/questions/319732/tips-techniques-for-high-performance-c-server-sockets
     "delays of up to 100s of milliseconds"

    If your messages are small,
    you could see a 200-ms delay due 'Nagle' algrithm
    http://support.microsoft.com/kb/214397
    http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.nodelay.aspx

    • Proposed as answer by Alan_chen Wednesday, September 1, 2010 5:52 AM
    • Marked as answer by Alan_chen Friday, September 3, 2010 3:42 AM
    Saturday, August 28, 2010 7:50 AM