none
Slow Performance RRS feed

  • Question

  • Hi!

    I'm working in a time-critical multi processed application made in C# 2.0... Almost all the time, the response time is quite good (something around 20 milliseconds)... Well, it's actually too slow, and we're working on that, trying to improve the overall performance...

    My problem is this:

    In the production environment, a 32 core + 64GB RAM machine + running OS WIN2003 Server x64 SP2,  the application freezes in nondeterministics circunstances... Being the application a real-time one, freezing by 20 seconds (sometimes is more than it) is something almost criminal...

    During this dreadful 20 seconds, all the Threads stop whatever they're doing... After this period of time, everything start to work again...

    Using a sniffer I found that TCP messages were arriving normally to our hardware, but the system (listening that tcp ports) didn't received anything... I've also analyzed almost all the performance counters collected, but I couldn't find anything strange (only this freezing time in my application, and, consequently, in the disk activity, because the application wasn't logging anything)...

    That are no other applications running on this machine... So, I don't know what more do I have to check...

    Do you have any idea, or suggestion about how to diagnose the problem?

    I wish you could help me in my search to find what is hapenning...

    This is a part of my app's Log:

    [05/05/2009 15:10:36.882] (440) TRACE   Thread ID: 440 [Message Sent (FIXApplication.SendMessage)]

    ??? NOTHING HAPPENING HERE ???

    [05/05/2009 15:10:56.194] (748) TRACE   Thread ID: 748 [(FIXApplication.fromApp)]

    Thank you in advance...

    Alexandre Eduardo de Oliveira

    Wednesday, May 27, 2009 6:22 PM

All replies

  • Check the memory pressure on the machine, you may be running into garbage collection issues. Check this article http://msdn.microsoft.com/en-us/library/cc713687.aspx and this http://channel9.msdn.com/shows/ARCast.TV/ARCastTV-Steve-Michelotti-of-eimagination-on-High-Performance-Web-Solutions/.

    HTH
    Wednesday, May 27, 2009 7:00 PM
  • Hmm, 20 seconds is rather long.  It can't be the finalizer thread, it has a 2 second timeout.  Use Perfmon.exe and look at the garbage collection counters to see if you can find a correlation. 

    You'll also want to edit the app.config file so you don't get server collection behavior.  Use the <gcServer> element , enabled="false".  You cannot assume you'll ever get time-critical real-time behavior with a garbage collector.
    Hans Passant.
    Wednesday, May 27, 2009 7:00 PM
    Moderator
  • As you asked me, I looked at performance counter once more trying to find some correlation between the "vacuum" time... Well, some considerations about:

    - The GC counters seems to be ok. There are no expensive collections being realized during the periodo of freezing...
    - The only performance counter that is changing (increasing) in the exact moment of the freezing situation is the File Read Operations/sec

    I'll try to found out what this counter is supposed to be, and check if there's a correlation with the problem...

    Thank you guys!

    Alexandre
    Thursday, May 28, 2009 8:03 PM
  • If you're doing a lot of IO try using the non-blocking operations like BeginRead/BeginWrite instead of synchronous Read/Write. This will free up your pool threads to pick up new tasks.
    Thursday, May 28, 2009 8:13 PM
  • Watch out for the paging file.  Nothing quite like a .NET program trying to page-in a bunch of JIT compiled code on an old machine with a heavily fragged paging file.  If that's it, tinker with Process.MinWorkingSet.  Or use Ngen.exe.

    Hans Passant.
    Thursday, May 28, 2009 11:42 PM
    Moderator
  • are you possible writing to a NAS or iSCSI storage that might incur packet losses, etc. causing IO operations to go crazy?
    Friday, May 29, 2009 12:31 AM