Challenging debugging problem. RRS feed

  • Question

  • I've never seen anything quite like this before and I'm out of ideas.

    I have a "client" (console) test program and a "server" (console) test program, these are being used to stress test new async client/server message passing code that sits on top of TCP sockets.

    The code behaves very well, clients can send approx a million messages per second to the server (both running on localhost) which is running in a loop on Socket.Accept(). Each time a client connects the severs starts an async message processor instance which internally uses only worker and IOCP threads, the coms code contains no sleeps, no blocking calls and uses no dedicated threads (nor does the client side, they both use same lib for IO).

    Now I have a problem, if I start say 40 instances of the client app, some of then encounter an exception, this clearly only occurs when twenty or so clients are fighting for CPU etc and is a bug I'd like to track down and fix.

    But this only happens when 30 or 40 are started via a .bat files that starts one per second then exits.

    I put a Console.Write in the exception handler and can see that it does hit this point, but then dies.

    I have followed the Console.WriteLine with a Sleep(-1), Debugger.Break(), Debugger.Launch() and a Console.ReadLine().

    None of these help, the process gets the exception, does the Console.WriteLine then dies.

    Because it always diesm the prompt to launch VS debugger and atttach to it etc, fails because the process has exited from the system.

    The code is clearlt doing something very bad that causes it to brutally die no matter what I try to do to keep it running.

    I can't just run a debug session in VS 2010, because the problme only start to happen when twenty or so instances are all starting up.

    I just tried manually starting a debug session of the EXE while a bunch were failing like this, buy that too failed, the VS2010 debug session just exited, so the problem did happen under that debug session but the exe still died.

    Does anyone know how to progress the investigation of such an isssue?

    All I can see is the very brief text from the Console.WriteLine() in the exception handler, but then it dies, I barely get a chance to read the text before the console window vanishes.

    Incidentally, by design an async worker or IO callback thread that gens an exception, will always call into the handler, so the exception migght be ocurring in any thread.

    I may add some sort of crude file write operation to record the details of the exceoption, but I'm still stumped at how to handle this kind of issue, I have many years experience of .Net and unmanaged code, OS level stuff etc and have never seen this kind of issue.





    Ok the logging helped, the exception is raised in the async handler that runs when a BeginSend() operation completes. Therefore my Sleep() and so on may fail because (I guess?) any attempt to freeze or hog an IOCP thread may be something the OS prevents...???



    Monday, January 17, 2011 7:51 PM

All replies