none
Asynchronous reading using native Overlapped RRS feed

  • Question

  • Hi,
    I'm using the managed Overlapped representation (System.Threading.Overlapped) of the native OVERLAPPED structure to read from a mini-filter driver using the FilterGetMessage native method.

    The Overlapped structure will be packed into a NativeOverlapped pointer with an IOCOmpletionCallback delegate and passed to the FilterGetMessage. When the reading finishes the delegate will be invoked.

    My problem is (i think) that the delegate always will be executed on the same thread (in my case often 7), regardless that the reading is started from 10 or more different threads, waiting for each other using waithandles.

    I get a lot of messages really (extremely) fast from the filter and when the delegate is always fired on thread 7 (mostly) it simply congests and kills the machine as the filter is pumping up messages from kernel mode.

    Are there anyway to get the delegates to fire on different threads ??

    Thanks, Soren
    Monday, September 28, 2009 10:38 PM

Answers

  • @Hans: He's using managed code to communicate with a filter driver.

    @webJose: The callbacks will be executed by a thread pool thread, not the originating thread.

    @Soren:

    By passing the driver handle to ThreadPool.BindHandle, you're adding that handle to a IOCP run by the threadpool. When the overlapped operation completes, the callback is on a thread pool thread.

    The thread pool is supposed to be self-tuning, adding more threads to its IOCP as necessary. So, I'm surprised they're all coming in on one thread if they are really completing so fast to cause problems. You can try increasing the number of IOCP threads by calling ThreadPool.SetMinThreads.

    If you still find you need to transfer threads, you may be able to use AsyncOperation. Or just turn around and do a ThreadPool.QueueUserWorkItem.

            -Steve
    Programming blog: http://nitoprograms.blogspot.com/
      Including my TCP/IP .NET Sockets FAQ
    MSBuild user? Try out the DynamicExecute task in the MSBuild Extension Pack source; it's currently in Beta so get your comments in!

    Microsoft Certified Professional Developer
    Tuesday, September 29, 2009 1:56 AM

All replies

  • Are you sure you are starting the read operations from different threads? Theoretically, the read completion callback function is called within the context of the thread that started the read operation.  Also note that the thread MUST be in an alertable wait state for the callback function to be executed.  You said you were waiting on thread handles.  If you are not using WaitForMultipleObjectsEx() or a similar, alertable wait function call, then the callback will never occur.

    So summarizing, I think you shouldn't have to do anything special to get the callback function to execute in the originating thread.


    MCP
    Monday, September 28, 2009 11:23 PM
  • This is weird.  You are using managed code in a filter driver?  Is there some kind of component you use that marshals the stuff to user mode?  That would be most likely your source of contention.

    Hans Passant.
    Tuesday, September 29, 2009 12:08 AM
    Moderator
  • @Hans: He's using managed code to communicate with a filter driver.

    @webJose: The callbacks will be executed by a thread pool thread, not the originating thread.

    @Soren:

    By passing the driver handle to ThreadPool.BindHandle, you're adding that handle to a IOCP run by the threadpool. When the overlapped operation completes, the callback is on a thread pool thread.

    The thread pool is supposed to be self-tuning, adding more threads to its IOCP as necessary. So, I'm surprised they're all coming in on one thread if they are really completing so fast to cause problems. You can try increasing the number of IOCP threads by calling ThreadPool.SetMinThreads.

    If you still find you need to transfer threads, you may be able to use AsyncOperation. Or just turn around and do a ThreadPool.QueueUserWorkItem.

            -Steve
    Programming blog: http://nitoprograms.blogspot.com/
      Including my TCP/IP .NET Sockets FAQ
    MSBuild user? Try out the DynamicExecute task in the MSBuild Extension Pack source; it's currently in Beta so get your comments in!

    Microsoft Certified Professional Developer
    Tuesday, September 29, 2009 1:56 AM
  • Stephen:

    Not my thread, but by asking is how I learn the most, so I hope that you don't mind my asking.

    ReadFile() and ReadFileEx()'s help topics @ MSDN Online state that the callback is performed in the context of the thread that called ReadFile() or ReadFileEx(); the callback is queued as an asynchronous procedure call, they say.  I can only imagine that it would have been the same under managed code.  Do you have lying around a link to some documentation that explains this difference?  I would very much like to learn about it.

    Thank you!
    MCP
    Tuesday, September 29, 2009 2:03 AM
  • No problem. Answering is how I learn the most. :)

    You are correct in that ReadFile/ReadFileEx will call back their Asynchronous Procedure Calls in the calling thread. However, the OP is not using APCs.

    Joe Duffy mentions that most (non-thread pool) waits in the CLR are alertable:
      http://www.bluebytesoftware.com/blog/2006/05/04/UsermodeAPCsAndManagedCode.aspx
    But in fact, APCs are not officially supported by the CLR; see this blog post from Eric Eilebrecht:
      http://blogs.msdn.com/ericeil/archive/2008/06/20/windows-i-o-threads-vs-managed-i-o-threads.aspx

    Anyway, the OP mentioned IOCompletionCallback, NativeOverlapped, and Overlapped. These are all classes used to manage the ThreadPool's IOCP (Input/Output Completion Port). So, we're dealing with the IOCP, not an APC.

    According to Joe Duffy's Concurrent Programming on Windows (which is a must-read, even though he gets a bit long-winded sometimes), the IOCP callbacks may be invoked on any thread associated with that IOCP. [I've stated elsewhere that Duffy's description of the various thread pools alone is worth the cost of the book, and here he comes through for me again].

             -Steve
    Programming blog: http://nitoprograms.blogspot.com/
      Including my TCP/IP .NET Sockets FAQ
    MSBuild user? Try out the DynamicExecute task in the MSBuild Extension Pack source; it's currently in Beta so get your comments in!

    Microsoft Certified Professional Developer
    Tuesday, September 29, 2009 2:52 AM
  • Hi,

    @Stephen: This is pretty much the senario.

    To my understanding there are default at least 5 (don't remember exactly) available on the threadpool, and they are all already fires up and ready to run. So that was my preferred choice as this just seem faster than firing up a new thread to handle the callback routine.

    I'll try increasing the number of threads in the ThreadPool, alternately I'll go create my own IAsyncResult implementation that will have it's own "ThreadPool".

    Thanks, I'll get back...

    Soren
    Tuesday, September 29, 2009 6:13 AM
  • Hi,

    @Stephen: This is pretty much the senario.

    To my understanding there are default at least 5 (don't remember exactly) available on the threadpool, and they are all already fires up and ready to run. So that was my preferred choice as this just seem faster than firing up a new thread to handle the callback routine.

    I'll try increasing the number of threads in the ThreadPool, alternately I'll go create my own IAsyncResult implementation that will have it's own "ThreadPool".

    Thanks, I'll get back...

    Soren

    How are you, Soren? I'm checking the status of the issue on your side. If you have any other questions, please feel free to post here.

    By the way, thanks to Stephen, webjose and Hans for your great inputs here!!

    Regards,
    Jialiang Ge
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Wednesday, October 7, 2009 4:52 AM
    Moderator
  • Well - it seems that my code is so fast (very humble) that the threadpool thread finishes before a new is needed - and therefore it'll usually be the same thread being picked up next time.

    The performance issue was lying elsewhere

    So as for now it's working as supposed too...

    Really noone came up with an answer (as there wasn't any), but Stephen get's the points :-)

    Thanks for inputs to all...
    Wednesday, October 7, 2009 10:06 AM