none
IOCP and I/O Threads RRS feed

  • Question

  • I implemented a WCF web service using the AsyncPattern.  This web service connects to other 3rd party web services asynchronously.  The basic flow is as follows:

    • Service Method is called
    • AsyncResult object is created containing a Result object
    • Business Logic Task object is created and is passed a reference to the AsyncResult
    • Business Logic Task is added to a Task List within a Singleton 
    • AsyncResult is returned to the Client
    • Singleton creates a background thread when it is created that processes requests until it is shutdown
    • 3rd Party Service is called asynchronously and is passed a callback to process the response.
    • Callback accesses the AsyncResult within the Task object, populates the Result object and calls the AsyncResult CompleteRequest method.

    My expectation was that the call to the 3rd Party's web service would be using IOCP and the callback would be invoked in an a separte IO Thread for each method call.  What I am seeing is that the callback is occurring on the Singleton's background worker thread.  Why is this the case and how do I achieve my goal?

    Wednesday, January 22, 2014 7:12 AM

Answers

  • The term 'worker thread' in .net/CLR typically just refers to any thread other than the Main thread that does some 'work' on behalf of the application that spawned the thread.  'Work' could really mean anything, including waiting for some I/O to complete.  The ThreadPool keeps a cache of worker threads because threads are expensive to create.

    The term 'I/O thread' in .net/CLR refers to the threads the ThreadPool reserves in order to dispatch NativeOverlapped callbacks from "overlapped" win32 calls (also known as "completion port I/O"). The CLR maintains its own I/O completion port, and can bind any handle to it (via the ThreadPool.BindHandle API).  Example here: http://blogs.msdn.com/junfeng/archive/2008/12/01/threadpool-bindhandle.aspx.  Many .net APIs use this mechanism internally to receive NativeOverlapped callbacks, though the typical .net developer won't ever use it directly.

    There is really no technical difference between 'worker thread' and 'I/O thread' -- they are both just normal threads.  But the CLR ThreadPool keeps separate pools of each simply to avoid a situation where high demand on worker threads exhausts all the threads available to dispatch native I/O callbacks, potentially leading to deadlock. (Imagine an application using all 250 worker threads, where each one is waiting for some I/O to complete).

    The developer does need to take some care when handling an I/O callback in order to ensure that the I/O thread is returned to the ThreadPool -- that is, I/O callback code should do the minimum work required to service the callback and then return control of the thread to the CLR threadpool. If more work is required, that work should be scheduled on a worker thread. Otherwise, the application risks 'hijacking' the CLR's pool of reserved I/O completion threads for use as normal worker threads, leading to the deadlock situation described above.

    Some good references for further reading: win32 I/O completion ports: http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx managed threadpool: http://msdn.microsoft.com/en-us/library/0ka9477y.aspx example of BindHandle: http://blogs.msdn.com/junfeng/archive/2008/12/01/threadpool-bindhandle.aspx

    Thursday, January 23, 2014 10:07 AM
  • Thanks for the response.  My architecture is built around a state machine pipeline where the callback method does just enough work to setup the next 3rd party request, and then invokes it from the IO thread.  It absolutely depends on the fact that the response comes back on a separate thread than the main thread that initially invoked it.  I did some more testing and found out that some times the response was performed on a ThreadPoolThread and some times it wasn't.

    I did some more research and ran into a discussion that lead me to a fix for my issue.  I did not implement the BeginGetRequestStream call, I was using GetRequestStream which then placed the HttpWebRequest object in a "synchronous" mode.  I really feel that the HttpWebRequest documentation should be updated to include this very interesting situation so that others don't pull their hair out trying to discover why it behaves the way it does.  Here's a paragraph from the discussion and a link to the discussion.  Anders states:

    This has nothing to do with IO Completion Ports making strange decisions this has to do with some implementation detail in the HttpWebRequest causing it to go synchronous when the first call is synchronous
    more specificly this call:
    Stream newStream = .GetRequestStream()

    you need to use BeginGetRequestStream instead even if you simply write:

    Stream newStream = .EndGetRequestStream(.BeginGetRequestStream(null, null))

    http://social.msdn.microsoft.com/Forums/vstudio/en-US/b8464aad-06b2-4a44-be72-eafa311c6821/webrequestbegingetresponse-is-blocking?forum=csharpgeneral

    Thanks again for your response, it proves how helpful these forums are because it was another discussion that helped point me in the right direction.

    Friday, January 24, 2014 7:19 PM

All replies

  • I was wrong about this particular problem.  I put some logging in and did find out that the responses were handled properly on IO Threads.

    The issue I did have and still am having problems understanding occurred in a simulator that I wrote for one of the 3rd Party services.  This 3rd Party service performs an out of band notification back to one of our RESTful WCF web services.  I developed a framework for writing .Net Proxies to communicate to RESTful services.  At its core is a class that implements an asynchronous HTTP Client, using HttpWebRequest's BeginGetResponse and Stream's BeginRead methods.  It appears that this does not use IO Threads when processing the BeginGetResponse and BeginRead callbacks.

    NOTE: I've been using the core HTTP Client for a few years and believe this worked correctly on IO Threads on .Net 3.5 or .Net 2.0, but haven't retested it for scalability issues when it has been recompiled with the newer frameworks.  Do all of the HttpWebRequest and Stream asynchronous methods still use IOCP and IO Threads?

    Wednesday, January 22, 2014 8:56 PM
  • Below is a trace for a single notification request from the 3rd Party Simulator:

    01/22/2014 13:40:58.777 - INFO - 3rdPartySim::BeginNotification => Active Requests: 1  ThreadID: 11  IsBackground: False  IsThreadPoolThread: False  AvailableWorker: 32767  AvailableIO: 1000  MaxWorker: 32767  MaxIO: 1000
    01/22/2014 13:40:58.793 - INFO - HTTPClient::PerformBeginRetrieve =>   ThreadID: 11  IsBackground: False  IsThreadPoolThread: False  AvailableWorker: 32767  AvailableIO: 1000  MaxWorker: 32767  MaxIO: 1000
    01/22/2014 13:41:02.069 - INFO - HTTPClient::ResponseCallback =>   ThreadID: 11  IsBackground: False  IsThreadPoolThread: False  AvailableWorker: 32767  AvailableIO: 1000  MaxWorker: 32767  MaxIO: 1000
    01/22/2014 13:41:02.069 - INFO - HTTPClient::ReadCallBack =>   ThreadID: 11  IsBackground: False  IsThreadPoolThread: False  AvailableWorker: 32767  AvailableIO: 1000  MaxWorker: 32767  MaxIO: 1000
    01/22/2014 13:41:02.069 - INFO - HTTPClient::ReadCallBack =>   ThreadID: 11  IsBackground: False  IsThreadPoolThread: False  AvailableWorker: 32767  AvailableIO: 1000  MaxWorker: 32767  MaxIO: 1000
    01/22/2014 13:41:02.069 - INFO - 3rdPartySim::ProcessNotificationResult => Active Requests: 0  ThreadID: 11  IsBackground: False  IsThreadPoolThread: False  AvailableWorker: 32767  AvailableIO: 1000  MaxWorker: 32767  MaxIO: 1000

    As can be seen, all of the callbacks execute on the same thread as the invoking thread.

    Below are snipets of the code for HTTPClient:

    private IAsyncResult PerformBeginRetrieve(String url, String a_sPostData,
    	AsyncCallback processWebResponse, Object clientState)
    {
        HTTPClientAsyncResult httpClientAsyncResult = null;
        WebRequestState webRequestState = null;
        WebRequest	webRequest = null;
    	try
    	{
            DisplayThreadInfo("HTTPClient::PerformBeginRetrieve");
    
            httpClientAsyncResult = new HTTPClientAsyncResult(processWebResponse, clientState);
            httpClientAsyncResult.HttpClient = this;
            webRequestState = new WebRequestState();
    		webRequestState.httpClientAsyncResult = httpClientAsyncResult;
            webRequest = WebRequest.Create(url);
    		webRequestState.request = webRequest;
    
            webRequest.Timeout = m_requestTimeout;
            ((HttpWebRequest)webRequest).ServicePoint.Expect100Continue = m_expect100Continue;
            ((HttpWebRequest)webRequest).ServicePoint.ConnectionLimit = m_defaultConnectionLimit;  // NOTE: This probably won't have an affect, but trying.  Notes say this may be app.config only setting.
    
            // Change request from GET to POST
            if (a_sPostData != "")
                PrepareWebRequestForPost((HttpWebRequest) webRequest, a_sPostData);
    
    
    
            if (allDone == null)
                allDone = new ManualResetEvent(false);
            else
                allDone.Reset();
    
            IAsyncResult result =
                (IAsyncResult) webRequest.BeginGetResponse(new AsyncCallback(ResponseCallback), webRequestState);
    
            ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, new WaitOrTimerCallback(TimeoutCallback), webRequestState, m_requestTimeout, true);
    	}
    	catch(WebException ex)
    	{
    		string msg = ex.Message;
    
    		if (processWebResponse != null)
    			ProcessWebResponse("", ex.Message, webRequestState);
    	} 
    	catch(Exception ex)
    	{
    		string msg = ex.Message;
    
    		if (processWebResponse != null)
                ProcessWebResponse("", ex.Message, webRequestState);
        }
    
        return httpClientAsyncResult;
    }
    
    private  void ResponseCallback(IAsyncResult asynchronousResult)
    {  
    	WebRequestState webRequestState = null;
    	Stream responseStream = null;
    
    	try
    	{
            DisplayThreadInfo("HTTPClient::ResponseCallback");
    
    		// Set the State of request to asynchronous.
    		webRequestState = (WebRequestState) asynchronousResult.AsyncState;
    		WebRequest  webRequest = webRequestState.request;
    
    		// End the Asynchronous response.
    		webRequestState.response =  webRequest.EndGetResponse(asynchronousResult);
    
            if (((HttpWebResponse)webRequestState.response).StatusCode == HttpStatusCode.OK)
                webRequestState.serverReceivedRequest = true;
    
    		// Read the response into a 'Stream' object.
    		responseStream = webRequestState.response.GetResponseStream();
    
    		webRequestState.responseStream=responseStream;
    
    		// Begin the reading of the contents of the HTML page.
    		responseStream.BeginRead(webRequestState.bufferRead, 0, BUFFER_SIZE, new AsyncCallback(ReadCallBack), webRequestState);
        
    	}
    	catch(WebException e)
    	{
            string msg = e.Message;
            string resp = "";
    
            if (webRequestState != null && webRequestState.httpClientAsyncResult != null && webRequestState.httpClientAsyncResult.m_clientAsyncCallback != null)
            {
                HttpWebResponse webResponse = (HttpWebResponse)e.Response;
                if (webResponse != null)
                {
                    StreamReader sr = new StreamReader(webResponse.GetResponseStream());
                    resp = sr.ReadToEnd();
                }
                if (webRequestState.errMsg != "")
                    msg = webRequestState.errMsg;
    
                ProcessWebResponse(resp, msg, webRequestState);
            }
    
    		if (responseStream != null)
    			responseStream.Close();
    	} 
    	catch(Exception e)
    	{
    		string msg = e.Message;
    
            if (webRequestState != null && webRequestState.httpClientAsyncResult != null && webRequestState.httpClientAsyncResult.m_clientAsyncCallback != null)
                ProcessWebResponse("", e.Message, webRequestState);
    
    		if (responseStream != null)
    			responseStream.Close();
    	}
        allDone.Set();
    }
    
    private   void ReadCallBack(IAsyncResult asyncResult)
    {
    	WebRequestState webRequestState = null;
    	Stream responseStream = null;
    
    	try
    	{
            DisplayThreadInfo("HTTPClient::ReadCallBack");
    
            // Cast the AsyncState to a WebRequestState; this is what we passed in to the
            // call to BeginRead.
    		webRequestState = (WebRequestState) asyncResult.AsyncState;
    		responseStream = webRequestState.responseStream;
    
    		int read = responseStream.EndRead( asyncResult );
    
    		// Read the contents of the HTML page.
    		if (read > 0)
    		{
                string contentEncoding = ((HttpWebResponse) (webRequestState.response)).CharacterSet;
                webRequestState.responseData.Append(DecodeToString(webRequestState.bufferRead, 0, read, contentEncoding));
    
                // Need to see if connection has closed
                //if (!asyncResult.IsCompleted)  -- Note this is the ReadCallback state not the HTTP Request and IsCompleted is wrong to check for connection closed.
                //{
    				responseStream.BeginRead(webRequestState.bufferRead, 0, BUFFER_SIZE, 
    					new AsyncCallback(ReadCallBack), webRequestState);
                //}
                //else
                //{
                //    webRequestState.processWebResponse(
    			//	    webRequestState.responseData.ToString(), "",
    			//	    webRequestState.clientState);
    
    			//    responseStream.Close();
                //}
    		}
    		else
    		{
                ProcessWebResponse(webRequestState.responseData.ToString(), "", webRequestState);
    			responseStream.Close();
    		}
    	}
    	catch(WebException e)
    	{
    		string msg = e.Message;
    
            if (webRequestState != null && webRequestState.httpClientAsyncResult != null && webRequestState.httpClientAsyncResult.m_clientAsyncCallback != null)
            {
                ProcessWebResponse("", msg, webRequestState);
            }
    
    		if (responseStream != null)
    			responseStream.Close();
    	} 
    	catch(Exception e)
    	{
    		string msg = e.Message;
    
            if (webRequestState != null && webRequestState.httpClientAsyncResult != null && webRequestState.httpClientAsyncResult.m_clientAsyncCallback != null)
            {
                ProcessWebResponse("", msg, webRequestState); 
            }
    
    		if (responseStream != null)
    			responseStream.Close();
    	}
    }

    Wednesday, January 22, 2014 10:04 PM
  • The term 'worker thread' in .net/CLR typically just refers to any thread other than the Main thread that does some 'work' on behalf of the application that spawned the thread.  'Work' could really mean anything, including waiting for some I/O to complete.  The ThreadPool keeps a cache of worker threads because threads are expensive to create.

    The term 'I/O thread' in .net/CLR refers to the threads the ThreadPool reserves in order to dispatch NativeOverlapped callbacks from "overlapped" win32 calls (also known as "completion port I/O"). The CLR maintains its own I/O completion port, and can bind any handle to it (via the ThreadPool.BindHandle API).  Example here: http://blogs.msdn.com/junfeng/archive/2008/12/01/threadpool-bindhandle.aspx.  Many .net APIs use this mechanism internally to receive NativeOverlapped callbacks, though the typical .net developer won't ever use it directly.

    There is really no technical difference between 'worker thread' and 'I/O thread' -- they are both just normal threads.  But the CLR ThreadPool keeps separate pools of each simply to avoid a situation where high demand on worker threads exhausts all the threads available to dispatch native I/O callbacks, potentially leading to deadlock. (Imagine an application using all 250 worker threads, where each one is waiting for some I/O to complete).

    The developer does need to take some care when handling an I/O callback in order to ensure that the I/O thread is returned to the ThreadPool -- that is, I/O callback code should do the minimum work required to service the callback and then return control of the thread to the CLR threadpool. If more work is required, that work should be scheduled on a worker thread. Otherwise, the application risks 'hijacking' the CLR's pool of reserved I/O completion threads for use as normal worker threads, leading to the deadlock situation described above.

    Some good references for further reading: win32 I/O completion ports: http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx managed threadpool: http://msdn.microsoft.com/en-us/library/0ka9477y.aspx example of BindHandle: http://blogs.msdn.com/junfeng/archive/2008/12/01/threadpool-bindhandle.aspx

    Thursday, January 23, 2014 10:07 AM
  • Thanks for the response.  My architecture is built around a state machine pipeline where the callback method does just enough work to setup the next 3rd party request, and then invokes it from the IO thread.  It absolutely depends on the fact that the response comes back on a separate thread than the main thread that initially invoked it.  I did some more testing and found out that some times the response was performed on a ThreadPoolThread and some times it wasn't.

    I did some more research and ran into a discussion that lead me to a fix for my issue.  I did not implement the BeginGetRequestStream call, I was using GetRequestStream which then placed the HttpWebRequest object in a "synchronous" mode.  I really feel that the HttpWebRequest documentation should be updated to include this very interesting situation so that others don't pull their hair out trying to discover why it behaves the way it does.  Here's a paragraph from the discussion and a link to the discussion.  Anders states:

    This has nothing to do with IO Completion Ports making strange decisions this has to do with some implementation detail in the HttpWebRequest causing it to go synchronous when the first call is synchronous
    more specificly this call:
    Stream newStream = .GetRequestStream()

    you need to use BeginGetRequestStream instead even if you simply write:

    Stream newStream = .EndGetRequestStream(.BeginGetRequestStream(null, null))

    http://social.msdn.microsoft.com/Forums/vstudio/en-US/b8464aad-06b2-4a44-be72-eafa311c6821/webrequestbegingetresponse-is-blocking?forum=csharpgeneral

    Thanks again for your response, it proves how helpful these forums are because it was another discussion that helped point me in the right direction.

    Friday, January 24, 2014 7:19 PM