Getting timeout when using Parallel.For and calling WebRequest.GetREsponse

Question

I have the following code that for every workOrder object, call a function that logins to a remote website.

Parallel.ForEach(workOrders, (WorkorderObject workorder) => { DoSomething(workorder); });

the function (simplified):

private void DoSomething(WorkorderObject workorder)
{
  WebRequest.Create(new Uri(URL)).GetResponse();
}

I am getting "The Operation has timed out" error on the 4th(usually) workorder.  I must have done something wrong, please tell me what is wrong with this code?

Thanks, and appreciate any guidance.

Answer 1

Are you closing the WebResponse? see: http://blogs.msdn.com/b/feroze_daud/archive/2004/01/21/61400.aspx

Anyway, IMHO you should not perform IO bound operations within a Parallel.ForEch because that causes threads to spend their time just waiting. The good aproach is to use Task.FromAsync when the underlying class implements the asynchronous pattern. Here you have an example:

 

public void ProcessWorkOrders(IEnumerable<WorkOrder> workOrders)
{
  var tasks = new List<Task<ProcessWorkOrderResult>>();
  string url = "some url";
  foreach (var workOrder in workOrders)
  {
    var task = GetProcessWorkOrderTask(url, workOrder);
    tasks.Add(task);
  }
  Task.WaitAll(tasks.ToArray());
}

public Task<ProcessWorkOrderResult> GetProcessWorkOrderTask(string url, WorkOrder workOrder)
{
  HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
  return Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null)
    .ContinueWith(t =>
    {
      try
      {
        if (t.IsFaulted)
        {
          throw t.Exception;
        }
        else if (t.IsCompleted)
        {
          return GetProcessWorkOrderResultFromWebResponse(t.Result, workOrder);
        }
        else
        {
          return null;
        }
      }
      finally
      {
        if (t.Result != null)
        {
          ((IDisposable)t.Result).Dispose();
        }
      }
    });
}

ProcessWorkOrderResult GetProcessWorkOrderResultFromWebResponse(WebResponse response, WorkOrder workOrder)
{
  using (var stream = response.GetResponseStream())
  {
    //TODO: Do response processing here
    return new ProcessWorkOrderResult();
  }
}

 

Answer 2

Thanks Jesus for your detail response.

Again I am a newbie so please bear with me.  From your code is it correct to say the parallel part starts on the line: Task.Factory.FromAsync?  which is in the GetProcessWorkOrderTask routine?

I did simplify the post to make it easier, but one of the things i left out was we have different regions that these workorders are located, and we cannot process the same region at the same time.  So each region will need to be in its own thread.  Can FromAsync call a subroutine? 

Answer 3

Task.Factory.FromAsync creates and starts a new task and returs inmediatly without waiting for completion. The code calls Taks.Fatory.FromAsync for work order 1, then calls it for work order 2 without waiting for the created task (for work order 1) to be completed, then calls it for work order 3 and so on. You end up with multiple tasks running in parallel.

Sorry I don't understand completly. Are you meaning you cannot process two work orders in the same region in the same thread, but you can process two work orders in different regions in the same thread? Please elaborate.

Also, Why you cannot? Perhaps we can help you to overcome this constraint.

Answer 4

It is a business requirement, that no two work orders in the same region can be processed at the same time.  And that we will have a thread for each region.

Maybe i should revise my original code to clarify the scenario (again, simplified):

Parallel.ForEach(regions, (RegionObject region) => { ProcessRegion(region); });

private void ProcessRegion(RegionObject region)
{

  workorderObject[] workorders = getWorkorders(region.ID);

  foeach (workorderObject workOrder in workorders)

  {

  doSomething()
  WebRequest.Create(new Uri(URL)).GetResponse();

  }
}

Thanks again Jesus for being so helpful, and patient.  I hope the new code made it clearer.

Answer 5

I think in your code, Paralle.ForEach is not necessarily useful. To repeat the advice of Jesus, you should parallelize calls WebRequest as the following code:

            Task<WebResponse>[] tasks = new Task<WebResponse>[urls.Length];

            for (int i = 0; i < urls.Length; i++)
            {
                WebRequest webRequest = WebRequest.Create(new Uri(urls[i]));
               
                tasks[i] = Task<WebResponse>.Factory.FromAsync(
                  webRequest.BeginGetResponse,
                  webRequest.EndGetResponse, TaskCreationOptions.LongRunning);                                               
            }
          
            try
            {
                Task.WaitAll(tasks);
            }
            catch (AggregateException ex)
            {
                ex.Handle((inner) =>
                {
                    if (inner is WebException)
                    {
                        WebException oce = inner as WebException;
                        // Log here
                        return true;
                    }
                    else
                    {
                        // this is an exception we don't know how
                        // to handle, so return false
                        return false;
                    }
                });
            }

            // get the responses ...
            foreach (var task in tasks)
            {
                Console.WriteLine("{0} {1}", task.Result.ResponseUri, task.Result.ContentLength);
            }

Hope is useful :-)

Answer 6

kumal,

 

Given your firm business requirements, I would suggest using a slight variation on your last code:

 

Parallel.ForEach(regions, region => 
{
  workorderObject[] workorders = getWorkorders(region.ID);
  foreach (var workOrder in workorders)
  {
   doSomething(workOrder);
   using(var r = WebRequest.Create(new Uri(URL)).GetResponse())
   {
    // optionally do something with "r" here, but this will, at a minimum, close the response. If there are many regions, this will be important!
   }
  }
}

This will force you to use one "thread" per region maximum, but at least give you parallelization at that level.  However, it is important to make sure to close your WebReponse objects as Jesus pointed out above.  The using statement will take care of that (plus allow you to use the response, if needed).

 

---

Reed Copsey, Jr. - http://reedcopsey.com