Operation Time-out Error

Question

Hi All,

I am getting following exceptions while making simultaneous requests to the java web service from windows forms application.

Exception 1 :  

The operation has timed out at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

Exception 2 :

The operation has timed out   at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request) at Microsoft.Web.Services3.WebServicesClientProtocol.GetResponse(WebRequest request, IAsyncResult result)at Microsoft.Web.Services3.WebServicesClientProtocol.GetWebResponse(WebRequest request) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

 

Web service calls are made to get data from windows forms application (framework 4.5) which is a multi-threaded application, running on windows server-2012 having capacity of 8 core processors, 32 GB RAM and 64 bit OS. 

We're getting the above exceptions only when we make simultaneous calls to the web service, i.e. for 100 simultaneous  calls, 30 requests are getting failed. 

We are consuming java web service and requests are handled via load balance, and the maximum client time-out configured is 10 seconds.

Could you please assist me in this regards. Thanks in advance.

Regrads,

Siva

Answer 1

We are consuming java web service

http://forums.asp.net/28.aspx/1?WCF+ASMX+and+other+Web+Services

Answer 2

The biggest issue with making HTTP calls is that you are limited to the # of simultaneous connections from the same machine to the same URL. This is partially a security feature. It used to be that you could only make 2 simultaneous calls to the same host at the same time. At least that used to be true, whether it is for you or not I cannot say.  Additionally your network adapter is limited in the # of connections it is allowed to make. Clients used to be limited to 10 simultaneous connections. As such I can see two possible causes for this issue.

1) Your calls are taking too long to execute and the 2 minute network timeout is kicking in as all your calls are serialized into the # of connections that are allowed. The solution for this is to not use so many calls. 100 is a lot anyway. Service calls should be broader to reduce the # that has to be made. If you cannot reduce the # of calls then you need to serialize them using ThreadScheduler or similar so that you can manage the # of outstanding calls.

2) Your service cannot handle that many calls at the same time and it is stalling too long causing the timeout. You would need to take a look at the service code and confirm it is not getting overloaded.

Michael Taylor
http://www.michaeltaylorp3.net

Answer 3

Thank you very much Michael for your response. 

1. Average time taken by Web Service to give response is 2 seconds and in worst case it is taking maximum of 5 seconds.

2. We had seen some instances where we had put the request on to web service e.g. at time 04:00:10 seconds and it is reaching to the destination server at 04:00:17 seconds.

3. As you said that only 2 simultaneous calls to the same host at the same time, we had tried increasing its value to 32 by making configuration settings in the app.config under connection setting as below, and it did not worked.

<configuration>
 <system.net>
  <connectionManagement>
   <add address="*" maxconnection="32" />
  </connectionManagement>
 </system.net>
</configuration>

Could you please clarify on, 2 simultaneous calls to the same host at the same time, is only for web browsers or for windows forms applications as well.

4.As soon as we start our application 60-100 sessions start simultaneously and they all try to put request on to the web service and during this time we noted that the time outs were peaked out. And the timeouts gradually comes down based on the traffic volumes. In one word the more volumes we process simultaneously the more timeouts we are experiencing.

5. We tried checking the port usage through net stats and there the status of the request is been shown as  Time_Wait and Close_Wait. Can we extract any abstract with these descriptions?

6.We tried mitigating the timeouts through some Config changes on one server which has 60 parallel sessions running on it. The Config changes we enhanced are timeout and retries(earlier timeout used to be 10secs and now we bought it down to 8secs, earlier no of retries used to be 3 now we bought it down to 2) and these Config changes has helped us to some extent in mitigating the no of timeouts. The issue still persists intermittently particularly when we restart the application for the first time and the server picks up heavy volume traffics.

7. Also, can you please help us on how to tweak "Network adapter connections limit".

Could you please help in this assert, Thank you in advance

Regards
Siva

Answer 4

The connectionManagement option is supposed to allow you to have more connections but in my experience it doesn't seem to help. You can confirm this however by monitoring your web server to see how many simultaneous connections it has to the client. If it is > 2 then the entry is working.

Note that the web server may throttle connections. Specifically if you're running a client OS and IIS then, historically, it limited the # of connections of any type to 10. For server there is no restriction. Again, looking at the server connection logs would help identify this.

If you are running into connection issues you should probably consider refactoring your calls. Do you really need to make that many calls at once? That is a lot of threads (assuming you're using threading) and your CPU isn't going to run that many anyway. In general you're only going to get 1 thread per logical CPU so a quad core processor is still going to break 60 calls up into 15 batches of 4 (in the best case).  Queuing calls would allow the OS to better manage your threads and probably eliminate the timeouts (although they can still happen).

You might also consider, if possible, evaluating your service interface. When you start talking about a SOA architecture then you try to reduce the # of calls and make them broader.  OOP doesn't work as well in this case. For example if you're trying to retrieve data about a component and you have 60 components that is 60 calls if each method returns info about 1 single component. In SOA you'd probably create a method that allows you to pass in a set of items and it returns the data in batch. This can reduce 60 calls to 1 (or larger if the data is large).

At the end of the day I don't know that you're going to be able to work around your timeouts given the # of requests you're making in the given timeframe. Timeouts are always possible irrelevant so you might also consider simply adding in some retry logic such that if the request times out you try again a couple of times before failing the call.

Answer 5

Thank you Every one giving the valuable information which help me to analyze the exception in different ways.

The Operation Time-out issue has been fixed by doing following settings.

1. By adding below Config setting in the web.config or in App.Config and setting maxconnection to 150

<configuration>
 <system.net>
  <connectionManagement>
   <add address="*" maxconnection="150" />
  </connectionManagement>
 </system.net>
< /configuration>

[OR]

2. From code by initializing the Servicepoint manager with request url and setting the connectionLimit property of servicePoint to 150 as Below

Using System.Net;

Uri objUri = new Uri(http://localhost/webservice.asmx); // Your Service URL
ServicePoint objSp = ServicePointManager.FindServicePoint(objUri);
bjSp.ConnectionLimit = 150;

Note : Via code you need to initialize the servicepoint every time and set the connection limit. Preferably use web Config setting which I advice,So no need to initialize every time.

 Regards,

Siva

---

Siva Eedala