locked
Simultaneous calls to services results in The request was aborted: The request was canceled

    Question

  • Hi Everyone

    I am making simultaneous calls to seperate web services on the same server.  The services have WSE 3 enabled using certificate based security.

    After a minute or so of running one them returns with the error "The request was aborted: The request was canceled", I am completley stumped, if I just make one of the calls then everything is ok.  Here is the stack trace from the exception:

       at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
       at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
       at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
       at System.IO.StreamWriter.Write(String value)
       at System.Xml.XmlTextWriter.WriteStartElement(String prefix, String localName, String ns)
       at System.Xml.XmlElement.WriteTo(XmlWriter w)
       at System.Xml.XmlElement.WriteContentTo(XmlWriter w)
       at System.Xml.XmlElement.WriteTo(XmlWriter w)
       at System.Xml.XmlElement.WriteContentTo(XmlWriter w)
       at System.Xml.XmlElement.WriteTo(XmlWriter w)
       at System.Xml.XmlElement.WriteContentTo(XmlWriter w)
       at System.Xml.XmlElement.WriteTo(XmlWriter w)
       at System.Xml.XmlDocument.Save(XmlWriter w)
       at Microsoft.Web.Services3.SoapEnvelope.Save(Stream outStream)
       at Microsoft.Web.Services3.Messaging.SoapPlainFormatter.Microsoft.Web.Services3.Messaging.ISoapFormatter.Serialize(SoapEnvelope envelope, Stream stream)
       at Microsoft.Web.Services3.Xml.SoapEnvelopeWriter.Finish()
       at Microsoft.Web.Services3.Xml.XmlWrappingWriter.Flush()
       at System.Web.Services.Protocols.SoapHttpClientProtocol.Serialize(SoapClientMessage message)
       at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
       at Washtec.Win.ServiceAgents.JobWS.JobsWSWse.GetTodaysJobs() in

    Any help on this is would be much appreciated, I'm using Visual Studio 2005, and the services were also built using visual studio 2005.

    Thanks in advance.

    Dan

     

    Saturday, January 21, 2006 8:41 PM

Answers

  • OK, I got a chance to digest the response. I'll post it below. The response included a suggestion to try. I did, and it seems to fix the problem. I don't know if there will be other consequences to the app, but we're making progress:

    Here's the response:

    I’ve finally been able to review the crash dump of the Request Canceled exception that you are getting. It turns out that when you start to make spin up your second set of threads to call the remote web service your ASP.NET application is waiting for a response from the web service but instead the web server resets the connection. Within the dump that I collected we see that there is an exception with the text of “The underlying connection was closed: A connection that was expected to be kept alive was closed by the server”. It appears that the code logic in .NET 2.0 is that when this occurs we will try to resend the web request, however because of the previous exception the m_Aborted property of the HttpWebRequest object is set to true and so we end up getting the Request Canceled exception that you are seeing.  

    So it appears that the BEA WebLogic web server is resetting the connection, whereas the ASP.NET application expects the connection to still be active. I cannot explain why you do not see this problem in .NET 1.1. We will need to capture a set of network traces in order to help determine this.

    Action Plan

    ==================

    We do not have an explanation of why your BEA WebLogic web server is resetting, thus closing, the connection with your ASP.NET application, and I currently can’t explain why you don’t see similar symptoms with your .NET 1.1 application that runs the same code. For our next troubleshooting steps I’d like for you to disable Http KeepAlives for the HttpWebRequest object in your .NET 2.0 application. This will force the client to establish another TCP session with the BEA Web server before sending out another web service requests.

     In order to set the KeepAlive property of the HttpWebRequest object to false you will need to override the GetWebRequest method within your client application. Please see Resolution D in the following article to see how to do this in your ASPX client page

     915599 You receive one or more error messages when you try to make an HTTP request in an application that is built on the .NET Framework 1.1 Service Pack 1

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;915599

    Friday, May 19, 2006 3:44 PM

All replies

  •  

    do you mean "One client pc calling the same service multiple times"

    or "two or more client pc's calling the service at the same time" ?

    if one client tries to open more than 2 connections to a web server it can be halted by the way HTTP works. by default http wants only 2 connections per client to keep from having a few clients flood a server with requests.

     

    if you have 2 or more clients calling the same web method then perhaps the method is blocking the other calls -- for example a sql data call innthe method is taking to long.

     

    also if the web server is iis on xp pro that has a connection limit on it that could block calls. if thats the case google for the subject and you wil find some ways to up xp pro to about 39 connections.

    Monday, January 23, 2006 1:45 PM
  • Hi figuerres

    Its actually one pc making both calls, and they fire prety much simultaneously.  The actual machine its calling into is an SBS 2003 server.

    Thanks

    Dan

    Monday, January 23, 2006 4:49 PM
  • This sounds like a networking issue, so I'm going to move this thread to the networking web forum. 

    Basically what is happening is one of the services is closing the connection when the client expects it to still be open.  You should consider getting a network trace to see what is going on on the wire:

    http://msdn2.microsoft.com/en-us/library/a6sbz1dx.aspx

    Daniel Roth

    Tuesday, January 24, 2006 1:15 AM
    Moderator
  • Yes Please get a netmon trace
    http://blogs.msdn.com/dgorti

    Or as Daniel says please get a system.net log file

    Tuesday, January 24, 2006 3:30 PM
    Moderator
  • Hi sorry for the delayed response.

    I tired to install netmon, but when I look in my windows components in Management and Monitoring Tools, I don't have network monitor listed.

    Friday, January 27, 2006 10:46 AM
  • Daniel,

    Did you ever figure out what was happening? I am experiencing something similar when I call a Java based webservice. Most of the time it works fine, but maybe 10% of the time I get the "The request was aborted: The request was canceled." error.

    Tuesday, May 09, 2006 10:50 PM
  • Heya Folks,

    Did either of you resolve this? I have a similar issue (also calling a Java web service) from a .net client.

    Thanks in advance,
    Billy
    Thursday, May 11, 2006 2:24 AM
  • Hi

    I never did find a proper answer to the problem in the end I had to just make the calls one after the other as each finished, which is a real pain.

    If anyone does have an answer for this then I am still interested as well.

    Sorry I can't be of much help

    Dan

    Thursday, May 18, 2006 8:37 PM
  • I opened a ticket with MSDN support for this problem. The person helping me took a bunch of crash dumps from a test application that reproduces this problem. I got a response this morning, but I haven't had a chance to digest it. I'll post what we discover up here.
    Friday, May 19, 2006 2:50 PM
  • OK, I got a chance to digest the response. I'll post it below. The response included a suggestion to try. I did, and it seems to fix the problem. I don't know if there will be other consequences to the app, but we're making progress:

    Here's the response:

    I’ve finally been able to review the crash dump of the Request Canceled exception that you are getting. It turns out that when you start to make spin up your second set of threads to call the remote web service your ASP.NET application is waiting for a response from the web service but instead the web server resets the connection. Within the dump that I collected we see that there is an exception with the text of “The underlying connection was closed: A connection that was expected to be kept alive was closed by the server”. It appears that the code logic in .NET 2.0 is that when this occurs we will try to resend the web request, however because of the previous exception the m_Aborted property of the HttpWebRequest object is set to true and so we end up getting the Request Canceled exception that you are seeing.  

    So it appears that the BEA WebLogic web server is resetting the connection, whereas the ASP.NET application expects the connection to still be active. I cannot explain why you do not see this problem in .NET 1.1. We will need to capture a set of network traces in order to help determine this.

    Action Plan

    ==================

    We do not have an explanation of why your BEA WebLogic web server is resetting, thus closing, the connection with your ASP.NET application, and I currently can’t explain why you don’t see similar symptoms with your .NET 1.1 application that runs the same code. For our next troubleshooting steps I’d like for you to disable Http KeepAlives for the HttpWebRequest object in your .NET 2.0 application. This will force the client to establish another TCP session with the BEA Web server before sending out another web service requests.

     In order to set the KeepAlive property of the HttpWebRequest object to false you will need to override the GetWebRequest method within your client application. Please see Resolution D in the following article to see how to do this in your ASPX client page

     915599 You receive one or more error messages when you try to make an HTTP request in an application that is built on the .NET Framework 1.1 Service Pack 1

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;915599

    Friday, May 19, 2006 3:44 PM
  • One more follow-up. I implemented the KeepAlives = FALSE and everything seems to be working great.
    Tuesday, May 23, 2006 7:31 PM
  • Well, we actually tried this solution without success :-( Only difference was that we implemented the GetWebRequest override in a WSE enabled proxy class. (IE, we wrote our web service classes, enabled WSE on them and then derived a new class from the proxy class that was generated with the WSE)

    Anyone who might have any other ideas as to why it still fails for us would be greatly appreciated.

    Paul.

     

    Monday, August 21, 2006 3:51 PM
  • Friday, February 02, 2007 2:15 PM by Craig Scheets

    THE REAL ASP.NET 2.0 SOLUTION:

    I fought this for a while and finally got the right code to fix this problem in ASP.Net 2.0.  As many of you know, ASP.Net 2.0 generates the proxy classes dynamically so you can't just simply edit them.  The key is in a new feature, 'partial classes'.  Below is everything you need to know to fix this in ASP.Net 2.0.

    First, my web reference declaration is:

    weather.ndfdXML

    The standard instantiation would then look like:

    weather.ndfdXML2 wxService = new weather.ndfdXML2();

    With this setup, I ran into the exact same problem everyone else has written about here and rarely solved.

    First, add a new class and make it a partial class (C# code posted):

    using System;

    using System.Net;

    using System.Web.Services.Protocols;

    namespace weather

    {

       public partial class ndfdXML2 : weather.ndfdXML

       {

           protected override System.Net.WebRequest GetWebRequest(Uri uri)

           {

               //throw new Exception("Custom WebRequest override code hit!!");

               System.Net.HttpWebRequest webRequest = (System.Net.HttpWebRequest)base.GetWebRequest(uri);

               webRequest.KeepAlive = false;

               webRequest.ProtocolVersion = HttpVersion.Version10;

               return webRequest;

           }

       }

    }

    You can optionally uncomment that new exception to verify that the code is indeed executing.

    Second, change you instantiation inside your code to the following:

    weather.ndfdXML2 wxService = new weather.ndfdXML2();

    Now everything works great!!

    Friday, March 16, 2007 10:13 AM
  • I'm having this same issue but my client is a BizTalk orchestration and is using the SOAP adaptor.  When a web reference is added to a BizTalk project the proxy file is a XSD file instead of a CS file.  The problem is I'm not sure how to override a method in a XSD file.  And the SOAP adaptor does the instantiation so we have no control over it.  So, I'm not sure how to implement the above solution within a BizTalk project.

    Monday, April 23, 2007 6:05 PM
  • Charles, have you had a chance to figure out how to deal with this problem. I am having the same issue consuming a Web service using SOAP adapter with BizTalk 2006.

    Thanks

    --Mike

    Thursday, May 17, 2007 11:17 PM
  • Me too!
    Wednesday, June 13, 2007 5:13 PM
  • Has anyone on this thread correlated the time of the connection being forcibly closed with the state of the worker process?  See the System section of the event log on the ASP.NET application tier.  Does the worker process get recycled when this happens?

     

    Here's an example of the event to look for (your reason may be different -- I'm forcing this to happen right now by setting the app pool to recycle the worker process every two minutes in order to repro a problem I'm working on).

     

    Event Type: Information
    Event Source: W3SVC
    Event Category: None
    Event ID: 1074
    Date:  8/21/2007
    Time:  05:07:15 PM
    User:  N/A
    Computer: MYDEV
    Description:
    A worker process with process id of '5900' serving application pool 'HelloWorld' has requested a recycle because the worker process reached its allowed processing time limit. 

    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

    Buck

    Wednesday, August 22, 2007 2:05 AM
    Moderator
  • In my case, I was able to reproduce the problem reliably with a test app, and the problem was caused by a bug in the handling of app pool recycling.  Working with an escalation engineer, we were able to verify that the problem I was seeing was fixed in Service Pack 2 for Windows 2003 Server.

     

    Buck

    Tuesday, August 28, 2007 12:34 AM
    Moderator
  • Yes, we verified the worker process is not being recycled.  We get this error up to 200 times per hour which is much more frequent than the worker process recycle.

     

     

    Tuesday, August 28, 2007 3:00 PM
  • I am posting this because we had a similar issue and solved it through a MSDN ticket.  A very knowledgeable engineer called me back and explained the issue.  This might help other people researching similar problems. 

     

    The problem is solved by changing two values on the ServicePointManager class.  The first is the DefaultPersistentConnectionLimit field.  This has been addressed in numerous other posts.  It defaults to 2 which is too low for most intensive apps.  So bump this up.  We went to 100.  The next property is the MaxServicePointIdleTime property.  This sets the suggested time a connection is held after its last use before it is released back to the "queue."  By default this is 100 seconds.  This is way too high, the Microsoft engineer suggested 2000-5000 milliseconds for a highly concurrent application.  The combination of these two settings resolved all of our The request was aborted: The request was canceled exceptions.  The real key is the idle settings.

     

    Many other posts refer to the Keepalive settings as the "work around" for this "bug."  Turning off Keepalive merely makes all connections immediately go back to the "queue."  So you are just avoiding the behavior that the Microsoft engineers designed into the framework.  Supposedly this was done to protect servers.  Hope this helps...

     

     

    Wednesday, October 03, 2007 7:30 PM
  • We changed the MaxServicePointIdleTime to 20 seconds which seemed to eliminate a lot of these errors although we do see the error about once a day.  This is much better than the 1000 times a day we were seeing it.  I will try to decrease this value to 5 seconds and let you know if this completely eliminates the error.

     

    Thursday, October 04, 2007 6:24 PM
  •  

    Hi All,

     

        This thread was of particular use to us as we had this exact problem. I was happy to finally hear of an actual solution to the problem. Now that I have this information I have been able to move on with a project that I was working on that needed asynchronous web service calls but I have stumbled into another problem almost immediately that I though someone here might be able to shed some light on.

     

    Just a little information on our web services for you. They are XML web services using WSE 3.0 and X.509 certificates.

    We now seem to be able to call Async versions of our web methods without getting the "The request was aborted: The request was canceled" anymore. However we are now getting a WSE exception with the following details in our web method call completion events

     

    Message: WSE910: An error happened during the processing of a response message, and you can find the error in the inner exception.  You can also find the response message in the Response property.

     

    InnerExceptionTongue Tiedystem.InvalidOperationException: WSE2256: Protection requirements not satisfied.
       at Microsoft.Web.Services3.Security.SecureConversationClientReceiveSecurityFilter.ValidateSecureConversationMessageSecurity(SoapEnvelope envelope, Security security, MessageProtectionRequirements response)
       at Microsoft.Web.Services3.Security.SecureConversationClientReceiveSecurityFilter.ValidateMessageSecurity(SoapEnvelope envelope, Security security)
       at Microsoft.Web.Services3.Security.ReceiveSecurityFilter.ProcessMessage(SoapEnvelope envelope)
       at Microsoft.Web.Services3.Pipeline.ProcessInputMessage(SoapEnvelope envelope)
       at Microsoft.Web.Services3.Xml.SoapEnvelopeReaderWrapper..ctor(SoapClientMessage message, String messageContentType)

     

    I have scoured the web looking for this WSE error code WSE2256 and i can't find it anywhere at all nor found anyone else that has reported the same kind of problem.

     

    If anyone can help out with this it would be really appreciated

    Cheers

    Paul.

    Friday, October 05, 2007 9:29 AM