WCF-BasicHttp adapter timeouts - no connections on WebService side RRS feed

  • Question

  • Helo,

    I am getting the exception shown below. BizTalk 2010 is able to call the service when
    there is very light load. But once I increase the load and the number of
    messsages to transmit increases after a while I keep getting this message for 
    ALL the calls to the service.

    On WebService side on IIS no connections are seen open. Application pool even goes to sleep ...

    When I restart BizTalk host instance, message flow resumes and no timeouts for a while (1 hour)

    Web service says reachable from BizTalk machine in all other ways.

    Adapter does not report any errors except for those timeouts...

    The adapter failed to transmit message going to send port "SP_IN_SyncCompany" with URL "http://Server/folder/service.asmx".

    It will be retransmitted after the retry interval specified for this Send Port.

    Details:"System.TimeoutException: The HTTP request to 'http://Server/folder/service.asmx' has exceeded the allotted timeout of 00:04:00.

    The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The request was aborted: The request was canceled. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result) --- End of inner exception stack trace --- Server stack trace: at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndRequest(IAsyncResult result) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at System.ServiceModel.Channels.IRequestChannel.EndRequest(IAsyncResult result) at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)".

    Friday, March 2, 2012 2:36 PM

All replies

  • it sounds like BizTalk adapter is running out of connections.

    You might try turning on ordered delivery on the send port that will single thread the message to the web service, but might help determine if it is a connection issues.

    do you have any of the Cumlative Updates on BizTalk 2010?

    Bill Chesnut | BizTalk Server MVP | Mexia Consulting | Melbourne Australia
    Please indicate "Mark as Answer" if this post has answered the question.

    Saturday, March 3, 2012 5:48 AM
  • Messages sent here take up to 4 minutes in some cases and there is at least 20-30 msg/minute traffic. Port waits for a response, so I can't turn on ordered delivery.

    I've got BT2010 CU2 installed on the system.

    Saturday, March 3, 2012 11:59 AM
  • What software is running the web service?  Is it running out of connections?  What retry setting do you have on the port?

    for that volume and that processing time the destination web service is going to have to support at least 120 to 150 concurrent connections, which guessing at a server that takes 4 minutes to return will be an issue.

    I think you will need to create an orchestration that throttles the number of messages being sent at once or do some sort of sharding so that the messages are sent via 6-10 send ports with those set to ordered delivery.

    Sorry, but I don't think this is going to be easy to fix.

    Bill Chesnut | BizTalk Server MVP | Mexia Consulting | Melbourne Australia
    Please indicate "Mark as Answer" if this post has answered the question.

    Sunday, March 4, 2012 12:49 AM
  • Web service runs on IIS and it is hardly a problem on web service side, because IIS log is clean (no connections are received on Web service side during timeouts). When I restart BT host instances connections appear for a while and all is well. After a random interval of time the problem repeats.

    IIS server has enough resources and connection limit is not reached at any point.

    Throttling is already configured on BizTalk host as not to flood the web service.

    Also backup transport is configured on send port, but this only delays the problem.

    Sunday, March 4, 2012 1:53 AM
  • What OS are you running BizTalk on?

    How have you configured Throttling on Biztalk?

    Bill Chesnut | BizTalk Server MVP | Mexia Consulting | Melbourne Australia
    Please indicate "Mark as Answer" if this post has answered the question.

    Sunday, March 4, 2012 4:38 AM
  • Hello Marius!

    As Bill states we need to know what type of throttling you have set these may have impact on the message flow out of BizTalk.

    Can you put som tracing on for the IIS where the message is sent and provide us with some more errors?

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

    Thursday, March 8, 2012 11:32 AM
  • Helo Tord,

    Destination system does not get any connections from BT when problem starts. So there are no errors (there are no failed or in fact any connections from BT acording to IIS log).
    BT stops sending messages (at least accordig to WireShark trace). My guess is that messages do not leave BT. I'd say it is throttling settings problem, but then messages shouldn't time out...

    System is a machine with 24 CPU cores and Windows Server 2008 R2 Enterprise (with SP1).

    Throttling parameters

    Resource based Throttling:
    Per CPU In-process messages - 1
    Internal Message queue size - 500

    Rate-Based Throttling->Delivery:

    Minimum number of samples - 5
    Sampling window duration - 20 s
    Rate overdrive factor - 100 %

    All other settings are default.

    Thursday, March 15, 2012 6:42 AM
  • Hi Marius,

    You can check if your system is throttling at runtime using PAL.

    Collect all important the performance counters when issue is happening and run the PAL on those counters to check whats going on in your system.

    You can also run MessageBoxViewer tool for more analysis of your biztalk engine configurations and identify any issues in case there are any.

    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.

    Thursday, March 15, 2012 7:04 AM
  • Hello Shailesh,

    BT is throttling when described problem occurs.
    Infact BT is there to handle messages as not to flood the WS. Timeouts don't seem to corelate with throttling state, but rather with extreme load of mesages (~2000-3000 msg at the same time is a problem). Shouldn't be a problem for BT though...

    Thursday, March 15, 2012 7:37 AM
  • Hi Marius,

    Please see the link below to understand how throttling works in BizTalk and there mitigation strategies.

    When BT throttles it actually slows down the processing which might overall causes bottleneck for end to end processing. There are different types of throttlings in BizTalk and they have there own mitigation strategies. So its important to understand what type of throttling is happening.

    Your issue you are reporting might be repurcation of lack of resources like allocated threads or allocated memory for host process etc. I would suggest to run the tool and counters and then fix the throttling and it will smoothen out your end to end processing flow.

    I hope this helps you.

    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.

    Thursday, March 15, 2012 8:28 AM