none
WCF clients start failing with TimeoutException as number of client processes increases RRS feed

  • Question

  • I'm seeing strange behavior from WCF client processes that manifests as the number of WCF client processes on the client machine increases.  The client process makes HTTP SOAP WCF requests to the server via a load balancer.  Normally, everything works fine with about 15 processes on the client machine communicating with the server.

    I started WCF and network traces on both the client and server while everything was working.  I increased the number of client processes to about 21 and I started seeing WCF failures on the client.  Decreased the number of client processes and everything returned to normal.

    Looking at the network traces, none of the processes which logged the WCF failures have any data in the network trace.  Below is the logged exception.  Any help is appreciated. 

    Thanks,

    ~James

    Exception - System.TimeoutException: The request channel timed out attempting to send after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to '<redacted uri>' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.
       at System.ServiceModel.Channels.HttpChannelUtilities.SetRequestTimeout(HttpWebRequest request, TimeSpan timeout)
       at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
       at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
       --- End of inner exception stack trace ---

    Server stack trace: 
       at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
       at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    ...


    Wednesday, March 9, 2016 9:54 PM

Answers

  • Figured this one out.  Root cause appears to be the confluence of two issues:

    1. Name resolution takes 7s
    2. winhttp!WinHttpGetProxyForUrl() calls into WinHttpAutoProxySvc which serializes calls across processes.

    As each serialized call into WinHttpAutoProxySvc takes 7s, the first eight concurrent callers will succeed (7s * 8 callers == 56s delay) while the 9th and subsequent callers time out at 60s.

    Workaround for slow DNS is to stop and disable WinHttpAutoProxySvc on the client machine.


    Thursday, March 10, 2016 7:47 PM

All replies

  • Hi 0xbaadf00d,

    According to your description, if increased the number of client processes to about 21, WCF service would throw an TimeOut exception, it seems that you need optimize your WCF Service performs. The easiest way is to modify the configuration file. for more information, please refer to following link.

    http://www.codeproject.com/Articles/133738/Quick-Ways-to-Boost-Performance-and-Scalability-of

    https://msdn.microsoft.com/en-us/library/ee377061(v=bts.10).aspx

    Best regards.

    Thursday, March 10, 2016 8:56 AM
  • Thanks for your reply!  Turns out the problem was not related to WCF limits at all.
    Thursday, March 10, 2016 7:45 PM
  • Figured this one out.  Root cause appears to be the confluence of two issues:

    1. Name resolution takes 7s
    2. winhttp!WinHttpGetProxyForUrl() calls into WinHttpAutoProxySvc which serializes calls across processes.

    As each serialized call into WinHttpAutoProxySvc takes 7s, the first eight concurrent callers will succeed (7s * 8 callers == 56s delay) while the 9th and subsequent callers time out at 60s.

    Workaround for slow DNS is to stop and disable WinHttpAutoProxySvc on the client machine.


    Thursday, March 10, 2016 7:47 PM