none
WCF hosted in winows service concurrent calls tcp 10061 error RRS feed

  • General discussion

  • hi, i have a WCF service hosted as a windows service on windows 2008 R2 Standard that uses NetTcpBinding and so far service worked fine. From recently as number of client requests increased dramatically due to change of client architecture (widgets are used on front and one widget is one wcf request, and average number of widgets on one page is 5, so i have 5 times more concurrent requests), form time to time client receives exception:

    The connection attempt lasted for a time span of 00:00:01.3622175. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.88.92:887. 

    So for example one client sends 5 requests and on 4 it gets appropriate answer but one of them is faulted.

    This is configuration of tcp binding:

    <netTcpBinding>
      <binding name="PlugInBinding" closeTimeout="00:01:00" openTimeout="00:10:00"
    receiveTimeout="00:20:00" sendTimeout="00:10:00" maxConnections="2000" listenBacklog="2000">
           <reliableSession enabled="false" />
            <security mode="None" />
      </binding>
    </netTcpBinding>

    Also I have a Service Throttling in Service Behavior

    <behaviors>
      <serviceBehaviors>  
         <behavior name="PlugInBehavior">
           <serviceMetadata httpGetEnabled="false" />
           <serviceDebug includeExceptionDetailInFaults="true" />
           <serviceThrottling maxConcurrentSessions="10000" maxConcurrentCalls="2000"/>
         </behavior>
      </serviceBehaviors>
    </behaviors>

    With this setup, i'm able to send about 350 requests concurrently to my WCF service without exception but when the number of concurrent calls is about 500, 50 or more request are not executed and they receive exception from beginning of the thread.

    I tried to look with wcf diagnostics but i could not find anything that i could understand:

    <Exception>
    <ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
    <Message>An existing connection was forcibly closed by the remote host</Message>
    <StackTrace>
    at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()
    at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)
    at System.ServiceModel.Channels.SocketConnection.OnReceiveAsyncCompleted(Object sender, SocketAsyncEventArgs e)
    at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
    at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncFailure(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
    at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
    at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
    </StackTrace>
    <ExceptionString>System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host</ExceptionString>
    <NativeErrorCode>2746</NativeErrorCode>
    </Exception>

    and wcf diagnostics on the client showed:

    <Exception>
    <ExceptionType>System.ServiceModel.EndpointNotFoundException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
    <Message>Could not connect to net.tcp://172.16.88.92:887/BankAdapterPlugin/AccountAccessNewAdapter. The connection attempt lasted for a time span of 00:00:01.4852565. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.88.92:887. </Message>
    <StackTrace>
    at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
    at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
    at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
    at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
    at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
    at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
    at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
    at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
    at Pexim.Ibank.BankAdapterPlugIn.ServiceContracts.IAccountAccess.GetAccountBalance(GetAccountBalanceRequest request)
    at Client.Program.Widgets(Object request)
    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Threading.ThreadHelper.ThreadStart(Object obj)
    </StackTrace>
    <ExceptionString>System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://172.16.88.92:887/BankAdapterPlugin/AccountAccessNewAdapter. The connection attempt lasted for a time span of 00:00:01.4852565. TCP error code 10061: No connection could be made because the target machine actively refused it 172.16.88.92:887.  ---&gt; System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.88.92:887
       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
       at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
       at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
       --- End of inner exception stack trace ---</ExceptionString>
    <InnerException>
    <ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
    <Message>No connection could be made because the target machine actively refused it 172.16.88.92:887</Message>
    <StackTrace>
    at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
    at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
    at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
    </StackTrace>
    <ExceptionString>System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 172.16.88.92:887
       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
       at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
       at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)</ExceptionString>
    <NativeErrorCode>274D</NativeErrorCode>
    </InnerException>
    </Exception>

    Does anybody have an idea of what i could do to push about 1000 concurrent requests through my wcf service?

    Friday, March 15, 2013 10:19 PM

All replies

  • Hi,

    You may check if there any limitation for concurrent TCP connections for Windows 2008. Take a look at a thread:

    http://social.technet.microsoft.com/Forums/en-US/winservergen/thread/fb7a6ef7-5a70-43a4-b0fe-c0252877467b/

    And do you try this on another machine with a different system?

    You can use SvcTraceViewer.exe to helps you analyze diagnostic traces that are generated by WCF.

    Best Regards.


    Haixia
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Tuesday, March 19, 2013 3:22 AM
    Moderator
  • Had the same issue

    http://stackoverflow.com/questions/34681487/thread-starvation-on-net-tcp-binding-tcp-error-code-10061

    From what I found that I might be running out of available ports. On Windows, when you release a port, it doesn't immediately go into an available state, but instead sits in a TIME_WAIT state for some interval. Until it leaves this state, no app can use this port. The time delay, the max number of ports, and the available port ranges are all different to the OS, XP vs Win7 vs Win2008 Server.

    There are two registry entries that can reduce this time interval: HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/TCPTimedWaitDelay

    and increase the max number of ports that can be opened by an app: HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/MaxUserPort


    • Edited by oerm85 Monday, January 11, 2016 2:23 PM
    Monday, January 11, 2016 2:19 PM