WCF Service call fail

Question

Hi,

I am getting WCF service call failed issue suddenly. All these service call are working fine and we have now working in Release2 of the project. We have't changed in the service logic. They are same as in the Release1.

The error is as below:

The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '10675199.02:48:05.4775807'.

Following warning messages are logged in the trace file.

Warning Messages:

1.)Description: MaxOutboundConnectionsPerEndpoint quota (10) has been reached, so the connection was closed and not reused by the listener.

Message ystem.ServiceModel.Channels.ServerMaxPooledConnectionsQuotaReached.aspx</TraceIdentifier><Description>MaxOutboundConnectionsPerEndpoint quota (10) has been reached, so the connection was closed and not reused by the listener.</

2.)Description: Handling an exception

Message:A TCP error (995: The I/O operation has been aborted because of either a thread exit or an application request) occurred while transmitting data.

Could anyone explain me how we can fix this issue? Do I need to specify/set any thing in the web.config file?

Please help me in this regard.

Padma

Answer 1

This might be due to heavy load on your service. You are hitting the limit of your service. 

Try adjusting your reader quotas like the following 

<readerQuotas maxDepth="64" maxStringContentLength="20480000" maxArrayLength="20480000"
              maxBytesPerRead="8152" maxNameTableCharCount="16384" /> 

---

Tanvir Huda Application Architect/Consultant

Answer 2

Hi,

I have given maximum values. Below is my web.config file details: Please let me know if I need to do any correction to solve this problem.

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

  <system.serviceModel>

    <behaviors>

      <endpointBehaviors>

        <behavior name="DisallowNtlm">

          <clientCredentials>

            <windows allowNtlm="false" />

          </clientCredentials>

        </behavior>

      </endpointBehaviors>

    </behaviors>

    <bindings>

      <wsHttpBinding>

        <binding name="largeMessageWSHttpBinding" openTimeout="23:00:00" closeTimeout="23:00:00" receiveTimeout="infinite" sendTimeout="infinite" maxReceivedMessageSize="2147483647" bypassProxyOnLocal="true" useDefaultWebProxy="false">

          <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>

        </binding>

              </wsHttpBinding>

      <basicHttpBinding>

        <binding name="largeMessageBasic" openTimeout="23:00:00" closeTimeout="23:00:00" receiveTimeout="infinite" sendTimeout="infinite" maxReceivedMessageSize="2147483647" bypassProxyOnLocal="true" useDefaultWebProxy="false">

          <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>

        </binding>

        <binding name="largeMessageBasicHttpBinding" openTimeout="23:00:00" closeTimeout="23:00:00" receiveTimeout="infinite" sendTimeout="infinite" maxReceivedMessageSize="2147483647" bypassProxyOnLocal="true" useDefaultWebProxy="false">

          <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>

        </binding>

     </basicHttpBinding>

      <customBinding>

        <binding name="largeMexHttpBinding">

          <textMessageEncoding>

            <readerQuotas maxNameTableCharCount="2000000" />

          </textMessageEncoding>

          <httpTransport />

        </binding>

      </customBinding>

      <netTcpBinding>

        <binding name="largeMessageNetTcpBinding" openTimeout="23:00:00" closeTimeout="23:00:00" receiveTimeout="infinite" sendTimeout="infinite" listenBacklog="200" maxBufferPoolSize="2147483647" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647" portSharingEnabled="true">

          <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>

        </binding>

              </netTcpBinding>

    </bindings>

       <extensions>

      <behaviorExtensions>

        <add name="objectPoolingBehavior" type="SyncSoft.XPlatform.V1.Services.Trace.ObjectPoolingConfigurationElement, ServiceTrace, Version=1.0.0.0, Culture=neutral, PublicKeyToken=610a37dd3ea7f658" />

      </behaviorExtensions>

    </extensions>

    <diagnostics performanceCounters="Off">

      <messageLogging

         logEntireMessage="true"

         logMalformedMessages="false"

         logMessagesAtServiceLevel="true"

         logMessagesAtTransportLevel="false"

         maxMessagesToLog="-1"

         maxSizeOfMessageToLog="8388608"/>

    </diagnostics>

  </system.serviceModel>

</configuration>

Answer 3

i dont think this is due to throttling.  

As per Dan Risby blog, When a quota is exceeded a QuotaExceededException is thrown.  Without Quotas a malicious message could attempt to access all available memory can create an OutOfMemoryException, or access all available stacks to cause a StackOverflowException.  The best part of the QuotaExceededException is that the message causing it can just be discarded and the service can keep on running.  Without quotas the resulting exceptions could cause the service to terminate.

<bindings>
    <netTcpBinding>
        <binding name="netTcp"
            maxReceivedMessageSize="2147483647"
            maxConnections="2147483647">
            <readerQuotas
                maxDepth="64"
                maxStringContentLength="2147483647"
                maxArrayLength="2147483647"
                maxBytesPerRead="4096"
                maxNameTableCharCount="16384"/>
        </binding>
    </netTcpBinding>
</bindings>
By the way , did you check your client configuration settings as well. May be the channel is getting faulted at the client side. May be service is still ok. 

---

Tanvir Huda Application Architect/Consultant

Answer 4

using System;
using System.Collections.Generic;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Security;
using System.Diagnostics;
using Microsoft.Practices.EnterpriseLibrary.Logging;
namespace SyncSoft.XPlatform.V1.Services.Client
{
    /// <summary>
    /// Manage creating and calling operations on a channel.
    /// Gracefully handle the case where a channel needs to be recreated because
    /// it was closed at the server end (e.g. by an application pool being recycled).
    /// CheckedClientBase catches MessageSecurityExceptions and retries the call.
    /// </summary>
    /// <typeparam name="Contract"></typeparam>
    public class CheckedClientBase<TContract> : CommunicationObject, IDisposable where TContract : class
    {
      
		private TContract currentChannel;
		private IClientChannel CurrentChannelObject { get { return (IClientChannel)currentChannel; } }
		private ChannelFactory<TContract> factory;
		private int numberRetries;
        private int throttleWait;
        public ChannelFactory Factory
        {
            get { return factory; }
        }
        [DebuggerNonUserCode]
        public void Dispose()
        {
            this.Close();
        }
        [DebuggerNonUserCode]
        protected CheckedClientBase() : this("*")
        {
        }
        [DebuggerNonUserCode]      
        protected CheckedClientBase(string endpointConfiguration)
        {
			factory = new ChannelFactory<TContract>(endpointConfiguration);
            Init();
        }
		[DebuggerNonUserCode]
		protected CheckedClientBase(string endpointConfiguration, string remoteAddress) :
			this(endpointConfiguration, new EndpointAddress(remoteAddress))
		{
		}
        [DebuggerNonUserCode]      
        protected CheckedClientBase(string endpointConfiguration, EndpointAddress remoteAddress)
        {
			factory = new ChannelFactory<TContract>(endpointConfiguration, remoteAddress);
            Init();
        }
        [DebuggerNonUserCode]
        protected CheckedClientBase(Binding binding, EndpointAddress remoteAddress)
        {
			factory = new ChannelFactory<TContract>(binding, remoteAddress);
            Init();
        }
		[DebuggerNonUserCode]
		protected CheckedClientBase(ChannelFactory<TContract> factory)
		{
			this.factory = factory;
			Init();
		}
        [DebuggerNonUserCode]      
        private void Init()
		{
            numberRetries = 2;
            throttleWait = 100;
			currentChannel = factory.CreateChannel();
        }
        protected delegate TReturn ServiceCall<TReturn>(TContract client);
        
        /// <summary>
        /// Try a service call, retrying on any transient communication errors for which the message
		/// can't have reached the service (e.g. closed security session due to server app domain recycled).
		/// Other retries are best left to the application.
        /// </summary>
        /// <param name="call"></param>
        //[DebuggerNonUserCode]  
        protected TReturn TryServiceCall<TReturn>(ServiceCall<TReturn> call)
        {
			this.ThrowIfDisposed();
            Exception firstCommExcp = null;
            int numCallRetries = 0;
            while (true)
            {
				if (currentChannel != null && CurrentChannelObject.State != CommunicationState.Opened)
				{
					AbortChannel();
				}
                if (currentChannel == null)
                {
                    try
                    {
                        currentChannel = factory.CreateChannel();
                        CurrentChannelObject.Open();
                    }
                    catch (EndpointNotFoundException ex)
                    {
                        Logger.Write(ex.Message, "General",1,101,TraceEventType.Error);
                        throw;
                    }
                    catch(CommunicationException)
                    {
                        
                    }
                    
                }
				try
				{
					return call(currentChannel);
				}
                  catch(EndpointNotFoundException ex)
                  {
                      Logger.Write(ex.Message, "General", 1, 101, TraceEventType.Error);
                      throw;
                  }
				catch (FaultException ex)
				{
                    Logger.Write(ex.Message, "General", 1, 101, TraceEventType.Error);
                    Logger.Write(ex.InnerException!=null ?ex.InnerException.Message:"No Inner Exception", "General",1, 101, TraceEventType.Error);
                    throw;
				}
				catch (MessageSecurityException cexcp)
				{
					AbortChannel();
					if (numCallRetries == 0)
					{
						firstCommExcp = cexcp;
					}
					if (numCallRetries <= numberRetries)
					{
						if (numCallRetries > 0)
						{
							System.Threading.Thread.Sleep(throttleWait);
						}
						numCallRetries++;
					}
					else
					{
						try
						{
							this.Abort();
						}
						catch
						{
                            throw;
                        }
                        throw;
						
					}
				}
				catch (CommunicationException ex)
				{
                    if (numCallRetries <= numberRetries)
                    {
				    this.AbortChannel();
                        numCallRetries++;
                    }
                    else
                    {
                        Logger.Write(ex.Message, "General",1, 101, TraceEventType.Error);
                        Logger.Write(ex.InnerException != null ? ex.InnerException.Message : "No Inner Exception", "General",1, 101, TraceEventType.Error);
                        throw;
                    }
				    
				}
				catch (TimeoutException ex)
				{
                    if (numCallRetries <= numberRetries)
                    {
                        this.AbortChannel();
                        numCallRetries++;
                    }
                    else
                    {
                        Logger.Write(ex.Message, "General",1, 101, TraceEventType.Error);
                        Logger.Write(ex.InnerException != null ? ex.InnerException.Message : "No Inner Exception", "General",1, 101, TraceEventType.Error);
                        throw;
                    }
                       
                }
            }
        }
		[DebuggerNonUserCode]
		private void CloseChannel()
		{
			try
			{
				CurrentChannelObject.Close();
				CurrentChannelObject.Dispose();
			}
			catch (CommunicationException)
			{
				AbortChannel();
			}
			finally
			{
				currentChannel = null;
			}
		}
		[DebuggerNonUserCode]
		private void CloseChannel(TimeSpan timeout)
		{
			try
			{
				CurrentChannelObject.Close(timeout);
				CurrentChannelObject.Dispose();
			}
			catch (CommunicationException)
			{
			}
			finally
			{
				currentChannel = null;
			}
		}
		[DebuggerNonUserCode]
		private void AbortChannel()
		{
			try
			{
				CurrentChannelObject.Abort();
			}
			catch (CommunicationException)
			{
			}
			finally
			{
				currentChannel = null;
			}
		}
		protected override void OnAbort()
		{
			if (currentChannel != null)
			{
				AbortChannel();
			}
		}
		protected override void OnClose(TimeSpan timeout)
		{
			if (currentChannel != null)
			{
				CloseChannel(timeout);
			}
		}
		protected override IAsyncResult OnBeginClose(TimeSpan timeout, AsyncCallback callback, object state)
		{
			if (currentChannel != null)
			{
				currentChannel = factory.CreateChannel();
			}
			return CurrentChannelObject.BeginClose(timeout, callback, state);
		}
		protected override IAsyncResult OnBeginOpen(TimeSpan timeout, AsyncCallback callback, object state)
		{
			if (currentChannel == null)
			{
				currentChannel = factory.CreateChannel();
			}
			return CurrentChannelObject.BeginOpen(timeout, callback, state);
		}
		protected override void OnEndClose(IAsyncResult result)
		{
			CurrentChannelObject.EndClose(result);
		}
		protected override void OnEndOpen(IAsyncResult result)
		{
			CurrentChannelObject.EndOpen(result);
		}
		protected override void OnOpen(TimeSpan timeout)
		{
			if (currentChannel == null)
			{
				currentChannel = factory.CreateChannel();
			}
			CurrentChannelObject.Open(timeout);
		}
		protected override TimeSpan DefaultOpenTimeout
		{
			get
			{
				return new TimeSpan(0, 1, 0);
			}
		}
		protected override TimeSpan DefaultCloseTimeout
		{
			get
			{
				return new TimeSpan(0, 1, 0);
			}
		}
	}
}

This the ClientBase code to create the channel

Please let me know if there is any issue this side.

Regards

Answer 5

Hi,

I have all max values set in my web.config file. Please help me if there is  any other issues where I need to look. This has become major issue now.

You can see my config file and client channel connection creation below.

Regards

Padma

Answer 6

Hi,

I am not getting any respone...

Please respond

Padma