一个与可靠性有关的问题？

Question

HI，各位大侠

小弟最近在用WCF开发服务端时遇到一个非常令人郁闷的问题，至今未解决，盼求能得到高人的指点。

两台机器，一台Win2003做服务端，一台WinXP做压力测试机，压力机用来模拟1500个用户在一秒内并发的情况。

压力机每隔两秒创建一个线程，线程创建完后立即开始工作，每个线程创建一个通道，通道打开后每隔一秒调用一次服务，调用完200次后关闭通道退出。

现在的问题就是，如果所调用服务采用net.tcp绑定并使用了回话可靠性<reliableSession enabled = "true">，当压力机添加的线程数在少于800个时，通道的打开和服务的调用都很正常，但当线程添加到800个以上时，服务端就突然卡住了，像是什么资源耗尽了，而且是突然卡住的，不是先快后慢。然后就是80%以上的通道掉线，出错的客服两边“都报套接字连接已中止”的异常，但是就不清楚是什么原因终止的。

如果使用去掉可靠性的net.tcp，或者使用WSHttp协议，就没有问题，1500个线程都能顺利的创建并完成任务后退出。

这个问题一直困扰了我一周的时间都没有解决。我曾怀疑是服务端的处理能力有限，换一台性能更强的机器测试，结果还是一样，还有最令人不解的是为什么是添加了可靠性后才会掉线，变得反而不可靠了。

一下是代码和配置，省去一些不相关的部分

服务端异常

<TraceIdentifier>http://msdn.microsoft.com/zh-CN/library/System.ServiceModel.Channels.SocketConnectionAbort.aspx</TraceIdentifier>
<Description>SocketConnection 已中止</Description>
<AppDomain>IBoxHost.exe</AppDomain>
<Source>System.ServiceModel.Channels.SocketConnection/21950498</Source>
</TraceRecord>
</DataItem>
</TraceData>
<System.Diagnostics xmlns="http://schemas.microsoft.com/2004/08/System.Diagnostics">
<LogicalOperationStack></LogicalOperationStack>
<Timestamp>29098404888</Timestamp>
<Callstack>
在 System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
在 System.Environment.get_StackTrace()
在 System.Diagnostics.TraceEventCache.get_Callstack()
在 System.Diagnostics.XmlWriterTraceListener.WriteFooter(TraceEventCache eventCache)
在 System.Diagnostics.XmlWriterTraceListener.TraceData(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, Object data)
在 System.Diagnostics.TraceSource.TraceData(TraceEventType eventType, Int32 id, Object data)
在 System.ServiceModel.Diagnostics.DiagnosticTrace.TraceEvent(TraceEventType type, TraceCode code, String description, TraceRecord trace, Exception exception, Object source)
在 System.ServiceModel.Diagnostics.TraceUtility.TraceEvent(TraceEventType severity, TraceCode traceCode, TraceRecord extendedData, Object source, Exception exception)
在 System.ServiceModel.Channels.SocketConnection.Abort(TraceEventType traceEventType, String timeoutErrorString, TransferOperation transferOperation)
在 System.ServiceModel.Channels.SocketConnection.Abort(String timeoutErrorString, TransferOperation transferOperation)
在 System.ServiceModel.Channels.SocketConnection.OnReceiveTimeout(Object state)
在 System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
在 System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
在 System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
在 System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
在 System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
在 System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
在 System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
在 System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
在 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

客户端异常

<ExceptionString>System.ServiceModel.CommunicationException: 套接字连接已中止。这可能是由于处理消息时出错或远程主机超过接收超时或者潜在的网络资源问题导致的。本地套接字超时是“00:00:59.9843750”。 ---&gt; System.Net.Sockets.SocketException: 远程主机强迫关闭了一个现有的连接。
   在 System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   在 System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
   --- 内部异常堆栈跟踪的结尾 ---</ExceptionString>
<InnerException>
<ExceptionType>System.Net.Sockets.SocketException, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>远程主机强迫关闭了一个现有的连接。</Message>
<StackTrace>
在 System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
在 System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
</StackTrace>
<ExceptionString>System.Net.Sockets.SocketException: 远程主机强迫关闭了一个现有的连接。
   在 System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   在 System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)</ExceptionString>
<NativeErrorCode>2746</NativeErrorCode>
</InnerException>
</Exception>
</TraceRecord>
</DataItem>
</TraceData>
<System.Diagnostics xmlns="http://schemas.microsoft.com/2004/08/System.Diagnostics">
<LogicalOperationStack></LogicalOperationStack>
<Timestamp>71042088658</Timestamp>
<Callstack>
在 System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
在 System.Environment.get_StackTrace()
在 System.Diagnostics.TraceEventCache.get_Callstack()
在 System.Diagnostics.XmlWriterTraceListener.WriteFooter(TraceEventCache eventCache)
在 System.Diagnostics.XmlWriterTraceListener.TraceData(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, Object data)
在 System.Diagnostics.TraceSource.TraceData(TraceEventType eventType, Int32 id, Object data)
在 System.ServiceModel.Diagnostics.DiagnosticTrace.TraceEvent(TraceEventType type, TraceCode code, String description, TraceRecord trace, Exception exception, Object source)
在 System.ServiceModel.Diagnostics.ExceptionUtility.ThrowHelper(Exception exception, TraceEventType eventType, TraceRecord extendedData)
在 System.ServiceModel.Diagnostics.ExceptionUtility.ThrowHelper(Exception exception, TraceEventType eventType)
在 System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
在 System.ServiceModel.Channels.SocketConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
在 System.ServiceModel.Channels.DelegatingConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
在 System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
在 System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
在 System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
在 System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
在 System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
在 System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.SyncWaiter.TryGetChannel()
在 System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.SyncWaiter.TryWait(TChannel& channel)
在 System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.TryGetChannel(Boolean canGetChannel, Boolean canCauseFault, TimeSpan timeout, MaskingMode maskingMode, TChannel& channel)
在 System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.TryGetChannelForOutput(TimeSpan timeout, MaskingMode maskingMode, TChannel& channel)
在 System.ServiceModel.Channels.ReliableChannelBinder`1.Send(Message message, TimeSpan timeout, MaskingMode maskingMode)
在 System.ServiceModel.Channels.SendReceiveReliableRequestor.OnRequest(Message request, TimeSpan timeout, Boolean last)
在 System.ServiceModel.Channels.ReliableRequestor.Request(TimeSpan timeout)
在 System.ServiceModel.Channels.ClientReliableSession.Open(TimeSpan timeout)
在 System.ServiceModel.Channels.ClientReliableDuplexSessionChannel.OnOpen(TimeSpan timeout)
在 System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
在 System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
在 System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
在 System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
在 System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
在 System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
在 System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
在 System.ServiceModel.Channels.ServiceChannelProxy.ExecuteMessage(Object target, IMethodCallMessage methodCall)
在 System.ServiceModel.Channels.ServiceChannelProxy.InvokeChannel(IMethodCallMessage methodCall)
在 System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
在 System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
在 System.ServiceModel.ICommunicationObject.Open(TimeSpan timeout)
在 System.ServiceModel.ClientBase`1.System.ServiceModel.ICommunicationObject.Open(TimeSpan timeout)
在 System.ServiceModel.ClientBase`1.Open()
在 DebugClient2.Program.ThreadFuncTestCallBack(Object obj)
在 System.Threading.ThreadHelper.ThreadStart_Context(Object state)
在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
在 System.Threading.ThreadHelper.ThreadStart(Object obj)

 

//服务契约



 [ServiceContract(SessionMode = SessionMode.Required)]



 //[DeliveryRequirements(RequireOrderedDelivery = true)]



 public interface INoramlIBServiceSessionMode



 {



 [OperationContract]



 string SumbitExsitString(string v);



 }







 [ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)]



 public class NoramlIBServiceSessionMode : INoramlIBServiceSessionMode



 {



 int record=0;



 //提交存在



 public string SumbitExsitString(string v)



 {



  Console.Write(" [{0}] ",record++);



  return v;



 }



 }

 

客户端代码

 

 



//添加线程



 static void MainDebugCourse3(string userid, string password)



 {



  for (int i = 0; i < 500; i++)



  {



  Thread thread1 = new Thread(new ParameterizedThreadStart(ThreadFuncTestPerSession));



  thread1.Start(new UserInfo(i.ToString(), password));



  Console.WriteLine("启动了第{0}个登录线程", i);



  Thread.Sleep(500);



  }



 }











 //线程函数



 static void ThreadFuncTestPreSession(object obj)



 {



  const int submittime = 200;



  try



  {







  Thread.Sleep(1000);







  //创建通道



  string pointer_addr = String.Format("net.tcp://192.168.0.2:8003/normal_service_session_address");



  EndpointAddress remoteNodeEndPointAddress = new EndpointAddress(pointer_addr);



  string pointer_config = "CustomBinding_INoramlIBServiceSessionMode";



  //ChannelFactory<INoramlIBServiceSessionMode> myfactory = new ChannelFactory<INoramlIBServiceSessionMode>(pointer_config, remoteNodeEndPointAddress);



  ChannelFactory<INoramlIBServiceSessionMode> myfactory = new ChannelFactory<INoramlIBServiceSessionMode>(pointer_config);



  INoramlIBServiceSessionMode channel = myfactory.CreateChannel();



  ((IChannel)channel).Closed += new EventHandler(ShowOffLine);



  ((IChannel)channel).Faulted += new EventHandler(ShowOffLineFaulted);



  if (((ICommunicationObject)channel).State != CommunicationState.Opened)



   ((ICommunicationObject)channel).Open();







  //发送心跳包



  for (int i = 0; i < submittime; i++)



  {



   channel.SumbitExsitString("s");







   Thread.Sleep(1100);



  }



  ((ICommunicationObject)channel).Close();



  }



  catch (FaultException<IBExceptionDetail> ex)



  {



  System.Console.WriteLine(ex.ToString() + ex.Detail.m_return_code.ToString());



  }



  catch (Exception ex_other)



  {



  System.Console.WriteLine(ex_other.ToString());



  }



 }

 

服务端配置

 <!--会话服务(TCP)-->



 <service behaviorConfiguration="NormalSerivceTCP_Behavior"



 name="IBNormalServiceLib.NoramlIBServiceSessionMode">



 <!--唯一使用的终结点-->



 <endpoint address="normal_service_session_address"



   binding="customBinding"



   bindingConfiguration="ServiceBigDataBinding"



   contract="IBNormalServiceLib.INoramlIBServiceSessionMode">



  <identity>



  <dns value="localhost" />



  </identity>



 </endpoint>



 



 <endpoint address="net.tcp://localhost:8004/" binding="mexTcpBinding" 







contract="IMetadataExchange" />



 <host>



  <baseAddresses>



  <add baseAddress="net.tcp://localhost:8003/" />



  </baseAddresses>



 </host>



 </service>







 <customBinding>



 <binding name="ServiceBigDataBinding" closeTimeout="00:20:00"



  openTimeout="00:20:00" receiveTimeout="03:00:00" sendTimeout="00:20:00" >



  <reliableSession flowControlEnabled="true" ordered="true" 







inactivityTimeout="03:00:00"/>



  <binaryMessageEncoding maxReadPoolSize="64" maxWritePoolSize="32" 







maxSessionSize="4096">



  <readerQuotas maxDepth="64" maxStringContentLength="131072" 







maxArrayLength="1638400"



   maxBytesPerRead="16384" maxNameTableCharCount="32768"/>  



  </binaryMessageEncoding>



  <tcpTransport manualAddressing="false" maxBufferPoolSize="524288" 







maxReceivedMessageSize="52428800"



   connectionBufferSize="65536" hostNameComparisonMode="StrongWildcard"



   channelInitializationTimeout="00:00:05" maxBufferSize="52428800"



   maxPendingConnections="100" maxOutputDelay="00:00:00.2000000"



   maxPendingAccepts="100" transferMode="Buffered" listenBacklog="400"



   portSharingEnabled="false" teredoEnabled="false">



  <connectionPoolSettings groupName="default" leaseTimeout="00:05:00" 







idleTimeout="00:02:00"



     maxOutboundConnectionsPerEndpoint="100"/>  



  </tcpTransport>



 </binding>



 </customBinding>











 <behaviors>



 <serviceBehaviors>



 <behavior name="NormalSerivceTCP_Behavior">







  <serviceThrottling



   maxConcurrentCalls="10000"



   maxConcurrentSessions="10000"



   maxConcurrentInstances="50000" />



  <serviceMetadata httpGetEnabled="false" />



  <serviceDebug includeExceptionDetailInFaults="true" />



 </behavior>



 </serviceBehaviors>



 </behaviors>

 

客户端配置

 <endpoint address="net.tcp://localhost:8003/normal_service_session_address"



 binding="netTcpBinding" bindingConfiguration="CustomBinding_INoramlIBServiceSessionMode"



 contract="ServiceReference_Session.INoramlIBServiceSessionMode"



 name="CustomBinding_INoramlIBServiceSessionMode">



 <identity>



  <dns value="localhost" />



 </identity>



 </endpoint>







 <binding name="CustomBinding_INoramlIBServiceSessionMode" closeTimeout="00:01:00"



  openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"



  transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions"



  hostNameComparisonMode="StrongWildcard" listenBacklog="10" maxBufferPoolSize="524288"



  maxBufferSize="65536" maxConnections="10" maxReceivedMessageSize="65536">



  <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"



  maxBytesPerRead="4096" maxNameTableCharCount="16384" />



  <reliableSession ordered="true" inactivityTimeout="00:10:00"



  enabled="false" />



  <security mode="None">



  <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />



  <message clientCredentialType="Windows" />



  </security>



 </binding>

Answer 1

如果请求很多。

并发了大量调用，服务端会对消息排队。

默认是1分钟内部处理就出错。

在你的服务的配置文件里。

很多TimeOut相关的属性。服务端设置了较大的值。但是客户端没有。

一样会出错，这里两端要匹配。receiveTimeout默认的就是10分钟。

receiveTimeout="00:10:00" .如果启用可靠传输，那么还有InactivityTimeOut都要设置一下。

 

---

Frank Xu Lei--谦卑若愚，好学若饥
专注于.NET平台下分布式应用系统开发和企业应用系统集成
Focus on Distributed Applications Development and EAI based on .NET
 

---

【老徐的网站】：http://www.frankxulei.com/

微软WCF中文技术论坛
微软WCF英文技术论坛

Answer 2

非常感谢版主的回复，我回去吧服务端和客户端的receiveTimeout和InactivityTimeOut都已经设置得很大了 03：00：00 ，但还是会出错？