none
WCF: Process uses sometimes 100% CPU RRS feed

  • Question

  • Hello,

    We have a self hoststed WCF-Servive that gets WCF-requests from many client stations (~200) and if it is neccesary route this WCF-requests to other sub-stations.

    Our Config:

    <netTcpBinding>
    	<binding name="XAMServerTCPEndpointBindung"
    			 closeTimeout="00:01:00"
    			 openTimeout="00:01:00"
    			 receiveTimeout="00:10:00"
    			 sendTimeout="00:01:00"
    			 transactionFlow="false"
    			 transferMode="Buffered"
    			 transactionProtocol="OleTransactions"
    			 hostNameComparisonMode="StrongWildcard"
    			 listenBacklog="10"
    			 maxBufferPoolSize="0"
    			 maxBufferSize="262144000"
    			 maxConnections="10"
    			 maxReceivedMessageSize="262144000">
    	  <readerQuotas maxDepth="256" maxStringContentLength="262144000" maxArrayLength="262144000"
    		  maxBytesPerRead="262144000" maxNameTableCharCount="262144000" />
    	  <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" />
    	  <security mode="None">
    		<transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
    	  </security>
    	</binding>
    </netTcpBinding>
      
    <behavior name="ServiceBehaviorMeta">
      <serviceMetadata httpGetEnabled="true" httpGetUrl="http://localhost:2121/XAMServer/meta"/>
      <dataContractSerializer maxItemsInObjectGraph="6553600"/>
      <serviceDebug includeExceptionDetailInFaults="true"/>
      <serviceThrottling maxConcurrentCalls="512" maxConcurrentInstances="1000"  maxConcurrentSessions="512"/>
    </behavior>
    
    <service behaviorConfiguration="ServiceBehaviorMeta" name="XAMServer.XAMRTDB">
    	<endpoint address="http://localhost:2121/XAMServer" binding="basicHttpBinding" bindingConfiguration="XAMServerHttpEndpoint" name="XAMServerHttpEndpoint" contract="XAMServer.IXAMRTDB"/>
    	<endpoint address="net.pipe://localhost/ServerPipeBinding" binding="netNamedPipeBinding" bindingConfiguration="XAMServerPipeBinding" name="XAMServerEndpointPipe" contract="XAMServer.IXAMRTDB"/>
    	<endpoint address="net.tcp://localhost:2131/ServerTCPBinding" binding="netTcpBinding" bindingConfiguration="XAMServerTCPEndpointBindung" name="XAMServerEndpointTCP" contract="XAMServer.IXAMRTDB"/>
    </service>


    This service runs for long time very well. But some times the CPU load of the process/server jumpts to 100% and never grow down.
    The server have 16BG RAM and 16 Cores with 2,5GHz. (Operating System Win2012R2 Stantard)

    We made crash dumps and found out that many threads (~66%) are locked at this time. We saw that the lock occures in the WCF-serializing but we don't know who is locking it an why.

    Some stack traces of the locks:

    OS Thread Id: 0x4310 (79)
            Child SP               IP Call Site
    000000000cb6b858 00007ff902c6177a [GCFrame: 000000000cb6b858] 
    000000000cb6ba50 00007ff902c6177a [GCFrame: 000000000cb6ba50] 
    000000000cb6ba88 00007ff902c6177a [HelperMethodFrame: 000000000cb6ba88] System.Threading.Monitor.Enter(System.Object)
    000000000cb6bb80 00007ff8f480a36c System.Runtime.Serialization.DataContract+DataContractCriticalHelper.GetBuiltInDataContract(System.String, System.String)
    000000000cb6bc10 00007ff8f480a1fd System.Runtime.Serialization.XmlObjectSerializerContext.ResolveDataContractFromKnownTypes(System.String, System.String, System.Runtime.Serialization.DataContract, System.Type)
    000000000cb6bc60 00007ff8f47f4a07 System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.String, System.String, System.Type, System.Runtime.Serialization.DataContract ByRef)
    000000000cb6bce0 00007ff8f47f56cf System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, Int32, System.RuntimeTypeHandle, System.String, System.String)
    000000000cb6bd40 00007ff88f6be374 DynamicClass.ReadCommunicationValueItemReplyBOOLFromXml(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext, System.Xml.XmlDictionaryString[], System.Xml.XmlDictionaryString[])
    000000000cb6bdb0 00007ff8f47fa97c System.Runtime.Serialization.ClassDataContract.ReadXmlValue(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext)
    000000000cb6be00 00007ff8f47f4a7f System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.String, System.String, System.Type, System.Runtime.Serialization.DataContract ByRef)
    000000000cb6be80 00007ff8f47f56cf System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, Int32, System.RuntimeTypeHandle, System.String, System.String)
    000000000cb6bee0 00007ff88f6bbdd0 DynamicClass.ReadArrayOfCommunicationValueItemReplyFromXml(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext, System.Xml.XmlDictionaryString, System.Xml.XmlDictionaryString, System.Runtime.Serialization.CollectionDataContract)
    000000000cb6bf60 00007ff8f47f8b81 System.Runtime.Serialization.CollectionDataContract.ReadXmlValue(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext)
    000000000cb6bfd0 00007ff8f47f49e2 System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.String, System.String, System.Type, System.Runtime.Serialization.DataContract ByRef)
    000000000cb6c050 00007ff8f47f4905 System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.Type, System.Runtime.Serialization.DataContract, System.String, System.String)
    000000000cb6c0a0 00007ff8f47eee46 System.Runtime.Serialization.DataContractSerializer.InternalReadObject(System.Runtime.Serialization.XmlReaderDelegator, Boolean, System.Runtime.Serialization.DataContractResolver)
    000000000cb6c120 00007ff8f47efb43 System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(System.Runtime.Serialization.XmlReaderDelegator, Boolean, System.Runtime.Serialization.DataContractResolver)
    000000000cb6c210 00007ff8f47eed0c System.Runtime.Serialization.DataContractSerializer.ReadObject(System.Xml.XmlDictionaryReader, Boolean)
    000000000cb6c260 00007ff8cd2683da System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter+PartInfo.ReadObject(System.Xml.XmlDictionaryReader, System.Runtime.Serialization.XmlObjectSerializer)
    000000000cb6c290 00007ff8cd268397 System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameterPart(System.Xml.XmlDictionaryReader, PartInfo, Boolean)
    000000000cb6c450 00007ff8cd2681dd System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeBody(System.Xml.XmlDictionaryReader, System.ServiceModel.Channels.MessageVersion, System.String, System.ServiceModel.Description.MessageDescription, System.Object[], Boolean)
    000000000cb6c4d0 00007ff8cd268054 System.ServiceModel.Dispatcher.OperationFormatter.DeserializeBodyContents(System.ServiceModel.Channels.Message, System.Object[], Boolean)
    000000000cb6c580 00007ff8cd266755 System.ServiceModel.Dispatcher.OperationFormatter.DeserializeReply(System.ServiceModel.Channels.Message, System.Object[])
    000000000cb6c710 00007ff8cd264492 System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(System.ServiceModel.Dispatcher.ProxyRpc ByRef)
    000000000cb6c7a0 00007ff8cd263ee6 System.ServiceModel.Channels.ServiceChannel.HandleReply(System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.ServiceModel.Dispatcher.ProxyRpc ByRef)
    000000000cb6c920 00007ff8cd2625b8 System.ServiceModel.Channels.ServiceChannel.Call(System.String, Boolean, System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.Object[], System.Object[], System.TimeSpan)
    000000000cb6cb60 00007ff8cd261e9c System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage, System.ServiceModel.Dispatcher.ProxyOperationRuntime)
    000000000cb6cbd0 00007ff8cd261313 System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage)
    000000000cb6cca0 00007ff8ed2b7464 System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)
    000000000cb6d1a8 00007ff8ee517292 [TPMethodFrame: 000000000cb6d1a8] XAMCommon.ServiceReference.IXAMRTDB.GetValuesID(XAMCommon.ServiceReference.CommunicationValueItemRequest[], Int32, System.String)
    000000000cb6d210 00007ff88f6de53a XAMCommon.ServiceReference.XAMRTDBClient.GetValuesID(XAMCommon.ServiceReference.CommunicationValueItemRequest[], Int32, System.String) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\Service References\ServiceReference\Reference.cs @ 13606]
    000000000cb6d260 00007ff88f6dbd12 XAMCommunication.XAMPLCCom.ProcessOnlineValues(System.Collections.Generic.List`1) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\XAMCom.cs @ 3052]
    000000000cb6d9c0 00007ff88f6d86ec XAMCommunication.XAMPLCCom.timer_Tick(System.Object, System.EventArgs) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\XAMCom.cs @ 2094]
    000000000cb6e0c0 00007ff88f3717bd XAMCommunication.XAMPLCCom.BackgroundWorkerLoopA(System.Object, System.ComponentModel.DoWorkEventArgs) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\XAMCom.cs @ 1432]
    000000000cb6e540 00007ff8ec43e04f System.ComponentModel.BackgroundWorker.WorkerThreadStart(System.Object)
    000000000cb6e998 00007ff8ee50a7f3 [HelperMethodFrame_PROTECTOBJ: 000000000cb6e998] System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr, System.Object[], System.Object, System.Object[] ByRef)
    000000000cb6ed70 00007ff8edd186e7 System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessageSink)
    000000000cb6ee50 00007ff8ed3139a5 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    000000000cb6efb0 00007ff8ed313719 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    000000000cb6efe0 00007ff8ed34216f System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
    000000000cb6f030 00007ff8ed34136a System.Threading.ThreadPoolWorkQueue.Dispatch()
    000000000cb6f598 00007ff8ee50a7f3 [DebuggerU2MCatchHandlerFrame: 000000000cb6f598] 


    OS Thread Id: 0x424 (80)
            Child SP               IP Call Site
    0000000005fab3b0 00007ff8ee644440 [GCFrame: 0000000005fab3b0] 
    0000000005fab480 00007ff8ee644440 [GCFrame: 0000000005fab480] 
    0000000005fab4b8 00007ff8ee644440 [HelperMethodFrame: 0000000005fab4b8] System.Threading.Monitor.Enter(System.Object)
    0000000005fab5b0 00007ff8f480a36c System.Runtime.Serialization.DataContract+DataContractCriticalHelper.GetBuiltInDataContract(System.String, System.String)
    0000000005fab640 00007ff8f480a1fd System.Runtime.Serialization.XmlObjectSerializerContext.ResolveDataContractFromKnownTypes(System.String, System.String, System.Runtime.Serialization.DataContract, System.Type)
    0000000005fab690 00007ff8f47f4a07 System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.String, System.String, System.Type, System.Runtime.Serialization.DataContract ByRef)
    0000000005fab710 00007ff8f47f56cf System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, Int32, System.RuntimeTypeHandle, System.String, System.String)
    0000000005fab770 00007ff88f6be374 DynamicClass.ReadCommunicationValueItemReplyBOOLFromXml(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext, System.Xml.XmlDictionaryString[], System.Xml.XmlDictionaryString[])
    0000000005fab7e0 00007ff8f47fa97c System.Runtime.Serialization.ClassDataContract.ReadXmlValue(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext)
    0000000005fab830 00007ff8f47f4a7f System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.String, System.String, System.Type, System.Runtime.Serialization.DataContract ByRef)
    0000000005fab8b0 00007ff8f47f56cf System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, Int32, System.RuntimeTypeHandle, System.String, System.String)
    0000000005fab910 00007ff88f6bbdd0 DynamicClass.ReadArrayOfCommunicationValueItemReplyFromXml(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext, System.Xml.XmlDictionaryString, System.Xml.XmlDictionaryString, System.Runtime.Serialization.CollectionDataContract)
    0000000005fab990 00007ff8f47f8b81 System.Runtime.Serialization.CollectionDataContract.ReadXmlValue(System.Runtime.Serialization.XmlReaderDelegator, System.Runtime.Serialization.XmlObjectSerializerReadContext)
    0000000005faba00 00007ff8f47f49e2 System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.String, System.String, System.Type, System.Runtime.Serialization.DataContract ByRef)
    0000000005faba80 00007ff8f47f4905 System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator, System.Type, System.Runtime.Serialization.DataContract, System.String, System.String)
    0000000005fabad0 00007ff8f47eee46 System.Runtime.Serialization.DataContractSerializer.InternalReadObject(System.Runtime.Serialization.XmlReaderDelegator, Boolean, System.Runtime.Serialization.DataContractResolver)
    0000000005fabb50 00007ff8f47efb43 System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(System.Runtime.Serialization.XmlReaderDelegator, Boolean, System.Runtime.Serialization.DataContractResolver)
    0000000005fabc40 00007ff8f47eed0c System.Runtime.Serialization.DataContractSerializer.ReadObject(System.Xml.XmlDictionaryReader, Boolean)
    0000000005fabc90 00007ff8cd2683da System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter+PartInfo.ReadObject(System.Xml.XmlDictionaryReader, System.Runtime.Serialization.XmlObjectSerializer)
    0000000005fabcc0 00007ff8cd268397 System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameterPart(System.Xml.XmlDictionaryReader, PartInfo, Boolean)
    0000000005fabe80 00007ff8cd2681dd System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeBody(System.Xml.XmlDictionaryReader, System.ServiceModel.Channels.MessageVersion, System.String, System.ServiceModel.Description.MessageDescription, System.Object[], Boolean)
    0000000005fabf00 00007ff8cd268054 System.ServiceModel.Dispatcher.OperationFormatter.DeserializeBodyContents(System.ServiceModel.Channels.Message, System.Object[], Boolean)
    0000000005fabfb0 00007ff8cd266755 System.ServiceModel.Dispatcher.OperationFormatter.DeserializeReply(System.ServiceModel.Channels.Message, System.Object[])
    0000000005fac140 00007ff8cd264492 System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(System.ServiceModel.Dispatcher.ProxyRpc ByRef)
    0000000005fac1d0 00007ff8cd263ee6 System.ServiceModel.Channels.ServiceChannel.HandleReply(System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.ServiceModel.Dispatcher.ProxyRpc ByRef)
    0000000005fac350 00007ff8cd2625b8 System.ServiceModel.Channels.ServiceChannel.Call(System.String, Boolean, System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.Object[], System.Object[], System.TimeSpan)
    0000000005fac590 00007ff8cd261e9c System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage, System.ServiceModel.Dispatcher.ProxyOperationRuntime)
    0000000005fac600 00007ff8cd261313 System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage)
    0000000005fac6d0 00007ff8ed2b7464 System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)
    0000000005facbd8 00007ff8ee517292 [TPMethodFrame: 0000000005facbd8] XAMCommon.ServiceReference.IXAMRTDB.GetValuesID(XAMCommon.ServiceReference.CommunicationValueItemRequest[], Int32, System.String)
    0000000005facc40 00007ff88f6de53a XAMCommon.ServiceReference.XAMRTDBClient.GetValuesID(XAMCommon.ServiceReference.CommunicationValueItemRequest[], Int32, System.String) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\Service References\ServiceReference\Reference.cs @ 13606]
    0000000005facc90 00007ff88f6dbd12 XAMCommunication.XAMPLCCom.ProcessOnlineValues(System.Collections.Generic.List`1) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\XAMCom.cs @ 3052]
    0000000005fad3f0 00007ff88f6d86ec XAMCommunication.XAMPLCCom.timer_Tick(System.Object, System.EventArgs) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\XAMCom.cs @ 2094]
    0000000005fadaf0 00007ff88f3717bd XAMCommunication.XAMPLCCom.BackgroundWorkerLoopA(System.Object, System.ComponentModel.DoWorkEventArgs) [D:\XAMControlE2\Dev\Dev2-MES-VASW3\X4Src\XAMCommon\XAMCom.cs @ 1432]
    0000000005fadf70 00007ff8ec43e04f System.ComponentModel.BackgroundWorker.WorkerThreadStart(System.Object)
    0000000005fae3c8 00007ff8ee50a7f3 [HelperMethodFrame_PROTECTOBJ: 0000000005fae3c8] System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr, System.Object[], System.Object, System.Object[] ByRef)
    0000000005fae7a0 00007ff8edd186e7 System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessageSink)
    0000000005fae880 00007ff8ed3139a5 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    0000000005fae9e0 00007ff8ed313719 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    0000000005faea10 00007ff8ed34216f System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
    0000000005faea60 00007ff8ed34136a System.Threading.ThreadPoolWorkQueue.Dispatch()
    0000000005faefc8 00007ff8ee50a7f3 [DebuggerU2MCatchHandlerFrame: 0000000005faefc8] 
    

    The lock info:

    !syncblk
    Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
       55 000000000679fb78          176         0 0000000000000000     none    00000000c0294690 System.Object
    -----------------------------
    Total           250
    CCW             1
    RCW             5
    ComClassFactory 0
    Free            1

    My be some one can help us.

    Thanks

    Monday, May 2, 2016 2:02 PM

Answers