none
The process was terminated due to an unhandled exception. This stops WCF-SQL host instance RRS feed

  • Question

  • Hello, friends, 

    I am experiencing a weird behavior in BizTalk. 

    There are around 25000 messages that BizTalk receives every 15 minutes that gets sent to MSMQ. I also commit these messages to SQL database using WCF-SQL adapter. I have 2 BizTalk application servers, each box has a host instance for WCF-SQL adapter. 

    BizTalk is struggling to commit the messages to the database. It is not throttling at all. It gives the following error in the event log and stops the host instances on the Both app boxes.

    I am using BizTalk server 2013 R2 with no CU's.

     I have started experiencing this issue only in recent times. It has been working fine for last 2 years. Since last year we haven't had any Windows Patches on the BizTalk Boxes as well.

    Please kindly guide me what could be the potential issue here.

    Application: BTSNTSvc64.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.ServiceModel.CommunicationObjectFaultedException

    Server stack trace: 
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.OneWayOperationSendPortRequestChannel`1.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.ServiceChannel.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)
       at System.ServiceModel.ICommunicationObject.BeginClose(System.AsyncCallback, System.Object)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].HandleTransmissionException[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Exception, Microsoft.BizTalk.Adapter.Wcf.Runtime.CallbackState`1<System.__Canon>)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].RequestCallback(System.IAsyncResult)
       at System.Runtime.AsyncResult.Complete(Boolean)

    Exception Info: System.Runtime.CallbackException
       at System.Runtime.AsyncResult.Complete(Boolean)
       at System.ServiceModel.Channels.ServiceChannel+SendAsyncResult.FinishSend(System.IAsyncResult, Boolean)
       at System.ServiceModel.Channels.ServiceChannel+SendAsyncResult.SendCallback(System.IAsyncResult)
       at System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)
       at Microsoft.ServiceModel.Channels.Common.Design.AdapterAsyncResult.SignalFinished(System.Object, System.Exception)
       at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()

    Wednesday, April 11, 2018 11:26 AM

All replies

  • Any other error logs on BizTalk, or on SQL?

    Next step would be to collect a WCF trace, esp. for the WCF-SQL Adapter.


    Thanks Arindam


    Wednesday, April 11, 2018 11:34 AM
    Moderator
  • Hi Arindam, 

    When I have enabled the trace, following error is getting written to the trace file.

    I don't see any other errors on either BizTalk or on SQL box.

    <E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent"><System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system"><EventID>131075</EventID><Type>3</Type><SubType Name="Error">0</SubType><Level>2</Level><TimeCreated SystemTime="2018-04-11T13:38:59.8439975Z" /><Source Name="System.ServiceModel" /><Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" /><Execution ProcessName="BTSNTSvc64" ProcessID="61688" ThreadID="36" /><Channel/><Computer>MyServerName</Computer></System><ApplicationData><TraceData><DataItem><TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error"><TraceIdentifier>http://msdn.microsoft.com/en-ZA/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier><Description>Throwing an exception.</Description><AppDomain>DefaultDomain</AppDomain><Source>Microsoft.ServiceModel.Channels.Common.Channels.AdapterRequestChannel/34916556</Source><Exception><ExceptionType>System.ServiceModel.CommunicationObjectFaultedException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The communication object, Microsoft.ServiceModel.Channels.Common.Channels.AdapterRequestChannel, cannot be used for communication because it is in the Faulted state.</Message><StackTrace>   at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
    
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.OneWayOperationSendPortRequestChannel`1.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.ServiceChannel.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
       at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
       at System.ServiceModel.Channels.ServiceChannelProxy.ExecuteMessage(Object target, IMethodCallMessage methodCall)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeChannel(IMethodCallMessage methodCall)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp;amp; msgData, Int32 type)
       at System.ServiceModel.ICommunicationObject.BeginClose(AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.HandleTransmissionException[ChannelType](Exception ex, CallbackState`1 state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend(IAsyncResult result, Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.SendCallback(IAsyncResult result)
       at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
       at Microsoft.ServiceModel.Channels.Common.Design.AdapterAsyncResult.SignalFinished(Object finishedResult, Exception finishedException)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
    </StackTrace><ExceptionString>System.ServiceModel.CommunicationObjectFaultedException: The communication object, Microsoft.ServiceModel.Channels.Common.Channels.AdapterRequestChannel, cannot be used for communication because it is in the Faulted state.</ExceptionString></Exception></TraceRecord></DataItem></TraceData><System.Diagnostics xmlns="http://schemas.microsoft.com/2004/08/System.Diagnostics"><LogicalOperationStack></LogicalOperationStack><Timestamp>9406485488563</Timestamp></System.Diagnostics></ApplicationData></E2ETraceEvent>




    Wednesday, April 11, 2018 1:44 PM
  • That doesn't look right, did you copy the config section from the shared link?

    Thanks Arindam

    Wednesday, April 11, 2018 1:53 PM
    Moderator
  • yes, I've copied the below section to the BizTalk Config file.

    <system.diagnostics>  
        <sources>  
          <source name ="System.ServiceModel" switchValue="Verbose">  
            <listeners>  
              <add name="xml" />  
            </listeners>  
          </source>  
          <source name ="System.ServiceModel.MessageLogging"   
                  switchValue="Verbose, ActivityTracing">          
            <listeners>  
              <add name="xml" />  
            </listeners>  
          </source>  
          <source name ="System.Runtime.Serialization" switchValue="Verbose">  
            <listeners>  
              <add name="xml" />  
            </listeners>  
          </source>  
       </sources>  
       <sharedListeners>  
          <add name="xml" type="System.Diagnostics.XmlWriterTraceListener"                
               traceOutputOptions="LogicalOperationStack"   
               initializeData="C:\log\WCFTrace.svclog" />  
       </sharedListeners>  
       <trace autoflush="true" />  
      </system.diagnostics>  
      <system.serviceModel>  
        <diagnostics>  
          <messageLogging   
               logEntireMessage="true"   
               logMalformedMessages="false"  
               logMessagesAtServiceLevel="true"   
               logMessagesAtTransportLevel="false"/>  
        </diagnostics>      
      </system.serviceModel>

    I am also getting following error.

    <ApplicationData><TraceData><DataItem><TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>UnhandledException</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>DefaultDomain</AppDomain><Exception><ExceptionType>System.Runtime.CallbackException, System.ServiceModel.Internals, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35</ExceptionType><Message>Async Callback threw an exception.</Message><StackTrace>   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend(IAsyncResult result, Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.SendCallback(IAsyncResult result)
       at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
       at Microsoft.ServiceModel.Channels.Common.Design.AdapterAsyncResult.SignalFinished(Object finishedResult, Exception finishedException)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()</StackTrace><ExceptionString>System.Runtime.CallbackException: Async Callback threw an exception. ---&amp;gt; System.ServiceModel.CommunicationObjectFaultedException: The communication object, Microsoft.ServiceModel.Channels.Common.Channels.AdapterRequestChannel, cannot be used for communication because it is in the Faulted state.
    
    Server stack trace: 
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.OneWayOperationSendPortRequestChannel`1.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.ServiceChannel.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
    
    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp;amp; msgData, Int32 type)
       at System.ServiceModel.ICommunicationObject.BeginClose(AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.HandleTransmissionException[ChannelType](Exception ex, CallbackState`1 state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
       --- End of inner exception stack trace ---
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishSend(IAsyncResult result, Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.SendCallback(IAsyncResult result)
       at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
       at Microsoft.ServiceModel.Channels.Common.Design.AdapterAsyncResult.SignalFinished(Object finishedResult, Exception finishedException)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()</ExceptionString><InnerException><ExceptionType>System.ServiceModel.CommunicationObjectFaultedException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The communication object, Microsoft.ServiceModel.Channels.Common.Channels.AdapterRequestChannel, cannot be used for communication because it is in the Faulted state.</Message><StackTrace>
    Server stack trace: 
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.OneWayOperationSendPortRequestChannel`1.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.ServiceChannel.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
    
    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp;amp; msgData, Int32 type)
       at System.ServiceModel.ICommunicationObject.BeginClose(AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.HandleTransmissionException[ChannelType](Exception ex, CallbackState`1 state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)</StackTrace><ExceptionString>System.ServiceModel.CommunicationObjectFaultedException: The communication object, Microsoft.ServiceModel.Channels.Common.Channels.AdapterRequestChannel, cannot be used for communication because it is in the Faulted state.
    
    Server stack trace: 
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.OneWayOperationSendPortRequestChannel`1.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.ServiceChannel.OnBeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult.InvokeClose()
       at System.ServiceModel.Channels.CommunicationObject.CloseAsyncResult..ctor(CommunicationObject communicationObject, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.CommunicationObject.BeginClose(TimeSpan timeout, AsyncCallback callback, Object state)
    
    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp;amp; msgData, Int32 type)
       at System.ServiceModel.ICommunicationObject.BeginClose(AsyncCallback callback, Object state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.HandleTransmissionException[ChannelType](Exception ex, CallbackState`1 state)
       at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)</ExceptionString></InnerException></Exception></TraceRecord></DataItem></TraceData><System.Diagnostics xmlns="http://schemas.microsoft.com/2004/08/System.Diagnostics"><LogicalOperationStack></LogicalOperationStack><Timestamp>9404274308573</Timestamp></System.Diagnostics></ApplicationData>



    Wednesday, April 11, 2018 1:57 PM
  • Can you also add as per the link-

    <system.diagnostics>  
        <sources>  
          <source name="Microsoft.ServiceModel.Channels" switchValue="Error">  
            <listeners>  
              <add name="xml" />  
            </listeners>  
          </source>  
          <source name="Microsoft.Adapters.Sql" switchValue="Information">  
            <listeners>  
              <add name="xml" />  
            </listeners>  
          </source>  
        </sources>  
        <sharedListeners>  
          <add name="xml" type="System.Diagnostics.XmlWriterTraceListener"   
       traceOutputOptions="LogicalOperationStack"   
              initializeData="C:\log\AdapterTrace.svclog" />  
        </sharedListeners>  
        <trace autoflush="true" />  
      </system.diagnostics> 


    Thanks Arindam

    Wednesday, April 11, 2018 9:22 PM
    Moderator
  • Hi Arindam, 

    I am applying CU8 to our environment. I will update you if I come right with this. It is weird that it is only happing to WCF-SQL adapter, that too on 2 boxes. 

    on the web, I see that people are having a similar type of issues with WCF-SAP adapter, Microsoft has given a patch for this as part of CU's. I am applying CU8 on BizTalk Server 2013 R2. 

    Regards

    Venu

    Thursday, April 12, 2018 12:23 PM