Biztalk 2006 R2 crashing with Dr Watson Error, caused by unhandled WCF exception in IReplyChannel RRS feed

  • Question

  • Hi

    We are running Biztalk 2006 R2 on Windows 2003 Server - 64 bit.

    We use the WCF-Custom adapter in Biztalk to receive messages from a Custom WCF Channel, using the Request/Response MEX Pattern. This reply channel makes use of a TcpListener to receive packets from clients that use a proprietary protocol for communication.

    About once a day, Biztalk crashes with the following Dr Watson error:

    Event Type: Error
    Event Source: BizTalk DW Reporting
    Event Category: None
    Event ID: 1000
    Date:  10/26/2009
    Time:  4:17:17 PM
    User:  N/A
    Computer: SERVERNAME
    Faulting application btsntsvc.exe, version 3.6.1404.0, stamp 4674b0a4, faulting module kernel32.dll, version 5.2.3790.4480, stamp 49c51f06, debug? 0, fault address 0x0002237e.

    After analyzing the crashdump, using the Windows Debugger, and viewing various log files, I came across the following exception:

    System.ServiceModel.Diagnostics.ExceptionUtility+InternalException: An error occurred within Windows Communication Foundation. Applications should not attempt to handle this error.
       at System.ServiceModel.Channels.LifetimeManager.DecrementBusyCount()
       at System.ServiceModel.Dispatcher.ChannelHandler.HandleReceiveComplete(RequestContext context)
       at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
       at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
       at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
       at XXXXX.SynapseReplyChannel.OnTcpClientReadComplete(IAsyncResult result)

    SynapseReplyChannel implements System.ServiceModel.Channels.IReplyChannel.

    The "OnTcpClientReadComplete" method is a callback that is called when the NetworkStream.BeginRead operation completes. This method checks the buffer to determine whether more asyncronous "NetworkStream.BeginRead" operations need to be made to complete the receiving of the packet, as per the protocol. When the entire packet has been read, the callback supplied to us by "WCF" is called, which is supposed to invoke the "EndTryReceiveRequest" method (from the IReplyChannel interface), which supplies a RequestContext object to WCF. This works almost all of the time, except for those occasions, about once a day, when calling the WCF supplied callback results in the above exception, which causes Biztalk to fault with a Dr Watson error, requiring a restart of the Biztalk process.

    So basically, it appears that unhandled exceptions thrown from within a callback, in the context of an asyncronous operation, within an implementation of the IReplyChannel, cause Biztalk(When using the Custom-WCF adapter) to crash with a Dr Watson error, as per the above. I have tried, purposefully, throwing a custom exception from within my callback and Biztalk crashes with this Dr Watson error. Either I'm doing something wrong or not understanding something fully(which is more likely), or could there be a bug somewhere in WCF/Biztalk? Any help/guidance would be greatly appreciated.

    The WCF exception above, specifically states that I should not handle the error, so I'm unsure as to what to do. I definitely think that the callback has to be called successfully, otherwise a RequestContext object will never be supplied to WCF and the message will be lost but then calling it sometimes results in this error which causes Biztalk to crash

    Another interesting thing to mention is that just before the crash happens, there is a significant jump in CPU usuage. Most of the time the CPU hovers around 3-8%. But just before a crash it starts spiking around 20%-40%.

    Anyway(after a lot of searching), I'm kind of at a dead end here and would really appreciate some help.

    Many thanks

    Tuesday, October 27, 2009 8:15 AM


  • Since you got a crash you should report this to MS Support. You are usually not charged if there is a bug that you are reporting. They will probably need to create a patch for this issue.

    If this answers your question, please use the "Answer" button to say so | Ben Cline
    • Marked as answer by marklinley Wednesday, October 28, 2009 6:38 AM
    Tuesday, October 27, 2009 5:25 PM