none
WCF-NetMsmq fails to retrieve messages from local MSMQ RRS feed

  • Question

  • Hello,

    We have a process that uses the WCF-NetMsmq adapter to receive messages from a local MSMQ queue located on the BizTalk server box. Using BTS2006 R2.

    This morning I noticed that hundreds of messages were sitting on the queue and BizTalk was not picking them up. This is the error message that was continuously showing in the Event Log:

     

     

    Event Type:   Warning
    Event Source: BizTalk Server 2006
    Event Category:      BizTalk Server 2006 
    Event ID:     5740
    Date:         3/22/2010
    Time:         4:36:21 AM
    User:         N/A
    Computer:     MTDFOIBIZ01
    Description:
    The adapter "WCF-NetMsmq" raised an error message. Details "System.ObjectDisposedException: Cannot access a disposed object.
    Object name: 'TransactionScope'.
       at System.Transactions.TransactionScope.Complete()
       at System.ServiceModel.Dispatcher.TransactionRpcFacet.ThreadLeave()
       at System.ServiceModel.Dispatcher.TransactionBehavior.ClearCallContext(MessageRpc& rpc)
       at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage7(MessageRpc& rpc)
       at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)".
    

     

     

    Restarting the host instance associated with the WCF-NetMsmq port solved the issue, and all messages got picked up.

    Any idea how to troubleshoot this issue? What could be causing this?

     

    Thanks in advance for the help!

    - Steve

    Monday, March 22, 2010 4:18 PM

Answers

  • Usually the default is to have throttling enabled. If you were using the default host then you should definitely move it to a separate dedicated host if you expect to have a lot of traffic. The default host has to work with so many other BizTalk processing requests - you may not need to customize the throttling on a dedicated host.

    From the earlier reply, I was guessing MSMQ must have gone down so thanks for confirming it did not.

    Thanks,


    If this answers your question, please use the "Answer" button to say so | Ben Cline
    Friday, March 26, 2010 5:07 PM
    Moderator

All replies

  • Hi Steve,

    This is just I theory or thinking. I think timeout occured on recieve location and therefor a handle/connection to MSMQ is lost. If it tries to connect after an interval and it fails. Or another error regarding configuration WCF-NetMsmq could cause the problem. Perhaps process explorer can help you troubleshoot the error. Restarting the host somehow resets everything. Troubleshooting MSMQ information can also be found here.

    HTH,

    Regards,

    Steef-Jan Wiggers
    http://soa-thoughts.blogspot.com
    If this answers your question please mark it accordingly 

     

     


    BizTalk
    Monday, March 22, 2010 9:00 PM
    Moderator
  •  

    Hi Steef-Jan,

    You might be right about a possible timeout issue. I have been looking at the MSDTC tracing logs. At the same exact time BizTalk was having issues accessing MSMQ, the MSDTC logs are also showing some "ABORT_DUE_TO_TRANSACTION_TIMER_EXPIRED" event. These do not appear once the host instance was later restarted. Here is an excerpt of the logs:

     

     

    pid=1196       ;tid=1240       ;time=03/22/2010-04:32:16.851   ;seq=287506     ;eventid=ABORT_DUE_TO_TRANSACTION_TIMER_EXPIRED   ;tx_guid=05dc2635-edf4-4c14-aa87-98baac1bcb78     ;"transaction timeout expired"
    pid=1196       ;tid=1240       ;time=03/22/2010-04:32:16.851   ;seq=287507     ;eventid=TRANSACTION_ABORTING                     ;tx_guid=05dc2635-edf4-4c14-aa87-98baac1bcb78     ;"transaction is aborting"
    pid=1196       ;tid=1240       ;time=03/22/2010-04:32:16.851   ;seq=287508     ;eventid=RM_ISSUED_ABORT                          ;tx_guid=05dc2635-edf4-4c14-aa87-98baac1bcb78     ;"abort request issued to resource manager #1003 for transaction enlistment #1"
    pid=1196       ;tid=39324      ;time=03/22/2010-04:32:16.851   ;seq=287509     ;eventid=RM_ACKNOWLEDGED_ABORT                    ;tx_guid=05dc2635-edf4-4c14-aa87-98baac1bcb78     ;"received acknowledgement of abort request from the resource manager #1003 for transaction enlistment #1"
    pid=1196       ;tid=39324      ;time=03/22/2010-04:32:16.851   ;seq=287510     ;eventid=TRANSACTION_ABORTED                      ;tx_guid=05dc2635-edf4-4c14-aa87-98baac1bcb78     ;"transaction has been aborted"
    pid=1196       ;tid=39324      ;time=03/22/2010-04:21:21.070   ;seq=287511     ;eventid=TRANSACTION_BEGUN                        ;tx_guid=a396d139-9ebc-42a8-a807-a8e3b4b5e89d     ;"transaction got begun, description : '<NULL>'"
    pid=1196       ;tid=39260      ;time=03/22/2010-04:21:21.070   ;seq=287512     ;eventid=RM_ENLISTED_IN_TRANSACTION               ;tx_guid=a396d139-9ebc-42a8-a807-a8e3b4b5e89d     ;"resource manager #1003 enlisted as transaction enlistment #1. RM guid = '54dbda6c-58ff-403f-a500-fdff6ce1909a'"
    pid=1196       ;tid=1240       ;time=03/22/2010-04:32:16.851   ;seq=287513     ;eventid=ABORT_DUE_TO_TRANSACTION_TIMER_EXPIRED   ;tx_guid=a396d139-9ebc-42a8-a807-a8e3b4b5e89d     ;"transaction timeout expired"
    pid=1196       ;tid=1240       ;time=03/22/2010-04:32:16.851   ;seq=287514     ;eventid=TRANSACTION_ABORTING                     ;tx_guid=a396d139-9ebc-42a8-a807-a8e3b4b5e89d     ;"transaction is aborting"
    pid=1196       ;tid=1240       ;time=03/22/2010-04:32:16.851   ;seq=287515     ;eventid=RM_ISSUED_ABORT                          ;tx_guid=a396d139-9ebc-42a8-a807-a8e3b4b5e89d     ;"abort request issued to resource manager #1003 for transaction enlistment #1"
    pid=1196       ;tid=39324      ;time=03/22/2010-04:32:16.851   ;seq=287516     ;eventid=RM_ACKNOWLEDGED_ABORT                    ;tx_guid=a396d139-9ebc-42a8-a807-a8e3b4b5e89d     ;"received acknowledgement of abort request from the resource manager #1003 for transaction enlistment #1"
    pid=1196       ;tid=39324      ;time=03/22/2010-04:32:16.851   ;seq=287517     ;eventid=TRANSACTION_ABORTED                      ;tx_guid=a396d139-9ebc-42a8-a807-a8e3b4b5e89d     ;"transaction has been aborted"
    pid=1196       ;tid=32880      ;time=03/22/2010-04:21:21.070   ;seq=287518     ;eventid=TRANSACTION_BEGUN                        ;tx_guid=4dc4e0b3-a7e8-447f-a0fa-69656bd86fe9     ;"transaction got begun, description : '<NULL>'"
    pid=1196       ;tid=39260      ;time=03/22/2010-04:21:21.070   ;seq=287519     ;eventid=RM_ENLISTED_IN_TRANSACTION               ;tx_guid=4dc4e0b3-a7e8-447f-a0fa-69656bd86fe9     ;"resource manager #1003 enlisted as transaction enlistment #1. RM guid = '54dbda6c-58ff-403f-a500-fdff6ce1909a'"
    pid=1196       ;tid=1240       ;time=03/22/2010-04:32:16.851   ;seq=287520     ;eventid=ABORT_DUE_TO_TRANSACTION_TIMER_EXPIRED   ;tx_guid=4dc4e0b3-a7e8-447f-a0fa-69656bd86fe9     ;"transaction timeout expired"
    

     

    What could be causing this? I am not really sure how Process Explorer could help me troubleshoot an adapter / host instance issue?

    Thanks,

    - Steve

     

    Monday, March 22, 2010 9:40 PM
  • Hi,

    A good resource for DTC tracing is this (http://blogs.msdn.com/distributedservices/archive/2009/02/07/the-hidden-tool-msdtc-transaction-tracing.aspx). If not process explorer, maybe process monitor can help.

    HTH,

    Regards,

    Steef-Jan Wiggers
    http://soa-thoughts.blogspot.com
    If this answers your question please mark it accordingly 


    BizTalk
    Monday, March 22, 2010 9:48 PM
    Moderator
  • First I would ask if your MSMQ queue is transactional? If so, then MSDTC would come into play. Are you sending the message from the local server or a remote server (this is possible even with private queues)? Although remote transactional reads are only possible on W2K8 or Vista. If it is a local queue it seems silly to make it transactional too because the write is on the same box anyway.

    If the MSMQ service went down, the BizTalk port could try a couple times and failing will stop. Your error refers to TransactionScope which means there was an ambient transaction, often because of a distributed transaction (leading to MSDTC). I would configure the WCF-netMSMQ port to not use a transaction so it does not get rolled together with MSDTC. You could also increase the WCF timeouts.

    Thanks,


    If this answers your question, please use the "Answer" button to say so | Ben Cline
    Tuesday, March 23, 2010 5:26 AM
    Moderator
  • Thanks for the replies.

    To answer Ben's questions, yes, the MSMQ queue is transactional and on the local server as well. We opted for a transactional queue as we wanted to ensure that each message is delivered only once, and the sender is notified in case of delivery failures. Any suggestion on where and what to look for exactly, that would explain what is causing this issue on a transactional queue?

    PS. The MSMQ service did not go down, in fact just restarting the BizTalk host instance solved the issue (all the documents got picked up from the local queue within seconds). The MSDTC timeouts are set to 10 minutes, the WCF timeouts are defaulted to 1 minute I believe.

     

    Thanks

    - Steve

    Tuesday, March 23, 2010 5:06 PM
  • Hi Steve,

    You may try test if the problem no longer happenes after clustering the adapter's host instances. Or if regularly restarting the host instance can be a workaround for you?

    A thorough troubleshooting of the problem should involve live debugging scenarios.

    Thanks.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
    Friday, March 26, 2010 10:10 AM
    Moderator
  • We think we might have found the issue, however it seems more like a BizTalk bug. We started having the same exact issue again this morning. Same warning messages in the Event Log. At the same time, the host instance associated to WCF-NetMsmq adapter started throttling due to high memory usage. Instead of restarting the host, we modified the throttling settings for the host so that it would allow for more memory to be used (instead of the default 25%). After applying this modifications, all documents got picked up from the queue instantly (and no throttling was happening as well).

    Can this be a BizTalk bug that occurs in throttling situations and WCF adapters? Is this a known issue?

     

    Also, replying to Wen-Jun: there is only a single host instance querying the local queue. Restarting the instance would be a temporary workaround, but not really a fix which is what we are looking for.

    Friday, March 26, 2010 2:51 PM
  • Usually the default is to have throttling enabled. If you were using the default host then you should definitely move it to a separate dedicated host if you expect to have a lot of traffic. The default host has to work with so many other BizTalk processing requests - you may not need to customize the throttling on a dedicated host.

    From the earlier reply, I was guessing MSMQ must have gone down so thanks for confirming it did not.

    Thanks,


    If this answers your question, please use the "Answer" button to say so | Ben Cline
    Friday, March 26, 2010 5:07 PM
    Moderator