locked
Soapfaults are no longer caught RRS feed

  • Question

  • Hi,

    Since a couple of days I'm running into a strange problem with all my projects that use webservices.
    Everytime when a soap exception occurs the message is no longer received by my orchestration.

    The transmitted message looks like this in the HAT:

    Microsoft.BizTalk.DefaultPipelines.PassThruTransmit Pipeline Started 22-2-2012 15:09:48,649 -1061153248

    The orchestration stays Started. The message flow view of the message above shows a Transmission Failure with exit code 0xC0C01620.

    For the record this all worked like a charm for all of my BizTalk projects but now fails for all of my BizTalk projects. It happens with all kind of soap faults. Even when I misconfigure the webservice URL in the sendport. The soap fault never reaches my orchestration.

    When I simulate the exact situation on a different environment it all works like a charm so I suspect this has something to do with this specific BizTalk environment.

    The question is what is wrong with this environment?? Any tips of how I can pinpoint the error?

    Wednesday, February 22, 2012 2:42 PM

Answers

  • Hi _FJ_

    This problem was eventually traced back to an access denied permission of a msdb stored procedure 'sp_help_jobhistory'

    The BizTalk operationsguide (for BizTalk2009) specifies that the BizTalk User Application group needs execute permission on this procedure. Right after granting this access BizTalk did work as one would expect.

    • Marked as answer by _FJ_ Thursday, March 21, 2013 2:43 PM
    Thursday, March 21, 2013 2:33 PM

All replies

  • Hi

    From your abouve error message it seems there is a transmission failure,Seems BizTalk Box is not able to even connect to the Web Service Box.

    Could you try opening the Web Service URL from The BizTalk Box, to see if there are any access issue from the Box? Do you see more details in BiZtalk event logs? What are they please?

    You could also check the network connectivity from the Box to the web services Box by running the tracert or Ping command.

    HTH

    Naushad

    Wednesday, February 22, 2012 3:45 PM
    Moderator
  • Check what was changed since the last successful work.

    Is it possible the Web-services were changed? Say some modifications in machine.config? Of update of the IIS?

    Could you compare the SOAP Fauld message formats for the successfull times and now? (changes in WS-Address: To, From headers; envelope format; SOAP version)?


    Leonid Ganeline [BizTalk MVP] BizTalkien: Advanced Questions: have fun - test your knowledge


    Wednesday, February 22, 2012 5:47 PM
    Moderator
  • Hi

    From your abouve error message it seems there is a transmission failure,Seems BizTalk Box is not able to even connect to the Web Service Box.

    Could you try opening the Web Service URL from The BizTalk Box, to see if there are any access issue from the Box? Do you see more details in BiZtalk event logs? What are they please?

    You could also check the network connectivity from the Box to the web services Box by running the tracert or Ping command.

    HTH

    Naushad


    In happy flow situations there is no problem at all. So all webservices can be reached. Only in case of soapfaults things go very wrong.
    So, webservice URI's can be opened from the BizTalk box. In the EventLog there is only one error:

    The Messaging Engine encountered an error while suspending one or more messages.

    Nothing wrong with network connections either....

    Wednesday, February 22, 2012 8:14 PM
  • Check what was changed since the last successful work.

    Is it possible the Web-services were changed? Say some modifications in machine.config? Of update of the IIS?

    Could you compare the SOAP Fauld message formats for the successfull times and now? (changes in WS-Address: To, From headers; envelope format; SOAP version)?


    Leonid Ganeline [BizTalk MVP] BizTalkien: Advanced Questions: have fun - test your knowledge


    I have like 10 BizTalk applications running and they all have the same problem now with webservices. (and they all worked a couple of weeks ago) There are like at least 10 different webservices on different machines. So I can't imagine it has something to do with those webservices. Like I said, when I test my projects locally there is no problem at all.

    I checked the soap faults but I don't really see weird things or differences in them. By the way, we only use the BasicHttp adapter, so no WS.

    I also picked one of the soap faults that go wrong and put it in a soapui mock service returning it to BizTalk on my local machine. Again no problem at all. That's another indication I think that there is something weird going on on the server.

    I also rebooted the server several times to no avail.

    Any other suggestions??

    Wednesday, February 22, 2012 8:21 PM
  • Hi

    Thanks for providing more details!

    I am sure there is no issues in your code since your solution is working fine in your dev farm? Could you check the properties of your send port? Like retry options? 
    For me it seems like the BizTalk engine/adapter is not able to handle the SOAP fault and not able to notify the engine about suspending the message or service?

    Could you check what is the routing properties(context) of the message when it gets suspended? 

    Just few links might help or give you insights about exception handling in Orchestrations

    handling soap faults by Sarvana

    One more thing,You could try running Message Box Viewer Tool, You can get it from here!, Please run it and check if you find any critical error/warning.

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com |@naushadalam

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!


    Wednesday, February 22, 2012 8:35 PM
    Moderator
  • Look to the last updates on your BizTalk server. Maybe Windows updates (related to .NET, WCF)?

    Check the Windows Update log.


    Leonid Ganeline [BizTalk MVP] BizTalkien: Advanced Questions: have fun - test your knowledge

    Wednesday, February 22, 2012 11:16 PM
    Moderator
  • Hi

    Could you try creating a new host and assign it to your send port? Just in case if there are issues with host. 

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com |@naushadalam

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!

    Wednesday, February 22, 2012 11:41 PM
    Moderator
  • Hi,

    In case you have a doubt on environmental issues you can run MessageBoxViewer tool for dignosis.

    http://blogs.technet.com/b/jpierauc/archive/2007/12/18/msgboxviewer.aspx

    Couple of other suggestions,

    Is there a possiblity of lack of trust between the two domains. I would suggest to test the service using some test tool from that machine.  

    1) Which environment is this ? In case its not production, Check if its possible to run a SOAPUI test on one of the machine to test the web service response.

    2) If its not possible to run the SOAPUI test, Please enable the WCF trace, take a look at, http://msdn.microsoft.com/en-us/library/ms732023.aspx

    3) One more way to trace your traffic to service would to be to use Fiddler tool, http://fiddler2.com/fiddler2/version.asp 

    I hope this will be helpful.


    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.
    http://shaileshbiztalk.blogspot.com/

    Thursday, February 23, 2012 6:42 AM
  • Hi

    Thanks for providing more details!

    I am sure there is no issues in your code since your solution is working fine in your dev farm? Could you check the properties of your send port? Like retry options? 
    For me it seems like the BizTalk engine/adapter is not able to handle the SOAP fault and not able to notify the engine about suspending the message or service?

    Could you check what is the routing properties(context) of the message when it gets suspended? 

    Just few links might help or give you insights about exception handling in Orchestrations

    handling soap faults by Sarvana

    One more thing,You could try running Message Box Viewer Tool, You can get it from here!, Please run it and check if you find any critical error/warning.

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com |@naushadalam

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!


    Well the soappfault message does not get suspended. I cannot even see it in BizTalk yet. When I use a network sniffer (Wireshark) I can see that a soap fault is returned. Nothing weird about the specific soap fault by the way. The request message stays on In process / Active, but it looks like the soap fault is never received. In the HAT I can see that the request message gets the servicecode: -1061153248. The orchestration remains dehydrated, not suspended.
    So I can't see any context properties of the soap fault since - it looks like - it's never entering BizTalk.
    Retry settings are 0 retries, failed message routing is enabled. I use it to get rid of suspended messages.

    I will try the messagebox viewer and report later. Thanks.

    Friday, February 24, 2012 3:46 PM
  • Hi,

    On the send port there is a property "Route Fault Messages" in messages tab. Did you try enabling this property.


    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.
    http://shaileshbiztalk.blogspot.com/

    Friday, February 24, 2012 4:03 PM
  • Fault is not suspended? Strange. Kind of lost correlation with port for Fault. Can you make sure the headers for the old Fault messages and new Faults are similar?

    Is it possible the Web service changed something in this Fauld related part?

    If you have WS logs, check them for Faults. Use SoapUi to test Fault and check the WCF log for all Fault nuances.

    Seems it is something in Web service, that is cause of your issue.


    Leonid Ganeline [BizTalk MVP] BizTalkien: Advanced Questions: have fun - test your knowledge

    Saturday, February 25, 2012 6:47 PM
    Moderator
  • Hi,

    You need to generate schemas again using WSDL, looks something got change in the webservice. Do they have multiple endpoints like http, net.tcp which returns different faults like SOAP_1.1 AND SOAP_1.2. I would suggest generate schemas to see the difference.

    Thanks

    Monday, February 27, 2012 8:52 PM
  • Hi

    Could you try creating a new host and assign it to your send port? Just in case if there are issues with host. 

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com |@naushadalam

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!

    Tried it, but it didn't solve the issue....thanks anyway.

    Tuesday, February 28, 2012 7:50 AM
  • Hi,

    On the send port there is a property "Route Fault Messages" in messages tab. Did you try enabling this property.


    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.
    http://shaileshbiztalk.blogspot.com/

    Propagate fault message is disabled. We don't use typed soap faults.

    Tuesday, February 28, 2012 7:52 AM
  • Fault is not suspended? Strange. Kind of lost correlation with port for Fault. Can you make sure the headers for the old Fault messages and new Faults are similar?

    Is it possible the Web service changed something in this Fauld related part?

    If you have WS logs, check them for Faults. Use SoapUi to test Fault and check the WCF log for all Fault nuances.

    Seems it is something in Web service, that is cause of your issue.


    Leonid Ganeline [BizTalk MVP] BizTalkien: Advanced Questions: have fun - test your knowledge

    The problem is we suddenly have this on all applications. So that would mean that all related webservices are changed. That does not seem logical to me. 
    And even if they are changed; those (simple untyped 1.1) soap faults should always enter the orchestration.

    Tuesday, February 28, 2012 7:55 AM
  • Hi,

    You need to generate schemas again using WSDL, looks something got change in the webservice. Do they have multiple endpoints like http, net.tcp which returns different faults like SOAP_1.1 AND SOAP_1.2. I would suggest generate schemas to see the difference.

    Thanks

    That was my first guess as well (and I also did that), but the issue occurs in all applications, not just for one webservice.

    In a new project that I'm working on on the troubled environment the soap fault sometimes enters BizTalk and sometimes it does not. Can't really find a pattern...

    To me it looks like a corrupted environment, corrupted WCF settings, but I don't know where to look.

    Tuesday, February 28, 2012 7:58 AM
  • Hi,

    Check to see if you can enable logging to get a pattern defined. Look at this article for more info.


    Regards,
    Bali
    MCTS: BizTalk Server 2010,BizTalk Server 2006 and WCF
    My Blog:dpsbali-biztalkweblog
    -----------------------------------------------------
    Mark As Answer or Vote As Helpful if this helps.

    Tuesday, February 28, 2012 8:03 AM
  • Hi _FJ_

    This problem was eventually traced back to an access denied permission of a msdb stored procedure 'sp_help_jobhistory'

    The BizTalk operationsguide (for BizTalk2009) specifies that the BizTalk User Application group needs execute permission on this procedure. Right after granting this access BizTalk did work as one would expect.

    • Marked as answer by _FJ_ Thursday, March 21, 2013 2:43 PM
    Thursday, March 21, 2013 2:33 PM