none
Different techniques of debatching in orchestration RRS feed

  • Question

  • What are the different techniques of debatching in orchestration? Is calling receive pipeline in orchestration the preferred solution?
    Friday, August 12, 2016 2:48 AM

Answers

  • Hello,

    Debatching in Orchestration can be done in two main ways:

    • By calling Receive pipeline in orchestration
    • By using some code.

    When I mean by some code, this can be

    • Using XPath
    • Nodelist in Atomic scope
    • Or pass the debatching logic to external .NET helper.

    For the different ways and the performance hits you can go through this link.

    There is great post written in below blog which shows performace benefit with Pipeline debatching

    https://connectedpawns.wordpress.com/2010/10/12/de-batching-within-an-orchestration-using-xpath-or-calling-a-pipeline/

    Microsoft has seen this requirement and seen people use various methods to debatch. So they came up with Pipeline Manager so the pipeline can be called in Orchestration to debatch, which is much better in terms of performance. But you lose the “Recoverable Interchange” option when you call Receive pipeline in orchestration.

    So what I do is chose the option based on quality of the interchange/batch I would receive. If interchange/batch is always going to be perfect and if I would not be facing any issues in debatch or format of the interchange, then I would prefer to go with calling Receive pipeline in orchestration. If message quality would vary and there are possibilities of many messages with in the message to fail during debatching i.e. if I need “Recoverable Interchange” and also there is need to debatch the message in orchestration, then I would use custom code to the debatch and handle the failed debatched messages internally.Its sometime better to handle the known devil/failure within the code rather than leaving to operational support to manage.

    Refer the discussion here: Call a receive pipeline in orchetsration for debatching

    How to debatch by calling Receive pipeline in orchestration?

    Refer: http://geekswithblogs.net/sthomas/archive/2005/06/16/44023.aspx

    http://tech-findings.blogspot.in/2013/07/debatchingsplitting-xml-message-in.html

       

    Basically you have to use the Expression Shape contains the following line of code:

    InputPipeline = Microsoft.XLANGs.Pipeline.
    XLANGPipelineManager.ExecuteReceivePipeline
    (typeof(CallReceivePipeline.ReceivePipeline),msgFullMessage);

    With InputPipeline defined as an Orchestration Variable of type

    Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages

    GetEachMessage will loop the collection using MoveNext like this:

    InputPipeline.MoveNext()

    Finally, the single messages are assigned inside the Message Assignment shape like this:

    msgSingle = new System.Xml.XmlDocument(); 
    InputPipeline.GetCurrent(msgSingle);



    Rachit Sikroria (Microsoft Azure MVP)

    Friday, August 12, 2016 2:49 AM
    Moderator
  • Why are you asking?  Do you have a specific scenario in mind?

    Between XPath and Pipeline, I would always to Pipeline unless there was a definitive reason that would not work.  Then it would be XPath.

    Also, many scenarios can be covered by a double hop of either parsing/debatching or debatching/debatching in separate Pipelines.  You would have to have a very very edge case to warrant a helper class.

    • Marked as answer by AditiBiz Friday, August 12, 2016 9:06 AM
    Friday, August 12, 2016 3:55 AM
    Moderator
  • Hello,

    Consider using call receive pipeline in Orchestration to debatch the messages, so you can easily loop through the detached messages in Orchestration. This approach is also better in terms of performance as you will create only one instance of orchestration for the each batch. i.e if your batch has say 100 messages in it. And if you debatch it in receive port, you would create 100 instances of orchestration which you will use to loop thru the each debatch message. By debatching the message in orchestration by calling the receive pipeline, you will have only one instance of orchestration per batch.


    Rachit Sikroria (Microsoft Azure MVP)

    • Marked as answer by AditiBiz Friday, August 12, 2016 9:06 AM
    Friday, August 12, 2016 4:37 AM
    Moderator

All replies

  • Hello,

    Debatching in Orchestration can be done in two main ways:

    • By calling Receive pipeline in orchestration
    • By using some code.

    When I mean by some code, this can be

    • Using XPath
    • Nodelist in Atomic scope
    • Or pass the debatching logic to external .NET helper.

    For the different ways and the performance hits you can go through this link.

    There is great post written in below blog which shows performace benefit with Pipeline debatching

    https://connectedpawns.wordpress.com/2010/10/12/de-batching-within-an-orchestration-using-xpath-or-calling-a-pipeline/

    Microsoft has seen this requirement and seen people use various methods to debatch. So they came up with Pipeline Manager so the pipeline can be called in Orchestration to debatch, which is much better in terms of performance. But you lose the “Recoverable Interchange” option when you call Receive pipeline in orchestration.

    So what I do is chose the option based on quality of the interchange/batch I would receive. If interchange/batch is always going to be perfect and if I would not be facing any issues in debatch or format of the interchange, then I would prefer to go with calling Receive pipeline in orchestration. If message quality would vary and there are possibilities of many messages with in the message to fail during debatching i.e. if I need “Recoverable Interchange” and also there is need to debatch the message in orchestration, then I would use custom code to the debatch and handle the failed debatched messages internally.Its sometime better to handle the known devil/failure within the code rather than leaving to operational support to manage.

    Refer the discussion here: Call a receive pipeline in orchetsration for debatching

    How to debatch by calling Receive pipeline in orchestration?

    Refer: http://geekswithblogs.net/sthomas/archive/2005/06/16/44023.aspx

    http://tech-findings.blogspot.in/2013/07/debatchingsplitting-xml-message-in.html

       

    Basically you have to use the Expression Shape contains the following line of code:

    InputPipeline = Microsoft.XLANGs.Pipeline.
    XLANGPipelineManager.ExecuteReceivePipeline
    (typeof(CallReceivePipeline.ReceivePipeline),msgFullMessage);

    With InputPipeline defined as an Orchestration Variable of type

    Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages

    GetEachMessage will loop the collection using MoveNext like this:

    InputPipeline.MoveNext()

    Finally, the single messages are assigned inside the Message Assignment shape like this:

    msgSingle = new System.Xml.XmlDocument(); 
    InputPipeline.GetCurrent(msgSingle);



    Rachit Sikroria (Microsoft Azure MVP)

    Friday, August 12, 2016 2:49 AM
    Moderator
  • Hi Aditi

    It depends on the size of the message that you are debatching and number of records in the batch.

    If the message is big in size (upwards of a few MB), pipeline debatching is preferred as it avoids loading entire messages in memory due to streamed processing and also it causes a single persistence point for the whole process. XPATH debatching induces a persistence point for each time you invoke XPATH in a loop.

    For files smaller in size, you can use any approach that you are comfortable with and the difference will barely be noticeable.

    Refer below link for performance comparison between using pipeline in orchestration and using XPATH for debatching-

    https://connectedpawns.wordpress.com/2010/10/12/de-batching-within-an-orchestration-using-xpath-or-calling-a-pipeline/

    You will find good references on the possible approaches below-

    http://social.technet.microsoft.com/wiki/contents/articles/31367.biztalk-de-batching-options.aspx

    https://jeremyronk.wordpress.com/2011/10/03/how-to-debatch-into-an-orchestration-with-a-pipeline/

    http://blogs.biztalk360.com/message-debatching-inside-biztalk-orchestration-with-targetnamespace/

    https://social.msdn.microsoft.com/Forums/en-US/fed3a06c-4795-4f03-a94e-31cc6c7e8d3b/debatching-a-message-in-an-orchestration?forum=biztalkgeneral

    http://blog.eliasen.dk/PermaLink,guid,6c7ac8ec-3f3e-49e4-a15a-76c736d30654.aspx


    Thanks Arindam


    Friday, August 12, 2016 3:02 AM
    Moderator
  • Why are you asking?  Do you have a specific scenario in mind?

    Between XPath and Pipeline, I would always to Pipeline unless there was a definitive reason that would not work.  Then it would be XPath.

    Also, many scenarios can be covered by a double hop of either parsing/debatching or debatching/debatching in separate Pipelines.  You would have to have a very very edge case to warrant a helper class.

    • Marked as answer by AditiBiz Friday, August 12, 2016 9:06 AM
    Friday, August 12, 2016 3:55 AM
    Moderator
  • Hi,

    It gets already debatched at rcv pipeline in first place. What is the situation which is causing this interesting requirement of debatching within pipeline . Aggregation is generally what you do in orchestration by calling pipeline.


    Regards Pushpendra K Singh

    Friday, August 12, 2016 4:25 AM
  • Hello,

    Consider using call receive pipeline in Orchestration to debatch the messages, so you can easily loop through the detached messages in Orchestration. This approach is also better in terms of performance as you will create only one instance of orchestration for the each batch. i.e if your batch has say 100 messages in it. And if you debatch it in receive port, you would create 100 instances of orchestration which you will use to loop thru the each debatch message. By debatching the message in orchestration by calling the receive pipeline, you will have only one instance of orchestration per batch.


    Rachit Sikroria (Microsoft Azure MVP)

    • Marked as answer by AditiBiz Friday, August 12, 2016 9:06 AM
    Friday, August 12, 2016 4:37 AM
    Moderator