none
how to processing files one by one in Orchestation RRS feed

  • Question

  • HI All,

    How to process files (In File Adapter) one after anotehr in BizTalk Orchestration.

    My requirement is , if i place two files(xml files) in the receive folder, biztalk should process one after anohter, that means biztalk orchestration should process one file and completes the full orchestation and  then process the another.

    why because  if i process one xml file, i am doing parsing the xml and inserting the xml data into stagging tables from stagging tables updating into master DB. if a process two xml files simulatneoulsy, both the files data will be inserting into same stagging tables. for that i want to process one after another. because each xml file consists of one country information. if i process one country xml file i need to update into that country destination db only.

    can anybody know to solve this problem.

    Regards,

    Syam

    Monday, September 19, 2011 8:03 AM

Answers

  • Ordered delivery parameter is only in the Send Ports!

    No such things in the Receive Ports.

    The ordered delivery can be forced only to the MSMQ transport adapter!!! So we can set the Ordered Delivery only for the MSMQ Receive Location parameter (it can be on the RL not RP layer!).

    Make sure you understand the FILE protocol does not support order [sequence]! It can emulate sequence. For example, if we assume the "Date Modified" file parameter works as the sequencer. But it is only artificial convention!

    ...

    ....

    would try to do a small sample using the ordered delivery with File Adapter and post the results.

    ...

    Say we have two files in the folder. File adapter read both of them. How do we know which file is first and which is the second?

    In MSMQ the firts message is on the top of queue, the last one is on the tail. MSMQ has notion of order.

    Files do not have such parameters as first of last. No such thing in the File protocole as first and last. The same is for the records in the SQL table.

    We could emulate order if all parts agree that, for example the order means the "[ascending] order in the list of the file names. Still bad solution, because, what we should do with new file that written to the folder while adapter read other files?


    Leonid Ganeline [BizTalk MVP] BizTalk: Sample: Context Routing and Throttling with orchestration
    Monday, September 19, 2011 10:33 PM
    Moderator

All replies

  • Hi,

    Use a Sequential Convoy pattern.

    http://msdn.microsoft.com/en-us/library/aa561843(v=bts.70).aspx

    Best regards

    Rasmus Jaeger

     

    Monday, September 19, 2011 8:30 AM
  • You will need to do this using a Sequential Convoy pattern (link that Ramus shared) and also you will have to set the Ordered Delivery property of the orchestration's receive port to True.



    Regards,
    Bali
    MCTS: BizTalk Server 2010,BizTalk Server 2006 and WCF
    Blog: http://dpsbali-biztalkweblog.blogspot.com
    -----------------------------------------------------
    Mark As Answer or Vote As Helpful if this helps.
    Monday, September 19, 2011 8:34 AM
  • Hi Bali/Rasmus,

    If i place more than one files, all my xml files are independent to each other. there is no related to each message.

    steps in my orchestration

    1. place the file in the file Adapter.

    2. parsing the xml file using .net code and inserting to stagging tables.

    3. using Some Stored procedures all the stagging tables data migrated to temp tables.

    4. from the temp tables i will check whether the data is new/updaste/delete and inserting to Master DB (final DB)

    5. deleting all the data in the stagging table and temporary tables , because for the next file processing.

    -------------------------

    After completing all the above steps..next xml input file should process

    Regards,

    Syam

     

    Monday, September 19, 2011 11:27 AM
  • Hi SyamMaddi,

                        You can  use sequential convoys,create correlation set on BTS.ReceiveLocation property as the Receive location is Same for all the Messages, and Set order Delivery to true.This will Solve.

     

     

    Please click on Is Answer if it Solved your problem

     

    Monday, September 19, 2011 11:51 AM
  • You will need to do this using a Sequential Convoy pattern (link that Ramus shared) and also you will have to set the Ordered Delivery property of the orchestration's receive port to True.



    Regards,
    Bali
    MCTS: BizTalk Server 2010,BizTalk Server 2006 and WCF
    Blog: http://dpsbali-biztalkweblog.blogspot.com
    -----------------------------------------------------
    Mark As Answer or Vote As Helpful if this helps.


    Ordered delivery parameter is only in the Send Ports!

    No such things in the Receive Ports.

    The ordered delivery can be forced only to the MSMQ transport receive  adapter!!! So we can set the Ordered Delivery only for the MSMQ Receive Location parameter (it can be on the RL not RP layer!).

    Make sure you understand the FILE protocol does not support order [sequence]! It can emulate sequence. For example, if we assume the "Date Modified" file parameter works as the sequencer. But it is only artificial convention!


    Leonid Ganeline [BizTalk MVP] BizTalk: Sample: Context Routing and Throttling with orchestration

    Monday, September 19, 2011 4:48 PM
    Moderator
  • The Sequential convoy will not help you.

    You have to implement the dublicate processing into the Destination Application. In your case it is the SQL. So implement simple  upsert logic in the stored procedure which write data to table.

    sp should check the "if i process one country xml file i need to update into that country destination db only." and insert only for a new ... and update for the old.

    Implemention of the same kind of logic in the transmittion layer would involve creating the storage in this layer and repeat the above logic there.  Too much burden.


    Leonid Ganeline [BizTalk MVP] BizTalk: Sample: Context Routing and Throttling with orchestration
    Monday, September 19, 2011 4:53 PM
    Moderator
  • Ordered delivery parameter is only in the Send Ports!

    No such things in the Receive Ports.

    The ordered delivery can be forced only to the MSMQ transport adapter!!! So we can set the Ordered Delivery only for the MSMQ Receive Location parameter (it can be on the RL not RP layer!).

    Make sure you understand the FILE protocol does not support order [sequence]! It can emulate sequence. For example, if we assume the "Date Modified" file parameter works as the sequencer. But it is only artificial convention!


    Leonid Ganeline [BizTalk MVP] BizTalk: Sample: Context Routing and Throttling with orchestration

    Well I thought ordered delivery should work in a receive shape of an orchestration. A bit more about this in the article on msdn: Ordered Delivery of Messages.

    The protocols underlying certain transports, such as FILE and HTTP, are not consistent with the notion of ordered delivery. However, even with such transports, if the port bound to the transport is marked for ordered delivery, then BizTalk Server enforces ordered delivery by ensuring that the transport does not get the next outbound message until the current one has been successfully sent. To achieve this, BizTalk Server passes each message to the transport's adapter in a single batch and waits until the adapter has successfully deleted the message from the message box before delivering the next message, in another batch, to the adapter.

    I think the requirement in this question can be achieved, if the orchestration is made an atomic transaction scope and follows ordered delivery pattern. 

    If time permits would try to do a small sample using the ordered delivery with File Adapter and post the results.


    Regards,
    Bali
    MCTS: BizTalk Server 2010,BizTalk Server 2006 and WCF
    Blog: http://dpsbali-biztalkweblog.blogspot.com
    -----------------------------------------------------
    Mark As Answer or Vote As Helpful if this helps.
    • Edited by DPS Bali Monday, September 19, 2011 6:17 PM
    Monday, September 19, 2011 6:17 PM
  • Ordered delivery parameter is only in the Send Ports!

    No such things in the Receive Ports.

    The ordered delivery can be forced only to the MSMQ transport adapter!!! So we can set the Ordered Delivery only for the MSMQ Receive Location parameter (it can be on the RL not RP layer!).

    Make sure you understand the FILE protocol does not support order [sequence]! It can emulate sequence. For example, if we assume the "Date Modified" file parameter works as the sequencer. But it is only artificial convention!

    ...

    ....

    would try to do a small sample using the ordered delivery with File Adapter and post the results.

    ...

    Say we have two files in the folder. File adapter read both of them. How do we know which file is first and which is the second?

    In MSMQ the firts message is on the top of queue, the last one is on the tail. MSMQ has notion of order.

    Files do not have such parameters as first of last. No such thing in the File protocole as first and last. The same is for the records in the SQL table.

    We could emulate order if all parts agree that, for example the order means the "[ascending] order in the list of the file names. Still bad solution, because, what we should do with new file that written to the folder while adapter read other files?


    Leonid Ganeline [BizTalk MVP] BizTalk: Sample: Context Routing and Throttling with orchestration
    Monday, September 19, 2011 10:33 PM
    Moderator
  • Hi,

    Leonid post is 100% correct.  The File Adapter does not support ordered delivery, but can emulate it.  However, the proposed solution by Bali, appears to be using the ordered delivery functionality to ensure that each inbound xml file is processed (through orchestration to staging/master db) before processing the next one.  It does not ensure the XML inbound files are processed in a specific order.  Changing the receive location batchsize to 1 from the default of 20 should pick one only one inbound XML file at a time.

    Again the proposed solution would only ensure that the files would be processed independant of each other, not in any specific order.  The OP would have to ensure that the proposed solution meets his needs.

    Thanks,

    William

    Sunday, September 25, 2011 3:30 PM