locked
How to Append instances of a file within Orchestration RRS feed

  • Question

  • Hi- I have in receive pipeline message getting debatched and creating 1000-3000 instances created and sent to file folder out. I batch this by using Append as FileCopy mode. So all these files are getting batched into 1.

    Is there a way instead of setting this in Admin console, I can do this Appending within Orchestration itself. If so , how ?


    MBH

    Wednesday, October 12, 2016 4:52 PM

All replies

  • Hi,

    You should stick with the pattern you have currently implemented. Any other aggregation patterns will not work since they mostly involve building up an XML file in memory. When a batch of 1000+ message will get submitted it will tak over 10 minutes to process the batch, which will be way too slow.

    If this approach is working and you are not facing any issue then stick with it rather than going with any other method. When dealing with very large batch any other pattern will be insufficient. 


    Rachit Sikroria (Microsoft Azure MVP)


    Wednesday, October 12, 2016 5:20 PM
    Moderator
  • Hi Rachit, I want to stick with my pattern itself but I am facing an issue.

    When I am Appending the debatched (~5000 xml files) in Send port location which is not properly formed XML.

    This huge XML file is taking 10 mins  appox to be written....Now my other Receive port (RP2) location is waiting to take this file and use Custom pipeline code to make this valid XML.

    So, as it is taking 10 mins to write the complete file, RP2 is picking the 1st minute file and only part of the file is being written. The rest of the instances are directly getting dehydrated for infinity time.

    How can I solve this ?


    MBH

    Wednesday, October 12, 2016 5:41 PM
  • Hello,

    Batching 5000 XMLs will take time and it is pretty normal.

    To prevent the RP2 to pick up the message, on your send port you can change the property of copy mode to create new and then select the use temporary file while writing. This way the destination system will only pick up the .xml files and hence wont pick up incomplete files.

    FILEtransportProperties.png


    Rachit Sikroria (Microsoft Azure MVP)

    Wednesday, October 12, 2016 6:17 PM
    Moderator
  • If I change from Append to Create New, I cannot batch the files. Main purpose I'm using Append is to include all instances in 1 file, if Mention create New it will create 5000 seperate files.

    MBH

    Wednesday, October 12, 2016 6:27 PM