none
Need Help with High Performance solution for a processing a BIG EDI file RRS feed

  • Question

  • I need to process some 277U EDI files which are >150MB.

    The Application is simple .. all i need to do is split the file into individual transactions get some required fields from each of them and make a SQL stored proc and pass those value and insert them into SQL table.

    The issue i see is when i use EDIReceive Pipeline it would create about 150K transactions and i am afraid that these messages would flood the message Box (would occupy about 2GB space).

    Would it be a good design to make all processing on Pipelines and avoid the messages from going into BizTalk Map?

    Is there a way i can use Orchestration and Map and still achieve performance?

    Thanks in Advance for your help!


    • Edited by raghus4228 Friday, June 26, 2015 7:08 AM
    Friday, June 26, 2015 6:44 AM

Answers

  • Hi Raghu,

    I think your approach is correct.

    As the incoming file is huge we cant avoid the pumping up the files to message box.

    I give one suggestion we have third party tools available to split the incoming EDI file as per configuration.

    Using this you can divide the file into desired chunks and then process each individual chunk in BizTalk.

    https://www.tallan.com/products/t-connect/edi-file-splitter/

    Please mark it as Answer if it answered your question.

    Thanks &Regards,

    Ammu4Biz

    • Marked as answer by Angie Xu Monday, July 6, 2015 3:32 AM
    Friday, June 26, 2015 7:28 AM
  • If you run this and the r/t processes in separate hosts, that will take care of most issues.

    If the r/t processes are still impacted, due to resource contention in the Windows instance, you can dial down the thread counts for this.  That will cause the 277's to process slower, but that's probably not a big deal.

    Keep in mind, there is no 'one way' to do this, you will have to test several configurations/app patterns to find the best one for your environment.

    • Marked as answer by Angie Xu Monday, July 6, 2015 3:32 AM
    Friday, June 26, 2015 8:11 PM
    Moderator

All replies

  • Hi Raghu,

    I think your approach is correct.

    As the incoming file is huge we cant avoid the pumping up the files to message box.

    I give one suggestion we have third party tools available to split the incoming EDI file as per configuration.

    Using this you can divide the file into desired chunks and then process each individual chunk in BizTalk.

    https://www.tallan.com/products/t-connect/edi-file-splitter/

    Please mark it as Answer if it answered your question.

    Thanks &Regards,

    Ammu4Biz

    • Marked as answer by Angie Xu Monday, July 6, 2015 3:32 AM
    Friday, June 26, 2015 7:28 AM
  • First, define 'performance'.  The 277U is not likely does not have any significant SLA attached to it.

    Second, don't worry about 'performance' unless you have an actual, repeatable and measurable problem.  150K messages is no problem for BizTalk and no problem for the MessageBox.  2GB is barely noticeable on disk.

    However, you will need to accommodate this in your BizTalk setup so you create at least 3 Hosts, one each for Receive, Orchestrations and Send.  This will avoid most Throttling situations.

    Also, in situation like this, the performance of the target SP is also critical, especially with locking and deadlocks.


    Friday, June 26, 2015 11:39 AM
    Moderator
  • Hi john,

    I want to understand few things over here.

    If  message is huge while it is receiving at receive pipeline stages the complete message gets loaded into memory(As a stream) 

    does it create any problem at memory

    Just want to understand how the streaming work here.

    Thanks& Regadrs,

    Ammu4Biz

    Friday, June 26, 2015 1:38 PM
  • Internally, the engine uses a VirtualStream somewhere down the stack which caches to disk for large messages.

    However, any 277 will likely not hit that threshold.

    Also, the composition of the files is a factor.  It's generally better to have more batches with fewer transactions.

    Any memory issue is relative to the hardware.

    Friday, June 26, 2015 2:51 PM
    Moderator
  • In an ideal BizTalk Environment 2GB is pretty small and should be handled pretty easily. But my problem is we have some real time transactions running on the same machine so we cannot afford to have 277 affecting the processing of the Real time transactions even by minimal.

    I am just looking for small tweaks to my application to enhance performace

     I am thinking of(doing all the processing in a Pipeline):

    1) after i split the file on EDI Recieve introduce a delay after each transaction so that they would not be flooded

    2) Call the Map from within the Pipeline

    3) Pass the Values from within the Pipeline to the SQL SP(the SQL SP will just insert record to DB)

    Can you think of what issues that can come up if i make the application it this way?

    Thanks,


    • Edited by raghus4228 Friday, June 26, 2015 7:01 PM
    Friday, June 26, 2015 6:49 PM
  • In an ideal BizTalk Environment 2GB is pretty small and should be handled pretty easily. But my problem is we have some real time transactions running on the same machine so we cannot afford to have 277 affecting the processing of the Real time transactions even by minimal.

    I am just looking for small tweaks to my application to enhance performace

     I am thinking of(doing all the processing in the pipeline):

    1) after i split the file on EDI Recieve introduce a delay after each transaction so that they would not be flooded

    2) Call the Map from within the Pipeline

    3) Pass the Values from within the Pipeline to the SQL SP(the SQL SP will just insert record to DB)

    Can you think of what issues that can come up if i make the application it this way?

    Thanks,


    • Edited by raghus4228 Friday, June 26, 2015 7:02 PM
    Friday, June 26, 2015 6:49 PM
  • If you run this and the r/t processes in separate hosts, that will take care of most issues.

    If the r/t processes are still impacted, due to resource contention in the Windows instance, you can dial down the thread counts for this.  That will cause the 277's to process slower, but that's probably not a big deal.

    Keep in mind, there is no 'one way' to do this, you will have to test several configurations/app patterns to find the best one for your environment.

    • Marked as answer by Angie Xu Monday, July 6, 2015 3:32 AM
    Friday, June 26, 2015 8:11 PM
    Moderator