none
Biztalk 2013 - Process Large EDI FILE >2MB RRS feed

  • Question

  • We receive an EDI file >2mb from trading partner. The file is passed through custom edi pipeline which has stages as archive, edi disassemble, policy, translation. We observed when the file goes through the pipeline it takes around 15 mins to process the file completely and it blocks other processes. We get many other fall outs during that time. 

    eg. 

    As2 messages get suspended with error as "The client connection might have been disconnected". 

    I think it is mostly due to the large size of the file.

    Did anyone face this kind of issue for larger files ?  Any solution is appreciated. 

    Monday, March 28, 2016 5:08 PM

Answers

  • Ok, here's what you can try next. I am assuming that you are using the EDIReceive pipeline for your tests.

    Change the highlighted values below to more than 2MB (say 3MB or 3072KB) at BizTalk Group Settings, and run your tests again -

    Thanks

    Arindam

    Tuesday, March 29, 2016 9:58 AM
  • The file has around 5100 transactions. The AS2 errors are not for this file(trading partner) but for other trading partners who are sending data via as2.

    Sorry, this is now a bit confusing.  Exactly what are you asking about?  Slow debatch time or the errors?

    Either way, 5100 transactions is a lot for such a small file but it now makes a bit more sense as to why it's taking so long.  Debatch time is driven mostly by number of output messages, not size.

    You may have to change your pattern to not debatch in the the initial EDI Receive.  Instead, use Preserve Interchange, ACKs and AS2 will still work fine, but you'd have to debatch later in your process.

    Here is a Wiki Article about working with Preserve Interchange: http://social.technet.microsoft.com/wiki/contents/articles/26691.biztalk-server-working-with-preserve-interchange-edi-xml-part-1.aspx

    Tuesday, March 29, 2016 12:13 PM

All replies

  • 2MB?  That's pretty small so it's unlikely any built in component.  How many transactions are in it?

    Sure, if it's coming in over AS2, the connection may very well timeout within 15 minutes, but that's probably only a side effect.

    You need to figure out exactly which step in your process is taking the time.  Can you process the 2MB EDI file with just the EDI Disassembler?  How long does that take.  Again, 2MB is small.

    Monday, March 28, 2016 5:38 PM
  • Hi Amita,

    Thank you for posting on MSDN forum.

    2mb file is not a large file and I suspect it could be because of the threshold value. You can certainly increase or decrease the threshold. 

    You are using custom pipeline, please try to use EDI default pipeline (EDI Disassembler) only and see the result.

    Please have a look into below article written by Tord Nordahl, that might be helpful for you,

    http://social.technet.microsoft.com/wiki/contents/articles/6983.biztalk-server-2010-host-throttling.aspx

    Please have a look into below article for more help.

    http://blogs.objectsharp.com/post/2005/10/23/Processing-a-Large-Flat-File-Message-with-BizTalk-and-the-SqlBulkInsert-Adapter.aspx


    Thanks,

    If my reply is helpful please mark as Answer or vote as Helpful.

    My blog | Twitter | LinkedIn

    This post is my own opinion and does not necessarily reflect the opinion or view of Microsoft, its employees, or other MVPs.

    Monday, March 28, 2016 5:42 PM
  • The file has around 5100 transactions. The AS2 errors are not for this file(trading partner) but for other trading partners who are sending data via as2. 

    When I ran the same file with EDI disassemble only, it took 4 mins to disassemble the file.

    When I ran the file with edi receive pipeline , with map on the send port, it took 7 mins for the whole process. 

    I am still not sure if this change will resolve the blocking issue/timeout errors which we get for other trading partner files during that time. 

    Monday, March 28, 2016 6:55 PM
  • Ok, here's what you can try next. I am assuming that you are using the EDIReceive pipeline for your tests.

    Change the highlighted values below to more than 2MB (say 3MB or 3072KB) at BizTalk Group Settings, and run your tests again -

    Thanks

    Arindam

    Tuesday, March 29, 2016 9:58 AM
  • The file has around 5100 transactions. The AS2 errors are not for this file(trading partner) but for other trading partners who are sending data via as2.

    Sorry, this is now a bit confusing.  Exactly what are you asking about?  Slow debatch time or the errors?

    Either way, 5100 transactions is a lot for such a small file but it now makes a bit more sense as to why it's taking so long.  Debatch time is driven mostly by number of output messages, not size.

    You may have to change your pattern to not debatch in the the initial EDI Receive.  Instead, use Preserve Interchange, ACKs and AS2 will still work fine, but you'd have to debatch later in your process.

    Here is a Wiki Article about working with Preserve Interchange: http://social.technet.microsoft.com/wiki/contents/articles/26691.biztalk-server-working-with-preserve-interchange-edi-xml-part-1.aspx

    Tuesday, March 29, 2016 12:13 PM