locked
Very large Positional Flat File ...Processing RRS feed

  • Question

  • Hi,

         I have a positional flat file which is > 0.5GB in size.Using the FF wizard the FF was transformed into an XML schema.
    The FF has header info.
    From the original FF records (having 50 items) we need only 5 items per record. Hence a transform was done to collect and calculate only the neccessary items.This is then loaded into the DB using bulkinsert.

         All this works fine when the size of the file is small.

    When it is > 3MB  the system errors out.

    What is the best approach to process large flat files ?
    What are settings in Biztalk that i need to be aware of for processing these large files ?

    Any good articles for Biztalk 2006 ?




    -Jerry
        

    Friday, July 11, 2008 4:23 PM

Answers

  • Avoid loading large messages into BizTalk orchestrations (and large >1MB in this context). Debatching file in pipeline and direct port binding can greatly improve performance. If you need mapping, keep message size under 10MB and apply maps on port level. See if "mapping" can be done in pipeline components in a stream like fashion, without map engine at all.

     

    Here's good article from BizTalk team, its old, a lot of things improved (particulary maps performance) but still a lot of it holds true. http://blogs.msdn.com/biztalk_core_engine/archive/2005/02/28/381700.aspx

     

    Friday, July 11, 2008 7:39 PM
    Answerer
  • The article below seems to be a good guideline from microsoft for dealing with large files.

    http://technet.microsoft.com/en-us/library/aa560481.aspx

    How can i do mapping in pipeline components using virtual streams ?
    Are there any good articles for debatching huge flatfiles in an efficient manner ?
    i did g throught debatching flatfiles ....but if the i/p is huge what is the best approach ?

    -jerry


    Sunday, July 13, 2008 6:49 PM

All replies

  • Avoid loading large messages into BizTalk orchestrations (and large >1MB in this context). Debatching file in pipeline and direct port binding can greatly improve performance. If you need mapping, keep message size under 10MB and apply maps on port level. See if "mapping" can be done in pipeline components in a stream like fashion, without map engine at all.

     

    Here's good article from BizTalk team, its old, a lot of things improved (particulary maps performance) but still a lot of it holds true. http://blogs.msdn.com/biztalk_core_engine/archive/2005/02/28/381700.aspx

     

    Friday, July 11, 2008 7:39 PM
    Answerer
  • The article below seems to be a good guideline from microsoft for dealing with large files.

    http://technet.microsoft.com/en-us/library/aa560481.aspx

    How can i do mapping in pipeline components using virtual streams ?
    Are there any good articles for debatching huge flatfiles in an efficient manner ?
    i did g throught debatching flatfiles ....but if the i/p is huge what is the best approach ?

    -jerry


    Sunday, July 13, 2008 6:49 PM