I have to pull the XML extract in to BizTalk again and do mapping and convert to edi. Since this is a big file , thought to do it using SSIS package to drop this file to a file folder and then receive this xml extract, debatch it and then
do the mapping , then I can batch and send the file.
-Prasad
I would recommend you first analyze what kind of data is coming in the message. It is very rare that large messages/files are interchanged for transactional activities. What I mean to say is most of the scenarios involve large files received as a day-end
activity. Do not come to conclusion that all messages should be handled in biztalk itself. If the message is more of data integration and less of business process integration, I would recommend you to use SSIS instead.
Few tips which might turn out to be useful if you choose BizTalk are below:
1) Try to avoid usage of Maps. If possible try to achieve the objective using promoted fields.
2) Most of the time problems are encountered when these large messages are handled in custom pipelines or .NET components. The reason is that the entire message is loaded into memory at once for any operations on the message. Always use Virtual streams while
reading message streams.
3) If the message size is substantially huge, debatching is the best technique. Develop a custom pipeline component to break the incoming message into smaller parts.
4) If you need to process the large message end-to-end (inbound to outbound as a single unit), then what you can do is debatch the message at inbound and aggregate all the message at the outbound using Sequential Convoys.
5) Adjust the "message size threshold". Any message larger than this size will be buffered to physical disk.
6) Adjust "message fragment size" property so that the message is split into fragments. MSDTC will be used to write into message box.
7) 64 bit version of SQL Server is recommended.
Refer: http://prashantbaker.blogspot.co.uk/2008/01/handling-large-size-messages.html
Rachit Sikroria (Microsoft Azure MVP)
