none
BizTalk 2006 R2 EDI Pipeline Processing Speed RRS feed

  • Question

  • Hi all,

     

    I am developing a process to accept HIPAA 834 files for membership maintenance received from our clients. I am dealing with 834 files that can up to 1 million members within them. These are obviously huge files and can vary in size from 200 MB to 1 GB. I am trying to figure out what are some possible options available to increase the translation time from the 834 file to the xEDI representation of the transactions. BizTalk seems to process the entire file in memory before sending out the xEDI. I assume it works this way so as to know if the file is vaild before passing out the xEDI and creating the 997 response transaction.

     

    My question is what can be done with either performance tuning settings or physical hardware to allow BizTalk to increase the speed at which it can parse the 834 file? I have made sure that I am not being throttled by setting the Process memory usage setting to 0 and Physical memory usage to 0 as well as turned off rate based throttling. Either way BizTalk doesn't seem to take advantage of more physical memory made available to it. The

    BTSNTSvc.exe process doesn't go much over 300MB of memory usage when the server has over 2GB of memory it can use. I have also tested in different CPU configurations ranging from 1 CPU to 4 CPUs. In either case by adding more CPUs to the BizTalk server it just seems to spread the processing time accross the CPUs and doesn't translate the file at any faster rate.

     

    Is this just a limitation of the EDI pipeline in that it can only translate EDI 834 files at a certain rate no matter how much physical horsepower I can make available to the server?

     

    Any help/ideas would be appreciated.

     

    Thanks,

    Joel

    Thursday, November 6, 2008 2:50 PM

All replies

  • The BizTalk EDI pipelines use streaming and hence dont load all the msg into memory. It does wait until the doc is processed to generate the 997s.

     

    I can check with other members and if I have any info, will share

    Monday, December 1, 2008 11:29 AM
    Moderator
  • Hi Joel,

     are you splitting the Hipaa subdocuments during processing? There were known performance issue when EDI interchange are split by BizTalk EDI. You can apply a quick fix released by our team available at:

     

    http://support.microsoft.com/default.aspx/kb/953388

     

    Thanks,

    Bhola

     

    Monday, December 1, 2008 12:18 PM
  • Sorry to Intervene. Does that hotfix fix the issue reported here http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=4029449&SiteID=1 ?

     

    thanks

    Monday, December 1, 2008 3:22 PM
  •  

    Hi Bhola,

    Yes we are splitting the HIPAA subdocuments during processing. We know about the known performance issue and have already applied the hotfix 954557 which includes the hotfix 953388 that you mentioned. We didn't see much if any performance gain on the translation speed of the 834 documents, but did see a gain in the 837 documents. This seems to be because the 837's have much more data in them for a given claim compared to the small amount of data for a given member in the 834.

     

    From the reponses on this forum it seems that the real issue is that the documents are streamed into memory and transformed in the xEDI documents and that process is a "black box" portion of BizTalk. I have plenty of system memory that could be utilitzed by BizTalk and I am really looking for a way to increase the amount of data from the document that BizTalk can stream into working memory. Are there any registry setting that could be utilized for this?

     

    I have already worked with the tuning properties available, but BizTalk still doesn't take advantage of the system memory that is available. For example, the BTSNTSvc.exe process doesn't seem to consume much over 300MB of system memory when translating an 834 document that has 200,000 member transactions within it, even when I set the Memory Usage throttling thresholds to 0 (zero). It seems again, like BizTalk is only allowed to consume a certain amount of the streamed file into memory.

     

    Any other suggestions? I have plenty of system memory and mulitple CPUs in the server that just are under utilized during the transformation process.

     

    Thanks,

    Joel

     

    Monday, December 1, 2008 4:54 PM
  •  gjoseph wrote:

    Sorry to Intervene. Does that hotfix fix the issue reported here http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=4029449&SiteID=1 ?

     

    thanks

     

    Hi gjoesph,

     

    No, the hotfix mentioned in this posting does not address the issue raised in the posting you linked to. That is still an open issue with support that is being worked on. Sorry, we wish that one was fixed too. :-(

     

    Joel

     

    Monday, December 1, 2008 4:56 PM
  • Thanks Joel for the update. I tried the hotfix and as you mentioned it did improve the 837 parsing performance. I am desperately waiting for the other hotfix.

     

    thanks

     

    Monday, December 1, 2008 9:08 PM
  • Can you escalate it through product support team so we can provide quickfix?

     

    Thanks,

    Bhola

     

    Wednesday, December 3, 2008 5:34 AM
  •  Bhola Meena[MSFT] wrote:

    Can you escalate it through product support team so we can provide quickfix?

     

    Thanks,

    Bhola

     

     

    Hi Bhola, was your response for me or for gjoseph? I just need to know if there is even an option to make BizTalk 06 R2 use all available resources on the server OR if the "black box" translation process from EDI document to xEDI only uses a fixed amount of server memory and CPU and that can't be changed?

     

    Joel

    Wednesday, December 3, 2008 2:46 PM
  • I have opened a ticket with the Support team to address the splitting issue.

     

    thanks

    Wednesday, December 3, 2008 2:59 PM
  • Just a quick status update here...
    Microsoft has been working on a hotfix for this splitting issue (as mentioned in http://social.msdn.microsoft.com/forums/en-US/biztalkediandas2/thread/b2765f46-b885-49cc-9f1b-60f85004a240/) and is close to finalizing it.
    It'll be available for public download soon.

    Gerard
    • Proposed as answer by providedBanned Friday, January 27, 2012 6:13 AM
    Saturday, February 14, 2009 12:18 AM