none
BizTalk orchestration issue while processing large message RRS feed

  • Question

  • Hello,

    i rceived a xml file 260MB   and my orchestration get started  and  remains  in "active" status.

    i think it stopped at construct shape   (mapping)

    yes it's a memory issue, or host instance issue and i would like what to do to resolve this without  debatching the input file.

    i  can  see  thate  we can change  the  configuration trough  groupe settings  but i do not know what to change excately

    do you have any idea on what to do?

    Thursday, July 6, 2017 2:40 PM

All replies

  • Well, 260MB is not that large.  Can you first determine exactly where is slowing down or stopping?

    As a test, can you apply that Map in the Port to see how long it takes?


    Thursday, July 6, 2017 2:53 PM
  • the probleme is that i am in go live environnement, so i can not just move the mapping from orchestration to send port.

    i have a scop in orchetration when i call a pipeline  to validate my xml file  and then a mapping.

    so my orchetration  reamins in active status when it came to this scop   for more  than 4hoours and counting.

    i  test the same file in quality env, and it's the same thing

    Thursday, July 6, 2017 2:56 PM
  • You can use BizTalk Performance Counters to determine processing

    4 hours is very long processing for such a simple task, the solution should never have left QA
    Why do you need an orchestration to validate and transform? Message only processing is preferable to have a low latency system

    /Peter

    Thursday, July 6, 2017 3:41 PM
  • Can you reproduce in a DEV environment?  Using a Port Map there.

    Somehow, you will have to identify the problem and the only way to do that is testing different scenarios.

    Thursday, July 6, 2017 5:29 PM
  • Hi ,

    If you have reproduced the issue in test then better to have proper design in place instead of trying to fix on environment level . The problem here is when you are trying to validate the incoming xml within scope it is throwing you exception or just unable to respond with exception message .

    The proper design would be to validate the message before it hits biztalk message box .Use Orchestration as workflow instead of  making it place of validation logic .

    Thanks

    Abhishek


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply

    Thursday, July 6, 2017 6:24 PM
  • thx Peter for your answer.

    the istutaion is that  this orchestration was  supposed  to process small  files  and it works  fine.

    it's just that  nowsome customers start  drop  some big files...

    so i need  to make  this work   now  even for those  big file 

    yes  i need  to  redevloppe my  orchestration  but at first i wnt to just   make it work by chnaging the settings: memory used....   through  biztlak groupe  settings...but i don't know  how!

    Friday, July 7, 2017 7:43 AM
  • Hello Abhishek

    At first  i don t want  to change   the design... the orchestration works   fine  for  small  files

    i just want  to know if there is any biztalk groupe  settings  (memory used.....)  to  make it works also  for big  files!

    Friday, July 7, 2017 7:47 AM
  • Most implementations will work with small files, with large files you need to to be extra carefull to get it right

    /Peter

    Friday, July 7, 2017 7:59 AM
  • Check this recent thread
    https://social.msdn.microsoft.com/Forums/en-US/0ecd349c-b4b6-4688-a437-2a0e82514c81/biztalk-xml-file-size-limit?forum=biztalkgeneral

    How Biztalk Processes large messages:
    https://msdn.microsoft.com/en-in/library/aa560481.aspx

    You have a couple of knobs to adjust: Large message fragment size and Large message threshold

    Read the above document and change configuration on the QA system at first and check overall BizTalk performance between each changes

    /Peter

    Friday, July 7, 2017 8:05 AM
  • The think is, it's difficult to provide advice unless we know what or at least where the problem is.  Otherwise, you're just randomly changing values which you should not be doing in PROD really :(.

    The fastest way to resolve this is to reproduce the problem in DEV and go from there.  Once we know exactly which step is halting, we can give you some direction.

    Friday, July 7, 2017 11:22 AM
  • i am agree  with you Johns

    that's why i reproduce  the same issue in dev  environnement  and  it stop  at  validation xsd pipline inside  my orchetration.

    with ut of memory error

    Friday, July 7, 2017 1:37 PM
  • Are you loading the message in an XmlDocument variable?

    /Peter

    Friday, July 7, 2017 3:21 PM
  • i have some   xml validation  pipeline   with an xsd doucment +  a mapping

    i think it fails/freezs at  xml validation  with out of memory error

    it's a test  performance to check  if we are able to process  this  kind of  file or do we  need  to  change biztalk groupe settings  or the orchetrstaion design

    Friday, July 7, 2017 3:24 PM
  • Hi ,

    I would suggest if you found the root cause for the exception which is in your case validation of xml based on xsd within Orchestration better option is to fix the issue on solution level instead of touching production environment setup . Biztalk process message on thread based and you do not know fix for one thing might break other processing parts or slow it down .

     


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply

    Saturday, July 8, 2017 5:36 AM
  • You need to establish proper processing with large messages in your QA

    Settings in production can and should be changed according to your requirements

    Start with QA I do hope you have similar hardware and configuration as in your setup

    1. Check with documentation for proper configuration of biztalk, eg tracking host, hosts for receiving, processing and sending at least
      If your configuration differs from recommended you need to establish this in QA first
    2. Proper documentation, write configuration down before/after changes make sure you can revert
    3. Create 24 hour baselines with PAL, know your environment
    4. After each change you need to check again create a new baseline with PAL, examine if have any change for the better or worse
    5. Iterate the above steps

    /Peter

    Saturday, July 8, 2017 8:05 AM
  • Can you post the error message here, so that we can try to understand what happened
    Sunday, July 9, 2017 4:14 AM
  • Hi,

    there is   no error message.

    the orchestration remains    in "active" status. that's it.

    when i  resubmit the same file in my test env  (with less memory), i got : out of memory error.

    Tuesday, July 11, 2017 7:49 AM
  • My advice would be to change the way messages are processed in the Biztalk.

    Best way would be to strip the message out to a physical location on the server and then once the rest of the processing is done, then pick up the message from the disk only when message is leaving the Biztalk environment.


    Mandar Dharmadhikari

    Monday, July 24, 2017 6:58 AM