none
Delay between Receive & Send RRS feed

  • Question

  • Hello,

    I have a simple messaging scenario. Receive Port pulls 'x' records from DB, Debatched & Publishes them to Message Box. 1 or more Send Ports pick the debatched messages, execute some task & return null.

    While the above solution works great under no load, there is a significant delay between Receive & Send under heavy load.

    There is almost 5-8 minutes of delay between the last line of dis assembler in the receive port & the first line of "Pre-Assembler" on the send port.

    This happens only under load & not constant.

    There could be a 1. delay in publishing the messages from receive to message box DB. 2. delay in subscribing the messages at the send.

    No throttling. Performance counter data looks normal. 

    How can I debug more to find the root cause?


    Tuesday, April 3, 2018 12:58 PM

Answers

  • Check

    • Backlog size (size of spool table)
    • Message delivery rate vs processing rate
    • Publishing request vs completion
    • Message delivery delay
    • Message publishing delay
    • Also if you have dehydrated messages (BTS Console -> Group Hub -> Running Service Instances

    rgds /Peter

    • Marked as answer by Shiv G Wednesday, April 4, 2018 12:39 PM
    Tuesday, April 3, 2018 1:28 PM
  • "under heavy load" things will generally slow down...so...

    Is the Receive Port and any processing, Orchestrations/Send Ports in different Hosts?  BizTalk will always prioritize Receive Operations over Send within a particular Host.

    • Marked as answer by Shiv G Wednesday, April 4, 2018 12:39 PM
    Tuesday, April 3, 2018 5:56 PM
    Moderator
  • Run a MBV/BHM report to see if the environment doesn't have any bottlenecks. If they look fine, you can drill down into specific areas of the flow(I would enable tracking) and narrow it down further.

    Thanks Arindam

    • Marked as answer by Shiv G Wednesday, April 4, 2018 12:40 PM
    Wednesday, April 4, 2018 11:53 AM
    Moderator

All replies

  • Check

    • Backlog size (size of spool table)
    • Message delivery rate vs processing rate
    • Publishing request vs completion
    • Message delivery delay
    • Message publishing delay
    • Also if you have dehydrated messages (BTS Console -> Group Hub -> Running Service Instances

    rgds /Peter

    • Marked as answer by Shiv G Wednesday, April 4, 2018 12:39 PM
    Tuesday, April 3, 2018 1:28 PM
  • Thanks for the reply.

    I checked those parameters.

    • Backlog size (size of spool table):  150-200 under load & getting cleared as normal.
    • Message delivery rate vs processing rate 
    • Publishing request vs completion
    • Message delivery delay - Always 0
    • Message publishing delay - Always 0

    For the receive host, Message Delivery Incoming Rate & Message Delivery Outgoing Rate are 0. Message publishing incoming rate & outgoing rate are between 0 & 1.

    For the send host, Message Delivery Incoming Rate & Message Delivery Outgoing Rate are between 0 & 1. Message publishing incoming rate & outgoing rate are 0. 

    No dehydrated / Ready to run instances in Admin console. There are ~100-150 active instances.

    There is no throttling or abnormal spike in memory / cpu.

    Adding more instances is not an option for me. Do you have any other suggestion/parameters to check?



    • Edited by Shiv G Tuesday, April 3, 2018 2:21 PM
    Tuesday, April 3, 2018 2:14 PM
  • "under heavy load" things will generally slow down...so...

    Is the Receive Port and any processing, Orchestrations/Send Ports in different Hosts?  BizTalk will always prioritize Receive Operations over Send within a particular Host.

    • Marked as answer by Shiv G Wednesday, April 4, 2018 12:39 PM
    Tuesday, April 3, 2018 5:56 PM
    Moderator
  • Yes, Receive & Send are in two different host instances. There are no orchestrations.
    Tuesday, April 3, 2018 7:14 PM
  • Shiv, 

    Break your testing into two phase first test your Receive side.

    If you are using Custom Pipeline, I would suggest to do the load testing only with Receive side with Heavy load and dump the data into folder location using pass through pipeline to see how fast it’s processing.

    If your receive side is doing fine try to look into your send side scenarios to improve the performance.

    Thanks,

    Chandra

    Tuesday, April 3, 2018 7:24 PM
  • Messages are active for a long time?

    Any web service calls, database calls or helper classes which does a lot of work in a map/transformation or pipeline component?

    /Peter


    • Edited by Peter Lykkegaard Tuesday, April 3, 2018 9:43 PM Added "pipeline component"
    Tuesday, April 3, 2018 9:42 PM
  • Run a MBV/BHM report to see if the environment doesn't have any bottlenecks. If they look fine, you can drill down into specific areas of the flow(I would enable tracking) and narrow it down further.

    Thanks Arindam

    • Marked as answer by Shiv G Wednesday, April 4, 2018 12:40 PM
    Wednesday, April 4, 2018 11:53 AM
    Moderator