none
Is restarting Biztalk host instance as a daily schedule job a best practice? RRS feed

  • Question

  • I'm a BizTalk newbie, and recently ran into some issue where the biztalk orchestration get stuck and i'm forced to restart host instance to get the message processing again.

    What i found weird is by doing a test, i can see that the biztalk host instance in task manager takes up a lot of memory and doesn't release them even after the orchestration goes into dehydrated mode.

    Is it due to some bad programming that i'm doing in my biztalk orchestration?

    FYI my orchestration uses a utility dll that calls a wcf for any SQL/database operation.

    Any ideas?

    Thanks


    thanks ~steve

    Monday, August 13, 2012 4:03 AM

Answers

  • Hi,

      Restarting hosts on daily basis is not a good practice. It indicates some issues with your code or your biztalk environment is missing some patches. Also, take a look at Host throttling indicators. Please mention if you see exceptions such as "System.OutOfMemoryException" if yes, please read "http://support.microsoft.com/kb/918643"

    Issues with your code

    Please take process dump of the BizTalk Host (before restarting host) using command "cscript ADPlus.vbs -hang -–CTCF –p (pid of host instance)  -o E:\Support\dump"

    ADPlus is available with WINDBG here "http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx"'

    to Locate exceptions using WINDBG check here "http://blogs.msdn.com/b/johan/archive/2008/01/31/using-windbg-hunting-exceptions.aspx"

    Host throttling indicators

    Read : http://msdn.microsoft.com/en-us/library/aa578302.aspx

    Message delivery throttling state
    A flag indicating whether the system is throttling message delivery (affecting XLANG message processing and outbound transports).
    ·    0: Not throttling
    ·    1: Throttling due to imbalanced message delivery rate (input rate exceeds output rate)
    ·    3: Throttling due to high in-process message count
    ·    4: Throttling due to process memory pressure
    ·    5: Throttling due to system memory pressure
    ·    9: Throttling due to high thread count
    ·    10: Throttling due to user override on delivery
    Message publishing throttling state
    A flag indicating whether the system is throttling message publishing (affecting XLANG message processing and inbound transports).
    ·    0: Not throttling
    ·    2: Throttling due to imbalanced message publishing rate (input rate exceeds output rate)
    ·    4: Throttling due to process memory pressure
    ·    5: Throttling due to system memory pressure
    ·    6: Throttling due to database growth
    ·    8: Throttling due to high session count
    ·    9: Throttling due to high thread count
    ·    11: Throttling due to user override on publishing

    Regards,

    Vijendra


    Monday, August 13, 2012 1:59 PM
  • It could be because of host throttling so refer to the performance counters and try to capture these for your environment. You may also use the Performance Analysis of Logs (PAL) Tool


    Please mark the post answered your question as answer, and mark other helpful posts as helpful, it'll help other users who are visiting your thread for the similar problem, Regards -Rohit Sharma (http://rohitt-sharma.blogspot.com/)

    Monday, August 13, 2012 4:08 AM
    Moderator

All replies

  • I'm a newbie in biztalk, and recently we had some issues with our production application.

    So the issue is this, after a few days of running, the issue would appear where the processing of the messages get stuck.

    And checking on Windows event log and our own internal application log, and biztalk administration console, we would find no errors.

    Only thing that we can see is that the stuck orchestration would appear in ready to run state.

    I would then restart the stuck orchestration host instance, and it would start to run.

    Setup: FYI, i'm using biztalk 2009, in windows server 2008 r2, 24GB of memory, intel xeon processor.

    For data access, we have a utility DLL which calls WCF to retrieve data from SQL Server 2008 R2.

    Orchestration are programmed with a timer utility to exit after four hours of non operation.

    Suspicion: My suspicion is that somehow the orchestration loads a lot of data into the memory for processing and when the orchestration dies, it doesn't clean up the garbage properly.

    Question: What does restarting host instance actually do in relation to the orchestration, etc? Under what conditon could biztalk host instance could get stuck without any errors in the orchestration? What tools can i use to check for bottle neck issue which could lead to this issue?

    Thanks for your help :)

    Monday, August 13, 2012 1:57 AM
  • It could be because of host throttling so refer to the performance counters and try to capture these for your environment. You may also use the Performance Analysis of Logs (PAL) Tool


    Please mark the post answered your question as answer, and mark other helpful posts as helpful, it'll help other users who are visiting your thread for the similar problem, Regards -Rohit Sharma (http://rohitt-sharma.blogspot.com/)

    Monday, August 13, 2012 4:08 AM
    Moderator
  • In your utility dll class, since you're doing SQL/database operation, you sure that connection is properly closed/disposed?


    Randy Aldrich Paulo

    MCTS(BizTalk 2010/2006,WCF NET4.0), MCPD | My Blog


    BizTalk Message Archiving - SQL and File
    Automating/Silent Installation of BizTalk Deployment Framework using Powershell >
    Sending IDOCs using SSIS

    Monday, August 13, 2012 6:42 AM
  • Be aware of the ReceiveTimeout option for WCF-SQL and the known bug (check this blog post). Also as mentioned when you hit this state it may also be a throttling state. Check your databases to make sure you are within recommended values, good tools you can use is the BizTalk Best Practice Analyzer and the Messagebox Viewer.

    Edit: And a schedules job to reset host instances is not best practices and in my opinion not recommended, locate the root cause of the problem and eliminate it.. ;)

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.



    Monday, August 13, 2012 7:08 AM
  • Hi,

      Restarting hosts on daily basis is not a good practice. It indicates some issues with your code or your biztalk environment is missing some patches. Also, take a look at Host throttling indicators. Please mention if you see exceptions such as "System.OutOfMemoryException" if yes, please read "http://support.microsoft.com/kb/918643"

    Issues with your code

    Please take process dump of the BizTalk Host (before restarting host) using command "cscript ADPlus.vbs -hang -–CTCF –p (pid of host instance)  -o E:\Support\dump"

    ADPlus is available with WINDBG here "http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx"'

    to Locate exceptions using WINDBG check here "http://blogs.msdn.com/b/johan/archive/2008/01/31/using-windbg-hunting-exceptions.aspx"

    Host throttling indicators

    Read : http://msdn.microsoft.com/en-us/library/aa578302.aspx

    Message delivery throttling state
    A flag indicating whether the system is throttling message delivery (affecting XLANG message processing and outbound transports).
    ·    0: Not throttling
    ·    1: Throttling due to imbalanced message delivery rate (input rate exceeds output rate)
    ·    3: Throttling due to high in-process message count
    ·    4: Throttling due to process memory pressure
    ·    5: Throttling due to system memory pressure
    ·    9: Throttling due to high thread count
    ·    10: Throttling due to user override on delivery
    Message publishing throttling state
    A flag indicating whether the system is throttling message publishing (affecting XLANG message processing and inbound transports).
    ·    0: Not throttling
    ·    2: Throttling due to imbalanced message publishing rate (input rate exceeds output rate)
    ·    4: Throttling due to process memory pressure
    ·    5: Throttling due to system memory pressure
    ·    6: Throttling due to database growth
    ·    8: Throttling due to high session count
    ·    9: Throttling due to high thread count
    ·    11: Throttling due to user override on publishing

    Regards,

    Vijendra


    Monday, August 13, 2012 1:59 PM
  • Hi Guys,

    Thanks for all the replies here so far.

    Just to give an update of the current situation, i think restarting the biztalk host instances on a daily basis definitely didn't help.

    Because there was one week when i restarted the host instances manually, i would encounter the biztalk getting stuck the next morning.

    Thus far it seems that biztalk is running ok even with the memory leak.

    I have been getting issues now for two mondays in a row, as during the weekends, we won't get any transactions to processed.

    I would have to restart the host instances then i would have no issues for the rest of the week...

    I'm currently running perfmon on the biztalk server using the PAL templates, i've analyze the dump.

    And even try running debug diagnostic tools on it.

    I think my results are still inconclusive.

    My senior did some testing on his other server and found that anti virus can cause the host instances to hang.

    Just wondering did any of you encountered any issues with antivirus?

    The system is using symantec endpoint protection.

    Please advise.

    Thanks


    thanks ~steve

    Wednesday, August 22, 2012 8:42 AM
  • It is not recommended to have an anti-virus agent running on your bts or corresponding sql servers. However, in many situations you can not escape this because of company policy.

    I have had issues where the anti-virus program scanned the mdf and ldf database files when ever a change occured to these files (which is almost constantly).

    Make sure you configure your anti-virus to exclude bts bits and logfiles from scanning.

    Wednesday, August 22, 2012 11:34 AM