none
High Memory Utilization and Job Failure RRS feed

  • Question

  • One of our clients reported high memory utilization on the BizTalk server which was progressively increasing.

    As per the report shared by them the memory utilization has been increasing since may 2016 and is constantly between 70-90%. The BizTalk installation at their end is on virtual machines done in sept 2014 and the version installed is 2013.

    Along with this some occurrences of failure of the MessageBox_UpdateStats_BizTalkMsgBoxDb job were also reported. Below is the error that was logged in the job history

    Executed as user: ABCD\ap.SQLSINV170032. ...E 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[Fragments] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[InstanceStateMessageReferences_MQ64BitHost1] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[Parts] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[BizTalkReceiveHost_DequeueBatches] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[BizTalkReceiveHost_MessageRefCountLog] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[MessageParts] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[BizTalkReceiveHostQ_Scheduled] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[DynamicStateInfo_BizTalkServerApplication] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[MessageRefCountLog1] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[InstanceStateMessageReferences_BizTalkServerApplication] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[MessageRefCountLog2] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 1 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[MQ64BitHost] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 2 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[BizTalkReceiveHostQ_Suspended] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 4 did not require update. [SQLSTATE 01000] (Message 15651)  Updating [dbo].[MessageRefCountLogTotals] [SQLSTATE 01000] (Message 15650)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  , update is not necessary... [SQLSTATE 01000] (Message 15653)  0 index(es)/statistic(s) have been updated, 2 did not require update...  The step failed.

    Any pointers to help in this regard is appreciated.

    Friday, August 5, 2016 10:07 AM

Answers

  • Hi,

    Is this behavior new? if yes most likely can be related to your BizTalk DB size, please try to run message box viewer, check your spool table and see if the numbers are above 1000. Also if you are using BizTalk 2013 and onward it comes integrated with BHM. You will be able to find MBV here on your server,

    D:\Program Files (x86)\Microsoft BizTalk Server 2013\SDK\Utilities\Support Tools\MsgBoxViewer

    the result of MBV for sure is going to give you some definite pointers:


    Also check if your BizTalk Back up/Purge/ Archive  jobs are enabled and executing as expected.

    Please check the following article:

    http://social.technet.microsoft.com/wiki/contents/articles/29180.biztalk-server-how-to-use-messagebox-viewer-to-compare-multiple-environments.aspx

    CRegards Pushpendra K Singh





    Tuesday, August 9, 2016 2:11 PM

All replies

  • Hi

    Is the memory utilization high on the BizTalk Server(s)? On each of your BizTalk Servers in the Group, you have to inspect from Task Manager -> Process Tab, which process(es) are consuming the memory.

    If these are BizTalk Host Instance processes, you have to then drill down into what all BizTalk artifacts(ports, orchestrations) are running in those Host Instances. How many Active/Running Instances are present in those Host Instances as seen from Admin Console? Do you get high memory at all times, or only when BizTalk is processing messages? Memory usage can go up during times of high load, but should come down eventually.

    Once you have the above information, we can offer you more advice.

    The SQL error that you see is possibly not related to the High Memory on BizTalk Server - you have to analyze that separately. Is the MessageBox_UpdateStats_BizTalkMsgBoxDb Job failing always? What about the other BizTalk SQL Agent Jobs - are they running fine?

    You can collect a MBV report and check the health of the BizTalk databases/SQL Jobs. In the BizTalk install folder, you will find the MBV tool in the "SDK\Utilities\Support Tools\MsgBoxViewer" path.


    Thanks Arindam





    Friday, August 5, 2016 10:11 AM
    Moderator
  • Hi mailrash2k,

    As Arindam pointed out,

    when you say memory utilization is High in BizTalk server , have you checked each and every server in the BizTalk group, application and SQL servers both. Also you should check the following

    1) What is the memory utilization on each server, what is the particular process which is clogging the memory? if it is a BizTalk service at all or not.

    2) If it is a service related to BizTalk, then check the number of active service instances in the group.

    3) Also Check if the message box performance is up to the mark.

    4) Check if the Message box size is under the upper threshold of generally allowed size of 5 GB(ideally it should be way smaller than that)

    5) Run the BHM tool to get the details about the environment...it will show if there is some critical warning which needs to be addressed.

    Once you identify the problem to be related to BizTalk then comes the deep dive into analysis of the reports and data you have at hand you may have to do these thing 

    1)Analyze the performance of the BizTalk Databases using the performance counters

    2) Analyze the performance of the BizTalk Host Instances using the performance counters

    3) Analyze the design of the applications if the problem is not at infrastructure level but rather at the design level....

    Also what is the frequency of failure of this job...does this occur only in conjunction with the memory clog??

    Regards,


    Mandar Dharmadhikari

    Friday, August 5, 2016 10:24 AM
    Moderator
  • HI,

    For this issue, it's recommended to look into the processors and SQL jobs, and check whether they work fine.

    At the same time you need also use latest BizTalk 2013 updates, and check whether it makes any difference.

    Regards,

    Angie


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Saturday, August 6, 2016 6:26 AM
  • Thanks Arindam for your advice. I will check with the client regarding the artifacts and host instances. The setup is simple, they have 1 Sql box and 1 BizTalk Box.

    Regarding the jobs, other than this job that they pointed out rest are fine. The failing job also doesn't fail persistently, it is random.

    I'll also advice them to share with us the MBV report.

    Thanks Again!

    Monday, August 8, 2016 1:29 AM
  • Thanks Mandar for your advice.

    The BizTalk setup at the client end is 1 SQL Box and 1 BizTalk Box. They have monitoring tools that alert when the memory utilization is high and it was reported only for the BizTalk Server. As you mentioned, i'll request them to let us know which are the host instances that are impacted the most and the artefacts tied to those as well as the DB size.

    Regarding the job failure, it happens randomly and it is only this job that is impacted the rest are fine.

    Will get back with more updates on this case.

    Thanks Again!

    Monday, August 8, 2016 1:33 AM
  • Thanks Angie for your advice. The client already has all the latest BizTalk 2013 updates installed. Will check with them on the processors and SQL jobs.

    Thanks Again!

    Monday, August 8, 2016 1:34 AM
  • Hi,

    Is this behavior new? if yes most likely can be related to your BizTalk DB size, please try to run message box viewer, check your spool table and see if the numbers are above 1000. Also if you are using BizTalk 2013 and onward it comes integrated with BHM. You will be able to find MBV here on your server,

    D:\Program Files (x86)\Microsoft BizTalk Server 2013\SDK\Utilities\Support Tools\MsgBoxViewer

    the result of MBV for sure is going to give you some definite pointers:


    Also check if your BizTalk Back up/Purge/ Archive  jobs are enabled and executing as expected.

    Please check the following article:

    http://social.technet.microsoft.com/wiki/contents/articles/29180.biztalk-server-how-to-use-messagebox-viewer-to-compare-multiple-environments.aspx

    CRegards Pushpendra K Singh





    Tuesday, August 9, 2016 2:11 PM