none
Issue with BizTalk host throttled because ProcessMemory exceeded the configured throttling limit. RRS feed

  • Question

  • Hi

    We are getting the error

    BizTalk host EhrP_Send throttled because ProcessMemory exceeded the configured throttling limit.

    For more information see the BizTalk throttling documentation at http://go.microsoft.com/fwlink/?LinkId=185578. I re

    I restart the 36 hosts that we have and the messages clear all out but then they start accumulating once again.

    The host are 64bit.

    I also tried to add a column in the query and it kills the Biztalk Administration console with a bunch of errors:

    ---------------------------------------------------------------------------------------------------------------

    Application: mmc.exe

    Framework Version: v4.0.30319

    Description: The process was terminated due to an unhandled exception.

    Exception Info: System.NullReferenceException

       at System.Windows.Forms.DataGridViewCell.PaintWork(Graphics graphics, Rectangle clipBounds, Rectangle cellBounds, Int32 rowIndex, DataGridViewElementStates cellState, DataGridViewCellStyle cellStyle, DataGridViewAdvancedBorderStyle advancedBorderStyle, DataGridViewPaintParts paintParts)

       at System.Windows.Forms.DataGridView.PaintColumnHeaders(Graphics g, Rectangle clipBounds, Boolean singleBorderAdded)

       at System.Windows.Forms.DataGridView.PaintGrid(Graphics g, Rectangle gridBounds, Rectangle clipRect, Boolean singleVerticalBorderAdded, Boolean singleHorizontalBorderAdded)

       at System.Windows.Forms.DataGridView.OnPaint(PaintEventArgs e)

       at Microsoft.BizTalk.SnapIn.Framework.Forms.DataGridViewEx.OnPaint(PaintEventArgs e)

       at System.Windows.Forms.Control.PaintWithErrorHandling(PaintEventArgs e, Int16 layer)

       at System.Windows.Forms.Control.WmPaint(Message& m)

       at System.Windows.Forms.Control.WndProc(Message& m)

       at System.Windows.Forms.DataGridView.WndProc(Message& m)

       at Microsoft.BizTalk.SnapIn.Framework.Forms.DataGridViewEx.WndProc(Message& m)

       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)

       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)

       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

    Exception Info: System.Reflection.TargetInvocationException

       at Microsoft.ManagementConsole.Internal.SnapInMessagePumpProxy.OnThreadException(Object sender, ThreadExceptionEventArgs e)

       at System.Windows.Forms.Application.ThreadContext.OnThreadException(Exception t)

       at System.Windows.Forms.Control.WndProcException(Exception e)

       at System.Windows.Forms.Control.ControlNativeWindow.OnThreadException(Exception e)

       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

    ---------------------------------------------------------------------------------------------------------------

    Faulting application name: mmc.exe, version: 6.3.9600.17415, time stamp: 0x545044c4

    Faulting module name: KERNELBASE.dll, version: 6.3.9600.18202, time stamp: 0x569e72bb

    Exception code: 0xe0434352

    Fault offset: 0x00015b68

    Faulting process id: 0x11d00

    Faulting application start time: 0x01d2669f5e5e8b9a

    Faulting application path: C:\Windows\SysWOW64\mmc.exe

    Faulting module path: C:\Windows\SYSTEM32\KERNELBASE.dll

    Report Id: 5b44d0cc-d294-11e6-80d0-005056ae59ca

    Faulting package full name:

    Faulting package-relative application ID:

    ---------------------------------------------------------------------------------------------------------------

    Fault bucket , type 0

    Event Name: CLR20r3

    Response: Not available

    Cab Id: 0

    Problem signature:

    P1: mmc.exe

    P2: 6.3.9600.17415

    P3: 545044c4

    P4: System.Windows.Forms

    P5: 4.6.1055.0

    P6: 563c0fe8

    P7: 1998

    P8: 0

    P9: N3CTRYE2KN3C34SGL4ZQYRBFTE4M13NB

    P10:

    Attached files:

    These files may be available here:

    C:\Users\adminsm\AppData\Local\Microsoft\Windows\WER\ReportArchive\AppCrash_mmc.exe_a1dfa682a7303bf2d73274a5f32b6b61e45ff22_d1af2f8a_349a720c

    Analysis symbol:

    Rechecking for solution: 0

    Report Id: 5b44d0cc-d294-11e6-80d0-005056ae59ca

    Report Status: 2048

    Hashed bucket:

    ---------------------------------------------------------------------------------------------------------------

    Faulting application name: mmc.exe, version: 6.3.9600.17415, time stamp: 0x545044c4

    Faulting module name: KERNELBASE.dll, version: 6.3.9600.18202, time stamp: 0x569e72bb

    Exception code: 0xc000041d

    Fault offset: 0x00015b68

    Faulting process id: 0x11d00

    Faulting application start time: 0x01d2669f5e5e8b9a

    Faulting application path: C:\Windows\SysWOW64\mmc.exe

    Faulting module path: C:\Windows\SYSTEM32\KERNELBASE.dll

    Report Id: 5c48b1c7-d294-11e6-80d0-005056ae59ca

    Faulting package full name:

    Faulting package-relative application ID:

    ---------------------------------------------------------------------------------------------------------------

    Fault bucket , type 0

    Event Name: APPCRASH

    Response: Not available

    Cab Id: 0

    Problem signature:

    P1: mmc.exe

    P2: 6.3.9600.17415

    P3: 545044c4

    P4: KERNELBASE.dll

    P5: 6.3.9600.18202

    P6: 569e72bb

    P7: c000041d

    P8: 00015b68

    P9:

    P10:

    Attached files:

    These files may be available here:

    C:\Users\adminsm\AppData\Local\Microsoft\Windows\WER\ReportArchive\AppCrash_mmc.exe_3c755d669d3e739155c717f194413fbd685af3fe_d1af2f8a_431a76b0

    Analysis symbol:

    Rechecking for solution: 0

    Report Id: 5c48b1c7-d294-11e6-80d0-005056ae59ca

    Report Status: 2048

    Hashed bucket:

    ---------------------------------------------------------------------------------------------------------------

    Wednesday, January 4, 2017 4:07 PM

Answers

  • Hi

    Since these are 64bit Hosts, you can safely increase the Process Virtual setting from default of 25 to 90(or even more) in the Hosts tab -> Resource Based Throttling tab. Make sure that the available RAM/memory on the box is sufficient first.

    You have to make this change for the Hosts that are running into this and restart the Host Instances.

    https://msdn.microsoft.com/en-us/library/ff629725.aspx


    Thanks Arindam

    Wednesday, January 4, 2017 6:17 PM
    Moderator
  • On that server, we have 24gb of RAM and we are using 20gb. Not sure if we would need more for the number of hosts.

    Thanks for the info, I will try it

    The message that was received probably required more processing resources than expected, or more messages arrived than expected.

    Then BizTalk reaches the limits of the available configured resources like memory. This often solves itself but it takes more time. In some scenarios the processing comes to a halt, because of this and BizTalk basically can't continue. If this is the case you should look at the performance counters which indicates if BizTalk is throttling or not and what the memory consumption is. This MSDN article explains this and guides you which performance counters to monitor:

    http://msdn.microsoft.com/en-us/library/aa578302(v=bts.20).aspx

    Most of the time if you have a out-of-the-box host instance or created one yourself with default settings, the allowed memory consumption is 25% of the available memory. You should look at these settings if this occurs more often.

    A 32-bit host instance can only address to a maximum of 2 GB but as you are already on 64-bit host instance that addresses a much higher memory space (up to 7 TB).

    So you can safely Increase the allowed Memory Usage for the Host (Right Click Group->Settings->Hosts->Resource-Based Throttling->Memory Usage->Process Virtual). This value signifies that BizTalk starts throttling when the Process Memory for a host reaches 25% of the total memory available.

    So if your total Memory is 24 GB. Then, 25 % of 24 GB = 6 GB is were the throttling occurs witth the currently configuration. If you increase the value to 90% then 90% of 24 GB = 21.6 GB is where the throttling will start.


    Rachit Sikroria (Microsoft Azure MVP)

    Thursday, January 5, 2017 3:24 AM
    Moderator
  • Hi 

    I would check the code, Are you usign any custom pipeline for processing the messages? 

    Does this happen only in production ? did you every performed load testing of the code ?

    You can always change the settings as suggested by pther members here but I would also suggest you to profile your code to understand what is causing the throttling to engine? 

    Run PAL gather charts/counters and study the patten. It might be some code at the end. 

    Hope this helps


    Cheers,HTH

    Naushad Alam http://alamnaushad.wordpress.com/

    When you see answers and helpful posts, please click Vote As Helpful , Propose As Answer , and/or Mark As Answer

    Thursday, January 5, 2017 5:00 PM
    Moderator

All replies

  • Hi

    Since these are 64bit Hosts, you can safely increase the Process Virtual setting from default of 25 to 90(or even more) in the Hosts tab -> Resource Based Throttling tab. Make sure that the available RAM/memory on the box is sufficient first.

    You have to make this change for the Hosts that are running into this and restart the Host Instances.

    https://msdn.microsoft.com/en-us/library/ff629725.aspx


    Thanks Arindam

    Wednesday, January 4, 2017 6:17 PM
    Moderator
  • On that server, we have 24gb of RAM and we are using 20gb. Not sure if we would need more for the number of hosts.

    Thanks for the info, I will try it

    Wednesday, January 4, 2017 6:19 PM
  • Hi,

    20GB out of 24 GB ? Can you run perfmon and verify which process is consuming larger chunk of the memory.

    I do suspect your environment need some investigation as there might be some memory leak  or issue with Biztalk databases . Try perfmon and Messagebox viewer and let us know the report 


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply

    Thursday, January 5, 2017 12:17 AM
  • On that server, we have 24gb of RAM and we are using 20gb. Not sure if we would need more for the number of hosts.

    Thanks for the info, I will try it

    The message that was received probably required more processing resources than expected, or more messages arrived than expected.

    Then BizTalk reaches the limits of the available configured resources like memory. This often solves itself but it takes more time. In some scenarios the processing comes to a halt, because of this and BizTalk basically can't continue. If this is the case you should look at the performance counters which indicates if BizTalk is throttling or not and what the memory consumption is. This MSDN article explains this and guides you which performance counters to monitor:

    http://msdn.microsoft.com/en-us/library/aa578302(v=bts.20).aspx

    Most of the time if you have a out-of-the-box host instance or created one yourself with default settings, the allowed memory consumption is 25% of the available memory. You should look at these settings if this occurs more often.

    A 32-bit host instance can only address to a maximum of 2 GB but as you are already on 64-bit host instance that addresses a much higher memory space (up to 7 TB).

    So you can safely Increase the allowed Memory Usage for the Host (Right Click Group->Settings->Hosts->Resource-Based Throttling->Memory Usage->Process Virtual). This value signifies that BizTalk starts throttling when the Process Memory for a host reaches 25% of the total memory available.

    So if your total Memory is 24 GB. Then, 25 % of 24 GB = 6 GB is were the throttling occurs witth the currently configuration. If you increase the value to 90% then 90% of 24 GB = 21.6 GB is where the throttling will start.


    Rachit Sikroria (Microsoft Azure MVP)

    Thursday, January 5, 2017 3:24 AM
    Moderator
  • Hi 

    I would check the code, Are you usign any custom pipeline for processing the messages? 

    Does this happen only in production ? did you every performed load testing of the code ?

    You can always change the settings as suggested by pther members here but I would also suggest you to profile your code to understand what is causing the throttling to engine? 

    Run PAL gather charts/counters and study the patten. It might be some code at the end. 

    Hope this helps


    Cheers,HTH

    Naushad Alam http://alamnaushad.wordpress.com/

    When you see answers and helpful posts, please click Vote As Helpful , Propose As Answer , and/or Mark As Answer

    Thursday, January 5, 2017 5:00 PM
    Moderator
  • Hi,

    to get insight into this specific issue (throttling) you can use the free version of our monitoring platform AIMS (www.aimsinnovation.com/aims-free). AIMS extracts a bunch of performance data from server, BizTalk hosts and for every port & orchestration (without bam or tracking), and you can then chart and correlate all the performance data for root-cause analysis.

    So instead of just adding more RAM without finding the reason, isolate the issue so you can perhaps fix it to reduce stress on memory.

    Thanks,

    Marius

    Wednesday, March 15, 2017 9:41 AM