none
BizTalk 2013 R2 Performance issues

    Question



    1. There is an unusually high number of Running and Ready service instance in the queue. Historically; the usual number of Running service instances used to be up to several hundreds, and the number of ready service instances was 0 to tens of them. However in recent days I've noticed that there is  a spike in the numbers plus some system administrators and users experience a lag - the messages arrive later than usual.

    1. The BizTalk server CPU: 10% to 30% memory 50%

    1. The SQL server behind BizTalk - CPU oscillates around 90% memory around 70%


    1. During the off-peak hours the number of running/ready instances drops down to close to 0

    This was not the case even several weeks ago, - the number of running / ready instances where anything from 0 or 1 to 100, if it was at several hundred I was considering it a highly unusual situation.  Also there was no change to the company structure , there is the same number of users doing exactly what they used to do, but the existing users report slow performance of the client systems, some of the messages/responses that used to arrive in minutes take now hours to get through. If those messages would be suspended, then it would be obvious that there is something wrong with the ports or the hosts or the adapters etc. In this case, the messages are stuck in the processing apparently for no reason.

    Any suggestions ?



    God bless you all :)

    Monday, July 10, 2017 10:31 AM

All replies

  • Use BizTalk Health Monitor to check current situation (thottling issues are most likely cause)
    https://www.microsoft.com/en-us/download/details.aspx?id=43716

    I assume the number of message has not exceed the usual thresholds?
    All BizTalk SQL jobs running as scheduled?
    Do you have errors in the Monitor BizTalk Server job? Investigate why and solve it using Terminator (out of production hours)

    Use PAL to investigate 24/7, create a baseline and check for changes on/off
    http://pal.codeplex.com/

    Please report your findings in order for us to help you further

    hth /Peter

    Monday, July 10, 2017 11:03 AM
  • Hi,

    Are all Biztalk jobs are running properly ? You need to check MessageBox_Message_Cleanup_BizTalkMsgBoxDb,

    MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb job and DTA archive and purge job . 

    Better run BizTalk Health Monitor to look for additional issues like spool size and what process is causing a deadlock in overall message processing .


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply

    Tuesday, July 11, 2017 5:44 AM
  • There are three ports that show most in the Dehydrated / Running / and Ready service Instances:

    sprt_2Way_P57X7001_StatusUpdate_WCF-SQL

    sprt_2Way_P57X7001_StatusUpdate_WCF-ORA

    sprt_2Way_CCCProjects_WCF-SQL


    1. Which host are these ports using ?

    sprt_2Way_P57X7001_StatusUpdate_WCF-SQL

    CCCSendHost

    sprt_2Way_P57X7001_StatusUpdate_WCF-ORA

    CCCSendHost

    sprt_2Way_CCCProjects_WCF-SQL

    CCCSendHost





    1. Check the Throttling settings on the CCCSendHost

    USE [BizTalkMgmtDb]

    select * from [dbo].[adm_HostSetting]  with (nolock)

    WHERE HostId = 8


    1. THe result:

    HostId

    PropertyName

    PropertyValue

    8

    AllowMultipleResponses

    -1

    8

    DehydrationBehavior

    2

    8

    LegacyWhitespace

    0

    8

    MessagingMaxReceiveInterval

    500

    8

    MessagingReqRespTTL

    20

    8

    MsgAgentPerfCounterServiceClassID

    0

    8

    SubscriptionPauseAt

    0

    8

    SubscriptionResumeAt

    0

    8

    ThrottlingBatchMemoryThresholdPercent

    1

    8

    ThrottlingDeliveryOverride

    0

    8

    ThrottlingDeliveryOverrideSeverity

    100

    8

    ThrottlingLimitToTriggerGC

    80

    8

    ThrottlingPublishOverride

    0

    8

    ThrottlingPublishOverrideSeverity

    100

    8

    ThrottlingSeverityDatabaseSize

    1

    8

    ThrottlingSeverityInflightMessage

    75

    8

    ThrottlingSeverityProcessMemory

    500

    8

    ThrottlingSpoolMultiplier

    10

    8

    ThrottlingTrackingDataMultiplier

    10

    8

    TimeBasedMaxThreshold

    1800

    8

    TimeBasedMinThreshold

    1

    8

    UseDefaultAppDomainForIsolatedAdapter

    0

    8

    XlangMaxReceiveInterval

    500


    Q1 : How do I find out if the threashold is met ?

    Q2:  Which values need to get changed ?


    God bless you all :)

    Friday, July 21, 2017 3:25 PM
  • Hi ,

    Generally if the throttling occurs on any particular handler/Host it gets displayed in the event log. So check if the event log shows the warning that the host has throttled.

    Also run BHM tool to generate the complete health report of the server to find out if there are any bottlenecks in the environment.

    Thanks


    Mandar Dharmadhikari

    Monday, July 24, 2017 5:31 AM
    Moderator

    1. Warning occurred in the Event Log indicating that there is a throttling error in BizTalk

    Event ID 7001 - BizTalk host … throttled

    BizTalk host CCCSendHost throttled because InflightMessageCount exceeded the configured throttling limit.

    For more information see the BizTalk throttling documentation at http://go.microsoft.com/fwlink/?LinkId=185578.

    Event ID 7002 - BizTalk host … has recovered from resource based throttling

    BizTalk host CCCSendHost has recovered from resource based throttling




    1. Go to CCCSendHost > Settings > Resource-Based Throttling


    What should be changed please ?


    God bless you all :)

    Monday, July 24, 2017 1:27 PM
  • Hi btsadministrator,

    Please help us with more information here.

    1) what is the memory installed on the Biztalk app server?

    2) Does the BHM report point out any thing different or funny going on the sever?

    3) I would also advise you to use the perfmon to monitor the environment when you have above bottleneck in the system, there is something in your environment that is eating up lot of memory.

    Refer Host Throttling Performance Counters this might help your figure what is causing the resource crunh. Also for reference other perfmon counter: BizTalk Server: Performance Counters 

    4) Ensure that the each host has dedicated purpose of processing, tracking, recieveing and sending.

    Now as for throttling you will have to play around with the settings above to find out what combinations work for you because you first of all need to isolate what and what kind of resource crunch is happening on your environment. To get you going, please refer the MSDN documentation on the resource based throttling and how to overcome it.

    How to Modify Resource Based Throttling Settings


    Mandar Dharmadhikari

    Tuesday, July 25, 2017 12:47 AM
    Moderator
  • The memory is not high:


    God bless you all :)


    Tuesday, July 25, 2017 8:44 AM
  • <style type="text/css"><!--td {border: 1px solid #ccc;}br {mso-data-placement:same-cell;}--></style>
    Warnings Report
    Critical Warnings: 0
    Item Caption Item Value
    Non Critical Warnings: 32
    Item Caption Item Value
    General
    Errors during Collect 4 - Check the STATUS Log file produced or error messages reported above in this HTML file !
    BizTalk Group
    Class Settings Changed see the corresponding Query Report to see the changes !
    BizTalk Databases - General
    SSO DB Not Clustered (this DB is critical as it keep encrypted ports properties) !
    LOG Db Growth for BizTalkMgmtDb Not recomended to use percentage (Current =10%,Def=10%) - Recommended for this Db=10240 KB ?!
    MDF Db Growth for BizTalkMsgBoxDb Current =102400 KB (Def=1024 KB) - Recommended for this Db=1024 KB ?!
    MDF Db Growth for BizTalkDTADb Current =102400 KB (Def=1024 KB) - Recommended for this Db=1024 KB ?!
    LOG Db Growth for BizTalkDTADb Not recomended to use percentage (Current =10%,Def=10%) - Recommended for this Db=102400 KB ?!
    MDF Db Growth for BAMPrimaryImport Current =102400 KB (Def=1024 KB) - Recommended for this Db=1024 KB ?!
    LOG Db Growth for BAMPrimaryImport Not recomended to use percentage (Current =10%,Def=10%) - Recommended for this Db=10240 KB ?!
    LOG Db Growth for BizTalkRuleEngineDb Not recomended to use percentage (Current =10%,Def=10%) - Recommended for this Db=1024 KB ?!
    LOG Db Growth for SSODB Not recomended to use percentage (Current =10%,Def=10%) - Recommended for this Db=1024 KB ?!
    Throttling Settings
    Some Settings were changed see the corresponding Query Report to see the changes !
    BizTalk Hosts
    Host "BizTalkServerApplication" Run Receive Location+Send Port+Orchestration (Recommended to dedicate this host to run either ReceiveLocations, Orchestrations, or SendPorts) !
    Host "BTS32" Run Receive Location+Send Port+Orchestration (Recommended to dedicate this host to run either ReceiveLocations, Orchestrations, or SendPorts) !
    Host "FTP32" Run Receive Location+Send Port (Recommended to dedicate this host to run either ReceiveLocations, Orchestrations, or SendPorts) !
    Ports, Pipelines & Orchs
    Total Receive Locations Disabled 6 !
    MsgBody Tracking for some ReceivePorts (9 RLs impacted) Yes (MsgBody can accumulate in MsgBox db if 'TrackedMessages_Copy' job is not running to send them to DTA) !
    Total Send Ports Stopped 8 !
    Total Orch Stopped 1 !
    "Start and End Shapes" tracking events for Orch. Enabled. Not recommended in Production as they can impact perfomances. ?!
    Adapters
    Non WCF SQL adapter used in some Receive Locations Prefer to use the WCF one which is more performant !
    Non WCF SQL adapter used in some Send Ports Prefer to use the WCF one which is more performant !
    DTA Tables
    DTA MarkLog Table Rows 68191 starts to be a large number - Known issue caused by BackupJob !
    MsgBox Tables
    MarkLog Table Rows in BizTalkMsgBoxDb 68191 starts to be a large number - Known issue caused by BackupJob !
    MsgBox Database Integrity
    Total Messages with RefCount = -1 8 (Large number can impact MsgBox Dbs size and so perfs) !
    Other Checks
    Server PRODBIZTALK01 =3.5) !"}" style="padding:2px 3px;vertical-align:bottom;background-color:#ffff00;font-family:'Segoe UI Light';font-size:12pt;">Running in VMware Virtual Platform (Supported on ESX >=3.5) !
    Server PRODSQL1010 =3.5) !"}" style="padding:2px 3px;vertical-align:bottom;background-color:#ffff00;font-family:'Segoe UI Light';font-size:12pt;">Running in VMware Virtual Platform (Supported on ESX >=3.5) !
    Special TCPIP Settings SynAttackProtect enabled on SQL Server PRODSQL1010 - can generate General Network errors !
    Tunning
    maxconnection' property Is not present in some BizTalk process config files - You can configure the number of concurrent connections that the SOAP adapter opens for a particular destination server by adding "maxconnection" entry !
    EventLog
    Transactions Timeouts The adapter "WCF-Custom" raised an error message. Details "System.TimeoutException: The timeout "00:10:00" expired while executing the function "TryReceive". at Microsoft.Adapters.OracleCommon.OracleCommonUtils.SetCommandTimeout(OracleCommand command, TimeoutHelper timeoutHelper) at Microsoft.Adapters.OracleCommon.OracleCommonUtils.ExecuteReader(OracleCommonConnectionWrapper connection, String commandText, IEnumerable`1 parameters, CommandBehavior commandBehavior, IOracleAdapterCommonBindingProperties properties, OracleCommonExecutionHelper executionHelper) at Microsoft.Adapters.OracleDB.OracleDBInboundContract.Polling_TryReceive_StatementPolling(OracleCommonExecutionHelper executionHelper, Message& wcfMessage) at Microsoft.Adapters.OracleDB.OracleDBInboundContract.TryReceive(TimeSpan timeout, Message& message, IInboundReply& reply) at Microsoft.ServiceModel.Channels.Common.Channels.AdapterInputChannel.TryReceive(TimeSpan timeout, Message& message) at System.ServiceModel.Dispatcher.InputChannelBinder.TryReceive(TimeSpan timeout, RequestContext& requestContext) at System.ServiceModel.Dispatcher.ErrorHandlingReceiver.TryReceive(TimeSpan timeout, RequestContext& requestContext)". - You can Increase the Timeout via 'machineSettings' maxTimeout property !
    SQL Servers
    Current Error log on PRODSQL1010 Error(s) found in SQL Error log - Check if these issues were fixed !
    Current Error log on PRODSQL1010 Fatal' error(s) found in SQL Error log - Check if this issue was fixed !

    Tuesday, July 25, 2017 3:17 PM
  • Hi btsAdministrator,

    Looking at the report, I found that there is an event log warning saying the WCF-Custom adapter timed out is this adapter running on the same host instance which is throttling? If it is then I believe the instance of the adapter still runs.

    Transactions Timeouts The adapter "WCF-Custom" raised an error message. Details "System.TimeoutException: The timeout "00:10:00" expired while executing the function "TryReceive". at Microsoft.Adapters.OracleCommon.OracleCommonUtils.SetCommandTimeout(OracleCommand command, TimeoutHelper timeoutHelper) at Microsoft.Adapters.OracleCommon.OracleCommonUtils.ExecuteReader(OracleCommonConnectionWrapper connection, String commandText, IEnumerable`1 parameters, CommandBehavior commandBehavior, IOracleAdapterCommonBindingProperties properties, OracleCommonExecutionHelper executionHelper) at Microsoft.Adapters.OracleDB.OracleDBInboundContract.Polling_TryReceive_StatementPolling(OracleCommonExecutionHelper executionHelper, Message& wcfMessage) at Microsoft.Adapters.OracleDB.OracleDBInboundContract.TryReceive(TimeSpan timeout, Message& message, IInboundReply& reply) at Microsoft.ServiceModel.Channels.Common.Channels.AdapterInputChannel.TryReceive(TimeSpan timeout, Message& message) at System.ServiceModel.Dispatcher.InputChannelBinder.TryReceive(TimeSpan timeout, RequestContext& requestContext) at System.ServiceModel.Dispatcher.ErrorHandlingReceiver.TryReceive(TimeSpan timeout, RequestContext& requestContext)". - You can Increase the Timeout via 'machineSettings'

    Mandar Dharmadhikari

    Wednesday, July 26, 2017 3:57 AM
    Moderator
  • Your installation is in a bit of mess
    Before anything else make sure your host instances are configured according to best practice

    • One host / host instance only for tracking
    • Hosts dedicated to receiving, processing (orchestrations) or sending
      Host instances can be divided into dedicated services for eg FTP, HTTP (WCF) etc

    Please observe that eg FTP adapter / Host instance is 32 bit only and HTTP can be 64 bit

    For your own sanity give the hosts a name which relates to the usage

    /Peter

    Wednesday, July 26, 2017 9:36 AM