Instances stuck in ready to run state and need host restarting

Question

 

Hi All,

      Our production has been experiencing a weird issue. All instances ( orchestrations and send ports) are in ready to run state and only restarting the host instances can solve the problem and then the issue will come back again after a couple of days. I created a simple application which just drops a message in a folder and sends the message out to a folder using a new host. I scheduled the application to run every half an hour. The issue happened last night for this simple application. At that time period, no any other messages in and out. The volume in our system is low all the time. We also configured SCOM to monitor throttling. I checked all the logs in both BizTalk and SQL server but could not find any error messages. No throttling alert triggered from SCOM either.

      We are using BizTalk 2010 in Windows 2008 R2 and SQL server 2008 R2.  Any suggestions are much appreciated.

 

Thanks,

David

---

David

Answer 1

Have you installed all cumulative updates for BizTalk Server 2010 ?

---

Christophe BRANDO...If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.

Answer 2

No, we have not.

---

David

Answer 3

There is an issue with BizTalk Server 2010 and SCOM

BizTalk Server 2010 CU2 solved this issue.

Hope this helps you

---

Christophe BRANDO...If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.

Answer 4

hi David,

it seems like i'm having the same issue as you but with biztalk server 2009.

just wondering were you able to solve the issue?

thanks

---

thanks ~steve

Answer 5

Hi Steve,

      No, we still cannot solve the problem. For now, we have script to restart all the hosts everyday.

David

---

David

Answer 6

We are experiencing the same problem (or at least it seems to be the same).

-BizTalk Server 2010

- The Problem occurs on both test (No SCOM) and production (SCOM)

-Single Server / Single Host / Single Host Instance

-2-3 days after restarting the host instance, it will stop Message Delivery but still Puslish Messages to the MsgBox (so the number of  running instances will increase and NOTHING is exiting the MsgBox again)

-NO throttling going on 

- Delivery state = 0

-Publish state = 0

- Thread Count approx = 180

-Nothing in the Event Viewer

As soon as we restart the host instance, Message Delivery will start!

We have not NOT installed any CU's I have to admit, since I often hear about new issues commencing when installing the CU's.

Morten la Cour

Answer 7

A little late in the game, but please check this

1. Sql Server Agent is running where the BizTalk databases are installed.

2. Job "MessageBox_DeadProcesses_Cleanup_BizTalkMsgBoxDb" is running where the BizTalk databases are installed. This particular job runs every minute. 

 From the description of this job, it has to say this:

This job detects when a host instance has stopped and releases all work locked by that host instance so it can be worked on by another host instance.

---

-Satya

Answer 8

Yes our SQL Server Agent is running, all jobs (backup, Purge Tracking) is configured, and there are no errors/warnings in the SQL Server Event Log!

Morten la Cour

Answer 9

Please, analyze when this issue is started.

I had similar issue. It were the new orchestrations with complex chained convoys. It created the deadlock in the MessageBox.

I've tried to solve this problem with Microsoft help, but finished only with redesign of these orchestrations. My guess is the orchestrations were too complex for the BizTalk.

BTW I mitigated this problem by switching off the tracking on the related artifacts.

---

Leonid Ganeline [BizTalk MVP] BizTalkien: Advanced Questions

Answer 10

Hi,

The orchestration is ready to run state means that the server is too busy or the resources are not available for orchestration to start. As you have confirmed that the load on server is very low and all the sql jobs are running properly i suggest you cross checked once. If all are running fine then check the Host.

Are all the host instances running are using same Host?

If yes then try to make three host one for orchestration, one for send port ,one for receive port.

If no, then check is the tracking is enable to that particular host instance?

If yes, then disable the tracking and create another host instance and enale tracking. Sometime, BizTalk uses a lot of resource while tracking messages. Kindly check this and reply.

Thanks

Answer 11

Hi,

Try and capture all the BizTalk related performance counters using PerfMon to generate a csv file so that we can generate an HTML (in easy readable format) to see if wverything is working fine. The ready to run state indeed means that server is too busy and the resources are not available for the respective instance to complete.

Please let me know if you need the exact steps to capture the performance counters.

Thanks,

Sumit

Answer 12

This might be a shot in the dark, but I just wanted to mention it since I've seen this happen once.

Have you checked your Anti virus or tried turning it off, I experienced with a user that after they installed antivirus BizTalk ended up putting all files in "Ready to run state".

Nevertheless, I would advice you to install all latest CU's, (Microsoft is going to say the same thing in case you open an incident towards them). Since there is a lot of problems that has been resolved with this, otherwise I hope you can upload some more information and report this as a bug towards Microsoft.

Best regards

Tord Glad Nordahl
Bouvet ASA, Norway
http://www.BizTalkAdmin.com | @tordeman

Please indicate ”Mark as Answer” if this post has answered the question.