Appfabric Autostart failing

Question

I have 2 application hosted in appfabric - one is a wf service with 4 xamlx workflows and the other is a wcf service.

After a reboot the event logs are filled with a number of errors (detailed below) and the applications fail to autostart. However, if you access the services (after a  delay as they load) - everything works properly.

I've set up two servers from scratch and autostart fails on both in the same way. Does anyone have any idea what's going wrong.

 

The errors are

Application Log

 

Error 1

eventid: 2281
description:  The worker process failed to initialize communication with the W3SVC and therefore could not be started. The data is the error.
data: 800700E7

 

 

Error 2

eventid: 2302
description: 

There was an error during processing of the managed application service auto-start for configuration path: 'MACHINE/WEBROOT/APPHOST/Default Web Site/[Service]. The error message returned is: 'An initialization error occurred while trying to preload an application.

Exception: System.TypeInitializationException

Message: The type initializer for 'System.Web.Compilation.CompilationLock' threw an exception.

StackTrace: at System.Web.Compilation.CompilationLock.GetLock(Boolean& gotLock)

at System.Web.Compilation.BuildManager.CheckTopLevelFilesUpToDate(StandardDiskBuildResultCache diskCache)

at System.Web.Compilation.BuildManager.Initialize()

at System.Web.Compilation.BuildManager.InitializeBuildManager()

at System.Web.HttpRuntime.HostingInit(HostingEnvironmentFlags hostingFlags, PolicyLevel policyLevel, Exception appDomainCreationException)

InnerException: System.InvalidOperationException

Message: Mutex could not be created.

StackTrace: at System.Web.Compilation.CompilationMutex..ctor(String name, String comment)

at System.Web.Compilation.CompilationLock..cctor()'. The worker process will be marked unhealthy and be shutdown. The data field contains the error code.

.
data: 80004005

 System Log

Error 3

eventid: 5011
description: A process serving application pool 'WorkflowV2' suffered a fatal communication error with the Windows Process Activation Service. The process id was '1684'. The data field contains the error number.
data: 80070218

 

Error 4

eventid: 5011
description: The AppFabric Workflow Management Service service hung on starting..

 

 Thanks

Simon

 

 

 

Answer 1

This turns out to be a bug in IIS that is going to be fixes in the next service release.

 

Simon

Answer 2

Hey I am hitting the same issue.  Do you know how you found out it is a AppFabric issue?  Is there a link or some MSDN notes on this?

Answer 3

Hi David,

Unfortunately I couldn't find any articles on this problem and so opened a support call with Microsoft. If you're interested it was case no 210120654515481002. You can probably raise a new call and associate it with that case if you want more information. There was no charge for the support case as it was recognised as a bug.

The conclusion was that it was too difficult to provide a hotfix as it involved some fundamental changes to IIS, but it would be fixed as part of the next service release to IIS (at some point in the future).

The problem was incredibly easy to reproduce and would go wrong 80% of the time on a clean install running a Microsoft sample wcf service application and so I suspect that if you are getting the event log messages  below you are experiencing the problem.

By the way, I started looking at this because the first access to a service running in appfabric, even with autostart on, was significantly slower than an equivalent wcf service running in a windows service.

One of the suggestions MS made was to switch off application pool recycling.

 

We spent several months working through the issue and this is an extract of the final email I received

Symptom

============

On a recycle, sometimes the worker process fails to spin up property and logs the following events in the event viewer. After this worker process crashes and a new one spins up and keeps on running though.

Microsoft-Windows-IIS-W3SVC-WP
Date: 12/14/2010 10:25:34 AM
Event ID: 2281
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: appserver4.ravensbay.co.uk
Description:
The worker process failed to initialize communication with the W3SVC and therefore could not be started. The data is the error.
[Error code in the event log is 800700E7 and this stands for ERROR_PIPE_BUSY]


System
Source: Microsoft-Windows-WAS
Date: 12/14/2010 10:25:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: appserver4.ravensbay.co.uk
Description:
A process serving application pool 'RouterApp' suffered a fatal communication error with the Windows Process Activation Service. The process id was '1840'. The data field contains the error number.
[Error code is 80070218 which stands for ERROR_PIPE_LISTENING]

 

Cause

========

This issue is caused due to timing issue between the IIS’s World Wide Web Publishing service and the IIS worker process and we have filed a bug for this against the IIS team.

 

Resolution

========== 

The Product group has agreed to fix this bug as a part of the next service pack release of the operating system.

 

I hope this helps

Simon

Answer 4

Since the last post in this thread is over 4 months old I hope I'm not too late,

W3SVC is depending on HTTP and WAS (check dependencies in "Administrative Tools, Services")
The "AppFabric Workflow Management Service" has no dependencies at all (but probably should have).
Services with startup type = "Automatic" but without depenencies are started alphabetically.

To overcome this I tried a work-around. I can easily do this because I work with a virtual machine so I'll allways have a fallback.

Change the startup type for "AppFabric Workflow Management Service" from "Automatic" into "Automatic (delayed start)" ("Administrative Tools, Services").
The OS will wait a short while (see http://www.sbsfaq.com/?p=2768) before starting these services.

Also do this for "Message Queuing" (MSMQ) otherwise it will fail to initialize properly (you may also manually start this service after a reboot, but that's more work).

The AppFabric autostart feature still doesn't for my service me but the conflict between AppFabric and W3SVC seems resolved so I can look for other causes

I'm running Windows Server 2008 R2 SP1 (x64) and AppFabric version SP5 QFE (file version 2.0.1493.5133).

Nico

Answer 5

Nico,

I came across your reply when hunting for another person's question. Do you think this issue might be resolved with a service dependency? You mention you think the AppFabric service needs dependencies. It is possible to add a dependency through the command-line: http://serverfault.com/questions/24821/how-to-add-dependency-on-a-windows-service-after-the-service-is-installed

Thanks,

Ben Cline

---

If this answers your question, please use the "Answer" button to say so | Ben Cline

Answer 6

Hi Nico,

I haven't looked at this for a while, but will follow your suggestion and get back to you. I found it really easy to reproduce and so hopefully shouldn't take too long - although I am snowed under with work at the moment.

One thing I would say, is that I also received these messages when recycling the application pools long after the server was booted and so I would be surprised if service dependencies would make a difference.

Simon