Bluescreen and applicationHost.config overwritten with zeroes

Question

User515338469 posted

I have an application which makes some changes to applicationHost.config. Every time I run this application, my system crashes with a bluescreen, and applicationHost.config is overwritten with zeroes.

I have been able to recover a recent valid applicationHost.config file from c:\inetpub\history and reproduce the problem every time I run my application.

This started happening after a Windows update that was installed yesterday (2019-10-10).

What exactly my application does is quite complicated. For example, it creates applications in IIS by running appcmd. Some modifications to the config are done by a .NET application, which uses Microsoft.Web.Administration. I also cannot tell at which exact point the crash happens, but I am working on getting more details.

System information:

Windows 10 1903 (it is a development machine)

Is this a known issue with the current Windows / IIS version?

What more information can I provide to get help, and what debugging steps would you recommend for me to do?

Answer 1

User515338469 posted

More research shows that my application also runs DISM to install IIS and IIS features, and that (according to logs) this may have been the last operation before the crash.

The DISM command-line is as follows:

dism /online /norestart /enable-feature /featurename:IIS-ApplicationDevelopment /featurename:IIS-ASPNET /featurename:IIS-ASPNET45 /featurename:IIS-CGI /featurename:IIS-CommonHttpFeatures /featurename:IIS-HealthAndDiagnostics /featurename:IIS-HttpCompressionDynamic /featurename:IIS-HttpErrors /featurename:IIS-HttpLogging /featurename:IIS-HttpRedirect /featurename:IIS-HttpTracing /featurename:IIS-IPSecurity /featurename:IIS-ISAPIExtensions /featurename:IIS-ISAPIFilter /featurename:IIS-LoggingLibraries /featurename:IIS-ManagementScriptingTools /featurename:IIS-NetFxExtensibility /featurename:IIS-NetFxExtensibility45 /featurename:IIS-Performance /featurename:IIS-RequestMonitor /featurename:IIS-RequestFiltering /featurename:IIS-Security /featurename:IIS-URLAuthorization /featurename:IIS-WebServer /featurename:IIS-WebServerManagementTools /featurename:IIS-WebServerRole /all

However, by running this command from the console, I cannot reproduce the crash.

Answer 2

User515338469 posted

DISM is innocent.

The crash is caused by a .NET app using Microsoft.Web.Administration to set some ISAPI and CGI Restriction rules. Given the current configuration, this app would not make any actual changes. Removing and adding the rules manually in IIS Manager does not cause a crash. Other operations using the same .NET app and Microsoft.Web.Administration do not cause crashes.

Answer 3

User515338469 posted

More shit has come to light:

The BSOD is caused by cldflt.sys, which according to various search results (often also related to BSOD situations) is a part of or used by Microsoft OneDrive.

The crash is caused by the second CommitChanges() call in this sequence:

1. 1. Remove ISAPI/CGI restriction
2. 2. CommitChanges()
3. 3. Add ISAPI/CGI restriction
4. 4. CommitChanges()

Another fun fact: When I ran this .NET application in the debugger in Visual Studio (which also reproduced the crash), the user settings file of the project was also overwritten with zeroes after the crash, causing Visual Studio to be unable to load the project until I deleted the .user file.

Answer 4

User690216013 posted

Where did you save your source code? It is strongly recommended that you don't save your code in OneDrive mapped folders as most of Microsoft development tools (IIS/Visual Studio) might not be tested to support such setup.

If you really want to back up your code periodically, learn to use a source code control system like Git.

Answer 5

User515338469 posted
Neither my source code, nor my VS project files, nor anything related to IIS or my application is stored in OneDrive.
This is why I don't understand why an API call to change the IIS configuration would cause a OneDrive driver to BSOD.

Answer 6

User-460007017 posted

Hi fw-flw,

Could you fix this issue by removing the update you installed in 10/10/2019? If this issue can be fixed by removing the update, We would know that some change applied in this update cause the crash.

It is recommended to restore the update as a workaround.

If you need to figure out the root cause. 

1.Please find the dump file generated when the server become BSOD. The location would be  C:\Winows\memory.dmp

2.We need to use dump analysis tool like WINDbg or Debug diagnostic tool to analyze the dump file

If the exception come from managed code, we need to figure out this issue come from native code or managed code. It will help us find the root cause.

If you are not expert in dump analysis, it is recommended to open a support ticket to https://support.microsoft.com/en-us.

Professional support engineer will help you handle this. If it is proved to be bug, they will help you report this issue and ask for a workaround or solution.

If the reply is helpful, it is appreciated if you could mark it as answer.

Best Regards,

Jokies Ding

Answer 7

User-2064283741 posted

I'm not sure this is an IIS issue.

Does changing the app host config by hand / appcmd result in this behavior?

Answer 8

User515338469 posted

Hi,

Changing the app host config by hand (using "IIS Manager" - the Microsoft one, not our app, which has the same name - see below) does not cause a crash.

Here is the output of WinDBG. I can also provide the complete memory dump, if it is still needed

(IISManager.exe is our app, which is a .NET 4.6.1 app that uses Microsoft.Web.Administration)

Also note that (as mentioned earlier), our IISManager.exe makes several changes to app host config BEFORE the change to ISAPI/CGI restrictions that do NOT cause a crash.

Microsoft (R) Windows Debugger Version 6.3.9600.17336 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Lab\MEMORY.DMP]
Kernel Bitmap Dump File: Only kernel address space is available


************* Symbol Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       https://msdl.microsoft.com/download/symbols
Symbol search path is: https://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 8 Kernel Version 18362 MP (12 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 18362.1.amd64fre.19h1_release.190318-1202
Machine Name:
Kernel base = 0xfffff801`7dc00000 PsLoadedModuleList = 0xfffff801`7e048210
Debug session time: Fri Oct 11 16:06:52.909 2019 (UTC + 2:00)
System Uptime: 0 days 0:35:56.657
Loading Kernel Symbols
......................................Page 20106bd67 too large to be in the dump file.
.........................
................................................................
................................................................
........................................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 000000e7`aa57c018).  Type ".hh dbgerr001" for details
Loading unloaded module list
................................
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 3B, {c0000005, fffff8017afcfbfe, ffffab8c3d1dd800, 0}

Probably caused by : cldflt.sys ( cldflt!HsmiFltPostECPCREATE+1da )

Followup: MachineOwner
---------

2: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff8017afcfbfe, Address of the instruction which caused the bugcheck
Arg3: ffffab8c3d1dd800, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

Debugging Details:
------------------


EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

FAULTING_IP: 
cldflt!HsmiFltPostECPCREATE+1da
fffff801`7afcfbfe f60201          test    byte ptr [rdx],1

CONTEXT:  ffffab8c3d1dd800 -- (.cxr 0xffffab8c3d1dd800;r)
rax=ffffc18df636fae8 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000014 rsi=ffffab8c3d1ded60 rdi=ffffc18dde39b720
rip=fffff8017afcfbfe rsp=ffffab8c3d1de1f0 rbp=ffffab8c3d1de240
 r8=0000000000000000  r9=7fffc18dde39b7a0 r10=fffff8017dc663d0
r11=ffffab8c3d1de1e0 r12=0000000000000014 r13=0000000000000000
r14=0000000000000000 r15=fffff8017af8d000
iopl=0         nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00050246
cldflt!HsmiFltPostECPCREATE+0x1da:
fffff801`7afcfbfe f60201          test    byte ptr [rdx],1 ds:002b:00000000`00000014=??
Last set context:
rax=ffffc18df636fae8 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000014 rsi=ffffab8c3d1ded60 rdi=ffffc18dde39b720
rip=fffff8017afcfbfe rsp=ffffab8c3d1de1f0 rbp=ffffab8c3d1de240
 r8=0000000000000000  r9=7fffc18dde39b7a0 r10=fffff8017dc663d0
r11=ffffab8c3d1de1e0 r12=0000000000000014 r13=0000000000000000
r14=0000000000000000 r15=fffff8017af8d000
iopl=0         nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00050246
cldflt!HsmiFltPostECPCREATE+0x1da:
fffff801`7afcfbfe f60201          test    byte ptr [rdx],1 ds:002b:00000000`00000014=??
Resetting default scope

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  0x3B

PROCESS_NAME:  IISManager.exe

CURRENT_IRQL:  0

ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre

LAST_CONTROL_TRANSFER:  from fffff8017afd0129 to fffff8017afcfbfe

STACK_TEXT:  
ffffab8c`3d1de1f0 fffff801`7afd0129 : ffffc18d`f636fae8 ffffab8c`3d1de360 00000000`00000000 00000000`00000000 : cldflt!HsmiFltPostECPCREATE+0x1da
ffffab8c`3d1de280 fffff801`81d03c03 : ffffc18d`f636fae8 ffffab8c`3d1de360 ffffc18d`f636fa00 00000000`00016bf0 : cldflt!HsmFltPostQUERY_OPEN+0x29
ffffab8c`3d1de310 fffff801`81d0243c : 00000000`00000000 ffffc18d`dae1dd00 ffffc18d`f4c87268 00000000`00000000 : FLTMGR!FltpPerformPostCallbacks+0x3e3
ffffab8c`3d1de3e0 fffff801`7dc89aac : ffffab8c`3d1de480 ffffab8c`3d1ded0c ffffc18d`dac1e8f0 ffffc18d`ee3e6010 : FLTMGR!FltpPostFsFilterOperation+0x2c
ffffab8c`3d1de410 fffff801`7e45010d : 00000000`00000000 ffffc18d`dae1ddc0 ffffab8c`3d1de540 fffff801`829ddda0 : nt!FsFilterPerformCompletionCallbacks+0x4c
ffffab8c`3d1de440 fffff801`7e3ead94 : 00000000`6d4e6f49 fffff801`7df6f06d ffffab8c`00000003 00000000`00000000 : nt!FsRtlQueryOpen+0xd1
ffffab8c`3d1de710 fffff801`7e1e62ba : fffff801`00000007 fffff801`7e1e5944 ffffab8c`3d1de950 00000000`00000000 : nt!IopQueryInformation+0x139ad4
ffffab8c`3d1de770 fffff801`7e1ecfcf : ffffc18d`dac1e8f0 ffffc18d`dac1e844 ffffc18d`f17f3010 00000000`00000000 : nt!IopParseDevice+0x8ea
ffffab8c`3d1de8e0 fffff801`7e1eb431 : ffffc18d`f17f3000 ffffab8c`3d1deb28 ffffc18d`00000240 ffffc18d`cfcfe640 : nt!ObpLookupObjectName+0x78f
ffffab8c`3d1deaa0 fffff801`7e457ec3 : 00000000`00000001 00000000`00000000 ffffab8c`3d1df090 ffffab8c`3d1deef8 : nt!ObOpenObjectByNameEx+0x201
ffffab8c`3d1debe0 fffff801`81d18063 : ffffab8c`3d1df000 ffffc18d`ed6ad9f0 ffffc18d`dd104a30 fffff801`81d076fb : nt!IoQueryInformationByName+0x263
ffffab8c`3d1dee90 fffff801`7af85c99 : ffffab8c`3d1df088 00000000`00000000 ffffab8c`3d1df088 fffff801`7dc6b455 : FLTMGR!FltQueryInformationByName+0x153
ffffab8c`3d1def40 fffff801`7af77924 : ffffab8c`3d1df088 00000000`00000000 00000000`00000000 00000000`00000000 : cldflt!FltQueryInformationByNameCallout+0x49
ffffab8c`3d1def90 fffff801`7afcf77d : 00000000`00000000 ffffab8c`3d1e0000 ffffab8c`3d1d9000 ffffc18d`ed6ad9f0 : cldflt!HsmExpandKernelStackAndCallout+0x44
ffffab8c`3d1defd0 fffff801`7afd0019 : ffffffff`0000ffff ffffc18d`f6666b38 ffffc18d`ed6adc80 ffffab8c`3d1df219 : cldflt!HsmiFltPreECPCREATE+0x34d
ffffab8c`3d1df140 fffff801`81d04a5d : ffffc18d`f66669b0 00000000`00000000 00000000`00000000 00000000`00000000 : cldflt!HsmFltPreCREATE+0x9
ffffab8c`3d1df170 fffff801`81d045a0 : ffffab8c`3d1df2f0 ffffab8c`3d1df300 00000000`00000000 00000000`00000000 : FLTMGR!FltpPerformPreCallbacks+0x2fd
ffffab8c`3d1df280 fffff801`81d3cd13 : fffff801`81d29060 00000000`00000090 00000000`00000000 00000000`000003a4 : FLTMGR!FltpPassThroughInternal+0x90
ffffab8c`3d1df2b0 fffff801`7dc31f39 : 00000000`00000000 fffff801`7e1e5905 00000000`00000000 00000000`00000000 : FLTMGR!FltpCreate+0x2f3
ffffab8c`3d1df360 fffff801`7dc30fe4 : 00000000`00000003 00000000`00000000 00000000`00000000 fffff801`7dc317a3 : nt!IofCallDriver+0x59
ffffab8c`3d1df3a0 fffff801`7e1e5ffb : ffffab8c`3d1df660 fffff801`7e1e5905 ffffab8c`3d1df5d0 ffffc18d`f63584e0 : nt!IoCallDriverWithTracing+0x34
ffffab8c`3d1df3f0 fffff801`7e1ecfcf : ffffc18d`dac1e8f0 ffffc18d`dac1e805 ffffc18d`f3973260 00000000`00000001 : nt!IopParseDevice+0x62b
ffffab8c`3d1df560 fffff801`7e1eb431 : ffffc18d`f3973200 ffffab8c`3d1df7a8 00000000`00000040 ffffc18d`cfcfe640 : nt!ObpLookupObjectName+0x78f
ffffab8c`3d1df720 fffff801`7e230300 : 00000000`00000001 000000e7`aa3dd4c8 00000000`00000001 00000000`00000000 : nt!ObOpenObjectByNameEx+0x201
ffffab8c`3d1df860 fffff801`7e22fac9 : 000000e7`aa3dd470 00000004`c0100080 000000e7`aa3dd4c8 000000e7`aa3dd488 : nt!IopCreateFile+0x820
ffffab8c`3d1df900 fffff801`7ddd2b15 : 00000000`00000000 00000000`00000000 00000000`00000000 000000e7`aa3dcb98 : nt!NtCreateFile+0x79
ffffab8c`3d1df990 00007ffe`7733cb64 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
000000e7`aa3dd3f8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffe`7733cb64


FOLLOWUP_IP: 
cldflt!HsmiFltPostECPCREATE+1da
fffff801`7afcfbfe f60201          test    byte ptr [rdx],1

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  cldflt!HsmiFltPostECPCREATE+1da

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: cldflt

IMAGE_NAME:  cldflt.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  0

STACK_COMMAND:  .cxr 0xffffab8c3d1dd800 ; kb

BUCKET_ID_FUNC_OFFSET:  1da

FAILURE_BUCKET_ID:  0x3B_cldflt!HsmiFltPostECPCREATE

BUCKET_ID:  0x3B_cldflt!HsmiFltPostECPCREATE

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x3b_cldflt!hsmifltpostecpcreate

FAILURE_ID_HASH:  {ff49d151-c405-fda0-3953-8b48357a52e0}

Followup: MachineOwner
---------

2: kd> lmvm cldflt
start             end                 module name
fffff801`7af70000 fffff801`7afe7000   cldflt     (pdb symbols)          C:\ProgramData\dbg\sym\cldflt.pdb\0698036E8827B2FF6ECB6676372B81FC1\cldflt.pdb
    Loaded symbol image file: cldflt.sys
    Image path: \SystemRoot\system32\drivers\cldflt.sys
    Image name: cldflt.sys
    Timestamp:        ***** Invalid (B7D0F1F2)
    CheckSum:         00079A82
    ImageSize:        00077000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Answer 9

User-460007017 posted

Hi fw-flw,

cldflt.sys belong to cloud file mini filter driver. Since the error message is thrown from native code. Could you fix this issue by rolling back these updates? I believe this issue is a compatibility issue.

I'm afraid you could only accept rolling back update as a workaround. 

If you need to fix this, You may need to support a ticket to Microsoft since PG need to collect business impact.

If the reply is helpful, it is appreciated if you could mark the reply as answer.

Best Regards,

Jokies Ding

Answer 10

User515338469 posted

Could you please give me a direct link to a website where I can submit a support ticket to Microsoft? I am finding it difficult to get past all the bullshit (such as knowledge bases, chatbots, etc.) Microsoft set up to keep people from submitting support tickets. This is obviously something that needs to be looked at by a human with some expertise and access to Microsoft developers.

Otherwise, thanks for the help so far. As a workaround, we are now doing what we should have been doing for a while, which is replacing our home-brew IIS management application with appcmd.

Answer 11

User-460007017 posted

Hi fw-flw,

If you need to contact Professional Microsoft support engineer, you could create a business request ticket from here:

https://support.microsoft.com/en-us/supportforbusiness/productselection

Best Regards,

Jokies Ding

Answer 12

User515338469 posted

Thank you for your help.

I have marked your reply as answer. We will get in touch with Microsoft support.