none
Application Crash: An ASP.NET via COM Interop crashes w3wp with first chance exception in IIS and heap corruption RRS feed

  • Question

  • Hi,

    Please see the following article: http://forums.iis.net/p/1166382/1939856.aspx

    I have applied latest security patches as of today on Windows Server 2003 SP2 and .NET 1.0, 1.1, 2.0, 3.0, 3.5 the issue is still not resolved however i believe it is .NET framework related.

    Apartment threaded COM DLL using IDistpatch late bound called XCSSParser uses MSXML 4's LoadXML method is causing above issue and heap corruption.

    Scenario is very simple and reproducable, also on UAT:

    1. Add a Class Library project having a direct reference from the registered COM object and a method using this:

            public string SaveXML2CSS(XmlDocument xml, string fullPath)
            {
                XML2CSSClass xml2css = null;
                //object comObject = null;
                string ErrorMessage = "";
                if (this.disposed)
                {
                    ErrorMessage = "Error: Object Disposed";
                    return ErrorMessage;
                }
    
                try
                {
                    xml2css = new XML2CSSClass();
                    if (xml2css != null)
                    {
                        string sxml = xml.InnerXml;
                        xml2css.LoadXML(sxml); //crashes here
                        string CSS = xml2css.CSS;
                        int ErrorCode = xml2css.ErrorCode;
                        ErrorMessage = xml2css.ErrorMessage;
                    }
                    //comObject = AppDomain.CurrentDomain.CreateInstanceFromAndUnwrap(AppDomain.CurrentDomain.SetupInformation.PrivateBinPath + "\\Interop.XCSSParser.dll", "XCSSParser.XML2CSSClass");
                    //if (comObject != null)
                    //{
                    //    // from the unwrapped object, get the type definition 
                    //    Type type = comObject.GetType();
    
                    //    // method parameters are passed in an object array. 
                    //    // Must set these up before invocation 
                    //    object[] oParms = new Object[] { xml.InnerXml };
    
                    //    // Invoke the method and cast the result to the correct 
                    //    // object type. 
                    //    type.InvokeMember("LoadXML", BindingFlags.InvokeMethod, null, comObject, oParms);
    
                    //    PropertyInfo pi = type.GetProperty("CSS");
                    //    string CSS = (string)pi.GetValue(comObject, null);
    
                    //    pi = type.GetProperty("ErrorMessage");
                    //    ErrorMessage = (string)pi.GetValue(comObject, null);
    
                    //    oParms = null;
                    //    oParms = new Object[] { fullPath, false };
                    //    type.InvokeMember("SaveAs", BindingFlags.InvokeMethod, null, comObject, oParms);
    
                    //    pi = type.GetProperty("ErrorMessage");
                    //    ErrorMessage = (string)pi.GetValue(comObject, null);
                    //}
                }
                finally
                {
                    //if(comObject!=null)
                    //    while (Marshal.ReleaseComObject(comObject) > 0) { }
                    if(xml2css!=null)
                        while (Marshal.ReleaseComObject(xml2css) > 0) { }
                }
                return ErrorMessage;
            }
    

    The XML message is 131072 bytes and 6187 xml elements. If i remove 1 element i get it working without issues.

    The XML message is 131072 bytes and 6187 xml elements. If i remove 1 element i get it working without issues but not with more elements. Is there any limitation recently introduced?

    2. Add a web site project with Default page and a Button1:

    using System;
    using System.Data;
    using System.Configuration;
    using System.Web;
    using System.Web.Security;
    using System.Web.UI;
    using System.Web.UI.WebControls;
    using System.Web.UI.WebControls.WebParts;
    using System.Web.UI.HtmlControls;
    using System.Xml;
    
    using COMWrapper;
    
    public partial class _Default : System.Web.UI.Page 
    {
        protected void Page_Load(object sender, EventArgs e)
        {
    
        }
        protected void Button1_Click(object sender, EventArgs e)
        {
            XmlDocument xml = new XmlDocument();
            xml.Load("d:\\temp\\css.xml");
            try
            {
                using (XCSSWrapper x = new XCSSWrapper())
                {
                    x.SaveXML2CSS(xml, "");
                }
            }
            catch (Exception e)
            { 
                //0x80010405 RPC_E_SERVERFAULT
            }
        }
    }

     The first it generates an RPC_E_SERVERFAULT exception and the last thing from .NET stack is mscorlib DemandSpecialSecurity. What special security is now required to call a COM object from .NET is that not possible if it is a late bound, may be it does request more heap or paging and have no access to OS resources?

    After the special security i see 3 times DllUnregisterServer and some Unk (Unknown) address. DebugDiag suggests that the COM object created a stack overflow. However if i call the same thing from a Console application or logged on user works fine, also works fine from WebDev.WebServer.exe. ProcessMonitor does not find anything obvious, however it does find some buffer overflows on different InprocServer32 keys which are now encrypted or assmeblified? Could it be that now this gets encrypted and some other DLLs still reading it unecrypted? Anyways, im not sure where the issue could be, but somehow it prevents the execution.

    Now if i go back second time to the default page and press the button, on LoadXML function this time, crashes the application with access violation.

    What is now the correct way of calling COM objects from ASP.NET or this should not be possible anymore?

    Thanks & Regards,

    Constantin

    Tuesday, April 13, 2010 12:27 AM

All replies

  • This is likely some native code or wrong PInvoke trashing memory. If you corrupt "the right" memory, it will crash. That's IMO why you are seeing it only ASP.NET. Try to avoid all suspicious code which manipulates with native memory and verify all PInvokes.

    These things in your code are suspicious:
    if (this.disposed) ... Does it mean that the object can be finalized?
    while (Marshal.ReleaseComObject(xml2css) > 0) { } ... I am not COM expert, but this doesn't feel right - see msdn docs and this blog post. Maybe I am wrong and this is correct ...

    You can: Try to get rid of all suspicious code (at least as a test) and see if you still have the crash.
    Or turn on PageHeap and GCStress (as adviced here) in your console application, maybe they will crash it and you can diagnose it there.

    -Karel

    Tuesday, April 13, 2010 5:11 AM
    Moderator
  • Hi,

    Thanks for the links i have Tess blog in my Favorites :). Ok, I wrapped it in a disaposable object just to test if the garbige collector would dispose this during the call. The issue occurs with this function outside of a disposable wrapper and without Marshal.ReleaseComObject. As you see in the commented code i have also tested with Invoke to see if it makes any difference. The native stack looks different but it breaks the same way in a call to oleaut32.dll. The same code work with the same xml message works fine from Console application. I cleaned that up to have just the code like below and it breaks as well, so I dont think it has to do with the garbage collector at least or Marshal function:

                    xml2css = new XML2CSSClass();
                    string sxml = xml.InnerXml;
                    xml2css.LoadXML(sxml); //crashes here

    Here is the auto generated code from the interop:

    [ComImport, Guid("F12F8ABE-354A-4F95-8915-C3F397A3E782"), TypeLibType((short) 2), ClassInterface((short) 0)]
    public class XML2CSSClass : IXML2CSS, XML2CSS
    {
        // Methods
        [MethodImpl(MethodImplOptions.InternalCall, MethodCodeType=MethodCodeType.Runtime), DispId(2)]
        public virtual extern void LoadXML([In, MarshalAs(UnmanagedType.BStr)] string sXML);
    ...
    } 

    I'm not sure what this DLL does since its third party freeware, but my guess it is using MSXML4's LoadXML intrnally looking at the ProcessMonitor on my console application.

    Here is the first exception where the problem comes from which is Stack Overflow:

    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (14ec.6a4): Stack overflow - code c00000fd (first/second chance not available)
    eax=00001370 ebx=02eefad0 ecx=02eb2608 edx=09730069 esi=0001f1b7 edi=096a4698
    eip=09365587 esp=02eef600 ebp=02eef610 iopl=0         nv up ei pl nz na po nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for XCSSParser.dll -
    XCSSParser!DllUnregisterServer+0x2886:
    09365587 8501            test    dword ptr [ecx],eax  ds:0023:02eb2608=00000000
    0:005> .reload
    ................................................................
    ..................................
    Loading unloaded module list
    .
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for XCSSParser.dll -
    0:005> !load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll
    0:005> !threadpool
    CPU utilization 81%
    Worker Thread: Total: 3 Running: 1 Idle: 2 MaxLimit: 200 MinLimit: 2
    Work Request in Queue: 0
    --------------------------------------
    Number of Timers: 7
    --------------------------------------
    Completion Port Thread:Total: 2 Free: 2 MaxFree: 4 CurrentLimit: 1 MaxLimit: 200 MinLimit: 2
    0:005> !threads
    ThreadCount: 9
    UnstartedThread: 0
    BackgroundThread: 9
    PendingThread: 0
    DeadThread: 0
    Hosted Runtime: no
                                          PreEmptive   GC Alloc           Lock
           ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
      16    1 125c 06721630   1808230 Enabled  00000000:00000000 07406d20     1 MTA (Threadpool Worker)
      20    2 16a0 06734be0      b230 Enabled  00000000:00000000 0671bd78     0 MTA (Finalizer)
      21    3  c88 073edc60    80a230 Enabled  00000000:00000000 0671bd78     0 MTA (Threadpool Completion Port)
      14    4  e54 073f59d8   880a230 Enabled  00000000:00000000 0671bd78     0 MTA (Threadpool Completion Port)
      22    5 1280 07406090      1230 Enabled  00000000:00000000 0671bd78     0 Ukn
      23    6  294 07868bc8   180b230 Enabled  00000000:00000000 0671bd78     0 MTA (Threadpool Worker)
      25    7  f80 066c7ae0    80a230 Enabled  00000000:00000000 0671bd78     0 MTA (Threadpool Completion Port)
      26    8 16c8 066fbb40   180b230 Enabled  00000000:00000000 0671bd78     0 MTA (Threadpool Worker)
      27    9 11c8 066f1a28   880b230 Enabled  00000000:00000000 0671bd78     0 MTA (Threadpool Completion Port)

    0:005> ~*e!clrstack
    OS Thread Id: 0xe54 (14)
    Failed to start stack walk: 80004005
    OS Thread Id: 0x125c (16)
    ESP       EIP    
    06c3f234 7c82860c [ComPlusMethodFrameStandaloneCleanup: 06c3f234] Interop.XCSSParser.XML2CSSClass.LoadXML(System.String)
    06c3f244 08f5274d COMWrapper.XCSSWrapper.SaveXML2CSS(System.Xml.XmlDocument, System.String)
    06c3f2a8 08f5261c COMWrapper.XMLUtils.SaveXML2CSS(System.Xml.XmlDocument, System.String)
    06c3f2f0 08f50a53 COMWrapper.CSSEditor.FetchDefaultCSSFile()
    06c3f334 08f50770 _Default.Button1_Click(System.Object, System.EventArgs)
    06c3f350 66799ec8 System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
    06c3f368 66799d2f System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
    06c3f380 66799f6b System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
    06c3f388 66095d9e System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
    06c3f394 66095cf5 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
    06c3f3a8 660a7f1e System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
    06c3f500 660a77a4 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
    06c3f538 660a76d1 System.Web.UI.Page.ProcessRequest()
    06c3f570 660a7666 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
    06c3f57c 660a7642 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
    06c3f590 08f501fe ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
    06c3f5a0 660adb16 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    06c3f5d4 6608132c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
    06c3f614 6608c5c3 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
    06c3f664 660808ac System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
    06c3f680 66083e1c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
    06c3f6b4 66083ac3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
    06c3f6c4 66082c5c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
    06c3f8d8 79f9811e [ContextTransitionFrame: 06c3f8d8]
    06c3f90c 79f9811e [GCFrame: 06c3f90c]
    06c3fa68 79f9811e [ComMethodFrame: 06c3fa68]
    OS Thread Id: 0x16a0 (20)
    Failed to start stack walk: 80004005
    OS Thread Id: 0xc88 (21)
    Failed to start stack walk: 80004005
    OS Thread Id: 0x1280 (22)
    Failed to start stack walk: 80004005
    OS Thread Id: 0x294 (23)
    Failed to start stack walk: 80004005
    OS Thread Id: 0xf80 (25)
    Failed to start stack walk: 80004005
    OS Thread Id: 0x16c8 (26)
    Failed to start stack walk: 80004005
    OS Thread Id: 0x11c8 (27)
    Failed to start stack walk: 80004005

    0:005> ~*kvn

       0  Id: 14ec.1100 Suspend: 1 Teb: 7ffdf000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0006fc08 7c827d29 77e61d1e 00000178 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0006fc0c 77e61d1e 00000178 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 0006fc7c 77e61c8d 00000178 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 0006fc90 5a364692 00000178 ffffffff 00000000 kernel32!WaitForSingleObject+0x12 (FPO: [2,0,0])
    04 0006fca0 5a366f27 00467e78 5a3af3dd 00000000 w3dt!WP_CONTEXT::RunMainThreadLoop+0x10 (FPO: [0,0,0])
    05 0006fca8 5a3af3dd 00000000 64711dcf 00000000 w3dt!UlAtqStartListen+0x2d (FPO: [0,0,0])
    06 0006fcb8 5a3bc315 01001418 010013e4 010012d0 w3core!W3_SERVER::StartListen+0xbd (FPO: [0,0,0])
    07 0006ff0c 0100187c 00000007 00464bd0 00000000 w3core!UlW3Start+0x26e (FPO: [3,144,4])
    08 0006ff44 01001a27 00000007 00464bd0 00466e18 w3wp!wmain+0x22a (FPO: [2,6,4])
    09 0006ffc0 77e6f23b 00000000 00000000 7ffd8000 w3wp!wmainCRTStartup+0x12f (FPO: [SEH])
    0a 0006fff0 00000000 010018f8 00000000 78746341 kernel32!BaseProcessStart+0x23 (FPO: [SEH])

       1  Id: 14ec.600 Suspend: 1 Teb: 7ffde000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 00b5fea0 7c827d19 7c80e5bb 00000002 00b5fef0 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 00b5fea4 7c80e5bb 00000002 00b5fef0 00000001 ntdll!NtWaitForMultipleObjects+0xc (FPO: [5,0,0])
    02 00b5ff48 7c80e4a2 00000002 00b5ff70 00000000 ntdll!EtwpWaitForMultipleObjectsEx+0xf7 (FPO: [SEH])
    03 00b5ffb8 77e6482f 00000000 00000000 00000000 ntdll!EtwpEventPump+0x27f (FPO: [1,18,0])
    04 00b5ffec 00000000 7c80e1fa 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

       2  Id: 14ec.1408 Suspend: 1 Teb: 7ffdd000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 02e1ff9c 7c826f69 7c83d454 00000001 02e1ffb0 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 02e1ffa0 7c83d454 00000001 02e1ffb0 00000000 ntdll!NtDelayExecution+0xc (FPO: [2,0,0])
    02 02e1ffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpTimerThread+0x47 (FPO: [1,2,0])
    03 02e1ffec 00000000 7c83d40d 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

       3  Id: 14ec.1038 Suspend: 1 Teb: 7ffdc000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 02e5ff70 7c8277f9 7c839f68 000000e4 02e5ffb0 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 02e5ff74 7c839f68 000000e4 02e5ffb0 02e5ffb4 ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 02e5ffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x3d (FPO: [1,7,0])
    03 02e5ffec 00000000 7c839f2b 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

       4  Id: 14ec.8ac Suspend: 1 Teb: 7ffdb000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 02eafe18 7c827859 77c885ac 00000130 02eaff74 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 02eafe1c 77c885ac 00000130 02eaff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc (FPO: [5,0,0])
    02 02eaff84 77c88792 02eaffac 77c8872d 001d1808 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198 (FPO: [0,14,0])
    03 02eaff8c 77c8872d 001d1808 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd (FPO: [1,0,0])
    04 02eaffac 77c7b110 001a4cd8 02eaffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d (FPO: [1,2,4])
    05 02eaffb8 77e6482f 001d1c60 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b (FPO: [1,0,0])
    06 02eaffec 00000000 77c7b0f5 001d1c60 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

    #  5  Id: 14ec.6a4 Suspend: 1 Teb: 7ffda000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00 02eef610 0936150c 09730068 096a46a0 09362c26 XCSSParser!DllUnregisterServer+0x2886
    01 02eef63c 09363e50 09730068 02eef67c 096a4698 XCSSParser+0x150c
    02 02eef660 09363b83 09730068 02eefa84 09375cd8 XCSSParser!DllUnregisterServer+0x114f
    03 02eef678 77c80193 096a4698 096a4738 02020202 XCSSParser!DllUnregisterServer+0xe82
    04 02eef694 77ce33e1 09363b52 02eef878 00000002 rpcrt4!Invoke+0x30
    05 02eefa94 77ce1968 07a4a570 07389a24 066e5a68 rpcrt4!NdrStubCall2+0x299 (FPO: [SEH])
    06 02eefaec 7d0e5684 07a4a570 066e5a68 07389a24 rpcrt4!CStdStubBuffer_Invoke+0x3f (FPO: [SEH])
    07 02eefb10 7778d01b 07a4a570 066e5a68 07389a24 oleaut32!CUnivStubWrapper::Invoke+0xc5 (FPO: [3,1,4])
    08 02eefb54 7778cfc8 066e5a68 073811b4 001dbc10 ole32!SyncStubInvoke+0x37 (FPO: [SEH])
    09 02eefb9c 776c120b 066e5a68 0738a218 073a2c70 ole32!StubInvoke+0xa7 (FPO: [6,7,4])
    0a 02eefc78 776c0bf5 07389a24 00000000 073a2c70 ole32!CCtxComChnl::ContextInvoke+0xec (FPO: [5,44,4])
    0b 02eefc94 776bc455 066e5a68 00000001 073a2c70 ole32!MTAInvoke+0x1a (FPO: [6,0,0])
    0c 02eefcc0 7778ced5 066e5a68 00000001 073a2c70 ole32!STAInvoke+0x48 (FPO: [7,2,4])
    0d 02eefcf4 7778cd66 d0908070 07389a24 073a2c70 ole32!AppInvoke+0xa3 (FPO: [7,1,4])
    0e 02eefdc8 7778c24d 066e5a10 001d2da0 00000000 ole32!ComInvokeWithLockAndIPID+0x2c5 (FPO: [2,41,4])
    0f 02eefdf0 776bc344 066e5a10 00000400 0674f2b8 ole32!ComInvoke+0xca (FPO: [1,3,4])
    10 02eefe04 776bc30f 066e5a10 02eefe94 776bc27c ole32!ThreadDispatch+0x23 (FPO: [1,0,0])
    11 02eefe1c 7739b6e3 001d77d8 00000400 0000babe ole32!ThreadWndProc+0xfe (FPO: [4,0,4])
    12 02eefe48 7739b874 776bc27c 0003209a 00000400 user32!InternalCallWinProc+0x28
    13 02eefec0 7739ba92 00000000 776bc27c 0003209a user32!UserCallWinProcCheckWow+0x151 (FPO: [SEH])

       6  Id: 14ec.94c Suspend: 1 Teb: 7ffd9000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 036aff24 7c8277f9 77e5bea2 0000016c 036aff80 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 036aff28 77e5bea2 0000016c 036aff80 036aff6c ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 036aff54 5a30248e 0000016c 036aff7c 036aff80 kernel32!GetQueuedCompletionStatus+0x29 (FPO: [5,4,0])
    03 036aff8c 5a3026ac 00000000 0046bc28 5a300000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x33 (FPO: [0,4,4])
    04 036affa0 5a301da9 0046baf0 00000000 00000000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x24 (FPO: [1,0,4])
    05 036affb8 77e6482f 0046bc28 00000000 00000000 w3tp!THREAD_MANAGER::ThreadManagerThread+0x39 (FPO: [1,0,0])
    06 036affec 00000000 5a301d70 0046bc28 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

       7  Id: 14ec.100c Suspend: 1 Teb: 7ffd7000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 036eff24 7c8277f9 77e5bea2 0000016c 036eff80 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 036eff28 77e5bea2 0000016c 036eff80 036eff6c ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 036eff54 5a30248e 0000016c 036eff7c 036eff80 kernel32!GetQueuedCompletionStatus+0x29 (FPO: [5,4,0])
    03 036eff8c 5a3026ac 00000000 0046bc78 5a300000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x33 (FPO: [0,4,4])
    04 036effa0 5a301da9 0046baf0 00000000 00000000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x24 (FPO: [1,0,4])
    05 036effb8 77e6482f 0046bc78 00000000 00000000 w3tp!THREAD_MANAGER::ThreadManagerThread+0x39 (FPO: [1,0,0])
    06 036effec 00000000 5a301d70 0046bc78 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

       8  Id: 14ec.34c Suspend: 1 Teb: 7ffd6000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0372ff24 7c8277f9 77e5bea2 0000016c 0372ff80 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0372ff28 77e5bea2 0000016c 0372ff80 0372ff6c ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 0372ff54 5a30248e 0000016c 0372ff7c 0372ff80 kernel32!GetQueuedCompletionStatus+0x29 (FPO: [5,4,0])
    03 0372ff8c 5a3026ac 00000000 0046bcc8 5a300000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x33 (FPO: [0,4,4])
    04 0372ffa0 5a301da9 0046baf0 00000000 00000000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x24 (FPO: [1,0,4])
    05 0372ffb8 77e6482f 0046bcc8 00000000 00000000 w3tp!THREAD_MANAGER::ThreadManagerThread+0x39 (FPO: [1,0,0])
    06 0372ffec 00000000 5a301d70 0046bcc8 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

       9  Id: 14ec.a88 Suspend: 1 Teb: 7ffd5000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0376ff24 7c8277f9 77e5bea2 0000016c 0376ff80 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0376ff28 77e5bea2 0000016c 0376ff80 0376ff6c ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 0376ff54 5a30248e 0000016c 0376ff7c 0376ff80 kernel32!GetQueuedCompletionStatus+0x29 (FPO: [5,4,0])
    03 0376ff8c 5a3026ac 00000000 0046bd18 5a300000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x33 (FPO: [0,4,4])
    04 0376ffa0 5a301da9 0046baf0 00000000 00000000 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x24 (FPO: [1,0,4])
    05 0376ffb8 77e6482f 0046bd18 00000000 00000000 w3tp!THREAD_MANAGER::ThreadManagerThread+0x39 (FPO: [1,0,0])
    06 0376ffec 00000000 5a301d70 0046bd18 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      10  Id: 14ec.fb4 Suspend: 1 Teb: 7ffd4000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 03a2fcec 7c827d19 7c83c7be 00000003 03a2fd34 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 03a2fcf0 7c83c7be 00000003 03a2fd34 00000001 ntdll!NtWaitForMultipleObjects+0xc (FPO: [5,0,0])
    02 03a2ffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWaitThread+0x161 (FPO: [1,168,0])
    03 03a2ffec 00000000 7c83c673 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      11  Id: 14ec.f10 Suspend: 1 Teb: 7ffaf000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 060ffa84 7c827d29 77e61d1e 000001c4 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 060ffa88 77e61d1e 000001c4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 060ffaf8 77e61c8d 000001c4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 060ffb0c 5a3b81ef 000001c4 ffffffff 00000000 kernel32!WaitForSingleObject+0x12 (FPO: [2,0,0])
    04 060fffb8 77e6482f 00000000 00000000 00000000 w3core!HTTP_COMPRESSION::CompressionThread+0x126 (FPO: [1,292,0])
    05 060fffec 00000000 5a3b80c9 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      12  Id: 14ec.10e4 Suspend: 1 Teb: 7ffae000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0613fe18 7c827859 77c885ac 00000130 0613ff74 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0613fe1c 77c885ac 00000130 0613ff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc (FPO: [5,0,0])
    02 0613ff84 77c88792 0613ffac 77c8872d 001d1808 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198 (FPO: [0,14,0])
    03 0613ff8c 77c8872d 001d1808 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd (FPO: [1,0,0])
    04 0613ffac 77c7b110 001a4cd8 0613ffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0x9d (FPO: [1,2,4])
    05 0613ffb8 77e6482f 00273eb8 00000000 00000000 rpcrt4!ThreadStartRoutine+0x1b (FPO: [1,0,0])
    06 0613ffec 00000000 77c7b0f5 00273eb8 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      13  Id: 14ec.92c Suspend: 1 Teb: 7ffad000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0617ff8c 7c826f69 7c81943a 00000001 0617ffac ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0617ff90 7c81943a 00000001 0617ffac 00000000 ntdll!NtDelayExecution+0xc (FPO: [2,0,0])
    02 0617ffb8 77e6482f 00278980 00000000 00000000 ntdll!RtlpIOWorkerThread+0x3f (FPO: [1,3,0])
    03 0617ffec 00000000 7c8193fb 00278980 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      14  Id: 14ec.e54 Suspend: 1 Teb: 7ffac000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 06bbfe74 7c8277f9 77e5bea2 0000023c 06bbfefc ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 06bbfe78 77e5bea2 0000023c 06bbfefc 06bbfebc ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 06bbfea4 79f5631c 0000023c 06bbfef4 06bbfefc kernel32!GetQueuedCompletionStatus+0x29 (FPO: [5,4,0])
    03 06bbff14 79f27d7b 00000000 00000001 00000000 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x141 (FPO: [1,17,4])
    04 06bbffb8 77e6482f 066e2da0 00000000 00000000 mscorwks!ThreadpoolMgr::intermediateThreadProc+0x49 (FPO: [1,1,0])
    05 06bbffec 00000000 79f27d35 066e2da0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      15  Id: 14ec.93c Suspend: 1 Teb: 7ffab000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 06bffe08 7c826f69 77e41ed5 00000000 06bffe4c ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 06bffe0c 77e41ed5 00000000 06bffe4c 00000000 ntdll!NtDelayExecution+0xc (FPO: [2,0,0])
    02 06bffe74 79e8d050 000001f4 00000000 7a86754e kernel32!SleepEx+0x68 (FPO: [SEH])
    03 06bffea8 79e8d06a 000001f4 00000000 7a867502 mscorwks!EESleepEx+0xbb (FPO: [2,5,4])
    04 06bffee4 79e7c7e5 000001f4 00000000 06bfffb8 mscorwks!__DangerousSwitchToThread+0x51 (FPO: [2,7,4])
    05 06bffef4 79eb7666 000001f4 3881f758 00000109 mscorwks!__SwitchToThread+0xd (FPO: [1,0,0])
    06 06bfffb8 77e6482f 00000000 00000000 00000000 mscorwks!ThreadpoolMgr::GateThreadStart+0xa1 (FPO: [1,19,4])
    07 06bfffec 00000000 79eb75d1 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      16  Id: 14ec.125c Suspend: 1 Teb: 7ffaa000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 06c3ea58 7c827d29 77e61d1e 0000025c 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 06c3ea5c 77e61d1e 0000025c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 06c3eacc 77e61c8d 0000025c ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 06c3eae0 7769c7ad 0000025c ffffffff 066e5a10 kernel32!WaitForSingleObject+0x12 (FPO: [2,0,0])
    04 06c3eafc 7778b5cb 001d20c8 066e5a10 00000000 ole32!GetToSTA+0x7c (FPO: [2,0,0])
    05 06c3eb1c 7778c38b 06c3ebe4 06c3ece4 00273dac ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xcb (FPO: [1,1,4])
    06 06c3ebfc 776c0565 00273dac 06c3ecf4 06c3ece4 ole32!CRpcChannelBuffer::SendReceive2+0xd3 (FPO: [3,50,4])
    07 06c3ec68 776c04fa 00273dac 06c3ecf4 06c3ece4 ole32!CAptRpcChnl::SendReceive+0xab (FPO: [3,19,4])
    08 06c3ecbc 77ce247f 00273dac 06c3ecf4 06c3ece4 ole32!CCtxComChnl::SendReceive+0x1a9 (FPO: [3,13,4])
    09 06c3ecd8 77ce252f 0738beac 06c3ed20 0300002c rpcrt4!NdrProxySendReceive+0x43 (FPO: [2,0,0])
    0a 06c3f0c0 77ce25a6 07a41e40 07a41fb0 06c3f0f8 rpcrt4!NdrClientCall2+0x206 (FPO: [SEH])
    0b 06c3f0e0 77c64f87 0000000c 00000008 06c3f21c rpcrt4!ObjectStublessClient+0x8b (FPO: [2,0,4])
    0c 06c3f114 79e82f10 02000007 79e7f082 ff3a5b3c rpcrt4!ObjectStubless+0xf
    0d 06c3f21c 08f5274d 06c3f28c 14334904 14341e60 mscorwks!MDInternalRO::GetCustomAttributeByName+0x1a (FPO: [5,0,0])
    *** WARNING: Unable to verify checksum for System.Web.ni.dll
    WARNING: Frame IP not in any known module. Following frames may be wrong.
    0e 06c3f344 66799ec8 14341df0 143bff2c 143c1134 0x8f5274d
    0f 06c3f360 66799d2f 143b7a94 143c1134 143b83e4 System_Web_ni+0x879ec8
    10 06c3f378 66799f6b 06c3f388 66095d9e 06c3f3a0 System_Web_ni+0x879d2f
    11 06c3f380 66095d9e 06c3f3a0 66095cf5 00000000 System_Web_ni+0x879f6b
    12 06c3f388 66095cf5 00000000 1833a734 143b83e4 System_Web_ni+0x175d9e
    13 06c3f3a0 660a7f1e 00000000 00000000 00000000 System_Web_ni+0x175cf5

      17  Id: 14ec.15e4 Suspend: 1 Teb: 7ffa9000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0702fe30 7c827d19 77e6202c 00000003 0702fe80 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0702fe34 77e6202c 00000003 0702fe80 00000001 ntdll!NtWaitForMultipleObjects+0xc (FPO: [5,0,0])
    02 0702fedc 77e62fbe 00000003 0702ff20 00000000 kernel32!WaitForMultipleObjectsEx+0x11a (FPO: [SEH])
    03 0702fef8 79f1a361 00000003 0702ff20 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [4,0,0])
    04 0702ff58 79f1a2be 7b3b746e 00000000 00000000 mscorwks!DebuggerRCThread::MainLoop+0xe9 (FPO: [0,14,4])
    05 0702ff88 79f1a1e5 7b3b745e 00000000 00000000 mscorwks!DebuggerRCThread::ThreadProc+0xe5 (FPO: [0,6,4])
    06 0702ffb8 77e6482f 00000000 00000000 00000000 mscorwks!DebuggerRCThread::ThreadProcStatic+0x9c (FPO: [1,6,4])
    07 0702ffec 00000000 79f1a19f 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      18  Id: 14ec.1484 Suspend: 1 Teb: 7ffa8000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0729fe50 7c827d29 77e61d1e 000002e8 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0729fe54 77e61d1e 000002e8 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 0729fec4 79e8c5f9 000002e8 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 0729ff08 79e8c52f 000002e8 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [SEH])
    04 0729ff58 79e8c54e ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [3,11,4])
    05 0729ff6c 79fcf2fe ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [3,0,0])
    06 0729ff98 79f23b6a 79fcd1d2 79f03bde 79f23b63 mscorwks!SVR::gc_heap::gc_thread_function+0x30 (FPO: [0,3,4])
    07 0729ffb8 77e6482f 06731fc0 00000000 00000000 mscorwks!SVR::gc_heap::gc_thread_stub+0x92 (FPO: [1,1,0])
    08 0729ffec 00000000 79f23b1c 06731fc0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      19  Id: 14ec.1698 Suspend: 1 Teb: 7ffa7000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 072dfd50 7c827d29 77e61d1e 000002e4 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 072dfd54 77e61d1e 000002e4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 072dfdc4 79e8c5f9 000002e4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 072dfe08 79e8c52f 000002e4 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [SEH])
    04 072dfe58 79e8c54e ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [3,11,4])
    05 072dfe6c 79fcf272 ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [3,0,0])
    06 072dfe98 79f23b6a 00000000 7c8276dd 072dfef8 mscorwks!SVR::gc_heap::gc_thread_function+0x59 (FPO: [0,3,4])
    07 072dffb8 77e6482f 06733428 00000000 00000000 mscorwks!SVR::gc_heap::gc_thread_stub+0x92 (FPO: [1,1,0])
    08 072dffec 00000000 79f23b1c 06733428 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      20  Id: 14ec.16a0 Suspend: 1 Teb: 7ffa6000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0731fcc4 7c827d19 77e6202c 00000002 0731fd14 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0731fcc8 77e6202c 00000002 0731fd14 00000001 ntdll!NtWaitForMultipleObjects+0xc (FPO: [5,0,0])
    02 0731fd70 77e62fbe 00000002 7a3b53d0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a (FPO: [SEH])
    03 0731fd8c 79fd4762 00000002 7a3b53d0 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [4,0,0])
    04 0731fdac 79fd4854 06734b20 0731feb0 06735470 mscorwks!SVR::WaitForFinalizerEvent+0x77 (FPO: [1,0,0])
    05 0731fdc0 79e9848f 0731feb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0x49 (FPO: [1,0,0])
    06 0731fdd4 79e9842b 0731feb0 0731fe5c 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a (FPO: [1,0,0])
    07 0731fe68 79e98351 0731feb0 7b087542 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3 (FPO: [1,28,4])
    08 0731fea4 79f074d4 0731feb0 00000000 07406d20 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a (FPO: [SEH])
    09 0731fecc 79f074e5 79fd4809 00000008 0731ff14 mscorwks!ManagedThreadBase_NoADTransition+0x32 (FPO: [2,7,0])
    0a 0731fedc 79f090b3 79fd4809 7b0874f2 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd (FPO: [1,0,0])
    0b 0731ff14 79f75715 00000000 00000001 ffffffff mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb (FPO: [SEH])
    0c 0731ffb8 77e6482f 06735470 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [1,1,0])
    0d 0731ffec 00000000 79f756cf 06735470 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      21  Id: 14ec.c88 Suspend: 1 Teb: 7ffa5000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0773fec4 7c826f69 77e41ed5 00000001 0773ff08 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0773fec8 77e41ed5 00000001 0773ff08 00000000 ntdll!NtDelayExecution+0xc (FPO: [2,0,0])
    02 0773ff30 79ec45f9 000016b5 00000001 0773ffa0 kernel32!SleepEx+0x68 (FPO: [SEH])
    03 0773ffac 79ec44a7 00000000 0773ffec 77e6482f mscorwks!ThreadpoolMgr::TimerThreadFire+0x6d (FPO: [0,21,4])
    04 0773ffb8 77e6482f 06c3ec74 00000000 00000000 mscorwks!ThreadpoolMgr::TimerThreadStart+0x57 (FPO: [1,0,0])
    05 0773ffec 00000000 79ec444f 06c3ec74 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      22  Id: 14ec.1280 Suspend: 1 Teb: 7ffa4000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0777fcac 7c827d29 77e61d1e 000002dc 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0777fcb0 77e61d1e 000002dc 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 0777fd20 79e8c5f9 000002dc ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 0777fd64 79e8c52f 000002dc ffffffff 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [SEH])
    04 0777fdb4 79e8c54e ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [3,11,4])
    05 0777fdc8 79efc1d5 ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [3,0,0])
    06 0777fe94 79f75715 07406090 00000000 00000000 mscorwks!AppDomain::ADUnloadThreadStart+0x308 (FPO: [1,40,4])
    07 0777ffb8 77e6482f 07406920 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [1,1,0])
    08 0777ffec 00000000 79f756cf 07406920 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      23  Id: 14ec.294 Suspend: 1 Teb: 7ffa3000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0780fc14 7c827d29 77e61d1e 00000230 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0780fc18 77e61d1e 00000230 00000000 0780fc5c ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 0780fc88 79e8c5f9 00000230 00009c40 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 0780fccc 79e8c52f 00000230 00009c40 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [SEH])
    04 0780fd1c 79e8c54e 00009c40 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [3,11,4])
    05 0780fd30 79ec43bb 00009c40 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [3,0,0])
    06 0780fdb0 79ec4849 066d9028 00009c40 00000000 mscorwks!ThreadpoolMgr::SafeWait+0x73 (FPO: [3,21,4])
    07 0780fe14 79f75715 00000000 00000000 00000000 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11c (FPO: [1,16,4])
    08 0780ffb8 77e6482f 078694c0 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [1,1,0])
    09 0780ffec 00000000 79f756cf 078694c0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      24  Id: 14ec.bec Suspend: 1 Teb: 7ffa1000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 08dbff70 7c8277f9 7c839f68 000000e4 08dbffb0 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 08dbff74 7c839f68 000000e4 08dbffb0 08dbffb4 ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 08dbffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x3d (FPO: [1,7,0])
    03 08dbffec 00000000 7c839f2b 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      25  Id: 14ec.f80 Suspend: 1 Teb: 7ffa2000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 090efec4 7c827d19 77e6202c 00000001 090eff14 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 090efec8 77e6202c 00000001 090eff14 00000001 ntdll!NtWaitForMultipleObjects+0xc (FPO: [5,0,0])
    02 090eff70 7a13c15d 00000001 066aa3b0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a (FPO: [SEH])
    03 090effb8 77e6482f 00000001 00000000 00000000 mscorwks!ThreadpoolMgr::WaitThreadStart+0x9a (FPO: [1,7,4])
    04 090effec 00000000 7a13c07c 066aa398 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      26  Id: 14ec.16c8 Suspend: 1 Teb: 7ffa0000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0926fb94 7c827d29 77e61d1e 00000230 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0926fb98 77e61d1e 00000230 00000000 0926fbdc ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    02 0926fc08 79e8c5f9 00000230 00009c40 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [SEH])
    03 0926fc4c 79e8c52f 00000230 00009c40 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [SEH])
    04 0926fc9c 79e8c54e 00009c40 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [3,11,4])
    05 0926fcb0 79ec43bb 00009c40 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [3,0,0])
    06 0926fd30 79ec4849 066d9028 00009c40 00000000 mscorwks!ThreadpoolMgr::SafeWait+0x73 (FPO: [3,21,4])
    07 0926fd94 79f75715 00000000 80a60456 00000000 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11c (FPO: [1,16,4])
    08 0926ffb8 77e6482f 066de570 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [1,1,0])
    09 0926ffec 00000000 79f756cf 066de570 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      27  Id: 14ec.11c8 Suspend: 1 Teb: 7ff3f000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 092afc74 7c8277f9 77e5bea2 0000023c 092afcfc ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 092afc78 77e5bea2 0000023c 092afcfc 092afcbc ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 092afca4 79f5631c 0000023c 092afcf4 092afcfc kernel32!GetQueuedCompletionStatus+0x29 (FPO: [5,4,0])
    03 092afd14 79f75715 00000000 092afd28 00000001 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x141 (FPO: [1,17,4])
    04 092affb8 77e6482f 066fc9b8 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [1,1,0])
    05 092affec 00000000 79f756cf 066fc9b8 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      28  Id: 14ec.678 Suspend: 1 Teb: 7ff3e000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0931ff70 7c8277f9 7c839f68 000000e4 0931ffb0 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0931ff74 7c839f68 000000e4 0931ffb0 0931ffb4 ntdll!NtRemoveIoCompletion+0xc (FPO: [5,0,0])
    02 0931ffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x3d (FPO: [1,7,0])
    03 0931ffec 00000000 7c839f2b 00000000 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

      29  Id: 14ec.137c Suspend: 1 Teb: 7ff3d000 Unfrozen
     # ChildEBP RetAddr  Args to Child             
    00 0935ff08 7c826f69 77e41ed5 00000000 0935ff4c ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    01 0935ff0c 77e41ed5 00000000 0935ff4c 77e61c96 ntdll!NtDelayExecution+0xc (FPO: [2,0,0])
    02 0935ff74 77e424fd 0000ea60 00000000 0935ffac kernel32!SleepEx+0x68 (FPO: [SEH])
    03 0935ff84 776bbb0f 0000ea60 06788eb0 776bbab4 kernel32!Sleep+0xf (FPO: [1,0,0])
    04 0935ff90 776bbab4 00000000 00000000 06788eb0 ole32!CROIDTable::WorkerThreadLoop+0x14 (FPO: [1,0,0])
    05 0935ffac 776b1704 00000000 0935ffec 77e6482f ole32!CRpcThread::WorkerLoop+0x26 (FPO: [0,1,0])
    06 0935ffb8 77e6482f 06788eb0 00000000 00000000 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20 (FPO: [1,0,4])
    07 0935ffec 00000000 776b16e4 06788eb0 00000000 kernel32!BaseThreadStart+0x34 (FPO: [SEH])

    Hm, as you see Thread 16 is calling the LoadXML function while Thread 20 is calling a Finalizer on it?

    Apart from that Failed to start stack walk 80004005 sounds like access denied as I tought a security issue.

    0:005> kv
    ChildEBP RetAddr  Args to Child             
    WARNING: Stack unwind information not available. Following frames may be wrong.
    02eef610 0936150c 09730068 096a46a0 09362c26 XCSSParser!DllUnregisterServer+0x2886
    02eef63c 09363e50 09730068 02eef67c 096a4698 XCSSParser+0x150c
    02eef660 09363b83 09730068 02eefa84 09375cd8 XCSSParser!DllUnregisterServer+0x114f
    02eef678 77c80193 096a4698 096a4738 02020202 XCSSParser!DllUnregisterServer+0xe82
    02eef694 77ce33e1 09363b52 02eef878 00000002 rpcrt4!Invoke+0x30
    02eefa94 77ce1968 07a4a570 07389a24 066e5a68 rpcrt4!NdrStubCall2+0x299 (FPO: [SEH])
    02eefaec 7d0e5684 07a4a570 066e5a68 07389a24 rpcrt4!CStdStubBuffer_Invoke+0x3f (FPO: [SEH])
    02eefb10 7778d01b 07a4a570 066e5a68 07389a24 oleaut32!CUnivStubWrapper::Invoke+0xc5 (FPO: [3,1,4])
    02eefb54 7778cfc8 066e5a68 073811b4 001dbc10 ole32!SyncStubInvoke+0x37 (FPO: [SEH])
    02eefb9c 776c120b 066e5a68 0738a218 073a2c70 ole32!StubInvoke+0xa7 (FPO: [6,7,4])
    02eefc78 776c0bf5 07389a24 00000000 073a2c70 ole32!CCtxComChnl::ContextInvoke+0xec (FPO: [5,44,4])
    02eefc94 776bc455 066e5a68 00000001 073a2c70 ole32!MTAInvoke+0x1a (FPO: [6,0,0])
    02eefcc0 7778ced5 066e5a68 00000001 073a2c70 ole32!STAInvoke+0x48 (FPO: [7,2,4])
    02eefcf4 7778cd66 d0908070 07389a24 073a2c70 ole32!AppInvoke+0xa3 (FPO: [7,1,4])
    02eefdc8 7778c24d 066e5a10 001d2da0 00000000 ole32!ComInvokeWithLockAndIPID+0x2c5 (FPO: [2,41,4])
    02eefdf0 776bc344 066e5a10 00000400 0674f2b8 ole32!ComInvoke+0xca (FPO: [1,3,4])
    02eefe04 776bc30f 066e5a10 02eefe94 776bc27c ole32!ThreadDispatch+0x23 (FPO: [1,0,0])
    02eefe1c 7739b6e3 001d77d8 00000400 0000babe ole32!ThreadWndProc+0xfe (FPO: [4,0,4])
    02eefe48 7739b874 776bc27c 0003209a 00000400 user32!InternalCallWinProc+0x28
    02eefec0 7739ba92 00000000 776bc27c 0003209a user32!UserCallWinProcCheckWow+0x151 (FPO: [SEH])

    0:005> !verifyheap
    -verify will only produce output if there are errors in the heap
    ------------------------------
    Heap 0
    total 4469 objects
    ------------------------------
    Heap 1
    total 34266 objects
    ------------------------------

    0:005> !finalizequeue
    SyncBlocks to be cleaned up: 0
    MTA Interfaces to be released: 0
    STA Interfaces to be released: 0
    ----------------------------------
    Heap 0
    generation 0 has 0 finalizable objects (06732ac8->06732ac8)
    generation 1 has 0 finalizable objects (06732ac8->06732ac8)
    generation 2 has 30 finalizable objects (06732a50->06732ac8)
    Ready for finalization 0 objects (06732ac8->06732ac8)
    ------------------------------
    Heap 1
    generation 0 has 0 finalizable objects (0673427c->0673427c)
    generation 1 has 0 finalizable objects (0673427c->0673427c)
    generation 2 has 75 finalizable objects (06734150->0673427c)
    Ready for finalization 0 objects (0673427c->0673427c)
    Statistics:
          MT    Count    TotalSize Class Name
    7a5e7394        1           20 Microsoft.Win32.SafeHandles.SafeProcessHandle
    7a5e733c        1           20 Microsoft.Win32.SafeHandles.SafeFileMapViewHandle
    7a5e72e4        1           20 Microsoft.Win32.SafeHandles.SafeFileMappingHandle
    7933491c        1           20 Microsoft.Win32.SafeHandles.SafeFileMappingHandle
    793348c4        1           20 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle
    7932e9cc        1           20 Microsoft.Win32.SafeHandles.SafeFileHandle
    793180c0        1           20 Microsoft.Win32.SafeHandles.SafeTokenHandle
    6615175c        1           20 System.Web.ClientImpersonationContext
    66147fa8        1           20 System.Web.PerfInstanceDataHandle
    7994f9c0        1           24 System.Threading.RegisteredWaitHandleSafe
    6614befc        2           24 System.Web.Configuration.ImpersonateTokenRef
    6614b374        1           32 System.Web.Compilation.CompilationMutex
    79323470        2           40 System.Security.Cryptography.SafeProvHandle
    79321de4        2          120 System.Runtime.Remoting.Contexts.Context
    79317a3c        6          120 Microsoft.Win32.SafeHandles.SafeWaitHandle
    7932a208        3          132 System.Threading.ReaderWriterLock
    79330fd4        3          168 System.Threading.Thread
    793170c4        7          168 System.Threading.TimerBase
    66148714        7          196 System.Web.DirMonCompletion
    7932b21c       14          280 Microsoft.Win32.SafeHandles.SafeRegistryHandle
    7932a1b0       38          608 System.WeakReference
    7a5e6bcc       10         1200 System.Diagnostics.PerformanceCounter
    Total 105 objects

    0:005> !eeheap -gc
    Number of GC Heaps: 2
    ------------------------------
    Heap 0 (06731fc0)
    generation 0 starts at 0x1035979c
    generation 1 starts at 0x10359790
    generation 2 starts at 0x10330038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    10330000 10330038  103597a8 0x00029770(169840)
    Large object heap starts at 0x18330038
     segment    begin allocated     size
    18330000 18330038  1833da48 0x0000da10(55824)
    Heap Size   0x37180(225664)
    ------------------------------
    Heap 1 (06733428)
    generation 0 starts at 0x1446dd2c
    generation 1 starts at 0x1446dd20
    generation 2 starts at 0x14330038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    14330000 14330038  1446dd38 0x0013dd00(1301760)
    Large object heap starts at 0x1a330038
     segment    begin allocated     size
    1a330000 1a330038  1a390078 0x00060040(393280)
    Heap Size  0x19dd40(1695040)
    ------------------------------
    GC Heap Size  0x1d4ec0(1920704)

    Above was with GCStress and PageHeap. Now this is just the first dump there are 2 more created 0x80010105 and 0xE0434F4D. What here could be reason?

    Thanks & Regards,

    Constantin

    Tuesday, April 13, 2010 8:34 PM