none
.net windows service Faulting application nnn.exe, version 2.0.1.0, stamp 4b055d77, faulting module kernel32.dll, version 5.2.3790.4480 RRS feed

  • Question

  • I'm running a windows service, on a 64bit server, that is on a sys.threading.timer, polling the processlist to verify that specific apps are running and starts them if they are not running.  It works fine for a number of .exe's all 32bit vb.net apps.  The service has no ui and all the vb.exe apps do have ui.  One app, the engineconsole won't start. It is a vb.net app that employees it's own timers and threadpool.  The logging in the engineconsole.formload doesn't get run so just starting the app is failing with the application kernal32.dll fault.  If I run the engineconsole.exe by itself without launching it from the service it runs fine on the server.

    Of course all of this runs as expected on the development machine.  I've researched this issue a bit and lots of posts with similar problems regarding windows services and application faults.  Do you have any suggestions for likely problems ?


    (specs)
    ms windows server 2003 R2 sp2
    dual quad xeon cpu's
    .net 2.0 with 3.0/3.5 installed
    development in visdev 2008

    Thursday, November 19, 2009 4:19 PM

Answers

  • Karel, I think I've found the issue.  Since running the application outright was running under my id against the database it was working but the default local id the application was running under when spawned from the service did not have access to the database.  The worst of it being that until I ran !clrstack against the memory dump I saw no evidence of database resource issues.  There is a call to the db in the constructor of the first object used in the app so it was failing their before initialization finishes.  Thanks.
    Monday, November 23, 2009 7:50 PM

All replies

  • 1) How do you start the apps from your service? (Which API do you use?)
    2) Is the failing app also 32-bit? (run corflags.exe on it and make sure it is not IL-only and if it is that it is 32-bit specific, or use TaskManager to figure out how does it run from command line)
    3) Get more info about the failure - post full log entry, attach native debugger to the process and get the native call stack (gflags allows you to start debugging session everytime a particular app runs).

    -Karel
    Friday, November 20, 2009 12:28 AM
    Moderator
  • Hello Karel.  Ok I'll gather that information but wanted to add another question.  The vb.net consoleengine.exe app uses oledb drivers which I know are an issue on 64bit but and haven't been able to remedy yet but when the application is run outright the oledb driver issue only becomes a problem when you attempt to use the driver.. when the service spawns the application does it do a full requisite check for libraries to be used thus failing because of the oledb requirement? 
    Friday, November 20, 2009 2:31 PM
  • I don't think that services are that special to do some full dependency tree check. However if you run it under debugger (see #3 above), you will find out soon if that's the case ...

    Friday, November 20, 2009 4:53 PM
    Moderator
  • Yes I am working on, no.3, the tools are not on the server in question and I don't have rights to install diagnostic tools so am nudging the systems folks to install them.
    Friday, November 20, 2009 5:10 PM
  • Ok Karel, here is more info.

    --This the Event Log message --

    Faulting application engineconsole.exe, version 2.0.1.0, stamp 4b08394b, faulting module kernel32.dll, version 5.2.3790.4480, stamp 49c51cdd, debug? 0, fault address 0x0000000000027ded.

     




    1.  This is the routine the windows service that spawns the application EngineConsole.exe

        Protected Sub FireUpProcess(ByVal execPathAndName As String, ByVal processName As String)
            Dim p As New ProcessStartInfo
            Try
                p.FileName = execPathAndName
                Process.Start(p)
                tmHlper.DebugLog("FireUpProcess", TaskManHelper.ENUMLOGMSGTYPE.INFO, "Process : " & processName & " STARTED")
            Catch ex As Exception
                tmHlper.DebugLog("FireUpProcess", TaskManHelper.ENUMLOGMSGTYPE.ERR, Err.Description)
                evLog.WriteEntry("Failed to start " & processName, EventLogEntryType.Error)
            End Try
        End Sub

    2.  The EngineConsole.exe app. is compiled with 'active any cpu' and I believe running under 64bit given that in taskmanager there is no '*32' next to the process name like some other processes have.  But given that the application runs as expected when not being spawned from the service I have been speculating that this is ok?

    3.
    Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.


    Loading Dump File [D:\logfiles\DebugDiag Stack Dumps\EngineConsole__PID__4960__Date__11_23_2009__Time_01_21_45PM__723__First chance exception 0XE0434F4D.dmp]
    User Mini Dump File with Full Memory: Only application data is available

    Comment: 'Dump created by DbgHost. First chance exception 0XE0434F4D'
    Symbol search path is: *** Invalid ***
    ****************************************************************************
    * Symbol loading may be unreliable without a symbol search path.           *
    * Use .symfix to have the debugger choose a symbol path.                   *
    * After setting your symbol path, use .reload to refresh symbol locations. *
    ****************************************************************************
    Executable search path is:
    Windows Server 2003 Version 3790 (Service Pack 2) MP (16 procs) Free x64
    Product: Server, suite: TerminalServer SingleUserTS
    Machine Name:
    Debug session time: Mon Nov 23 13:21:46.000 2009 (GMT-5)
    System Uptime: 0 days 2:21:17.812
    Process Uptime: 0 days 0:00:04.000
    .........................................................
    Missing image name, possible paged-out or corrupt data.
    Loading unloaded module list
    ....
    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (1360.560): CLR exception - code e0434f4d (first/second chance not available)
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for kernel32.dll -
    kernel32!PulseEvent+0x60:
    00000000`77d4dd50 4881c4c8000000  add     rsp,0C8h
    0:000> .loadby sos mscorwks
    0:000> .chain
    Extension DLL search Path:
        C:\Program Files\Debugging Tools for Windows (x64)\WINXP;C:\Program Files\Debugging Tools for Windows (x64)\winext;C:\Program Files\Debugging Tools for Windows (x64)\winext\arcade;C:\Program Files\Debugging Tools for Windows (x64)\pri;C:\Program Files\Debugging Tools for Windows (x64);C:\Program Files\Debugging Tools for Windows (x64)\winext\arcade;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files (x86)\Dell\SysMgt\oma\bin;C:\Program Files (x86)\Dell\SysMgt\idrac;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\PROGRA~1\CA\SHARED~1\SCANEN~1
    Extension DLL chain:
        C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\sos: image 2.0.50727.3053, API 1.0.0, built Fri Jul 25 11:08:32 2008
            [path: C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\sos.dll]
        dbghelp: image 6.11.0001.404, API 6.1.6, built Wed Feb 25 21:10:27 2009
            [path: C:\Program Files\Debugging Tools for Windows (x64)\dbghelp.dll]
        ext: image 6.11.0001.404, API 1.0.0, built Wed Feb 25 21:10:26 2009
            [path: C:\Program Files\Debugging Tools for Windows (x64)\winext\ext.dll]
        exts: image 6.11.0001.404, API 1.0.0, built Wed Feb 25 21:10:17 2009
            [path: C:\Program Files\Debugging Tools for Windows (x64)\WINXP\exts.dll]
        uext: image 6.11.0001.404, API 1.0.0, built Wed Feb 25 21:10:20 2009
            [path: C:\Program Files\Debugging Tools for Windows (x64)\winext\uext.dll]
        ntsdexts: image 6.1.7015.0, API 1.0.0, built Wed Feb 25 21:09:22 2009
            [path: C:\Program Files\Debugging Tools for Windows (x64)\WINXP\ntsdexts.dll]
    0:000> !dumpheap stat
    PDB symbol for mscorwks.dll not loaded
    *** WARNING: Unable to verify checksum for EngineConsole.exe
    *** ERROR: Module load completed but symbols could not be loaded for EngineConsole.exe
    *** ERROR: Module load completed but symbols could not be loaded for xpsp2res.dll
    *** WARNING: Unable to verify checksum for Dispatcher.dll
    *** ERROR: Module load completed but symbols could not be loaded for Dispatcher.dll
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for System.Data.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for System.Transactions.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for rsaenh.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for dssenh.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for user32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for msvcr80.dll -
    *** WARNING: Unable to verify checksum for Microsoft.VisualBasic.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for Microsoft.VisualBasic.ni.dll
    *** WARNING: Unable to verify checksum for System.Xml.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Xml.ni.dll
    *** WARNING: Unable to verify checksum for System.Configuration.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Configuration.ni.dll
    *** WARNING: Unable to verify checksum for System.Runtime.Remoting.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Runtime.Remoting.ni.dll
    *** WARNING: Unable to verify checksum for System.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.ni.dll
    *** WARNING: Unable to verify checksum for System.Drawing.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Drawing.ni.dll
    *** WARNING: Unable to verify checksum for System.Windows.Forms.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Windows.Forms.ni.dll
    *** WARNING: Unable to verify checksum for mscorlib.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for mscoree.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for mscorjit.dll -
    *** WARNING: Unable to verify checksum for System.Data.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Data.ni.dll
    *** WARNING: Unable to verify checksum for System.EnterpriseServices.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.EnterpriseServices.ni.dll
    *** WARNING: Unable to verify checksum for System.Transactions.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Transactions.ni.dll
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for iphlpapi.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ole32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for hnetcfg.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ws2help.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for wshtcpip.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for mswsock.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for uxtheme.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ws2_32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for netapi32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for kerberos.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for security.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for userenv.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for msasn1.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for crypt32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for imm32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for cryptdll.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdsapi.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for schannel.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for psapi.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for msv1_0.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for dnsapi.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for wldap32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for secur32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for winrnr.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for rasadhlp.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for shlwapi.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for comctl32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for shell32.dll -
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for msvcrt.dll -
             Address               MT     Size
    total 0 objects
    Statistics:
                  MT    Count    TotalSize Class Name
    Total 0 objects
    0:000> .ecxr
    rax=0000000000129518 rbx=0000000000129540 rcx=0000000000129040
    rdx=00000000000000e0 rsi=0000000077ee6798 rdi=00000000c0000026
    rip=0000000077d4dd50 rsp=0000000000129520 rbp=000000000012da90
     r8=0000000000000000  r9=0000000000000000 r10=0000000000000038
    r11=0000000000129560 r12=0000000000129640 r13=ffffffff80131904
    r14=0000000000000001 r15=0000000000000001
    iopl=0         nv up ei pl nz na pe nc
    cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000200
    kernel32!PulseEvent+0x60:
    00000000`77d4dd50 4881c4c8000000  add     rsp,0C8h
    0:000> .reload
    *********************************************************************
    * Symbols can not be loaded because symbol path is not initialized. *
    *                                                                   *
    * The Symbol Path can be set by:                                    *
    *   using the _NT_SYMBOL_PATH environment variable.                 *
    *   using the -y <symbol_path> argument when starting the debugger. *
    *   using .sympath and .sympath+                                    *
    *********************************************************************
    0:000> .sympath
    Symbol search path is: <empty>
    Expanded Symbol search path is: <empty>
    0:000> .sympath+\
    Symbol search path is: \
    Expanded Symbol search path is: \
    0:000> .sympath+
    Symbol search path is: \
    Expanded Symbol search path is: \
    0:000> reload
    Last set context:
                ^ Bad register error in 'reload'
    0:000> .sympath+ \\myserver\myshare\symbols\x86fre\symbols
    Symbol search path is: \;\\myserver\myshare\symbols\x86fre\symbols
    Expanded Symbol search path is: \;\\myserver\myshare\symbols\x86fre\symbols
    WARNING: Inaccessible path: '\\myserver\myshare\symbols\x86fre\symbols'
    0:000> .sympath+ myserver\myshare\symbols\x86fre\symbols
    Symbol search path is: \;\\myserver\myshare\symbols\x86fre\symbols;myserver\myshare\symbols\x86fre\symbols
    Expanded Symbol search path is: \;\\myserver\myshare\symbols\x86fre\symbols;myserver\myshare\symbols\x86fre\symbols
    WARNING: Inaccessible path: '\\myserver\myshare\symbols\x86fre\symbols'
    WARNING: Inaccessible path: 'myserver\myshare\symbols\x86fre\symbols'
    0:000> .reload
    .........................................................
    Missing image name, possible paged-out or corrupt data.
    Loading unloaded module list
    ....
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for kernel32.dll -
    0:000> x fs_rec!*
                   ^ Couldn't resolve 'x fs_rec'
    0:000> lm
    start             end                 module name
    00000000`00400000 00000000`00456000   EngineConsole   (deferred)            
    00000000`19940000 00000000`19c07000   xpsp2res   (deferred)            
    00000000`19c10000 00000000`19c48000   Dispatcher   (deferred)            
    00000000`1b6d0000 00000000`1b9ba000   System_Data   (deferred)            
    00000000`1e230000 00000000`1e279000   System_Transactions   (deferred)            
    00000000`68000000 00000000`6804c000   rsaenh     (deferred)            
    00000000`68100000 00000000`6813f000   dssenh     (deferred)            
    00000000`77c20000 00000000`77d2c000   user32     (deferred)            
    00000000`77d40000 00000000`77eb3000   kernel32   (export symbols)       kernel32.dll
    00000000`77ec0000 00000000`77ffb000   ntdll      (deferred)            
    00000000`78130000 00000000`781f9000   msvcr80    (deferred)            
    00000642`3c860000 00000642`3ca77000   Microsoft_VisualBasic_ni   (deferred)            
    00000642`46f40000 00000642`475e5000   System_Xml_ni   (deferred)            
    00000642`49120000 00000642`49263000   System_Configuration_ni   (deferred)            
    00000642`4eee0000 00000642`4efdd000   System_Runtime_Remoting_ni   (deferred)            
    00000642`74880000 00000642`75284000   System_ni   (deferred)            
    00000642`75bc0000 00000642`75df8000   System_Drawing_ni   (deferred)            
    00000642`75fa0000 00000642`77036000   System_Windows_Forms_ni   (deferred)            
    00000642`78000000 00000642`78ed9000   mscorlib_ni   (deferred)            
    00000642`7ee50000 00000642`7eeb6000   mscoree    (deferred)            
    00000642`7eed0000 00000642`7f054000   mscorjit   (deferred)            
    00000642`7f330000 00000642`7fce1000   mscorwks   (deferred)            
    00000642`b7380000 00000642`b7bb9000   System_Data_ni   (deferred)            
    00000642`bb760000 00000642`bb86d000   System_EnterpriseServices_ni   (deferred)            
    00000642`bc500000 00000642`bc5e5000   System_Transactions_ni   (deferred)            
    000007ff`57040000 000007ff`57071000   iphlpapi   (deferred)            
    000007ff`57140000 000007ff`573c5000   ole32      (deferred)            
    000007ff`6d200000 000007ff`6d293000   hnetcfg    (deferred)            
    000007ff`77140000 000007ff`7714c000   ws2help    (deferred)            
    000007ff`77170000 000007ff`7717b000   wshtcpip   (deferred)            
    000007ff`771b0000 000007ff`7722d000   mswsock    (deferred)            
    000007ff`77230000 000007ff`772af000   uxtheme    (deferred)            
    000007ff`77310000 000007ff`77340000   ws2_32     (deferred)            
    000007ff`77370000 000007ff`77409000   netapi32   (deferred)            
    000007ff`77410000 000007ff`774c5000   kerberos   (deferred)            
    000007ff`77530000 000007ff`77535000   security   (deferred)            
    000007ff`7c680000 000007ff`7c78a000   userenv    (deferred)            
    000007ff`7d340000 000007ff`7d36c000   msasn1     (deferred)            
    000007ff`7d370000 000007ff`7d4cf000   crypt32    (deferred)            
    000007ff`7d500000 000007ff`7d539000   imm32      (deferred)            
    000007ff`7dc90000 000007ff`7dc9f000   cryptdll   (deferred)            
    000007ff`7dcb0000 000007ff`7dcd4000   ntdsapi    (deferred)            
    000007ff`7dd50000 000007ff`7dd92000   schannel   (deferred)            
    000007ff`7e380000 000007ff`7e390000   psapi      (deferred)            
    000007ff`7e500000 000007ff`7e545000   msv1_0     (deferred)            
    000007ff`7e8f0000 000007ff`7e93e000   dnsapi     (deferred)            
    000007ff`7e950000 000007ff`7e9b6000   wldap32    (deferred)            
    000007ff`7e9c0000 000007ff`7e9e2000   secur32    (deferred)            
    000007ff`7e9f0000 000007ff`7e9fb000   winrnr     (deferred)            
    000007ff`7ea00000 000007ff`7ea07000   rasadhlp   (deferred)            
    000007ff`7ef60000 000007ff`7effb000   shlwapi    (deferred)            
    000007ff`7f000000 000007ff`7f187000   comctl32   (deferred)            
    000007ff`7f190000 000007ff`7fb9b000   shell32    (deferred)            
    000007ff`7fc00000 000007ff`7fc86000   msvcrt     (deferred)            
    000007ff`7fc90000 000007ff`7fd2c000   gdi32      (deferred)            
    000007ff`7fd30000 000007ff`7fec9000   rpcrt4     (deferred)            
    000007ff`7fee0000 000007ff`7ffe6000   advapi32   (deferred)            
    Missing image name, possible paged-out or corrupt data.

    Unloaded modules:
    00690064`00023ee6 00690064`00673f4d   Unknown_Module_00690064`00023ee6
    45d6ccab`00000002 45d6ccab`00113dca   dll    
    00000000`00099000 00000000`00099001   PI32.dll
    000007ff`7dd50000 000007ff`7dd92000   schannel.dll
    0:000> !sym noisy
    noisy mode - symbol prompts on
    0:000> .reload
    .........................................................
    Missing image name, possible paged-out or corrupt data.
    Loading unloaded module list
    ....
    DBGHELP: \kernel32.pdb - file not found
    DBGHELP: \dll\kernel32.pdb - file not found
    DBGHELP: \symbols\dll\kernel32.pdb - file not found
    DBGHELP: \\myserver\myshare\symbols\x86fre\symbols\kernel32.pdb - file not found
    DBGHELP: \\myserver\myshare\symbols\x86fre\symbols\dll\kernel32.pdb - file not found
    DBGHELP: \\myserver\myshare\symbols\x86fre\symbols\symbols\dll\kernel32.pdb - file not found
    DBGHELP: myserver\myshare\symbols\x86fre\symbols\kernel32.pdb - file not found
    DBGHELP: myserver\myshare\symbols\x86fre\symbols\dll\kernel32.pdb - file not found
    DBGHELP: myserver\myshare\symbols\x86fre\symbols\symbols\dll\kernel32.pdb - file not found
    DBGHELP: kernel32.pdb - file not found
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for kernel32.dll -
    DBGHELP: kernel32 - export symbols

    Monday, November 23, 2009 6:58 PM
  • Karel, I think I've found the issue.  Since running the application outright was running under my id against the database it was working but the default local id the application was running under when spawned from the service did not have access to the database.  The worst of it being that until I ran !clrstack against the memory dump I saw no evidence of database resource issues.  There is a call to the db in the constructor of the first object used in the app so it was failing their before initialization finishes.  Thanks.
    Monday, November 23, 2009 7:50 PM
  • Glad to hear you found what caused your problem!
    -Karel
    Monday, November 23, 2009 11:12 PM
    Moderator