none
Trying to solve a re-occuring sql server crash by using WinDbg RRS feed

  • Question

  • Hi All,

    My SQL Server crashed a few times... and the error in the sql server error log viewer was the same - "...fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION..."

    Since sql server saved a minDump file prior to crashing, I'm trying to read this file with WinDbg.exe to further understand where the problem is coming from....

    I'm doing as guided on the web. Since my SQL Server machine is not connected to the web and I need to get the symbols from the microsoft symbol server than I didn't installed the WinDbg on my computer and loaded the mini-dump file onto it. But I'm running into a few issues.

    Firstly when I open the mini-dump file I get this:

    **** (start of past 1)

    Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
    Copyright (c) Microsoft Corporation. All rights reserved.


    Loading Dump File [F:\SQLDump0032.mdmp]
    User Mini Dump File: Only registers, stack and portions of memory are available

    Comment: 'Program fault handler'
    Comment: '03/24/14 16:02:05 spid 111 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION reading address 0000000100000000 at 0x000000006FD98875
    '
    WARNING: Whitespace at end of path element
    Symbol search path is: http://msdl.microsoft.com/download/symbols

    Executable search path is:
    Windows 7 Version 7601 (Service Pack 1) MP (24 procs) Free x64
    Product: Server, suite: Enterprise TerminalServer SingleUserTS
    Machine Name:
    Debug session time: Mon Mar 24 16:02:06.000 2014 (GMT+2)
    System Uptime: 1 days 12:01:02.814
    Process Uptime: 0 days 1:45:13.000
    ................................................................
    .................................
    Loading unloaded module list
    ..........................
    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (10e0.1f9c): Access violation - code c0000005 (first/second chance not available)
    ntdll!NtWaitForSingleObject+0xa:
    00000000`7751135a c3              ret

    *** (end of paste 1)

    Ok, so I checked things out on the web and ntdll!NtWaitForSingleObject seems to be a lock issue (which is quite surprising - why would sql server crash from that especially if we have a deadlock mechanism).

    Anyways, I wanted to investigate further and did .excr:

    **** (start of paste 2)

    0:000> .ecxr
    rax=0000000004445ff0 rbx=0000000000000002 rcx=0000000021976e90
    rdx=00000000823661a0 rsi=0000000000000000 rdi=00000000219774a8
    rip=000000006fd98875 rsp=0000000021976e18 rbp=0000000021977168
     r8=0000000004446b40  r9=0000000021977d40 r10=000000006fd80000
    r11=0000000000000000 r12=0000000021977d40 r13=0000000100000000
    r14=0000000000000000 r15=0000000000000000
    iopl=0         nv up ei pl nz na po nc
    cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
    msvcr80!output_l+0x2b5:
    00000000`6fd98875 410fb67d00      movzx   edi,byte ptr [r13] ds:00000001`00000000=??

    **** (end of paste 2)

    I really don't understand much, so I realized that though i've defined the "symbol file path", I probably still need to run ".reload /f" but there are many errors especially one of sqlservr.exe (which is the most important) - why is this happening is it because the mini-dump is read from a machine different there where the dump happened (or maybe because this machine also doesn't have sql server installed on it???), anyway that's what I got:

    *** (start of paste 3)

    0:000> .reload /f
    .Unable to load image C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe, Win32 error 0n2
    *** WARNING: Unable to verify timestamp for sqlservr.exe
    ........Unable to load image C:\Windows\winsxs\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\msvcr80.dll, Win32 error 0n2
    *** WARNING: Unable to verify timestamp for msvcr80.dll
    .Unable to load image C:\Windows\winsxs\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_88df89932faf0bf6\msvcp80.dll, Win32 error 0n2
    *** WARNING: Unable to verify timestamp for msvcp80.dll
    .Unable to load image C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlos.dll, Win32 error 0n2
    *** WARNING: Unable to verify timestamp for sqlos.dll

    ..............ECT........

    *** (end of paste 3)

    Moving on I also tried to run the command "!locks" and it also failed:

    *** (start of paste 4)

    0:000> !locks
    NTSDEXTS: Unable to resolve ntdll!RTL_CRITICAL_SECTION_DEBUG type
    NTSDEXTS: Please check your symbols

    *** (end of paste 4)

    So, basically I need some help....:

    1. How do I go about finding out why sql server keeps crashing (every few hours)?

    2. How do I read the mini-dump and load symbols correctly to WinDbg.exe (do I need to run on the same machine as when the sql server that crashed is - or do i need to be a machine that has sql server on it?

    3. Also, how do I accurately understand from WinDbg.exe what is exactly going on and where does the crash originate...

    Thanks in advance,

    Dror

    Tuesday, March 25, 2014 5:49 PM

Answers

  • Thanks for the reply:-)

    I've run:

    SELECT  SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

    and it yielded those results:

    10.50.2500.0 SP1 Enterprise Edition (64-bit)

    ***

    product version = 10.50.2500.0

    product level = SP1 

    edition = Enterprise Edition (64-bit)

    ***

    Is this the latest and how do I know if it's the latest - can you please refer me to a link where there is a descritpion of what is available and what each product version / level / edition is?

    Thanks again,

    Dror

    SP1 is not the latest for SQL 2008 R2.  See http://support.microsoft.com/kb/2527041


    Dan Guzman, SQL Server MVP, http://www.dbdelta.com

    Wednesday, March 26, 2014 11:54 AM

All replies

  • The first thing you need to do is install the latest Service Pack and test if the problem is resolved.  If not, then install the latest Cumulative Update.

    Without knowing what version you are running it is impossible to guess.  This error is an internal error, there are outside causes, however it is most likely fixed already.

    Please see:

    http://support.microsoft.com/kb/321185

    Tuesday, March 25, 2014 6:59 PM
    Moderator
  • Thanks for the reply:-)

    I've run:

    SELECT  SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

    and it yielded those results:

    10.50.2500.0 SP1 Enterprise Edition (64-bit)

    ***

    product version = 10.50.2500.0

    product level = SP1 

    edition = Enterprise Edition (64-bit)

    ***

    Is this the latest and how do I know if it's the latest - can you please refer me to a link where there is a descritpion of what is available and what each product version / level / edition is?

    Thanks again,

    Dror

    Wednesday, March 26, 2014 11:10 AM
  • Thanks for the reply:-)

    I've run:

    SELECT  SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

    and it yielded those results:

    10.50.2500.0 SP1 Enterprise Edition (64-bit)

    ***

    product version = 10.50.2500.0

    product level = SP1 

    edition = Enterprise Edition (64-bit)

    ***

    Is this the latest and how do I know if it's the latest - can you please refer me to a link where there is a descritpion of what is available and what each product version / level / edition is?

    Thanks again,

    Dror

    SP1 is not the latest for SQL 2008 R2.  See http://support.microsoft.com/kb/2527041


    Dan Guzman, SQL Server MVP, http://www.dbdelta.com

    Wednesday, March 26, 2014 11:54 AM