none
Exception happened when running extended stored procedure

    Question

  • Hello,

    We experience a problem with running our custom extended stored procedure on one installation.

    The exception message is the following:

    Exception happened when running extended stored procedure 'xp_customsp' in the library 'customdll.dll'. SQL Server is terminating process 52. Exception type: Win32 exception; Exception code: 0xc0000005. 

    We do not experience this problem at about 50 other installations, so it is the only server which gives the problem (it is Sql Server 2008 R2 as like a couple of dozens of other installations).

     

    Could anyone help me to undersatnd what can casue the problem and how can I investigate it (keeping in mind that it is the customer's server with several hundreds of usersa working with DB)?

    Thanks in advance.

     

    Monday, January 10, 2011 9:37 AM

Answers

  • Hey Olexandr,

    Given the code is in production, their are limited steps that can be done to isolate the cause of the exception.  First, as Balmu mentioned, SQL Server could have generated a *.mdmp file for when the exception happened which would be located in the LOG folder where SQL Server also puts the SQL Server Errorlog.

    You could use Windbg to analyze the dump using the symbols for your dll to isolate the root cause.  More information about using Windbg can be found by going to http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx.

    If debugging the memory dump does not resolve the issue or you are not sure how to use windbg to resolve the issue, the other option is to modify the code within the DLL to put in code that will help isolate where the error is being raised.  You could have it write errors to the Application Event log or a log file of your own.  You could even enhance your errorhandling within the dll to include the line of code it is failing on so that you know where in your code it is failing. In other words put in code that helps debug where the failure is occurring and why.

    0xc0000005 is as Balmu pointed out an Access Violation - Reading or writing to an inaccessible memory location.  You could get some help about Access Violations in http://msdn.microsoft.com/en-us/magazine/cc163311.aspx.  It is unlikely a permissions issue would result in you having an Access Violation.  However, if your code expected a pointer to have the address of file path for example and the variable was never set because when you attempt to set the variable the folder or location was inaccessible and the code didn't handle that situation, you code could be attempting to resolve a null pointer and that would result in an Access Violation.

    I hope this information helps.

    Sincerely,

    Rob Beene, MSFT

    Thursday, January 13, 2011 6:38 PM
    Answerer

All replies

  • Is it SQL Server 2000?
    Best Regards, Uri Dimant SQL Server MVP http://dimantdatabasesolutions.blogspot.com/ http://sqlblog.com/blogs/uri_dimant/
    Monday, January 10, 2011 9:48 AM
    Answerer
  • No, it is Sql Server 2008 R2
    Monday, January 10, 2011 10:18 AM
  • 0xc0000005 is Access Violation. Since its your own code you need to debug using symbols of your dll. You would have dump generated in LOG folder which you can load in windbg and get the line of code causing problem.
    Balmukund Lakhani | Please mark solved if I've answered your question, vote for it as helpful to help other user's find a solution quicker
    --------------------------------------------------------------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
    --------------------------------------------------------------------------------
    My Blog: http://blogs.msdn.com/blakhani
    Team Blog: http://blogs.msdn.com/sqlserverfaq
    Monday, January 10, 2011 10:20 AM
    Moderator
  • Thank you for your reply.

    How can I 'debug' it without disturbing active users?

    Which types of Access Violation can cause the problem? Is it access to the stored procedure from Sql Server or access to some files from extended procedure code?

    Monday, January 10, 2011 11:56 AM
  • Hi Olexandr Soukhoy,

     

    Is this extended stored procedure written on Microsoft .Net Framework?

    If so, please try the following steps:

    1.       Change the Permission Level to Unsafe
    in Visual Studio, please right-click on this project and choose Properties. Then please switch to Database page. Choose Unsafe from the dropdown list under Permission Level.

    2.       Alter this database and SET TRUSTWORTHY ON
    please try the following statement:
    ALTER DATABASE <database name> SET TRUSTWORTHY ON

     

    If anything is unclear, please let me know.


    Regards,
    Tom Li
    Wednesday, January 12, 2011 8:14 AM
    Moderator
  • Hi Tom Li,

     

    The procedure is written on C++. 

    Also, notice the exception occurs not every time when the extended SP is used, but occasionally. 

    One hint - the SP refers to file system. Could it be sth in Sql Server permission model which may occasionally block calls to files on disk?

     

    Regards,

    Olexandr

     

     

     

    Thursday, January 13, 2011 7:49 AM
  • Hey Olexandr,

    Given the code is in production, their are limited steps that can be done to isolate the cause of the exception.  First, as Balmu mentioned, SQL Server could have generated a *.mdmp file for when the exception happened which would be located in the LOG folder where SQL Server also puts the SQL Server Errorlog.

    You could use Windbg to analyze the dump using the symbols for your dll to isolate the root cause.  More information about using Windbg can be found by going to http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx.

    If debugging the memory dump does not resolve the issue or you are not sure how to use windbg to resolve the issue, the other option is to modify the code within the DLL to put in code that will help isolate where the error is being raised.  You could have it write errors to the Application Event log or a log file of your own.  You could even enhance your errorhandling within the dll to include the line of code it is failing on so that you know where in your code it is failing. In other words put in code that helps debug where the failure is occurring and why.

    0xc0000005 is as Balmu pointed out an Access Violation - Reading or writing to an inaccessible memory location.  You could get some help about Access Violations in http://msdn.microsoft.com/en-us/magazine/cc163311.aspx.  It is unlikely a permissions issue would result in you having an Access Violation.  However, if your code expected a pointer to have the address of file path for example and the variable was never set because when you attempt to set the variable the folder or location was inaccessible and the code didn't handle that situation, you code could be attempting to resolve a null pointer and that would result in an Access Violation.

    I hope this information helps.

    Sincerely,

    Rob Beene, MSFT

    Thursday, January 13, 2011 6:38 PM
    Answerer
  • Thank you Rob for your help.

    I got the dumps from the server and installed WinDbg.

    Here is what I get when debugging:

     

     

    This dump file has an exception of interest stored in it.

    The stored exception information can be accessed via .ecxr.

    (5f8.eb4): Access violation - code c0000005 (first/second chance not available)

    Unable to load image C:\Windows\System32\ntdll.dll, Win32 error 0n2

    *** WARNING: Unable to verify timestamp for ntdll.dll

    *** ERROR: Module load completed but symbols could not be loaded for ntdll.dll

    ntdll+0x4fd9a:

    00000000`774afd9a c3              ret


    I see similar problems reported on MSDN forums but without clear answer what the mesasge may mean :(

    Regards,
    Olexandr

     

    Thursday, January 27, 2011 9:26 AM
  • Hey Olexandr,

    To resolve the symbol error for NTDLL, you need to download or use the public symbols.  http://msdn.microsoft.com/en-us/library/b8ttk8zy.aspx

    Also, the first step is to do .ecxr.  This will take you to the exception thread stack.  Then do kpn100.  This will dump up to the first 100 functions in the stack.  Look in the stack to find the dll that raised the exception.  You will then want to attempt to isolate what caused the exception.

    You can also do !analyze -vv and this will also help with analyzing the dump and pull back the stack that raised the error.

    I hope this helps!

    Sincerely,

    Rob Beene - MSFT

    Friday, January 28, 2011 9:09 PM
    Answerer
  • Hey Olexandr,

    Here is more information about using the Microsoft Public Symbol Servers with Windbg.


    http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx which states the following:

    To use the Microsoft Symbol Server

    1.

    Make sure you have installed the latest version of Debugging Tools for Windows.

    2.

    Start a debugging session.

    3.

    Decide where to store the downloaded symbols (the "downstream store"). This can be a local drive or a UNC path.

    4.

    Set the debugger symbol path as follows, substituting your downstream store path for DownstreamStore.

    SRV*DownstreamStore*http://msdl.microsoft.com/download/symbols

    For example, to download symbols to c:\websymbols, you would add the following to your symbol path:
    SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

    Friday, January 28, 2011 9:16 PM
    Answerer
  • Hey Olexandr,

    It seems one of my posts didn't make it.  To resolve the issues with symbols for ntdll.dll, you need to use the public symbols from Microsoft as noted in my other post.

    Once you have the dump open, you do the following:

    .ecxr

    kpn100  --This will list the top100 functions in the stack that resulted in the access violation.  You then isolate the function/dll that caused the error.

    optionally, you can run the following:

    !analyze -vv

    This will automatically find the offending stack and show it to you and give you some information as to what may have contributed to the access violation.  You will still need to isolate the function that caused the problem.  You can view the Local variables by looking at locals on the View menu and move the focus to that function by using the Call stack menu option in the view menu.

    I hope this information helps!

    Sincerely,

    Rob Beene - MSFT

    Friday, January 28, 2011 9:21 PM
    Answerer
  • Thank you Robert.

    With the steps you provided I found the next lines:

    FAULTING_IP: 

    dtSearchExtendedSP+b135

    000007fe`f3b0b135 0fbe0408        movsx   eax,byte ptr [rax+rcx]

     

    As I understand it signifies that I have some troubles with byte pointers.

    I will try to make some special debug version for the server to investigate further

    Wednesday, February 02, 2011 10:58 AM
  • Btw here is the result of !analyze -vv:

     

    *******************************************************************************

    *                                                                             *

    *                        Exception Analysis                                   *

    *                                                                             *

    *******************************************************************************

     

    GetPageUrlData failed, server returned HTTP status 404

    URL requested: http://watson.microsoft.com/StageOne/sqlservr_exe/2009_100_1600_1/4bb6b40b/dtSearchExtendedSP_dll/0_0_0_0/4a8120a5/c0000005/0000b135.htm?Retriage=1

     

    FAULTING_IP: 

    dtSearchExtendedSP+b135

    000007fe`f3b0b135 0fbe0408        movsx   eax,byte ptr [rax+rcx]

     

    EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)

    ExceptionAddress: 000007fef3b0b135 (dtSearchExtendedSP+0x000000000000b135)

       ExceptionCode: c0000005 (Access violation)

      ExceptionFlags: 00000000

    NumberParameters: 2

       Parameter[0]: 0000000000000000

       Parameter[1]: 0000000021a8f000

    Attempt to read from address 0000000021a8f000

     

    PROCESS_NAME:  sqlservr.exe

     

    ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

     

    EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

     

    EXCEPTION_CODE_STR:  c0000005

     

    EXCEPTION_PARAMETER1:  0000000000000000

     

    EXCEPTION_PARAMETER2:  0000000021a8f000

     

    READ_ADDRESS:  0000000021a8f000 

     

    FOLLOWUP_IP: 

    dtSearchExtendedSP+b135

    000007fe`f3b0b135 0fbe0408        movsx   eax,byte ptr [rax+rcx]

     

    WATSON_BKT_PROCVER:  2009.100.1600.1

     

    WATSON_BKT_PROCSTAMP:  4bb6b40b

     

    WATSON_BKT_MODULE:  dtSearchExtendedSP.dll

     

    WATSON_BKT_MODVER:  0.0.0.0

     

    WATSON_BKT_MODSTAMP:  4a8120a5

     

    WATSON_BKT_MODOFFSET:  b135

     

    BUILD_VERSION_STRING:  6.1.7600.16385 (win7_rtm.090713-1255)

     

    MOD_LIST: <ANALYSIS/>

     

    NTGLOBALFLAG:  0

     

    ANALYSIS_SESSION_HOST:  UKR1-306M-2003

     

    ANALYSIS_SESSION_TIME:  02-02-2011 12:27:59.0823

     

    THREAD_ATTRIBUTES: 

     

    [ GLOBAL ]

     

        Global     PID: [1528]

        Global     Thread_Count: [1]

        Global     LoadedModule_Count: [99]

        Global     UnloadedModule_Count: [14]

        Global     PageSize: [4096]

        Global     ModList_SHA1_Hash: [8e856cb18e1f38be94c7a1a9275b14010ec1504c]

        Global     ProcessName: [sqlservr.exe]

        Global     CommandLine: ["d:\program files\microsoft sql server\mssql10_50.mssqlserver\mssql\binn\sqlservr.exe" -smssqlserver]

        Global     Desktop_Name: []

        Global     Debugger_CPU_Architecture: [X86]

        Global     CPU_ProcessorCount: [4]

        Global     CPU_MHZ: [2300]

        Global     CPU_Architecture: [X64]

        Global     CPU_Family: [16]

        Global     CPU_Model: [2]

        Global     CPU_Stepping: [3]

        Global     CPU_VendorString: [<unavailable>]

        Global     GFlags: [0]

        Global     SystemUpTime: [3 days 7:48:55.000]

        Global     SystemUpTime: [287335]

        Global     ProcessUpTime: [3 days 7:48:10.000]

        Global     ProcessUpTime: [287290]

        Global     CurrentTimeDate: [Mon Jan 10 15:01:12.000 2011 (UTC + 1:00)]

        Global     CurrentTimeDate: [1294668072]

        Global     CustomDumpFlags: [10380]

        Global     MiniDump

     

    [ THREAD ]

     

      00 Id: 5f8.eb4

     

        Frame[00]  Is_OriginalExceptionThread

        Frame[00]  Stack_Frames_Extraction_Time_(ms): [0x0]

        Frame[00]  ThreadStartAddress: [msvcr80!endthreadex]

        Frame[00]  ThreadStartAddress: [0x0000000075083810]

        Frame[00]  Thread_LastError: [0x3f0]

        Frame[00]  Thread_LastStatus: [0xc000007c]

        Frame[00]  ThreadLocale: [0x406]

        Frame[00]  BadReadAddress: [0x21a8f000]

        Frame[00]  badptr_READ: [0x0000000021a8f000]

        Frame[00]  Number_of_Stack_Frames: [0x8]

        Frame[00]  Bad_Frame_Count: [0x7]

        Frame[00]  Ignored_Frame_Count: [0x0]

        Frame[00]  Frames_not_in_stack_range: [0x0]

        Frame[00]  NotSysEnter

        Frame[00]  Is_SuspectHighUserTime

        Frame[00]  Is_DefiniteHighUserTime

        Frame[00]  badptr_Arch_AX: [0x0000000021a8eca4]

        Frame[00]  badptr_Arch_BX: [0x00000000230eab08]

        Frame[00]  badptr_Arch_CX: [0x000000000000035c]

        Frame[00]  badptr_Arch_DX: [0x0000000000000012]

        Frame[00]  null_Arch_SI

        Frame[00]  Arch_DI_Register: [0x00000000230ea9c0]

        Frame[00]  badptr_Arch_SP: [0x00000000230e0660]

        Frame[00]  badptr_Arch_BP: [0x00000000230ed0e0]

        Frame[00]  NX_LoadedModule_Arch_IP: [0x000007fef3b0b135]

        Frame[00]  badptr_msr_r8: [0x00000000000000f9]

        Frame[00]  badptr_msr_r9: [0x000000000531b510]

        Frame[00]  OneBit_msr_r10

        Frame[00]  badptr_msr_r11: [0x00000004dc57a630]

        Frame[00]  badptr_msr_r12: [0x00000004dc57aa80]

        Frame[00]  badptr_msr_r13: [0x00000004dc57a740]

        Frame[00]  badptr_msr_r14: [0x0000000497ac3650]

        Frame[00]  null_msr_r15

        Frame[00]  Instruction_Pointer: [0x000007fef3b0b135]

        Frame[00]  ip_not_executable: [0xf3b0b135]

        Frame[00]  IP_Biased

        Frame[01]  Frame: [0x00000000230eab08]

        Frame[01]  ip_is_call_value_Arch_ip

        Frame[02]  Frame: [0x0000000000000021]

        Frame[03]  Frame: [0x00000000230e05a0]

        Frame[04]  Frame: [0x00000000230e87b0]

        Frame[05]  Frame: [0x0000000000000034]

        Frame[06]  Frame: [0x00000000000000e7]

        Frame[00]  Stack_Attribute_Extraction_Time_(ms): [0x1b6]

     

     

    FAULTING_THREAD:  0000000000000eb4

     

    PROBLEM_CLASSES: 

     

    INVALID_POINTER_READ

        Tid    [0xeb4]

        Frame  [0x00]: dtsearchextendedsp

     

    CALL

        Tid    [0xeb4]

        Frame  [0x01]: unknown!unknown

        Failure Bucketing

     

     

    BUGCHECK_STR:  APPLICATION_FAULT_INVALID_POINTER_READ_CALL

     

    PRIMARY_PROBLEM_CLASS:  INVALID_POINTER_READ_CALL

     

    DEFAULT_BUCKET_ID:  INVALID_POINTER_READ_CALL

     

    IP_ON_HEAP:  00000004dc57a740

    The fault address in not in any loaded module, please check your build's rebase

    log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may

    contain the address if it were loaded.

     

    FRAME_ONE_INVALID: 1

     

    LAST_CONTROL_TRANSFER:  from 00000004dc57a740 to 000007fef3b0b135

     

    STACK_TEXT:  

    00000000`230e0660 00000004`dc57a740 : 00000000`230eab08 00000000`00000021 00000000`230e05a0 00000000`230e87b0 : dtSearchExtendedSP+0xb135

    00000000`230e0668 00000000`230eab08 : 00000000`00000021 00000000`230e05a0 00000000`230e87b0 00000000`00000034 : 0x4`dc57a740

    00000000`230e0670 00000000`00000021 : 00000000`230e05a0 00000000`230e87b0 00000000`00000034 00000000`000000e7 : 0x230eab08

    00000000`230e0678 00000000`230e05a0 : 00000000`230e87b0 00000000`00000034 00000000`000000e7 00000000`00000000 : 0x21

    00000000`230e0680 00000000`230e87b0 : 00000000`00000034 00000000`000000e7 00000000`00000000 00000000`00000000 : 0x230e05a0

    00000000`230e0688 00000000`00000034 : 00000000`000000e7 00000000`00000000 00000000`00000000 cccccccc`ffffffff : 0x230e87b0

    00000000`230e0690 00000000`000000e7 : 00000000`00000000 00000000`00000000 cccccccc`ffffffff cccccccc`cccccccc : 0x34

    00000000`230e0698 00000000`00000000 : 00000000`00000000 cccccccc`ffffffff cccccccc`cccccccc cccccccc`cccccccc : 0xe7

     

     

    STACK_COMMAND:  ~0s; .ecxr ; kb

     

    FAULT_INSTR_CODE:  804be0f

     

    SYMBOL_STACK_INDEX:  0

     

    SYMBOL_NAME:  dtSearchExtendedSP+b135

     

    FOLLOWUP_NAME:  MachineOwner

     

    MODULE_NAME: dtSearchExtendedSP

     

    IMAGE_NAME:  dtSearchExtendedSP.dll

     

    DEBUG_FLR_IMAGE_TIMESTAMP:  4a8120a5

     

    FAILURE_BUCKET_ID:  INVALID_POINTER_READ_CALL_c0000005_dtSearchExtendedSP.dll!Unknown

     

    BUCKET_ID:  X64_APPLICATION_FAULT_INVALID_POINTER_READ_CALL_dtSearchExtendedSP+b135

     

    WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/sqlservr_exe/2009_100_1600_1/4bb6b40b/dtSearchExtendedSP_dll/0_0_0_0/4a8120a5/c0000005/0000b135.htm?Retriage=1

     

    ANALYSIS_SESSION_ELAPSED_TIME: 10a9

     

    Followup: MachineOwner

    ---------

     

    Wednesday, February 02, 2011 11:29 AM
  • Hey Olexandr,

    Yes, I would agree. The problem appears to be an invalid pointer.  Could be the result of not initializing the pointer or it not getting set correctly.  Possibly taking a look at the values in the registers - ptr [rax+rcx] may help give a clue.  Also, see if the locals shows anything? Possibly a variable is not set as expected?

    Hope this helps.

    Sincerely,

    Rob Beene, MSFT

    Wednesday, February 02, 2011 5:44 PM
    Answerer
  • Finally the problem is resolved. It was a buggy pointer usage in C++ code.

    Thanks to everyone for help.

    Wednesday, May 11, 2011 12:02 PM
  • Finally the problem is resolved. It was a buggy pointer usage in C++ code.

    Thanks to everyone for help.

    Thanks for providing update.
    Balmukund Lakhani | Please mark solved if I've answered your question, vote for it as helpful to help other user's find a solution quicker
    --------------------------------------------------------------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
    --------------------------------------------------------------------------------
    My Blog: http://blogs.msdn.com/blakhani
    Team Blog: http://blogs.msdn.com/sqlserverfaq
    Wednesday, May 11, 2011 3:54 PM
    Moderator