none
A severe error occurred on the current command. The results, if any, should be discarded

    Question

  •  

    Hello.

    I happened to notice backups on one of our SQL instance began to fail unexpectedly. Looking the job history I found out that they have failed with the error:

    Executing the query "EXECUTE master.dbo.xp_delete_file 0,N'I:\Backups\BLAH_BLAH\Full',N'bak',N'2011-05-03T21:00:58'  " failed with the following error: "A severe error occurred on the current command.  The results, if any, should be discarded.  A severe error occurred on the current command.  The results, if any, should be discarded.".

    Seeing this my first thought was to go through the SQL server errorlog. But after executing master..xp_readerrorlog I see:

    Msg 0, Level 11, State 0, Line 0

    A severe error occurred on the current command.  The results, if any, should be discarded.

    Msg 0, Level 20, State 0, Line 0

    A severe error occurred on the current command.  The results, if any, should be discarded.

    (I tried the GUI too - same error). So I manually loaded the errorlog into a notepad. Lot of stack dumps generated. The start of each dump looked like this.

     

    2011-05-08 00:00:25.13 spid79      * BEGIN STACK DUMP:

    2011-05-08 00:00:25.13 spid79      *   05/08/11 00:00:25 spid 79

    2011-05-08 00:00:25.13 spid79      *

    2011-05-08 00:00:25.13 spid79      *

    2011-05-08 00:00:25.13 spid79      *   Exception Address = 000007FF7FC51D71 Module(msvcrt+0000000000051D71)

    2011-05-08 00:00:25.13 spid79      *   Exception Code    = c0000005 EXCEPTION_ACCESS_VIOLATION

    2011-05-08 00:00:25.13 spid79      *   Access Violation occurred reading address FFFFFFFFFFFFFFFF

    2011-05-08 00:00:25.13 spid79      * Input Buffer 198 bytes -

    2011-05-08 00:00:25.13 spid79      *             EXECUTE master.dbo.xp_delete_file 0,N'I:\Backups\blah',N'ba

    2011-05-08 00:00:25.13 spid79      *  k',N'2011-05-01T00:00:23'

    So the backups were taken; but it has a problem doing the cleanup. 

    The event logs too had almost the same thing to say. 

    -- For the errorlog.

    Exception happened when running extended stored procedure 'xp_readerrorlog' in the library 'xpstar90.dll'. SQL Server is terminating process 68. Exception type: Win32 exception; Exception code: 0xc0000005.

    --For xp_delette_file

    Exception happened when running extended stored procedure 'xp_delete_file' in the library 'xpstar90.dll'. SQL Server is terminating process 91. Exception type: Win32 exception; Exception code: 0xc0000005.

    The SQL server version is 

    Microsoft SQL Server 2005 - 9.00.3042.00 (X64) 

    Feb 10 2007 00:59:02 

    Copyright (c) 1988-2005 Microsoft Corporation

    Enterprise Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 2)

     

    Any pointers to resolve these issues is highly appreciated.

    Thank you.

     

     

    Monday, May 09, 2011 11:17 AM

Answers

  • Since all the errors are centered around calls to Extended Stored Procedures I would suspect your system got into memory pressure due to heavy multipage allocations and you possibly hit a out of memory state that lead to the stack dump occuring, and now you have residual issues from the dump.  Have you tried restarting the SQL Server instance since the stack dump occurred?  

    Unfortunately, in the case of stack dumps, trying to solve them over the forums is usually pretty unproductive because it usually requires cracking the dump files with windbg and the SQL Server symbols to identify the dump exception and then walk the call stack for further troubleshooting.  This is something that Product Support Services does as a part of support call when they encounter a SQL Server dumping out an exception.

    I notice that you are on SQL Server 2005 Service Pack 2 while the latest Service Pack is Service Pack 4 http://support.microsoft.com/kb/913089.  You might try updating to the latest Service Pack which has been out for a while now, as this would be one of the recommended steps from Product Support Services in troubleshooting the issue as well.  What is the Servers configuration like?  Hardware configuration; #cpu, RAM installed, etc...  SQL Server configuration; max server memory setting, lock pages in memory being used, max degree of parallelism settings, etc....


    Jonathan Kehayias | Senior Consultant, SQLSkills.com
    SQL Server MVP | Microsoft Certified Master: SQL Server 2008
    Feel free to contact me through My Blog or Twitter
    Please click the Mark as Answer button if a post solves your problem!

    • Marked as answer by aG_ Tuesday, May 10, 2011 11:53 AM
    Monday, May 09, 2011 12:04 PM
    Moderator

All replies

  • Whats the SQL Server Build ..
    Abhay Chaudhary OCP 9i, MCTS/MCITP (SQL Server 2005, 2008, 2005 BI) ms-abhay.blogspot.com/
    Monday, May 09, 2011 12:02 PM
  • Since all the errors are centered around calls to Extended Stored Procedures I would suspect your system got into memory pressure due to heavy multipage allocations and you possibly hit a out of memory state that lead to the stack dump occuring, and now you have residual issues from the dump.  Have you tried restarting the SQL Server instance since the stack dump occurred?  

    Unfortunately, in the case of stack dumps, trying to solve them over the forums is usually pretty unproductive because it usually requires cracking the dump files with windbg and the SQL Server symbols to identify the dump exception and then walk the call stack for further troubleshooting.  This is something that Product Support Services does as a part of support call when they encounter a SQL Server dumping out an exception.

    I notice that you are on SQL Server 2005 Service Pack 2 while the latest Service Pack is Service Pack 4 http://support.microsoft.com/kb/913089.  You might try updating to the latest Service Pack which has been out for a while now, as this would be one of the recommended steps from Product Support Services in troubleshooting the issue as well.  What is the Servers configuration like?  Hardware configuration; #cpu, RAM installed, etc...  SQL Server configuration; max server memory setting, lock pages in memory being used, max degree of parallelism settings, etc....


    Jonathan Kehayias | Senior Consultant, SQLSkills.com
    SQL Server MVP | Microsoft Certified Master: SQL Server 2008
    Feel free to contact me through My Blog or Twitter
    Please click the Mark as Answer button if a post solves your problem!

    • Marked as answer by aG_ Tuesday, May 10, 2011 11:53 AM
    Monday, May 09, 2011 12:04 PM
    Moderator
  • Whats the SQL Server Build ..
    Abhay Chaudhary OCP 9i, MCTS/MCITP (SQL Server 2005, 2008, 2005 BI) ms-abhay.blogspot.com/

    It was listed in the original post:

     

    The SQL server version is Microsoft SQL Server 2005 - 9.00.3042.00 (X64)
    Feb 10 2007 00:59:02
    Copyright (c) 1988-2005 Microsoft Corporation
    Enterprise Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 2)

    Jonathan Kehayias | Senior Consultant, SQLSkills.com
    SQL Server MVP | Microsoft Certified Master: SQL Server 2008
    Feel free to contact me through My Blog or Twitter
    Please click the Mark as Answer button if a post solves your problem!

    Monday, May 09, 2011 12:09 PM
    Moderator
  • Yes , I noticed it late Jon ...
    Abhay Chaudhary OCP 9i, MCTS/MCITP (SQL Server 2005, 2008, 2005 BI) ms-abhay.blogspot.com/
    Monday, May 09, 2011 12:11 PM
  • Access Violation occurred reading address FFFFFFFFFFFFFFFF <-- Any thread that is going to try to reach this region in memory will be thrown out as this region is not suppose to be accessed ...Ideally , this should not happen as the functioin should not return a value such that this address is accessed ..I would say first apply SP and see if this is fixed .Otherwise PSS can assist you with the AV dump debugging as no one outside has Symbol files ..
    Abhay Chaudhary OCP 9i, MCTS/MCITP (SQL Server 2005, 2008, 2005 BI) ms-abhay.blogspot.com/
    Monday, May 09, 2011 12:27 PM
  • Access Violation occurred reading address FFFFFFFFFFFFFFFF <-- Any thread that is going to try to reach this region in memory will be thrown out as this region is not suppose to be accessed ...Ideally , this should not happen as the functioin should not return a value such that this address is accessed ..I would say first apply SP and see if this is fixed .Otherwise PSS can assist you with the AV dump debugging as no one outside has Symbol files ..
    Abhay Chaudhary OCP 9i, MCTS/MCITP (SQL Server 2005, 2008, 2005 BI) ms-abhay.blogspot.com/

    Microsoft has a public Symbol server that anyone can use with windbg to get access to the debugger symbols for SQL Server:

    http://support.microsoft.com/kb/311503

    I have blogged in the past about how to use the SQL Server symbols to open dump files, or to gain access to callstack information produced by Extended Events.  You don't have to be a member of the PSS team to read most dump files produced by SQL Server.  Certain complex problems may require private symbols to materialize portions of the callstack but those cases are not as common as you'd think.


    Jonathan Kehayias | Senior Consultant, SQLSkills.com
    SQL Server MVP | Microsoft Certified Master: SQL Server 2008
    Feel free to contact me through My Blog or Twitter
    Please click the Mark as Answer button if a post solves your problem!

    Monday, May 09, 2011 12:32 PM
    Moderator
  • I tired using it a few times in the past , but i found it slow since its loaded via internet ..Also , even if we load the symbols ,we still get the symbols not available when we load the mdmp files ..so my experience was not that good ...you might have had different experience though ...
    Abhay Chaudhary OCP 9i, MCTS/MCITP (SQL Server 2005, 2008, 2005 BI) ms-abhay.blogspot.com/
    Monday, May 09, 2011 12:39 PM
  • Recycling the services did the trick... and Yes! the issues were memory related (confirmed it using DMV queries). Also we'll hopefully patch the instance to SP4 in the coming days (we are hoping to get a maintenance window soon).

     

    Jon and Abhay - Thanks a bunch for the inputs and pointers. That definitely helped us to figure the cause of the problem.

     

     

    Tuesday, May 10, 2011 11:52 AM