none
Incorrect stack on access violation - need help reconstructing - possible heap corruption unmanaged C++ console app RRS feed

  • Question

  • This is a release build with /O2 optimizations.  The program processes requests for delivering files via FTP.

    I've tried to follow the two techniques in these links to try and reconstruct the stack, but have not been able to do so. I cannot seem to match up the EBP & previous EBP values.

     http://www.dumpanalysis.org/blog/index.php/2007/07/25/reconstructing-stack-trace-manually/

    http://msdn.microsoft.com/en-us/library/ff552143(VS.85).aspx

    This is how the functions are laid out.

    int main(int argc, char *argv[])

           {

           initialize_program(argc, argv);

     

           if (process_ftp_request())

                  {

                  // log error

                  return(1);

                  }

     

           return(0);} 

     

    I would expect the stack to look like:

      FtpPutFile

       process_put_command

        process_an_ftp_command

         process_script_array

          execute_commands

           process_ftp_request

            main 

     

    0:000> r

    eax=00000000 ebx=009eb0f0 ecx=00000000 edx=081d0001 esi=009eb0f0 edi=00001000

    eip=5051c5d0 esp=0012e474 ebp=0012e528 iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246

    thirdpartyftp!FtpUploadFileW+0xc884:

    5051c5d0 8b4008          mov     eax,dword ptr [eax+8] ds:0023:00000008=????????

    FAILURE_BUCKET_ID:  NULL_CLASS_PTR_DEREFERENCE_c0000005_thirdpartyftp.dll!FtpUploadFileW

    BUCKET_ID:  APPLICATION_FAULT_NULL_CLASS_PTR_DEREFERENCE_INVALID_POINTER_READ_thirdpartyftp!FtpUploadFileW+c884

    I am not including the call stack listing, as data from the file being FTP'd was on the call stack (overrun? parameter?).  However, I am including 'highlights' from the call stack in the hopes that someone will recognize a pattern.  I had to leave out signif portions due to size of post message.

    This is the “bottom” of the stack and appears to be ‘garbage’ – based on the program logs, ‘initialize_program' can’t be on the call stack!

    Why are there 2 occurances of kernel32!BaseProcessStart for the same thread??

     

    0:000> dds 00126000 00130000

    0012fe94  7c829f80 ntdll!CheckHeapFillPattern+0x64

    0012fe98  ffffffff

    ...

    0012fec0  fffffffe

    0012fec4  78134c58 msvcr80!free+0xec [f:\dd\vctools\crt_bld\self_x86\crt\src\free.c @ 115]

    0012fec8  781f72f0 mfc80!CObject::operator delete+0xa [f:\dd\vctools\vc7libs\ship\atlmfc\include\afx.inl @ 102]

    ...

    0012fedc  78138ced msvcr80!_except_handler4

    0012fee0  d779e223

    0012fee4  fffffffe

    0012fee8  78134c58 msvcr80!free+0xec [f:\dd\vctools\crt_bld\self_x86\crt\src\free.c @ 115]

    0012feec  781f72f0 mfc80!CObject::operator delete+0xa [f:\dd\vctools\vc7libs\ship\atlmfc\include\afx.inl @ 102]

    0012fef0  00d26d20

    0012fef4  781f2e01 mfc80!ATL::CStringData::Release+0x17 [f:\dd\vctools\vc7libs\ship\atlmfc\include\atlsimpstr.h @ 113]

    0012fef8  00d26d20

    0012fefc  00401ccd FTP_Processor!FTP_Processor::initialize_program+0x43d

    ...

    0012ff10  00401ce8 FTP_Processor!FTP_Processor::initialize_program+0x458

    ...

    0012ff6c  0042f247 FTP_Processor!__sse2_available_init+0x4f1c

    0012ff70  00000006

    0012ff74  00401857 FTP_Processor!main+0x17

    ...

    0012ff80  0042992a FTP_Processor!__tmainCRTStartup+0x10f [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 597]

    ...

    0012ffc4  77e6f23b kernel32!BaseProcessStart+0x23

    ...

    0012ffe4  77e61a60 kernel32!_except_handler3

    ...

    0012fff8  00429a6f FTP_Processor!mainCRTStartup [f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 404]

    0012fffc  00000000

    00130000  78746341

     

    This is the “middle” of the stack and from the program’s log file, makes a *little* more sense

     

     

    0012dc00  0000003a

    0012dc04  00429c00 FTP_Processor!__report_gsfailure+0x4c

    ...

    012dd2c  7c828290 ntdll!_except_handler3

    0012dd30  7c827d19 ntdll!NtWaitForMultipleObjects+0xc

    0012dd34  7c826d49 ntdll!ZwClose+0xc

    0012dd38  77e63eb3 kernel32!CloseHandle+0x59

    ...

    0012dd44  77e767c5 kernel32!UnhandledExceptionFilter+0x81a

    ...

    0012dd4c  00430022 FTP_Processor!_imp__FtpOpenDirectoryA+0x2

    ...

    0012dd58  0044004e FTP_Processor!_TI2?AVbad_allocstd+0x3ef6

     

    ...

    0012de18  77e76615 kernel32!UnhandledExceptionFilter+0x5f9

    ...

    0012de28  77e769aa kernel32!UnhandledExceptionFilter+0x856

    ...

    0012de5c  7c82a144 ntdll!RtlpAllocateFromHeapLookaside+0x13

    ...

    0012decc  77ecd204 kernel32!BasepProcessCurrentDirPlacement+0x54

    ...

    0012def0  c0000005

    0012def4  5051c5d0 thirdpartyftp!FtpUploadFileW+0xc884

    ...

    0012df60  0012df84

    0012df64  7c82b5ae ntdll!RtlpInsertFreeBlock+0x118

    ...

    0012df88  7c833a5e ntdll!RtlpExtendHeap+0x2c8

    ...

    0012dfa4  7c829f03 ntdll!RtlFreeHeap+0x1ba

    ...

    0012dfbc  7c82c7c5 ntdll!RtlAllocateHeap+0x6f7

    ...

    0012e010  7c82b5ae ntdll!RtlpInsertFreeBlock+0x118

    0012e014  0012e044

    0012e018  78132c78 msvcr80!__set_flsgetvalue+0xd

    0012e01c  0000000b

    0012e020  0012e044

    0012e024  78132e24 msvcr80!_getptd_noexit+0x72

    ...

    0012e038  7c827739 ntdll!ZwQueryVirtualMemory+0xc

    0012e03c  77e47f45 kernel32!_ValidateEH3RN+0xb6

    0012e040  ffffffff

    0012e044  77e6f248 kernel32!`string'+0x88

    ...

    0012e060  0012dd4c

    0012e064  77e6f000 kernel32!GetDriveTypeW+0x2b8

    0012e068  0012e0c0

    0012e06c  77e61a60 kernel32!_except_handler3

    0012e070  77e769b0 kernel32!`string'+0xc

    0012e074  ffffffff

    0012e078  77e769aa kernel32!UnhandledExceptionFilter+0x856

    0012e07c  77e76a20 kernel32!BaseProcessStart+0x39

    0012e080  0012e0a0

    0012e084  77e61ac1 kernel32!_except_handler3+0x61

    ...

    0012e0e4  77e61a60 kernel32!_except_handler3

    ...

    0012e0f0  0012e104

    0012e0f4  7c808bbc ntdll!RtlCallVectoredContinueHandlers+0x15

    0012e0f8  0012e18c

    0012e0fc  0012e1a8

    0012e100  7c88aa10 ntdll!RtlpCallbackEntryList

    0012e104  0012e174

    0012e108  7c83161e ntdll!RtlDispatchException+0x11f

    ...

    0012e174  0012e528

    0012e178  7c827769 ntdll!NtRaiseException+0xc

    0012e17c  7c828599 ntdll!KiUserExceptionDispatcher+0x29

    ...

    0012e198  5051c5d0 thirdpartyftp!FtpUploadFileW+0xc884

    ...

    0012e1cc  ffffffff

    0012e1d0  004146fe FTP_Processor!FTPStaticLib::process_new_list_command+0xfe

    ...

    0012e25c  0012e528

    0012e260  5051c5d0 thirdpartyftp!FtpUploadFileW+0xc884

    ...

    0012e27c  004146fe FTP_Processor!FTPStaticLib::process_new_list_command+0xfe

    ...data on call stack

    0012e474  5051cf75 thirdpartyftp!FtpUploadFileW+0xd229

    ...

    0012e48c  505117e8 thirdpartyftp!FtpUploadFileW+0x1a9c

    ...data on call stack

    0012e6a0  5053204b thirdpartyftp!FtpUploadFileW+0x222ff

    ...data on call stack

    0012eec8  5053204b thirdpartyftp!FtpUploadFileW+0x222ff

    ...

    0012f520  7c828290 ntdll!_except_handler3

    0012f524  7c82a120 ntdll!CheckHeapFillPattern+0x54

    0012f528  ffffffff

    0012f52c  00000000

    0012f530  00000000

    0012f534  5055420c thirdpartyftp!FtpUploadFileW+0x444c0

    0012f538  af704a4e

    0012f53c  0012f5c8

    0012f540  5050f1aa thirdpartyftp!FtpPutFileA+0x63

    ...

    0012f564  78137021 msvcr80!_LocaleUpdate::_LocaleUpdate+0x14

    0012f568  00ce9fe8

    0012f56c  78166924 msvcr80!_ismbcspace_l+0x12

    ...

    0012f590  781669df msvcr80!_ismbcspace+0xb

    ...

    0012f59c  781f49da mfc80!ATL::CStringT<char,StrTraitMFC_DLL<char,ATL::ChTraitsCRT<char> > >::TrimLeft+0x13

    ...

    0012f5b0  004177d6 FTP_Processor!FTPStaticLib::parse_next_argument+0x116 //odd- this is called several times in process_put_command, but shouldn’t be on stack when thirdpartyftp!FtpPutFile is called.

    ...

    0012f5cc  00415eff FTP_Processor!FTPStaticLib::process_put_command+0x13f

    ...

    0012f5e8  781f6bd9 mfc80!ATL::CStringT<char,StrTraitMFC_DLL<char,ATL::ChTraitsCRT<char> > >::Format

    ...

    0012f5f0  0012f64c

    0012f5f8  78137d42 msvcr80!_mbscmp+0xf

    ...

    0012f610  00000003

    0012f614  00411e14 FTP_Processor!FTPStaticLib::process_an_ftp_command+0x3d4

    ...

    0012f644  00000000

    0012f648  781f3e4f mfc80!ATL::CStringT<char,StrTraitMFC_DLL<char,ATL::ChTraitsCRT<char> > >::CStringT<char,StrTraitMFC_DLL<char,ATL::ChTraitsCRT<char> > >+0xd

    0012f64c  781f6bd9 mfc80!ATL::CStringT<char,StrTraitMFC_DLL<char,ATL::ChTraitsCRT<char> > >::Format

    0012f650  004118b4 FTP_Processor!FTPStaticLib::process_script_array+0x124

    ...

    0012f680  00000001

    0012f684  00404855 FTP_Processor!FTP_Processor::execute_commands+0x85

    ...

    0012f6a8  00000000

    0012f6ac  0040224f FTP_Processor!FTP_Processor::process_ftp_request+0x3bf

    0012f6b0

    • Moved by billb08 - MSFT Friday, November 5, 2010 7:56 PM related to Question about Windbg, not VS debugger (From:Visual Studio Debugger)
    Thursday, October 28, 2010 10:28 PM

All replies

  • More info:

    This is on a prod server and use of gflags.exe for pageheaps is out of the question.  It is unlikely that we could reproduce this in a MO/test environment. 

    I know that when we attempted to deliver the file to the endpoint, that they are ‘doing something’ with the file before it is fully delivered and that is causing the issue.

    I need to know if the dump is worthless, or if I can gain any useful info from it.

    My hope by posting this is:

    1.) to get a meaningful error message out of the dump that would indicate what they are doing.

    2.)If our code is causing the corruption – to correct it.

    3.) If the third party ftp DLL is not handling the issue/causing it, to have them correct it.

    Also, as we have had other dumps form this program where the stack looks similar (multiple initialize_program items on the stack) and I have been unable to figure out what the issue is.

    Thursday, October 28, 2010 10:29 PM
  • Hi,

    Your issue falls into a category that we are not able to resolve using the forums. There are various

    support options such as advisory and per issue. Please visit the below link to see the various paid

    support options that are available to better meet your needs.

    http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone

    Alliance and Premier VSIP membership includes a complimentary MSDN subscription, which includes 4

    professional support incidents. These can be used to initialize a support request with Microsoft's

    Customer Support Services. Some versions of Visual Studio include a number of free support incidents

    as well. See the "Technical Support Incidents" topic for details.


    bill boyce
    Monday, November 8, 2010 1:57 PM
  • Hi,

    !heap -s -v    will verify the heap and show corruption(s)

    You could try this, trix for finding "real stack "behind" the exception"  (it has helped me a several times):
    --------
    0:001> ~*e s -d poi(@$teb+8) poi(@$teb+4) 1003f   <== or 1001f

    you should have the addressed where to find the context record.
    Do a `.cxr <address>;kb` on the addressed returned by the previous command,
    and look at them for possible problems/errors.
    -----------
    copied from:
    http://groups.google.com/group/microsoft.public.windbg/browse_thread/thread/e0270232f2560e5e/08b20827422f3d42?hl=no&lnk=gst&q=real+stack+%22behind%22+the+exception#08b20827422f3d42

    Regards
    Kjell Gunnar

    Tuesday, November 9, 2010 12:27 PM
  • Kjell,

    Thanks for the info.

    I ran the following.  Does the 'bad' on the four rows indicate an issue?  If so, is there a way to get to the source that did the allocation?

    Is there a document that explains what each of the columns in the summary means?

    0:000> !heap -s -v   
    LFH Key                   : 0x3a103ff6
      Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
                        (k)     (k)    (k)     (k) length      blocks cont. heap
    -----------------------------------------------------------------------------
    .00150000 00000002    1024    248    248     10     2     1    0      0   L 
    .00250000 00008000      64     12     12     10     1     1    0      0     
    ..00390000 00001002    1088    108    108     38     4     1    0      0   L 
    .003d0000 00001002      64     12     12      4     1     1    0      0   L 
    ...003f0000 00001002    3136   1196   1204    124    27     4    0      0   L 
    .009e0000 00001002      64     52     52      9     1     1    0      0   L 
    .009f0000 00001002      64     16     16      2     2     1    0      0   L 
    .00a00000 00001002      64     40     40      7     1     1    0      0   LFH
    .00a10000 00001002      64     40     40      7     2     1    0      0   L 
    .00b20000 00001003     256     24     24      3     1     1    0    bad     
    .00b60000 00001003     256      4      4      2     1     1    0    bad     
    .00ba0000 00001003     256      4      4      2     1     1    0    bad     
    .00be0000 00001003     256      4      4      2     1     1    0    bad     
    .00d40000 00000002    1024     20     20      4     1     1    0      0   L 
    .00f60000 00001002      64     52     52     14     3     1    0      0   L 
    .00f70000 00001002      64     48     48     40     2     1    0      0   L 
    .00f80000 00011002     256     12     12      4     1     1    0      0   L 
    .00fc0000 00001002      64     16     16      8     1     1    0      0   L 
    .00fd0000 00001002      64     12     12      4     1     1    0      0   L 
    .00fe0000 00001002      64     52     52     18     2     1    0      0   L 
    ..01110000 00001002    1088     72     72     27     6     1    0      0   L 
    .01120000 00001002      64     52     52     18     2     1    0      0   L 
    .01130000 00001002      64     52     52     18     2     1    0      0   L 
    .01140000 00001002      64     52     52     18     2     1    0      0   L 
    .01150000 00001002      64     64     64     41     1     0    0      0   L 
    ..01160000 00001002    1088    340    340     68     2     1    0      0   L 
    -----------------------------------------------------------------------------

    I also ran your other suggestion.  It didn't yield any different info than what I got from the !analyze -v command.  I still have incomplete stack info.

    0:000> ~*e s -d poi(@$teb+8) poi(@$teb+4) 1003f
    0012c908  0001003f 00000000 00000000 00000000  ?...............
    0:000> .cxr 0012c908;kb
    eax=00000000 ebx=00000000 ecx=8004301a edx=00910001 esi=009e4538 edi=00000000
    eip=5051ca4c esp=0012cbd4 ebp=0012cbe8 iopl=0         nv up ei pl zr na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246
    thirdpartyftp!FtpUploadFileW+0xcd00:
    5051ca4c 837b1068        cmp     dword ptr [ebx+10h],68h ds:0023:00000010=????????
      *** Stack trace for last set context - .thread/.cxr resets it
    ChildEBP RetAddr  Args to Child             
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0012cbe8 50512b15 009e4538 00000000 00000000 thirdpartyftp!FtpUploadFileW+0xcd00
    0012ccb0 00000000 00000000 00000000 00000000 thirdpartyftp!FtpUploadFileW+0x2dc9

    Tuesday, November 9, 2010 3:45 PM
  • Hi
    Your heap looks OK for me, don’t know what they mean by “bad” in the lock count (maybe it’s a hex value),
    but it’s usually present in otherwise healthy processes.
    When trying to reconstruct the stack, you must realize that on x86 there are numerous calling sequences and some of them don’t
    maintain the “EBP   to previous EBP” pattern.
    Take a closer look on the  MSDN “Manually Walking a Stack”, and try to follow the stack backwards from ESP.

    <            sample from my latest received dump here           >
    eax=00000000 ebx=0012cd68 ecx=0012cd40 edx=7c90eb94 esi=00000000 edi=7ffd4000
    eip=7c90eb94 esp=0012cd40 ebp=0012cddc iopl=0         nv up ei pl zr na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
    ntdll!KiFastSystemCallRet:
    7c90eb94 c3              ret
    0:000> dds 0012cd40 l3
    0012cd40  7c90e9ab ntdll!ZwWaitForMultipleObjects+0xc
    0012cd44  7c8094f2 kernel32!WaitForMultipleObjectsEx+0x12c
    0012cd48  00000002
    0:000> ub 7c90e9ab
    ntdll!ZwWaitForMultipleObjects:
    7c90e99a 90              nop
    < some lines cut here>
    ntdll!ZwWaitForMultipleObjects:
    7c90e99f b80e010000      mov     eax,10Eh
    7c90e9a4 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
    7c90e9a9 ff12            call    dword ptr [edx]    <-- Yep here is a call
    7c90e9ab c21400     ret     14h                     <-- and 7c90e9ab is most likely the return address
    7c90e9ae 90              nop

    Regards

    Kjell GUnnar

    Wednesday, November 10, 2010 7:13 PM