locked
Mismatch between loaded module and downloaded image from symbol server

    Question

  • Hi

    A while back I posted a question about implementing ICorDebugDataTarget::ReadVirtual. Rick gave a really good answer and I've gotten quite far in my implementation. I've now run across a strange situation.

    My implementation of ReadVirutal() has been requested to read 16 bytes from address 76F61000. As per Rick's answer I scanned the memory ranges, none of which contained the address. I then scanned the loaded modules to see if there was a hit there. I found a hit with secur32.dll: image base at 76F50000 and end address 76F63000, size of image is 77824 bytes.

    I doubled checked the loaded module information using WinDbg (you'll notice my information is the same as WinDbg):

    0:034> lm v m secur32
    start    end        module name
    76f50000 76f63000   secur32    (deferred)             
        Image path: C:\WINDOWS\system32\secur32.dll
        Image name: secur32.dll
        Timestamp:        Tue Jun 16 08:59:00 2009 (4A3742B4)
        CheckSum:         0001D441
        ImageSize:        00013000
        File version:     5.2.3790.4530
        Product version:  5.2.3790.4530
        File flags:       0 (Mask 3F)
        File OS:          40004 NT Win32
        File type:        2.0 Dll
        File date:        00000000.00000000
        Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

    So my implementation of ReadVirtual() has correctly identified where the piece of memory is located. Since this is a partial memory dump I do not have secur32.dll available as part of the crash dump, which means I need to try and locate a matching copy of the file and read directly from it.

    I checked on my local disk to see if I had a matching file and I didn't, so I tried the symbol server instead. I've successfully got the symbol server to retrieve the file for me, but here's the catch: it's not the same file?

    I asked for secure32.dll with a timestamp of "4A3742B4" and image size of "00013000" (77824 bytes) to be downloaded, but the retrieved file is only 65 536 bytes when downloaded.

    The memory I've been asked to read is 16 bytes from position 69632 (0x11000) which doesn't exist since the downloaded image doesn't match what was loaded at runtime, even though I passed the correct information to the symbol server?

    So my question is: how can the downloaded image (with correct timestamp/image size) not match what was loaded at the time of the crash? This doesn't seem to be isolated to secur32.dll either, I've also noticed on: gdi32.dll, msvcrt.dll and ws2_32.dll. Asside from those few most of the downloaded files all seem correct though.

    fyi - the crash was from a Win2003 32bit box, and I'm working on a Win8 64bit box. In my naiveity I can't see that being a problem since I told the symbol server exactly which version of secur32.dll I wanted? I can't see secur32.dll having the same timestamp/image size across different platforms?

    ps - my app is configured for "Any CPU" and is running as a 32bit app.

    Cheers,

    Greg


    Edit: Make sure to read both of Noah's answers, you need to take relocations into account as well.

    • Edited by Greg Nagel Sunday, October 21, 2012 12:01 PM Extra info on answers
    Wednesday, October 10, 2012 3:22 PM

Answers

  • Hi Greg,

      I think what you are seeing is a difference between physical file offsets and RVA (Relative virtual address). When the OS loads a dll into memory it does something a little more sophisticated than copying the file byte-for-byte into virtual memory. Each dll is in PE format and the spec for that format can be found here:

    http://msdn.microsoft.com/en-us/library/windows/hardware/gg463119.aspx

    In the header portion of PE there is a table that lists sections of the file and what RVA they should be mapped to. RVA just refers to an offset in memory relative to the base load address for the module. For example assume the OS is going to load secur32.dll at 76f50000 and the table says there is a section of file starting at file offset 0x800, length 0x39 bytes, and it is mapped to RVA 0x1000. That means bytes [0x800, 0x839) of the file will located in memory at [0x76f51000, 0x76f51039). Generally the sections mapped in memory are more spread out than they were in the file (different alignment requirments). In order to satisfy the debugger's request for memory in this region you have to do the reverse of what the OS does and map the memory location back to the file offset which would have been placed there.

    The SizeOfImage field that you used to look up the image is not (as you noticed) the size of the file on disk, instead it is the size of the image after it has been mapped into the virtual address space. 'SizeOfImage' actually names a specific data field inside one of the structs in the PE header.

        In terms of implementing this I've seen two different approaches. One approach that may not always work but is quick and dirty, is to call LoadLibary on the module to get the OS to load it into the debugger process. Then you can read memory within the debugger process to see what got mapped where without understanding anything about how the OS does its job. One reason this might not work is if the OS loads the module in the debugger process at a different load address than in the debuggee. You could attempt to account for that, but dealing with relocations (see the PE spec for more details) would make this no longer very cheap.

       Another approach which I imagine is what most serious debuggers use is to replicate the OS loader semantics themselves without calling LoadLibrary. The exact mechanics of this should be covered in the PE specification so you need not guess about OS internals. This is more work than LoadLibrary but should give a more reliable solution and you don't have the side effect of loading random libraries into your debugger process.

    HTH,

      -Noah

    • Proposed as answer by Mike FengModerator Friday, October 12, 2012 8:55 AM
    • Marked as answer by Greg Nagel Sunday, October 21, 2012 12:00 PM
    Wednesday, October 10, 2012 8:14 PM
    Moderator

All replies

  • Hi Greg,

      I think what you are seeing is a difference between physical file offsets and RVA (Relative virtual address). When the OS loads a dll into memory it does something a little more sophisticated than copying the file byte-for-byte into virtual memory. Each dll is in PE format and the spec for that format can be found here:

    http://msdn.microsoft.com/en-us/library/windows/hardware/gg463119.aspx

    In the header portion of PE there is a table that lists sections of the file and what RVA they should be mapped to. RVA just refers to an offset in memory relative to the base load address for the module. For example assume the OS is going to load secur32.dll at 76f50000 and the table says there is a section of file starting at file offset 0x800, length 0x39 bytes, and it is mapped to RVA 0x1000. That means bytes [0x800, 0x839) of the file will located in memory at [0x76f51000, 0x76f51039). Generally the sections mapped in memory are more spread out than they were in the file (different alignment requirments). In order to satisfy the debugger's request for memory in this region you have to do the reverse of what the OS does and map the memory location back to the file offset which would have been placed there.

    The SizeOfImage field that you used to look up the image is not (as you noticed) the size of the file on disk, instead it is the size of the image after it has been mapped into the virtual address space. 'SizeOfImage' actually names a specific data field inside one of the structs in the PE header.

        In terms of implementing this I've seen two different approaches. One approach that may not always work but is quick and dirty, is to call LoadLibary on the module to get the OS to load it into the debugger process. Then you can read memory within the debugger process to see what got mapped where without understanding anything about how the OS does its job. One reason this might not work is if the OS loads the module in the debugger process at a different load address than in the debuggee. You could attempt to account for that, but dealing with relocations (see the PE spec for more details) would make this no longer very cheap.

       Another approach which I imagine is what most serious debuggers use is to replicate the OS loader semantics themselves without calling LoadLibrary. The exact mechanics of this should be covered in the PE specification so you need not guess about OS internals. This is more work than LoadLibrary but should give a more reliable solution and you don't have the side effect of loading random libraries into your debugger process.

    HTH,

      -Noah

    • Proposed as answer by Mike FengModerator Friday, October 12, 2012 8:55 AM
    • Marked as answer by Greg Nagel Sunday, October 21, 2012 12:00 PM
    Wednesday, October 10, 2012 8:14 PM
    Moderator
  • Great thanks Noah, that makes complete sense. Should actually have thought of that myself. I definitely prefer the 2nd approach, so I'll try it over the weekend and post my results here.

    Cheers.

    Greg

    Thursday, October 11, 2012 5:36 AM
  • Hi Noah

    I've coded my updated routine and have a few interesting things to add to this thread.

    The first thing I noticed when I started testing the new routine was that I was getting read requests for memory that didn't match up with any of the sections. It didn't take long to figure out that the requests were for PE header information (I eye-balled the addresses and they all fell within a few bytes of the ImageBase), so I updated my code to allow reads directly from the image if they fall within the "header area" (my definition of "header area" is very broad at the moment ;) ).

    The second this is more important - I have little to no know idea of how to actually test my code. The first I did was to use dumpbin.exe as a reference point. Based on that I seem to be reading all of the information correctly: headers, data directories (not that I need them in this situation), section headers & sections and even section data. Luckily dumpbin.exe does a really good job of display raw section data, so I've been able to verify my reads against that. So based on what dumpbin says as far as I'm concerned I'm reading data corectly.

    However, in my original question I posted a reference to this post in which Rick mentioned that the WinDbg "db" command should be doing the same thing I'm trying to do (if I understood him correctly). This is where I have a bit of a problem: my read matches dumpbin, but niether matches the output from "db".

    The output from "db" is:

    0:034> db 77c50000 77c50064
    77c50000  4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00  MZ..............

    While the data according to dumpbin/me is

      77C50000: C1 28 0F 84 5B B8 01 00 8B 45 DC 40 8B 7D D4 EB  Á(..[¸...EÜ@.}Ôë

    So which of us is right? Is "db" really doing what I'm trying to do? Or is "db" doing what I'm trying to do but maybe picking up a different version of the file?

    One last thing - "db" doesn't work on my local box, I just get "????" as the output. I had to run it on the 2003 server box at work to get a result (thus my comment about "db" getting different versions of the file). Although the 2003 server box I used was not the one where the crash dump originated from.

    Ps - the attempted read is for: 0x40 bytes from address 0x77c50000 in file rpcrt4.dll (timestamp = 0x4c6ba77d, SizeOfImage = 0x000a0000)

    Thanks

    Greg


    • Edited by Greg Nagel Saturday, October 13, 2012 10:43 AM Included more info on "db" not working locally.
    Saturday, October 13, 2012 10:31 AM
  • Do you know if the image was rebased? Each image has a preferred load address that is assigned statically at build time, however the OS does not have to load the image at this preferred address. I notice that windbg indicates the memory starting at 77c50000 begins with 'MZ'. That 2 byte sequence always marks the first two bytes in a PE, and combined with large alignment of this address I would guess an image has been loaded at 77c50000. The read comes from file offset 0 of whatever image has been loaded there.

    Can you check using windbg's 'lm' command to confirm that it thinks a module is starting at 77c50000. I would guess your tool and windbg have different opinions about what module is loaded at that address, and since windbg is consuming the runtime data rather than the static build-time data, most likely windbg is correct.

    You can use toolhelp snapshot to get a listing of all modules in the process and their actual load address (which may differ from the static preferred load address).

    HTH,

     -Noah

    Tuesday, October 16, 2012 10:10 PM
    Moderator
  • Thanks Noah: spot on, the module was actually loaded at 77c20000, not 77c50000. I've adjusted my code and everything seems to work fine, by which I mean it doesn't crash ;)

    My app seems to run through loading the crash dump without any obvious errors, but OpenVirtualProcess() never returns a valid ICorDebugProcess instance, it just returns 80131C44 (CORDBG_E_NOT_CLR) for every module.

    I think this is valid in this case though, as the partial dump I'm trying to load is from a CLR 2.0 process, not a CLR V4.0 process. I tried mdbg to see what it would do, and it crashed with an AccessViolation, so for now I'm going to assume my version is mostly correct. I'll get a CLR4 partial crash dump and see how that goes.

    Thanks for your help,

    Greg

    Sunday, October 21, 2012 12:00 PM
  • The OpenVirtualProcess behavior you saw was correct and by design. The only clr dlls that are recgonized are those for .Net 4.0 and higher (because 4.0 was the first version in which managed dump debugging was supported). I trust when you tried a dump using v4.0 clr.dll you no longer got that error.

    As for mdbg that is unfortunate, but I could imagine nobody ever tested it in the unsupported 2.0 dump debugging scenario to verify that it handled the error gracefully. Thanks for letting me know about it,

     -Noah

    Friday, November 23, 2012 9:32 AM
    Moderator