none
Weird behavior calling into HeapAlloc RRS feed

  • Question

  • Hi --

     

    I'm developing a virtual machine platform that among other things allows me to call out to C++ Windows OS functions whose address I pass into it.  I have a lot of code that uses this mechanism, so I'm pretty confident that it generally functions correctly.  But I am now encountering some weird behavior when I call out to the HeapAlloc function.

    When I trace through the call, the stack looks like it is set up correctly and the stack pointer seems to be correct.  Moreover as I start to execute the HeapAlloc itself, I trace through the code execution and see that the instructions (this is the disassembly of Microsoft's HeapAlloc implementation) seem to be manipulating the stack correctly.  Here's what the first eight or so lines of code look like that the initial HeapAlloc entry point jumps to:

     7C9100C4 push 204h
    7C9100C9 push 7C9101E0h
    7C9100CE call 7C90E8CB
    7C9100D3 mov ebx,dword ptr [ebp+8]
    7C9100D6 mov dword ptr [ebp-1Ch],ebx
    7C9100D9 xor edi,edi
    7C9100DB mov dword ptr [ebp-30h],edi
    7C9100DE mov byte ptr [ebp-1Eh],0

    I step over the first two push instructions and then step into the function called on line 3.  Everything appears normal during this execution (which basically adjusts the stack frame size -- adding the 0x204 bytes that specified in the first line above before returning).  Instruction by instruction stepping through this called routine appears to execute correctly and I observe the changes to the stack induced by this code.  But when I execute the return instruction, I'm brought back to the right place in the code snippet above (it comes back to the fourth line right after the call, as it should).  But for some reason the yellow symbol that marks one's place in stepwise debugging has turned from its normal yellow to green.  And worse, if I step over the next instruction (the mov ebx,dword ptr [ebp+8] on the fourth line above), the first time through this function execution stops executing line by line and I am proceeds automatically to the next place in code that I have a breakpoint (which is fairly far away from the code I was stepping through.  So bizarre.  The second time through the function when execution tries beyond the same point in the function an exception is thrown.

    Does any of this look familiar to any of you.  And do you have any ideas for me about how to troubleshoot this problem.  I'm sure it's something I've done (it always is), but I don't even begin to have a clue on how to track this down.

    Any thoughts would be greatly appreciated.  Thanks.

    Mike

     

    Tuesday, September 14, 2010 3:48 AM

Answers