none
Works in Debug but not in Release RRS feed

  • Question

  • I have a project that uses a dll. That dll calls

           [DllImport("kernel32.dll")]
           public static extern IntPtr VirtualAlloc(IntPtr address, Int64 size, uint flag, byte flag2);

    When I compile the program in Debug and then run it everything is fine. When I compile the program in Release and then run it, it crashes. I looked at Event Viewer and see

    Framework Version: v4.0.30319
    Descriptioin: The process was terminated due to an unhandled exception
    Exception Info: System.OutOfMemoryException
    Event ID: 1026

    Interestingly, when I ran the same release on a different computer everything is fine. The system where it works has more memory. However, I would assume the debug version uses more memory than the release version. Hence, if it is really a memory issue then the debug version should crash before the release version does.

    Wednesday, February 7, 2018 10:01 PM

Answers

  • Your P/Invoke signature is wrong. VirtualAlloc should be defined like this. 

    [DllImport("kernel32.dll", SetLastError = true, ExactSpelling = true)]
    public static extern IntPtr VirtualAlloc ( IntPtr address, IntPtr size, uint flag, unit flag2 );
    

    Your original signature is passing a single byte to the call which means it is getting garbage for the other values. Probably why you're getting 0 back. Getting the last error will indicate what went wrong. Most likely it is working in debug because the debugger pads values to help detect these kinds of things.

    The second signature you posted only works in x86 situations. Trying that on an x64 system would result in similar behavior. By using IntPtr for the size it will be 4 bytes on x86 and 8 bytes on x64 which is how size_t works.


    Michael Taylor http://www.michaeltaylorp3.net

    • Marked as answer by hulamula Thursday, February 8, 2018 8:29 PM
    Thursday, February 8, 2018 6:30 PM
    Moderator

All replies

  • The usual reason for this sort of thing is that there is an uninitialised variable somewhere, and the debug version is setting it to zero while the release version is leaving it as some random number. The result is that the debug version always works but the release version only works sometimes.

    These sorts of bugs can be real swines to track down. If possible, try putting MessageBox calls at various points in the code to see if you can narrow down the search area.

    // Some code.
    
    MessageBox.Show("Message1"); // If this messagebox pops up, we know the preceeding code is okay.
    
    // More code.
    
    MessageBox.Show("Message2"); // If we get here, the "more code" must be okay.
    
    // etc.


    • Edited by Ante Meridian Wednesday, February 7, 2018 10:48 PM Didn't comment out the comment.
    Wednesday, February 7, 2018 10:31 PM
  • One more piece of information:

    The function first calls the above VirtualAlloc with VirtualAlloc(IntPtr.Zero, ((Int64)1024, 0x2000, 0x04) but that call return 0

    There is anther DllImport

           [DllImport("kernel32.dll")]
           public static extern IntPtr VirtualAlloc(IntPtr address, int size, uint flag, byte flag2);

    So after the first VirtualAlloc returns 0, it tries to call the second one where size is of type int. It uses the following call:

    VirtualAlloc(IntPtr.Zero, test, 0x1000, 0x04);

    This one also fails and then the program itself throws an exception. 

    So the amount of memory that is being allocated is rather small (1024) so this has to be something else. Maybe something dll related. Note that the system on which the program does not work in release mode is a new installation of Windows 7 x64 with all updates installed.


    Wednesday, February 7, 2018 10:34 PM
  • Thanks for the response but I do not believe that is the issue here. Our messages crossed but please see my additional information below. Basically, the exception is thrown by the program because it was not able to allocate memory. Also the release version works on one computer but not another and the variable should be used/initialized exactly the same (the error happens before user interaction). So the question is why it wasn't able to allocate a mere 1024 bytes? I suspect is has something to do with the DllImport but I am not sure.
    Thursday, February 8, 2018 2:13 AM
  • And one more piece of information. As mentioned, VirtualAlloc memory fails in Release. When I uncheck the option "Optimize code" then the Release also starts working. That's the only difference.
    Thursday, February 8, 2018 4:21 PM
  • Your P/Invoke signature is wrong. VirtualAlloc should be defined like this. 

    [DllImport("kernel32.dll", SetLastError = true, ExactSpelling = true)]
    public static extern IntPtr VirtualAlloc ( IntPtr address, IntPtr size, uint flag, unit flag2 );
    

    Your original signature is passing a single byte to the call which means it is getting garbage for the other values. Probably why you're getting 0 back. Getting the last error will indicate what went wrong. Most likely it is working in debug because the debugger pads values to help detect these kinds of things.

    The second signature you posted only works in x86 situations. Trying that on an x64 system would result in similar behavior. By using IntPtr for the size it will be 4 bytes on x86 and 8 bytes on x64 which is how size_t works.


    Michael Taylor http://www.michaeltaylorp3.net

    • Marked as answer by hulamula Thursday, February 8, 2018 8:29 PM
    Thursday, February 8, 2018 6:30 PM
    Moderator
  • CoolDadTx, that was it! Thanks a lot!!!!

    There is a 64-bit version and 32-bit version because the program first tries the 64-bit one and if that fails it calls the 32-bit version. I am not sure this actually makes any sense as the program is just compiled for 64-bit systems so wouldn't run on 32-bit systems and 64-bit systems always have the system32\kernel32.dll. Hence, having just the 64-bit one should be fine (I didn't write any of this code so I don't know the background).

    The error I got was something about not sufficient memory which didn't make any sense as I only requested 1024 bytes. Also what threw me off is that the release version was working on one system but not the other. That's why I was looking at the DLL stuff. I am not sure why it would work on one system but not the other with everything being the same. Maybe the systems have a different compiler that creates the binary from the bytecode and arranges data differently.

    Thursday, February 8, 2018 8:08 PM