none
Malloc crashes my program in Vista but works fine in XP

    Question

  • Hi All,

    I have a program that call malloc. It crashes my program in windows vista. It works fine though in windows XP. I am stuck with this for few days and still havent found the solution.

    The thing is that it is the malloc() that crashes. That leaves me unable to check whether malloc successfully allocates memory or not.

    What are the possible causes that malloc crashes? I understand if free/delete crashes the program but this is first time I am facing this problem.

    I have tried several other memory allocation function as well. They all crash in vista but work fine in XP.

    Thank you for your help.

    Best regards,
    Pancilobak
    Thursday, July 30, 2009 8:37 AM

Answers

  • This error usually means you've got a memory corrupting bug. Using the debug page heap is usually the best way to track this down:

    http://support.microsoft.com/default.aspx/kb/286470

    Ignore the pageheap.exe section and go to the "Using GFLAGS With a Single Process Page Heap" section. Replace the lsass.exe in the command line with your own app. No path, just YourAppName.exe

    Gflags itself is part of the Debugging Tools for Windows which you'll have to download and install:
    http://www.microsoft.com/whdc/devtools/debugging/default.mspx

    Enable gflags page heap for your process and with luck you'll get an AV at the point of the memory corruption bug. PageHeap is designed to AV when overwritting buffers or otherwise accessing memory you don't own. In a nutshell it adds nonwritable pages bounding your allocations (among other checks) so if you touch one of those you AV. It's probably better to run a retail build as well (with PDB's so you can debug) as the debug builds add in padding when doing allocations. It can increase memory usage significantly for your app so don't forget to turn it off.
    • Marked as answer by Wesley Yao Wednesday, August 5, 2009 3:03 AM
    Saturday, August 1, 2009 1:26 AM

All replies

  • How much are you trying to allocate?

    Are you using the memory afterwards but going outside the bounds?

    Thursday, July 30, 2009 8:50 AM
  • hi,

    i just tried to allocate 16 bytes (16 chars)...

    i have check it....it didnt go outside the bounds.

    the funny thing is this crash on vista but works fine in xp.
    Thursday, July 30, 2009 8:52 AM
  • That sounds as if the problem lies somewhere else and it only manifests when you try to allocate memory.
    Can you post some sourcecode?
    Thursday, July 30, 2009 8:54 AM
  • Hi,

    Following is some of the codes:

    static int CallbackPSKFunction (gnutls_session_t a_pSession, const char *a_pcUserName, gnutls_datum_t* a_pKey)
    {
    	char temp[3];
    	int i;
    	
    	temp[2] = NULL; 
    	a_pKey->data = (unsigned char *) malloc (16);  //crash here
    if(a_pKey->data == NULL)
    return E_MEM_ERROR; for (i=0; i<16; i++) { memcpy(temp, g_pcUserKey+(i*2), 2); a_pKey->data[i] = (unsigned char) strtol(temp, NULL, 16); } a_pKey->size = 16; return E_SUCCESS; }
    the above function is a callback function pointer pass into GNUTLS's  method which is second parameter to function below:

    gnutls_psk_set_server_credentials_function(gnutls_psk_server_credentials_t cred, gnutls_psk_server_credentials_function * func);

    best regards,
    Pancilobak
    Thursday, July 30, 2009 9:01 AM
  • Vista has a different heap manager.  Quite a bit better at detecting corruption and letting you know about it than the one in XP.  You have a memory bug in your code, you just happened to get away with it in XP.  Good luck finding it.

    Hans Passant.
    Thursday, July 30, 2009 9:02 AM
    Moderator
  • Is "a_pKey" valid,

    it looks like a callback too, so is it thread safe? (multiple callbacks at same time etc)
    Thursday, July 30, 2009 9:04 AM
  • Hi,

    a_pKey is valid. In older codes, we actually checked it and it is always valid. So we removed the checking of a_pKey in above codes.

    I have verified that it is the malloc that crashes.

    Best regards,
    Pancilobak
    Thursday, July 30, 2009 9:06 AM
  • Try

    unsigned char* pData = (unsigned char *) malloc (16); 

    Does that crash? If not I suspect a_pKey is either null or pointing to some bad place...

    Thursday, July 30, 2009 9:08 AM
  • Hi,

    I tried unsigned char* pData = (unsigned char *) malloc (16); 

    it succeeded. Seems there is error with a_pKey.

    if(a_pKey == NULL)
        Utils::ShowWarningMsg("a_pKey == NULL");
    if(a_pKey->data == NULL)
        Utils::ShowWarningMsg("a_pKey->data == NULL");
    
    unsigned char* pData = (unsigned char *) malloc (16);  
    	
    a_pKey->data = pData; //still crash here
    for (i=0; i<16; i++) 
    {
       memcpy(temp, g_pcUserKey+(i*2), 2);
       a_pKey->data[i] = (unsigned char) strtol(temp, NULL, 16);
    }
    a_pKey->size = 16;

    I am not sure if it crashed during assignment of pData value to a_pKey->data...
    Thursday, July 30, 2009 9:23 AM
  • what is the value (address) of a_pKey, I suspect 0xfefefefe or similar meaning it is unitalised and coming through as garbage... where does it come from/get created?
    Thursday, July 30, 2009 9:24 AM
  • EDITED!!!

    the function is a callback function and passed into GNUTLS method.

    so the value come from GNUTLS DLL.

    Actually I wrote a small program just to test that part outside my current program. It actually crashes.

    I know the following is not safe, somehow it didnt crash in vista and xp/

    	unsigned char buf[16];
    temp[2] = NULL; a_pKey->data = buf;// DIDNT CRASH
    if(a_pKey->data == NULL)
    return E_MEM_ERROR; for (i=0; i<16; i++) { memcpy(temp, g_pcUserKey+(i*2), 2); a_pKey->data[i] = (unsigned char) strtol(temp, NULL, 16); } a_pKey->size = 16;
    I have no idea what happen
    • Edited by Pancilobak Thursday, July 30, 2009 9:57 AM
    Thursday, July 30, 2009 9:37 AM
  • Hi ALl,

    After debugging and setting some break point in VS 2008, I managed to narrow down the bug.

    I have this problem: Access violation reading location [some address] in ntdll.dll.

    It seems that it didnt crash within the callback function. Instead it crashed on handshake() method which belong to the DLL.
    Friday, July 31, 2009 9:22 AM
  • Of course the fact that it is crashing in ntdll.dll doesn't mean that is where to look for the bug. Typically the memory corruption occurs much earlier in the user code.
    Friday, July 31, 2009 3:49 PM
  • This error usually means you've got a memory corrupting bug. Using the debug page heap is usually the best way to track this down:

    http://support.microsoft.com/default.aspx/kb/286470

    Ignore the pageheap.exe section and go to the "Using GFLAGS With a Single Process Page Heap" section. Replace the lsass.exe in the command line with your own app. No path, just YourAppName.exe

    Gflags itself is part of the Debugging Tools for Windows which you'll have to download and install:
    http://www.microsoft.com/whdc/devtools/debugging/default.mspx

    Enable gflags page heap for your process and with luck you'll get an AV at the point of the memory corruption bug. PageHeap is designed to AV when overwritting buffers or otherwise accessing memory you don't own. In a nutshell it adds nonwritable pages bounding your allocations (among other checks) so if you touch one of those you AV. It's probably better to run a retail build as well (with PDB's so you can debug) as the debug builds add in padding when doing allocations. It can increase memory usage significantly for your app so don't forget to turn it off.
    • Marked as answer by Wesley Yao Wednesday, August 5, 2009 3:03 AM
    Saturday, August 1, 2009 1:26 AM
  • Definitely a memory corruption bug from my experience. Just fixed a similar problem in my code. Try to comment out writes to malloced memory earlier on just before the crashing malloc. You'll eventually get to the point that the program get's past the crashing malloc. That last commented write to malloced data will probably tell you that that written object is incorrectly malloced (probably the size was too small) or that you're indexing beyond the object.
    Tuesday, May 4, 2010 3:48 PM