locked
realloc not allocating additional memory RRS feed

  • Question

  • Hi,

    I am using VS2012 32 bit and the cpp compiler. Windows 8.1, I have 16GB of RAM.

    uiOK = realloc( uiOK, (uiMax * sizeof( UINT) ) )

    uiMax is increased in 50k increments. uiOK returns NULL when uiMax is at 4,200,000

    Sometimes it works, more often it does not. When it fails, 3.9GB are in use and 11.5GB are available. Anyone got any pointers as to what is going on? Any help appreciated.


     

    Tuesday, May 19, 2015 3:31 PM

Answers

  • before I increment the memory just  before realloc returns a NULL the size of the memory for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me. With 11 GB available I would have thought realloc could have found 0.4GB of unbroken memory? Unless it has to find it within the 2GB reserved for the process assuming that this is what happens?

    64 bit seems to be the road to go down. With SSD discs is this also not an alternative?

    >before I increment the memory just  before realloc returns a NULL the size of the memory
    >for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me.

    2GB is the max memory for *everything* in the program: code, variables, constants. etc.,
    not just the malloced/realloced memory. Also as I indicated before, if what *is* available
    is fragmented then there may not be a *contiguous* block large enough to satisfy the
    request - even if there is enough free in total (in many free blocks each of which is too
    small for the request.)

    >With 11 GB available I would have thought realloc could have found 0.4GB
    >of unbroken memory? Unless it has to find it within the 2GB reserved for
    >the process

    It has to be found *within* the 2GB address space of your program. It wouldn't matter if
    you had 100GB available on the machine - the allocation is from *virtual* memory.

    >64 bit seems to be the road to go down

    Have you set the project property: Linker->System: Enable Large Addresses

    to

    Support Addresses Larger Than 2 Gigabytes (/LARGEADDRESSAWARE)

    >With SSD discs is this also not an alternative?

    An alternative how? Feel free to use temporary disc files instead of heap if that's what
    you want. Or play around with memory-mapped files:

    Managing Memory-Mapped Files
    https://msdn.microsoft.com/en-us/library/ms810613.aspx

    - Wayne

    • Marked as answer by Shu 2017 Thursday, May 28, 2015 2:07 AM
    Wednesday, May 20, 2015 8:54 AM
  • In 'processes' in task manager I am nowhere near 4GB but I am having a hard time.

    If you in fact do not appear to be exceeding the max limit on per process memory, note as
    well that a failed heap allocation request - whether from malloc/realloc/calloc/new/new[] -
    can be symptomatic of heap corruption. Issues such as buffer overruns, or using an invalid
    pointer, or freeing/deleting an allocation more than once, etc. can corrupt the heap
    manager (or its control data) with the result that subsequent heap requests such as an
    allocation may fail because the manager's status is in an indeterminate state.

    In such cases a thorough code review and use of analytics, static code checking tools, etc.
    is needed to debug the program.

    - Wayne

    • Marked as answer by Shu 2017 Thursday, May 28, 2015 2:08 AM
    Friday, May 22, 2015 12:17 AM

All replies

  • rockmate, is Pavel's answer clear to you?
    Tuesday, May 19, 2015 9:41 PM
  • I am using VS2012 32 bit and the cpp compiler. Windows 8.1, I have 16GB of RAM.

    uiOK = realloc( uiOK, (uiMax * sizeof( UINT) ) )

    uiMax is increased in 50k increments. uiOK returns NULL when uiMax is at 4,200,000

    Sometimes it works, more often it does not. When it fails, 3.9GB are in use and 11.5GB are available.

    In addition to the 32-bit address space limitations already mentioned. and explained in
    some detail at these links:

    Memory Limits for Windows Releases
    http://msdn.microsoft.com/en-us/library/aa366778.aspx

    RAM, Virtual Memory, PageFile and all that stuff
    http://members.shaw.ca/bsanders/WindowsGeneralWeb/RAMVirtualMemoryPageFileEtc.htm

    note that realloc/malloc/calloc must acquire their allocations as a single contiguous block
    of memory. If realloc can expand the current allocation by adding contiguous memory to
    the end of the current allocation - assuming that it's free - then allocating a new chunk
    of the complete requested size may not be necessary. But if the trailing memory space after
    the current allocation isn't free, then a new allocation of the requested size must be made
    and the contents of the original allocation copied to it before releasing the original
    allocation. Clearly, the larger the allocation request becomes and the more fragmented
    the available heap becomes, the more likely it is that the request will fail.

    Possible solutions include:

    (1) Try to avoid reallocating as much as possible. Try to anticipate max size needed and
    allocate it initially rather than piecemeal.

    (2) Increase the available address space from 2GB to 3GB/4GB by using "large address aware".

    (3) Build the application as 64-bit rather than 32-bit to get an 8TB address space.

    - Wayne

    Tuesday, May 19, 2015 10:53 PM
  • Thanks Brian, Pavel and Wayne, before I increment the memory just  before realloc returns a NULL the size of the memory for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me. With 11 GB available I would have thought realloc could have found 0.4GB of unbroken memory? Unless it has to find it within the 2GB reserved for the process assuming that this is what happens?

    I have tried allocating memory to the pointers at the start but this seems to slow the program right down.

    64 bit seems to be the road to go down. With SSD discs is this also not an alternative?

    Wednesday, May 20, 2015 8:12 AM
  • before I increment the memory just  before realloc returns a NULL the size of the memory for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me. With 11 GB available I would have thought realloc could have found 0.4GB of unbroken memory? Unless it has to find it within the 2GB reserved for the process assuming that this is what happens?

    64 bit seems to be the road to go down. With SSD discs is this also not an alternative?

    >before I increment the memory just  before realloc returns a NULL the size of the memory
    >for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me.

    2GB is the max memory for *everything* in the program: code, variables, constants. etc.,
    not just the malloced/realloced memory. Also as I indicated before, if what *is* available
    is fragmented then there may not be a *contiguous* block large enough to satisfy the
    request - even if there is enough free in total (in many free blocks each of which is too
    small for the request.)

    >With 11 GB available I would have thought realloc could have found 0.4GB
    >of unbroken memory? Unless it has to find it within the 2GB reserved for
    >the process

    It has to be found *within* the 2GB address space of your program. It wouldn't matter if
    you had 100GB available on the machine - the allocation is from *virtual* memory.

    >64 bit seems to be the road to go down

    Have you set the project property: Linker->System: Enable Large Addresses

    to

    Support Addresses Larger Than 2 Gigabytes (/LARGEADDRESSAWARE)

    >With SSD discs is this also not an alternative?

    An alternative how? Feel free to use temporary disc files instead of heap if that's what
    you want. Or play around with memory-mapped files:

    Managing Memory-Mapped Files
    https://msdn.microsoft.com/en-us/library/ms810613.aspx

    - Wayne

    • Marked as answer by Shu 2017 Thursday, May 28, 2015 2:07 AM
    Wednesday, May 20, 2015 8:54 AM
  • Thank you
    Wednesday, May 20, 2015 10:36 AM
  • On 19/05/2015 17:31, rockmate wrote:

    I am using VS2012 32 bit and the cpp compiler. Windows 8.1, I have 16GB of RAM.

    uiOK = realloc( uiOK, (uiMax * sizeof( UINT) ) )

    uiMax is increased in 50k increments. uiOK returns NULL when uiMax is at 4,200,000



    Sometimes it works, more often it does not. When it fails, 3.9GB are in use and 11.5GB are available.

    In addition to what others already suggested (for example, to move to a 64-bit build), I'd like to add that since you are using C++, you may want to use C++ convenient tools and idioms instead of old C functions like realloc().

    In particular, you may want to use std::vector and its resize() method (I don't know about your particular scenario; maybe vector::reserve() could be a valid option as well).

    This has several advantages, including automatic memory cleanup thanks to std::vector's destructor, making it simpler to write exception-safe code, and a simpler programming interface as well.

    For example, compare your line of code with the following simpler one:

    // vector<UINT> v;
    v.resize(uiMax);
    

    Giovanni

    Thursday, May 21, 2015 10:34 AM
  • Hi Giovani,

    thanks, I actually use MFC for the GUI and 'C' for the other stuff. But that does seem easier. Is there anywhere in windows where I can see the memory being used by the APP? In 'processes' in task manager I am nowhere near 4GB but I am having a hard time.

    Thursday, May 21, 2015 10:47 AM
  • Is there anywhere in windows where I can see the memory being used by the APP?

    You could try:

    Process Explorer v16.05
    https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    Example info for a running application:

    - Wayne

    Friday, May 22, 2015 12:07 AM
  • In 'processes' in task manager I am nowhere near 4GB but I am having a hard time.

    If you in fact do not appear to be exceeding the max limit on per process memory, note as
    well that a failed heap allocation request - whether from malloc/realloc/calloc/new/new[] -
    can be symptomatic of heap corruption. Issues such as buffer overruns, or using an invalid
    pointer, or freeing/deleting an allocation more than once, etc. can corrupt the heap
    manager (or its control data) with the result that subsequent heap requests such as an
    allocation may fail because the manager's status is in an indeterminate state.

    In such cases a thorough code review and use of analytics, static code checking tools, etc.
    is needed to debug the program.

    - Wayne

    • Marked as answer by Shu 2017 Thursday, May 28, 2015 2:08 AM
    Friday, May 22, 2015 12:17 AM