realloc not allocating additional memory

Question

Hi,

I am using VS2012 32 bit and the cpp compiler. Windows 8.1, I have 16GB of RAM.

uiOK = realloc( uiOK, (uiMax * sizeof( UINT) ) )

uiMax is increased in 50k increments. uiOK returns NULL when uiMax is at 4,200,000

Sometimes it works, more often it does not. When it fails, 3.9GB are in use and 11.5GB are available. Anyone got any pointers as to what is going on? Any help appreciated.

 

Answer 1

rockmate, is Pavel's answer clear to you?

Answer 2

I am using VS2012 32 bit and the cpp compiler. Windows 8.1, I have 16GB of RAM.

uiOK = realloc( uiOK, (uiMax * sizeof( UINT) ) )

uiMax is increased in 50k increments. uiOK returns NULL when uiMax is at 4,200,000

Sometimes it works, more often it does not. When it fails, 3.9GB are in use and 11.5GB are available.

In addition to the 32-bit address space limitations already mentioned. and explained in
some detail at these links:

Memory Limits for Windows Releases
http://msdn.microsoft.com/en-us/library/aa366778.aspx

RAM, Virtual Memory, PageFile and all that stuff
http://members.shaw.ca/bsanders/WindowsGeneralWeb/RAMVirtualMemoryPageFileEtc.htm

note that realloc/malloc/calloc must acquire their allocations as a single contiguous block
of memory. If realloc can expand the current allocation by adding contiguous memory to
the end of the current allocation - assuming that it's free - then allocating a new chunk
of the complete requested size may not be necessary. But if the trailing memory space after
the current allocation isn't free, then a new allocation of the requested size must be made
and the contents of the original allocation copied to it before releasing the original
allocation. Clearly, the larger the allocation request becomes and the more fragmented
the available heap becomes, the more likely it is that the request will fail.

Possible solutions include:

(1) Try to avoid reallocating as much as possible. Try to anticipate max size needed and
allocate it initially rather than piecemeal.

(2) Increase the available address space from 2GB to 3GB/4GB by using "large address aware".

(3) Build the application as 64-bit rather than 32-bit to get an 8TB address space.

- Wayne

Answer 3

Thanks Brian, Pavel and Wayne, before I increment the memory just  before realloc returns a NULL the size of the memory for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me. With 11 GB available I would have thought realloc could have found 0.4GB of unbroken memory? Unless it has to find it within the 2GB reserved for the process assuming that this is what happens?

I have tried allocating memory to the pointers at the start but this seems to slow the program right down.

64 bit seems to be the road to go down. With SSD discs is this also not an alternative?

Answer 4

before I increment the memory just  before realloc returns a NULL the size of the memory for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me. With 11 GB available I would have thought realloc could have found 0.4GB of unbroken memory? Unless it has to find it within the 2GB reserved for the process assuming that this is what happens?

64 bit seems to be the road to go down. With SSD discs is this also not an alternative?

>before I increment the memory just  before realloc returns a NULL the size of the memory
>for this and another pointer amount to 0.4GB. I am well within the 2GB allocated to me.

2GB is the max memory for *everything* in the program: code, variables, constants. etc.,
not just the malloced/realloced memory. Also as I indicated before, if what *is* available
is fragmented then there may not be a *contiguous* block large enough to satisfy the
request - even if there is enough free in total (in many free blocks each of which is too
small for the request.)

>With 11 GB available I would have thought realloc could have found 0.4GB
>of unbroken memory? Unless it has to find it within the 2GB reserved for
>the process

It has to be found *within* the 2GB address space of your program. It wouldn't matter if
you had 100GB available on the machine - the allocation is from *virtual* memory.

>64 bit seems to be the road to go down

Have you set the project property: Linker->System: Enable Large Addresses

to

Support Addresses Larger Than 2 Gigabytes (/LARGEADDRESSAWARE)

>With SSD discs is this also not an alternative?

An alternative how? Feel free to use temporary disc files instead of heap if that's what
you want. Or play around with memory-mapped files:

Managing Memory-Mapped Files
https://msdn.microsoft.com/en-us/library/ms810613.aspx

- Wayne

Answer 5

Thank you

Answer 6

On 19/05/2015 17:31, rockmate wrote:

I am using VS2012 32 bit and the cpp compiler. Windows 8.1, I have 16GB of RAM.

uiOK = realloc( uiOK, (uiMax * sizeof( UINT) ) )

uiMax is increased in 50k increments. uiOK returns NULL when uiMax is at 4,200,000



Sometimes it works, more often it does not. When it fails, 3.9GB are in use and 11.5GB are available.

In addition to what others already suggested (for example, to move to a 64-bit build), I'd like to add that since you are using C++, you may want to use C++ convenient tools and idioms instead of old C functions like realloc().

In particular, you may want to use std::vector and its resize() method (I don't know about your particular scenario; maybe vector::reserve() could be a valid option as well).

This has several advantages, including automatic memory cleanup thanks to std::vector's destructor, making it simpler to write exception-safe code, and a simpler programming interface as well.

For example, compare your line of code with the following simpler one:

// vector<UINT> v;
v.resize(uiMax);

Giovanni

Answer 7

Hi Giovani,

thanks, I actually use MFC for the GUI and 'C' for the other stuff. But that does seem easier. Is there anywhere in windows where I can see the memory being used by the APP? In 'processes' in task manager I am nowhere near 4GB but I am having a hard time.

Answer 8

Is there anywhere in windows where I can see the memory being used by the APP?

You could try:

Process Explorer v16.05
https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Example info for a running application:

- Wayne

Answer 9

In 'processes' in task manager I am nowhere near 4GB but I am having a hard time.

If you in fact do not appear to be exceeding the max limit on per process memory, note as
well that a failed heap allocation request - whether from malloc/realloc/calloc/new/new[] -
can be symptomatic of heap corruption. Issues such as buffer overruns, or using an invalid
pointer, or freeing/deleting an allocation more than once, etc. can corrupt the heap
manager (or its control data) with the result that subsequent heap requests such as an
allocation may fail because the manager's status is in an indeterminate state.

In such cases a thorough code review and use of analytics, static code checking tools, etc.
is needed to debug the program.

- Wayne