none
GC doesn't free up memory RRS feed

  • Question

  • I have got two PCs. They differ in performance, memory, etc, but both are XP SP2 machines running .NET Framework 2.0. I am working on a VB.NET application, using VS 2005.

    When I run my application on one PC (the old one), it seems to work. But when I run the exact same application on the other PC, it runs about a minute and then it runs out of memory. How is this possible?

    I have basic knowledge about the GC, but still cannot figure out the difference in behaviour. Let's assume I consume unmanaged memory in my application (I use third party libraries that are SUPPOSED to be managed). But then the memory shouldn't be deallocated by the GC on the first PC, right?

    On the machine where things seem to work, the auto GC runs like every 10 sec (gen 0).
    On the machine where things does not work, it runs once after like 10 secs and not again until the app crashes.

    The objects that are created, at a rate of approx 1 per sec, are quite large, about 15-20 MB. They must be since they consist of image data that are analysed. But they are truly temporary and run out of scope at the End Sub statement. And it is not like the GC does not keep up in speed, because on the old, low-performance machine with less memory, everything works.

    If I run GC.Collect(0) once a sec, it keeps memory usage at a steady level on both machines. But this isn't good practice...

    So I am confused. Could the .NET CLR have become corrupt on the second PC? I have run the repair procedure on the .NET Framework 2.0, without any improvement.

    Thanks.

    A former Javaman
    Monday, December 8, 2008 1:46 PM

Answers

  • Large objects (> 85 K) go to the Large Object Heap (LOH), and they are on Generation 2. Only with generation 2 (full collections) will memory be decommitted to the OS.  So clearing Gen0 does no good in this case. Take a look at the LoH article below for the list of conditions that will cause automatic collection.

    The error is that the GC cannot commit (reserve) the memory from the OS. How much memory is free on the problem machine when this happens? Is windows managing the memory? Are there any "memory managers" on the pc?

    If you think that the 3rd party utils may be unmanaged or using unmanaged resources, you can use Memory Pressure to deal with them. For example, if the code uses a com object that is 25MB, .NET is only storing the reference (a few KB), causing an incorrect memory size to be known by the GC. AddMemoryPressure will tell the GC that the com object is really using 25MB of memory and will allow the GC to perform its operations using the right amount.



    Check this article for more details on the Loh. http://msdn.microsoft.com/en-us/magazine/cc534993.aspx

    Check this one on Memory Pressure for unmananged objects. http://blogs.msdn.com/jmeier/archive/2006/10/09/Performance-Guideline_3A00_-Use-AddMemoryPressure-while-consuming-Unmanaged-Objects-through-COM-Interop.aspx




    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Monday, December 8, 2008 2:48 PM
  • Don't arbitrarily add memory pressure. That should be used when you are working with unmanaged resources, which it sounded like you may be doing (because you arent sure of the 3rd party component). It is meant to allow you to more correctly inform the GC of allocation needs that are outside the CLR. Adding 500MB at the start if you only have 200 available is going to make the GC behave more erraticly.

    If you have a COM component that loads a 16 MB file, then you would Add 16MB of memory pressure before loading the com object, and then release the 16 MB after the component has unloaded.

    The goal is not to control the garbage collection process, but to make sure it has all the information it needs to run properly.

    To see if the 3rd party component is a managed or unmanaged resource, you can try a few things.
     - Open the assembly in reflector to see if any unmanaged dll's are requested. Check the methods that you are using. See if there are any late bound calls to COM or other components.  If it is calling COM / ServicedComponent items, is the 3rd party code calling Dispose() on the objects?
     - Open the assembly with dependency walker to see if it references any COM / non .NET / unmanaged dll's.

    What is the goal of the application? What is it doing fine on one machine and not fine on another? Is it using FTP, opening text files, processing database records?

    Based on the results of the CLR per counters and the size of the objects, it sounds like there is indeed some unmanaged resources that need to be cleaned up. Do the objects have a .Dispose() method, or can you call ServicedComponent.Dispose(obj) on it?

    You may also need to contact the 3rd party manufacturer for help with this.


    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Tuesday, December 9, 2008 1:37 PM
  • Images are not stored in managed memory, they take unmanaged heap space.  That's why the Image class has the Dispose() method.  The unmanaged memory will be released when the garbage collector and finalizer runs.  But you'll get into trouble when your program doesn't allocate enough *managed* memory.  Explicitly calling Dispose() is required to stay out of trouble.
    Hans Passant.
    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Tuesday, December 9, 2008 2:42 PM
    Moderator
  • That doesn't sound good.  If this 3rd party library is written in unmanaged code, it is simply leaking memory and there's nothing you can do about it.  However, if it recommends you call GC.Collect(), there's a spark of hope that it is just poorly designed managed code.  Calling GC.Collect() is however a poor workaround.  It will force managed objects that you created into a later generation.  Where they'll clog up memory until the gen1 or 2 collection happens.  In other words, it will cause *your* code to perform worse.  This is a choice between a rock and a hard place, one you shouldn't be forced to make.  You have justification to complain.
    Hans Passant.
    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Tuesday, December 9, 2008 5:35 PM
    Moderator
  • Hmya, that's nonsense.  That's what IDisposable and the Dispose() method was designed to solve.  GC.Collect() is your only workaround, I guess.
    Hans Passant.
    • Marked as answer by magja Thursday, December 11, 2008 8:54 AM
    Wednesday, December 10, 2008 3:48 PM
    Moderator
  • I agree that it is nonsense.

    For this library, you'll probably have to call GC.Collect() and follow it with GC.WaitForPendingFinalizers(). The second call will block until all possible unmanaged memory has been released.

    Either way, this is a very poorly designed library.

           -Steve
    • Marked as answer by magja Thursday, December 11, 2008 8:54 AM
    Wednesday, December 10, 2008 7:59 PM
  • That would make me want to roll my own. How good can that stuff be if they are doing that?
    • Marked as answer by magja Thursday, December 11, 2008 8:54 AM
    Wednesday, December 10, 2008 9:05 PM

All replies

  • Large objects (> 85 K) go to the Large Object Heap (LOH), and they are on Generation 2. Only with generation 2 (full collections) will memory be decommitted to the OS.  So clearing Gen0 does no good in this case. Take a look at the LoH article below for the list of conditions that will cause automatic collection.

    The error is that the GC cannot commit (reserve) the memory from the OS. How much memory is free on the problem machine when this happens? Is windows managing the memory? Are there any "memory managers" on the pc?

    If you think that the 3rd party utils may be unmanaged or using unmanaged resources, you can use Memory Pressure to deal with them. For example, if the code uses a com object that is 25MB, .NET is only storing the reference (a few KB), causing an incorrect memory size to be known by the GC. AddMemoryPressure will tell the GC that the com object is really using 25MB of memory and will allow the GC to perform its operations using the right amount.



    Check this article for more details on the Loh. http://msdn.microsoft.com/en-us/magazine/cc534993.aspx

    Check this one on Memory Pressure for unmananged objects. http://blogs.msdn.com/jmeier/archive/2006/10/09/Performance-Guideline_3A00_-Use-AddMemoryPressure-while-consuming-Unmanaged-Objects-through-COM-Interop.aspx




    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Monday, December 8, 2008 2:48 PM
  • Thanks, Andrew. I have done some further testing (and reading).

    If I force a full garbage collection once per sec the memory usage (according to task manager) is constant. I make the following conclusion; There are no unmanged memory leaks and hence, the problem is solely related to the LOH. Correct?

    But the question is why the GC doesn't fire? I have no memory managers running, it is a new, clean PC with XP SP2. When the application crashes, the amount of free memory (according to task manager) is between 100-200 MB (of 2 GB).

    Of course there might be unmanaged memory in use that I should inform the GC about, using AddMemoryPressure, but if I for example add a pressure of 500 MB at application start, the behaviour is still the same. Which I think is strange, I would have expected the GC to fire at an earlier stage.

    A former Javaman
    Monday, December 8, 2008 8:59 PM
  • Now I have been trying to diagnose my problem looking at .NET CLR memory performance counters. But I am only getting more confused.

    When I run my application (where the memory gets eaten up without the GC stepping in), the counters tell me the following:

    Allocated Bytes/s = /Average/ = ~22 kB
    Allocated Bytes Tot = ~2.2 MB
    Amount of Bytes all heaps = ~500 kB

    At the same time the system total memory decreases approx 15 MB/s!

    These observations is totally contradictory to my previous conclusions...

    So, to sum it up:
     
    1. (Previous post) Firing a full GC all the time keeps memory usage constant. => No unmanaged memory leaks.
    2. (This post) There is almost no (relatively speaking) managed memory allocations. => There must be unmanaged memory leaks.

    Thanks.
    A former Javaman
    Tuesday, December 9, 2008 9:10 AM
  • Don't arbitrarily add memory pressure. That should be used when you are working with unmanaged resources, which it sounded like you may be doing (because you arent sure of the 3rd party component). It is meant to allow you to more correctly inform the GC of allocation needs that are outside the CLR. Adding 500MB at the start if you only have 200 available is going to make the GC behave more erraticly.

    If you have a COM component that loads a 16 MB file, then you would Add 16MB of memory pressure before loading the com object, and then release the 16 MB after the component has unloaded.

    The goal is not to control the garbage collection process, but to make sure it has all the information it needs to run properly.

    To see if the 3rd party component is a managed or unmanaged resource, you can try a few things.
     - Open the assembly in reflector to see if any unmanaged dll's are requested. Check the methods that you are using. See if there are any late bound calls to COM or other components.  If it is calling COM / ServicedComponent items, is the 3rd party code calling Dispose() on the objects?
     - Open the assembly with dependency walker to see if it references any COM / non .NET / unmanaged dll's.

    What is the goal of the application? What is it doing fine on one machine and not fine on another? Is it using FTP, opening text files, processing database records?

    Based on the results of the CLR per counters and the size of the objects, it sounds like there is indeed some unmanaged resources that need to be cleaned up. Do the objects have a .Dispose() method, or can you call ServicedComponent.Dispose(obj) on it?

    You may also need to contact the 3rd party manufacturer for help with this.


    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Tuesday, December 9, 2008 1:37 PM
  • Images are not stored in managed memory, they take unmanaged heap space.  That's why the Image class has the Dispose() method.  The unmanaged memory will be released when the garbage collector and finalizer runs.  But you'll get into trouble when your program doesn't allocate enough *managed* memory.  Explicitly calling Dispose() is required to stay out of trouble.
    Hans Passant.
    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Tuesday, December 9, 2008 2:42 PM
    Moderator
  • Andrew, nobugz, thanks.

    I thought that the GC didn't release unmanaged memory, but I suppose I was wrong, it just doesn't SEE it, right?

    I have done some more digging. The 3rd party library handles, as you say, image data in unmanaged space. I basically use the API like this...

    imageTool.InputImage = myImage
    imageTool.RunAnalysis()
    mySecondImage = imageTool.OutputImage

    ...but even if all objects are Shared (VB.NET), i.e., reused all the time, system memory is consumed at a high rate (~ 15MB per analysis). This must depend of some memory duplication, etc, done by the 3rd party stuff, right? But since I cannot reference this "invisible" data I cannot explicitely dispose it.

    The manufacturer of this 3rd party library suggests calling GC.Collect() every fourth time the code runs....it actually says so in the documentation!
    A former Javaman
    Tuesday, December 9, 2008 4:07 PM
  • That doesn't sound good.  If this 3rd party library is written in unmanaged code, it is simply leaking memory and there's nothing you can do about it.  However, if it recommends you call GC.Collect(), there's a spark of hope that it is just poorly designed managed code.  Calling GC.Collect() is however a poor workaround.  It will force managed objects that you created into a later generation.  Where they'll clog up memory until the gen1 or 2 collection happens.  In other words, it will cause *your* code to perform worse.  This is a choice between a rock and a hard place, one you shouldn't be forced to make.  You have justification to complain.
    Hans Passant.
    • Marked as answer by magja Thursday, December 11, 2008 8:55 AM
    Tuesday, December 9, 2008 5:35 PM
    Moderator
  • Below a quotation from our 3rd party API supplier.
    I have replaced their product name with X, don't want to throw garbage at them :-)

    This company is world leading in their area, i.e., machine vision algorithms, but perhaps not .NET.

    " The .NET Framework provides automatic memory management that allocates and releases memory as your application runs. However, X uses large blocks of unmanaged memory to store acquired images. Since the .NET Framework is not aware of the unmanaged memory, it is possible to use up the unmanaged heap before the .NET Framework runs the garbage collector. To prevent this problem you must periodically invoke the garbage collector yourself to free both the managed and unmanaged memory for images that are no longer being used. "

    Thanks for the feedback!


    A former Javaman
    Wednesday, December 10, 2008 3:21 PM
  • Hmya, that's nonsense.  That's what IDisposable and the Dispose() method was designed to solve.  GC.Collect() is your only workaround, I guess.
    Hans Passant.
    • Marked as answer by magja Thursday, December 11, 2008 8:54 AM
    Wednesday, December 10, 2008 3:48 PM
    Moderator
  • I agree that it is nonsense.

    For this library, you'll probably have to call GC.Collect() and follow it with GC.WaitForPendingFinalizers(). The second call will block until all possible unmanaged memory has been released.

    Either way, this is a very poorly designed library.

           -Steve
    • Marked as answer by magja Thursday, December 11, 2008 8:54 AM
    Wednesday, December 10, 2008 7:59 PM
  • That would make me want to roll my own. How good can that stuff be if they are doing that?
    • Marked as answer by magja Thursday, December 11, 2008 8:54 AM
    Wednesday, December 10, 2008 9:05 PM
  • Thanks for your help! Now I am going to have a serious chat with our supplier.


    A former Javaman
    Thursday, December 11, 2008 8:53 AM