none
high percentage of time spent in GC - workaround ? RRS feed

  • Question

  • I'm facing the following problem:

    My program needs to perform some calculations that involve allocating big amounts of data. The calculations may last minutes. Upon performing calculations memory blocks are continually allocated->used->freed. (this cannot be changed)

    I have found that when the program has had initialized some structures with data, the calculations speed drastically drops down. As I later found - more time was spent in GC compared to situation when these structures were not [fully] initialized. My assumption is that GC needs more time to traverse more complicated object graph when searching for objects that could be removed from managed heap.

    In general, the calculations are running within a loop:
    for(...)
    {
      //perform single calculation
      Calculate();
    }

    Calculate call involves allocating [say] 50 MB on the heap, organized within some object hierarchy. When Calculate is run in a tight loop, it is imminent that memory is exhausted, and GC needs to be run to find free space. The problem is that when the mentioned structures are fully initialized - GC needs to spent a lot of time traversing that structures. This renders high percentages of time spent in GC and overall poor performance.

    The question is:
    Is it possible to give a hint to CLR memory manager  (or a switch when launching exe) so that it reserves huge amount of memory from operating system, so that GC would be run less frequently ? - before performing such an operation (known to be consuming large memory) ?

    Now my program heap might look like this:

    U - used
    F - free

    UUUUUUUUUUUUUUUFFFF

    I would like it to look like:

    UUUUUUUUUUUUUUUFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

    in this way single GC collection would reclaim much more free memory, allowing program to run longer without calling again into GC. Of course this would cost higher consumption of os-process memory (priate bytes), but if this would be the remedy for the problem, i'd accept it. 


    Thanks in advance,
    Michal

    Wednesday, March 10, 2010 4:26 PM

Answers

  • Hi,
    actually I do not have much choice. I use components that when being called do create new objects (and return as return-value). Exposed API does not allow to pass-in old object and reuse it's buffers or sth like this.

    But:

    I have found a reason for slowdown - and fortunately it was my fault: In the loop I was calling GC.Collect. When this statement was removed, the code begun performing almost as fast as when heap was not burdened with many objects. I believe that collections were still taking place, but only 0 and 1 generations were examined - allowing to speed up collection.  Each time the loop was passing, only new object should be collected (instantiated the last time loop has passed), so it was not necessary to inoke full collection - as I had implicitly caused.

    So, the lesson learend is that it is not reasonable (in most cases i suppose) to be smarter than GC is. And finding GC.Collect statements (and confirming that these are really necessary) in source code should be obligatory step in development process.

    Best regards,
    Michal

    Tuesday, March 30, 2010 5:10 PM
  • GC.Collect should be indeed used only for debugging purposes (when you need to test your code, etc.). Using GC.Collect in production code is a high risk that you will end up in similar situation as you did here. Just trust GC that it will do good job and investigate perf issues only when they arise and fix them properly (i.e. not by invoking GC explicitly).

    In other words: Remove all GC.Collect calls in your code. Do not try to confirm that they are necessary. They never are.

    -Karel

    Tuesday, March 30, 2010 7:41 PM
    Moderator

All replies

  • Check out Latency Modes for the garbage collector at MSDN - http://msdn.microsoft.com/en-us/library/bb384202.aspx

    GCSettings.LatencyMode = GCLatencyMode.LowLatency;
    in the System.Runtime namespace will make the collection be more relaxed during critical operations. You should reset the latency mode after your operations has finished.

    Regards,
    Mikael Svenson

    Search Enthusiast - MOSS MCTS
    http://techmikael.blogspot.com/ - http://www.comperio.no/
    Wednesday, March 10, 2010 8:55 PM
  • GCSettings.LatencyMode = GCLatencyMode.LowLatency;


    Hi Mikael,
    in my scenario it does not seem to do the trick. I cannot see expected increase in consumed memory (and acompanying speedup/{less time spent in GC}). According to the documentation, this may be remedy to deffer collection (in order to avoid it happen in a short, time critical block), provided that the code does not request more and more memory.  In my scenario i do request more and more memory (according to performance counters it is 150 MB/sec)

    Regards,
    Michal
    Thursday, March 11, 2010 11:37 AM
  • Read up on Garbage Collection Notifications at http://msdn.microsoft.com/en-us/library/cc713687.aspx. It might help you tune your application and control the garbage collection.

    Specifically experiment with the RegisterForFullGCNotification method. Hope this helps you a bit on your way.

    Another option is to allocate your huge memory objects in unmanaged memory.

    Regards,

    Mikael Svenson


    Search Enthusiast - MOSS MCTS
    http://techmikael.blogspot.com/ - http://www.comperio.no/
    Wednesday, March 24, 2010 12:40 PM
  • Do you really need to allocate on the heap every time?  This would be slow in the Win32 world too. Is it possible for you to allocate a 50MB area on the heap and re-use it in the loop rather than constantly create a new one?
    Phil Wilson
    Wednesday, March 24, 2010 5:37 PM
  • Hi,
    actually I do not have much choice. I use components that when being called do create new objects (and return as return-value). Exposed API does not allow to pass-in old object and reuse it's buffers or sth like this.

    But:

    I have found a reason for slowdown - and fortunately it was my fault: In the loop I was calling GC.Collect. When this statement was removed, the code begun performing almost as fast as when heap was not burdened with many objects. I believe that collections were still taking place, but only 0 and 1 generations were examined - allowing to speed up collection.  Each time the loop was passing, only new object should be collected (instantiated the last time loop has passed), so it was not necessary to inoke full collection - as I had implicitly caused.

    So, the lesson learend is that it is not reasonable (in most cases i suppose) to be smarter than GC is. And finding GC.Collect statements (and confirming that these are really necessary) in source code should be obligatory step in development process.

    Best regards,
    Michal

    Tuesday, March 30, 2010 5:10 PM
  • GC.Collect should be indeed used only for debugging purposes (when you need to test your code, etc.). Using GC.Collect in production code is a high risk that you will end up in similar situation as you did here. Just trust GC that it will do good job and investigate perf issues only when they arise and fix them properly (i.e. not by invoking GC explicitly).

    In other words: Remove all GC.Collect calls in your code. Do not try to confirm that they are necessary. They never are.

    -Karel

    Tuesday, March 30, 2010 7:41 PM
    Moderator
  • GC.Collect should be indeed used only for debugging purposes (when you need to test your code, etc.). Using GC.Collect in production code is a high risk that you will end up in similar situation as you did here. Just trust GC that it will do good job and investigate perf issues only when they arise and fix them properly (i.e. not by invoking GC explicitly).

    In other words: Remove all GC.Collect calls in your code. Do not try to confirm that they are necessary. They never are.

    -Karel

    In most cases I agree with you, but I there are a few cases where you have to invoke it manually, especially on 32bit systems.

    In the following scenario a manual GC will keep your application alive instead of throwing an out of memory exception.

    You allocate a lot of objects which are referenced, around 800mb (which seems to be the limit on 32bit before it gives up :) , and you have high CPU usage.

    If you release a good size of these objects after a while, say 300mb,  and instantly re-allocate new ones, then you will get an out of memory exception if you don't force a GC before the re-allocation, since the high CPU load will defer the runtime from executing a GC when you actually need it.

    But, it's very very rare that you actually need to force a GC, and it should be avoided as you say.

    Regards,

    Mikael Svenson


    Search Enthusiast - MOSS MCTS
    http://techmikael.blogspot.com/ - http://www.comperio.no/
    Wednesday, March 31, 2010 7:12 AM
  • Yes, you are right Mikael. Here's probably the best answer:
    http://blogs.msdn.com/ricom/archive/2004/11/29/271829.aspx (especially Rule #1)
    http://vineetgupta.spaces.live.com/blog/cns!8DE4BDC896BEE1AD!1104.entry?sa=807618166 (4.4.2 GC.Collect)
    4.4.2 GC.Collect
    * You typically do not know better than the GC to figure when to do a collection
    * It completely messes up the GC tuning
    * The only scenario to use it is if on some rare, non-repeating event, a lot of old objects die, and without doing GC.Collect, the GC does not collect them for a long enough period of time for it to impact the app.

    -Karel

    Wednesday, March 31, 2010 3:51 PM
    Moderator
  • There is another scenario where forced GC's are nessecary and that is if woking with unmanaged objects that allocates large amount of memory that the GC is not aware of. Add in some object pooling of those unmanaged objects and you're bound to get into trouble.

    There are indeed situations where the GC algorithm just isn't good enough. But I shouldn't complain too much. Most of the time its a wonderful feature.

    /Calle


    - Still confused, but on a higher level -
    Thursday, April 1, 2010 8:21 AM
  • There is another scenario where forced GC's are nessecary and that is if woking with unmanaged objects that allocates large amount of memory that the GC is not aware of. Add in some object pooling of those unmanaged objects and you're bound to get into trouble.

    There are indeed situations where the GC algorithm just isn't good enough. But I shouldn't complain too much. Most of the time its a wonderful feature.

    /Calle


    - Still confused, but on a higher level -

    Wouldn't you get around this by using the GC.Add/Remove-MemoryPressure, letting the GC know that you have allocated unmanaged memory?

     

    Regards,

    Mikael Svenson


    Search Enthusiast - MOSS MCTS
    http://techmikael.blogspot.com/ - http://www.comperio.no/
    Thursday, April 1, 2010 9:32 AM
  • Wouldn't you get around this by using the GC.Add/Remove-MemoryPressure, letting the GC know that you have allocated unmanaged memory?

     

    Regards,

    Mikael Svenson


    Search Enthusiast - MOSS MCTS
    http://techmikael.blogspot.com/ - http://www.comperio.no/


    With respect to the thread owner Michal, I feel I must answer this anyway.

    You are right Mikael, using Add/Remove mem pressure is suitable in this situations, but what if the library you are working with hasn't implemented this.

    I have recently worked on a project that involved image processing with a large amount of unmanaged memory allocation/deallocations. Surprisingly, I found that calling GC.Collect at specified intervals rather than making the GC aware of the unmanaged memory gave the best performance. It seemed that when the GC knew about the unmanaged memory it started making collections very frequently. Altering the GC latency did not help. I think this was because all images were short lived, allocated and then deallocated shortly after.
    It is of course in rare situations like this when you need to work the GC more manually.

    /Calle


    - Still confused, but on a higher level -
    Friday, April 2, 2010 1:30 PM
  • Wouldn't you get around this by using the GC.Add/Remove-MemoryPressure, letting the GC know that you have allocated unmanaged memory?

     

    Regards,

    Mikael Svenson


    Search Enthusiast - MOSS MCTS
    http://techmikael.blogspot.com/ - http://www.comperio.no/


    With respect to the thread owner Michal, I feel I must answer this anyway.

    You are right Mikael, using Add/Remove mem pressure is suitable in this situations, but what if the library you are working with hasn't implemented this.

    I have recently worked on a project that involved image processing with a large amount of unmanaged memory allocation/deallocations. Surprisingly, I found that calling GC.Collect at specified intervals rather than making the GC aware of the unmanaged memory gave the best performance. It seemed that when the GC knew about the unmanaged memory it started making collections very frequently. Altering the GC latency did not help. I think this was because all images were short lived, allocated and then deallocated shortly after.
    It is of course in rare situations like this when you need to work the GC more manually.

    /Calle


    - Still confused, but on a higher level -

    Third party libraries :) Can't live without them can we. And an excellent point indeed. When using components we don't have control over trial and error comes into play.

    I think this thread has discussed a lot around the GC, and proven that it's not for the faint of heart and that you need a good understanding of what's going on before starting to tune it.

    Regards,

    Mikael Svenson 


    Search Enthusiast - MOSS MCTS
    http://techmikael.blogspot.com/ - http://www.comperio.no/
    Friday, April 2, 2010 1:46 PM