none
GC still pauses whole app under SustainedLowLatency LatencyMode despite it consumes 0.25GB and another 30GBs are available RRS feed

  • Question

  • We are developing an application that needs to meet strict SLA for response times (order of milliseconds). Our app is well within those limits despite when GC pauses the app.

    We changed the app config to ensure server gc collection:

    <runtime>
        <gcServer enabled="true" />
        <gcConcurrent enabled="true" />
      </runtime>

    During the business hours we change the LatencyMode:

    GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;

    Despite those steps we still see (several times per hour) cases where the whole application is being paused for usually up to 100ms, but we saw even a case where it was stopped for 0.5 second (!).

    Our app is running on .NET 4.5 on a machine with 4 CPUs and 32GB RAM, and it is the only resources demanding app on that machine.

    Is it expected that with such a configuration GC would still pause all threads of a running application? Are there other steps (before investing to manage memory ourselves and setting GC Mode to LowLatency) to prevent GC from pausing all threads simultaneously?

    Tuesday, November 12, 2013 3:29 PM

Answers

All replies

  • Hi Jan,

    Per my understanding, you don’t want the garbage collector affect critical business. I think you are correct configuration with GC. I assume you should ignore some details. When you set GC to low sustainedlowlatency periods, generation 2 collections are background used. But full blocking collections may still occur if the system is under memory pressure. Did you create much instances so that GC enforce full blocking collections? Please refer to this below page to see how to monitor a full blocking collection occurs. http://msdn.microsoft.com/en-US/library/dd997285(v=vs.110).aspx.

    My suggestion is when you execute critical business, set GC to sustainedlowlatency mode. Other times to use the default recovery mode.

    Hope useful to you.

    Best Regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, November 13, 2013 3:32 AM
    Moderator
  • Thank you Hetro for looking into this!

    We are continuously creating a very large number of (usually quite small) objects, with majority of them being very short lived. Unfortunately we need to stick to our SLA for 12 hours a day almost continuously - that's also why the app has a very generous HW setting (it's using around 1% of available free memory).

    I tried your suggestions and here are few observations and questions:

    - GC.CollectionCount(2) number is increasing faster than number of notifications I get through WaitForFullGCComplete - is that expected? (does maybe GC.CollectionCount(2) count even background collections?)

    - According to http://msdn.microsoft.com/en-us/library/cc713687(v=vs.110).aspx the notifications should be raised only for blocking collections (Only blocking garbage collections raise notifications. When the <gcConcurrent> configuration element is enabled, background garbage collections will not raise notifications.) and according to http://msdn.microsoft.com/en-us/library/system.runtime.gclatencymode(v=vs.110).aspx full blocking collections under LowLatencySustainableMode should happen only under memory pressure - however in our case those notifications are raised few times a minute despite there is still plenty of memory available on the OS (few dozens gigs compared to ~0.25GB consumed by our process) and GC.Collect is not being explicitly called.

    - When attempting to set GCSettings.LatencyMode to GCLatencyMode.LowLatency the property still stays in GCLatencyMode.SustainedLowLatency mode - are there any scenarios when .NET prevents flipping to GCLatencyMode.LowLatency mode?

    The main concern remains - why is the GC suspending the whole process despite there is still many dozen times more available memory on the OS, compared to what is the process currently consuming?

    We will welcome and investigate any suggestions promising to improve the situation.

    Thanks and regards

    Jan

    Wednesday, November 13, 2013 9:46 AM
  • Hello,

    NET program could not hold all your memory address. I think the root cause is the large numbers of instances you are creating and mark the GC feel to do full blocking collections. Sometimes we will meet "OOM" while the operating system still has much memory left.

    See two MSDN blogs for more information.

    http://blogs.msdn.com/b/expressionencoder/archive/2010/03/10/9976633.aspx

    http://blogs.msdn.com/b/calvin_hsia/archive/2010/09/27/10068359.aspx

    Can you try to modify the business algorithms, try to avoid to create so many instances during a short time?

    Best Regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, November 14, 2013 2:25 AM
    Moderator
  • Hetro,

    Unfortunately we cannot create any less instances.

    However - our process is 64bit and according to task manager or process explorer there are still 30GBs of free available memory - so OS should not get into memory pressure (btw. is there any way how to monitor this exact situation - low memory notifications from the system?).

    If the GC is still doing blocking Gen 2 collections in low latency modes just because of large number of existing instance then it is either the GC bug or documentation bug here http://msdn.microsoft.com/en-us/library/bb384202(v=vs.110).aspx. Or is there anything we miss which can cause SustainedLowLatency mode work different than expected?

    Do you see any reason why setting LowLatency mode (GCSettings.LatencyMode = GCLatencyMode.LowLatency;) has actually no effect and Latency mode stays in SustainedLowLatencyMode?

    Thursday, November 14, 2013 7:42 AM
  • In order to provide more evidence - here is RamMap view on one of the boxes exhibiting the behavior (as it can be seen there s still plenty of unused memory left):

    RamMap View

    Thanks

    Jan

    Thursday, November 14, 2013 7:53 AM
  • Hi Jan,

    No matter how much physical memory you install in a computer. Your application will limit the actual memory available for it. See this page for more. http://www.codeproject.com/Articles/483475/Memory-Limits-in-a-NET-Process. Unfortunately we could not set ma heap size in .NET unless you host the CLR yourself in a process.

    As i know from your descripion, you app is more like SQL Server or IIS, and want to control over memor and avoid paging. I recommend you have a look at CLR Hosting APIs. Because the hosted application are hosts for other managed apps, and they control how much memory the hosted applications are allowed to have.

    Some starter info could be found here. http://msdn.microsoft.com/en-us/magazine/cc163567.aspx

    Best Regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, November 15, 2013 9:04 AM
    Moderator
  •  

    Hetro,



    I doubt CLR hosting would make any difference - there must be a much more simpler answer.
    Our app is a 64bit .NET 4.5 app on 64 bit OS - so it should have access to all the available physical memory (even according to the article you mentioned), definitely it shouldn't stop around 0.25GB.



    Let's forget about our app and let's have a look at the example here: http://msdn.microsoft.com/en-US/library/system.gc.registerforfullgcnotification(v=vs.110).aspx

    If you build it and run it as is (just changing it to 64bit and make sure to target .NET 4.5), then the working set grows quite rapidly and there are Gen 2 collections (reported through GC.CollectionCount(2)), however there is no blocking Gen 2 collection (GC.WaitForFullGCComplete() doesn't return)
    That's probably expected as by default the concurrent gen 2 garbage collection is being used (according to http://msdn.microsoft.com/en-us/library/yhwwzef8(v=vs.110).aspx)



    If you add following into the application config:

    <runtime>
        <gcConcurrent enabled="false" />
      </runtime>

    Than for each collection reported by GC.CollectionCount(2) there is a collection reported by GC.WaitForFullGCComplete() - meaning that all Gen 2 collections are blocking - this is again expected.



    Now let's focus on unexpected scenarios:



    1) Unnecessary blocking Gen2 collections during LowLatency modes

    Set the latency mode to GCLatencyMode.SustainedLowLatency or GCLatencyMode.LowLatency on the beginning of the example. You can do so by adding following code to the beginning of main:

                    Console.WriteLine("Starting... Current GC mode: {0}, Is server GC: {1}",
                           GCSettings.LatencyMode, GCSettings.IsServerGC);
                    Console.WriteLine("Setting GCSettings to SustainedLowLatency");
                    GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;
                    Console.WriteLine("Current GC mode: {0}, Is server GC: {1}",
                           GCSettings.LatencyMode, GCSettings.IsServerGC);

    You can now see that each Gen2 was a blocking Gen2 collection - this is consistent with documentation. What is not consistent is that those collections are called even if obviously not needed (plant of physical memory still available). If you place breakpoint on the line which identifies new Gen2 collection:

         // Show collection count when it increases:
         Console.WriteLine("Gen 2 collection count: {0}", GC.CollectionCount(2).ToString());

    You will find out that with LowLatency modes or without them - Gen2 collections are still being called after approximately same number of allocations.
    If you comment out lines clearing the memory (comment call of OnFullGCApproachNotify(); and OnFullGCCompleteEndNotify();), than - if behaving consistently with the documentation - during LowLatency modes app should get OutOfMemory exception after trying to perform Gen2 collection first time - as it should perform it only if there is a memory pressure from the OS - and since this example app still holds on all memory, and there is no compaction for LowLatency modes (according to msdn docs), then there is nowhere to get more memory from. Instead of that, it calls Gen2 collection quite many times.

    So again - my question is: Why is the blocking Gen2 collection called during LowLatency modes when OS is not clearly under memory pressure (I got first Gen2 collection with working set around 45MB)?



    2) Cannot set LowLatency mode for NET 4.5 app with gcServer enabled.

    Now modify the app.config so that it contains:

      <runtime>
        <gcServer enabled="true" />
      </runtime>

    And try to set LowLatency GC mode. You can try e.g. the following code:

                    Console.WriteLine("Starting... Current GC mode: {0}, Is server GC: {1}",
                           GCSettings.LatencyMode, GCSettings.IsServerGC);
                    Console.WriteLine("Setting GCSettings to SustainedLowLatency");
                    GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;
                    Console.WriteLine("Current GC mode: {0}, Is server GC: {1}",
                           GCSettings.LatencyMode, GCSettings.IsServerGC);

                    Console.WriteLine("Setting GCSettings to LowLatency");
                    GCSettings.LatencyMode = GCLatencyMode.LowLatency;
                    Console.WriteLine("Current GC mode: {0}, Is server GC: {1}",
                           GCSettings.LatencyMode, GCSettings.IsServerGC);



    The GCSetting.LatencyMode will not flip to LowLatency mode.

    So my question remains: Why GC prevents you from setting it to LowLatency mode (under server gc mode)?


    Feel free to pull in FTE responsible for the GC latency modes. We are more than willing to provide repros, code snippets, memory dumps or so.

    We cannot accept blocking of the whole process while there are still dozen gigs of free memory.

    Thanks and regards
    Jan









    Friday, November 15, 2013 2:40 PM
  • Since your app is so small could you have two instances running at the same time with only one beeing "active"? The "inactive" one can call GC.Collect and take its deserved rest while your can continue in your active app to service requests. If you want to prevent pauses caused by GC you should be preallocating your objects so at runtime nothing bad can happen anymore. For hard realtime systems you are not allowed to allocate stuff to meet the required response time. Exceptions are also a no go due to the unkown performance characteristics how long they take to unwind.
    Sunday, November 17, 2013 2:21 PM
  • We are welcoming all suggestions for improvements - In fact some of the suggested techniques are in our long term plan (multiple concurrent instances, possibly in NLB cluster). But I'm always trying to approach simpler solutions first (which - I hope - in this situation is guiding GC to use more of the available memory)

    Let's please completely forget about our app, and let's focus on some very easy example (like e.g. the demonstration code here: http://msdn.microsoft.com/en-US/library/system.gc.registerforfullgcnotification(v=vs.110).aspx) - then let's try to find answers to the following two questions:

    1) Why it is not possible to set LatencyMode to LowLatency when GC is in server mode? Is that documented somewhere?

    2) Why under LowLatency or SustainableLowLatency modes there are still same number of Gen2 collections as under Interactive mode and why is the majority of those Gen2 collections blocking?

    My next step is to use CreateMemoryResourceNotification to subscribe to system-wide event that GC also uses to trigger full collection and verify if blocking Gen2 collections under LowLatency modes are started only under this condition (as documented on msdn) - but my guess is that that wouldn't be a case with dozens of gigs of free memory available.


    Monday, November 18, 2013 11:08 AM
  • I have two thoughts on this subject that come from a GC performance experience from about 4 years ago.

    1.  How many of the objects being collected have finalizers?  If many, occasional spikes in finalizers will generate occasional spikes in GC execution time.  I am guessing this is not the case - perhaps your app just makes a lot of strings.

    2.  Does heap fragmentation deteriorate over time?  I am suggesting that this might be the case with your app.

    An experiment may help decide if 1 or 2 is a factor in your perf problem.  I suggest you measure the effect of periodic GC calls, ie you call GC rather than leave GC entirely up to .NET.  You should call Collect, WaitForPendingFinalizers, and Collect again.  Try a few variations such as how often to make the calls and parameters to Collect().  If you collect every n seconds, you expect your app to spike performance every n seconds.  A collection of such data might point you in the right direction.  For example, a GC every 2 seconds might make your problem go away.  I make no such guarantees, but this is an easy experiment to try and it might yield a hint.

    Monday, November 18, 2013 12:48 PM
  • Thanks Amercer for suggestion. We were actually performing performance measurements and that was the way how we discovered the strange behavior.

    But again - let's completely forget about our application, as that drags attention and efforts to different directions. Even on a very basic application - like this one http://msdn.microsoft.com/en-US/library/system.gc.registerforfullgcnotification(v=vs.110).aspx (please read my previous replies for more details), which allocates blocks of byte arrays (means no finalizers, no potential for fragmentation) - exhibits the same strange behavior.

    So I need to again repeat my questions that remained completely unanswered:

    1) Why it is not possible to set LatencyMode to LowLatency when GC is in server mode? Is that documented somewhere?

    2) Why under LowLatency or SustainableLowLatency modes there are still same number of Gen2 collections as under Interactive mode and why is the majority of those Gen2 collections blocking?

    Monday, November 18, 2013 6:23 PM
  • Hi Jan,

    > 1) why it is not possible to set LatencyMode to LowLatency when GC is in server mode? Is that documented somewhere?

    Enables garbage collection that is more conservative in reclaiming objects. Full collections occur only if the system is under memory pressure, whereas generation 0 and generation 1 collections might occur more frequently. This mode is not available for the server garbage collector. –MSDN

    You could find the below explanation on this page.http://msdn.microsoft.com/en-us/library/system.runtime.gclatencymode(v=vs.110).aspx.

    > 2) why under LowLatency or SustainableLowLatency modes there are still same number of Gen2 collections as under Interactive mode and why is the majority of those Gen2 collections blocking?

    If you exceed the LOH threshold, it will trigger a Gen2 collection. If you’re allocating a lot of short-lived large objects (about 85k), it will be a problem. See CLR Inside Out: Large Object Heap Uncovered.

    If possible, please post a simple project to help us to reproduce your issue. This will helpful.

    At last, here is a page about fundamentals of garbage collection. http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx#what_happens_during_a_garbage_collection

    Best Regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, November 19, 2013 9:31 AM
    Moderator
  • Great thanks Hetro for your patience and for providing answer to the first question and helpful links!

    Let's now focus on number 2)

    Is exceeding LOH threshold size mentioned somewhere as trigger for blocking gen2 collection under low latency modes?

    I created a small test that you can run under Intercative, LowLatency and SustainableLowLatency modes.

    • Under Interactive mode it triggers Gen2 collections regularly and all of them seem to be non-blocking. This is great
    • Under LowLatency mode it doesn't trigger any Gen2 collections until under low memory pressure. This is great
    • Under SustainableLowLatency it triggers Gen2 collections with the same rate as under Interactive mode, however all of them are blocking(!), also it triggers them even despite low memory condition is not met.

    It is allocating only blocks of 500 bytes - so it shouldn't touch LOH at all (apart from some interned strings), interestingly enough few hundred MB are allocated on LOH according to perfmon (but majority is still in Gen2 heap)

    And here is the fully functional test code:

    namespace GCExperiments
    {
        using System;
        using System.Collections.Generic;
        using System.Runtime;
        using System.Runtime.InteropServices;
        using System.Threading;
    
        public static class Win32Imports
        {
            public static int LowMemoryResourceNotification = 0;
    
            [DllImport("kernel32.dll", SetLastError = true)]
            public static extern IntPtr CreateMemoryResourceNotification(int notificationType);
    
            [DllImport("kernel32.dll", SetLastError = true)]
            public static extern bool QueryMemoryResourceNotification(IntPtr hResNotification, out bool isResourceStateMet);
        }
    
        public class GCModesTest
        {
            public static void Main(string[] args)
            {
                GCModesTest t = new GCModesTest();
                t.Run();
            }
    
            private List<byte[]> _load = new List<byte[]>();
            private IntPtr _hResNotification;
    
            public void Run()
            {
                Console.WriteLine(
                    "Press 'I' for test under (I)nteractive GC mode, press 'S' for test under (S)ustainableLowLatency mode and 'L' for test under (L)owLatency mode");
    
                GCLatencyMode requestedMode;
    
                switch (Console.ReadKey(true).Key)
                {
                    case ConsoleKey.I:
                        requestedMode = GCLatencyMode.Interactive;
                        break;
                    case ConsoleKey.S:
                        requestedMode = GCLatencyMode.SustainedLowLatency;
                        break;
                    case ConsoleKey.L:
                        requestedMode = GCLatencyMode.LowLatency;
                        break;
                    default:
                        Log("Unknown option, exiting");
                        return;
                }
    
                GCSettings.LatencyMode = requestedMode;
                Console.WriteLine("Current GC mode: {0}, Is server GC: {1}",
                                  GCSettings.LatencyMode, GCSettings.IsServerGC);
    
                _hResNotification = Win32Imports.CreateMemoryResourceNotification(Win32Imports.LowMemoryResourceNotification);
    
                GC.RegisterForFullGCNotification(1, 1);
                ThreadPool.QueueUserWorkItem(o => WaitForFullGCProc());
    
                RunAllocations();
            }
    
            private void RunAllocations()
            {
                try
                {
                    int lastCollCount = 0;
                    int newCollCount = 0;
                    int allocations = 0;
    
                    while (true)
                    {
                        _load.Add(new byte[500]);
                        allocations++;
                        newCollCount = GC.CollectionCount(2);
                        if (newCollCount != lastCollCount)
                        {
                            // Show collection count when it increases:
                            Log("Allocations: {0}. Gen 2 collection count: {1}, from that blocking collections: {2}",
                                allocations, GC.CollectionCount(2), _gcFullollectionNotificationsCount);
                            lastCollCount = newCollCount;
                        }
                    }
                }
                catch (OutOfMemoryException)
                {
                    Console.WriteLine("Out of memory.");
                }
            }
    
            private int _gcFullollectionNotificationsCount = 0;
    
            private void WaitForFullGCProc()
            {
                // CheckForNotify is set to true and false in Main. 
                while (true)
                {
                    // Check for a notification of an approaching collection.
                    GCNotificationStatus s = GC.WaitForFullGCApproach();
                    if (s == GCNotificationStatus.Succeeded)
                    {
                        Log("GC Approach Notification raised.");
                    }
                    else if (s == GCNotificationStatus.Canceled)
                    {
                        Log("GC Approach Notification cancelled.");
                        break;
                    }
                    else
                    {
                        // This can occur if a timeout period 
                        // is specified for WaitForFullGCApproach(Timeout)  
                        // or WaitForFullGCComplete(Timeout)   
                        // and the time out period has elapsed. 
                        Log("GC Approach Notification not applicable.");
                        break;
                    }
    
                    // Check for a notification of a completed collection.
                    s = GC.WaitForFullGCComplete();
                    if (s == GCNotificationStatus.Succeeded)
                    {
                        bool isResourceStateMet;
                        bool succeeded = Win32Imports.QueryMemoryResourceNotification(_hResNotification,
                                                                                      out isResourceStateMet);
    
                        if (!succeeded)
                        {
                            Log("Call to QueryMemoryResourceNotification failed!");
                        }
    
                        _gcFullollectionNotificationsCount++;
                        Log(
                            "GC Completed Notification raised. Count: {0}, Gc gen 2 count: {1}, Is Low memory condition: {2}",
                            _gcFullollectionNotificationsCount, GC.CollectionCount(2), isResourceStateMet);
                    }
                    else if (s == GCNotificationStatus.Canceled)
                    {
                        Log("GC Completed Notification not applicable.");
                        break;
                    }
                    else
                    {
                        // Could be a time out.
                        Log("GC Completed Notification not applicable.");
                        break;
                    }
                }
            }
    
            private void Log(string format, params object[] args)
            {
                Console.WriteLine(format, args);
            }
        }
    }

    Thanks!
    Jan

     


    • Edited by Jan Krivanek Wednesday, November 20, 2013 6:31 AM formal mistake in question
    Tuesday, November 19, 2013 5:12 PM
  • Have you looked at using a WeakReference for these objects to cache them and re-use them? Just a thought, but maybe something like this would reduce the GC collections.

    http://msdn.microsoft.com/en-us/library/system.weakreference(v=vs.110).aspx

    Code example at the bottom.


    Phil Wilson

    Monday, November 25, 2013 6:30 PM
  • Thanks Phil for suggestion - however as I mentioned I'm mainly interested in discovering why SustainableLowLatency GC mode performs blocking Gen2 collections despite there is no OS memory pressure which is in contrary with MSDN documentation here http://msdn.microsoft.com/en-us/library/system.runtime.gclatencymode(v=vs.110).aspx

    Sample code I posted here demonstrates this behavior.

    Profiling and tuning of our app is a different thing that happens in parallel with this investigation - here I want to focus solely on documented GC behavior.

    Thanks
    Jan

    Monday, November 25, 2013 7:25 PM
  • Hi Jan,

    since you are already deep into the internals I doubt there is someone outside MS who is able to answer your question. You should really open a call at MS directly and talk to the GC devs.

    Yours,

      Alois Kraus

    Monday, November 25, 2013 10:42 PM
  • Hi Hetro,

    So the sample project that you requested wasn't helpful at all?

    Should I open a an issue?

    Thanks
    Jan

    Thursday, November 28, 2013 6:58 AM
  • Hi Jan,

    As Alois has said, you have deep into the core of GC collection. It’s better to contact to our GC product group to get a more effective response. Your scenario was special one which was more like SQL Server, requires a unique garbage collection algorithm.

    Regerads,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, November 29, 2013 2:26 AM
    Moderator
  • Hi Jan,

    Did you ever an answer to your quite good questions?

    I am about to try setting latency to sustainedlowlatency, but if it is broken

    then i'd rather not waste the time.

    thanks,

    --tom

    Wednesday, May 28, 2014 6:05 PM
  • Hi Tom,

    We did some deeper research in this field (mainly we started to collect ETW events traced by GC) and we've found that SustainedLowLatency mode is indeed beneficial, however to achieve very strict goals (e.g. no app freez above certain time threshold) you need to completely refactor your app.

    Collection of Gen0 and Gen1 always take 'stop the World' approach - meaning it will freez all your threads. Under SustainedLowLatency mode, the Gen2 collections were still surprisingly happening even without existing system wide memory pressure - however it turned out that those are not the culprit of the app freezes at all. Gen0 and Gen1 collections were.

    Under those circumstances (Gen0 and Gen1 collections prooved to cause frequent freezes above 100ms), we were left out with need to completely change our memory usage patterns - small and very frequently created objects needs to be pooled; code that uses a large memory blocks but is ok with freezes needs to be pushed to separate processes etc.

    Overall - SustainedLowLatency is beneficial, but it might not be sufficient to meet your goals (don't forget to precisely set those up front).

    Jan

    Thursday, May 29, 2014 10:37 AM