none
CLR Profiler heap graph missing 85% of the allocated objects RRS feed

  • Question

  • When looking at the Heap Graph report in CLR profiler, it appears to be missing 85%+ of the allocated objects.

    Here is a screenshot:


    The allocation graph correctly shows 174Mb in the root.

    Expected: The heap graph also shows 174Mb in the root, and shows the relevant reference paths to those allocations.

    Actual: The heap graph shows only 26Mb in the root!

    I can see 84Mb worth of strings on the heap with !dumpheap -stat, and this number also shows correctly in the "Allocation Graph". However, the Heap Graph that shows the type reference paths is missing almost all of these objects.

    This makes the Heap Graph completely useless for memory profiling, since its not actually showing references to most of the larger memory allocations.

    Can someone from the CLRProfiler team assist here? The link below has the source CLRProfiler log, and the corresponding !dumpheap -stat output.

    https://drive.google.com/folderview?id=0B8bTEQhK15d6WXZuMkppLWo1TEk&usp=sharing

    Best,

    Mike




    Mike Volodarsky

    Thursday, July 11, 2013 8:07 PM

Answers

  • 1. Sort of. dumpheap reports objects that aren't referenced but that haven't yet been collected. Try specifying -live to get only referenced objects. I'm not sure what !traverseheap includes but anyway the CLR Profilers will only show referenced objects in a heap graph.

    2. Well, you could say it is. Garbage collection is a rather intensive process, there's no need to do it all the time. The GC collects memory now and then when more memory is needed and a threshold is reached.

    3. Software can have bugs so that's a possibility. But I can't say I ever noticed something like this.

    I tested code like the following:

    for (int i = 0; i < 1000; i++) {
        float[] f = new float[1000];
    }
    
    After the for loop ends dumpheap shows that 218 float array objects are still on the heap but only one of them is "live". A dump taken with !traverseheap shows the same thing in CLR profilers, only one array is still rooted.

    Saturday, July 13, 2013 5:01 AM
    Moderator

All replies

  • Seems normal to me, the allocation graph and the heap graph are quite different. The heap graphs shows the state of the heap at a moment in time and only includes live objects. The allocation graphs shows who allocated what and it includes allocation of objects that may be dead when the heap dump is taken.
    Friday, July 12, 2013 5:32 AM
    Moderator
  • Hey Mike,

    Thanks for the reply!

    That makes sense.  My concern is that the discrepancy between the objects still on the heap and the objects considered rooted (I did verify by debugging the CLRProfiler sample) is very high.  My expectation would be that the GC would get rid of more of the objects not alive anymore. I want to eliminate the suspicion that objects that are indeed rooted are for whatever reason not being considered rooted.

    Almost every time I've used the CLRProfiler in the past, I was constantly suprised by how small the % of rooted memory reported was, compared to GC heap sizes/!dumpheap -stat. The possible options I can think of are:

    1. The "!dumpheap -stat" command and !TraverseHeap (that I used to generate the CLRProfiler log) report objects that have been already been collected by the GC.

    2. The GC is very slow to collect objects.

    3. CLRProfiler is missing roots to objects that are indeed alive and taking memory.

    Are you aware of anything that may cause this to happen? E.g. the roots reported by the SOS "!TraverseHeap" command that generated the CLRProfiler log missing some legitimate roots.

    I am making a test which will allocate a bunch of objects, then do a full gc, then do a snapshot, and make sure all objects I did not release actually are reported as rooted.  I'll post here once I have the results.

    Best,

    Mike


    Mike Volodarsky

    Friday, July 12, 2013 3:50 PM
  • 1. Sort of. dumpheap reports objects that aren't referenced but that haven't yet been collected. Try specifying -live to get only referenced objects. I'm not sure what !traverseheap includes but anyway the CLR Profilers will only show referenced objects in a heap graph.

    2. Well, you could say it is. Garbage collection is a rather intensive process, there's no need to do it all the time. The GC collects memory now and then when more memory is needed and a threshold is reached.

    3. Software can have bugs so that's a possibility. But I can't say I ever noticed something like this.

    I tested code like the following:

    for (int i = 0; i < 1000; i++) {
        float[] f = new float[1000];
    }
    
    After the for loop ends dumpheap shows that 218 float array objects are still on the heap but only one of them is "live". A dump taken with !traverseheap shows the same thing in CLR profilers, only one array is still rooted.

    Saturday, July 13, 2013 5:01 AM
    Moderator
  • Mike,

    I think you are right. I ran some of my own tests with the same conclusion. Thanks for your help!

    Best,

    Mike


    Mike Volodarsky

    Saturday, July 13, 2013 8:51 PM
  • Hi, all.  I spoke with some of the experts on our team, and would just like to confirm for you...

    SOS !dumpheap (w/out -live) and the CLRProfiler allocation graph both show everything ever allocated, whereas CLRProfiler heap graph only shows what's still live.

    It's also worth noting that using SOS to get heap info is error-prone, as it does not consider dependent handles (ConditionalWeakTableElements) or collectible assembly roots, and could theoretically be attempting to traverse the heap at unsafe points during execution (e.g., while objects are being written to).

    When possible we'd recommend using PerfView (http://www.microsoft.com/en-us/download/details.aspx?id=28567) for memory, as well as CPU, analysis or at least CLRProfiler's collection engine for memory analysis (as opposed to SOS) to avoid the issues above.

    Thanks,
    Dave

    Tuesday, July 16, 2013 10:29 PM
  • Hey David,

    Thanks for chiming in.  I was suspecting that some roots were missing from the SOS output.  

    Couple followup questions, if you dont mind, so we can finalize this:

    1. Do you have a sample command we can use with PerfView to export the CLRProfiler heap log?

    2. Can you take a look at my question here: http://social.msdn.microsoft.com/forums/windowsazure/fr-fr/b6aa9bfc-cc07-410f-b819-c6e084c9ec9f/sos-link-a-traverseheap-object-root-with-a-thread-id.  I am having trouble getting any response on the forums.

    Thanks again for your help here.

    Best,

    Mike



    Mike Volodarsky

    Tuesday, July 16, 2013 11:09 PM
  • Doing a PerfView -> Memory -> Take Heap Snapshot will bring up the dialog box and you can select the 'Clr Profiler format' checkbox.

    You can do this from the command line.  Use PerfView /? for command line options.

    Note that the Clr Profiler format is deprecated and has been removed in future versions of PerfView.

        

    Wednesday, July 17, 2013 9:57 PM