none
Need help interpreting System.OutOfMemoryException issue RRS feed

  • Question

  • Hello,

    I have ASP.NET application failing with OutOfMemoryException and have memory dump for that application. Memory dump size is 1.4 GB.
    Some usefull statistics I got out of memory dump is.
    It appears that GC .NET heap is ~320 MB in size, total .NET heap size ~1.1 GB, Virtual Memory used is ~1.7 GB, largest available block is 45 MB.
    What can I make out of it?

    I run also through DebugDiag leak tracking and output suggested stuff below.

    ONETUTIL.DLL is responsible for 259.33 MBytes worth of outstanding allocations
    iisutil.dll is responsible for 35.79 MBytes


    1. !address -stat
        TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
       46767000 ( 1154460) : 55.05%    65.68%    : RegionUsageIsVAD
       14b4e000 (  339256) : 16.18%    00.00%    : RegionUsageFree
        e6b1000 (  236228) : 11.26%    13.44%    : RegionUsageImage
        397b000 (   58860) : 02.81%    03.35%    : RegionUsageStack
          e6000 (     920) : 00.04%    00.05%    : RegionUsageTeb
       12c26000 (  307352) : 14.66%    17.48%    : RegionUsageHeap
              0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
           1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
           1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
           1000 (       4) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
           Tot: 7fff0000 (2097088 KB) Busy: 6b4a2000 (1757832 KB)
    -------------------- Type SUMMARY --------------------------
        TotSize (      KB)   Pct(Tots)  Usage
       14b4e000 (  339256) : 16.18%   : <free>
       13f82000 (  327176) : 15.60%   : MEM_IMAGE
        14b3000 (   21196) : 01.01%   : MEM_MAPPED
       5606d000 ( 1409460) : 67.21%   : MEM_PRIVATE
    -------------------- State SUMMARY --------------------------
        TotSize (      KB)   Pct(Tots)  Usage
       549fc000 ( 1386480) : 66.11%   : MEM_COMMIT
       14b4e000 (  339256) : 16.18%   : MEM_FREE
       16aa6000 (  371352) : 17.71%   : MEM_RESERVE
    Largest free region: Base 6e630000 - Size 02c80000 (45568 KB)
    
    
    0:183> !eeheap -gc
    Number of GC Heaps: 4
    ------------------------------
    Heap 0 (00116a58)
    generation 0 starts at 0x05ee5cf4
    generation 1 starts at 0x05ea5774
    generation 2 starts at 0x02890038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    29da42b0 7b45382c  7b46a268 0x00016a3c(92732)
    0012b208 7a72c42c  7a74d308 0x00020edc(134876)
    00120b28 790d5588  790f4b38 0x0001f5b0(128432)
    02890000 02890038  05eebd00 0x0365bcc8(56999112)
    Large object heap starts at 0x12890038
     segment    begin allocated     size
    12890000 12890038  1348dfd0 0x00bfdf98(12574616)
    Heap Size  0x42b0b28(69929768)
    ------------------------------
    Heap 1 (00117f50)
    generation 0 starts at 0x0a88b9f4
    generation 1 starts at 0x0a88b6e4
    generation 2 starts at 0x06890038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    06890000 06890038  0a88fff4 0x03ffffbc(67108796)
    Large object heap starts at 0x14890038
     segment    begin allocated     size
    14890000 14890038  16692358 0x01e02320(31466272)
    Heap Size  0x5e022dc(98575068)
    ------------------------------
    Heap 2 (001190a0)
    generation 0 starts at 0x0ba0ad30
    generation 1 starts at 0x0ba0ad24
    generation 2 starts at 0x0a890038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    0a890000 0a890038  0e383a20 0x03af39e8(61815272)
    Large object heap starts at 0x16890038
     segment    begin allocated     size
    16890000 16890038  17a25c80 0x01195c48(18439240)
    Heap Size  0x4c89630(80254512)
    ------------------------------
    Heap 3 (0011ae68)
    generation 0 starts at 0x114666d8
    generation 1 starts at 0x1144d080
    generation 2 starts at 0x0e890038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    0e890000 0e890038  1146a6e4 0x02bda6ac(45983404)
    Large object heap starts at 0x18890038
     segment    begin allocated     size
    18890000 18890038  1984a920 0x00fba8e8(16492776)
    Heap Size  0x3b94f94(62476180)
    ------------------------------
    GC Heap Size  0x128d13c8(311235528)
    
    
    !dumpheap -stat 
    790fd8b4   103660      5804960 System.Collections.Hashtable
    7910234c   291035      6984840 System.Collections.ArrayList
    7a7603a4   476632      7626112 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
    79122414    62205     11792668 System.Int32[]
    7912254c   329494     12627804 System.Object[]
    7912273c    11855     20559796 System.Byte[]
    79122610   103752     22895616 System.Collections.Hashtable+bucket[]
    000ffa08      329     59644340      Free
    790f9244   680630    101597984 System.String
    Wednesday, October 14, 2009 7:26 PM

Answers

  • I don't think the ArrayList is the problem, or switching to List<T> will solve anything directly.  It will, however, help you diagnose the problem.

    Unfortunately, finding the specific list that isn't being unrooted, when you have 291,000+ of them, is difficult.  If you're using strongly typed lists, instead of ArrayList, you'll at least know what type of list (by what it's containing) is eating the most memory, which in turn will help you narrow down where the problem lies in your code.

    Switching to List<T> from ArrayList will, however, help you maintain your codebase over time.

    The issue is one of two things:

    1) Your application is just large enough (and has enough users) at this point that you're hitting hte limits of 32 bit applications.  If this is the case, your choices are either to move to 64bit hardware, or to split your application up into tiers so each tier can run separately, on a separate system, to avoid the limits.

    2) Your application is holding onto some objects, at some point, that are storing large amounts of memory.  Its not ever releasing them.  Hashtables or ArrayLists declared as static, or sitting in a singleton, are typical culprits.  You'll need to make sure that these objects are cleared out at some point, or set to null, so that they are found by the GC as unrooted.



    Reed Copsey, Jr. - http://reedcopsey.com
    • Marked as answer by artisticcheese Wednesday, October 14, 2009 8:55 PM
    Wednesday, October 14, 2009 8:54 PM
    Moderator

All replies

  • Well, you're getting this because you're hitting the upper cap of what works in 32bit .NET applications, which is somewhere between about 1.3-1.6gb of RAM.  Once you hit that point, .NET will start spewing OutOfMemoryExceptions on you.

    The easiest solution is to move your ASP.NET app to 64bit Windows, and throw more memory at it.

    However, looking at your !dumpheap, there is some interesting things there.  It looks like you have a whole lot of Hashtable and ArrayList instances.  Using ASP.NET, you should consider moving these to the generic Dictionary<TKey,TValue> and List<T> classes, as having the type safety will make your life easier, but also make it easier to track which of your lists and hashes is being allocated so many times.

    My guess is you're creating hashtables and arraylists like crazy, and rooting them somehow.  Right now, you've got 103660 Hashtable instances on the heap...

    Reed Copsey, Jr. - http://reedcopsey.com
    Wednesday, October 14, 2009 7:38 PM
    Moderator
  • 1. But those ArrayList and HashTables total take only 12 MB worth of size. How this cause memory exhaustion?
    2. I understand that I hit 2 GB limit but I don't understand where all this memory went to. I understand .NET heap use 1.1 GB out of it but where is the rest?
    Wednesday, October 14, 2009 8:03 PM
  • 1) The ArrayList and HashTable instances take up 12MB worth of size.  However, if they are holding references to other objects, those object's memory will not be included in this total.  My guess is that they're what is holding many of the 680630 string references which are taking up a lot more memory, as well as the hash table buckets (23 MB), etc... The 12MB is just the memory used by the arraylist and hashtable alone, but not what they're "holding"...  Chances are, what they're holding is what is eating up your memory.

    2) .NET heap is 1.1, GC heap is 320mb.  Between the two, you're at 1.4GB.  You'll find that, for .NET, there isn't a 2GB cap - in reality, it's more like a 1.4GB cap (usually between 1.3-1.6, depending on the application style).  Unfortunately, that's a limitation of .NET - I run into this a lot... Right at about 1.4GB, you start hitting out of memory errors.  You'll find that this is a very common issue .


    Reed Copsey, Jr. - http://reedcopsey.com
    Wednesday, October 14, 2009 8:09 PM
    Moderator
  • So you do you see something out of ordinary or just another heavily used ASP.NET application?
    I mean do I troubleshoot this and try to find out what is inside those Arrays etc and fix code etc or try to move to 64 bit (which might or might not be possible).
    Wednesday, October 14, 2009 8:19 PM
  • Well, as I mentioned in the first message-

    The best thing you could do, from what I see, is probably migrate your Hashtable instances to Dictionary<TKey,TValue>, and your ArrayList to List<T>.

    There is really no reason to use the non-generic collection classes, since you're targeting .NET 2+ (you have to for ASP.NET).

    This has many, many benefits.  It'll help you in this situation, since you'll more easily see which of the specific dictionaries and lists is causing a problem (each type of <T> in the List<T> instances will show up as a separate type in !dumpheap -stat).  This will make it much, much easier to track down and debug.

    It also provides you compile time type safety, much better performance with value type collections, etc.  This would be my first step to debug this.

    Reed Copsey, Jr. - http://reedcopsey.com
    Wednesday, October 14, 2009 8:28 PM
    Moderator
  • Well, problem is that this is SharePoint and hence I doubt I would be able to do much if it's SharePoint code. I have to talk to developers and check if they are using those Collections for anything.
    So you feel this exccessive use of ArrayList etc is causing memory to be hold when not needed anymore? Will Generics not exhibit this behavior?

    There is nothing in finalize queue, would that mean that issue is not what is seems to be?

    0:183> !finalizequeue
    SyncBlocks to be cleaned up: 4
    MTA Interfaces to be released: 0
    STA Interfaces to be released: 0
    ----------------------------------
    ------------------------------
    Heap 0
    generation 0 has 1 finalizable objects (37306904->37306908)
    generation 1 has 21 finalizable objects (373068b0->37306904)
    generation 2 has 6698 finalizable objects (37300008->373068b0)
    Ready for finalization 1 objects (37306908->3730690c)
    ------------------------------
    Heap 1
    generation 0 has 1 finalizable objects (374345ac->374345b0)
    generation 1 has 0 finalizable objects (374345ac->374345ac)
    generation 2 has 7529 finalizable objects (3742d008->374345ac)
    Ready for finalization 1 objects (374345b0->374345b4)
    ------------------------------
    Heap 2
    generation 0 has 5188 finalizable objects (37360f7c->3736608c)
    generation 1 has 0 finalizable objects (37360f7c->37360f7c)
    generation 2 has 2013 finalizable objects (3735f008->37360f7c)
    Ready for finalization 0 objects (3736608c->3736608c)
    ------------------------------
    Heap 3
    generation 0 has 3 finalizable objects (372d27b4->372d27c0)
    generation 1 has 9 finalizable objects (372d2790->372d27b4)
    generation 2 has 5930 finalizable objects (372ccae8->372d2790)
    Ready for finalization 0 objects (372d27c0->372d27c0)
    Wednesday, October 14, 2009 8:31 PM
  • I don't think the ArrayList is the problem, or switching to List<T> will solve anything directly.  It will, however, help you diagnose the problem.

    Unfortunately, finding the specific list that isn't being unrooted, when you have 291,000+ of them, is difficult.  If you're using strongly typed lists, instead of ArrayList, you'll at least know what type of list (by what it's containing) is eating the most memory, which in turn will help you narrow down where the problem lies in your code.

    Switching to List<T> from ArrayList will, however, help you maintain your codebase over time.

    The issue is one of two things:

    1) Your application is just large enough (and has enough users) at this point that you're hitting hte limits of 32 bit applications.  If this is the case, your choices are either to move to 64bit hardware, or to split your application up into tiers so each tier can run separately, on a separate system, to avoid the limits.

    2) Your application is holding onto some objects, at some point, that are storing large amounts of memory.  Its not ever releasing them.  Hashtables or ArrayLists declared as static, or sitting in a singleton, are typical culprits.  You'll need to make sure that these objects are cleared out at some point, or set to null, so that they are found by the GC as unrooted.



    Reed Copsey, Jr. - http://reedcopsey.com
    • Marked as answer by artisticcheese Wednesday, October 14, 2009 8:55 PM
    Wednesday, October 14, 2009 8:54 PM
    Moderator
  • So you do you see something out of ordinary or just another heavily used ASP.NET application?
    I mean do I troubleshoot this and try to find out what is inside those Arrays etc and fix code etc or try to move to 64 bit (which might or might not be possible).

    Hello Lorry

    To find out what's inside the Arrays or Hashtables:

    Please first run !dumpheap -type <the type of Array or Hashtable>. If there are a lot of Arrary or HashTable objects, the output will be very long. To restrict the output number, you can specify the start and end search address. Please refer to the doc of !dumpheap for more details: !help dumpheap

    Pick some Array or Hashtable objects from the output of !dumpheap -type. Run !do <address of the Array or Hashtable object> to see the details of the object. You will know what's inside it.

    To find the gc root of the array or Hashtable objects:

    Run !gcroot <address of the Array or Hashtable object>

    Finding out who is keeping so many objects alive is useful for troubleshooting this out-of-memory issue.


    Regards,
    Jialiang Ge
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Thursday, October 15, 2009 4:27 AM
    Moderator