Windgb - !Dumpheap -stat and !eeheap -gc produce "Error Requesting Heap Segment" errors RRS feed

  • Question

  • Ok so let me start out by saying I'm learning and need help.  I have been reading as much of Tess' articles as possible, i've installed her app and walked through some issues but that hasn't given me the insight needed to figure this one out.  I am trying to debug a 32 bit app that is run on a x64 box running in 32Bit IIS.  The issue is that our memory on a large asp.net 2.0 web application is growing out of control (1.5-2 gigs, then the app stops responding).  These are the steps I have taken:

    1) Looked at Performance Monitor and turned on .clr memory trace. 
        - we see that the gen 2 memory is the source of the problem.  I see that it has 1.9 gigs out of a the 2.2 gigs of reserved bits.
    2) Perform a memory dump
        - using adplus.vbs -quiet -hang -p <<PROCESSID>>
        -  the dump size is about 2.25 gig
    3) Open 32 bit windbg on the same server that is running the app
        - Set the symbol path to the microsoft server
        - Open the dump file

        - .loadby sos mscorwks
        - .load wow64exts
        - !sw
              this took a while to figure out, without doing this I couldn't get the correct version of mscorwks to load.  I can use SOS functions now.

    4) Attempt to Analyze my dump 
        - !dumpheap -stat and !eeheap -gc both seem to blow up when trying to read the heap, Anyone have any ideas why?  I'm not sure where to go from here.  I've looked at every website I can find and just dont know where to go next.  Am I maybe taking the dump incorrectly or is there something else I'm missing?

    Thanks in advance,

        0:000:x86> !dumpheap -stat
        Error requesting heap segment 0000000088010000
        Failed to retrieve segments for gc heap
        Unable to build snapshot of the garbage collector state

       - !eeheap -gc causes:
    0:000:x86> !eeheap -gc
    Number of GC Heaps: 4
    Heap 0 (00000000001b2b50)
    generation 0 starts at 0x00000000b62f1568
    generation 1 starts at 0x00000000b608c4d0
    generation 2 starts at 0x0000000010010038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    0000000003b4ad18 000000007b463c40  000000007b47a744 0x0000000000016b04(92932)
    00000000001c8820 000000007a733370  000000007a754b98 0x0000000000021828(137256)
    00000000001bb2c0 00000000790d8620  00000000790f7d8c 0x000000000001f76c(128876)
    0000000010010000 0000000010010038  000000001400fd70 0x0000000003fffd38(67108152)
    000000002c010000 000000002c010038  000000003000fdc4 0x0000000003fffd8c(67108236)
    0000000039f00000 0000000039f00038  000000003deffa7c 0x0000000003fffa44(67107396)
    0000000055950000 0000000055950038  000000005994fe98 0x0000000003fffe60(67108448)
    Error requesting heap segment 0000000088010000
    Large object heap starts at 0x0000000020010038
     segment    begin allocated     size
    0000000020010000 0000000020010038  0000000020cd3418 0x0000000000cc33e0(13382624)
    Heap Size  0x10d1a1e0(282173920)
    Heap 1 (00000000001b3c28)
    generation 0 starts at 0x00000000ba24d954
    generation 1 starts at 0x00000000ba0a824c
    generation 2 starts at 0x0000000014010038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    0000000014010000 0000000014010038  000000001800fe84 0x0000000003fffe4c(67108428)
    0000000031f00000 0000000031f00038  0000000035efffe4 0x0000000003ffffac(67108780)
    0000000041f00000 0000000041f00038  0000000045efff9c 0x0000000003ffff64(67108708)
    Error requesting heap segment 0000000080010000
    Large object heap starts at 0x0000000022010038
     segment    begin allocated     size
    0000000022010000 0000000022010038  00000000230e30f8 0x00000000010d30c0(17641664)
    Heap Size  0xd0d2e1c(218967580)
    Heap 2 (00000000001b53a0)
    generation 0 starts at 0x00000000be2ea060
    generation 1 starts at 0x00000000be0c18a0
    generation 2 starts at 0x0000000018010038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    0000000018010000 0000000018010038  000000001c00f464 0x0000000003fff42c(67105836)
    0000000008d10000 0000000008d10038  000000000cd0f7f8 0x0000000003fff7c0(67106752)
    0000000035f00000 0000000035f00038  0000000039efff88 0x0000000003ffff50(67108688)
    0000000045f00000 0000000045f00038  0000000049eff310 0x0000000003fff2d8(67105496)
    000000006d110000 000000006d110038  000000007110f624 0x0000000003fff5ec(67106284)
    Error requesting heap segment 000000008c010000
    Large object heap starts at 0x0000000024010038
     segment    begin allocated     size
    0000000024010000 0000000024010038  0000000024071008 0x0000000000060fd0(397264)
    Heap Size  0x1405e3d0(335930320)
    Heap 3 (00000000001b6630)
    generation 0 starts at 0x00000000b214863c
    generation 1 starts at 0x00000000b20cd13c
    generation 2 starts at 0x000000001c010038
    ephemeral segment allocation context: none
     segment    begin allocated     size
    000000001c010000 000000001c010038  000000002000fe9c 0x0000000003fffe64(67108452)
    0000000028010000 0000000028010038  000000002c00ff2c 0x0000000003fffef4(67108596)
    000000003df00000 000000003df00038  0000000041effd88 0x0000000003fffd50(67108176)
    0000000051950000 0000000051950038  000000005594fe94 0x0000000003fffe5c(67108444)
    Error requesting heap segment 0000000084010000
    Large object heap starts at 0x0000000026010038
     segment    begin allocated     size
    0000000026010000 0000000026010038  00000000260549b0 0x0000000000044978(280952)
    Heap Size  0x1004427c(268714620)
    GC Heap Size  0x41e8f648(1105786440)

    • Edited by Don Tan Friday, June 26, 2009 6:04 PM Fixing Thread Title bug
    Thursday, June 18, 2009 9:27 PM


  • As I said, get a good minidump, don't wait until OOM.

    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Thursday, June 25, 2009 2:14 PM
    Friday, June 19, 2009 4:14 PM

All replies

  • Without sounding trite, but what are you creating that goes into Gen2 in the code? Do you specifically gen that large of objects?

    If so, looking at why those are not being properly disposed of may be the key to better running of the app.
    William Wegerson (www.OmegaCoder.Com)
    Thursday, June 18, 2009 10:37 PM
  • I agree there are problems, thats why I'm trying to figure out what those gen 2 objects are so i can fix the code.  But I can't fugure out how to analyze the dump to figure that out becuase of the issues above.  Any guidance on how to do that?

    Friday, June 19, 2009 1:34 PM
  • Well, it doesn't look like you are going to get anywhere with the minidump you've got.  You don't have to wait until you get OOM to find a leak, leaky objects should be identifiable well before that.  Using Windbg to analyze minidumps requires wearing a pointy hat.  Consider one of the many excellent commercially available memory profilers for a friendlier user interface.
    Hans Passant.
    Friday, June 19, 2009 2:40 PM
  • Yeah, I can go that route, I was just wondering if there was something that some people with a pointy hat around here could help me with?  I have googled it, checked every group I can find, tried multiple dumps (btw these are not mini dumps, they are full dumps from what I understand) and still not avail.  Any ideas?
    Friday, June 19, 2009 2:48 PM
  • As I said, get a good minidump, don't wait until OOM.

    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Thursday, June 25, 2009 2:14 PM
    Friday, June 19, 2009 4:14 PM