none
Memory Leak .Net application RRS feed

  • Question

  • Hi All,

    I am facing serious problem with my application which is under production and would request immediate assistance in analyzing the memory dump.

    Operating system: windows 2003,
    Front End : .Net application along with cystal report / business objects
    Users access the aspx webpages for transactions.
    Back End : SQL database
    IIS 6.0

    Behavior : w3wp.exe consuming high memory , after some time users application hang , they would not be able to submit any data.
    In order to over come the problem , we are killing w3wp process , so that new instance gets started and everything works for certain time.

    Here is the memory analysis done so far

    ========================================================================================
    Phase 1:
    The issue is not related to .Net application because the GC Heap size is not increased:
    GC Heap Size  0x47b90a0(75206816) -- 74 MB ~

    ======================================================================
    Phase 2:
    checking for virtual alloc or heap alloc allocations. .Net memory uses virtualalloc but, heap is higher than vad.( so concentrating on native heap)
    Pct(Tots)          Pct(Busy)                      Usage
                             29             Regionusageisvad
                            59             regionusageheap

    ==========================================================================
    Phase 3:
    Analyzing the debug diag output:
    I see the below dll's from the debug diag output which are suspicious:

    rptdefmodel.dll == 66 MB outstanding allocations
    rptdefmodel!dllgetclassobject+3e722: 60.11 MB
    rptdefmodel+64c6: 6.09 MB

    Ntdll.dll
    ntdll!rtlinitializecriticalsectionandspincount+63 == 26.MB
    ntdll!rtlcreateheap+5fc:== 186.kb.

    System Up-Time   5 day(s) 12:40:08
    Process Up-Time   03:03:21

    The complete memory was built up in 3 hours 3 minutes.( check in process up-time )

    outstanding allocation summary:
    Number of allocations   4,820,935 allocations
    Total outstanding handle count   257 handles
    Total size of allocations   157.06 MBytes
    Tracking duration   01:21:44

    During the leak track turned on , we have allocated 157.MB and 257 handles , so rptdefmodel.dll should be compared to 157 MB
    Leaktrack has been enabled for 1 hour 21 minutes out of 3 hours.

    ==================================================================
    Phase 4:
    I have verified the perfmon counter on w3wp.exe process for #bytes in all heaps vs. private bytes

    Private byte was at peak ( 100 in perfmon) for certain time frame 30 % and then decreased where as #bytes in all heaps was having varing output ( i see a increase and decrease in graph ( 100 - 0 again 100 - 0 ) behvavior

    ================================================================
    Phase 5:
    Assemblies information:
    we havent created any application layer domain so default CLR created 3 app domain
    Default are: system domain, shared domain and Domain with specific name
    Below output shows the assemblies which have got loaded:

    0:000> !dumpdomain
    --------------------------------------
    Domain 2: 000fe218
    LowFrequencyHeap: 000fe23c
    HighFrequencyHeap: 000fe288
    StubHeap: 000fe2d4
    Stage: OPEN
    SecurityDescriptor: 001031a8
    Name: /LM/W3SVC/1/Root/application name
    Assembly: 000a7bf8

    Shared Assemblies which are stored in GAC:
    system.web.dll
    system.dll
    System.Configuration.dll ( MSIL )
    System.Xml.dll
    Microsoft.JScript.dll]
    System.Transactions.dll
    System.EnterpriseServices.dll
    System.Drawing.dll
    System.Windows.Forms.dll
    CrystalDecisions.ReportAppServer.ClientDoc.dll
    CrystalDecisions.ReportAppServer.ClientDoc.dll
    CrystalDecisions.ReportAppServer.DataSetConversion.dll
    CrystalDecisions.ReportAppServer.DataDefModel.dll
    CrystalDecisions.ReportAppServer.Controllers.dll]
    CrystalDecisions.ReportAppServer.CommLayer.dll
    CrystalDecisions.ReportAppServer.Controllers.dll
    CrystalDecisions.ReportAppServer.CubeDefModel.dll
    CrystalDecisions.ReportAppServer.ReportDefModel.dll
    BusinessObjects.Licensing.KeycodeDecoder.dll
    CustomMarshalers.dll
    CrystalDecisions.ReportAppServer.XmlSerialize.dll
    System.Web.Services.dll
    CrystalDecisions.ReportSource.dll
    CrystalDecisions.Enterprise.Framework.dll

    Assemblies loaded in Private Assemblies which are stored onto disk
    App_Code.DLL
    DataAccess.DLL
    System.Data.dll
    SystemBase.DLL
    CrystalReports.Engine.dll
    CrystalDecisions.Shared.dll
    App_global.asax.DLL
    System.Web.Mobile.dll
    System.ServiceModel.dll
    System.ServiceModel.dll
    System.Web.RegularExpressions.dll
    CrystalDecisions.Web.dll
    SMDiagnostics.dll]
    App_web_xxxx ( our custom files)

    ================================================================
    Phase 6:
    All the assemblies are loaded from the disk and all the assemblies have appropriate path.

    I am hitting a block from here, please assist me:
    as per debug diag output it say that rptdefmodel.dll is consuming 66 MB and ntdll!rtlinitializecriticalsectionAndspincount is occupying 26MB , so I have also analyze another debug diag output where the same rptdefmodel is consuming memory

    So how should I proceed to understand where is the leak ??
    I ran clrstack and saw most of the functions belong to crystalreport , as I said earlier crystalreport is embedded into our webapp,
    Am I going in right track ? please suggest.

    Tuesday, January 26, 2010 2:07 PM

Answers

  • I don't know exactly how to collect the call stacks as I didn't have to track down native memory leak in couple of years myself. However Tess's blog (http://blogs.msdn.com/tess/archive/2010/01/14/debugging-native-memory-leaks-with-debug-diag-1-1.aspx) shows call stacks, so I guess it is possible to get them. Did you read that article? Did you use her steps?

    If you don't have PDBs, you can still get partial call stacks. And as I mentioned, just check out which component/DLL calls into rptdefmodel.dll. Also you can for sure find public PDBs on Microsoft symbol server (search the net).
    If DebugDiag doesn't work, you can always log every call stack by setting conditional breakpoint in windbg, but that will take a lot of time and work. I would strongly suggest you to try to make DebugDiag work.

    Was your analysis correct? ... I don't know for sure, but it looks sound. Phase #5 and #6 are irrelevant IMO to the memory leak, but that is not important.

    -Karel
    • Marked as answer by SamAgain Tuesday, February 9, 2010 4:05 AM
    Wednesday, January 27, 2010 6:24 PM
    Moderator

All replies

  • Check out Tess's latest blog post on investigating native memory leaks (http://blogs.msdn.com/tess/archive/2010/01/14/debugging-native-memory-leaks-with-debug-diag-1-1.aspx).

    Get the callstack(s) which allocate in Crystal report DLL (rptdefmodel.dll) and see which component is calling there. Then follow up with owner of that component and/or with Crystal Reports to see if it is a known problem and if it is a memory leak in Crystal Report or in the caller into Crystal Report (or even higher in the stack).

    -Karel

    Tuesday, January 26, 2010 5:03 PM
    Moderator
  • Hi Karel,

    Thank you very much for your response, i am deep digged into this problem and not finding a way.

    "Get the callstack(s) which allocate in Crystal report DLL (rptdefmodel.dll) and see which component is calling there. "

    can you give me some hints / pointers how to get this , because i do not have the pdb files for rptdefmodel.dll , is there any other way through windbg to check for the call stack and disassemble them.

    Another question == Was my analysis correct ? also i am running SQL server on the same server as IIS where i have deployed .net app.

    Please help me
    Wednesday, January 27, 2010 2:04 PM
  • I don't know exactly how to collect the call stacks as I didn't have to track down native memory leak in couple of years myself. However Tess's blog (http://blogs.msdn.com/tess/archive/2010/01/14/debugging-native-memory-leaks-with-debug-diag-1-1.aspx) shows call stacks, so I guess it is possible to get them. Did you read that article? Did you use her steps?

    If you don't have PDBs, you can still get partial call stacks. And as I mentioned, just check out which component/DLL calls into rptdefmodel.dll. Also you can for sure find public PDBs on Microsoft symbol server (search the net).
    If DebugDiag doesn't work, you can always log every call stack by setting conditional breakpoint in windbg, but that will take a lot of time and work. I would strongly suggest you to try to make DebugDiag work.

    Was your analysis correct? ... I don't know for sure, but it looks sound. Phase #5 and #6 are irrelevant IMO to the memory leak, but that is not important.

    -Karel
    • Marked as answer by SamAgain Tuesday, February 9, 2010 4:05 AM
    Wednesday, January 27, 2010 6:24 PM
    Moderator