none
What WMI queries to troubleshoot System.OutOfMemoryException RRS feed

  • Question

  • We have a VB.NET 3.0 SP2 Winform app that "imports" records from a file and inserts/updates rows in a SQL Server 2005 SP3 database...

    It intermittently gets OutOfMemoryExceptions after having imported a large number (sometimes 1000's)  of records.  The app catches the exception, logs info about it and then continues importing records. It continues to get exceptions for a while (perhaps for 100's of subsequent records) but then appears to resume importing OK again, without getting any further exceptions.

    But unfortunately, the problem only occurs on production systems at our customer's site... we've never been able to reproduce it on our dev systems... even if we start with the same data.

    So... I would like to instrument the app using WMI and have it create a log of memory usage as each incoming record is imported. That way, we can at least determine whether we've got a leak that's occurring gradually and increases as each record is imported (ie, an "inter-record" leak) - OR whether there's something about those particular records (ie, an "intra-record" leak). That would really help determine where to look.

    I've only used WMI once in the past to get the number or cores and installed memory: SelectQuery("Win32_Processor") followed by GetPropertyValue("NumberOfCores") and SelectQuery("Win32_PhysicalMemory") followed by GetPropertyValue("Capacity")   

    My questions:

    What WMI queries are appropriate for tracking virtual memory usage and other values that might help troubleshoot this problem?  

    Are there other "tools" that I could repeatedly invoke (and write their results to a text message log)?

    Can System.OutOfMemory be caused by a shortage of swap/disk space?  I've seen some posts that imply it can. If so, I suppose I also need to issue WMI queries for swap and/or disk space.

    Could this problem be caused by the GC not getting a chance to run (ie, a uni-processor system, etc)?  And will adding WMI queries change whether the GC gets a chance to run, and thus distort what I'm trying to monitor,  akin to the Heisenberg uncertainty principle?

    Thanks,

    DT 


    Friday, June 12, 2009 2:50 PM

Answers

  • You could use PerformanceCounters to see what's going on with the garbage collected heap.  You can see their names with Perfmon.exe.  However, doing anything at all in an OOM condition isn't likely to succeed.  It doesn't sound like you have a leak, OOM caused by leaks are permanent.  You are probably just storing too much data.  That's recoverable when the code stops processing data and object references disappear, allowing the garbage collector to make space again.

    You should be able to get an in-house repro for this, just make sure you run queries and/or process volumes of data that are at least comparable to what the production machine is doing.  Preferrably at least a factor of 5 more than the typical load.  Also beware of Server GC, it runs less frequently than Workstation GC.  Switching to a 64-bit operating system is the quick solution.

    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Thursday, June 18, 2009 11:28 AM
    Friday, June 12, 2009 4:24 PM
    Moderator

All replies

  • You could use PerformanceCounters to see what's going on with the garbage collected heap.  You can see their names with Perfmon.exe.  However, doing anything at all in an OOM condition isn't likely to succeed.  It doesn't sound like you have a leak, OOM caused by leaks are permanent.  You are probably just storing too much data.  That's recoverable when the code stops processing data and object references disappear, allowing the garbage collector to make space again.

    You should be able to get an in-house repro for this, just make sure you run queries and/or process volumes of data that are at least comparable to what the production machine is doing.  Preferrably at least a factor of 5 more than the typical load.  Also beware of Server GC, it runs less frequently than Workstation GC.  Switching to a 64-bit operating system is the quick solution.

    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Thursday, June 18, 2009 11:28 AM
    Friday, June 12, 2009 4:24 PM
    Moderator
  • Yes, I was thinking about using both WMI queries and inspecting the PerformanceCounters in the .NET CLR Memory category.  But let's assume I find the problem is that the app is simply out-running the GC... how can I allow the GC to catch-up?

    DT
    Friday, June 12, 2009 5:10 PM
  • An app can't outrun the garbage collector, it runs in response to allocation requests.  It can however keep too many objects referenced.

    Hans Passant.
    Friday, June 12, 2009 5:22 PM
    Moderator