locked
Out of Memory Exception RRS feed

  • Question

  • I have written a .NET Framework 1.1 Component which is called from either an ASP.NET Form or a command line program. Using the command line option, the Component is called repeatedly, once for each file in a folder, each of which is processed and uploaded into a SQL Server database. The command line program uses a ThreadPool to process multiple files in parallel.

    My problem is that the command line program apparently leaks! (I understand this isn't supposed to happen anymore, but it is!) After processing through a large number of files (maybe 100 or more) -- which might take several hours on my development XP machine -- the command line program terminates with an Unhandled Out of Memory Exception.

    Tracking the process Virtual Bytes counter while the program is executing, I can see Virtual Bytes increasing during the course of execution.

    I am looking for some ideas on how to find and fix this problem.
    Note:

    (1) I have the Framework 1.1 Service Pack applied

    (2) I would watch the .CLR Memory counters during execution if I could, but they seem to have been wiped out when I installed the VS Studio 2005 Beta
    Wednesday, November 2, 2005 10:34 PM

Answers

  • The key words you mentioned are "files" and "threads".  Its very likely that you are leaking memory because you are not properly disposing of the System.IO.Stream for the file you have open. 

    The GC will reclaim memory only for objects with no root reference.  This means that the object cannot be referenced by any live objects in the AppDomain.  If you're opening the files in another thread in such a way that the Stream never goes out of scope or looses its root reference, then the GC will not be able to reclaim it. 

    When working with streams always try to use the temporary memory usage pattern.  This pattern is simply the use of the "using(resource) { ... }" block in C# (.NET 1.0+), the "Using resource ... End Using" block in VB.NET (.NET 2.0) or simply a Try...Finally block which explicitly calls the Dispose() method on the stream.



    void Start()
    {
       foreach(FileInfo fi in new DirectoryInfo(somePath).GetFiles())
       {
          using(Stream s = fi.Open(FileMode.Open))
          {
             // ... do work

          } // at this point the Stream s goes out of scope and the "using" block calls Dispose()
       }
    }

     

    Friday, November 4, 2005 5:59 AM

All replies

  • The key words you mentioned are "files" and "threads".  Its very likely that you are leaking memory because you are not properly disposing of the System.IO.Stream for the file you have open. 

    The GC will reclaim memory only for objects with no root reference.  This means that the object cannot be referenced by any live objects in the AppDomain.  If you're opening the files in another thread in such a way that the Stream never goes out of scope or looses its root reference, then the GC will not be able to reclaim it. 

    When working with streams always try to use the temporary memory usage pattern.  This pattern is simply the use of the "using(resource) { ... }" block in C# (.NET 1.0+), the "Using resource ... End Using" block in VB.NET (.NET 2.0) or simply a Try...Finally block which explicitly calls the Dispose() method on the stream.



    void Start()
    {
       foreach(FileInfo fi in new DirectoryInfo(somePath).GetFiles())
       {
          using(Stream s = fi.Open(FileMode.Open))
          {
             // ... do work

          } // at this point the Stream s goes out of scope and the "using" block calls Dispose()
       }
    }

     

    Friday, November 4, 2005 5:59 AM
  • Ron,

    That is a good thought, but I don't think that is the problem. I will double-check though.

    The worker thread calls a Component method to process the file, passing a file name string. The File Stream Open and Close are called entirely within the scope of the Component Method. All private objects allocated inside the Component should go out of scope when the Component Method returns.

    In theory.

    The leaking only occurs when I use a ThreadPool to process multiple files currently.

    If the File Stream was the problem, I would think I'd be having the same leak behavior when I run through a folder serially, which is another option in the same program. Looping through the contents of folder serially -- without the ThreadPool -- works steady as a rock.

    I am investigating another possibility suggested by a different, but possibly related thread. This has to do with the Dataset object not being garbage collected when it goes out of scope. If they were not being garbage collected, the Dataset objects that are created would create a significant leak.

    Since the file data is read into a dataset object, and from there into SQL Server, that idea looks plausible. I had previously tried calling the dataset Dispose() method explicitly, but the remark in the other thread is that Dispose does nothing in this case. (???) That got me to thinking.

    While there are still files to process, my thread Dispatcher routine re-drives the same set of worker threads over and over again. Is it possible there is something about the ThreadPool that affects dereferencing of some (bulky) object? Maybe GC cannot tell for sure that this or that object is no longer referenced since the worker thread never really terminates. An explicit call to Dispose() should set things right, but what if the Dispose() Method is a stub? From the other thread, you have to subclass Dataset & issue a GC.ReRegisterForFinalize(Me) in the Constructor.

    I am getting ready to try that.

    I am also looking into the use of SOS to see what is actually going on inside. If I have to guess about which private objects are resisting gc in this set of circumstances, I will be here until Easter.

    Thanks.
    Friday, November 4, 2005 3:39 PM
  • Hi Mark,

     I had a same problem in my solution, whenever I called the dataset.dispose method I got the Out of memory exception.. Can you please help me on this..?

     

    Regards,

    Sambath R

    Tuesday, March 20, 2007 10:07 AM