locked
Filestreaming System.OutofMemoryException RRS feed

  • Question

  • Hi,
    I am getting an System.OutofmemoryException when using Filestream to load zip file. the file is 521MB. My code is:

    //Insert using Filestream, file into SQL Server Table
            private void btnInsert_Click(object sender, EventArgs e)
            {
                OpenFileDialog openFileDlg = new OpenFileDialog();
                openFileDlg.InitialDirectory = Directory.GetCurrentDirectory();
                if (openFileDlg.ShowDialog() == DialogResult.OK)
                {
                    FileInfo fi = new FileInfo(openFileDlg.FileName);
                    FileStream fs = new FileStream(fi.FullName, FileMode.Open, FileAccess.Read);
                    BinaryReader rdr = new BinaryReader(fs);
                    byte[] fileData = rdr.ReadBytes((int)fs.Length);
                    rdr.Close();
                    fs.Close();

    My program dies on this line: byte[] fileData = rdr.ReadBytes((int)fs.Length);
    i have also attached the detail for the System.OutofMemoryException.  I am runing this on a Windows 7, 64bit

    machine.

    thanks for your help.

    Sharon
    Thursday, March 8, 2012 3:05 PM

Answers

  • Only testing and time will be able tell you that. If the code is configured accordingly I would try running it in such a manner that simulates the type of environment it will be expected to operate in. E.g. if it's supposed to be able to go non-stop and semi-rigorous then try running it overnight and see if it dies with the same problem.

    It could be a result of a) Memory Leak in the application meaning you should probably try tracking it down or at least have the application restart perhaps on a daily basis. or b) It could be a bigger problem related to the overall memory usage of all of the applications on the system after a duration of say a few days or a few weeks.

    Sometimes computers or software just needs to be rebooted/restarted for optimal performance.

    If your company has the money and is willing to spend it I would recommend

    http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

    It is a great means of tracking memory usage and leaks in applications. Its 500 dollars well spent by any company where this can end up causing time and money loss later down the line and is a great tool to have in a developer's aresenal. 500 dollars spent now might save the company 10k-250k down the line.

    • Edited by MarcD Thursday, March 8, 2012 7:44 PM added memory profile links
    • Marked as answer by Leo Liu - MSFT Friday, March 16, 2012 4:28 AM
    Thursday, March 8, 2012 7:42 PM
  • Hi, 

    Also, it is advisable wrapping your stream with using statement to release all resources used by stream immediately.


    If this post answers your question, please click "Mark As Answer". If this post is helpful please click "Mark as Helpful".

    Saturday, March 10, 2012 1:11 AM
  • The fact you rebooted and now can do the job indicates to me that there was a possible memory leak.  Quite common actually.  Just make sure to close all your streams. When you are debugging.  Kris example above is perfect for the system to automatically do it based on method exit.  You can also add code to the DISPOSE method of the window or look into TRY CATCH FINALLY and put it in the FINALLY code.

    JP Cowboy Coders Unite!

    Saturday, March 10, 2012 3:12 AM

All replies

  • Can you post here the complete stack trace?

    Paulo Morgado

    Thursday, March 8, 2012 3:18 PM
  • Hi, 521MB are too large to store in memory. this is why it's causing that exception.

    Bilhan silva

    Thursday, March 8, 2012 4:25 PM
  • 512MB should be a piece of cake for a x64 bit system....unless you are loading many of these files/have a small ammount. How much RAM do you have? A 4 gig or higher system should probably be used for doing this all at once.

    What may be happening is the file stream uses virtual memory instead of real memory to read in the individual chunks. Try reading it in in smaller chunks:

    System.IO.FileStream fs = null;
                    List<byte> data = new List<byte>();
                    while (fs.Position < fs.Length)
                    {
                        data.AddRange(rdr.ReadBytes(128));
                    }

    If this is happening only on a second or third time reading, you might want to try wrapping the streams in Using statements to force them to be cleaned up and not rely on the garbage collector to get to it:

    using (FileStream fs = new FileStream(fi.FullName, FileMode.Open, FileAccess.Read))
    {
                    using (BinaryReader rdr = new BinaryReader(fs))
    {
     ...
    }
    }

    Hope this helps!

    Thursday, March 8, 2012 4:39 PM
  • Can you, also, post here the stack trace of the exception?

    Paulo Morgado

    Thursday, March 8, 2012 4:54 PM
  • Stack Trace:

    System.OutOfMemoryException was unhandled
      Message=Exception of type 'System.OutOfMemoryException' was thrown.
      Source=mscorlib
      StackTrace:
           at System.IO.BinaryReader.ReadBytes(Int32 count)
           at GEOINT.frmForm.btnInsert_Click(Object sender, EventArgs e) in C:\DEVELOPMENT\GEOINT\GEOINT\Form1.cs:line 35
           at System.Windows.Forms.Control.OnClick(EventArgs e)
           at System.Windows.Forms.Button.OnClick(EventArgs e)
           at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
           at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
           at System.Windows.Forms.Control.WndProc(Message& m)
           at System.Windows.Forms.ButtonBase.WndProc(Message& m)
           at System.Windows.Forms.Button.WndProc(Message& m)
           at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
           at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
           at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
           at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
           at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
           at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
           at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
           at System.Windows.Forms.Application.Run(Form mainForm)
           at GEOINT.Program.Main(String[] args) in C:\DEVELOPMENT\GEOINT\GEOINT\Program.cs:line 26
           at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
           at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
           at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
           at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
           at System.Threading.ThreadHelper.ThreadStart()
      InnerException:

    Thank you for your help on this.

    Sharon

    Thursday, March 8, 2012 5:06 PM
  • Kind of unrelated but you should probably actually be using the FILESTREAM that Microsoft added to SQL Server 2008 if you are on 2008 or greater and assuming it licenses for the edition of sql you are using and have no plans on supporting alternate DB (such as Oracle or MySQL, etc) as they may not support this (e.g. it's not ansi compliant, they've not implemented it) etc.

    Links are included and embedded below:

    Getting started with FILEStream storage

    http://msdn.microsoft.com/en-us/library/bb933995(v=sql.100).aspx

    Using FILESTEAM Storage in client applications

    http://msdn.microsoft.com/en-us/library/bb933877(v=sql.100).aspx

    There is a SqlFileStream class documentation

    http://msdn.microsoft.com/en-us/library/system.data.sqltypes.sqlfilestream.aspx

    Sample code on using it.

    http://geekswithblogs.net/aghausman/archive/2009/03/16/saving-and-retrieving-file-using-filestream-sql-server-2008.aspx

    A link that I forget where it goes to.

    http://msdn.microsoft.com/en-us/library/bb933972(v=sql.100).aspx

    Thursday, March 8, 2012 5:27 PM
  • thank you. I'll  check these links out. I am on SQL Server 2008 R2.

    Sharon

    Thursday, March 8, 2012 5:30 PM
  • You might be running into memory fragmentation, although I wouldn't expect it for 128 bytes.

    Anyway, try this to see if that's the problem:

    List<byte> data = new List<byte>();
    byte[] buffer = new byte[65536];
    int count;
    while (fs.Position < fs.Length)
    {
        count = rdr.Read(buffer, 0, 65536);
        data.AddRange(buffer.Take(count));
    }


    Paulo Morgado

    Thursday, March 8, 2012 5:57 PM
  • Hi,

    As I mentioned I have a machine that has 8 GB of memory. I had to shut down my machine to take it home last night. When I tried to load the file again today it loaded with no problem. Normally I will only be loading one file at a time, but in this case I was testing my Development database and was loading 28 files one right after the other.

    Now my question is, since it is working ok now, is there a problem I still need to fix?

    Sharon

    Thursday, March 8, 2012 7:37 PM
  • Only testing and time will be able tell you that. If the code is configured accordingly I would try running it in such a manner that simulates the type of environment it will be expected to operate in. E.g. if it's supposed to be able to go non-stop and semi-rigorous then try running it overnight and see if it dies with the same problem.

    It could be a result of a) Memory Leak in the application meaning you should probably try tracking it down or at least have the application restart perhaps on a daily basis. or b) It could be a bigger problem related to the overall memory usage of all of the applications on the system after a duration of say a few days or a few weeks.

    Sometimes computers or software just needs to be rebooted/restarted for optimal performance.

    If your company has the money and is willing to spend it I would recommend

    http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

    It is a great means of tracking memory usage and leaks in applications. Its 500 dollars well spent by any company where this can end up causing time and money loss later down the line and is a great tool to have in a developer's aresenal. 500 dollars spent now might save the company 10k-250k down the line.

    • Edited by MarcD Thursday, March 8, 2012 7:44 PM added memory profile links
    • Marked as answer by Leo Liu - MSFT Friday, March 16, 2012 4:28 AM
    Thursday, March 8, 2012 7:42 PM
  • MarcD,

    Thanks so much for your help.

    Sharon

    Thursday, March 8, 2012 7:44 PM
  • Have you tried allocating the buffer yourself and then use the filestreams Read method instead?
    Friday, March 9, 2012 6:39 PM
  • Hi, 

    Also, it is advisable wrapping your stream with using statement to release all resources used by stream immediately.


    If this post answers your question, please click "Mark As Answer". If this post is helpful please click "Mark as Helpful".

    Saturday, March 10, 2012 1:11 AM
  • The fact you rebooted and now can do the job indicates to me that there was a possible memory leak.  Quite common actually.  Just make sure to close all your streams. When you are debugging.  Kris example above is perfect for the system to automatically do it based on method exit.  You can also add code to the DISPOSE method of the window or look into TRY CATCH FINALLY and put it in the FINALLY code.

    JP Cowboy Coders Unite!

    Saturday, March 10, 2012 3:12 AM
  • One other thing, everyone who repsonded to this post is just guessing.  When you have problem like these you may be able to run the .NET Profiller for your application to see what's going on.  In addition, you could look into using WINDBG but it takes a while to learn.

    JP Cowboy Coders Unite!

    Saturday, March 10, 2012 3:14 AM
  • Mr. Javaman,

    I'm doing a .close would that work just as good as a DISPOSE?

    sHARON

    Friday, March 16, 2012 2:30 PM
  • Mr. Javaman,

    I'm doing a .close would that work just as good as a DISPOSE?

    sHARON


    Yes.
    Friday, March 16, 2012 2:38 PM
  • Thanks Jared. Thanks everyone.
    Friday, March 16, 2012 4:02 PM