none
Memory Stream (Memory overflow exception at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count))

    Question

  • Hello
              I am dealing with very large image data which is upto 1 GB. I am writing the data into memory stream using Stream.Write(data, 0, data.Length).(where data is in byte[]) when the capacity of the stream goes beyond 435142656 i.e. 414.99 MB(approx)  it throws the out of memory exception. As per my knowledge memory reserve for each CLR object is 2 GB. My data is under the limit of 2 GB then why I am getting this error ??

    Any Clue ??

    Regards
    Mahesh  
    Monday, June 15, 2009 1:27 PM

Answers

  • Well, you could write a managed wrapper for the Windows API functions for using Memory Mapped Files.

    C# 4 will provide such a wrapper class (6 years too late for us, we wrote our own wrapper a long time ago...).

    I can't post all our copyrighted code here unfortunately, but if you google for it, I'm sure there are wrapper classes in the public domain.
    • Marked as answer by Mahesh Dubey Tuesday, June 16, 2009 1:32 PM
    Tuesday, June 16, 2009 9:36 AM

All replies

  • I believe that that is the default maximum memory capacity for a Visual Studio application. 
    As for your Memory Overflow ... 

    Buffered Stream.  Look into it.  A BufferedStream can be used with any other stream object, not just a network stream as shown in the sample.
    Mark the best replies as answers. "Fooling computers since 1971."
    Monday, June 15, 2009 1:44 PM
    Moderator
  • Not too sure about the 2GB per object limit but do know that if thats true it'll only apply to 64 bit platforms, in the 32 bit world you are limited to 2GB per process.

    Monday, June 15, 2009 1:46 PM
  • In theory, I agree. I'm a little worried by the fact that MemoryStream.Length is, as you'd expect, a long (int64) but MemoryStream.Capacity is an int (int32)!

    Seems like a mistake to me...
    Monday, June 15, 2009 1:51 PM
  • Not too sure about the 2GB per object limit but do know that if thats true it'll only apply to 64 bit platforms, in the 32 bit world you are limited to 2GB per process.


    Yes.  I think you are right.  The OP complained about getting the error at 0.5GB.  I think that limit is the default app size imposed by Visual Studio.
    Mark the best replies as answers. "Fooling computers since 1971."
    Monday, June 15, 2009 1:51 PM
    Moderator
  • A recent article on Eric Lippert's blog exposes issues similar to this, and it might be relevant to you. In particular where it states:

    "An “out of memory” error almost never happens because there’s not enough storage available; as we’ve seen, storage is disk space, and disks are huge these days. Rather, an “out of memory” error happens because the process is unable to find a large enough section of contiguous unused pages in its virtual address space to do the requested mapping."

    Source: http://blogs.msdn.com/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx

    Regards,
    Fernando.
    I always try to Keep it Sharp & simple.
    Monday, June 15, 2009 2:07 PM
  • I tried a wee experiment:

    using System;
    using System.IO;
    
    namespace ConsoleApplication1
    {
        class Program
        {
            static void Main()
            {
                MemoryStream ms = new MemoryStream();
    
                byte[] buffer = new byte[128*1024*1024];
    
                for (int i = 0; i < 1024; ++i)
                {
                    ms.Write(buffer, 0, buffer.Length);
                    Console.WriteLine("Length (MB) = " + ms.Length/(1024*1024) + ", Capacity (MB) = " + ms.Capacity/(1024*1024));
                }
            }
        }
    }
    
    Results on Windows 7 64-bit with 8GB RAM:

    Compiled as 64 bit:

    Length (MB) = 128, Capacity (MB) = 128
    Length (MB) = 256, Capacity (MB) = 256
    Length (MB) = 384, Capacity (MB) = 512
    Length (MB) = 512, Capacity (MB) = 512
    Length (MB) = 640, Capacity (MB) = 1024
    Length (MB) = 768, Capacity (MB) = 1024
    Length (MB) = 896, Capacity (MB) = 1024
    Length (MB) = 1024, Capacity (MB) = 1024
    Length (MB) = 1152, Capacity (MB) = 1152
    Length (MB) = 1280, Capacity (MB) = 1280
    Length (MB) = 1408, Capacity (MB) = 1408
    Length (MB) = 1536, Capacity (MB) = 1536
    Length (MB) = 1664, Capacity (MB) = 1664
    Length (MB) = 1792, Capacity (MB) = 1792
    Length (MB) = 1920, Capacity (MB) = 1920

    Unhandled Exception: System.IO.IOException: Stream was too long.
       at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
       at ConsoleApplication1.Program.Main() in C:\Test\cs3.5\ConsoleApplication1\Program.cs:line 16

    Compiled as 32 bit:

    Length (MB) = 128, Capacity (MB) = 128
    Length (MB) = 256, Capacity (MB) = 256
    Length (MB) = 384, Capacity (MB) = 512
    Length (MB) = 512, Capacity (MB) = 512

    Unhandled Exception: OutOfMemoryException.

    Conclusion

    Interesting... ;)
    Monday, June 15, 2009 2:09 PM
  • Yes, my experiment seems to confirm your hypothesis. :)
    Monday, June 15, 2009 2:11 PM
  • Matthew, could you try it with 2 MemoryStreams? Perhaps is a stream limitation rather than process'.

    Just a thought.

    I always try to Keep it Sharp & simple.
    Monday, June 15, 2009 2:12 PM
  • Matthew, could you try it with 2 MemoryStreams? Perhaps is a stream limitation rather than process'.

    Just a thought.

    I always try to Keep it Sharp & simple.

    With 2 memory streams:

    using System;
    using System.IO;
    
    namespace ConsoleApplication1
    {
        class Program
        {
            static void Main()
            {
                MemoryStream ms1 = new MemoryStream();
                MemoryStream ms2 = new MemoryStream();
    
                byte[] buffer = new byte[128*1024*1024];
    
                for (int i = 0; i < 1024; ++i)
                {
                    ms1.Write(buffer, 0, buffer.Length);
                    Console.WriteLine("MS1: Length (MB) = " + ms1.Length/(1024*1024) + ", Capacity (MB) = " + ms1.Capacity/(1024*1024));
                    Console.WriteLine("Total allocated = " + GC.GetTotalMemory(false)/(1024*1024));
    
                    ms2.Write(buffer, 0, buffer.Length);
                    Console.WriteLine("MS2: Length (MB) = " + ms2.Length/(1024*1024) + ", Capacity (MB) = " + ms2.Capacity/(1024*1024));
                    Console.WriteLine("Total allocated = " + GC.GetTotalMemory(false)/(1024*1024));
                }
            }
        }
    }
    

    x86:

    MS1: Length (MB) = 128, Capacity (MB) = 128
    Total allocated = 256
    MS2: Length (MB) = 128, Capacity (MB) = 128
    Total allocated = 384
    MS1: Length (MB) = 256, Capacity (MB) = 256
    Total allocated = 640
    MS2: Length (MB) = 256, Capacity (MB) = 256
    Total allocated = 896
    MS1: Length (MB) = 384, Capacity (MB) = 512
    Total allocated = 1408
    Unhandled Exception: OutOfMemoryException.

    x64:


    MS1: Length (MB) = 128, Capacity (MB) = 128
    MS2: Length (MB) = 128, Capacity (MB) = 128
    MS1: Length (MB) = 256, Capacity (MB) = 256
    MS2: Length (MB) = 256, Capacity (MB) = 256
    MS1: Length (MB) = 384, Capacity (MB) = 512
    MS2: Length (MB) = 384, Capacity (MB) = 512
    MS1: Length (MB) = 512, Capacity (MB) = 512
    MS2: Length (MB) = 512, Capacity (MB) = 512
    MS1: Length (MB) = 640, Capacity (MB) = 1024
    MS2: Length (MB) = 640, Capacity (MB) = 1024
    MS1: Length (MB) = 768, Capacity (MB) = 1024
    MS2: Length (MB) = 768, Capacity (MB) = 1024
    MS1: Length (MB) = 896, Capacity (MB) = 1024
    MS2: Length (MB) = 896, Capacity (MB) = 1024
    MS1: Length (MB) = 1024, Capacity (MB) = 1024
    MS2: Length (MB) = 1024, Capacity (MB) = 1024
    MS1: Length (MB) = 1152, Capacity (MB) = 1152
    MS2: Length (MB) = 1152, Capacity (MB) = 1152
    MS1: Length (MB) = 1280, Capacity (MB) = 1280
    MS2: Length (MB) = 1280, Capacity (MB) = 1280
    MS1: Length (MB) = 1408, Capacity (MB) = 1408
    MS2: Length (MB) = 1408, Capacity (MB) = 1408
    MS1: Length (MB) = 1536, Capacity (MB) = 1536
    MS2: Length (MB) = 1536, Capacity (MB) = 1536
    MS1: Length (MB) = 1664, Capacity (MB) = 1664
    MS2: Length (MB) = 1664, Capacity (MB) = 1664
    MS1: Length (MB) = 1792, Capacity (MB) = 1792
    MS2: Length (MB) = 1792, Capacity (MB) = 1792
    MS1: Length (MB) = 1920, Capacity (MB) = 1920
    MS2: Length (MB) = 1920, Capacity (MB) = 1920

    Unhandled Exception: System.IO.IOException: Stream was too long.
       at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
       at ConsoleApplication1.Program.Main() in C:\Test\cs3.5\ConsoleApplication1\Program.cs:line 17

    Conclusion

    A 32 bit C# program can allocate at least 1.5GB memory. Memory Streams seem limited to 512MB.
    A 64 bit program can allocate way more memory, and Memory Streams are limited to 2GB.
    Monday, June 15, 2009 2:35 PM
  • Thank you all for clearing my doubt. From above postings I concluded that it is not possible to buffered a  data after a certain ammount in memory streams. Is there any other alternative to handle it ?? One common solution  is to introduced the disk.But I am trying to avoid disk IO operation because of performance issue. 

    Regards
    Mahesh

    Tuesday, June 16, 2009 9:24 AM
  • Well, you could write a managed wrapper for the Windows API functions for using Memory Mapped Files.

    C# 4 will provide such a wrapper class (6 years too late for us, we wrote our own wrapper a long time ago...).

    I can't post all our copyrighted code here unfortunately, but if you google for it, I'm sure there are wrapper classes in the public domain.
    • Marked as answer by Mahesh Dubey Tuesday, June 16, 2009 1:32 PM
    Tuesday, June 16, 2009 9:36 AM
  • Thanks Matthew
                            This is what I am searching for I had created wrapper for memory mapped file and use it successfully for handling the large data.

    Thank You All For Responding The Thread

    Regards 
    Mahesh 
    Tuesday, June 16, 2009 1:32 PM
  • Mubey, I believe there's also an issue do to the fact that MemoryStream issues contiguous memory. If you were to separate this into chunks of say 256 MB, you'll probably have no problems. Maybe this inmediate solution may work for you and might be worth trying it out.

    Just see if you are able to have 4 or 5 MemoryStreams with 256 MB each.
    I always try to Keep it Sharp & simple.
    Tuesday, June 16, 2009 8:53 PM
  • I tried the following and it worked. It managed to allocate a little over 1.5 GB of memory in MemoryStreams. This PC has 1.5GB of physical memory, and I can tell you, as for how it responded after allocating it, it definitely did allocate it.

    static MemoryStream CreateStream(int capacityInMB)
    {
        MemoryStream stream = new MemoryStream(capacityInMB * 1024 * 1024);
        byte[] buffer = new byte[1024 * 1024];
        for (int i = 0; i < 1024 * 1024; i++)
        {
            buffer[i] = 255;
        }
        for (int i = 0; i < capacityInMB; i++)
        {
            stream.Write(buffer, 0, buffer.Length);
        }
        return stream;
    }
    
    static void Main(string[] args)
    {
        MemoryStream str1 = CreateStream(256);
        Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
        MemoryStream str2 = CreateStream(256);
        Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
        MemoryStream str3 = CreateStream(256);
        Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
        MemoryStream str4 = CreateStream(256);
        Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
        MemoryStream str5 = CreateStream(256);
        Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
        MemoryStream str6 = CreateStream(256);
        Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
        MemoryStream str7 = CreateStream(128);
        Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
        Console.WriteLine("Done");
        Console.ReadLine();
        GC.KeepAlive(str1);
        GC.KeepAlive(str2);
        GC.KeepAlive(str3);
        GC.KeepAlive(str4);
        GC.KeepAlive(str5);
        GC.KeepAlive(str6);
        GC.KeepAlive(str7);
    }

    The output was:

    257
    513
    770
    1027
    1284
    1537
    1666
    Done

    So it seems that this approach may be valid for your particular issue.

    Also, the KeepAlives aren't necessary, but just in case... for the skepticals.

    PS: My PC is fine by the way... almost back to normal ;)
    I always try to Keep it Sharp & simple.
    Tuesday, June 16, 2009 9:15 PM
  • Thanks Fernando;

                            Unfortunately this approach doesnt work for me. I got out of memory exception while creating the third stream i.e. After allocating 513 MB.Using Multiple stream as suggested by you buffered the data slighty more than single stream but this quantity is not  sufficient for me as I have to buffer 1 GB of data . I am using 32 bit platform with 1 GB of RAM.

    Regards
    Mahesh

    Wednesday, June 17, 2009 6:16 AM
  • any final solution wiht full source code sample ?

    www.kiquenet.com/profesional

    Wednesday, December 12, 2012 8:20 PM