Memory Stream (Memory overflow exception at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count))

Question

Hello
          I am dealing with very large image data which is upto 1 GB. I am writing the data into memory stream using Stream.Write(data, 0, data.Length).(where data is in byte[]) when the capacity of the stream goes beyond 435142656 i.e. 414.99 MB(approx)  it throws the out of memory exception. As per my knowledge memory reserve for each CLR object is 2 GB. My data is under the limit of 2 GB then why I am getting this error ??

Any Clue ??

Regards
Mahesh  

Answer 1

I believe that that is the default maximum memory capacity for a Visual Studio application. 
As for your Memory Overflow ... 

Buffered Stream.  Look into it.  A BufferedStream can be used with any other stream object, not just a network stream as shown in the sample.

---

Mark the best replies as answers. "Fooling computers since 1971."

Answer 2

Not too sure about the 2GB per object limit but do know that if thats true it'll only apply to 64 bit platforms, in the 32 bit world you are limited to 2GB per process.

Answer 3

In theory, I agree. I'm a little worried by the fact that MemoryStream.Length is, as you'd expect, a long (int64) but MemoryStream.Capacity is an int (int32)!

Seems like a mistake to me...

Answer 4

Not too sure about the 2GB per object limit but do know that if thats true it'll only apply to 64 bit platforms, in the 32 bit world you are limited to 2GB per process.

Yes.  I think you are right.  The OP complained about getting the error at 0.5GB.  I think that limit is the default app size imposed by Visual Studio.

---

Mark the best replies as answers. "Fooling computers since 1971."

Answer 5

A recent article on Eric Lippert's blog exposes issues similar to this, and it might be relevant to you. In particular where it states:

"An “out of memory” error almost never happens because there’s not enough storage available; as we’ve seen, storage is disk space, and disks are huge these days. Rather, an “out of memory” error happens because the process is unable to find a large enough section of contiguous unused pages in its virtual address space to do the requested mapping."

Source: http://blogs.msdn.com/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx

Regards,
Fernando.

---

I always try to Keep it Sharp & simple.

Answer 6

I tried a wee experiment:

using System;
using System.IO;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main()
        {
            MemoryStream ms = new MemoryStream();

            byte[] buffer = new byte[128*1024*1024];

            for (int i = 0; i < 1024; ++i)
            {
                ms.Write(buffer, 0, buffer.Length);
                Console.WriteLine("Length (MB) = " + ms.Length/(1024*1024) + ", Capacity (MB) = " + ms.Capacity/(1024*1024));
            }
        }
    }
}

Results on Windows 7 64-bit with 8GB RAM:

Compiled as 64 bit:

Length (MB) = 128, Capacity (MB) = 128
Length (MB) = 256, Capacity (MB) = 256
Length (MB) = 384, Capacity (MB) = 512
Length (MB) = 512, Capacity (MB) = 512
Length (MB) = 640, Capacity (MB) = 1024
Length (MB) = 768, Capacity (MB) = 1024
Length (MB) = 896, Capacity (MB) = 1024
Length (MB) = 1024, Capacity (MB) = 1024
Length (MB) = 1152, Capacity (MB) = 1152
Length (MB) = 1280, Capacity (MB) = 1280
Length (MB) = 1408, Capacity (MB) = 1408
Length (MB) = 1536, Capacity (MB) = 1536
Length (MB) = 1664, Capacity (MB) = 1664
Length (MB) = 1792, Capacity (MB) = 1792
Length (MB) = 1920, Capacity (MB) = 1920

Unhandled Exception: System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at ConsoleApplication1.Program.Main() in C:\Test\cs3.5\ConsoleApplication1\Program.cs:line 16

Compiled as 32 bit:

Length (MB) = 128, Capacity (MB) = 128
Length (MB) = 256, Capacity (MB) = 256
Length (MB) = 384, Capacity (MB) = 512
Length (MB) = 512, Capacity (MB) = 512

Unhandled Exception: OutOfMemoryException.

Conclusion

Interesting... ;)

Answer 7

Yes, my experiment seems to confirm your hypothesis. :)

Answer 8

Matthew, could you try it with 2 MemoryStreams? Perhaps is a stream limitation rather than process'.

Just a thought.

---

I always try to Keep it Sharp & simple.

Answer 9

Matthew, could you try it with 2 MemoryStreams? Perhaps is a stream limitation rather than process'.

Just a thought.

---

I always try to Keep it Sharp & simple.

With 2 memory streams:

using System;
using System.IO;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main()
        {
            MemoryStream ms1 = new MemoryStream();
            MemoryStream ms2 = new MemoryStream();

            byte[] buffer = new byte[128*1024*1024];

            for (int i = 0; i < 1024; ++i)
            {
                ms1.Write(buffer, 0, buffer.Length);
                Console.WriteLine("MS1: Length (MB) = " + ms1.Length/(1024*1024) + ", Capacity (MB) = " + ms1.Capacity/(1024*1024));
                Console.WriteLine("Total allocated = " + GC.GetTotalMemory(false)/(1024*1024));

                ms2.Write(buffer, 0, buffer.Length);
                Console.WriteLine("MS2: Length (MB) = " + ms2.Length/(1024*1024) + ", Capacity (MB) = " + ms2.Capacity/(1024*1024));
                Console.WriteLine("Total allocated = " + GC.GetTotalMemory(false)/(1024*1024));
            }
        }
    }
}

x86:

MS1: Length (MB) = 128, Capacity (MB) = 128
Total allocated = 256
MS2: Length (MB) = 128, Capacity (MB) = 128
Total allocated = 384
MS1: Length (MB) = 256, Capacity (MB) = 256
Total allocated = 640
MS2: Length (MB) = 256, Capacity (MB) = 256
Total allocated = 896
MS1: Length (MB) = 384, Capacity (MB) = 512
Total allocated = 1408
Unhandled Exception: OutOfMemoryException.

x64:

MS1: Length (MB) = 128, Capacity (MB) = 128
MS2: Length (MB) = 128, Capacity (MB) = 128
MS1: Length (MB) = 256, Capacity (MB) = 256
MS2: Length (MB) = 256, Capacity (MB) = 256
MS1: Length (MB) = 384, Capacity (MB) = 512
MS2: Length (MB) = 384, Capacity (MB) = 512
MS1: Length (MB) = 512, Capacity (MB) = 512
MS2: Length (MB) = 512, Capacity (MB) = 512
MS1: Length (MB) = 640, Capacity (MB) = 1024
MS2: Length (MB) = 640, Capacity (MB) = 1024
MS1: Length (MB) = 768, Capacity (MB) = 1024
MS2: Length (MB) = 768, Capacity (MB) = 1024
MS1: Length (MB) = 896, Capacity (MB) = 1024
MS2: Length (MB) = 896, Capacity (MB) = 1024
MS1: Length (MB) = 1024, Capacity (MB) = 1024
MS2: Length (MB) = 1024, Capacity (MB) = 1024
MS1: Length (MB) = 1152, Capacity (MB) = 1152
MS2: Length (MB) = 1152, Capacity (MB) = 1152
MS1: Length (MB) = 1280, Capacity (MB) = 1280
MS2: Length (MB) = 1280, Capacity (MB) = 1280
MS1: Length (MB) = 1408, Capacity (MB) = 1408
MS2: Length (MB) = 1408, Capacity (MB) = 1408
MS1: Length (MB) = 1536, Capacity (MB) = 1536
MS2: Length (MB) = 1536, Capacity (MB) = 1536
MS1: Length (MB) = 1664, Capacity (MB) = 1664
MS2: Length (MB) = 1664, Capacity (MB) = 1664
MS1: Length (MB) = 1792, Capacity (MB) = 1792
MS2: Length (MB) = 1792, Capacity (MB) = 1792
MS1: Length (MB) = 1920, Capacity (MB) = 1920
MS2: Length (MB) = 1920, Capacity (MB) = 1920

Unhandled Exception: System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at ConsoleApplication1.Program.Main() in C:\Test\cs3.5\ConsoleApplication1\Program.cs:line 17

Conclusion

A 32 bit C# program can allocate at least 1.5GB memory. Memory Streams seem limited to 512MB.
A 64 bit program can allocate way more memory, and Memory Streams are limited to 2GB.

Answer 10

Thank you all for clearing my doubt. From above postings I concluded that it is not possible to buffered a  data after a certain ammount in memory streams. Is there any other alternative to handle it ?? One common solution  is to introduced the disk.But I am trying to avoid disk IO operation because of performance issue. 

Regards
Mahesh

Answer 11

Well, you could write a managed wrapper for the Windows API functions for using Memory Mapped Files.

C# 4 will provide such a wrapper class (6 years too late for us, we wrote our own wrapper a long time ago...).

I can't post all our copyrighted code here unfortunately, but if you google for it, I'm sure there are wrapper classes in the public domain.

Answer 12

Thanks Matthew
                        This is what I am searching for I had created wrapper for memory mapped file and use it successfully for handling the large data.

Thank You All For Responding The Thread

Regards 
Mahesh 

Answer 13

Mubey, I believe there's also an issue do to the fact that MemoryStream issues contiguous memory. If you were to separate this into chunks of say 256 MB, you'll probably have no problems. Maybe this inmediate solution may work for you and might be worth trying it out.

Just see if you are able to have 4 or 5 MemoryStreams with 256 MB each.

---

I always try to Keep it Sharp & simple.

Answer 14

I tried the following and it worked. It managed to allocate a little over 1.5 GB of memory in MemoryStreams. This PC has 1.5GB of physical memory, and I can tell you, as for how it responded after allocating it, it definitely did allocate it.

static MemoryStream CreateStream(int capacityInMB)
{
    MemoryStream stream = new MemoryStream(capacityInMB * 1024 * 1024);
    byte[] buffer = new byte[1024 * 1024];
    for (int i = 0; i < 1024 * 1024; i++)
    {
        buffer[i] = 255;
    }
    for (int i = 0; i < capacityInMB; i++)
    {
        stream.Write(buffer, 0, buffer.Length);
    }
    return stream;
}

static void Main(string[] args)
{
    MemoryStream str1 = CreateStream(256);
    Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
    MemoryStream str2 = CreateStream(256);
    Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
    MemoryStream str3 = CreateStream(256);
    Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
    MemoryStream str4 = CreateStream(256);
    Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
    MemoryStream str5 = CreateStream(256);
    Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
    MemoryStream str6 = CreateStream(256);
    Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
    MemoryStream str7 = CreateStream(128);
    Console.WriteLine(GC.GetTotalMemory(false) / (1024 * 1024));
    Console.WriteLine("Done");
    Console.ReadLine();
    GC.KeepAlive(str1);
    GC.KeepAlive(str2);
    GC.KeepAlive(str3);
    GC.KeepAlive(str4);
    GC.KeepAlive(str5);
    GC.KeepAlive(str6);
    GC.KeepAlive(str7);
}

The output was:

257
513
770
1027
1284
1537
1666
Done

So it seems that this approach may be valid for your particular issue.

Also, the KeepAlives aren't necessary, but just in case... for the skepticals.

PS: My PC is fine by the way... almost back to normal ;)

---

I always try to Keep it Sharp & simple.

Answer 15

Thanks Fernando;

                        Unfortunately this approach doesnt work for me. I got out of memory exception while creating the third stream i.e. After allocating 513 MB.Using Multiple stream as suggested by you buffered the data slighty more than single stream but this quantity is not  sufficient for me as I have to buffer 1 GB of data . I am using 32 bit platform with 1 GB of RAM.

Regards
Mahesh

Answer 16

any final solution wiht full source code sample ?

---

www.kiquenet.com/profesional