none
Not able to write more than 256MB in Memory Stream 32 bit machine RRS feed

  • Question

  • Hi<o:p></o:p>

    We are trying to write more than 256 MB data in Memory stream(C# .Net Memory Stream). While writing "Outofmemory” exception is thrown in 32 bit machines. <o:p></o:p>

    If we try to write more than 256 MB data 32 bit machine with i5 Intel processor with 4 GB ROM, it is working fine.<o:p></o:p>

    Note: if we try to write more than 256 MB data in 64 bit also, we didn't get any issues. It is working fine.<o:p></o:p>

    Thanks in advance,<o:p></o:p>

    Rajasekar<o:p></o:p>


    Thursday, February 13, 2014 10:42 AM

Answers

  • // You may need to check this for exceptions, see msdn library
    string path = Path.GetTempFileName();
    
    FileStream fileStream = new FileStream(path, FileMode.Create);

    Both Path and FileStream are in the System.IO namespace. You can write to the filestream as you did to the memorystream.

    Make sure you dispose the stream and delete the temp file when you're done with it.


    https://fraktalysator.codeplex.com

    Friday, February 14, 2014 10:26 AM

All replies

  • How are you writing this data ? Could you provide some sample code? Also, what is your scenario for holding more than 256MB of data in a MemoryStream?

    ----------------------- Sujay Sarma {Unbounded;}

    Thursday, February 13, 2014 10:51 AM
  • Hi

    Thanks for prompt support.

    Sample code:

             ///inputStream is more than 256 MB(i.e : 302 MB)

               int streamLength, length, padLength;
                byte padValue = 0;
                int intByte;
           
                Stream outputStream = new MemoryStream();
                do
                {
                    intByte = inputStream.ReadByte();
                    if (intByte != -1)
                    {
                        padValue = (byte)intByte;
                        outputStream.WriteByte(padValue);
                       
                    }
                } while (intByte != -1);

    I have tested in different OS/Machines?

    OS

    ROM(GB)

    Platform

    Working?

    Windows 7

    16

    64

    OK

    Windows 8

    2

    64

    OK

    Windows server 2012

    1

    64

    OK

    Windows 7(Onsite-English OS)

    4

    32

    OK

    Windows 8

    1

    32

    NG

    Windows XP

    2

    32

    NG

    Windows 7

    3

    32

    NG

    Windows 7(Onsite-Japanese OS)

    4

    32

    NG

    We need to know following info

    Is this issue dependent on 32bit/64 bit?

    Is this issue dependent on Processor type?

    Is this issue dependent on ROM Size?

    Is this a .Net Frame work limitation?

    Thanks,

    Rajasekar


    Thursday, February 13, 2014 11:03 AM
  • You havent said why you want to load up your memory with a 300 MB data stream... There could be a cheaper way to do this.

    ----------------------- Sujay Sarma {Unbounded;}

    Thursday, February 13, 2014 12:13 PM
  • Hi Sujay,

    We are decrpting the Data using  CryptoStream . while decrpting we need to remove the padding from the decrpted stream.

    Inoreder to remove the pading, we are looping this. Please find the full code

    ///inputStream is more than 256 MB(i.e : 302 MB)

               int streamLength, length, padLength;
                byte padValue = 0;
                int intByte;
           
                Stream outputStream = new MemoryStream();
                do
                {
                    intByte = inputStream.ReadByte();
                    if (intByte != -1)
                    {
                        padValue = (byte)intByte;
                        outputStream.WriteByte(padValue);
                       
                    }
                } while (intByte != -1);
                streamLength = (int)outputStream.Length;
                //
                //Get padding value and length of padding
                padLength = (int)padValue + 1;
                length = streamLength - padLength;
                outputStream.SetLength(length);
                outputStream.Position = 0;
                return outputStream;
           

    Thanks in advance,

    Rajasekar

                  

    Thursday, February 13, 2014 12:19 PM
  • Just after the constructor for MemoryStream, could you add this line and check?

    long streamSize = 302 * 1024 * 1024; // 302 MB

    outputStream.SetLength( streamSize );

    If you hit an error on the SetLength, then it could point to a memory sizing issue.

    Also check on the systems on which it is failing, what is the size of the virtual memory. While that code is running, keep a watch in either Resource Monitor (server OS) or Task Manager (details pane, all OS) on the amount of memory your application is eating.


    ----------------------- Sujay Sarma {Unbounded;}

    Thursday, February 13, 2014 12:26 PM
  • what is the type of the inputstream?

    Also, is this method called just once at the time and the memorystream disposed or are you holding several of this memorystream in the memory?


    https://fraktalysator.codeplex.com


    • Edited by ArgusMagnus Thursday, February 13, 2014 12:29 PM
    Thursday, February 13, 2014 12:27 PM
  • Hi

    I have followed the below,it throws out of exception only.

    >>long streamSize = 302 * 1024 * 1024; // 302 MB

    >>outputStream.SetLength( streamSize );

    Please find the below detailed source code

     Rijndael aes = RijndaelManaged.Create();
                aes.BlockSize = 128;
                aes.KeySize = 128;
                aes.Mode = CipherMode.CBC;
                aes.Padding = PaddingMode.None;

     ICryptoTransform crptTrsfm;

      crptTrsfm = aes.CreateDecryptor(aesKey, aesIV);
      crptMode = CryptoStreamMode.Read;

     Stream decrptStream = new CryptoStream(inputStream, crptTrsfm, crptMode);
     retStream = RemovePadding(decrptStream);

    //////////////////////////////////////////////////////////

    private Stream RemovePadding(Stream inputStream)
            {
                int streamLength, length, padLength;
                byte padValue = 0;
                int intByte;
           
                Stream outputStream = new MemoryStream();
                do
                {
                    intByte = inputStream.ReadByte();
                    if (intByte != -1)
                    {
                        padValue = (byte)intByte;
                        outputStream.WriteByte(padValue);
                       
                    }
                } while (intByte != -1);
                streamLength = (int)outputStream.Length;
                //
                //Get padding value and length of padding
                padLength = (int)padValue + 1;
                length = streamLength - padLength;
                outputStream.SetLength(length);
                outputStream.Position = 0;
                return outputStream;
            }

    Thanks in advance,

    Rajasekar

    Thursday, February 13, 2014 12:56 PM
  • Argus,

    This method is called only once.

    Thursday, February 13, 2014 2:12 PM
  • Im not sure about this, but I believe that MemoryStream works similar to a List<byte>. The default constructor allocates a byte array of a default length and each time the something is written to the MemoryStream and the underlying byte array is to small to hold the entire data, the underlying array is resized, meaning a new larger array is allocated and the data is copied from the old to the new array. Thus in the worst case, the MemoryStream requires double the memory you think it does for the time it copies the data from the old to the new array.

    If you know the approx amount of bytes the memorystream will have to hold, you could call a constructor which lets you specify a initial capacity. But if there isn't a really good reason to do this with a memorystream, I would use a temp file and a FileStream..


    https://fraktalysator.codeplex.com

    • Proposed as answer by Sujay Sarma Friday, February 14, 2014 2:12 PM
    Thursday, February 13, 2014 3:25 PM
  • Hi Argus,

    Thanks for your support.

    I'm new to this work.

    could you please guide me to use temp file and filestream.

    Thanks,

    Rajasekar

    Friday, February 14, 2014 6:30 AM
  • // You may need to check this for exceptions, see msdn library
    string path = Path.GetTempFileName();
    
    FileStream fileStream = new FileStream(path, FileMode.Create);

    Both Path and FileStream are in the System.IO namespace. You can write to the filestream as you did to the memorystream.

    Make sure you dispose the stream and delete the temp file when you're done with it.


    https://fraktalysator.codeplex.com

    Friday, February 14, 2014 10:26 AM