locked
Memory Stream Help RRS feed

  • Question

  • Hi,

    I would like to ask a question about MemoryStream, my syntax is like this,

    FtpWebResponse response = request.GetResponse() as FtpWebResponse;
    Stream reader = response.GetResponseStream();
    byte[] downloadedData = new byte[0];
    MemoryStream memStream = new MemoryStream();
                    byte[] buffer = new byte[1024]; //downloads in chuncks
    
                    while (true)
                    {
                        //Application.DoEvents(); //prevent application from crashing
    
                        //Try to read the data
                        int bytesRead = reader.Read(buffer, 0, buffer.Length);
    
                        //Nothing was read, finished downloading
                        if (bytesRead == 0)
                            break;
                        //Write the downloaded data
                        else
                            memStream.Write(buffer, 0, bytesRead);
                        
                    }
                    
                    //Convert the downloaded stream to a byte array
                    downloadedData = memStream.ToArray();
                    
                    //Clean up
                    reader.Close();
                    memStream.Close();
                    response.Close();

    I am trying to download CSV file with size of  514,105,022 bytes form FTP, but it returns OutOfMemoryException on my production server but not in development server. Is there anything I need to configure on my production server to make it work like on development server.

    Thanks and Regards,

    Auliman Arga

    Tuesday, March 20, 2012 3:35 AM

Answers

All replies

  • Hi

    Almost you are trying to load 500 MB in RAM.This is most expensive.

    you can't except 500 MB free space in production server.

    So Write downloaded chunks to file stream instead of memory stream.



    Thirukumaran.C

    Tuesday, March 20, 2012 4:24 AM
  • Can you replace your MemoryStream to StreamWriter?

    StreamWriter sw = new StreamWriter(@"D:\words.csv");


    Regards, Nighting Liu

    Tuesday, March 20, 2012 4:43 AM
  • Yeah, I already try to change the application using file stream but my manager told me not to change it before we find the real problem, because production server has greater specification than development server and still wondering why it get the Out of Memory Exception on Production server.

    Do you have any idea??

    Regards,

    Auliman Arga

    Tuesday, March 20, 2012 5:35 AM
  • when you run it go to taskmgr and check the performance... or check the processes it'll tell you how much ram are used and then check your product server for free Rams...
    Sometimes even if it has greater specs your server may be using more than the other and thus lacking free memory ...

    Peter Koueik


    Tuesday, March 20, 2012 6:03 AM
  • Use WebClient.DownloadDataAsync.
    Tuesday, March 20, 2012 6:21 AM
  • Hi

    Almost you are trying to load 500 MB in RAM.This is most expensive.

    you can't except 500 MB free space in production server.

    So Write downloaded chunks to file stream instead of memory stream.



    Thirukumaran.C


    Actually 2 * 500 MB.  500 in memory stream and 500 in array.
    Tuesday, March 20, 2012 6:23 AM
  • Hi Peter,

    Already did that too. the application only use memory until around 710 MB(actually it always different each time I tried it, but most of them are around this number) before it got the exception.
    And the production server still got 15 GB available memory, meanwhile on Development server only have 2,3 GB available memory. That is why I got so confused with this issue, because I think the file size hasn't hit the MemoryStream threshold yet...

    Regards,

    Auliman Arga



    • Edited by Auliman Tuesday, March 20, 2012 6:42 AM
    Tuesday, March 20, 2012 6:33 AM
  •  check this link !:
    http://stackoverflow.com/questions/5104327/download-a-file-from-a-external-sever-through-asp-net-c-sharp

    Peter Koueik

    • Proposed as answer by Neddy Ren Wednesday, March 21, 2012 7:02 AM
    Tuesday, March 20, 2012 6:39 AM
    • Proposed as answer by Neddy Ren Wednesday, March 21, 2012 7:02 AM
    • Marked as answer by Auliman Wednesday, March 21, 2012 7:10 AM
    Tuesday, March 20, 2012 6:54 AM
  • Peter and JohnWein,

    Thanks for the workaround of this MemoryStream, I'll try that.

    But I am still wondering the real cause of this Exception, is it because the Production server configuration?? because I didn't get the exception on development server.  Already searched it around 4 hours and still can't find any clue about this.

    Thanks and regards,

    Auliman Arga 

    Tuesday, March 20, 2012 6:58 AM
  • check these links for probable answer to your question about why this may be happening:

    http://stackoverflow.com/questions/2819081/memorystream-and-large-object-heap
    http://stackoverflow.com/questions/6700341/system-outofmemory-exception-when-calling-memorystream-toarray-after-serializi
    http://stackoverflow.com/questions/1767096/c-sharp-outofmemoryexception-on-memorystream-writing

    Peter Koueik

    • Proposed as answer by Neddy Ren Wednesday, March 21, 2012 7:02 AM
    • Marked as answer by Auliman Wednesday, March 21, 2012 7:10 AM
    Tuesday, March 20, 2012 7:03 AM
  • Thanks to everyone who answer my question. Really appreciate all your helps, now I get more knowledge about memory usage in programming even though it is still so complex for me to fully understand it :) .

    Hope you all have good days. Cheers!!

    Best Regards,

    Auliman Arga

    Wednesday, March 21, 2012 7:18 AM