none
FTP OutOfMemory

    Question

  • Hi,

    The following code is directly from the MSDN website to upload a file using FTP.

    using System;  
    using System.IO;  
    using System.Net; using System.Text;  
     
    namespace Examples.System.Net  
    {  
        public class WebRequestGetExample  
        {  
            public static void Main ()  
            {  
                // Get the object used to communicate with the server.  
                FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://www.contoso.com/test.htm");  
                request.Method = WebRequestMethods.Ftp.UploadFile;  
     
                // This example assumes the FTP site uses anonymous logon.  
                request.Credentials = new NetworkCredential ("anonymous","janeDoe@contoso.com");  
                  
                // Copy the contents of the file to the request stream.  
                StreamReader sourceStream = new StreamReader("testfile.txt");  
                byte [] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());  
                sourceStream.Close();  
                request.ContentLength = fileContents.Length;  
     
                Stream requestStream = request.GetRequestStream();  
                requestStream.Write(fileContents, 0, fileContents.Length);  
                requestStream.Close();  
     
                FtpWebResponse response = (FtpWebResponse)request.GetResponse();  
          
                Console.WriteLine("Upload File Complete, status {0}", response.StatusDescription);  
          
                response.Close();  
                }  
            }  
        }  



    I am using this to upload SQL Server Backup files to a local FTP Server.
    The backups vary in size from 1k to 150,000 k and there are about 10 of them.
    The problem is that after about 250000k of files the sub fails at the line 

    byte [] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());

    with OutOfMemoryException

    No one file is causing the problem because if the order of the files is changed then the oom occurs in a different place. The issue appears to be with the bytearray and it not being disposed or GC'd.

    I added the line

    fileContents = null;

    this does not make any difference.

    Please can anybody advise....

    Kind Regards...

    • Edited by Funklet Thursday, October 02, 2008 12:58 PM
    Thursday, October 02, 2008 10:42 AM

Answers

  • There's definitely room for improvement in the code. For example, I see no reason to read the file as text using StreamReader and then convert back to binary using Encoding.UTF8, that's just a waste of memory. Instead open the file directly as binary using a FileStream.

    Also, to deal with files of any size, I'd recommend transfering the file in chunks instead of reading it all into memory at once. So I'd write something like

    byte[] buffer = new byte[10240];    // Read/write 10kb  
    using (FileStream sourceStream = new FileStream("testfile.txt", FileMode.Open))  
    {  
        int bytesRead;  
        do {  
            bytesRead = sourceStream.Read(buffer, 0, buffer.Length);  
            requestStream.Write(buffer, 0, bytesRead);  
        } while (bytesRead > 0);  



     


    Mattias, C# MVP
    • Marked as answer by Funklet Thursday, October 02, 2008 12:36 PM
    Thursday, October 02, 2008 11:25 AM

All replies

  • There's definitely room for improvement in the code. For example, I see no reason to read the file as text using StreamReader and then convert back to binary using Encoding.UTF8, that's just a waste of memory. Instead open the file directly as binary using a FileStream.

    Also, to deal with files of any size, I'd recommend transfering the file in chunks instead of reading it all into memory at once. So I'd write something like

    byte[] buffer = new byte[10240];    // Read/write 10kb  
    using (FileStream sourceStream = new FileStream("testfile.txt", FileMode.Open))  
    {  
        int bytesRead;  
        do {  
            bytesRead = sourceStream.Read(buffer, 0, buffer.Length);  
            requestStream.Write(buffer, 0, bytesRead);  
        } while (bytesRead > 0);  



     


    Mattias, C# MVP
    • Marked as answer by Funklet Thursday, October 02, 2008 12:36 PM
    Thursday, October 02, 2008 11:25 AM
  • Mattias,

    Many thanks for your swift response on this one.

    I have now seen quite a lot of posts regarding OOM with byte[] and it seems the only thing is to avoid having large byte[] in the first place.

    For those landing here with the same problem here is the complete function with the suggestion from Mattias included.

    It is essential that the path exists when uploading the file otherwise you will get a WebException.

            private static bool FTPUploadFile(String serverPath, String serverFile, FileInfo LocalFile, NetworkCredential Cred)  
            {  
                FtpWebResponse response = null;  
                try 
                {  
                
                FtpWebRequest request = (FtpWebRequest)WebRequest.Create(serverPath + serverFile);  
                request.Method = WebRequestMethods.Ftp.UploadFile;  
                request.Credentials = Cred;  
                byte[] buffer = new byte[10240];    // Read/write 10kb     
                using (FileStream sourceStream = new FileStream(LocalFile.ToString(), FileMode.Open))  
                    {  
                        using (Stream requestStream = request.GetRequestStream())  
                        {  
                        int bytesRead;  
                        do 
                        {  
                            bytesRead = sourceStream.Read(buffer, 0, buffer.Length);  
                            requestStream.Write(buffer, 0, bytesRead);  
                        } while (bytesRead > 0);  
                    }    
                    response = (FtpWebResponse)request.GetResponse();  
                    response.Close();  
     
                }  
                   return true;  
                }  
                catch(Exception ex)  
                {  
                    Console.WriteLine("Error in FTPUpload - " + ex.Message);  
                    if (response != null)  
                    {  
                        response.Close();  
                    }  
                    return false;  
                }  
            } 
    Thursday, October 02, 2008 12:41 PM