none
FtpWebRequest class: The underlying connection was closed: The server committed a protocol violation RRS feed

  • Question

  • Hi everyone, as mentioned in the title, i have an problem with FtpWebRequest.

    I'm developing an application that connect to about 1000 ftp server and upload one file to each of them.
    I read this article and use it in my code:
    https://msdn.microsoft.com/en-us/library/ms229715(v=vs.110).aspx

    Everything works fine when i upload to each FTP server synchronously (using only one thread).
    But you know, 1000 server so it's a little bit slow, about 4 hour for all of them.  So i use multithread approach.

    Since that, an exception was thrown when GetRequestStream method reached.
    I also tried: 

    KeepAlive = false; (default is true).

    EnableSsl = false;

    only about 150 - 200 connect requests are success, anothers thrown an exception.

    Sorry i can't show my code but i'm sure that each thread is completely isolate.

    the only thing i can privode to you is my stacktrace (logged by log4net):

    08-10-2016 10:59:53 ERROR : [MB.UPDATER.BL.UpdaterBL] - ConnectionErrorException: Connect to server error:  ---> System.Net.WebException: The underlying connection was closed: The server committed a protocol violation.
       at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
       at System.Net.FtpWebRequest.RequestCallback(Object obj)
       at System.Net.CommandStream.Dispose(Boolean disposing)
       at System.IO.Stream.Close()
       at System.IO.Stream.Dispose()
       at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
       at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
       at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
       at System.Net.FtpWebRequest.GetRequestStream()
       at MB.UPDATER.BL.UpdaterBL.UploadAndVerrifyFileToFtp(String localfilepath, UpdateProfile profile, Byte[] fileArr, Action`2 progressReport) in d:\Thang\UpdateApiWebChiliConsole\MB.UPDATER.BL\UpdaterBL.cs:line 440
       --- End of inner exception stack trace ---
       at MB.UPDATER.BL.UpdaterBL.UploadAndVerrifyFileToFtp(String localfilepath, UpdateProfile profile, Byte[] fileArr, Action`2 progressReport) in d:\Thang\UpdateApiWebChiliConsole\MB.UPDATER.BL\UpdaterBL.cs:line 455
       at MB.UPDATER.BL.UpdaterBL.<>c__DisplayClasse.<>c__DisplayClass16.<BeginUpload>b__b() in d:\Thang\UpdateApiWebChiliConsole\MB.UPDATER.BL\UpdaterBL.cs:line 261


    Thanks.



    Sunday, October 9, 2016 12:39 PM

Answers

  • Hi Lê Khả Sỹ,

    Thanks for posting more useful code.

    Based on your code, I would suggest you add a try catch statement before WebRequest.Create.

    Check if there is same name  between profile.ServicesName and filename parameters or some other error. I wonder if there is any error on your server, just based on the client code, it looks good for me.

    Best regards,

    Kristin

     

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Tuesday, October 11, 2016 6:20 AM
  • You could try System.Net tracing as described in How to: Configure Network Tracing. Perhaps that would show what exactly goes wrong.

    If the .NET Framework has a bug that only occurs when you generate a flood of FTP requests like this, perhaps you can work around it by adding e.g. Thread.Sleep(200) between the Task.Run calls. That would make it slower but it should still take less than the original 4 hours. It's hard to say how long the sleep would have to be, though. If I ended up doing this, I might run a second loop with a longer sleep for those servers whose FtpWebRequests threw the exception in the first loop.

    Tuesday, October 11, 2016 9:39 PM

All replies

  • Hi

    >>I'm developing an application that connect to about 1000 ftp server and upload one file to each of them.

    You should not attempt to do this through your code.  What happens if one of the servers is down?  You would need to detect this, and then sync the file later when it came back up.

    Based on your scenario, I would suggest you use Shared Storage. The files are saved to a network share accessible by both servers.  You don't need to copy each file to 1000 FTP servers. Actually, we normally use some code to upload multiple files to an FTP server asynchronously.

    Best regards,

    Kristin


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, October 10, 2016 7:31 AM
  • Hi Kristin,

    All these FTP server is client's websites in my company's local network. This tool is kind of updater for these server.
    The project manager doesn't want to waste time for a version control system because this is a old system and will be replaced soon.

    If some servers are down, it will be logged into a file and will be updated manually in another day, we accept that.

    however, I still want to know why that exception is thrown.
    thank you.

    Monday, October 10, 2016 8:47 AM
  • Hi Lê Khả Sỹ,

    Just based on your error information, I cannot locate where the error is.

    I would suggest you debug your code step by step.

    Here is a  example demonstrates using asynchronous operations to upload a file to an FTP server.

    using System;
    using System.Net;
    using System.Threading;
    
    using System.IO;
    namespace Examples.System.Net
    {
        public class FtpState
        {
            private ManualResetEvent wait;
            private FtpWebRequest request;
            private string fileName;
            private Exception operationException = null;
            string status;
    
            public FtpState()
            {
                wait = new ManualResetEvent(false);
            }
    
            public ManualResetEvent OperationComplete
            {
                get {return wait;}
            }
    
            public FtpWebRequest Request
            {
                get {return request;}
                set {request = value;}
            }
    
            public string FileName
            {
                get {return fileName;}
                set {fileName = value;}
            }
            public Exception OperationException
            {
                get {return operationException;}
                set {operationException = value;}
            }
            public string StatusDescription
            {
                get {return status;}
                set {status = value;}
            }
        }
        public class AsynchronousFtpUpLoader
        {  
            // Command line arguments are two strings:
            // 1. The url that is the name of the file being uploaded to the server.
            // 2. The name of the file on the local machine.
            //
            public static void Main(string[] args)
            {
                // Create a Uri instance with the specified URI string.
                // If the URI is not correctly formed, the Uri constructor
                // will throw an exception.
                ManualResetEvent waitObject;
    
                Uri target = new Uri (args[0]);
                string fileName = args[1];
                FtpState state = new FtpState();
                FtpWebRequest request = (FtpWebRequest)WebRequest.Create(target);
                request.Method = WebRequestMethods.Ftp.UploadFile;
    
                // This example uses anonymous logon.
                // The request is anonymous by default; the credential does not have to be specified. 
                // The example specifies the credential only to
                // control how actions are logged on the server.
    
                request.Credentials = new NetworkCredential ("anonymous","janeDoe@contoso.com");
    
                // Store the request in the object that we pass into the
                // asynchronous operations.
                state.Request = request;
                state.FileName = fileName;
    
                // Get the event to wait on.
                waitObject = state.OperationComplete;
    
                // Asynchronously get the stream for the file contents.
                request.BeginGetRequestStream(
                    new AsyncCallback (EndGetStreamCallback), 
                    state
                );
    
                // Block the current thread until all operations are complete.
                waitObject.WaitOne();
    
                // The operations either completed or threw an exception.
                if (state.OperationException != null)
                {
                    throw state.OperationException;
                }
                else
                {
                    Console.WriteLine("The operation completed - {0}", state.StatusDescription);
                }
            }
            private static void EndGetStreamCallback(IAsyncResult ar)
            {
                FtpState state = (FtpState) ar.AsyncState;
    
                Stream requestStream = null;
                // End the asynchronous call to get the request stream.
                try
                {
                    requestStream = state.Request.EndGetRequestStream(ar);
                    // Copy the file contents to the request stream.
                    const int bufferLength = 2048;
                    byte[] buffer = new byte[bufferLength];
                    int count = 0;
                    int readBytes = 0;
                    FileStream stream = File.OpenRead(state.FileName);
                    do
                    {
                        readBytes = stream.Read(buffer, 0, bufferLength);
                        requestStream.Write(buffer, 0, readBytes);
                        count += readBytes;
                    }
                    while (readBytes != 0);
                    Console.WriteLine ("Writing {0} bytes to the stream.", count);
                    // IMPORTANT: Close the request stream before sending the request.
                    requestStream.Close();
                    // Asynchronously get the response to the upload request.
                    state.Request.BeginGetResponse(
                        new AsyncCallback (EndGetResponseCallback), 
                        state
                    );
                } 
                // Return exceptions to the main application thread.
                catch (Exception e)
                {
                    Console.WriteLine("Could not get the request stream.");
                    state.OperationException = e;
                    state.OperationComplete.Set();
                    return;
                }
    
            }
    
            // The EndGetResponseCallback method  
            // completes a call to BeginGetResponse.
            private static void EndGetResponseCallback(IAsyncResult ar)
            {
                FtpState state = (FtpState) ar.AsyncState;
                FtpWebResponse response = null;
                try 
                {
                    response = (FtpWebResponse) state.Request.EndGetResponse(ar);
                    response.Close();
                    state.StatusDescription = response.StatusDescription;
                    // Signal the main application thread that 
                    // the operation is complete.
                    state.OperationComplete.Set();
                }
                // Return exceptions to the main application thread.
                catch (Exception e)
                {
                    Console.WriteLine ("Error getting response.");
                    state.OperationException = e;
                    state.OperationComplete.Set();
                }
            }
        }
    }
    
    
    For more information, please refer to

    ttps://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest(v=vs.100).aspx

    Since you have 1000 FTP server, you should create multiple FtpWebRequest classes, please have a try.

    Best regards,

    Kristin


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, October 11, 2016 2:29 AM
  • Hi Kristin Xie,

    Now i can post a "minimum version" of my code.
    it's preaty simple:

    void Copy(byte[] arr, Stream desStream)
            {
                var bufferSize = 8 * 1024;
                var looptime = arr.Length / bufferSize;
                var oddSize = arr.Length - looptime * bufferSize;
    
                // copy
                for (int i = 0; i < looptime; i++)
                {
                    var buffer = new byte[bufferSize];
                    Array.Copy(arr, i * bufferSize, buffer, 0, bufferSize);
                    desStream.Write(buffer, 0, bufferSize);
                }
    
                // copy odd
                var oddbuffer = new byte[oddSize];
                Array.Copy(arr, arr.Length - oddSize, oddbuffer, 0, oddSize);
                desStream.Write(oddbuffer, 0, oddSize);
            }
    
    
    
    private void UploadAndVerrifyFileToFtp(string filename, UpdateProfile profile, byte[] fileData)
    {
                string filePathOnServer = string.Format("ftp://{0}/httpdocs/{1}", profile.ServicesName, filename);
                FtpWebRequest request = (FtpWebRequest)WebRequest.Create(filePathOnServer);
                request.Method = WebRequestMethods.Ftp.UploadFile;
    
                request.Credentials = new NetworkCredential(profile.ServerUser.Normalize(), profile.ServerPassword.Normalize());
    
                request.KeepAlive = false;
    
                request.EnableSsl = false;
    
                using(Stream ftpStream = request.GetRequestStream()) // this line sometime throw an exception
    	    {
    	        Copy(fileArr, ftpStream)
    	    }
    }
    
    void Main()
    {
    	var filename = "file.php";
    
    	var UpdateList = GetUpdateProfiles();
    
    	byte[] UploadData = GetUploadData();
    
    	Queue<Task> Tasks = new Queue<Task>();
    
    	foreach (var item in UpdateList)
    	{
    		var profile = item;
    		Task task = Task.Run(new Action(() =>
    		{
    			UploadAndVerrifyFileToFtp(filename, profile, UploadData );
    		}
    		Tasks.Enquque(task);
    	}
    
    	while (Tasks.Count != 0)
    	{
    		Tasks.Dequeue().Wait();
    	}
    }

    i don't know what's wrong with my code.
    Implement another code could take time, so i just can compare your code and my code, find the different.
    Unfortunately, i don't see anything special.


    Tuesday, October 11, 2016 3:49 AM
  • Hi Lê Khả Sỹ,

    Thanks for posting more useful code.

    Based on your code, I would suggest you add a try catch statement before WebRequest.Create.

    Check if there is same name  between profile.ServicesName and filename parameters or some other error. I wonder if there is any error on your server, just based on the client code, it looks good for me.

    Best regards,

    Kristin

     

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Tuesday, October 11, 2016 6:20 AM
  • You could try System.Net tracing as described in How to: Configure Network Tracing. Perhaps that would show what exactly goes wrong.

    If the .NET Framework has a bug that only occurs when you generate a flood of FTP requests like this, perhaps you can work around it by adding e.g. Thread.Sleep(200) between the Task.Run calls. That would make it slower but it should still take less than the original 4 hours. It's hard to say how long the sleep would have to be, though. If I ended up doing this, I might run a second loop with a longer sleep for those servers whose FtpWebRequests threw the exception in the first loop.

    Tuesday, October 11, 2016 9:39 PM
  • Turn out, it's a .net bug.
    So i use another ftp lib called DotNetFTPLiblary: 
    https://sourceforge.net/projects/dotnetftplib/
    it solve my problem.
    Tuesday, April 25, 2017 2:34 AM