FtpWebRequest exception when closing RequestStream

Question

I've just moved an application to a new machine.  This one is XP x64 the last one was XP x86.

The application uploads files (among other things) and no longer works doing so.  The problem is that when I close the stream returned by GetRequestStream an exception 'System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.' is thrown.

The stack trace looks like this

 

   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.FtpWebRequest.RequestCallback(Object obj)
   at System.Net.CommandStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.FtpWebRequest.RequestCallback(Object obj)
   at System.Net.CommandStream.Abort(Exception e)
   at System.Net.CommandStream.CheckContinuePipeline()
   at System.Net.FtpWebRequest.DataStreamClosed(CloseExState closeState)
   at System.Net.FtpDataStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
   at System.Net.FtpDataStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at EKProcessor.FTPUpload.FTPFile(String& StatusDescription)

My code is this

FtpWebRequest ftp = (FtpWebRequest)WebRequest.CreateDefault(ftpFile);
string userName;
string password;
string[] splittedUser = FtpUser.Split(new char[] { ':' });
userName = splittedUser[0];
password = splittedUser[1];
ftp.Credentials = new NetworkCredential(userName, password);
ftp.Method = WebRequestMethods.Ftp.UploadFile;
ftp.UseBinary = true;
ftp.KeepAlive = false;
ftp.UsePassive = true;
ftp.Timeout = FtpRequestTimeout;
ftp.ReadWriteTimeout = FtpReadWriteTimeout;

using (Stream requestStream = ftp.GetRequestStream())
{

  int bufferLength = FtpBufferSize;
  byte[] buffer = new byte[bufferLength];
  int count = 0;
  int readBytes = 0;
  using (FileStream stream = File.OpenRead(localFile))
  {
    long length = stream.Length;
    do
    {
      readBytes = stream.Read(buffer, 0, bufferLength);
      if (readBytes > 0)
      {
        requestStream.Write(buffer, 0, readBytes);
        count += readBytes;
        log.Info(string.Format("Written {0} bytes of {1}", count, length));
      }
    }
    while (readBytes != 0);
  }
  log.Info(string.Format("About to close requestStream for {0}, written {1} bytes", localFile, count));

I've set the timeouts to larger values than necessary after reading comments on similar problems

The file I'm uploading appears to be entirely there despite the exception.

Can anyone suggest what I can try next? It does seem as if it's connected with the OS (x64 rather than x86), but I'm not sure where that gets me!

Iain

  requestStream.Close();  // ** HERE I CRASH
  log.Info(string.Format("Have closed requestStream for {0}", localFile));
      }

---

Iain Downs

Answer 1

It seems like a timeout issue.  Does this happen for all sized files?

You could take a network trace with Netmon to see if the server (or something downstream in your network) is closing the connection prematurely.

I to not see in your code where you are calling ftp.GetResponse().  Make sure you do that.  Also you can try turning pipelining off ftp.Pipelined=false;

 

 

---

Jeff Sanders (MSFT)

Answer 2

Thanks, Jeff.

It seems to happen for all sized files.

What seems to happen is that the process successfully uploads the whole file (at the maximum bandwidth - which isn't a lot).  Then it tries to close the connection.  At this point you see some intermittent traffic on the network connection and 20 seconds later you get the exception.

I don't call get response because I have an exception in closing the request before I get there!  I was surprised to see in the docs that you close the request before you open the response, but that's how it seems to be.

I don't know what you mean by ftp.Pipelined=false;  I can't find such a property on the FtpWebRequest object.

I'm sort of hoping to avoid using NetMon.  I get tto use it once every two or three years, so everytime is a new learning experience :(!

 

Iain

---

Iain Downs

Answer 3

Iaim,

That is good information (that the file seems to upload).  Sorry about the Pipelined statement.  It is supported on the underlying HttpWebRequest object not the FtpWebRequest.  It was a shot in the dark anyhow.  What you could do is this:

HttpWebRequest aReq= (HttpWebRequest)WebRequest.CreateDefault(ftpFile);
aReq.Pipelined=false;

But my guess is that this is not the real issue.  Give it a try.

You can take a .Net trace of the success and failed trace to see what differences their may be: http://blogs.msdn.com/b/jpsanders/archive/2009/03/24/my-favorite-system-net-trace-configuration-file-dumps-process-id-and-date-time-information.aspx

Finally compare the versions of the framework.  Make sure you are running with all the latest updates by going to Windows Update.

 

FtpWebRequest ftp = (FtpWebRequest)aReq;

---

Jeff Sanders (MSFT)

Answer 4

In the end this seems to have been a timeout issue.

WHen I set the timeouts to -1 it started working again.

I'm not entirely clear why it stopped working in the move from one machine to another, but there you are!

 

Iain

---

Iain Downs

Answer 5

Oops.  Ignore that lost comment.  It's just been a while since I last looked at the problem.

In fact the problem seems to be a 'known' one, just not one with any clear answers.

With the timeout set to -1, and with files over a certain size, the routine above hangs in the exit from the using requestStream.  Or in the requestStream.Close() method if this is called explicitly.

The problem occurs ONLY on one machine (that I know of) which is an XP x64 SP2 machine.

the problem occurs in my app and in a simple test program which is pretty much as above.

If I do NOT close the requestStream then I cannot open the response stream (it hangs here).

If I do not close the requestStream and do not try and open the response stream, then the first upload works but susequent uploads may work, may fail with 'file unavailable (550)' or may hang during the process of getting the requestStream.

So;  a machine dependent problem where the close of an FtpRequestStream hangs after a large amount of data (in my case > 6MB or so ) is written to it.  And one which you can find a dozen or so times in the groups, but with no clear solution.

I really have no idea what to try next...

Iain

---

Iain Downs

Answer 6

I've dusted off netmon and tried to do something with it.  I don't get it.

On my machine I get a parse of the FTP component so I can see the ftp commands winging their way backwards and forwards.

On the machine which doesn't work, I only get TCP packets and, as far as I can see these are purely the data being uploaded - that is to say the directory change PASV and other commands simply aren't shown.

On both machines I've downloads the latest x64 netmon and used it as is.  At least I think so.  I'm not entirely clear if I need to do something to get a parser running or if it is automatic.  On the machine which doesn't work properly it looks almost as if the ftp elements are being parsed out.

Any pointers much appreciated.

 

Iain

 

 

---

Iain Downs

Answer 7

In Netmon go to 'Tools', 'Options', click on the 'Parser Profiles' tab and select the 'Default' parser and click 'Set as Active'.  That one and the 'Widnows' one are the two most popular.

---

Jeff Sanders (MSFT)

Answer 8

I've had a bit more success with WireShark.

As best I can tell, the preamble goes up in both cases and all the data.  On the working machine there is then a sequence

DEST->226 Transfer Complete

Source->QUIT

DEST->221

On the non functioning machine this does not happen.  I'm not clear if this is because there is some sort of TCP end of stream command missing or if there is some other reason.

Iain

---

Iain Downs

Answer 9

Without being able to fully troubleshoot the traces myself, I am afraid trying to get support via the user forum will not be enough for you.

One last thing that may be some help is .NET tracing.  You may be able to detect what is different between the two machines.  See:

http://blogs.msdn.com/b/jpsanders/archive/2009/03/24/my-favorite-system-net-trace-configuration-file-dumps-process-id-and-date-time-information.aspx

and

http://msdn.microsoft.com/en-us/library/ty48b824.aspx

 

---

Jeff Sanders (MSFT)

Answer 10

What I've done as a work around is to catch the exception (caused more or less by the timeout) and discard it.  I also do not attempt to get the response from the ftp server.

This feels entirely wrong, but it seems to have fixed the problem for the moment.  If I get time I may try and follow this up with MSDN support to try and get a proper fix in place!

 

Iain

---

Iain Downs

Answer 11

Hi,

is some new info about this problem? I have this problem too.

 

Thx

Martin

Answer 12

Did you follow all the troubleshooting steps listed above?  What does your .NET trace show?

---

Jeff Sanders (MSFT)

Answer 13

Hey guys,

I know this is an old post now... but I have a similar problem and it seems there are still no real answers for it.

Using FtpWebRequest to download a file and write using a FileStream I can download a smallish file 30MB+ without any problems. When I try to download 90MB+ I get a WebException at ResponseStream.Read().

The download completes and then I get the exception (it's a zip file and even after the exception I can unzip it no problem). I used wireshark and .Net tracing and I found with .Net the 226 (transfer complete) never arrives at the end and with Wireshark I see it come in as [TCP Retransmission] this happens for my work PC and another testers PC but not my home PC?

I can do the same as Iain and just ignore the error and continue (which definitely feels wrong) but unfortunately I really want to use WebClient which using FtpWebRequest itself suffers the same problem only it seems I can't get the exception it will just sit there for a long time...

I think I could override some bits here and there to make sure it works but it's a little out of my depth at the moment (still new to programming) and it really seems like a hack rather than a fix.

If anyone has ever come across a solution for this please let us know :)

Dave

Answer 14

Hey Dave,

When you say you don't see the 226, are you saying the server never sends the 226?  If so, there is a problem with your network or the server.

-Jeff

---

Jeff Sanders (MSFT)