Error 404 'Not found' encoding problem?

Question

I have the following problem:

I am trying to download files from a web server using this code:

Public Function DownloadFileWithProgress(ByVal URL As String, ByVal Location As String) As Boolean
            Dim FS As FileStream = Nothing
            Try
                mCurrentFile = GetFileName(URL)
                Dim wRemote As WebRequest
                Dim bBuffer As Byte()
                ReDim bBuffer(256)
                Dim iBytesRead As Integer
                Dim iTotalBytesRead As Integer

                FS = New FileStream(Location, FileMode.Create, FileAccess.Write)
                wRemote = WebRequest.Create(URL)
                Dim myWebResponse As HttpWebResponse
                myWebResponse = wRemote.GetResponse
                RaiseEvent FileDownloadSizeObtained(myWebResponse.ContentLength)
                Dim sChunks As Stream = myWebResponse.GetResponseStream
                Do
                    iBytesRead = sChunks.Read(bBuffer, 0, 256)
                    FS.Write(bBuffer, 0, iBytesRead)
                    iTotalBytesRead += iBytesRead
                    If myWebResponse.ContentLength < iTotalBytesRead Then
                        RaiseEvent AmountDownloadedChanged(myWebResponse.ContentLength)
                    Else
                        RaiseEvent AmountDownloadedChanged(iTotalBytesRead)
                    End If
                Loop While Not iBytesRead = 0
                sChunks.Close()
                FS.Close()
                RaiseEvent FileDownloadComplete()
                Return True
            Catch ex As Exception
                If Not (FS Is Nothing) Then
                    FS.Close()
                    FS = Nothing
                End If
                RaiseEvent FileDownloadFailed(ex)
                Return False
            End Try
        End Function

I must admit that I did not write the original code myself but got it from a generous code project. However it suits my needs very well and functions perfectly as long as I don't use non_ASCII characters in the names of the folders and files on the web server.

If I use non-ASCII character  I get error 404 'not found'.

This is disastrous because my project is multilingual and I need to be able to use all characters FRANÇAIS ESPAÑOl etc. and Chinese etc if possible.

I know it is an encoding issue. I solved a similar problem involving an FTP request:

 Dim sr As New StreamReader(MyFTPWebRequest.GetResponse().GetResponseStream(), Encoding.Default) By adding Encoding.Default I was able to get the list of files that came from: MyFTPWebRequest.Method = WebRequestMethods.Ftp.ListDirectory

Should I be downloading using an FTPWebRequest? or can an HTTPWebRequest be adapted in the same way?

I used the above code because I find the FTPWebRequest system locks up the GUI and I cannot show the downloading progress.

Any enlightenment would be greatly appreciated.

Answer 1

I'm not sure which is the correct method to use.  One will work.  Your special characters are 8 bit ascii and you need to convert to 7 bit ascii.  I think methods 1 & 3 will give the same results, but not 100% sure.  Method 1 may just truncate the MSB (most significant) bit and method 3 may truncate MSB and remove special characters (0 to 15).

Method 1
        Dim URL As String = "FRANÇAIS ESPAÑOl"
        Dim utf7 As New System.Text.UTF7Encoding()
        Dim encodedBytes As Byte()
        encodedBytes = utf7.GetBytes(URL)
Method 2
        Dim URL As String = "FRANÇAIS ESPAÑOl"
        Dim utf8 As New System.Text.UTF8Encoding()
        Dim encodedBytes As Byte()
        encodedBytes = utf8.GetBytes(URL)
Method3
        Dim URL As String = "FRANÇAIS ESPAÑOl"
        Dim ascii As New System.Text.ASCIIEncoding()
        Dim encodedBytes As Byte()
        encodedBytes = ascii.GetBytes(URL)

---

jdweng

Answer 2

Are the URLs http?  Can you show use example of URLs that you're trying to access.  That'll let us see what format of encoding is being used there.  Are they using international character in the domain name or in the path part or both?  It might be Punycode for instance.

---

http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, please vote and/or mark the question answered. Available for contract programming.

Answer 3

Please pass the URL to System.Web.HttpUtility.UrlEncode() to convert it to format that can be safely passed to web servers.

Answer 4

Thanks to all for your help.

Unfortunately I haven't been able to apply any of it yet.

How would I use the encoded bytes methods? They presumably encode a byte at a time and would have to be part of a stream routine?

(I'm not convinced about using UTF7 either because the WebClient object has an Encoding property and I've tried all available encodings to no avail.)

The System.Web.HttpUtility.UrlEncode() method looks promising. I can add System.Web to the Imports but then I get 'HttpUtility' is not a member of 'Web' when I try to use it!?

The URLs are http, I think. (They always begin with http://). They are addresses of media files and txt files to be downloaded from an FTP site. They are part of an educational application that I have written. The non ASCII characters are not in the site name but in the folder and file names in the path. The folders are created and the files uploaded by another (FTP) part of my app so I could solve the problem by forbidding the use of non ASCII characters but I would prefer not to.

As I remarked in my previous post, perhaps I should abandon the HttpWebRequest download and use an FTPWebRequest download instead? I have tested this and it works.

Could anyone please explain why? Why does the FTPWebRequest not encounter the same encoding problem as the HttpWebRequest? I don't have to specify any encoding. I just pass the FTP address including ç é ê à etc. and there is no problem. Does the FTP system automatically detect the encoding and will it always do this?

Answer 5

First try using a webbrowser to get to the site.  See if you can connect.  You can check the webbrowser encoding setting by using gthe menu view - encondingg.  If you are going just going to upload/dowload files then you should be using FTP.  You can test FTP from a webrowser by putting into the webpbrowser a URL like FTP://ftp.microsoft.com.  For FTP to work you need to have the FTP service running on the server. 

---

jdweng

Answer 6

You need to "Add reference" to System.Web in order to see it if your project is not a web project.

The default System.dll does contain namespace for System.Web, but it only contains classes for setting permissions and nothing else.

You can actually check where the class is located in "View" -> "Obbect broswer" of Visual Studio to check which DLL contains what classes. The "Search" function these is helpful in lots of time because you can create custom set for other components too (E.g.: the classes for MS Office (VSTO) is a good choice).

Answer 7

First question can you access the files through a 'real' webbrowser MSIE/firefox/chrome?  Make sure you can, or maybe the webserver is itself incapable of sharing the files...

Anyway There can be differences how different HTTP devices encode non-ASCII characters in a URL.  All that really needs to work on the wide web is that if the webbrowser gets a URL from a webpage and uses the exact sequence of bytes included in the HTTP request there then it will work. Taking a printable URL containing non-ASCII characters and then converting it to a sequence of ASCII bytes to send in the HTTP protocol won't always work if the two systems use even slightly different conversion schemes.

For the two clients you are using FTP works for you and HTTP doesn't but it might be different for a different pair of endpoints.

HttpWebRequest doesn't do any URL encoding it just uses the Uri class.  So you should be able to do the simple test:

    string txt = <printable Unicode-ish URL>
    var url = new Uri(txt);

And compare what series of bytes is in the URL.  The URL specification describes using % signs to encode special characters, so you'll likely see a bunch of them (with two hex digits after each) if the URL contains non-ASCII characters.

Note that in my opinion using UrlEncode and HttpEncode etc are red-herrings.  They are use when one is storing data **within** a URL so would make the URL completely unlike a URL.

As I said before seeing an example of your filenames would really help.  Don't tell us the hostname of the web/ftp server if you want, just show use the filename parts.

---

http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, please vote and/or mark the question answered. Available for contract programming.

Answer 8

Thanks for your reply.<u1:p></u1:p>

No, I can't access the files with a 'real' browser.<u1:p></u1:p>

When I try to go here:<u1:p></u1:p>

www.mediacours.com/test/français/Méduses.jpg<u1:p></u1:p>

I get this:<u1:p></u1:p>

The requested URL /test/français/Méduses.jpg was not found on this server.<u1:p></u1:p>

As you say, "maybe the webserver is itself incapable of sharing the files" even though it's French!. Bearing this in mind, I have made my own coding system which will allow users to write correctly in French, Italian etc. As there are only about 70 characters involved I have used numerical codes between two markers "=" and "+" (These are forbidden for use with filenames in my program.)<u1:p></u1:p>

Here is an example:  Les élèves français ont dû être déterminés. becomes  Les =43+l=42+ves fran=41+ais ont d=60+ =44+tre d=43+termin=43+s. !!<u1:p></u1:p>

It looks pretty strange but it seems to work perfectly. I decided to do this when I read somewhere that the only certain thing about FTP sites  is that they can all use ASCII.<u1:p></u1:p>

This solves the problem for European languages that use Latin characters. I'll have to find something else for other languages. I was thinking of uploading html files containing columns of names with corresponding ASCII file/folder names  like 顰飝衈羺冐=A1 etc. A bit complicated ... and can Chinese FTP sites use ASCII, I wonder. <u1:p></u1:p>

Here's my system in case anyone else wants to use FTP sites to store non-ASCII files.

Probably not the best written code but the computer seems to understand it :-)

Public Function TextToFTPText(ByVal txt As String)
        Dim TheChar As Char
        Dim NumberCode As String
        Dim Done As Boolean
        txt = txt.Replace(" ", "_")
        Do Until Done = True
            For i As Integer = 0 To txt.Length - 1
                If i = txt.Length - 1 Then
                    Done = True
                End If
                TheChar = txt.Chars(i)
                If Asc(TheChar) > 127 Then
                    NumberCode = ExtCharToNumber(TheChar)
                    If NumberCode = -1 Then
                        Return ";" & TheChar
                        Exit Function
                    End If

                    NumberCode = "=" & NumberCode & "+"
                    txt = txt.Replace(TheChar, NumberCode)
                    Exit For
                End If
            Next
        Loop
        Return txt
    End Function

    Public Function FTPTextToText(ByVal txt As String)
        Dim NumberCode As String
        Dim Done As Boolean
        Dim StrBld As New StringBuilder
        Dim LastChar As Byte = txt.Length - 1
        Do Until Done = True
            For i As Integer = 0 To LastChar
                If i = LastChar Then
                    Done = True
                End If
                If Not txt.Chars(i) = "=" Then
                    StrBld.Append(txt.Chars(i))
                Else
                    If txt.Chars(i + 2) = "+" Then
                        NumberCode = txt.Chars(i + 1)
                        i += 2
                    Else
                        NumberCode = txt.Chars(i + 1) & txt.Chars(i + 2)
                        i += 3
                    End If
                    If i >= LastChar Then
                        Done = True
                    End If
                    StrBld.Append(NumberToExtChar(NumberCode))
                End If
            Next
        Loop
        txt = StrBld.ToString
        txt = txt.Replace("_", " ")
        Return txt
    End Function

    Public Function ExtCharToNumber(ByVal TheChar As Char)
        Dim ReturnValue As Integer = -1
        Dim TheChars As String = "Œ;Ž;ž;Ÿ:À;Á;Â;Ã;Ä;Å;Æ;Ç;È;É;Ê;Ë;Ì;Í;Î;Ï;Ð;Ñ;Ò;Ó;Ô;Õ;Ö;Ø;Ù;Ú;Û;Ü;Ý;Þ;ß;à;á;â;ã;ä;å;æ;ç;è;é;ê;ë;ì;í;î;ï;ð;ñ;ò;ó;ô;õ;ö;ø;ù;ú;û;ü;ý;þ;ÿ;Ŵ;Ý;Ŷ;ŵ;ý;ŷ;Ð"
        Dim CharDetail() As String = TheChars.Split(";")
        For i As Integer = 0 To CharDetail.Length - 1
            If CharDetail(i) = TheChar Then
                ReturnValue = i
                Exit For
            End If
        Next
        Return ReturnValue
    End Function

    Public Function NumberToExtChar(ByVal TheNumber As Byte)
        Dim TheChars As String = "Œ;Ž;ž;Ÿ:À;Á;Â;Ã;Ä;Å;Æ;Ç;È;É;Ê;Ë;Ì;Í;Î;Ï;Ð;Ñ;Ò;Ó;Ô;Õ;Ö;Ø;Ù;Ú;Û;Ü;Ý;Þ;ß;à;á;â;ã;ä;å;æ;ç;è;é;ê;ë;ì;í;î;ï;ð;ñ;ò;ó;ô;õ;ö;ø;ù;ú;û;ü;ý;þ;ÿ;Ŵ;Ý;Ŷ;ŵ;ý;ŷ;Ð"
        Dim CharDetail() As String = TheChars.Split(";")
        If TheNumber < CharDetail.Length Then
            Return CharDetail(TheNumber)
        Else
            Return "0"
        End If
    End Function

Answer 9

FTP has two mdoes of operation.  An Ascii mode and a Binary mode.  Ascii mode will translae CR/LF from one operating system to another operating system.  Yo were using HttpWebResponse so I didn't recommend FTP.

Thei website has the FTPwebrequest class

http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx

You can set to binary by setting the usebinary property to true

http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.usebinary.aspx

---

jdweng

Answer 10

Note that UrlEncode() have overload that contains Encoding parameter to let you specify the encoding to use when coding the path. If not specified, Unicode (UTF16, i.e. the internal encoding used by .NET framework) will be assumed.

You can try and see what encoding works for you (like UTF8, or your native code page).

It also helps to know if the server is hosted on Windows. Since NTFS v5+ will handle ANSI<=> Unicode translation and homographic equivalent Unicode characters automatically, if it's hosted on Windows Encoding is rarely needed to be specified.

Answer 11

Thanks for the info. I'm not sure if it's all relevant.

I probably did not make my problem clear at the outset.

I'm not trying to access files on a particular server. (I'm just usiing my own FTP site to test my program.) My program basically uploads media files (jpg, mpg, mp3 etc.) txt and rtf files to the user's FTP site. These files are subsequently downloaded by other users or in some cases used directly as the URL property of a Windows media player object.

I chose FTP because it still seems to be the most widely used system for upload/download. Maybe I should be using Windows Azure?

I cannot know in advance what encoding systems will be used on users' FTP sites can I? Unless I haven't understood your suggestions and there is a way to detect FTP encoding remotely.

I have read various articles on the web about FTP and it seems that it was invented at a time when ASCII was the system and no one was thinking about using international characters. The subsequent additional encoding systems seem a bit like "band aid" solutions to me and it seems there is no universal encoding system used on FTP sites except ASCII. Correct me if I'm wrong.

So, for the time being I will use my own system which "translates" latin extended characters "é" to "=43+" (ASCII only). I encode the folder and file addresses before uploading and decode them if and when files are downloaded.

Answer 12

When they invented FTP they were still using 75 baud tty terminals.  The RFC standards were developed by IEEE which includes FTP and were written to operate with legacy software on many different type computers.  Using ASCII mode no character were translated except the CR and LF.  Unix operating systems were just using CR and DOS was using CR/LF so FTP was performing translation when going between different operating systems.  FTP in binary mode also encoded and decoded the data so allow the data to get passed through computers that were looking for specifal characters like EOT to terminate messages.

FTP is binary mode is doing exactly the same as you are doing.  The encoding and decoding method FTP uses is defined in the RFC specifications and is standard.

---

jdweng

Answer 13

Okay, so it isn't web afterall. (Since you mentioned the file is on web server and ask if you can modify HTTPWebRequest to do this I assumed using web download is an alternative)

In that case I'll suggest that you login to the FTP server using "ftp" command from Windows, and issue "literal feat" command to see if the FTP server supports "UTF8" encoding. (You should be able to see a line with "UTF8" there if it is, you can also see a line beginning with "LANG" to see what codepage does it support)

We can determine what to do once the result is there.

The following is a sample screen:

C:\>ftp ftp.microsoft.com
Connected to ftp.microsoft.akadns.net.
220 Microsoft FTP Service
User (ftp.microsoft.akadns.net:(none)): anonymous
331 Anonymous access allowed, send identity (e-mail name) as password.
Password:
230-Welcome to FTP.MICROSOFT.COM. Also visit http://www.microsoft.com/downloads.
230 User logged in.
ftp> literal feat
211-Extended features supported:
 LANG EN*
 UTF8
 AUTH TLS;TLS-C;SSL;TLS-P;
 PBSZ
 PROT C;P;
 CCC
 HOST
 SIZE
 MDTM
 REST STREAM
211 END
ftp> quit
221 Thank you for using Microsoft products.
C:\>

As you can see, that FTP server supports UTF8. And if UTF8 mode is not specified, it supports using LANG command to change non-Unicode encoding to other English based locale (EN*)

Answer 14

Thanks for your replies. I don't understand everything.

Joel are you saying that there is a better system of overcoming the text encoding problems than mine. (As mine works, I feel inclined to stick with it.)

Cheong, I'm afraid it took me some time to figure out what you meant by sample screen and maybe I still haven't understood. Are you talking about using the DOS prompt to connect to the FTP server? I tried to do this. I get as far as password but then I can't enter the password. The characters I type never appear.

Are you saying that I can contact the server before doing any uploading/downloading and find out what encoding system it's using?

I can't find any methods in FTPWebRequest that do this. How should I be doing it?

Answer 15

I'm just saying you should stick with the standard method that has been around for 30 years.  Why write code that already is built into the FTP functionaliaty.

---

jdweng

Answer 16

Thanks. That's what I thought you meant. But how do I do it? You've probably already explained but I haven't understood.

How do I upload a file like Méduses.jpg and then download it again successfully? I can upload it but if I try to download it using the "é" character it won't be found.

Answer 17

Change the declaration as below.  The FTPWebRequest inherits all the properties of the WebRequest.

From

Dim wRemote As WebRequest

To

Dim wRemote As FTPWebRequest

then add following

wRemote.UseBinary =

True

---

jdweng

Answer 18

Thanks again. You're right it works! :-)

However, could you please confirm that the same procedure is unavailable for use with an HttpWebRequest?

Why would I want to use an HttpWebRequest? Because I want the user to see a progress bar and for this I need to get the file size. wRemote..Method = WebRequestMethods.Ftp.GetFileSize produces an error message. I assume the server cannot process this command. I could try to parse the results of wRemote.Method = WebRequestMethods.Ftp.ListDirectoryDetails but these seem to vary considerably between different servers and it might be a bit hit and miss. The Http download gets the filesize before downloading:

wRemote = WebRequest.Create(URL)
Dim myWebResponse As WebResponse = wRemote.GetResponse
RaiseEvent FileDownloadSizeObtained(myWebResponse.ContentLength)

Answer 19

HttpWebRequest and FTPWebrequest are siblings, both inherits all properties from WebRequest, but don't share non inherited properties/methods.

---

jdweng

Answer 20

Sorry for a certain amount of dumbness.

Can I get the file size using FTPWebRequest Yes/No?

Answer 21

Sorry again. It seems I can. Thanks :-)

Answer 22

I guess I'm boring you. Please bear with me. UseBinary is not a property of WebRequest and not a property of HttpWebRequest therefore I can't use it if the WebRequest is an HttpWebRequest, right?

I have found that using FtpWebRequest locks up the GUI until the operation is complete. To overcome this I use a separate exe for uploading. To enable the user to see what's happening the 2 apps communicate via a system I have made  using ManagedWinApi which displays a progress bar and the number of bytes uploaded.

I found a code project downloader (which uses HttpWebRequest). I preferred it to my uploading system because it doesn't lock up the GUI. That's why I ended up using a combination of HttpWebRequest and FtpWebrequest.

Is there any way to get round the "lock up problem" when an FtpWebRequest is used? (Maybe there is something in the HttpWebRequest downloader code that I haven't understood which allows it to overcome this problem).

If I can't find a way round the problem I guess I'll just have to write a third exe for downloading.

Answer 23

this is the webpage I posted on Tuesday.  Using the Receive Async method will prevent the lockup.  The Sync method is a blocking function that waits until all the data is received before continueing.

http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx

---

jdweng

Answer 24

I looked here: http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx

I can't find "Receive Async" or "Sync"

You say "Using the Receive Async method will prevent the lockup." This sounds promising! It's exactly what need. Where is it, please?

Answer 25

There is an example on the page.  It uses a Callback Event to perform the reading.  Above the example the webpage says this :

"The following code example demonstrates using asynchronous operations to upload a file to an FTP server"

---

jdweng

Answer 26

I used the code from this example in my uploader exe which I made about a month ago. Do you mean that you have altered the code recently?

If not, I don't understand because my uploading code blocks the GUI.

Of course, I may have altered something inadvertantly but in that case it seems surprising that my version works at all.

Answer 27

See webpage below.  Read the two sentaences I extracted fro the webpage.

http://msdn.microsoft.com/en-us/library/system.net.webrequest.getrequeststream

The GetRequestStream method initiates a request to send data to the Internet resource and returns a Stream instance for sending data to the Internet resource.

The GetRequestStream method provides synchronous access to the Stream. For asynchronous access, use the BeginGetRequestStream and EndGetRequestStream methods.

---

jdweng

Answer 28

Many thanks for your patience. I'm getting warmer. I'm calling it a day (my brain hurts) but I'll have another crack at it tomorrow.

I didn't understand when you said you "posted" the webpage on Tuesday. I imagined you must be the guy in charge of MSDN webpages! :-)

Answer 29

No! I just meant the I pu the link in my responmse on Tuesday.  Get some sleep.

---

jdweng

Answer 30

So I see you're getting progress.

But to followup the question (so others can try on the web later), yes, it's the FTP client in command prompt. The password will not be displayed, but it's standard convention that if anonymous login is allowed, you have to submit your email address as password. For most systems I don't bother to use real one, and just enter "guest@" there. Press <Enter> key afterwards.

Of course to connect to your own FTP server, you have to supply your username and password to do the testing.

FEAT command is defined in RFC-2389 to determine supported features of FTP server. The FTP client come with Windows doesn't support this command natively, but fortunately it has literal command that allow you to submit raw FTP command to the server, so it's a handy way to check what options do you have on hand when facing FTP transfer problems.

Answer 31

Hi Cheong,

I tried using the command prompt again and this time succeeded in sending the literal feat command. The result was disappointing. I got this:

MDIM

REST STREAM

SIZE

It doesn't seem very useful. Any thoughts?

Answer 32

Try putting the URL into t your webbrowser which may contain a proxy to allow youto connect.  the REST Stream I thinmk indicates that you just didn't connect.  the stream just timed out.  Some FTP sites have limits on the size of the file that can be transfered.  Try a smaller file.  On UNIX systems I sometimes had to use the split command to transfer files in pieces and then recombine the files.  That was 20 years ago.

---

jdweng

Answer 33

That means your FTP server does not support i18n features. All filenames need to be in the same code page as the server, or character with illegal "code points" (that is, the byte value of individual characters) in the filename will be replaced.

If possible, I recommand you to suggest upgrade the server if it's a public facing one because it's also a hint that the FTP server itself has some age. But that's a seperate problem.

Answer 34

Hi,

Thanks for your replies.

Topic 1: The FTP server.

You're right. It is an old server and I had an email from the hosting company recently saying that it is going to be shut down this summer and my site will migrate to a new server.

Topic 2: My attempts to create an FTP downloading system that does not lock up the user interface.

I'm getting nowhere. I can't get the filesize. When I use an ftpWebRequest I get -1

Here's the code:

Try
            Dim myCredentials = New NetworkCredential(UserName, Password)
            'Dim FileSize As Long
            MyFTPRequest = FtpWebRequest.Create("ftp://[e-mail]/www/MDC1/KHCLKLH3/teachers/CAT_Carol/FRANÇAIS/Chinese.rtf")
            MyFTPRequest.Credentials = myCredentials
            MyFTPRequest.Method = WebRequestMethods.Ftp.DownloadFile
            MyFTPRequest.UseBinary = True
            MyFTPRequest.Timeout = 10000
            MyFTPRequest.ReadWriteTimeout = 10000
            MyFTPRequest.KeepAlive = False
            MyFTPRequest.UseBinary = True
            MyFTPResponse = CType(MyFTPRequest.GetResponse, FtpWebResponse)
            MsgBox(MyFTPResponse.ContentLength)
            'MsgBox(FileSize)
            Dim responseStream As Stream = MyFTPResponse.GetResponseStream()
            Dim writestream As New FileStream("c:\test\testing.rtf", FileMode.Create)
            Const bufferLength As Integer = 2048
            Dim buffer As Byte() = New Byte(bufferLength - 1) {}
            Dim count As Integer = 0
            Dim readBytes As Integer = 0

            Do
                readBytes = responseStream.Read(buffer, 0, bufferLength)
                writestream.Write(buffer, 0, readBytes)
                count += readBytes
                'Update progress bar in Mediacours_Perso.exe
                'If ProgressWindowIsOpen() = True Then
                '    UpdateProgress(count)
                'End If
            Loop While readBytes <> 0
            responseStream.Close()
            writestream.Close()
        Catch ex As Exception
            MsgBox(ex.ToString)
        End Try

I cannot even replicate the system that works which uses this code:

Public Function DownloadFileWithProgress(ByVal URL As String, ByVal Location As String) As Boolean
            Dim FS As FileStream = Nothing
            Try
                mCurrentFile = GetFileName(URL)
                Dim wRemote As WebRequest
                Dim bBuffer As Byte()
                ReDim bBuffer(256)
                Dim iBytesRead As Integer
                Dim iTotalBytesRead As Integer

                FS = New FileStream(Location, FileMode.Create, FileAccess.Write)
                wRemote = WebRequest.Create(URL)
                Dim myWebResponse As WebResponse = wRemote.GetResponse
                RaiseEvent FileDownloadSizeObtained(myWebResponse.ContentLength)
                Dim sChunks As Stream = myWebResponse.GetResponseStream
                Do
                    iBytesRead = sChunks.Read(bBuffer, 0, 256)
                    FS.Write(bBuffer, 0, iBytesRead)
                    iTotalBytesRead += iBytesRead
                    If myWebResponse.ContentLength < iTotalBytesRead Then
                        RaiseEvent AmountDownloadedChanged(myWebResponse.ContentLength)
                    Else
                        RaiseEvent AmountDownloadedChanged(iTotalBytesRead)
                    End If
                Loop While Not iBytesRead = 0
                sChunks.Close()
                FS.Close()
                RaiseEvent FileDownloadComplete()
                Return True
            Catch ex As Exception
                If Not (FS Is Nothing) Then
                    FS.Close()
                    FS = Nothing
                End If
                RaiseEvent FileDownloadFailed(ex)
                Return False
            End Try
        End Function

If I use this as its author intended i.e. as part of his downloader class it works. If I try to use it elsewhere it doesn't.

I notice that it uses a webrequest (neither http nor ftp). If I try to use a webrequest like this to download a file with non ASCII characters in the name the file size returned is -1 (which is probably normal because it hasn't found the file). When I try to use it with a file with ASCII only characters it reurns the file size as 0 ! :-(

I may have to settle for using the clunky systems I've invented because they work.

Another question: I've tried uploading files with Chinese characters. I can't do it with my code and nor can FileZilla do it. Is this because the server is old? Would the ftpWebRequest UseBinary property allow the use of Chinese characters on a more recent server? If it would then I really should try to find an ftp downloading system and not use the WebRequest downloader.

Answer 35

Another question: I've tried uploading files with Chinese characters. I can't do it with my code and nor can FileZilla do it. Is this because the server is old? Would the ftpWebRequest UseBinary property allow the use of Chinese characters on a more recent server? If it would then I really should try to find an ftp downloading system and not use the WebRequest downloader.

Like what I've said before, your FTP server doesn't support i18n features, so the filename it can see is limited to the current locale of the server it's running.

If you prefer you can do UrlEncode() to the filename to make sure it can go the same across systems no matter whether it has i18n support. The FTP commands such as SIZE will surely work as intended this way. Note that you'll need to remember running UrlDecode() to get it's original filenameif you need to display it on a web page .

Answer 36

Thanks for your reply.

I think there still 2 topics:

Getting the file size with an ftpWebrequest.

I'm not sure that this is possible. On this page http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx it says:

"ContentLength - Infrastructure: Gets or set a value that is ignored by the FTPWebRequest class." What does "ignored" mean in this context? Could this explain why I always get a value of -1 when I try to get the ContentLength of an ftpWebRequest? (The 0 value I mentioned in my last message was, in fact, correct. I was testing with a very small file and I guess it's size was approximated to 0 bytes. I can get ContentLength if I use WebRequest but not if I use ftpWebRequest.

Using UrlEncode.

I added a reference to System.Web and I now have access to  System.Web.HttpServerUtility.UrlTokenEncode()

Is this what you meant by UrlEncode?

The problem is I don't know how to use it because the input value is Byte. How do I go from a string a characters to a Byte value?

Are you saying that I could use this to encode any string of characters including non latin ones? If so, this must be the definitive solution! (But it sounds too good to be true :-))

Answer 37

Thanks for your reply.

I think there still 2 topics:

Getting the file size with an ftpWebrequest.

I'm not sure that this is possible. On this page http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx it says:

"ContentLength - Infrastructure: Gets or set a value that is ignored by the FTPWebRequest class." What does "ignored" mean in this context? Could this explain why I always get a value of -1 when I try to get the ContentLength of an ftpWebRequest? (The 0 value I mentioned in my last message was, in fact, correct. I was testing with a very small file and I guess it's size was approximated to 0 bytes. I can get ContentLength if I use WebRequest but not if I use ftpWebRequest.

No. You have to set FtpWebRequest.Method to WebRequestMethods.Ftp.GetFileSize and issue the request (i.e.: issue the SIZE command to FTP server, that from your previous FEAT result, is supported by your server). After sending the request you can read the response that is always string representation of filesize in bytes. (Something like perform NetworkStream.Read() and then send result to System.Text.Encoding.Default.GetString().Trim() and then pass it to Int32.Parse() )

Alternatively you may issue WebRequestMethods.Ftp.ListDirectoryDetails command to retrieve filenames and size of all files in the folder. Just note that if your FTP server is on *nix system, the returned size in this way could be in blocks (i.e.: 512 bytes) instead of bytes or kilobytes.

Using UrlEncode.

I added a reference to System.Web and I now have access to  System.Web.HttpServerUtility.UrlTokenEncode()

Is this what you meant by UrlEncode?

The problem is I don't know how to use it because the input value is Byte. How do I go from a string a characters to a Byte value?

Are you saying that I could use this to encode any string of characters including non latin ones? If so, this must be the definitive solution! (But it sounds too good to be true :-))

No, what I was referring to is HttpServerUtility.UrlEncode(), it accepts string as input. Just pass the whole filename to it and it'll encode whatever characters that's not safe in an URL to escaped sequence in ASCII range, therefore makes it safe to use in applications that neither support foreign codepages nor Unicode. The characters that is already safe to be in an URL is left untouched.

Answer 38

Many thanks for your reply. I have managed to find and use HttpServerUtility.UrlEncode and HttpServerUtility.UrlDecode. They are brilliant! Exactly what I needed. I can apply them immediately because they will replace the encoding system I invented and encode all types of character French, Chinese etc. etc. :-)

I was not able to use WebRequestMethods.Ftp.GetFileSize. You wrote "i.e.: issue the SIZE command to FTP server, that from your previous FEAT result, is supported by your server." Did you perhaps mean "is not supported"? I had already thought about Ftp.ListDirectoryDetails but was put off by the fact that different types of server seem to produce wildly different types of list.

Anyway, it doesn't really matter any more because with UrlEncode I can use the WebRequest which will now be able to handle the file and folder paths. I must remember to forbid the use of "%" for file and folder names. :-)

Answer 39

Afterthought:  I guess you did mean "is supported" because SIZE was in the FEAT result but I definitely can't get the size with the FtpWebRequest.

Answer 40

It's strange that the SIZE string is returned from FEAT command but is not supported, something is wrong there. If you still want to use that command you can post the code for getting size here for me to take a look.

Regarding forbidding the use of "%" in filename, let me reassure you that "%" would be escaped to "%25" too so no need to forbid the use of it if it's supported in the filesystem.

EDIT: So I got something wrong after trying the command. Here's sample code that works:

            FtpWebRequest req = (FtpWebRequest)WebRequest.Create("ftp://ftp.microsoft.com/developr/readme.txt");
            req.Method = WebRequestMethods.Ftp.GetFileSize;
            req.Credentials = new NetworkCredential("anonymous", "guest@");
            FtpWebResponse resp = (FtpWebResponse)req.GetResponse();
            try
            {
                string[] strresponse = resp.StatusDescription.Split(' ');
                if (strresponse[0] == "213")
                {
                    int filesize = Int32.Parse(strresponse[1], System.Globalization.CultureInfo.InvariantCulture);
                    Console.WriteLine("Filesize for readme.txt is {0}", filesize);
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error: {0}", ex.Message);
            }
            Console.ReadKey();

What I got it wrong is that you have to get the response from the response stream (I got that impression because I was writing the FTP client myself and always executing the command that way, but apparently .NET framework made it simplier by let you get it from StatusDescription.

Answer 41

Hi,

Thanks for your reply.

This line: Dim resp As FtpWebResponse = DirectCast(req.GetResponse(), FtpWebResponse) produces this error (translated from French):

The remote server sent an error: (550) File not available(for example, file not found, access denied).

(Le serveur distant a retourné une erreur : (550) Fichier non disponible (par exemple, fichier introuvable, accès impossible).

But the file is definitely there because I can download it using the WebRequest download code.

BTW I shall be out of internet reach for about a week so if I don't reply ... maybe in a week's time :-)

Answer 42

FTP supports downloading with wildcards like get * and mget * (multiple files).  You can avoid the problem with the name of the file if you use the wildcard.

---

jdweng

Answer 43

When you say "the problem" are we talking about the same thing? Downloading is not the problem. The problem is getting the size of the file before downloading.

Where should the "get *" be put?

Can it be part of this code?

 FtpWebRequest req = (FtpWebRequest)WebRequest.Create("ftp://ftp.microsoft.com/developr/readme.txt");
            req.Method = WebRequestMethods.Ftp.GetFileSize;
            req.Credentials = new NetworkCredential("anonymous", "guest@");
            FtpWebResponse resp = (FtpWebResponse)req.GetResponse();
            try
            {
                string[] strresponse = resp.StatusDescription.Split(' ');
                if (strresponse[0] == "213")
                {
                    int filesize = Int32.Parse(strresponse[1], System.Globalization.CultureInfo.InvariantCulture);
                    Console.WriteLine("Filesize for readme.txt is {0}", filesize);
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error: {0}", ex.Message);
            }
            Console.ReadKey();

Answer 44

this posting has been going on for so long I forgot we were ataching the file size issue.  Yo can always extract the file size from the reults of a FTP LS or DIR command.

---

jdweng

Answer 45

OK, Thanks. As I said once before I don't have any problems now. I can use the WebRequest downloader code because the System.Web.HttpUtility.UrlEncode/UrlDecode allows me to create ASCII only folder and file names for use on any FTP site.

I think Cheong is curious to know why getting the size of the file on this server is impossible despite the fact that the server returned SIZE after a FEAT request.

Answer 46

Indeed. If the server is implementing FTP standard, there's no reason that a feature that it "explicitly declared as supported" not working.

FTP error code 550 is usually related to file path or command syntax. If Your FTP server has verbose debugging log and don't think it's too much trouble, you can wipe the sensitive detail and post here to give me a look.

I visit technical forums to sample real world problems so that I can prepare myself before actually have to face them personally. Therefore if I believe it's very probable that I'd face the type of problem in the future, I don't mind going extra mile to discover the root cause.

Answer 47

Hi,

I've not had access to the internet for a few days. I can't find anything that looks like a debugging log. I've tried using different paths and files but I always get the same error. As I mentioned earlier this server is about to be phased out. Could its age be causing the problem? I've looked on the host company's forum but not found anything relevant. Sorry I can't help much. I've put a question here http://forum.nuxit.com/discussions/obtenir-la-taille-d-un-fichier-t1022.html

Answer 48

I saw one interesting suggestion here. See if it applies to your case.

Answer 49

Yes, the symptoms seem the same but I don't really understand the way the ftp address is put together.

I don't use "ftp://ftp ..." I have tried this but I always get a "not connected" error.

The adddress I use is like this: "ftp://[username]@ftp6.provider.net/[foldername]/[filename]"

Should I be using something else? The address format I use allows me to upload/download delete etc. using both FtpWebRequest and WebRequest but it returns error 550 if I attempt to get the file size with an FtpWebRequest.

Answer 50

Try ftp://[username]@ftp6.provider.net//[foldername]/[filename] . The additional "/" after hostname can trick it to think it's relative to root folder. And on FTP servers that guards "upload level from root" (Mind you, some of them don't. For example, the FTP server that come with Solaris in my school didn't guard it.), it should make it start recursing from your FTP user's home folder.

See if it works. If it doesn't, perheps you can also try ftp://[username]@ftp6.provider.net/~/[foldername]/[filename] where "~" means home folder for current user.

Answer 51

No, both addresses return error 550.