downloading multi-files via http?

Question

- Could u share your ideas?

My intent is to download more than 100 files from web server. Assume that my web server is http://111,222,333,444/ and i have 100 files(with small size) named "A1","A2"..."A100" within Data folder likes

http://111.222.333.444/Data/A1 .

 

Which is the best way to download all of these files with highest connection speed? 

 

Because of i applied WebRequest , WebResponse, and multi-threading(1 thread for 1 file),it has too long delay when connect to each file.Could u give me an adviced how to do this?

 

 

Answer 1

Multi-threading (as in posting the same message over and over again in this forum) is not a good idea...

Most webservers allow you to have only a limited number of concurrent connections.. (And if i'm not mistaken .net is also limited to have maximum two connections at the same time.)

Answer 2

Without describing the whole scenario of the problem, and if you don't show the source, don't espect good answers.

I don't know what are you mean with WebRequest and WebResponse, 'case WebClient is the way to go.

Answer 3

ok,this is the core problem code(this is not the real code but it's real step.)

 

Code Snippet

ArrayList filenameList=new ArrayList();
filenameList.Add("http://111.222.333.444/Data/A1");
filenameList.Add("http://111.222.333.444/Data/A2");
filenameList.Add("http://111.222.333.444/Data/A3");
...
filenameList.Add("http://111.222.333.444/Data/A10000");

 

for(int i=0;i<filenameList.Count;i++)
{
   Thread workThread=new Thread(new ParametrizeThreadStart(downloadMethod));
   workThread.Start(filenameList[i]);
}
.
.
.
public void downloadMethod(object TargetUrl)
{

   string Target=TargetUrl as string
   HttpWebRequest req = (HttpWebRequest)WebRequest.Create(Target);
   req.Timeout = 100000 * 3;
   req.Proxy = new WebProxy(new Uri("http://localhost:3128"));
   resp = (HttpWebResponse)req.GetResponse();
   Stream stm = resp.GetResponseStream();
   int ReadCount;
   byte[] buffer = new byte[65535];
   FileStream f = File.Open(SaveDirectory + "/" + filename, FileMode.Create
                  , FileAccess.Write, FileShare.None);
   if (stm.CanRead)
   {
     while ((int)(ReadCount = stm.Read(buffer, 0, buffer.Length)) > 0)
     {
         f.Write(buffer, 0, ReadCount);
     } 
   }
   f.Flush();
   f.Close();
}

 

 

 

Likes that,the main problem is "resp = (HttpWebResponse)req.GetResponse();" becuase it has too much delay time.

Can i use multi-threading like this?

Answer 4

I was especting you will also answer these questions:
-is this web server yours
-does you download files regulary, and all of them, or just new ones

If this server is yours, then you can create some application on the server, that will prepare an arhive file with all files, so you will download one file only.
If you download these files once, and after only new ones, then thinking about performance of downloading all files is unecessary.
If you did need to download all files always, then surely starting the download process for all files at once is not the faster way. You need to create some logic of downloading maximum 3 - 5 files. For example IE don't allow to download more then 2 files at once, because they think this is optimal way. You of course can increase this limit to more then 2. Other download manager applications also have some limits for number of files that are downloaded in one moment and that number is in most cases 3.
Now to return of your method of downloading file. I think that in .NET best way of downloading files is using WebClient class. This class have several way of doing this like, downloading whole file, like reading the file at once, or reading the file asinchronously. Which way you will chose depends of size of files, and on your application requrements. If you like to show the progress of download process, then asynchronous is only option. This option is best for bigger files. By big file these days, my criteria is > 50KB. If your files are very small then DownloadFile method can do the job. If you like to download more then one file then controling two or more threads will be the way to go. Every thread will have a method for downloading one file. When that job will be finished, then you will terminate the tread and start new one with next file in the list for downloading.

I have code that do many things, like showing progress for every file, and total progress, but code use arrays of files to be downloaded, do also other things, so if i post the whole source it will be very hard to extract what matters to you. So i made some correction to the code to be usefull. This code uses asynchronous way of downloading file and need to be packed to be able to use it in separate thread:

private WebClient _WebClient;

private Stream _Stream;

private FileStream _File;

private byte[] _DataBuffer = new byte[8192];

private long _TotalDownloadSize;

private long _TotalDownload;

private long _FileSize;

private int _BytesTransfered;

private int _Increment = 0;

 

private void Download()

{

    string url = "http://111.222.333.444/Data/A10000.extension";

    string localPath = "C:\\Data\\A10000.extension";

    _WebClient = new WebClient();

    _Stream = _WebClient.OpenRead(url);

    _File = new FileStream(localPath, FileMode.Create);

    _Stream.BeginRead(_DataBuffer, 0, 8192, AsynDownload, null);

}

 

private void AsynDownload(IAsyncResult ar)

{

    try

    {

        int intCount = _Stream.EndRead(ar);

        if (intCount < 1)

        {

            _Stream.Close();

            _File.Close();

            _WebClient.Dispose();

                FileComplete();

 

            return;

        }

 

        _BytesTransfered += intCount;

        _File.Write(_DataBuffer, 0, intCount);

        DisplayBar();

        _Stream.BeginRead(_DataBuffer, 0, 8192, AsynDownload, null);

    }

    catch (Exception e)

    {

        MessageBox.Show(e.Message);

    }

}

 

private delegate void FileCompleteInvoker();

 

private void FileComplete()

{

    if (InvokeRequired)

    {

        BeginInvoke(new FileCompleteInvoker(FileComplete));

        return;

    }

    _Increment++;

    _TotalDownload += _FileSize;

    progressUpdate.Value = (int)(100d / _TotalDownloadSize * _TotalDownload);

    Application.DoEvents();

    Download();

}

 

private delegate void BarInvoker();

 

private void DisplayBar()

{

    if (InvokeRequired)

    {

        BeginInvoke(new BarInvoker(DisplayBar));

        return;

    }

    progressFile.Value = (int)(100d / _FileSize * _BytesTransfered); //progressbar control for file progress

    progressUpdate.Value = (int)(100d / _TotalDownloadSize * (_TotalDownload + _BytesTransfered)); //progressbar control for total progress

    Application.DoEvents();

}