Download or upload of file, supporting progress reporting

Question

Hi...

I guess this is must be a common thing to look for. Do you know any articles covering this topic? I'm using WCF of .NET 3 in VS 2005.

Thanks

Answer 1

Hi,

 

if you use an object which derivates from the Stream class you can use the BytesRead property to get info om how many bytes have been read.

 

Regards

 

Régis Baccaro

Answer 2

Can you be a little more specific. There is not BytesRead property in stream class.

Answer 3

For downloading files, since you'd get the stream to read from it, it's quite simple to calculate the progress (see example below). For uploading files, if you want to show the progress on the client, you'd need to somehow know how much of the stream has been read (by the server) at any given point. One way to do that is to create a new class, deriving from System.IO.Stream, which knows how to "count" how many bytes have been read; this class would know much has been read compared to the total size of the file.

 

public class Post1832432

{

    [MessageContract]

    public class MyFileInfo

    {

        [MessageHeader]

        public string FileName;

        [MessageHeader]

        public long FileSize;

        [MessageBodyMember]

        public Stream Stream;

        public MyFileInfo() { }

        public MyFileInfo(Stream stream, string fileName, long fileSize)

        {

            this.Stream = stream;

            this.FileSize = fileSize;

            this.FileName = fileName;

        }

    }

    [MessageContract]

    public class DownloadFileRequest

    {

        [MessageBodyMember]

        public readonly string FileName;

        public DownloadFileRequest() { }

        public DownloadFileRequest(string fileName)

        {

            this.FileName = fileName;

        }

    }

    [ServiceContract]

    public interface IFileManager

    {

        [OperationContract]

        MyFileInfo DownloadFile(DownloadFileRequest request);

        [OperationContract]

        void UploadFile(MyFileInfo fileInfo);

    }

    [ServiceBehavior(IncludeExceptionDetailInFaults = true)]

    public class MyService : IFileManager

    {

        public MyFileInfo DownloadFile(DownloadFileRequest request)

        {

            FileInfo fi = new FileInfo(request.FileName);

            MyFileInfo result = new MyFileInfo(File.OpenRead(request.FileName), request.FileName, fi.Length);

            return result;

        }

        public void UploadFile(MyFileInfo fileInfo)

        {

            FileStream fs = File.Create(fileInfo.FileName);

            byte[] buffer = new byte[10000];

            int bytesRead;

            do

            {

                bytesRead = fileInfo.Stream.Read(buffer, 0, buffer.Length);

                fs.Write(buffer, 0, bytesRead);

            } while (bytesRead > 0);

            fs.Close();

            fileInfo.Stream.Close();

        }

    }

    public static void Run()

    {

        string baseAddress = "http://localhost:8000/FileService";

        ServiceHost host = new ServiceHost(typeof(MyService), new Uri(baseAddress));

        host.AddServiceEndpoint(typeof(IFileManager), GetBinding(), "");

        host.Open();

        ChannelFactory<IFileManager> factory = new ChannelFactory<IFileManager>(GetBinding(), new EndpointAddress(baseAddress));

        IFileManager manager = factory.CreateChannel();

        string remoteFileName = "FileOnServer.bin";

        MyFileInfo fileInfo = manager.DownloadFile(new DownloadFileRequest(remoteFileName));

        FileStream fs = File.Create("Client_" + fileInfo.FileName);

        int bytesRead = 0;

        long totalBytesRead = 0;

        byte[] buffer = new byte[10000];

        do

        {

            bytesRead = fileInfo.Stream.Read(buffer, 0, buffer.Length);

            fs.Write(buffer, 0, bytesRead);

            totalBytesRead += bytesRead;

            double currentProgress = ((double)totalBytesRead) / fileInfo.FileSize;

            Console.WriteLine("Read so far {0:.00}% of the file", currentProgress * 100);

        } while (bytesRead > 0);

        ((IClientChannel)manager).Close();

        factory.Close();

        host.Close();

    }

    private static Binding GetBinding()

    {

        BasicHttpBinding binding = new BasicHttpBinding();

        binding.TransferMode = TransferMode.Streamed;

        binding.MaxReceivedMessageSize = int.MaxValue;

        return binding;

    }

}

 

Answer 4

One simple Stream class that can track progress:

public class MyCountingStream : Stream

{

    private FileStream file;

    private long bytesRead;

    public MyCountingStream(FileStream file)

    {

        this.file = file;

        this.bytesRead = 0;

    }

    public double GetProgress()

    {

        return ((double)bytesRead) / file.Length;

    }

    public override bool CanRead

    {

        get { return true; }

    }

    public override bool CanSeek

    {

        get { return false; }

    }

    public override bool CanWrite

    {

        get { return false; }

    }

    public override void Flush() { }

    public override long Length

    {

        get { throw new Exception("The method or operation is not implemented."); }

    }

    public override long Position

    {

        get { throw new Exception("The method or operation is not implemented."); }

        set { throw new Exception("The method or operation is not implemented."); }

    }

    public override int Read(byte[] buffer, int offset, int count)

    {

        int result = file.Read(buffer, offset, count);

        bytesRead += result;

        return result;

    }

    public override long Seek(long offset, SeekOrigin origin)

    {

        throw new Exception("The method or operation is not implemented.");

    }

    public override void SetLength(long value)

    {

        throw new Exception("The method or operation is not implemented.");

    }

    public override void Write(byte[] buffer, int offset, int count)

    {

        throw new Exception("The method or operation is not implemented.");

    }

}

 

Answer 5

This is great. This class was exactly what I was missing!

Thank both of you!

Answer 6

I'm back again! Everything works fine. I can know get the progress of both download and upload operations.

 

It seems though that DownloadFile method of server should also use some king of MyCountingStream class. This is to be notified when downloading is completed, in order to close the input stream on server side. Since the execution of this method on server side is completed before the actual download is done by the client, server method cannot close the stream. This results in the file being locked, even after the downloading is completed.

 

Is this correct?

 

I discovered this while debugging the whole idea. If I invoke DownloadFile to get a file from a specific location from server and then I try to upload that file to the same location on server, I get a 'File is being used by another process' exception.

Answer 7

 

if use like that your server file always locked..

Answer 8

If use like what?

Read my post above. I think we are talking about the same thing. But it is not locked in all cases.

Answer 9

One idea is to close the FileStream when the client is done reading it:

 

    public override int Read(byte[] buffer, int offset, int count)

    {

        int result = file.Read(buffer, offset, count);

        bytesRead += result;

        if (result == 0) { file.Close(); }

        return result;

    }

 

Or even better: override the Close method on the stream:

    public override void Close()

    {

        file.Close();

    }

 

This way when the client closes the stream and the message is terminated, the stream will be closed.

 

Answer 10

Thanks for your reply. I'm already doing the first suggestion but it does not help. It seems that closing the stream on client side does not release the server side file.

About the second I'm not sure what you mean. If you mean this should be done on client side, there is nothing to override there. When downloading I'm not using the suggested class from the previous posts, since client has the control of the downloading and progress can be read directly from the download stream. That class is used only when uploading, since then the server has the control of the process but the client needs to now then progress.

If you mean that this must be done on server, then we are talking about the same thing.

Answer 11

Here is the solution I have found. Just before the end of the Download function on server side, I start a thread using a method that accepts the stream as input (thread is started using parametrized thread start). After the thread is started the previous thread that accepted the call returns the response to the client and finishes. The method started on separate thread has an endless loop and queries the position of the stream every 3 seconds (a Thread.Sleep(3000) is used so that it does not allocate much processing time). If the position of the stream remains the same for more than 3 seconds the stream is closed.

Answer 12

 

.... method.......

mFileStream = new MemoryStream(ReadFully(fs, fs.Length));

fs.close;

return mFileStream ;

.............

 

 

private byte[] ReadFully(Stream stream, long initialLength)

{

if (initialLength < 1)

{

initialLength = 32768;

}

byte[] buffer = new byte[initialLength];

int read = 0;

int chunk;

while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0)

{

read += chunk;

if (read == buffer.Length)

{

int nextByte = stream.ReadByte();

if (nextByte == -1)

{

return buffer;

}

byte[] newBuffer = new byte[buffer.Length * 2];

Array.Copy(buffer, newBuffer, buffer.Length);

newBuffer[read] = (byte)nextByte;

buffer = newBuffer;

read++;

}

}

byte[] ret = new byte[read];

Array.Copy(buffer, ret, read);

return ret;

}

 

 

 

 

we are using this method its actually working...

 

Answer 13

You are not specifying what you are using where... but I suppose you are using the first piece of code on server and the rest on client. If this is the case it seems that you are reading completely the file in memory and then you are sending the stream to client. Isn't it so? If yes, it's sufficient for small files but what about large files? Isn't it time and resource consuming?

Answer 14

yes, and we are trying large file up 50+MB and so fast stream into memory,

and another issue

 

if same file trying download, other downlaod request must not waiting first download process done. we solve problem like that, may be good solution somewhere. but we must use this solution while find a other

Answer 15

Consider trying the solution I propose above (Yesterday, 8:36 PM).

In terms of scalability it's better not to load 50MB in memory. This is a lot of data. If 3 users request the same file your server will take up 150MB or RAM.

And if you open the file properly (see bellow) there is not locking problem while reading. Ofcourse if someone is trying to upload or delete the file someone else is downloading, the file will be locked.

System.IO.FileStream stream = new System.IO.FileStream(filePath, System.IO.FileMode.Open, System.IO.FileAccess.Read);

Answer 16

yes, you right when use FileAccess.Read file not locked for read and when service dispose not locked also. and i changed my codes. i was 50, 150 MB file transfer succesfully, but when i try 200+ service doesnt response anything, did you know anything this isssue. i am using IIS 7.0 Vista.

 

Answer 17

I'm back...

Read my complete article on this subject in Code Project. It appears that properly closing the stream on server side was easier than we thought!

http://www.codeproject.com/WCF/WCF_FileTransfer_Progress.asp

Answer 18

Hi guys,

I have to implement a webservice that transfers synchronizatin packages but the packages are all 40+ MB. I searched  the net for a suitable solution to download/upload huge files from/to WCF services. It seems the only way that people have used till now is the streaming method. This seems to me to be a very unstable solution. What if the upload or download process breaks just before the completion of the process? It doesn't seems to support any kind of resume. Also it doesn't seem to me that its a reliable solution to download a package of 50MB over the internet.

I encountered something called Buffered transport but i haven't found any sample about that yet. Any one has any ideas to help me out, please?

Answer 19

Hi Osama,

Support for resuming... mmm... very interesting! I'm gonna have to do this!