TCP File Transfer in WCF

Question

Hi, I made a Windows Communication Foundation Application that sends

files via TCP from a client to a Service host. These files are converted in an

Array of bytes (byte[] file) and the client calls the service sending the byte

Array as a parameter. I configured the App.config file to support large size

messages (maxBufferSize="2147483647").

When I send a 150MB file the App runs OK

THE PROBLEM IS...

when I send a larger file (say 300MB) it gives me an error:

"Failed to allocate a managed memory buffer of 536870912 bytes. The amount of available memory may be low."

I don't know how to configure that buffer... it looks like that is the

Application buffer

Any ideas what could be???

PS: I have 1,5G of RAM so I think RAM is not a problem

Answer 1

Make sure you also set the reader quotas to high enough values.  Some common settings are demonstrated below

<netTcpBinding>

<binding name="tcpNone" >

<readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647"   />

</binding>

 

For file transfer, streaming is  the way to go.  You avoid large memory alloctions, and therefore limits on file size, and your recieving code can stream straight to a file.   See http://msdn2.microsoft.com/en-us/library/ms731913.aspx

 

Answer 2

Thanx Martin, I was doing that already...

One thing that I notice and maybe could help to figure this out is when I send a large file by wsHttpBinding

it gives me the same error but the managed memory buffer is 268435456 bytes (half of the tcpBinding)...

Answer 3

The binary encoder used for the TCP binding is more space efficient than the text encoder used for the Http binding.  However binary encoder  doesn't do compression.  It could be that the shape of your data is amenable to being made smaller by the binary encoder.

Answer 4

Also, did you try streaming?   Hugh buffered transfers (such as 150 MB) aren't recommended.   They are generally quite memory intensive.   I'd strongly recommend streaming for this scenario.

Answer 5

How can I do this? Just putting  transferMode="Streamed" in the binding??

I did that and it gave me this error :

System.OutOfMemoryException

Answer 6

Yes you just set the transferMode attribute.

What are you doing with the data in your service?  Do you really need to buffer the contents of the whole file in memory?  If so, Buffered messages are reasonable and more convenient.  If that's the case, your app design sounds pretty memory intensive and it may be suspectible to OOMs.   If  your file is text, remember that each character is stored as unicode in memory and  consumes 2 bytes.   A 300MB file may actually consume 600MB RAM. 

If your service writes the data to a file or can work with it in smaller chunks, streaming is the way to go.  it's much  more memory efficient.   If you are going to stream and store large amounts of data in memory, consider allocating the required amount of memory up front in a byte array for example or a StringBuilder.   Using a collection that keeps allocating larger  and larger chunks of memory as it needs it would be memory intensive.

Answer 7

Another thought.  If you are going to send a 300 or 600 MB buffered  message, WCF must allocate memory as it handles the message so there's more memory overhead beyond the message size itself.

Answer 8

I'm trying to send the file by stream...

I don't know if I'm doing this right but instead of passing a byte Array, I'm sending the filestream:

proxy.StoreFile(fs, fileName); //(fs is the FileStream and fileName a string)

Now I'm getting this kind of error:

Type 'System.IO.FileStream' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute.

my Service Class if this:

[ServiceBehavior()]

public class FileStorageServiceClass : IFileStorageService

{

#region IFileStorageService Members

public void StoreFile(FileStream file, string fileName)

{

     Console.WriteLine("StoreFile Service Ready");

     Console.WriteLine(fileName);

     Console.WriteLine(Convert.ToString(file.Length));

     CopyFileStream(file);

}

#endregion

private void CopyFileStream(FileStream file)

{

     FileStream fout;

     int i;

     fout = new FileStream(@"c:\teste2\" + file.Name, FileMode.Create);

     do

     {

          i = file.ReadByte();

          if (i!=-1)

          fout.WriteByte((byte)i);

     }while(i != -1);

     }

}

Answer 9

Check out http://msdn2.microsoft.com/en-us/library/ms789010.aspx which I think will answer your questions.   The message body may only have a single stream as a parameter.  If you wish to send a file name with the call as well, you can use a message contract and send the file name parameter as a header.

Answer 10

OK Martin I'll see this tomorrow at work,

thanx

Answer 11

Wshttp binding 1 byte = 13 bytes ... dont do it.  If you want to send a byte[] over http convert it to base64 string

Regards

 

Ben

Answer 12

Thanx for the tip Benk, I'm sending files by Stream, it looks good now...

What I'm trying to do now is place a Progressbar or something in the Client side to show the state of the action...

the reason is simple: if I send a larger file the application will be doing the streaming and the client sees nothing about

the progress of the action, and he could think that the App is somehow blocked...

Can you give me some advices in how to do this?

thanx

 

Answer 13

If you use a streaming transport mode, you can create a memory stream over your byte buffer and pass that for the stream parameter to your remoted method.   You don't have to do base64 encoding. 

To do a progress bar:

- use a streaming transport mode

- pass a stream to the method call. include the stream size in a header parameter

-  read the stream in increments in your service and update your progress bar based on bytes read vs total bytes in the stream.

Answer 14

Thanx Martin...

Well my ProgressBar is supposed to be in the client Form...

How does the Service inform the client that the ProgressBar must be incremented?

regards

Answer 15

I've done something similar, but I used chunking to send large files.  On the client side open a stream to your data, and then fill a byte array and send that byte array to the service.  Keep appending the file on the service as it receives more data.   From what I've read, streaming is not at all recommended for reliable transfer of binary data, chunking is the way to go.   You can easily be updating your progress on the client as the data is transfered by monitoring the offset of where you are at in the stream.

http://blogs.msdn.com/yassers/archive/2006/01/21/515887.aspx

Cheers

Answer 16

Thanx for the advice zuzinet,

I was simply passing a Stream as a parameter to the Service Operation:

Stream fs = new FileStream(FileNamePath, FileMode.Open);

...

proxy.StoreFile(streamMesg);

The streamMesg is a MessageContract with the Stream in it.

I didn't use any array of bytes because I was having out of memory problems the last time I used them

So I had no idea how I could catch the operation progress in the client side...

Maybe Chunking would do the work, but can you tell me how it works??

Cheers

Answer 17

Thats odd that you had out of memory problems, as if you break it up into reasonably sized chunks you shouldn't have any issues at all.  I used 65K for my chunk size, and it works great.    Ensure you have set your service throttling parameters and any other configuration params properly for larger message sizes.  The example below is stripped down, but you'll get the idea.   For progress in the client just throw a callback or something in the stream loop.

// contract

[OperationContract]
void AppendFile(string fileName, byte[] data, int offset, int count);

// service

public void AppendFile(string fileName, byte[] data, int offset, int count)
{
     using(FileStream f = new FileStream(fileName,FileMode.Append,FileAccess.Write))
     {
           f.Write(data,offset,count);
     }
}

// CLIENT

using(FileStream fs = new FileStream("c:\\myfile.zip",FileMode.Open))
{

byte[] buffer = new byte[ 65000 ];
int bytesRead = 0;

// get a proxy to our service
ChannelFactory<IWCFPerfTest> factory = new ChannelFactory<IWCFPerfTest>("MyEndpoint");
IWCFPerfTest proxy = factory.CreateChannel();

// iterate through the file filling a buffer and sending it to the service
while ( ( bytesRead = fs.Read( buffer, 0, 65000 ) ) > 0 )
{
     proxy.AppendFile("c:\\myfile.zip",buffer,0,bytesRead);
}

// close our connection
factory.Close();
}

 

Answer 18

Below are some configuration params that will need to be adjusted to allow you to transfer larger messages.

For the binding:

<binding name="perfTCPBinding" maxConnections="500" maxBufferSize="104857600" maxReceivedMessageSize="104857600">

<readerQuotas maxDepth="104857600"

maxStringContentLength="104857600"

maxArrayLength="104857600"

maxBytesPerRead="104857600"

maxNameTableCharCount="104857600" />

For the service behaviour (these numbers are quite high, set them to whatever you think your service can handle)

<serviceThrottling

maxConcurrentCalls="500"

maxConcurrentSessions="500"

maxConcurrentInstances="500"

/>

 

Answer 19

Thanx zuzinet, I'll try this now...

I'll give you some news about this later

Regards

Answer 20

That worked perfectly zuzinet...

I finnaly matched to make a working progressbar!!!

Although the time spent sending medium files (300 MB) is larger than sending them via FIleStream, when you start

sending larger files(+ 1GB) you don't notice the difference... In a matter of fact... they are lower...

I think I will go with the chunking system though

Thanx for the time spent with this

Regards

Answer 21

As Martin mentioned ,  you will be reading or writing it in increments - it would be good if you know the file size before hand maybe a seperate method then you can show an accurate %.

 

but basically from stream read...

 

      while (numBytesToRead > 0 )
        {
            // Read may return anything from 0 to numBytesToRead.
            int n = s.Read(bytes, numBytesRead, numBytesToRead);
            // The end of the file is reached.
            if (n==0)
                break;

         

         //TODO update UI  that numBytesRead

            numBytesRead += n;
            numBytesToRead -= n;
        }
        s.Close();
Ben

 

Answer 22

Thanx for the information BenK,  knowing the file size give us many ways for managing the stream progress...

 

my code is not exactly similar to yours but they both do the trick

Answer 23

Excellent question - I've just run into the same problem!!!
Since noone in this thread knows the aswer to it (noone even knows what's causing the problem :):) ), I can only conclude that it is a bug in WCF :)
I have a 32 GB of RAM available on my system, configured the TCP binding correctly (all values set to max size of int: 2^31-1, in both client and service configs), and still getting this exception when I try to trasfer 0.5 GB of data.
Back-doors like streaming is not an option in my case, and I'm not trying to trasfer a large file but a very large byte array.
Does anyone know at least what's the problem and why is this happening?

Answer 24

I think that it is not necessarily a bug in WCF. (It is more our fault for not accepting the defaults.)

Threads like this one exacerbate the issue, imho.

For example (again, today)...why run the maxDepth to the same value as the maxArrayLength? (maxDepth is in levels, not bytes.) By the same token, how can you expect more than one client if your maxBufferPoolSize is the same as your maxBufferSize?

These are reasons to have a communications professional write network code, and not necessarily a reason to invent an API that allows everyone to do so. (I'm not complaining, I like WCF alot.)

In a thread like this one, the OP gets their code running (sometimes), and that's that...on to the next disaster. In your case, you probably want to examine the messages before they go out. You can actually set breakpoints to see how big the array is, I think. Also, Fiddler2 is a great way to see what is going on between the service and the client. (I had looked at the size of the outgoing stream, but that was before it was encoded.)

I moved my service's config into the lightoff code of the service...it gets configured the way I want it, and then I can generate client code and config from the running service. (However, the client config is not going to be correct, if I have manipulated the values in question from the defaults.) It is much simpler and more productive for me this way. I am doing the same thing on my client side, so that I can also inspect the messages myself.

At any rate....could you please just make a new thread with your issue and both config files? (I promise not to tell you to run all the values up that high...I'll work with what you have and want...it helps me, too.)

Answer 25

Anyway,
I've managed to figure out 1/2 of this puzzle, and here's what I've managed to figure out so far:

1) The problem has nothing to do with the config file; believe me - you'll get an error/exception if the value exceeds the boundaries, so this is not an issue !!! BTW, I DID started with the default configuration, we all did (if you read more carefully ...)
2) The problem has nothing to do with the RAM memory; I have a LOT of RAM available ...
3) It acts the same both with the Debug and Release builds.
4) While trying to send a message, there are (at least) 3 magic numbers that appear in the exceptions: the mentioned 512 MB (536870912), 256 MB, and 128 MB.

Here's the deal:
a) If you try to send a message with size <= 128 MB - it will work!!! (magic #1)

b) If you try to send a message with the size between 128 MB (magic #2) and 512 MB (magic #3), you'll get the message:
"Failed to allocate a managed memory buffer of 268435456 bytes. The amount of available memory may be low."
Now this is funny becuse it will come up even if you try to send a message with the size less then 256 MB! I didn't require 256MB, I'm sending 253 MB :)

c) The error message celobateira got happens when you try to send a message larger then 512 MB.

Finally ....
5) What's causing the problem is ... believe it or not ... the Visual Studio! If you run the same executables from the command line - it will work just fine !!!

As I've mentioned, this is just a 1/2 of the puzzle - the aswer to the WHAT question. Now if someone knows WHY it happens and how to configure my VS not to cause exceptions, it would be great!