none
Keeping the network pipe full - uploading big files with WCF Net/TCP binding (binary encoding)

    Question

  • I just did a performance test comparing streaming to chunking. The test file is 293 MB. According to one of the online bandwidth test websites, my cable broadband upload speed should transfer that (cross-country, USA, which is where the server is) in around 12 minutes.

    Interestingly, both methods - streaming and chunking - took around 18 minutes. In my implementation, streaming reads 4K at a time and is totally synchronous. The chunking test uses 4 MB blocks (an arbitrary number; I haven't experimented with it yet) and is asynchronous on both sides (which presumably slows things down slightly).

    I only include the numbers as relative references, since obviously I haven't been very scientific. My question is this: theoretically, what's the better architecture for keeping the network pipe full throughout the entire upload?

    My suspicion is that both of my current approaches are suffering from periods in which the pipe (i.e. the network segment between the client and server) is empty. For example, the streaming server endpoint reads 4K, but while it's copying that to the local file, the pipe is empty, correct? In the chunking case, the client doesn't send chunk n+1 until it receives acknowledgement that chunk n was received by the server.

    Presumably, the optimum solution is one in which the client stuffs data into the pipe as fast as it can (which is to say as fast as TCP will allow it to) and doesn't wait for any server acknowledgement (again, relying on TCP to do its job). Offhand, I think what that would look like is an interface that allows the client to setup some context with the server, make a bunch of "one-way" calls to upload the data, and then maybe a call at the end to ask the server to hash the file just in case.

    Is that the right approach if optimum bandwidth usage is the goal?

    Friday, September 04, 2009 2:43 AM

All replies

  • Hi Dan,

    Based on your post, your concern is the network efficiency: whether the all data is transferred in short time so as to improve network throughout. Buffered transfers hold the entire message in a memory buffer until the transfer is complete. Streamed transfers only buffer the headers and expose the message body as a stream, from which smaller portions can be read at a time. Streamed transfers can improve the scalability of a service by eliminating the need for large memory buffers. Whether changing the transfer mode will improve scalability depends on the size of the messages being transferred. Large message sizes favor using streamed transfers.

    In Buffered mode if we transfer the large data, this will consume much memory in Service process. If we set InstanceContextMode to PerCall and there are some clients which access the WCF service simutaneously, this will heavily affect the service performance. I think that Streamed Mode is one best method here than Buffered mode. In this scenario I think that you can increase the MaxBufferSize value to improve the performance for Streamded Mode WCF service.

    Please refer to the following articles:
    http://nirajrules.wordpress.com/2009/08/03/mtom-vs-streaming-vs-compression-%E2%80%93-large-attachments-over-wcf/
    http://www.haveyougotwoods.com/archive/2008/04/14/wcf-message-streaming.aspx

    Or you can also have a look at WCF REST programming model to upload data to WCF service in this scenario:
    http://msdn.microsoft.com/en-us/library/cc656724.aspx
    Best regards,
    Riquel
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Monday, September 07, 2009 6:06 AM
    Moderator
  • Yes, I understand that much. My question is how to keep the network pipe full throughout the upload.

    Again, my understanding of Streaming is that once the service receives the amount of data requested, transmission will pause until another request is issued. During that time, there's no data in transit, which from a utilization perspective is 'wasteful'. Does that make sense?

    An alternative streaming architecture would be one in which the service requests the whole file up front, and the runtime notifies the server chunk by chunk as the data arrives up the network stack. In theory, that would perform well as long as the service can keep up with the network. However, the runtime doesn't seem to work that way. To confirm, is WCF pre-fetching data from the stream before the server asks for it?

    Otherwise, I'm wondering if a buffered-mode chunking approach, driven by the client, as proposed above, would be the closest approximation of that.

    Monday, September 07, 2009 3:39 PM
  • Hi Dan,

    I think that you need to increase Maxbuffersize for Stream mode or increase chunk size for chunking to reduce communication overhead to improve network efficiency. You also need to consider how many request your service will handle. If the service is required to support many clients's request, we need to decrease the above value so as to reduce memory consumption for service performance. I don't think that we can only consider network throughput.


    Best regards,
    Riquel
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Tuesday, September 08, 2009 9:14 AM
    Moderator
  • Hi Dan,
       your question is prety good.I d like to share my points with you guys,Hope it can help you.
    As we know ,WCF has two transfer mode:buffer and Streaming.

    1.Streaming mode,I do not think that channel is empty,when Server try to read data.
       in your case,if you set the read size is 4kb everytime,WCF  Service instance maybe apply a memory block with 8kb or more .
       for example:memory blocks:|-----(1)--4k------|-----(2)---4k-------|.when WCF start receiving data ,it stores the data at 2nd block,when the 2nd block is full,WCF will invoke a nother thead to read the data and move it to other place.at the same time ,WCF service keeps receiving data from the channel,and put the data at 1st block,When the 1st block is full,Another thread will start to move the data,WCF also can receive data and put them in 2nd block.We asume that, the receive and move threads has the same work efficency.
       So ,WCF can reuse the memory blocks.It does not to wait all data comes here,then to move them together.
      It is WCF Steaming mode works process.
      Actually,WCF maybe apply lots of blocks one time.It is can  advoid memory read and write locks.
      So ,you are correct,WCF will pre-fetch data from the stream memory blocks.
    2.No matter TcpBinding or HttpBinding ,they all use TCP/IP to transfer data.as you know,TCP/IP is a two-layer program. The higher layer, Transmission Control Protocol, manages the assembling of a message or file into smaller packets that are transmitted over the Internet and received by a TCP layer that reassembles the packets into the original message. The lower layer, Internet Protocol, handles the address part of each packet so that it gets to the right destination. Each gateway computer on the network checks this address to see where to forward the message.
       Http also use TCP/IP,HTTP is an application protocol that runs on top of the TCP/IP suite of protocols (the foundation protocols for the Internet).
     Http:  http://searchwindevelopment.techtarget.com/sDefinition/0,,sid8_gci214004,00.html
    TCP:http://searchnetworking.techtarget.com/sDefinition/0,,sid7_gci214173,00.html
    3.It is difficult to control the channel work state,
       for WCF also needs to create channel for diffierent bindings.
       No matter what binding we use,WCF will transport data though the channel over tcp/ip.
       it will try to  transport all data and keep  data validation.
       WCF does not expose any API like this to control the underlying data transpotation,like ,when channle is not busy,let it start work.
      
    4.I also posted a thread about WCF thread model:Does WCF use or Supports IOCP?
       you know,WCF use IOCP model for  you can read this book:Network Programming for Microsoft Windows, Second Edition.
    It introduces all Windows IO model:blocking, select, WSAAsyncSelect, WSAEventSelect, overlapped I/O, and completion port.I found some useful words for this question on chapter 8.
       Here is the thread:
    1.Does WCF use or Supports IOCP?:http://social.msdn.microsoft.com/Forums/en-US/wcf/thread/3f45500d-6de0-4c90-8a62-f08e1ede053e/
    2.Chinese version:
    http://www.cnblogs.com/frank_xl/archive/2009/07/17/1525191.html
       
       if you know these models,you can know your question better.
       WCF uses IOCP model.
     
       Hopes these words can help you guys.

    Regards
    Frank Xu Lei--谦卑若愚,好学若饥
    专注于.NET平台下分布式应用系统开发和企业应用系统集成
    Focus on Distributed Applications Development and EAI based on .NET
    欢迎访问老徐的中文技术博客:Welcome to My Chinese Technical Blog
    欢迎访问微软WCF中文技术论坛:Welcome to Microsoft Chinese WCF Forum
    欢迎访问微软WCF英文技术论坛:Welcome to Microsoft English WCF Forum
    Tuesday, September 08, 2009 2:08 PM
  • Frank - thanks - that is indeed helpful. Regarding your point #1, how did you confirm that WCF is pre-fecthing - did you use NetMon?
    Tuesday, September 08, 2009 2:19 PM
  • Hi   Dan,
         Actually not,I learned it when I try to find the 4th question about IOCP.WCF works with managed codes,You also can learn more about Memory allocation and management of .NET from book:Applied Microsoft .NET framework Programming.
        this book:
        Network Programming for Microsoft Windows, Second Edition.
        if really a good book  about network programming,I learned much from it.you can read it when you are free.

    Regards.
    Frank Xu Lei--谦卑若愚,好学若饥
    专注于.NET平台下分布式应用系统开发和企业应用系统集成
    Focus on Distributed Applications Development and EAI based on .NET
    欢迎访问老徐的中文技术博客:Welcome to My Chinese Technical Blog
    欢迎访问微软WCF中文技术论坛:Welcome to Microsoft Chinese WCF Forum
    欢迎访问微软WCF英文技术论坛:Welcome to Microsoft English WCF Forum
    Tuesday, September 08, 2009 2:38 PM
  • Can any of the Microsoft folks confirm whether WCF streaming does pre-fetching?
    Tuesday, September 08, 2009 2:53 PM