locked
Sending Large Data from WCF Service and OutOfMemoryException RRS feed

  • Question

  • Hello,

    I am trying to return a dataset which has around 150,000 rows (around 60MB data and still can't say huge data as I thought) of datatable from WCF Service to Client and it gives me an OutOfMemoryException. I do binary serialize for dataset and still it doesn't make any senses. maxRequestLength is set to around 100MB and maxItemsInObjectGraph is set to 10485760 and any other length are set to 2147483647. I am using BasicHttpBinding with Buffered. I was googling about returning large data and it was all said to use Streamed. Without using Streamed, coz I don't want to change my design, is there any workaround for this?



    Nilar

    Wednesday, October 23, 2013 2:19 PM

All replies

  • I think you would have to set TransferMode to Streamed to send the data. Don't think there is any workaround.

    http://msdn.microsoft.com/en-us/library/ms789010.aspx


    Gaurav Khanna | Microsoft VB.NET MVP | Microsoft Community Contributor

    Wednesday, October 23, 2013 7:05 PM
  • Thanks,

    "Don't think there is any workaround." means there is no workaround??


    Nilar

    Thursday, October 24, 2013 3:43 AM
  • Hi,

    Yes, it seems that you have to use the stream. The following information is mentioned in this MSDN document, please try to check it:

    Windows Communication Foundation (WCF) transports support two modes for transferring messages:
    • Buffered transfers hold the entire message in a memory buffer until the transfer is complete. A buffered message must be completely delivered before a receiver can read it.
    • Streamed transfers expose the message as a stream. The receiver starts processing the message before it is completely delivered.
    • Streamed transfers can improve the scalability of a service by eliminating the requirement for large memory buffers. Whether changing the transfer mode improves scalability depends on the size of the messages being transferred. Large message sizes favor using streamed transfers.

    So if we use the buffered mode with large data, maybe we will meet the OutOfMemoryException.

    Best Regards,
    Amy Peng


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, October 24, 2013 6:44 AM
  • Hi Thanks again,

    What about Chunking channel? I found that topic when i tried to find out about streaming. I am still getting no idea about Chunking channel but it sounds good as i read here. But what should I use? Streaming or Chunking.

    Any suggestion would be appreciated.

    Thanks/


    Nilar

    Sunday, October 27, 2013 11:04 AM
  • Hi,

    When sending large messages using Windows Communication Foundation (WCF), it is often desirable to limit the amount of memory used to buffer those messages. One possible solution is to stream the message body (assuming the bulk of the data is in the body). However some protocols require buffering of the entire message. Reliable messaging and security are two such examples. Another possible solution is to divide up the large message into smaller messages called chunks, send those chunks one chunk at a time, and reconstitute the large message on the receiving side. The application itself could do this chunking and de-chunking or it could use a custom channel to do it.

    One of the reasons customers use Chunking channel to transfer large data is when they want to use Reliable Messaging (WS-RM) since one needs to buffer data to use this protocol. In short, this protocol provides reliability at the SOAP layer (something which is provided by TCP at the transport layer).

    Streaming (BasicHttp, Tcp, Pipe) can be a good solution when you don’t want to increase the load on your servers though unlike buffering this doesn’t allow you to leverage on WCF’s message based security & reliability. When you want to use streaming though, your OperationContract should use only one instance of Stream class (details here) in parameter list or as return type.

    Best Regards,
    Amy Peng


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, October 28, 2013 7:05 AM
  • Hi Amy, Thank you very much for your great supporting.

    The reason why I don't want to choose streaming is "reliability". It can be broken anytime during the data transferring. But Chunking is also quite tough for me since this is the first time investigating and the sample program is also difficult to understand(even the language, I am doing vb) for me, but I am trying to understand what they want to say. I want to get some help from though who implemented successfully with chunking.
    So I think if you guys support me to success with chunking at least providing me with some idea or some coding would be a great help. If I couldn't manage with Chunking, I will consider for streaming.

    Thanks/

    Nilar


    • Edited by Nilar Ko Monday, October 28, 2013 8:22 AM
    Monday, October 28, 2013 8:22 AM
  • Well, I am stuck again.

    The chunking channel sample provided by here is an IDuplexSessionChannel  base. What I actually needed is Request/Response design that I am using BasicHttpBinding currently.

    Is it possible to create custom chunking channel with that Request/Response pattern??

    Or Is there any other solution for sending large data and OutOfMemoryException? 

    I don't really want to use streaming since i do care for a data reliability.

    Any suggestion would be appreciated.

    Thanks.


    Nilar

    Monday, November 4, 2013 11:58 AM
  • Is it possible to create custom chunking channel with that Request/Response pattern??

    Hi,

    This article will be what you want:
    Chunks using WCF:http://www.codeproject.com/Articles/102656/Downloading-in-Chunks-using-WCF-and-Silverlight .

    Wednesday, November 6, 2013 3:10 AM
  • Thank you Diana, 

    I will check your link. 

    Aside from that, now I am concerning about Message Structure of Chunking Channel Sample. 

    Is Chunking only doing for Stream type??

    What about DataContract or any other object types?

    What my requirement is something like that. 

    <OperationContract()> _
    Function DoWork(ByRef p1 As Param1, ByVal p2 As Param2, ByRef p3 As Param3) As Boolean

    Those Param1, 2, 3 are DataContract types. 

    As far as i know XMLDictionaryWriter and Reader of chunking channel doesn't implement those types. Is there anything that i can customize in ChunkingSample program according to my requirement. For example Serialize my object to XML? Is it possible or not?

    Any suggestions would be appreciated. 

    Thanks


    Nilar

    Wednesday, November 6, 2013 4:23 AM