none
Passing 3MB DataSet via WCF - Best Practice RRS feed

  • Question

  • Hi

    In our application we have several DataSets with about 2-4MB of size.
    Each DataSet contains of only one single DataTable.

    The data has to come from the server to a client via WCF. Client and server are in the same LAN. What are best practices to pass the data?

    The only thing I think I can be sure about is that I should use binary encoding for the binding.

    The following questions are still not clear for me:

    1. Which binding should I use - netTCP or WSHttp?
    2. I know that a DataSet produces overhead, but does this really matter in my case?
    3. If the DataSet produces to much overhead, what should I do best to get the data to the client fast?
    4. Should I use streaming or better the standard buffering?
    5. Which timeout and max size settings do I have to take care of (MaxReceiveSize, SendTimeout and so on)?
    6. What do I have forgotten to pass the data quickly?
    Thanks for your thoughts about this subject.

    Additionally to that: What else do I have to mention if my DataSet becomes about 10MB of size?

    Regards
    Bastian
    Thursday, October 19, 2006 5:21 PM

Answers

All replies

  •  

    Interesting, i was working on a similar DataExchange  project and we needed  a decoupled solution so i choose NetMsmqBinding. You could use NetTcpBinding with binary or MTOM for better performance.

    For large DataSet, i recommend you chuck records and  sent records in batches as Datapackets. Here is how i have the DataPacket Contract. I have a configurable  RowsetSize that will allow me to control the no.of records send. The GroupID and the batchID are just used for tracking the flow of data.

    [DataContract]

    public class DataPacket

    {

    Guid m_groupID = Guid.Empty;

    int m_batchID = 0;

    DataTable m_data = null;

     

    [DataMember]

    public Guid GroupID

    {

    get { return m_groupID; }

    set {

    m_groupID = value;

    }

    }

    [DataMember]

    public int BatchID

    {

    get { return m_batchID; }

    set { m_batchID = value; }

    }

    [DataMember]

    public DataTable Data

    {

    get { return m_data; }

    set { m_data = value; }

    }

    }

     

     

    Thursday, October 19, 2006 7:40 PM
  • Sounds interesting.
    What do I have to do on the client to use the packets of data that have arrived yet, while further packet are coming over the wire?

    Can I do it like this:
    1. Create a empty DataTable on the client
    2. Bind DataTable on a DataGrid
    3. Add DataRows from the first packet that arrived (Rows will be visible in the DataGrid at once)
    4. Call the service again untill all packets have been arrived
    5. On arriving of a packet add new DataRows to the client DataTable
    Can I simulate a filling of the DataGrid in the background?

    Or the question I wanted to ask simply is: I understand how you get the data in packets over the wire, but how do you work with the data on the client while you are still receiving more packets?

    Regards
    Bastian
    Friday, October 20, 2006 7:25 AM
  • Today I learned the following:

    Passing database data via DataSet and WCF takes approximately twice as long as converting data to a string array, sending it via WCF and converting it back to a DataSet in the client.

    Wow I didn't know how much overhead a DataSet produces, but the Stopwatch doesn't lie...

    The WsHttp is about 20% slower than NetTcp Binding.

    These are the facts from my test and development environment, so I don't know if the results will be the same in your cases, but I thikn they will be similar...

    Regards
    Bastian
    Friday, October 20, 2006 2:24 PM
  • (1)

    please check this blog,binary gives better performance,if you want to use wshttp,please use MTOM

    http://blogs.msdn.com/yassers/archive/2006/01/21/515887.aspx

    (2)

    About dataset,somebody suggested this trick before,i didn't try it yet,This may work

    You can set the following properties:

    ds.SchemaSerializationMode = SchemaSerializationMode.ExcludeSchema;
    ds.RemotingFormat = SerializationFormat.Binary;

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=462032&SiteID=1

     

    Friday, October 20, 2006 2:44 PM
  • I heard about that


    ds.SchemaSerializationMode = SchemaSerializationMode.ExcludeSchema;
    ds.RemotingFormat = SerializationFormat.Binary;

    But unfortunately it does not work. WCF throws an exception and say something about that it is not allowed to overwrite the remoting format - sorry I can't remember the exception message at the moment.

    Bastian
    Friday, October 20, 2006 2:54 PM
  • Bastian,

    Are you using any of the features of Dataset in the server side?

    How often is this service called?

    How many connections simultaneous connections do you plan to support in your service?

    Thanks

    Thursday, March 1, 2007 6:35 AM
    Moderator
  • I checked the SchemaSerializationMode and RemotingFormat DataSet options with WCF. Both work properly though the second one has no effect on WCF perfoprmance as the RemotingFormat is considered in .NET Remoting only. When I exclude schema from the DatSet being sent the WCF works a bit better but in any case the .NET Remoting is faster when sending DataSet.

     

    My performance comparison can be found here: WCF vs. Remoting (with DataSet)- performance comparison

     

    Marcin Celej

    Thursday, April 12, 2007 9:25 PM
  • The service is called very often and I think with about 2-5 simultaneous connections.

    My solution is to convert the DataSet into a string (just the needed content) --> transfer it --> convert it back to DataSet after transfer.

    This solution works about 10 times faster than the standard way.
    Friday, April 20, 2007 9:00 AM
  • Dear all,

     

    I have a similar case for my project of building an alarming system.

    I have a component which collect and send event data that I need to show and archives in database.

    The way I have done it is as follow :

     

     - I have an initial component that collects alarms at a frequency of 500ms pooling ( It does not means that I willhave alarms each time but could happen that I receive a lot).

     

    - Then I have build  windows service which host a remote component which is charge to buffer incoming alarms to database.I have using tcp chanel for that and binary formatter.

     

    - Then I have by business logic which is informed when new data has been inserted to databse, to collect them and display to users. AS soon as my read alarms gets read , it is removed from the buffer table. In that solution I have all the time to display my alarms has they are buffered by my service with time stamp. When ready for display I how then in the order of appearence.

     

    I have here two differente process: one which is bufferring, and an other which is reading

     

    regards

    serge

    Saturday, May 5, 2007 2:50 PM
  • Honestly, I have dropped WCF serialization/deserialization altogether, it has way too much overhead for me. I use something similar to this AltSerializer solution posted on CodeProject:

     

    http://www.codeproject.com/cs/library/AltSerializer.asp

     

     

    The performance is dramatically better and it allows me to use one interface to pass any serializable object. Benchmark the AltSerializer: its very fast!

     

    Trevor

     

    • Proposed as answer by Ranjancse Tuesday, March 17, 2009 6:30 AM
    Monday, May 7, 2007 7:29 AM
  • interesting design choice to send 3mb of data in a single call. does the user need to see all 3mb of this data all at once? Why a dataset instead of strongly typed collection of objects?

     

    I'm no expert of course, i just wonder if there isn't a better way to deliver data to your users than sending them a big 3meg chunk all at once. Surely they aren't looking at all 3 megs at once.

    Monday, May 7, 2007 5:39 PM
  • Hi,
        Just wanted to know two things.

    1. How do I configure the size limit of Objects to be sent using WCF.
    2. I am not able to serialize Datatable using WCF.

    Regards,
        Jasaz

    • Proposed as answer by Ranjancse Tuesday, March 17, 2009 6:30 AM
    Sunday, December 7, 2008 5:32 AM
  • 1) Answer for your first query is You can configure in say App.config or Web.config of WCF Client application
    Just you have to modify the  maxReceivedMessageSize="2147483647"

    Regards,
    Ranjan.D

    Tuesday, March 17, 2009 6:37 AM
  • Hi there this the solution:

     

     

    ' Declare a datatable and dataset

     

    Dim table As DataTable = New DataTable

     

    Dim DS As DataSet = New DataSet

     

    ' Set the Datatble

    table.Columns.Add(

    "Code", GetType(String))

    table.Columns.Add(

    "Name", GetType(String))

    table.Columns.Add(

    "Address", GetType(String))

     

     

    ' Fill the Datatable

     

    Dim Row As DataRow

    Row = table.Rows.Add

    Row.Item(

    "Code") = "0000"

    Row.Item(

    "Name") = "Peter Jhons"

    Row.Item(

    "Address") = "NY"

     

     

    ' Second record

    Row = table.Rows.Add

    Row.Item(

    "Code") = "0001"

    Row.Item(

    "Name") = "Charlie Brown"

    Row.Item(

    "Address") = "Manhatan"

     

     

    ' Add the table to Datatable

    DS .Tables.Add(table)

     

    ' Set the remotingFormat

    DS .RemotingFormat = SerializationFormat.Binary

     

    ' Return the Table from the DatatSet

     

    Return DS .Tables(0)

    Thursday, October 7, 2010 5:35 PM
  • I am also having the same problem I need to load about 45,000 rows of records using wcf service Please any Idea the service keeps throwing exception once the data table rows is higher than 4000. I need to stream the wcf service
    • Edited by Ig Sam Tuesday, October 11, 2011 3:49 PM
    Tuesday, October 11, 2011 3:48 PM
  • I assume it is a web page/windows app that displays the data. Do you think showing 45000 rows to user is going to help? I'd suggest pagination. If its not for a human interface then write a bacth component which does heavy lifting for you either using binary stream or calling the paginated rows sequentially and aggregate it.
    Wednesday, October 12, 2011 6:57 AM