Passing 3MB DataSet via WCF - Best Practice

Question

Hi

In our application we have several DataSets with about 2-4MB of size.
Each DataSet contains of only one single DataTable.

The data has to come from the server to a client via WCF. Client and server are in the same LAN. What are best practices to pass the data?

The only thing I think I can be sure about is that I should use binary encoding for the binding.

The following questions are still not clear for me:

1. Which binding should I use - netTCP or WSHttp?
2. I know that a DataSet produces overhead, but does this really matter in my case?
3. If the DataSet produces to much overhead, what should I do best to get the data to the client fast?
4. Should I use streaming or better the standard buffering?
5. Which timeout and max size settings do I have to take care of (MaxReceiveSize, SendTimeout and so on)?
6. What do I have forgotten to pass the data quickly?

Thanks for your thoughts about this subject.

Additionally to that: What else do I have to mention if my DataSet becomes about 10MB of size?

Regards
Bastian

Answer 1

 

Interesting, i was working on a similar DataExchange  project and we needed  a decoupled solution so i choose NetMsmqBinding. You could use NetTcpBinding with binary or MTOM for better performance.

For large DataSet, i recommend you chuck records and  sent records in batches as Datapackets. Here is how i have the DataPacket Contract. I have a configurable  RowsetSize that will allow me to control the no.of records send. The GroupID and the batchID are just used for tracking the flow of data.

[DataContract]

public class DataPacket

{

Guid m_groupID = Guid.Empty;

int m_batchID = 0;

DataTable m_data = null;

 

[DataMember]

public Guid GroupID

{

get { return m_groupID; }

set {

m_groupID = value;

}

}

[DataMember]

public int BatchID

{

get { return m_batchID; }

set { m_batchID = value; }

}

[DataMember]

public DataTable Data

{

get { return m_data; }

set { m_data = value; }

}

}

 

 

Answer 2

Sounds interesting.
What do I have to do on the client to use the packets of data that have arrived yet, while further packet are coming over the wire?

Can I do it like this:

1. Create a empty DataTable on the client
2. Bind DataTable on a DataGrid
3. Add DataRows from the first packet that arrived (Rows will be visible in the DataGrid at once)
4. Call the service again untill all packets have been arrived
5. On arriving of a packet add new DataRows to the client DataTable

Can I simulate a filling of the DataGrid in the background?

Or the question I wanted to ask simply is: I understand how you get the data in packets over the wire, but how do you work with the data on the client while you are still receiving more packets?

Regards
Bastian

Answer 3

Today I learned the following:

Passing database data via DataSet and WCF takes approximately twice as long as converting data to a string array, sending it via WCF and converting it back to a DataSet in the client.

Wow I didn't know how much overhead a DataSet produces, but the Stopwatch doesn't lie...

The WsHttp is about 20% slower than NetTcp Binding.

These are the facts from my test and development environment, so I don't know if the results will be the same in your cases, but I thikn they will be similar...

Regards
Bastian

Answer 4

(1)

please check this blog,binary gives better performance,if you want to use wshttp,please use MTOM

http://blogs.msdn.com/yassers/archive/2006/01/21/515887.aspx

(2)

About dataset,somebody suggested this trick before,i didn't try it yet,This may work

You can set the following properties:

ds.SchemaSerializationMode = SchemaSerializationMode.ExcludeSchema;
ds.RemotingFormat = SerializationFormat.Binary;

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=462032&SiteID=1

 

Answer 5

I heard about that


ds.SchemaSerializationMode = SchemaSerializationMode.ExcludeSchema;
ds.RemotingFormat = SerializationFormat.Binary;

But unfortunately it does not work. WCF throws an exception and say something about that it is not allowed to overwrite the remoting format - sorry I can't remember the exception message at the moment.

Bastian

Answer 6

Bastian,

Are you using any of the features of Dataset in the server side?

How often is this service called?

How many connections simultaneous connections do you plan to support in your service?

Thanks

Answer 7

I checked the SchemaSerializationMode and RemotingFormat DataSet options with WCF. Both work properly though the second one has no effect on WCF perfoprmance as the RemotingFormat is considered in .NET Remoting only. When I exclude schema from the DatSet being sent the WCF works a bit better but in any case the .NET Remoting is faster when sending DataSet.

 

My performance comparison can be found here: WCF vs. Remoting (with DataSet)- performance comparison

 

Marcin Celej

Answer 8

The service is called very often and I think with about 2-5 simultaneous connections.

My solution is to convert the DataSet into a string (just the needed content) --> transfer it --> convert it back to DataSet after transfer.

This solution works about 10 times faster than the standard way.

Answer 9

Dear all,

 

I have a similar case for my project of building an alarming system.

I have a component which collect and send event data that I need to show and archives in database.

The way I have done it is as follow :

 

 - I have an initial component that collects alarms at a frequency of 500ms pooling ( It does not means that I willhave alarms each time but could happen that I receive a lot).

 

- Then I have build  windows service which host a remote component which is charge to buffer incoming alarms to database.I have using tcp chanel for that and binary formatter.

 

- Then I have by business logic which is informed when new data has been inserted to databse, to collect them and display to users. AS soon as my read alarms gets read , it is removed from the buffer table. In that solution I have all the time to display my alarms has they are buffered by my service with time stamp. When ready for display I how then in the order of appearence.

 

I have here two differente process: one which is bufferring, and an other which is reading

 

regards

serge

Answer 10

Honestly, I have dropped WCF serialization/deserialization altogether, it has way too much overhead for me. I use something similar to this AltSerializer solution posted on CodeProject:

 

http://www.codeproject.com/cs/library/AltSerializer.asp

 

 

The performance is dramatically better and it allows me to use one interface to pass any serializable object. Benchmark the AltSerializer: its very fast!

 

Trevor

 

Answer 11

interesting design choice to send 3mb of data in a single call. does the user need to see all 3mb of this data all at once? Why a dataset instead of strongly typed collection of objects?

 

I'm no expert of course, i just wonder if there isn't a better way to deliver data to your users than sending them a big 3meg chunk all at once. Surely they aren't looking at all 3 megs at once.

Answer 12

Hi,
    Just wanted to know two things.

1. How do I configure the size limit of Objects to be sent using WCF.
2. I am not able to serialize Datatable using WCF.

Regards,
    Jasaz

Answer 13

1) Answer for your first query is You can configure in say App.config or Web.config of WCF Client application
Just you have to modify the  maxReceivedMessageSize="2147483647"

Regards,
Ranjan.D