none
Using Serialization/Deserialization versus flat file for Interprocess Communication? RRS feed

  • Question

  • I have a process A that gathers data and hands it over for processing to process B. The amount of data is anywhere near 650MB or more, the transport should be as fast as possible, yet painless (if possible). A and B are different processes, so the scenario could be considered Inter Process Communication.

    Here is what I've done and found so far:

    1. Process A created an object filled with all information and serialized it into a file. B deserialized and processed the data.
      - the computational time on the serialization is OK.
      - the deserialization is many times bigger than serialization to the point that I consider it useless because it takes too long.
    2. I do not serialize/deserialize instead use a flat file for the communication.
      - it's quick and same speed for read and write
      - lacks flexibility and is not very elegant either.
    3. I used namedPipes to transfer the serialized object between processes A and B, which is elegant but ever slower because of the issue in #1 and the fairly poor performance of namedPipes and big data?

     

    Questions:

    1. Can somebody please advise me on how to best deal with my requirements?
    2. Explain to me why deserialization is soooo slow (I understand that depending on type a lot of object have to be created) and what better approach one should take.

    Does somebody even have example code for .NET 3.5 or .NET 4.x:

    1. using namedPipes in my context (I'm aware of the standard 'Hello World' stuff)
    2. exchanging LOTS of DATA between processes in the fastest possible way.

     

    Hints are much appreciated.
    Dirk

     p.s. all tests have been conducted on a 2x4core machine with 16GB RAM and SSD harddrive. It's not the machine.

    Tuesday, October 19, 2010 12:56 PM

Answers

  • Please, take a look on protobuf-net:

    http://code.google.com/p/protobuf-net/

    It will speed-up impressively both serialization and deserialization.

     

    Good Luck


    ...
    • Marked as answer by SamAgain Sunday, October 31, 2010 2:59 PM
    Sunday, October 24, 2010 11:11 AM
  • You can implement your own serializer - to reduce the volume of data exchanged and increase the speed of serialization and deserialization - but it might involved maintenance !! But if speed is really important - you'll increase speed a lot.

    I suggest you pack some DSeries to send them by 10 or 100 according the average size, through a pipe or even IP.

    This way, you might start deserialization BEFORE having received everything. The entire process will be done faster (even if you're not implementing your own serializer).

    Kind regards,


    Kind regards, Sebastien G. - Axiomtek Deutschland GmbH
    Wednesday, October 20, 2010 11:32 AM

All replies

  • Hello,

    Can you describe the objects you are exchanging :

    • how many classes
    • kind of data inside
    • quantity

    It'll be easier to give advice if we know "details" about the data you're exchanging. if you're sending thousands of simple objects is not the same than exchanging tones of very complex ones.

    Thanks,


    Kind regards, Sebastien G. - Axiomtek Deutschland GmbH
    Wednesday, October 20, 2010 5:11 AM
  • It's a single class, please see code parts below

    [Serializable]
    public class DSeries
    {
      private string _qk;
      private string _tsid;
      private double[] _hst;
      private double[] _fst;
      private double[] _modl;

    ....

    which I fill a LIST with

    public static List<DSeries> ResultCollection = new List<DSeries>();
     
    ...

    the list can contain up to 1000000 (1 million) entries wich is around 630MB if serialized into a file using the BinaryFormatter.


    Thanks for looking into this.

    Wednesday, October 20, 2010 10:43 AM
  • You can implement your own serializer - to reduce the volume of data exchanged and increase the speed of serialization and deserialization - but it might involved maintenance !! But if speed is really important - you'll increase speed a lot.

    I suggest you pack some DSeries to send them by 10 or 100 according the average size, through a pipe or even IP.

    This way, you might start deserialization BEFORE having received everything. The entire process will be done faster (even if you're not implementing your own serializer).

    Kind regards,


    Kind regards, Sebastien G. - Axiomtek Deutschland GmbH
    Wednesday, October 20, 2010 11:32 AM
  • Please, take a look on protobuf-net:

    http://code.google.com/p/protobuf-net/

    It will speed-up impressively both serialization and deserialization.

     

    Good Luck


    ...
    • Marked as answer by SamAgain Sunday, October 31, 2010 2:59 PM
    Sunday, October 24, 2010 11:11 AM