Parsing byteArray data in TCP Socket Server

Question

I am using a socket client to send data to a socketServer :
 
At socket Client :
 
   string  strUser = "[Photo][User] + "\n"   ;    where \n  is  LF (Line feed) is the seperator
 
  byte[] Ar_user =  Encoding.UTF8.GetBytes(strUser)

  byte[]  Ar_TTL =  Ar_user +  PhotoByte[]  ( where PhotoByte[] is the actual photo)

 using this method to combine array :
 
    byte[] rv = new byte[ a1.Length + a2.Length + a3.Length ];
     System.Buffer.BlockCopy( a1, 0, rv, 0, a1.Length );
     System.Buffer.BlockCopy( a2, 0, rv, a1.Length, a2.Length );
     System.Buffer.BlockCopy( a3, 0, rv, a1.Length + a2.Length, a3.Length );

suppose the Ar_TTL is sent by a Socket client to a socket Server.
 
At socket server, the problem I want to solve:
 
1) Is above combine array workable ?
 
2) can I use this (\n) line feed inside this Ar_TTL ? will there be any of this in PhotoByte[]? what will be the best separator to use?

3) How to parse to get the [Photo][User]  and the actual PhotoByte[]  from Ar_TTL ? or is there other way to  do parsing?
 
This is what I used and not sure it is workable  :
 
             data = new byte[_client.ReceiveBufferSize];
 
            _client.GetStream().BeginRead(data, 0, System.Convert.ToInt32        (_client.ReceiveBufferSize),  ReceiveMessage, null);

     public void ReceiveMessage(IAsyncResult ar)
         {
                    int bytesRead;
 
                    const int LF = 10;
                      int i = 0;
                     int start = 0;
                      byte[]  receivePhoto;

                    while (data[i] != 0)
                     {
                         if (i + 1 > bytesRead)
                         {
                             break;
                         }
                         if (data[i] == LF)
                         {
                             //- How to parse and get the photoByte[] ?

                          }
 
                    }
 
     }

Thanks

Answer 1

Are you aware that you are dealing with a stream of bytes, not discrete messages?  The bytes sent by the client can be repackaged up and be received in different groupings than the client sent them.  But the order will always be the same.

For example.  The client may send:

HELLO

COOL

WORLD

Then the receiver may receive

HEL

LOCO

OLWOR

L

D

(* I'm putting the contents of each buffer on a new line for illustration purposes only.  There are no artificial new-line characters added to the stream.)

The receive buffer will contain at least one byte.  This is because there's no point in being notified of 0 bytes of data.  In fact, when the callback says that the number of bytes received is zero, that actually means that connection has been lost.  If your receive buffer is large enough, you might even just receive the whole thing at once.

But the bytes sent over a TCP stream have no magic delimiters.

Imagine writing those words to a file with no new-line characters.  Here's what's in the file:

HELLOCOOLWORLD

Where are the boundaries?  They're gone!  You have to insert your own delimiters manually.

I could choose to use a * as a delimiter.  So I would then send:

HELLO*

COOL*

WORLD*

And my stream looks like this:

HELLO*COOL*WORLD*

But the receiver will still get it in random chunks.  So it might get 4 discrete notifications like this:

HEL

LO*CO

OL*WORL

D*

Now it's at least possible to parse the bytes and merge them together to form messages and look for *'s to delimit messages.

You could technically use a new-line character as a delimiter but be aware that the receiver will still get random chunks of the bytes, and the delimiters might be in the middle of the receive buffer and you will always need to be responsible for merging buffers together, or handling what happens when you have multiple messages (and therefore multiple delimiters) in the same buffer.

The agreement on the format of the data in the stream is called a protocol and the act of introducing boundaries to break the stream into groups called messages is called message framing.

---

But what if you want to be able to send a * or a new-line?  You might not want to designate any character as a delimiter.  You might want to send binary data.  So instead you can choose a binary protocol.  An example of a binary protocol is where you first write out a fixed-width size field, (for example a 4 byte integer, least significant byte first), that says how many of the following bytes are part of the message.  Then you can send out:

{0x05}{0x00}{0x00}{0x00}HELLO{0x04}{0x00}{0x00}{0x00}COOL{0x05}{0x00}{0x00}{0x00}WORLD

And this is easy enough to parse to put back together into messages too.  You might choose a smaller size field, or you might go with a variable length encoding or whatever you want.

Or you can use HTTP.  That's a well defined protocol that handles all these issues too.

Answer 2

I have no idea what you're trying to accomplish with the BlockCopy stuff.  I don't believe it is right though as it appears you're overwriting the existing array on each call (you specify 0 as the offset I believe). 

In general try to avoid working with byte arrays explicitly.  Here's how I'd do it

byte[] data = null;
using (var stream = new MemoryStream())
{
   using (var writer = new BinaryWriter(stream))
   {
      writer.Write("Whatever" + "\n");
      writer.Write(PhotoByte);
   };

   //Better yet just pass it directly to the socket client
   data = stream.ToArray();  
};

Write(string) writes the string out length prefixed so it is a little different than your code.  If you want to write out the string encoded differently then add an extension method.  But see below as to why you probably don't want to.

In your current implementation you're using \n as a delimiter but I wouldn't do that since you're working with binary.  In general you'll want to use a fixed size header to indicate the start of each of your messages.  When network issues start occurring you'll need to resync your client and server (assuming you're not doing UDP or something where you don't care).  Without a fixed size header you're going to have no way of knowing whether the byte stream you're reading is the beginning or middle of a message.  In general a fixed size header contains a starter signature string followed by the length of the header.  Within the header you generally include any additional information you might be interested in such as the type of message being sent and the additional amount of data after the header that you want to read.  Whenever you read the socket you'll ignore everything up to the signature (as it is probably part of a previous message that failed).  Then you read the rest of your fixed size header.  You can then determine how to process the request on the server based upon the header information.  This might involve reading more data.

If you follow the above approach then you'd first read the header which would tell you want the message is and you can ensure that you're at the start of the request.  Maybe something like this:

public struct MessageHeader
{
   //Should be short but random enough to not appear in the data stream
   public static readonly ValidSignature = new char[] { 'M', 'I', 'N', 'E' };

   public char[] Signature;  //4 byte fixed signature
   public long HeaderSize;
   public int MessageType;
   ...
   public long PayloadSize;

   public static MessageHeader CreateMessageHeader ( )
   {
      var header = new MessageHeader();
      Array.Copy(Signature, ValidSignature, 4);
      return header;
   }

   public bool HasValidSignature ( )
   {  
       return (Enumerable.SequenceEquals(Signature, ValidSignature);                
   }
}

Once you've read the header you'd verify the signature.  The header would tell you how much additional data to read.  You'd then read the rest of the data and start processing based upon the message type.  In the case of a photo you'd know to read the length-prefixed string followed by the byte array.  You can use BinaryReader to read the data without having to work with the byte array at all.  Note that if you use a length prefixed string you no longer need the \n delimiter either.

Michael Taylor - 3/21/2013
http://msmvps.com/blogs/p3net

Answer 3

I really appreciate both of your advices and suggestions. Both of you have opened up my understanding in socket sending data. However, I am new in socket and hope you both can guide me along.

I want to  create a multi-User socket app where the socket server will forward or send the data to respective clients base on using these protocols :

1) Sender sends  a message to a receiver in this way :  [Msg][Username] + Message + "\n"

   At Socket Server, it will look for the "\n" which indicate the end of line, then handle accordingly.
   it will send this text-message to user with this username.

2) Requester sends photo request : [Photo][Username] + Message + "\n"

responder sends back the photo to requester:  [Photo][Username] + PhotoByte[]

The problems now  at Socket Server  for (2)-protocol
--------------------------------------------

2a) What will be the workable protocol to use so that socket server knows what to do.

Old approach :    [Photo][Username] + "\n "  + PhotoByte

- With understanding from both of you ,  this "\n" is not useful here.

New approach :   [Photo][Username] + PhotoByte

-With understanding now, Socket server will receive it in random order BUT in SAME order.

So now, how my Socket will work with consideration of this protocol : [Photo][Username]+PhotoByte?

With below logic :
- Socket server will look for the "\n"  for  Text-message data . This is fine.
- Socket server can not handle for case like [Photo][Username] + PhotoByte

Would appreciate if you show me the logic in handling both of Text-Message and Photo-Message?
Sorry did not mention earlier that this socketServer need to handle Text and Photo. As this photoByte thing is really new to me, I can only handle text-message for this while loop.

 

public void ReceiveMessage(IAsyncResult ar)
          {
                     int bytesRead; 
                     const int LF = 10;
                      int i = 0;
                      int start = 0;
                      byte[]  receivePhoto;
 
                    while (data[i] != 0)
                      {
                          if (i + 1 > bytesRead)
                          {
                              break;
                          }
                          if (data[i] == LF)
                          {
                              //- How to parse and get the photoByte[] ?
 
                          } 
                     } 
      }

 Thanks.

 

 

 

 

 

 

Answer 4

You cannot reliably send string messages from one machine to another using \n as the delimiter.  The issue is that binary data can be translated into any data type so there is no reliable way of knowing that you've received a \n.  It could be an integral 13, part of a double, a Boolean value, etc.  The only time textual messages work is if you're building a simple echo client/server (which most examples use).  If echo is all you really want then MSDN has lots of examples on it.  But it won't work reliably in professional code.  You need to move away from the entire concept of sending text from a client to a server. 

You need to send binary messages (in which a string can be sent as part of it).  Going back to my original code where you have a message header before each data then you'd first send a message header for the Msg command.  The payload can then be just the string message.  You don't need [Msg] or any delimiter.  You can use BinaryWriter to handle writing the message header and the message itself.

var header = MessageHeader().Create();
header.MessageType = MessageType.Message;  //Eliminates need for [Msg]
header.PayloadSize = //size of the text message to send in bytes

writer.Write(header);  //Create an extension method that writes out the header info
writer.Write(textMessage);  //Writes out the raw text

//Server side
var header = reader.ReadHeader();  //Extension method to read in message header, ignores anything that isn't part of valid header
switch(header.MessageType)
{
   case MessageType.Message : ProcessTextMessage(header, reader); break;
   case MessageType.Photo: ProcessPhotoMessage(header, reader); break;
};

The Process methods will then read the appropriate data (string, binary[], whatever).  If the client goes down part way through the message then the server won't bother processing the request.  If the client sends a partial message and then starts to send another one then the server will probably respond to the second message. 

Michael Taylor - 3/23/2013
http://msmvps.com/blogs/p3net

Answer 5

Hi Michael,

Thank for your explanation. I am still not able to implement your suggestion for this multi-user socket app. Here, I explain what I do and hope you help me to implement where required.

//-1-   on Client side:

I am using Socket v4.0 or higher to create socket client and to connect to TCP socket server.

       socket cleint (1) <--------> Socket Server App <---------> Socket client(2)

Note on server side:

The Socket Server App  will keep a record of the socket Client by their IPaddr when they are connected.

//-2-  after clients connected to the socket server , use this protocol for communication among clients

  Message Type :   example : 
 
  [Msg]  for sending text-message
  [Photo]  for sending photo or image

  -- Create a text-message to send :

     [Message Type][username] +  strMessage + "\n"   

 //-3-   Client(1) send a message to client(2)
    
   string strMsg = "[Msg][Username] + "Hi, What's up" + "\n";

 //-- Send strMsg in byte[]  in below method:

    SocketClient.Send(strMsg)
 |
 V

public string Send(string data)
        {
            string response = "Send Operation Timeout";

            //************
            // We are re-using the _socket object that was initialized in the Connect method

            if (_socket != null)
            {
                // Create SocketAsyncEventArgs context object
                SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs();

                // Set properties on context object
                socketEventArg.RemoteEndPoint = _socket.RemoteEndPoint;
                socketEventArg.UserToken = null;

                // Inline event handler for the Completed event.
                // Note: This even handler was implemented inline in order to make this method self-contained.
                socketEventArg.Completed += new EventHandler<SocketAsyncEventArgs>(delegate(object s, SocketAsyncEventArgs e)
                {
                    response = e.SocketError.ToString();

                    // Unblock the UI thread
                    _clientDone.Set();
                });

               //*******    Send strMsg in byte[]

                byte[] payload = Encoding.UTF8.GetBytes(data);
                socketEventArg.SetBuffer(payload, 0, payload.Length);

                // Sets the state of the event to nonsignaled, causing threads to block
                _clientDone.Reset();

                // Make an asynchronous Send request over the socket
                _socket.SendAsync(socketEventArg);

                // Block the UI thread for a maximum of TIMEOUT_MILLISECONDS seconds.
                // If no response comes back within this time then proceed
                _clientDone.WaitOne(TIMEOUT_MILLISECONDS);
            }
            else
            {
                response = "Socket is not initialized";
            }

            return response;
        }

 

  //-4-  The Socket server App can handle the stream and forward the Message to Client(2)

using System.Net.Sockets.TcpListener listener

public void ReceiveMessage(IAsyncResult ar)
           {
                      int bytesRead; 
                     const int LF = 10;
                       int i = 0;
                       int start = 0;
                       byte[]  receivePhoto;
 
                     while (data[i] != 0)
                       {
                           if (i + 1 > bytesRead)
                           {
                               break;
                           }
                           if (data[i] == LF)
                           {
                           
                           //-- Get the Message Type and username out from the stream

                          // send the data to client who is represented by username here.

                           } 
                     } 
      }

======= The problems now

-(a)  Client's IPAddr

The client's IPAddr is identified when they are first connected to socketServer ( in above step1). 

-(b)  using version4.0 Socket client's method to send Byte[].

 I used it to send text Data to socket server. It is OK.
   
example :    string data = "[Msg][Username] + "Hi, What's up" + "\n";  

                byte[] payload = Encoding.UTF8.GetBytes(data);

                socketEventArg.SetBuffer(payload, 0, payload.Length);

                _socket.SendAsync(socketEventArg);        
 
This protocol , socket server app can parse to get the MessageType and username as in (4) and send it to the respective client.

Problem(s)
 
Problem(C) :
How to define a protocol that allows socket server app to identify message Type and Username for Photo-message  as well as  text-message ? and how Socket server side handling sending/receiving photo?

C 1) For sending Photo, to use this socket method in (b) to send. So,  I think of using this protocol

[Photo][username] + "\n" + PhotoByte[]  but the socket server logic will have problem with "\n"

C 2) with respect to your Method of using MessageHeader

1)  How Client(1) send a request to client(2)  for text-message as well as photo-message?

2) How the socket server handle the stream from (1)

3) Client(2) send the photo to Client(1)

Can I use your suggestion here :

byte[] data = null;

 using (var stream = new MemoryStream())
 {
    using (var writer = new BinaryWriter(stream))
    {
       writer.Write("Whatever" + "\n");   <-- can use  [Photo][Usernam] + "\n"  ??
       writer.Write(PhotoByte);               <-- the actual photo
    };
 
   //Better yet just pass it directly to the socket client

    data = stream.ToArray();  

};

- send by this socket own method :

  byte[] payload = Encoding.UTF8.GetBytes(data);

  socketEventArg.SetBuffer(payload, 0, payload.Length);

  _socket.SendAsync(socketEventArg); 

Thanks

 

 

 

 

 

Answer 6

The code that I keep posting solves the problems you mentioned.  But you keep going back to sending text to the server.  I don't understand why you are adamant about sending "[Msg]blah\n" from the server.  It is going to have all the problems you keep mentioning.  The code I posted allows you to send any message from client to server and is pretty much defacto code that is used in every client/server system I've ever seen implemented.  Could you get a text message to work?  Yes.  Is it worth it?  Probably not.

Here's more sample code.

public abstract class SocketMessage
{
   protected SocketMessage ( MessageType messageType )
   {
      Header = MessageHeader.Create();
      Header.MessageType = messageType;
   }

   public void ToStream ( Stream stream )
   {
      //Get message payload
      var data = GetMessageData();
      Header.PayloadSize = data.Length();
      Header.Data = data;

      //Send to stream
      ...
   }

   protected abstract byte[] GetMessageData ();

   protected MessageHeader Header { get; private set; }
}

public class TextMessage : SocketMessage
{
   public TextMessage ( ) : base(MessageType.Text)
   { }

   public string Text { get; set; }

   protected override byte[] GetMessageData () 
   {
      return Encoding.Ansi.GetBytes(Text);   
   }
}

public class PhotoMessage : SocketMessage
{
   public PhotoMessage : base(MessageType.Photo)
   { }

   public byte[] Photo { get; set; }

   protected override byte[] GetMessageData ()
   {
      return Photo; 
   }
}

//Send a text message
var msg = new TextMessage() { Text = "Hello" };
msg.ToStream(clientStream);

//Send a photo
var msg = new PhotoMessage() { Photo = ... };
msg.ToStream(clientStream);

It doesn't get any easier than that for the client.  The only thing you need to implement is the conversion of the message to a byte array by using BinaryWriter.  On the server side the opposite approach occurs where you'll reader the header to identify the type of message, instantiate the correct class instance and then let it process the byte stream.  You'll likely have a separate Request and Response type for each type of message.  The above code would have defined the Request type for the 2 message types you mentioned.  Remember, you won't be sending raw text across the wire.  Since you're working with photos you have to use binary data otherwise your server isn't going to handle photos properly.

Michael Taylor - 3/24/2013
http://msmvps.com/blogs/p3net

Answer 7

Hi Michael,

Sorry for Not being a good listener. I read many times. Now ,I understand you had explained what need to be done.  Is it just that I am not able to follow as your methods implementation are advance level but I do understand the required parts like Headersize, Playloadsize and MessageType.

(1) //-- your method of sending

public void ToStream ( Stream stream )
   {
      //Get message payload
      var data = GetMessageData();
      Header.PayloadSize = data.Length();
      Header.Data = data;

      //Send to stream
      ...
   }

//--  Can I use this below method :

suppose , I use simple way to send data from client side  as follows:

string MessageType ="Photo"    //  or "Text"   ( this base on if I send photo or text message)

int PayloadSize =  photobyte.length /1024 ( Kb)  or just PhotoByte.Lenght ?

byte[] Photo = PhotoByte[]

your method:
The only thing you need to implement is the conversion of the message to a byte array by using BinaryWriter

//--  so the message will be sent in this order:   MessageType + int PayloadSize + Photo

byte[] data = null;
 
 using (var stream = new MemoryStream())
  {
     using (var writer = new BinaryWriter(stream))
     {
        writer.Write( MessageType + PayloadSize);  
        writer.Write(Photo);              
     };
 
    //Better yet just pass it directly to the socket client
 
    data = stream.ToArray();  

};

//--- send the above data using this created client socket's method (version 4.0 socket) on client side.

This client socket is created and connected to the socket Server App :
 
  byte[] payload = Encoding.UTF8.GetBytes(data);

  socketEventArg.SetBuffer(payload, 0, payload.Length);

  _socket.SendAsync(socketEventArg);   //-- using this method to send

 

 

(2) How to handle this in Server side using TCP Socket Server?

public ClientMessage(TcpClient client)
{
           
    //-- the incoming client
     _client = client;
    data = new byte[_client.ReceiveBufferSize];
  _client.GetStream().BeginRead(data, 0, System.Convert.ToInt32(_client.ReceiveBufferSize), ReceiveMessage, null);

 }
   
 
      public void ReceiveMessage(IAsyncResult ar)
      {

        // ?  How to read to get the data

           MessageType, Int PhotoLenght, PhotoByte[]

       }

 

(3) With reference Wyck method:using binary protocol of fixed-width size field, (for example a 4 byte integer, least significant byte first), that says how many of the following bytes are part of the message.  Then you can send out:
 
{0x05}{0x00}{0x00}{0x00}HELLO{0x04}{0x00}{0x00}{0x00}COOL{0x05}{0x00}{0x00}{0x00}WORLD

How to generate this {0x05} to represent the word lenght?  Is this means :

  string 1stMsg = "HELLO";
  int 1stMsgLenght = 1stMsg.Lenght
  string str1stHex = "{0x" + 1stMsgLenght.ToString();  
  string strHexConstant = "{0x00}{0x00}{0x00}";

 string strHexAndMsgdata = str1stHex + strHexConstant + "HELLO" + "{0x04}{0x00}{0x00}{0x00}COOL{0x05}{0x00}{0x00}{0x00}  WORLD";

then use  byte[] payload = Encoding.UTF8.GetBytes(strHexAndMsgdata);

If use this , how to parse at the server side?

 public void ReceiveMessage(IAsyncResult ar)
      {
        // ?  How to read to get the :

         (nbr) MessageType + (nbr) Int PhotoLenght, PhotoByte[]
       }

 

I hope you can understand the way I explain.

Thanks

Answer 8

1) MessageType should be an integral value or enum so you can easily determine what type of message it is.
Use photoByte.Length for the payload size as it would be in bytes.

2) Sending data

//Not tested
public void ToStream ( Stream stream )
{
   //Data
   var data = GetMessageData();     
   header.PayloadSize = data.Length;
   
   //Warning: BinaryWriter will close the stream when it is done
   using (var writer = new BinaryWriter(stream))
   {
      //Write header
      writer.Write(header.Signature);
      writer.Write(header.HeaderSize);
      writer.Write(...);
      
      writer.Write(header.PayloadSize);
      writer.Write(data);      
   };
}

public void FromStream ( Stream stream )
{
   //Read the next message - reads the entire thing
   var message = ReadMessage(stream);

   using (var temp = new MemoryStream(message))
   {
      using (var reader = new BinaryReader(temp))
      {
         header.Signature = reader.ReadChar(4);
         header.HeaderSize = reader.ReadInt32();
         ...
         header.PayloadSize = reader.ReadInt32();
         data = reader.ReadBytes(header.PayloadSize);
      };
   };

   //Now you can do things like create a strongly typed message instance based upon the message type or whatnot
}

private byte[] ReadMessage ( Stream stream )
{
   //Find the start of the next message by scanning the bytes until we find the message signature
   //Once we find the message signature then read in the entire message header (it is fixed size)
   //Once we've done that we also know the payload size
   //so go ahead and read it in now as well
}

3) I have no idea what you're trying to do here.  If you're using BinaryReader then you don't need any of that code.  The above mentioned code is converting every message to a byte[] so you can call any of the socket write methods that accept a byte[]. The same goes for the server side. It doesn't matter whether you make sync or async read/write calls because everything is a byte array.

Michael Taylor - 3/25/2013
http://msmvps.com/blogs/p3net

Answer 9

OK here's a littler server whose sole purpose is to provide message framing.

The code for a client is similar except that you connect instead of listen.

The protocol is like this:  ( [header] [body] ) *

[header] is 4 bytes.  integer, LSB first : length of message

[body] is length of message bytes.

I didn't do any exception handling...you can do that yourself.

I didn't do any protocol validation (like checking to see if lengths are valid, or if they are zero.)

I didn't do much to optimize it or make it efficient.

Key points:

• Use a socket.
• Use asynchronous BeginXXX EndXXX techniques
• BitConverter can be used to convert between byte representation of integers.
• use UTF8 string encoding to convert strings to bytes and bytes to strings.
• Use an accumulator (list of bytes) to save up bytes between Receive callbacks.
• Use a finite state machine to keep track of Protocol state (parsing state) between receive calls.
• Note that the events will called from the ThreadPool thread that is executing the socket callback, so the event handler will likely need to Invoke a call back to the UI thread to do anything interesting.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net.Sockets;
using System.Net;
using System.Runtime.InteropServices;

namespace LittleServer
{
    public delegate void ProcessMessageDelegate( byte[] message );
    public delegate void DisconnectedDelegate();

    interface IClient
    {
        event ProcessMessageDelegate ProcessMessage;
        event DisconnectedDelegate Disconnected;

        void SendRawBytes( byte[] bytes );
        void SendProtocolMessage( byte[] message );
    }

    class Server
    {
        const int port = 8000;
        const int backlog = 10;
        public delegate void ProcessConnectionDelegate( IClient client );
        public event ProcessConnectionDelegate ProcessConnection;

        Socket listenSocket;
        public void Start()
        {
            listenSocket = new Socket( AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp );
            IPEndPoint ep = new IPEndPoint( IPAddress.Any, port );
            listenSocket.Bind( ep );
            
            listenSocket.Listen( backlog );
            Accept();
        }

        void Accept()
        {
            listenSocket.BeginAccept( new AsyncCallback( this.OnAccept ), null );
        }


        void OnAccept( IAsyncResult asyncResult )
        {
            Socket clientSocket = listenSocket.EndAccept( asyncResult );
            Client client = new Client( clientSocket );
            this.ProcessConnection( client );
            client.Receive();
            Accept();
        }
    }

    class Client : IClient
    {
        Socket socket;
        List<byte> accumulator = new List<byte>();
        int bodyLength;
        byte[] buffer = new byte[4096];

        public event ProcessMessageDelegate ProcessMessage = delegate { }; // Trick: always have one empty handler so we don't have to check for null
        public event DisconnectedDelegate Disconnected = delegate { }; // Trick: always have one empty handler so we don't have to check for null

        public Client( Socket socket )
        {
            this.socket = socket;
        }

        public void Receive()
        {
            this.socket.BeginReceive( buffer, 0, buffer.Length, SocketFlags.None, new AsyncCallback( this.OnReceive ), null );
        }

        void OnReceive( IAsyncResult asyncResult )
        {
            int numBytesReceived = socket.EndReceive( asyncResult );
            if( numBytesReceived == 0 )
            {
                // disconnected.
                Disconnected();
            }
            else
            {
                Parse( buffer, numBytesReceived );
                Receive();
            }
        }

        void Parse( byte[] buffer, int numBytes )
        {
            for( int i = 0; i < numBytes; i++ )
            {
                ParseByte( buffer[i] );
            }
        }

        enum State
        {
            Header,
            Body
        }

        State state = State.Header;

        void ParseByte( byte b )
        {
            switch( state )
            {
            case State.Header:
                accumulator.Add( b );
                if( accumulator.Count == 4 )
                {
                    bodyLength = BitConverter.ToInt32( accumulator.ToArray(), 0 );
                    accumulator.Clear();
                    state = State.Body;
                }
                break;
            case State.Body:
                accumulator.Add( b );
                if( accumulator.Count == bodyLength )
                {
                    ProcessMessage( accumulator.ToArray() );
                    accumulator.Clear();
                    state = State.Header;
                }
                break;
            }
        }

        void IClient.SendRawBytes( byte[] message )
        {
            socket.BeginSend( message, 0, message.Length, SocketFlags.None, new AsyncCallback( this.OnSend ), null );
        }

        void IClient.SendProtocolMessage( byte[] message )
        {
            byte[] header = BitConverter.GetBytes( message.Length );
            socket.BeginSend( new List<ArraySegment<byte>>() {new ArraySegment<byte>(header), new ArraySegment<byte>(message)}, SocketFlags.None, new AsyncCallback( this.OnSend ), null );
        }

        void OnSend( IAsyncResult asyncResult )
        {
            int bytesSent = socket.EndSend( asyncResult );
        }

        event ProcessMessageDelegate IClient.ProcessMessage
        {
            add { this.ProcessMessage += value; }
            remove { this.ProcessMessage -= value; }
        }

        event DisconnectedDelegate IClient.Disconnected
        {
            add { this.Disconnected += value; }
            remove { this.Disconnected -= value; }
        }
    }
}

Here's an example of setting up this server.  This particular usage expects the bytes of each [body] to be a UTF8 encoded string.  It also assumes you have a RichTextBox on your form called richTextBox1.

private void Form1_Load( object sender, EventArgs e )
{
    Server server = new Server();
    server.ProcessConnection += delegate( IClient client )
    {
        this.Invoke( (Action)delegate
        {
            this.richTextBox1.AppendText( "Client connected\n" );
        } );
        client.ProcessMessage += delegate( byte[] message )
        {
            this.Invoke( (Action)delegate
            {
                this.richTextBox1.AppendText( string.Format( "Client Message: {0} bytes: {1}\n", message.Length, string.Join( ",", message ) ) );
                this.richTextBox1.AppendText( Encoding.UTF8.GetString( message ) + "\n" );
                client.SendProtocolMessage( Encoding.UTF8.GetBytes( "OK" ) );
            } );
        };
        client.Disconnected += delegate
        {
            this.Invoke( (Action)delegate
            {
                this.richTextBox1.AppendText( "Client disconnected\n" );
            } );
        };
    };
    server.Start();
}

Without writing a client, you can test this with telnet.

connect to localhost 8000

So to send "TEST" you would have to send the bytes 4, 0, 0, 0, (that's the length of "TEST") then the UTF-8 encoded string "TEST".

Type the following (see below) to test it with telnet: (Boldface stuff is key strokes, normal face text is just literal text)  (And in case you're not familiar with sending bytes via the ALT+NumPad technique, then read this.)

telnet
o localhost 8000 ENTER
ALT+NumPad0+NumPad0+NumPad4
ALT+NumPad0+NumPad0+NumPad0
ALT+NumPad0+NumPad0+NumPad0
ALT+NumPad0+NumPad0+NumPad0
TEST
Ctrl+]
c

Answer 10

Hi Michael,

Many thanks to all of you. I tried very best to learn.  Below is the first method I tried out.

//----On client : I use this protocol  [Msg][usr][what]

            string Msg = "Msg";           
            string Usr = "David";
            string strWhat =" Hi, what's up";

            byte[] data = null;

           using (var stream = new MemoryStream())
            {
                using (var writer = new BinaryWriter(stream))
                {
                    writer.Write(Msg);
                    writer.Write(Usr);
                    writer.Write(strWhat);
                };

                //Better yet just pass it directly to the socket client
                data = stream.ToArray();             

            };

    //-- send it by socket client here :
         string strMsg2 = client.SendStream(data);

 

//-- on server side using TCP SocketServer:

byte[] data;

 public Client(TcpClient client)
        {
            _client = client;
         
            data = new byte[_client.ReceiveBufferSize];
            _client.GetStream().BeginRead(data, 0, System.Convert.ToInt32(_client.ReceiveBufferSize), ReceiveMessage,     null);
        }

public void ReceiveMessage(IAsyncResult ar)
        {
            int bytesRead;
         
                lock (_client.GetStream())
                {
                    bytesRead = _client.GetStream().EndRead(ar);
                }
                               
                   Stream stm = new MemoryStream(data);

                    BinaryReader br = new BinaryReader(stm);

                    //Seek to the start of the file
                    br.BaseStream.Seek(0, SeekOrigin.Begin);

                    //Read from the file and store the values to the variables

                    string strMsg = br.ReadString();

                    string strUsername = br.ReadString();

                    string  strWhat  = br.ReadString();
                  
                    //Display the data on the console
                    Console.WriteLine("Msg :" +strMsg);
                    Console.WriteLine("User:" + strUsername);
                    Console.WriteLine("What :" + intLength);
                   
                    //Close the stream and free the resources

                    br.Close();

                

     }

 

Questions

1) If I choose this protocol in which data is sent in this order : [Msg][Usr][What], then on the server I will receive in this order? Is above code correctly implmented or what you meant ?

which to use memoryStream or stream or bufferedStream for better performance?

2) How do I know  on the server side that the ReceiveMessage()  get ALL the stream or byte from Client?
 
 bytesRead = _client.GetStream().EndRead(ar) , is this will confirm that?

3) What will happen when many people send at the same time, will the Socket server receive the Data from client in ORDER ?

 

 Thanks

 

 

 

Answer 11

1) Since you're using TCP the packets may arrive in any order but they will be put back together in the order they were sent prior to you receiving the packet.  That is one of the benefits of TCP (vs UDP) so no you don't have to worry about ordering.

Stream is the base class so you won't use it directly.  MemoryStream is used to store data in memory and is generally used as an intermediary stream.  BufferedStream is used when you need buffering and is generally only beneficial on streams that are slow (i.e. network or file).  Your socket already provides you the NetworkStream instance so you don't need to create your own.  Depending upon how you decided to go you'll likely either read the data directly from the stream (or via BinaryReader) into a byte array or you could use a MemoryStream.  Either approach is fine.  You wouldn't gain anything by using a BufferedStream in this case.

2) You can't.  That is why I kept saying that your approach of passing strings from the client to the server wasn't going to work well.  A stream is nothing more than a list of bytes.  A set of bytes can be translated any # of ways so your app has to understand it.  Hence the use of a message header that is fixed size (so you can read the entire header into memory) in combination with a fixed signature in the header (a char array generally) ensures that you can find the start of each message and know how much to read.  The process of reading the data is a distinct step from translating it into a useful message.  Going back to your sample code your client is now at least using length-prefixed strings so you should be able to read the 3 strings back in without any issues PROVIDED your server is always starting at the beginning of the sent message.  Unfortunately if the server only partially read any previous message (for whatever reason) or the client only sent a partial message then your server is going to be reading garbage because it has no way of identifying the beginning of the message.  For a simple app or an app where the client/server are guaranteed to always read/write the entire message this isn't an issue.  You'll have to decide whether it is important for your app.

You cannot read to the end of the stream because the read will be in blocks (5 bytes here, 10 bytes there).  You need to know how many bytes to read and keep reading until you've gotten it all.  Since the strings are length prefixed you know how much to read (read the first length, then that many chars, read the next length, then that many chars, etc).

3) Each client that connects to your server will connect using a different socket. Each socket is bidirectional between the client and server. Each socket is tied to the client on the other end so on a single socket you will always get the messages (ultimately) in the order the client sent them to you (again, the benefit of TCP). However your server could be processing multiple requests at the same time on different clients. In general a server uses multiple threads to manage the sockets. For small #s of clients or cases where you can limit the # then a single thread created when each client connects is reasonable. For servers where there can be large #s of clients (i.e. IIS) then a thread pools is generally used.  In either case each thread is working with a single socket at a time while a listener thread is listening on the port for new connection requests.

Writing a TCP client/server system is not trivial.  I strongly recommend that you either read up on the various issues you're going to have to solve or you buy a good book on the topic.  There are quite a few books around that discuss how to implement a client/server system to meet various needs.

Michael Taylor - 3/27/2013
http://msmvps.com/blogs/p3net

Answer 12

Hi Wyck,

At this moment, I able to send data using BinaryReader&writer. I am also trying on your latest approach which is of advance level. Allow me to use this simple method and to learn each different method to understand how socket can be used to adapt to different situations .

You had mentioned the below method which indicate the length of message.

1) How to generate this 4 byte integer?
this is what I will do: a)Use string.length to get each message's length out first for text-message b) use image.length for photo message.

if the integer is 5 , how to generate a 4 byte integer as indicated by you as follows:
{0x05}{0x00}{0x00}{0x00}

string  strClientMsg ="{0x05}{0x00}{0x00}{0x00}HELLO{0x04}{0x00}{0x00}{0x00}COOL{0x05}{0x00}{0x00}{0x00}WORLD"

2) after (1), can I use this to conver to byte[]

 byte[] payload = Encoding.UTF8.GetBytes(strClientMsg);

3) Using your method.  I use it this way: 

On client side :
 
  byte[]  ClientDataByte =  5[Msg] + 6[user] + int length of photo  + photoByte[]

byte[] payload = Encoding.UTF8.GetBytes(ClientDataByte);
socketEventArg.SetBuffer(payload, 0, payload.Length);

On server side:

4) This is my problem on server side. what to use and How to read out each parts like :

a) 5[Msg] , 6 [user] for text-message

b) photo : length of photo  , photoByte[] 

Thanks. I will move on the advance level later.

Answer 13

To Michael,

I am working with the simple approach first to learn the differences. Agreed your notes. Will pay attention each steps in using your approach to this problem. I read your explanation many times as this socket is exciting thing which required some exercises to understand.

I am using your BinaryReader and Writer method as a starting point. I am able to send data from client. But I saw something not right as below:

on client side:

1) using Binarywriter, It works . Example  for this Message : [Msg][user]

                 byte[] data = null;
                   
                    using (var stream = new MemoryStream())
                    {
                        using (var writer = new BinaryWriter(stream))
                        {
                            writer.Write(Msg);
                            writer.Write(Usr);
                        
                        };

                        data = stream.ToArray();

                    };

 1(b)  //- pass data to  Client Socket to send

byte[] payload =data
socketEventArg.SetBuffer(payload, 0, payload.Length);

  1(c) //-- Over server side:  read out[Msg][user]

                 MemoryStream ms = new MemoryStream(data);
                    BinaryReader br = new BinaryReader(ms);

                    //Seek to the start of the file
                    br.BaseStream.Seek(0, SeekOrigin.Begin);

                    //Read out each part

                    string strMsg = br.ReadString();
                    string strUsername = br.ReadString();
                                  
                    //Display the data on the console
                    Console.WriteLine("Msg :" + strMsg + strUsername);

2) if I use this socket client to send this way:

string strClientMsg = "[Msg][user]"

byte[] payload = Encoding.UTF8.GetBytes(ClientMsg);
socketEventArg.SetBuffer(payload, 0, payload.Length);

on Server side : base on 1(c) method ,  I always get this  Msg][user],  missing "[ "

is Encoding.Ascii work different than UTF8 ??

3) say, I want to use this Method as in (1) with changes as follows:

    [Msg][user] + int  (forphotoLength)   + PhotoByte

          writer.Write(Msg);
          writer.Write(Usr);
          Writer.Write( PhotoLength)
          writer.Write(PhotoByte)

On server side:

//Read out each part accodingly base on the order of the protocol

      string strMsg = br.ReadString();
      string strUsername = br.ReadString();
      int  intLengthPhoto = br.readint32();

      How to read this PhotoByte?  use ReadByte or readBytes ?

Is this correct way to read ?

right now, I am trying to learn to use your advance method as well. Will report back later.

Thanks.

Answer 14

2) ReadString reads a length prefixed string.  Therefore you have to write out the string as length prefixed.  To do that use the BinaryWriter.Write method and pass it the string.  Don't convert it to a byte array first.  There is no need to.

var msg = "Hello";
writer.Write(msg);  //length prefixed

var msg = reader.ReadString(); //length prefixed

3) Use ReadBytes and pass it the length that you previously read.  It will return the bytes as an array.

Michael Taylor - 3/29/2013
http://msmvps.com/blogs/p3net