Send large files over TCP Sockets....

Question

Hello community,

I created this Socket client-server library where everything seemed to work as
i was sending simple strings and custom objects across the wire. Things got strange
as soon as i started to create the feature of sending files....small files are ok and
everything works as expected, but...when i want to send big files i get errors.

Then while researching about send large files over TCP Sockets and related subjects,
i created a split/assemble file operation to split the file to be sent over and assemble in the other side!
I should mention that TCP Server serves the files all over the clients, means one client to send file to another,
has to go through the TCP Server, as it is working proxy like, and while it is having all related socket references
sends the file,messages to the recipient.

While testing and debugging, everything seems to kinda work, when i am in slow motion (breakpoint-debugging-watching) :).
But soon as i try to send over the server in full speed the files, while grabbing their names into a list, and foreaching with
a Send() inside as every other object seems like Send() hangs up for a reason, and never completes the task.

I am assuming that packets are sent before the other side of the socket finished working with the previous byte[]s
and so there is no waiting for data but as far as i know there is no need to be in Receive(), the socket will grab the packets
waiting on the side soon as it gets in the Receive() state.

I use Async sockets to grab the data and a Sync sicket Send() Receive() only to clarify data information. As you already know,
Send(Information), BeginSend(Data), Receive(Information), BeginReceive(Data).

I should mention that files are moving in different port as the objects just for separation as i thought this strategy better!

Anyone with a solution? I was thinking something like slow down the sending between the files somehow, get inform when a file ended
receiving and continue sending another or i don't know...so the veterans here should suggest a fix.

There is another concern about the buffersize, what is the prefered size when sending receiving? I have 1024 right now and receiving data
while checking the full size of data from the information Receiv().

Thank you in advance!

Answer 1

The first thing to check is that you are properly doing message framing: http://nitoprograms.blogspot.com/2009/04/message-framing.html

A Send may block if too many bytes have not been acknowledged by the receiving side, e.g., the receiving window is full.

You shouldn't have to worry about the buffer sizes. Just leave them at the default.

       -Steve

---

Programming blog: http://nitoprograms.blogspot.com/
  Including my TCP/IP .NET Sockets FAQ

Microsoft Certified Professional Developer

Answer 2

Dear Steve,

As i learned so far, message framing is done with sending first the actual size file with a synchronous Send()
and the send the data Async. In fact with the size i send the file name too.

"A Send may block if too many bytes have not been acknowledged by the receiving side"

Is any workaround for this? any idea how to slow down the send between files? I thought starting the send in a different thread
and Sleep() that thread for sometime in every send, but i dont like this logic so much.

I wonder if there is anyway to know, when the other side acknowledged the whole packets, (means finished that part of file)
and waiting again for another?

G.Tas

Answer 3

Can you post the code for the receiving side? Also, you said that you are getting errors when you send large files. Can you tell us what errors you are getting?

As Stephen said, if you dont drain the receiving socket quickly enough, the receciving window will fill up and then your send will block.

Post the code and we can see what can be going wrong, along withh the details of errors you are seeing.

---

feroze
--
My blog
Instruction on how to create a tracelog with your System.Net application

Answer 4

http://rapidshare.com/files/301038905/Sockets.zip.html

I uploaded sample of the code with some comments in the sendfile method,
while i was trying today after some fix (not in the send receive side of sockets but in handling and sendig files)
i noticed without sending splited files the file is sent and received, but i give you the error on a png image inside the zip file,
where it comes up without any reason at all. I tried over 500mbytes file, and seemed to work, but i feel and know that i should
make it more perfect and understand it better, without doubt of working properly, at any time.

Basic problem is if i should fix the problem with spliting and sending files in parts, or
try to understand why these strange values as you can see in the png image.

So, give me an opinion and help me on clearing this out.

Thank you for helping so far.

Answer 5

I havent looked at the code in detail, just the WaitForData() method.

Looking at the error  you are getting, which is actually an IndexOutOfRangeException, in the following lines:


                    byte[] temp = new byte[1024];
                    int readBytes = stateClient.Socket.Receive(temp, 0, temp.Length, SocketFlags.None);
                    if (readBytes < 1) throw new SocketException();
                    int dataLength = BitConverter.ToInt32(temp, 0);
                    int command = BitConverter.ToInt32(temp, 4);
                    int parameterLength = BitConverter.ToInt32(temp, 8);
                    int typeOfObjectLength = BitConverter.ToInt32(temp, 12);

                    stateClient.NumBytes = dataLength;
                    stateClient.Command = (CommandInOrder)command;
                    >>>> stateClient.Parameter = Encoding.UTF8.GetString(temp, 16, parameterLength);
                    stateClient.TypeOfTheObject = Encoding.UTF8.GetString(temp, 16 + parameterLength, typeOfObjectLength);

The exception is happening in the highlighted line.

What I think is happening is that you are always assuming that you get all the data in one call to Socket.Receive(). This is not correct. TCP is a streaming protocol, which means that if the server sends 10 bytes of data, the receive will not always get 10 bytes. The receive might complete 10 times with 1 byte each, or once with the whole 10 bytes, or anything in between.

So, in the above case, you have to first make sure that you got all the data before you can decode it.

So, something like...

int dataBytesAvailable = readBytes - 16 // 16 is the size of the packet header.
int dataBytesNeeded = parameterLength + typeOfObjectLength;
int dataBytesRemaining = dataBytesNeeded - dataBytesAvailable;

// allocate an array for the data bytes. We will put all data bytes here.
byte [] dataArray = new byte[dataBytesNeeded];
// copy the remaining data from temp array into this array
int index = 0;
Array.Copy(temp, 16, dataBytesAvailable, dataArray, 0);
index = dataBytesAvailable; // the next byte of data should start at this index.
if (dataBytesAvailable <dataBytesNeeded)
{
           // issue another read
           while(dataBytesRemaining > 0)
           {
                   readBytes = socket.Receive(dataArray, index, dataBytesRemaining);
                   index += readBytes;
                   dataBytesRemaining -= readBytes;
           }
}

// at the end of this, you have all data you need in dataArray
stateClient.Parameter = Encoding.UTF8.GetString(dataArray, 0, parameterLength);
stateClient.TypeOfTheObject = Encoding.UTF8.GetString(dataArray, parameterLength, typeOfObjectLength);

There might be slight bugs in the code, but you get the idea. Also, the same assumption applies for reading the packet header - you cannot assume that you got all your packet header bytes in one call to Socket.Receive().

Good luck!

---

feroze
--
My blog
Instruction on how to create a tracelog with your System.Net application

Answer 6

Thank you Feroze, i will imply this in my code and then mark your answer.
This makes lot of sense, and as i figure out, i was lucky working so far good!

This should work! Im back with news...

Answer 7

Dear Feroze,

Trying to figure your code, i see that this starts breaking my code and not working properly!
Have you seen that i send only the info and then starting asynchronous Receive to get the data?
If im not sending all the data as you say and i understand this on how TCP works then why my
code hangs again in the Receive inside the While?
Here is what i wrote with your telling:

                byte[] temp = new byte[1024];
                int readBytes = stateClient.Socket.Receive(temp, 0, temp.Length, SocketFlags.None);
                if (readBytes < 1) throw new SocketException();
                <br/>
                int dataLength = BitConverter.ToInt32(temp, 0);
                int command = BitConverter.ToInt32(temp, 4);
                int parameterLength = BitConverter.ToInt32(temp, 8);
                int typeOfObjectLength = BitConverter.ToInt32(temp, 12);

                // Testing
                int dataBytesAvailable = readBytes - 16;
                int dataBytesNeeded = dataLength + command + parameterLength + typeOfObjectLength;
                int dataBytesRemaining = dataBytesNeeded - dataBytesAvailable;

                byte[] dataArray = new byte[dataBytesNeeded];
                int index = 0;
                Array.Copy(temp, 16, dataArray, 0, dataBytesAvailable);
                index = dataBytesAvailable;
                if (dataBytesAvailable < dataBytesNeeded)
                {
                    while (dataBytesRemaining > 0)
                    {
                       Code waits again --> readBytes = stateClient.Socket.Receive(dataArray, index, dataBytesRemaining, SocketFlags.None);<br/>
                        // Then data comes and everything goes wrong!!!<br/>
                        index += readBytes;
                        dataBytesRemaining -= readBytes;
                    }
                }

                stateClient.NumBytes = dataLength;
                stateClient.Command = (CommandInOrder)command;
                stateClient.Parameter = Encoding.UTF8.GetString(dataArray, 0, parameterLength);
                stateClient.TypeOfTheObject = Encoding.UTF8.GetString(temp, parameterLength, typeOfObjectLength);

Answer 8

While debugging yesterday and limiting the problem, i am now seeing a problem in the client side, as before,
but now sometimes, while all values are ok and everything should work properly, stops receiving data and gets out of
the ReadCallback method, im posting the method in the client side:

        private void WaitForData(State stateClient)
        {
            try
            {
                ReceiveDone.Reset();
                // 
                byte[] temp = new byte[1024];
                int readBytes = stateClient.Socket.Receive(temp, 0, temp.Length, SocketFlags.None);
                if (readBytes < 1) throw new SocketException();

                //<br/>
                int dataLength = BitConverter.ToInt32(temp, 0);
                int command = BitConverter.ToInt32(temp, 4);
                int parameterLength = BitConverter.ToInt32(temp, 8);
                int typeOfObjectLength = BitConverter.ToInt32(temp, 12);

                stateClient.NumBytes = dataLength;
                stateClient.Command = (CommandInOrder)command;
                stateClient.Parameter = Encoding.UTF8.GetString(temp, 16, parameterLength);
                stateClient.TypeOfTheObject = Encoding.UTF8.GetString(temp, 16 + parameterLength, typeOfObjectLength);

                // 
                switch (stateClient.Command)
                {
                    default:
                        stateClient.Socket.BeginReceive(stateClient.ByteBuffer, 0, stateClient.ByteBuffer.Length,
                                                            SocketFlags.None, new AsyncCallback(ReadCallBack),
                                                            stateClient);
                        ReceiveDone.WaitOne();
                        break;
                }

            }
            catch (SocketException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = stateClient.ClientInfo,
                    Message = ex.Message,
                    SocketError = ex.ErrorCode.ToString()
                };
                if (OnSocketError != null) OnSocketError(e);
                stateClient.Socket.Close();
            }
        }

        private void ReadCallBack(IAsyncResult ar)
        {
            State stateClient = (State)ar.AsyncState;
            Socket handlerClient = stateClient.Socket;
            //  
            // <br/>
            try
            {
                int readBytes = handlerClient.EndReceive(ar);
                stateClient.MemStream.Write(stateClient.ByteBuffer, 0, readBytes);
                stateClient.NumBytes -= readBytes;
                if (stateClient.NumBytes == 0) <--noticed some rare times for no reason at all never get to 0 and just returns
with no error causing the socket to lose track and stop working, what is going on? i assume that this should work so far.
                {
                    switch (stateClient.Command)
                    {
                        case CommandInOrder.File:
                            HandleFile(stateClient);
                            ReceiveDone.Set();
                            WaitForData(stateClient);
                            break;
                        default:
                            RaiseData(stateClient);
                            ReceiveDone.Set();
                            WaitForData(stateClient);
                            break;
                    }
                }
                else if (stateClient.NumBytes > 0)
                {
                    // <br/>
                    handlerClient.BeginReceive(stateClient.ByteBuffer, 0, readBytes,
                                         SocketFlags.None, new AsyncCallback(ReadCallBack),
                                         stateClient);
                }

            }
            catch (SocketException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = stateClient.ClientInfo,
                    Message = ex.Message,
                    SocketError = ex.ErrorCode.ToString()
                };
                if (OnSocketError != null) OnSocketError(e);
                stateClient.Socket.Close();
            }
            catch (ObjectDisposedException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = stateClient.ClientInfo,
                    Message = ex.Message,
                    SocketError = string.Empty
                };
                if (OnSocketError != null) OnSocketError(e);
                stateClient.Socket.Close();
            }
        }

I would appreciate any help on that?

Answer 9

You're not doing message framing correctly. You need to buffer.
  http://nitoprograms.blogspot.com/2009/04/message-framing.html
  http://nitoprograms.blogspot.com/2009/04/sample-code-length-prefix-message.html

For example, what happens if Socket.Receive returns 3 in WaitForData? How would your code respond to that?

       -Steve

---

Programming blog: http://nitoprograms.blogspot.com/
  Including my TCP/IP .NET Sockets FAQ

Microsoft Certified Professional Developer

Answer 10

Ok Steve, i will read those carefully and do message frame in the Socket.Receive too.
Something similar like what i am doing with the async pattern!

Answer 11

As i learned so far, message framing is done with sending first the actual size file with a synchronous Send()
and the send the data Async. In fact with the size i send the file name too.

Nope Nope Nope.  Wrong wrong wrong.

Synchronous send does not mean the message is automatically framed.  Synchronous just means that the bytes were delivered to the network subsystem on the sending machine before the Send function returns.  The bytes will get transmitted to the receiving machine later, and you don't know how many bytes at a time will be sent.

With TCP, it's best to think of sending a buffer of 50 bytes as 50 sends of 1 byte each. 

Message framing comes from of the format of the bytes of transmitted data , not from of the arguments of the calling API (Send) or the size of the buffers exchanged between your program and the network subsystem.

Answer 12

Encode the filename string into a buffer using your favourite encoding, like UTF8.
Send 4 bytes.  These 4 bytes will be N , the length of the encoded filename buffer sent in natural (little) endian order.
Then send N bytes.   These N bytes will be the encoded filename buffer bytes.
Then send 8 bytes.  These 8 bytes will be L , the length of the file as an 8 byte integer in natural (little) endian order.
Then send L bytes.  These L bytes will be the bytes of the file itself.

Obviously if N or L are very large, you will have to break it up into pieces.  This is really the easy part though.  Just break the buffer into manageable pieces and send the pieces in order .

Answer 13

As I said at the bottom of my post, you might still have a bug because you are assuming that you can read all the info (or header) in one call to Socket.Receive(). I think you might be able to simplify your solution if you wrap the Socket in a BinaryReader instead...

byte[] temp = new byte[1024];

NetworkStream ns = new NetworkStream(stateClient.Socket);
BinaryReader br = new BinaryReader(ns);

int dataLength = br.ReadInt32();
int command = br.ReadInt32();
int parameterLength = br.ReadInt32();
int typeOfObjectLength = br.ReadInt32();

stateClient.NumBytes = dataLength;
stateClient.Command = (CommandInOrder)command;

// read in the string
br.Read(temp, 0, parameterLength);
stateClient.Parameter = Encoding.UTF8.GetString(temp, 0, parameterLength);

// read in the next string
br.Read(temp, 0, typeOfObjectLength);
stateClient.TypeOfTheObject = Encoding.UTF8.GetString(temp, 0, typeOfObjectLength);

This way, you dont have to worry about buffering on your side.

feroze
--
My blog
Instruction on how to create a tracelog with your System.Net application

Answer 14

        private void WaitForData(State stateClient)
        {
            try
            {
                ReceiveDone.Reset();
                byte[] temp = new byte[1024];
                int readBytes = stateClient.Socket.Receive(temp, 0, temp.Length, SocketFlags.None);
                if (readBytes < 1) throw new SocketException();

                // Test
                byte[] infoLength = new byte[4];
                Array.Copy(temp, 0, infoLength, 0, readBytes);
                while (readBytes < 4)
                {
                    int readBytesAgain = stateClient.Socket.Receive(infoLength, readBytes, infoLength.Length, SocketFlags.None);
                    readBytes += readBytesAgain;
                }

                int infoLengthInt = BitConverter.ToInt32(infoLength, 0);
                byte[] infoMessage = new byte[infoLengthInt];
                readBytes = stateClient.Socket.Receive(infoMessage, 0, infoMessage.Length, SocketFlags.None);
                //Array.Copy(temp, 0, infoLength, 0, readBytes);
                while (readBytes < infoLengthInt)
                {
                    int readBytesAgain = stateClient.Socket.Receive(infoMessage, readBytes, infoMessage.Length, SocketFlags.None);
                    readBytes += readBytesAgain;
                }

                int dataLength = BitConverter.ToInt32(infoMessage, 0);
                int command = BitConverter.ToInt32(infoMessage, 4);
                int parameterLength = BitConverter.ToInt32(infoMessage, 8);
                int typeOfObjectLength = BitConverter.ToInt32(infoMessage, 12);

                stateClient.NumBytes = dataLength;
                stateClient.Command = (CommandInOrder)command;
                stateClient.Parameter = Encoding.UTF8.GetString(infoMessage, 16, parameterLength);
                stateClient.TypeOfTheObject = Encoding.UTF8.GetString(infoMessage, 16 + parameterLength, typeOfObjectLength);
                // Test

                 switch (stateClient.Command)
                {
                    default:
                        stateClient.Socket.BeginReceive(stateClient.ByteBuffer, 0, stateClient.ByteBuffer.Length,
                                                            SocketFlags.None, new AsyncCallback(ReadCallBack),
                                                            stateClient);
                        ReceiveDone.WaitOne();
                        break;
                }

            }
            catch (SocketException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = stateClient.ClientInfo,
                    Message = ex.Message,
                    SocketError = ex.ErrorCode.ToString()
                };
                if (OnSocketError != null) OnSocketError(e);
                stateClient.Socket.Close();
            }
        }

        private void ReadCallBack(IAsyncResult ar)
        {
            State stateClient = (State)ar.AsyncState;
            Socket handlerClient = stateClient.Socket;
            
            try
            {
                int readBytes = handlerClient.EndReceive(ar);
                stateClient.MemStream.Write(stateClient.ByteBuffer, 0, readBytes);
                stateClient.NumBytes -= readBytes;
                if (stateClient.NumBytes == 0)
                {
                    switch (stateClient.Command)
                    {
                        case CommandInOrder.File:
                            HandleFile(stateClient);
                            ReceiveDone.Set();
                            WaitForData(stateClient);
                            break;
                        default:
                            RaiseData(stateClient);
                            ReceiveDone.Set();
                            WaitForData(stateClient);
                            break;
                    }
                }
                else if (stateClient.NumBytes > 0)
                {
                                  handlerClient.BeginReceive(stateClient.ByteBuffer, 0, readBytes,
                                         SocketFlags.None, new AsyncCallback(ReadCallBack),
                                         stateClient);
                }

            }
            catch (SocketException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = stateClient.ClientInfo,
                    Message = ex.Message,
                    SocketError = ex.ErrorCode.ToString()
                };
                if (OnSocketError != null) OnSocketError(e);
                stateClient.Socket.Close();
            }
            catch (ObjectDisposedException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = stateClient.ClientInfo,
                    Message = ex.Message,
                    SocketError = string.Empty
                };
                if (OnSocketError != null) OnSocketError(e);
                stateClient.Socket.Close();
            }
        }

         public void SendData(byte[] data, CommandInOrder commandEnum, byte[] parameter, string objectType)
        {
            SendDone.Reset();
            int commandInt = (int)commandEnum;
            byte[] command = BitConverter.GetBytes(commandInt);
            byte[] typeOfObject = Encoding.UTF8.GetBytes(objectType);
            int dataBytes = data.Length;
            byte[] dataBytesLength = BitConverter.GetBytes(dataBytes);
            int commandBytes = command.Length;
            int parameterBytes = parameter.Length;
            byte[] parameterLength = BitConverter.GetBytes(parameterBytes);
            int parameterLengthBytes = parameterLength.Length;
            int typeOfObjectBytes = typeOfObject.Length;
            byte[] objectTypeLength = BitConverter.GetBytes(typeOfObjectBytes);
            int objectTypeLengthBytes = objectTypeLength.Length;
            int sumBytes = dataBytesLength.Length + commandBytes + parameterBytes + parameterLengthBytes +
                            typeOfObjectBytes + objectTypeLengthBytes;

            byte[] allData = new byte[sumBytes];
            dataBytesLength.CopyTo(allData, 0);
            command.CopyTo(allData, dataBytesLength.Length);
            parameterLength.CopyTo(allData, dataBytesLength.Length + command.Length);
            objectTypeLength.CopyTo(allData, dataBytesLength.Length + command.Length + parameterLength.Length);
            parameter.CopyTo(allData, dataBytesLength.Length + command.Length + parameterLength.Length + objectTypeLength.Length);
            typeOfObject.CopyTo(allData, dataBytesLength.Length + command.Length + parameterLength.Length + objectTypeLength.Length + parameter.Length);

            try
            {
                // Test
                byte[] alldataLength = BitConverter.GetBytes(sumBytes);
                int allDataLengthBytes = alldataLength.Length;
                SocketClient.Send(alldataLength, 0, allDataLengthBytes, SocketFlags.None);
                // Test

                
                SocketClient.Send(allData, 0, sumBytes, SocketFlags.None);

    
                if (dataBytes != 0)
                {
                    SocketClient.BeginSend(data, 0, data.Length,
                                           SocketFlags.None, new AsyncCallback(EndSendClient),
                                           SocketClient);
                    SendDone.WaitOne();
                }
            }
            catch (SocketException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = ThisClient,
                    Message = ex.Message,
                    SocketError = ex.ErrorCode.ToString()
                };
                if (OnSocketError != null) OnSocketError(e);
                SocketClient.Close();
            }
            catch (ObjectDisposedException ex)
            {
                SocketErrorEventArgs e = new SocketErrorEventArgs
                {
                    Client = ThisClient,
                    Message = ex.Message,
                    SocketError = ex.ObjectName
                };
                if (OnSocketError != null) OnSocketError(e);
                SocketClient.Close();
            }
        }

         private void EndSendClient(IAsyncResult ar)
        {
            Socket client = (Socket)ar.AsyncState;
            client.EndSend(ar);
            SendDone.Set();
        }

How about this? Sending 4 bytes length of the info, sending the info where includes the length of the actual data (file, object, whatever) and async receive
the data! ? Is this correct for your telling?

Answer 15

There is still a bug in your receive loop, where you are waiting to read 4 bytes.

 

                // Test
                byte[] infoLength = new byte[4];
                Array.Copy(temp, 0, infoLength, 0, readBytes);
                while (readBytes < 4)
                {
                    int readBytesAgain = stateClient.Socket.Receive(infoLength, readBytes, infoLength.Length, SocketFlags.None);
                    readBytes += readBytesAgain;
                }

You are not decreasing the #bytes remaining by the #bytes already read.

See this article where I show how to serialize an ICMP header into the socket, and deserialize it. It should help you with your situation.

http://blogs.msdn.com/feroze_daud/archive/2005/10/24/484260.aspx

---

feroze
--
My blog
Instruction on how to create a tracelog with your System.Net application