UDP server storing received data using pointers.

Question

I currently have a udp server that works. I am currently using the socket.receivefrom method and passing a byte array to store the data in.

What I would like to be able to do is have a large byte array buffer, with a pointer to where the received data should be stored.

I have defined the buffer on the stack using:

unsafe
{
    byte* pData = stackalloc byte[8014];
}


I then tried to recive the data using:

serverSocket.beginreceivefrom(pData, SocketFlags.none, ref epSender, new AsyncCallback(onReceive), epsender);

but I get the error:

Cannot convert byte* to byte[]

Answer 1

Do you really need to stackalloc your buffer?

Answer 2

Hi Louis,

I need to stackalloc because I want the buffer to be independant from the garbage collector.

How would you tackle this problem?

Answer 3

I see: for some reason, you want the memory for the buffer to be reclaimed at the end of your method.

May I ask why? I don't see a way to do exactly what you want. But maybe it's not what you really need.

Answer 4

Thanks again Louis for your reply.

The program is receiving 8014 bytes of data every 25ms. What I would like is a thread that receives the data and puts it into a buffer. Another thread will process the data from the buffer.

In C++ you can declare a buffer and then use pointers to point to the position in the buffer where the received data should be stored and then have another pointer that reads from the same buffer. I can achieve a similar thing in C# if I receive data in 1 byte array and then copy it into the buffer array. But this copy process takes up processing time.

If I use stackalloc the buffer array will remain for the duration of the program, and will not be moved each time the garbage collector kicks in. I am trying to keep unnecessary data movement to a minimum.

Answer 5

I suppose you're making your stackalloc on your Main method. If not, it will be reclaimed as soon as the method returns.

If all you want is to fix the byte array in place, use the 'fixed' keyword.

byte[] buffer = new byte[8014];
fixed(byte* p = buffer)
{
    // do whatever you want with the buffer array here, it won't move
}

Answer 6

Thanks again Louis for your reply.

The program is receiving 8014 bytes of data every 25ms. What I would like is a thread that receives the data and puts it into a buffer. Another thread will process the data from the buffer.

In C++ you can declare a buffer and then use pointers to point to the position in the buffer where the received data should be stored and then have another pointer that reads from the same buffer. I can achieve a similar thing in C# if I receive data in 1 byte array and then copy it into the buffer array. But this copy process takes up processing time.

If I use stackalloc the buffer array will remain for the duration of the program, and will not be moved each time the garbage collector kicks in. I am trying to keep unnecessary data movement to a minimum.

This all sounds a little unclear.

It seems that one thread must receive 8014 byte blocks from the network and put that block into a shared buffer.

Then another thread must access the buffer at some arbitrary slot and process (some of ?) that data block.

If so, then surely the network read thread will always need to write to the entire 8014 byte buffer, from the start of the buffer, yes?

It also seems that this "other" thread can simply access the buffer freely and arbitrarily (assuming we know it has a complete, full 8014 byte block) yes?

If you use a fixed size 8014 byte buffer and you always receive a fixed size 8014 byte block, then there is no need for any flexibility, because if you "write" data at any position other than the start of the buffer, you will (attempt to) overwrite memory surely?

You need to explain a little more.

Does the 8014 byte block represent some collection of smaller messages/items?

Do you need to accumulate a multiple arbitrary number of instances of the 8014 byte message?

I have seen and designed stuff like this many times, but I am not sure I follow you 100%.

Are you perhaps seeking some form of circular buffer that supports simultaneous read/write operations?

Have you looked into using a Stream?

Hugh

HI, have a look at this page, it may help you.

---

Hugh Moran - http://www.morantex.com

Answer 7

Hi,

I will try and explain a little better.

I am receiving 8014 bytes of data from the network every 25ms.

The buffer I am going to use will be 5 x 8014 bytes (40070 bytes) allowing for 5 chunks of data to be stored. Data received on the network will be put into the buffer. I will have 2 pointers 1 will point to the data in position of the buffer and the other will point to the data out position of the buffer. I will also have another buffer bool dataAvailable[5] which will be set to true when data is available and set to false when data has been read.

Data will be put into the buffer when the onReceive is called.

A thread will then check the dataAvailable buffer to see if data is available to be processed. If it is the 8014 bytes from the data out pointer will be processed.

Example

Data Received thread:

1)i = 0
2)Read data into buffer at data in pointer.
3)Set dataAvailale[i] to True.
3)i + 1 (if i = 5 then i = 0)
4)data in pointer + 8014 (if i = 0 reset data in pointer to start of buffer)
5) goto 2

Read Data buffer:

1) x = 0
2) if dataAvailable[x] = true else sleep and to to 2.
3) Process data from buffer at data out pointer to data out pointer + 8014.
4) Set dataAvailable[x] to False
5) x + 1 (if x = 5 then x = 0)
6)data out pointer + 8014 (if x = 0 reset data out pointer to start of buffer)
7) goto 2

The problem I have is: I need to find a way to pass a pointer instead of a byte array to the UDP beginreceivefrom method. If I can do this I think I can do the rest. I hate to say this, but I can do this easyily in C++.

The other way I can do it is by receiving the data into the byte array, then copy it into the larger buffer (specifying the start position and length). But copying arrays is time consuming and I can't afford to miss any data from the network.

Is there another way to do this in C#.

Really appreciate the help so far!

Answer 8

I still don't see why you'd need pointers.
You just need 5 arrays of 8014 bytes each.

byte[][] buffer = new byte[5][];
for(int i=0; i<5; i++)
    buffer[i]=new byte[8014];

Data received thread

1)i = 0
2)Read data into buffer[i]
3)Set dataAvailale[i] to True.
3)i + 1 (if i = 5 then i = 0)
5) goto 2

Read data buffer

1) x = 0
2) if dataAvailable[x] = true else sleep and to to 2.
3) Process data from buffer[x][0] to buffer[x][8013]
4) Set dataAvailable[x] to False
5) x + 1 (if x = 5 then x = 0)
7) goto 2

Answer 9

I had my pointers hat on, I hadn't though of that! I will give that a try.

Only problem is it will be affected by the garbage collector, but I gues I will have to live with that.

Thanks

Answer 10

As I said previously, if you absolutely need to have it fixed in place, you can use the 'fixed' keyword.
But since it is 5 arrays, I think you need to fix each one:

fixed(byte* p0 = buffer[0], p1 = buffer[1], p2 = buffer[2], p3 = buffer[3], p4 = buffer[4])
{
    // do whatever you want with the buffer array here, it won't move
}

Answer 11

I had my pointers hat on, I hadn't though of that! I will give that a try.

Only problem is it will be affected by the garbage collector, but I gues I will have to live with that.

Thanks

OK

Well see the link at the end of my first comment (above).

You need what is called a "Ring Buffer" widely used in comms protocol coding, that link has some C# classes that do all you need.

These buffers allow continuous asynchronous reading/writing to take place because internall the ring buffer maintains two "pointers".

Hugh

---

Hugh Moran - http://www.morantex.com