none
NetworkStream.BeginRead. Streaming images/video. Dealing with not getting entire image in one read. RRS feed

  • Question

  • I know, don't reinvent the wheel, use an existing library. I'm in a situation where I need to do this without a 3rd party library due to various reasons and I need to engineer this up myself. 

    I'm creating a video client server solution for work.  The server allows for a client connection and will send one image at a time to a client this seems to operate just fine. 

    My assumption was that for every Write from the server, I would have a 1 to 1 read on the client which I would then turn the bytes into an image and display them in a form control. 

    However, that's not the case.  The problem is on the client side when calling NetworkStream.BeginRead(...) because I'm not guaranteed to receive the full image bytes in one read. Which really complicates things because the callback specified in BeginRead operates on a different thread.  

    Is there a way to call BeginRead and and get the entire image from the server in one shot before the application loops back around and calls BeginRead again? Because ultimately the second BeginRead would contain bytes from the previous image AND the next image being sent from the server. 

    OR... Is this just the nature of the beast and I have to constantly, from the callback, fill some buffer with the byte data, then have some other process that knows how to pluck each image out of that buffer, all the while dealing with threads and locking the data...

    Any guidance here would be awesome! 

    Thanks.


    Rick

    Friday, February 8, 2019 6:50 PM

All replies

  • No, BeginRead is never guaranteed to return all the data in a single read. There are too many factors involved such that this would even be remotely possible. There are several different solutions to this problem however.

    The simplest solution is to wrap NetworkStream in a BinaryReader. Then use ReadBytes to read the exact amount of data you need. That method will loop until the specified # of bytes are read. Unfortunately this would require a change in your architecture because: a) you need to know the size of the image in advance and b) it is sync.

    For A this is a common problem that people writing TCP protocols run into. The solution is almost always to ensure that you're passing messages (not just data) between the client and server. The message has a well-defined header that specifies (amongst other things) the size of the payload to expect. The client/server can then read the fixed-size header, determine what message it is and how big. Then use ReadBytes to read the remaining data in. After all the data is read (ignoring streaming messages) then it can be processed. If all the data doesn't arrive then there was a network issue and the message should be ignored.

    For B you can make the call async (instead of BeginRead) and accomplish the same thing. This will ripple up your code but is preferable to using the older BeginRead stuff.

    As for the multi thread thing it is generally common to have each client either have a dedicated thread (if you don't have too many) or have each client request be handled by a worker thread. In their case the thread will have its own buffer to write to so there shouldn't be any threading issues. Once the request has been handled the buffer is no longer needed. If you have yet another thread that is trying to process the various requests from multiple clients then each request thread, after it has retrieved all the data should add it to a shared worker queue that the processing thread listens on. A ReaderWriterLock is designed for this kind of architecture and allows the processing thread to sleep when there is nothing to do. And there can be any # of worker threads adding (completed) messages to the queue for the processor to run against.


    Michael Taylor http://www.michaeltaylorp3.net

    Friday, February 8, 2019 10:09 PM
    Moderator
  • Hi Michael... Thanks for the info.  Let me describe what I have so far.

    I have a server that is already working.  For every image pulled from the web cam, it will create a new byte array that contains the size of the image AND the image itself... SO, the first 4 bytes represent the length of the image. Then it gets sent using NetworkStream.Send( …. )

    This all works just fine. My issue is on the client side. My brain just gets scrambled and I'm having a hard time trying to approach this one small part at a time. 


    In general... On the client side, is it better to call NetworkStream.Read(…)  or BeginRead()
    It seems like it would be easier to use "Read()".

    Assuming this would be in a loop 

    1) Call NetworkStream.Read( … )
    2) Pull off first 4 bytes to get the size of the image. Store it into imgSize
    3) Put remaining bytes into new buffer.
    4) Is new buffer.Length >= imgSize.
    5) Have full image, do something with it.
    6) Loop back around calling Read again, this time only requesting the delta between how much we got the first time and image size.

    There would be a little more to it than that... But... 

    I have not worked on this code in about a month, I got side tracked a while back. It seemed like I tried this approach and I ran into blocking issues.  Seems as though when you call Read, it blocked the server for some reason until I did a Write back to the server.


    Rick

    Friday, February 8, 2019 11:30 PM
  • "In general... On the client side, is it better to call NetworkStream.Read(…)  or BeginRead()"

    Read is blocking and BeginRead is not. If you need non blocking then consider using Task method with Read call.

    Since the first bytes are the size then I would wrap the NetworkStream with BinaryReader. Then make the 2 calls to it.

    using (var reader = new BinaryReader(stream))
    {
       //Read the size
       var size = reader.ReadInt32();
    
       //Read the data
       if (size > 0)
       {
          var data = reader.ReadBytes(size);
       };
    };

    Some caveats to this code. Readers will close the stream when they are disposed. For network streams then tends not to be an issue because of how the socket works but if it causes problems then you'll have to use the constructor overload to tell it not to.

    Secondly ReadBytes will allocate an array each time so if you're calling this in a tight loop then you can eat up memory pretty fast. If this is your case then you might do better to allocate a single array buffer that you can reuse. Since you mentioned a webcam I assume the image sizes are pretty static so a single buffer would be more efficient. However you can't use ReadBytes anymore. The reader does have a Read method that can be used but at that point it is probably just easier to stick with the stream.

    //Not verified ...
    var size = GetImageSize();
    
    var offset = 0;
    var remaining = size;
    while (read < size)
    {
       var read = stream.Read(buffer, offset, remaining);
       remaining -= read;
       offset += read;
    };
    
    
    You could optimize this and there may be an off by one error but that is the gist of the code.
     


    Michael Taylor http://www.michaeltaylorp3.net

    • Proposed as answer by Stanly Fan Wednesday, February 13, 2019 2:46 AM
    Friday, February 8, 2019 11:54 PM
    Moderator
  • Hi Michael...  As mentioned I have already been working on this code and was making modifications between our posts.... Here is what I have changed so far.... Or the direction I'm going with Read() and blocking... I had already just written a way to Read()… Then extract the image size, and create a new image buffer without the size tacked on.

    This would eventually have checking to see if I have the full image like your  while loop... Just haven't gotten that far.

            public void StartVideoStream()
            {
                if( _networkStream == null )
                    throw new Exception( "The network stream is null" );
                byte[] buff = new byte[RECEIVE_SIZE];
                int numRead = 0;
    
                numRead = _networkStream.Read( buff, 0, RECEIVE_SIZE );
    
                int imgSize = GetImageSize( buff );
                byte[] image = TrimOffImageSize( buff );
            }

    Extract the size of the incoming image.

            private int GetImageSize( byte[] imgBuff )
            {
                byte[] size = new byte[ 4 ];
                for( int x = 0; x <= 4; x++ )
                    size[ x ] = imgBuff[ x ];
                return BitConverter.ToInt32( size, 0 );
            }


    Create a new byte array that only has the image data

            private byte[] TrimOffImageSize( byte[] imgBuff )
            {
                int newLen = imgBuff.Length - 4;
                byte[] newBuff = new byte[ newLen];
                Array.Copy( imgBuff, 4, newBuff, 0, newLen );
                return newBuff;
            }

    Would you say that this accomplishes the same thing you'd done with the Binary Reader?  Is there an advantage to using your method over what I already wrote?

    Also, I was thinking about making the call StartVideoStream an async method.. So it would become 

    public Task StartVideoStreamAsync( )
    {...}
    This way from the GUI I could just call it and await it.  Ultimatly, I'd like my client class to have an event that can publish a full image back to the calling GUI.  The GUI would subscribe to this event and display the new image/video stream.


    Rick



    • Edited by m00n Saturday, February 9, 2019 12:12 AM
    Saturday, February 9, 2019 12:08 AM
  • If your code is working then feel free to use it but it is very inefficient on memory. You are allocating 3 byte arrays per image - 1 for the read buffer, 1 to store the temporary length and then another to trim it off. It is far more efficient to simply read the 4 bytes you need for the length, then allocate a single array (if you don't want to reuse the buffer). As your receive_size value gets larger it'll become more inefficient. 

    Your code also doesn't handle the fact that the image won't entire get transmitted the first read. You still need a loop (or maybe you just left it off). It also doesn't handle the (very rare) case of the first read not even reading 4 bytes. To be honest I'd use BinaryReader. It is 2 lines of code (for the reads), handles the buffering and is more efficient than you'd probably write yourself.


    Michael Taylor http://www.michaeltaylorp3.net

    Saturday, February 9, 2019 8:14 AM
    Moderator
  • Do you have to throttle down reading from a network stream?   I have a function that is being called every time System.Timers.Timer ticks which if I set to something low like 10milliseconds, my app will fail eventually.  But if I slow it down to say 30 millisecond intervals, my code seems to run ok. 

    I got to wondering IF the timer was executing faster than the function was executing, basically, two different tick events executing the same function which would cause reading the stream from one tick, to interfere with the next tick.  I did notice that this was the case in a few iterations, however, despite blocking the code from running if it wasn't fully complete, I was still running into the same issues.

    The issue seems to be that I'm not always getting the proper imgSize when pulling off the first 4 bytes of the stream.  Sometimes I would get numbers there were WAY too large or I would end up with negative numbers.  Almost as if after it had ran a little while, when trying to grab the image size, it was reading data from the old image... As if the stream had not repositioned itself fast enough from the reading the actual image bytes from the previous image. 

    The only way that I've been able to successfully get this to be reliable is to slow the timer down to around 30 milliseconds.   

    This code snippet shows the gist of trying to grab the image size, then pulling the actual image off the stream. There is some other code going on, but this is the real core of it all. 

    int imgSize     = 0;
    int offset      = 0;
    int read        = 0;
    int remaining   = 0;
    
    _networkStream.Read( _imgSizeBuff, 0, 4 );
    imgSize = BitConverter.ToInt32( _imgSizeBuff, 0 );
    
    while( read < imgSize )
    {
        //Task.Delay( 2 );
        read = _networkStream.Read( _destImageBuffer, offset, remaining );
        remaining -= read;
        offset += read;
    }
    


    Rick

    Saturday, February 16, 2019 7:36 AM
  • "I got to wondering IF the timer was executing faster than the function was executing"

    When using a System.Timers.Timer the call is occurring on secondary threads so yes you can have reentrancy if a single call takes longer than the interval. 10ms is really fast so this is likely to happen given that you're making a network call. Streams are not thread safe so calling the same stream on different threads at the same time is going to cause problems like you're seeing.

    Because you're using System.Timers.Timer I would recommend you set the AutoReset property to false. The timer will fire once. In your handler, after you've done the work then call Start to restart it for the next interval. This will prevent reentrancy.

    Note that you cannot set the Interval property to an arbitrarily low value. Refer to the MSDN documentation on it. The resolution is limited to the clock resolution. In Win 7 that was 15ms. So you're 10ms timer isn't going to run that fast. Honestly at that resolution you're trying to do real-time and that isn't going to work out. Use  more realistic interval instead. 100ms is still going to be pretty fast given that you're making a network call. Ultimately you're stalled by the network speed and the remote server sending the data. Even at 100ms you're not going to be getting your data that fast most likely.


    Michael Taylor http://www.michaeltaylorp3.net

    Saturday, February 16, 2019 5:23 PM
    Moderator
  • Hi Michael.  I noticed that if I set the timer to 100,it's WAY too slow and makes the displaying of images look like it's running in slow motion.  I THINK I've found a sweet spot for simi live streaming at about 30.  I've set the timer's AutoReset to false in a function not posted here.  But you can see where I restart it in the snippet below. 

    Here is my code, it's working, but I know it's still probably going to need some tweaking.  What I'm not showing here is all the connecting to the network and all that.  Just the pulling of the images from the stream, then the event I fire that spits the image out to a Windows Form client. 

    Any feedback is most welcome!! :)

            /// <summary>
            /// Called by System.Timer.Timer on specified interval. Assumes network stream is connected
            /// </summary>
            /// <param name="source"></param>
            /// <param name="e"></param>
            private void FetchNextImage( Object source, System.Timers.ElapsedEventArgs e )
            {
                Debug.WriteLine( "IMAGE (" + _imageNumber + ")" );
                if( !_networkStream.CanRead )
                    return;
    
                // Pulls the next image off the network stream
                if( !FillBuffer( ) )
                    return;
    
                if( _destImageBuffer != null )
                {
                    // Send new image out on the new image event handler so a client can do something with it
                    _imageArg.ImageBuffer = _destImageBuffer;
                    OnNewImage?.Invoke( this, _imageArg );
                    _imageNumber++;
                }
    
                // Tell timer to run. 
                _FPSTimer.Start( );
            }
    
    
            /// <summary>
            /// Pulls the next image off the network stream.
            /// </summary>
            /// <returns>True if successful, false otherwise.</returns>
            private bool FillBuffer( )
            {
                int imgSize     = 0;
                int offset      = 0;
                int read        = 0;
                int remaining   = 0;
    
                // Each image sent from server is packed with first 4 bytes representing the size of the image.
                // Pull the image size off.
                _networkStream.Read( _imgSizeBuff, 0, 4 );
                imgSize = BitConverter.ToInt32( _imgSizeBuff, 0 );
    
                if( imgSize <= 0 )
                    return false;
    
                // Clear out the buffer from previous read and resize it according to the new image size.
                if( _destImageBuffer != null )
                    Array.Clear( _destImageBuffer, 0, _destImageBuffer.Length );
                Array.Resize( ref _destImageBuffer, imgSize );
    
                // Debugging print statement
                PrintOut( imgSize );
    
                // We know the size of the next image, pull exactly that many bytes off the stream.
                remaining = imgSize;
                while( read < imgSize )
                {
                    Task.Delay( 1 );
                    read        = _networkStream.Read( _destImageBuffer, offset, remaining );
                    remaining   -= read;
                    offset      += read;
                }
    
                // Succesfully got the image into the classes global image buffer
                return true;
            }
    
    
            private void PrintOut( int imgSize )
            {
                string s = "";
                for( int i = 0; i < _imgSizeBuff.Length; i++ )
                    s += "[" + _imgSizeBuff[ i ] + "]";
    
                Console.WriteLine( "\t\t\tIMAGE NUMBER " + _imageNumber +
                                    "\n\t\t\tIMAGE SIZE = " + imgSize +
                                    "\n\t\t\tIMAGE SIZE BYTE ARRAY" + s );
            }


    Rick


    • Edited by m00n Saturday, February 16, 2019 7:42 PM
    Saturday, February 16, 2019 7:40 PM