none
SerialPort - 101

    General discussion

  • Much of the difficulty I see here, and in other forums, surrounding the use of Windows serial port is centered around an assumption.

    That assumption is that the serial port receives data in chunks that are conveniently packaged in the format we want.  If you are talking about terminal stuff that is true, they are called lines and it is easy.  But when that is not the case panic sets in.

    The second thing I see is people fighting the DataReceived Event Handler.  My advice is don’t.  Treat it like a spoiled child that is never going to learn and just give it whatever it wants, whenever it wants.

    Lastly, a byte can contain 256 values.  The standard encoder for the serial port is for 7-bit data.  If you have 8 bit data you need to change the encoding.  I use

    System.Text.Encoding.GetEncoding("Windows-1252") always and have not had any problems.

    edit - The encoder is only used when working with characters and strings.  The proper 1-1 encoder should be

    From MSDN BLOG "The only encoding that converts all characters with a value 0-255 to a single byte with the corresponding value and vice versa when converting bytes to characters is the “Western European (ISO)”(28591) encoding."

    end edit

     I wanted fast code, and to write the code in such a way that it could be used for any application.

     I have successfully sent, received, processed, and updated the UI at close to 1Mbps (USB Serial Converter) using the techniques I will describe.  My test bed is a USB SerialConverter to breakout box with Pins 2 & 3 hot wired.

     **DataReceived Event Handler**

     Mine is short, sweet, simple.  I read the number of bytes available (.BytesToRead) into a buffer of bytes of the correct size.  I store that buffer of bytes into a queue. 

     Depending on how the UI is being driven I then raise an event, start a delegate, or do nothing if it is timer or user interaction based.

     That is it!  Any time it wants to fire, it can.

    edit - In a thread on StackOverflow one of the heavyweight posters (+55,000 points) contended that this event will fire repeatedly if the data is not read.  That is incorrect.  This event fires only when new data is received.

     **So how does the data get to the user…**

     Ask yourself what are the possible scenarios of how you might want data?  You might want:

         -one byte

        -a number of bytes

        -a character

        -a string

        -a string delimited by a character  (like lines)

        -etc.

     And what if you want 2 bytes, then 4 characters, then a byte, and then a variable length line?

     At this point my DataReceived Event Handler has placed any data received into a queue.  Here is an example. I have a GPS receiver that sends NMEA sentences (lines ending with CRLF). This is an example of one of the sentences:

     $GPGSV,3,1,11,10,75,053,29,29,52,311,32,24,50,298,30,02,39,073,30*77

     The serial ports DataReceived Event Handler might (usually does on my PC) fire several times with chunks of that data.  The queue might look like this:

         Queue entry – data

        1   $

        2   GPGSV,3,1,11,10

        3   ,75,053,29,29,52,311,

        4   32,24,50,298,30,02,39,073,30*77

     The approach I took, with my limited knowledge of the programming tool at the time, was to write several routines that pieced the data back together.  The common routine for all of those created a user buffer by removing entries from the queue and piecing them back together.  The other routines then access the user buffer when they need data, and in the case of string based functions, converts it.  If there is not enough data to satisfy the request the routines return empty strings or zero length byte arrays.

    Lets say I used a Timer control to read, process the sentences, and update the UI from my GPS receiver.  In my Timer routine I have a function call to my version of readline.  The first few times it is called it might return empty strings because the entire sentence has not been received.  Eventually all of the data is in and I get the entire sentence back, because it is actually all there.

        At time 1

        Queue entry – data

        1   $

        2   GPGSV,3,1,11,10

         User buffer – empty

        At time 2

        Queue entry – data

        1    ,75,053,29,29,52,311,

        2   32,24,50,298,30,02,39,073,30*77

        User buffer – $ GPGSV,3,1,11,10

       

        At time 3

        Queue empty   

        User buffer – $ GPGSV,3,1,11,10,75,053,29,29,52,311, 32,24,50,298,30,02,39,073,30*77

     The only imperative is that when you have a device that sends data all the time is that you process it in a timely manner.  If not, as you can imagine, you run out of memory. 

     

    During my testing of the USB to SerialPort converter I was surprised to find that it was capable of speeds near 1Mbps and I made the mistake of not doing that.  I was using a user interface driven (click a button, see the data) routine that read all of the data and displayed the last chunk, when I received a high priority interrupt from Honey.  You know the one, take the trash out, change the light bulb, etc.  I left the test bed running and I had received an out of memory exception while servicing the Honey interrupt.

    What prompted me to write this was something I read in another thread.  The poster had a problem because he said the device he was interfacing with only provided a 1 ms. gap in the data as its protocol.  As it turned out the protocol was a delimiter, followed by a length byte, followed by the data.  Then the problem was that that wasn’t enough to know that you actually had good data.  The posters point was this:

        message_delimiter - 1 byte

        4 - length - 1 byte

        (4 data bytes) - 4 bytes

    But if you look at it this way, a **valid** message is always bounded by the message_delimiter:

        message_delimiter - 1 byte

        4 - length - 1 byte

        (4 data bytes) - 4 bytes

        message_delimiter - 1 byte

     

    Then you can safely process the data.


     


    Looking for work - Zip 65101 http://www.vbforums.com/showthread.php?t=552774
    • Edited by dbasnett Sunday, October 17, 2010 2:53 PM
    Friday, April 16, 2010 1:07 PM

All replies

  • In another forum* it was pointed out that the protocol (delimiter,length,data,delimiter,length,data,etc) could have an error that would go undetected.  Looking at a frame of data with 01 as the delimiter:

    01 04 11 11 01 ff - what was sent- 4 data bytes with the delimiter as part of the data

    01 02 11 11 01 ff - what was received (1 bit error) - a valid 2 data byte frame

    Eventually, if you implement the protocol correctly an error will get detected.  Unfortuantely that first two byte frame was accepted.  The example I gave was another posters problem, and he said that was what the manufacturer's protocol was. ref.  http://stackoverflow.com/questions/2647052/timing-of-reads-from-serial-port-on-windows

    My opinion is that it is about the poorest protocol possible.

    If you are building a product THE simplest you should consider is:

    delimiter, length, data, datachecksum, delimiter

    The simplest I would do would be:

    delimiter, length, data, datachecksum, enddelimiter

     

    *thanks to markh44@stackoverflow


    Looking for work - Zip 65101 http://www.vbforums.com/showthread.php?t=552774
    Saturday, April 17, 2010 12:21 PM
  • A very crude sample program that uses a timer to process the data received. My first choice is always a timer.

     

    Public Class Form1

        'this example was written with a SerialPort LoopBack
        'which means that everything sent is received

        Private Sub Button1_Click(ByVal sender As System.Object, _
                                  ByVal e As System.EventArgs) Handles Button1.Click

            Button1.Enabled = False
            Try 'very general error catching for this sample only
                If SerialPort1.IsOpen Then 'if the serial port is open close it
                    SerialPort1.Close()
                    Debug.WriteLine("Closed")
                Else 'otherwise open it
                    SerialPort1.Encoding = System.Text.Encoding.GetEncoding(28591) 'one-to-one encoding
                    SerialPort1.ReceivedBytesThreshold = 1 'the default <<<<<
                    SerialPort1.BaudRate = 9600
                    SerialPort1.DtrEnable = True
                    'etc.  other settings if needed
                    SerialPort1.PortName = "COM6" ' the port on my pc that has loop back
                    SerialPort1.Open()
                    Dim t As New Threading.Thread(AddressOf SendSome) 'used to simulate send, since this is both send / receive
                    t.IsBackground = True
                    t.Start()
                    Debug.WriteLine("Opened")
                End If
            Catch ex As Exception
                Debug.WriteLine("OC " & ex.ToString)
            End Try
            Button1.Enabled = True
        End Sub

        Dim PRNG As New Random
        Const sendLimit As Integer = 256 'max characters sent per test

        Private Sub SendSome()
            Dim sendCT As Integer = 0 'count of bytes sent
            Do
                Dim ct As Integer = PRNG.Next(1, 33) 'random buffer size
                sendCT += ct 'count chars sent
                Dim b(ct - 1) As Byte 'buffer
                PRNG.NextBytes(b) 'random data
                Try 'very general error catching for this sample only
                    SerialPort1.Write(b, 0, ct) 'write buffer to serial port
                Catch ex As Exception
                    Debug.WriteLine("S " & ex.ToString)
                End Try
                Threading.Thread.Sleep(PRNG.Next(0, 51)) 'random delay between writes to simulate "real-world"
            Loop While sendCT < sendLimit
            Debug.WriteLine("SendDone")
        End Sub

        Private Sub SerialPort1_DataReceived(ByVal sender As System.Object, _
                                             ByVal e As System.IO.Ports.SerialDataReceivedEventArgs) _
                                         Handles SerialPort1.DataReceived

            If SerialPort1.BytesToRead = 0 Then Exit Sub
            If Threading.Monitor.TryEnter(qLock) Then
                Dim br As Integer = SerialPort1.BytesToRead 'get number of bytes available
                Dim b(br - 1) As Byte 'set receive buffer
                br = SerialPort1.Read(b, 0, br) 'read bytes available
                Dim bl As New List(Of Byte) 'add bytes just read to queue
                bl.AddRange(b)
                q.Enqueue(bl)
                Threading.Monitor.Exit(qLock)
            End If
        End Sub

        Dim q As New Queue(Of List(Of Byte))
        Dim qLock As New Object

        Private Sub Timer1_Tick(ByVal sender As System.Object, _
                                ByVal e As System.EventArgs) Handles Timer1.Tick

            If Threading.Interlocked.Read(q.Count) = 0 Then Exit Sub
            If Threading.Monitor.TryEnter(qLock) Then
                Debug.Write(q.Count.ToString & " TMR ")
                Dim bl As List(Of Byte)
                bl = q.Dequeue
                Debug.WriteLine(bl.Count.ToString)
                Threading.Monitor.Exit(qLock)
            End If

        End Sub

        Private Sub Form1_Shown(ByVal sender As Object, _
                                ByVal e As System.EventArgs) Handles Me.Shown
            Timer1.Interval = 50 'watch for received data
            Timer1.Start()
        End Sub
    End Class


    Subnet Calculator / Planner       Serial Port       Random
    • Edited by dbasnett Sunday, January 30, 2011 6:41 PM
    Thursday, August 19, 2010 5:25 PM
  • A loop back can be created by placing a jumper between pins 2 & 3.
    Subnet Calculator / Planner      Serial Port      Random
    Thursday, August 19, 2010 5:32 PM
  • I gave up a long time ago helping people with serial port issues, because so many get it wrong - precisely because of your first statement.

    Usually, I treat every bit of data received as a byte, and have never had problems with encoding; these bytes get thrown into a circular buffer and then everything up from there is serial port (data acquisition) independent - that is, all data interpretation functions are valid for Ethernet IP, Serial, Shared Memory and Reflective Memory.

    Of all the underlying connection mechanisms, serial is the most flawed - it's slow and (relatively) error prone. To that end, I usually treat a serial message as the equivalent of UDP: with handshaking built into the user protocol. True, there is often very little choice when connecting to a third party device.

    The most serious flaw that programmers make, in my opinion, is trying to feed the serial port data directly up to the user interface. If you consider a serial port as ones and zeroes, unless the user is expecting to see a bunch of ones and zeroes, stop feeding it to your form! Break down the application into 3 levels:

    1. Data acquisition - the existence of a serial port is not known from this point forward;

    2. Data Manipulation - turn those bits into real world data (temperatures, speeds, etc.)

    3. Present the data - bargraphs, whirly-gigs, etc.

    While the UI is used to parametrize the data acquisition device (in this case, a serial port) that's where its involvement with the port should end.

    As a side note, I usually have a 4 byte delimiter, as it fits nicely into a 32 bit integer, represents a floating point number, as well as represents 4 ASCII (single byte) readable characters.

    Essentially, I follow Black Box Programming: each component adds value to the system as a whole, has inputs and outputs, the inner workings is irellavant to the other connected Black Boxes. A Black Box may contain one or more other black boxes.

    In this way, the 'serial port black box' has a serial port physical connection, a few parametrizations, a bytes in and bytes out. Once connected, all you care about is that you receive bytes and can send bytes (no encoding issues, no byte swapping issues, no hand shaking issues, no threading issues).


    Stephen J Whiteley
    Thursday, August 19, 2010 6:54 PM
  • I agree.  I no longer jump on SerialPort questions because of the reasons you pointed out, plus the posters often don't understand the SerialPort.  This thread is my catch all.

    Usually Dick or Carsten jumps in and redirects poters else where.

    How are you doing?  For some time I didn't notice you posting.  Hope all is well.


    Subnet Calculator / Planner      Serial Port      Random
    Thursday, August 19, 2010 7:00 PM
  • I agree.  I no longer jump on SerialPort questions because of the reasons you pointed out, plus the posters often don't understand the SerialPort.  This thread is my catch all.

    Usually Dick or Carsten jumps in and redirects poters else where.

    How are you doing?  For some time I didn't notice you posting.  Hope all is well.


    Subnet Calculator / Planner       Serial Port       Random

    Only so much time for forum stuff. If you go to web sites that actually have information and discussions that go beyond the attention span of your average internet user (approximately 27 seconds or 140 characters, whichever is less) there's not much time left to help out those that fit into the aforementioned group ;)

    Stephen J Whiteley
    Friday, August 20, 2010 2:01 PM
  • Your and MSDN's encoding won't work for non-english windows. This works for me:
    SerialPort1.Encoding = System.Text.Encoding.Default

    Tested with xbee module and windows 7 with central european encoding as default.
    Friday, July 06, 2012 2:12 AM
  • The only encoding that 'works' for text transmission is the encoding that matches that used at the other end of the serial transmission pair.  The only rule that you can state that is completely reliable is that things will probably work correctly if the encoding is the same at both ends.

    The communication might work if the encoding is different at the two ends, depending on the data size, what the two different encodings are, the actual data that is being transmitted, and the definition of what constitutes success.  Text can be successfully transmitted with very different encodings if the content of that text is limited to a subset of available characters.  Text can be successfully transmitted with different encodings if that conversion from one to the other is actually what was required for that particular transmission (which is frequently the case).   That is what lies behind the vast number of different (and sometimes conflicting) 'rules' about how to get reliable text transmission.

    To get an exact duplicate of what was sent, use bytes not text.  Then, if necessary, work out what decoding is needed for those bytes to meet the requirements of the particular circumstance.

    Friday, July 06, 2012 3:18 AM
  • I am a software engineer myself for the last 13 years and have dealt with many serial communication issues over the years. However, one shouldn't assume right away that when an individual posts question(s) for help with their programming issues that they have not done their homework. Often times they have and they usually use the Internet as the last resort like me for help. I am pretty smart when it comes to serial communication, but even I needed help few times.

    This one time I was harassed by a fellow programmer for posting a question related to serial communication - "inability to program" or "Crap." At that time, I was developing software under Windows 7 using Delphi RAD XE IDE with AsyncPro VCL component for serial communication. It just so happened that my IDE altered my system timer resolution, because I was using an event based timer. That caused my application's serial communication to fail - with IDE running worked fine but without it failed. This I never encountered ever before. So, being confused I posted a question on Stackoverflow.com.

    http://stackoverflow.com/questions/6139026/serial-communication-rts-and-windows-7

    One programmer did help me. In fact, her answer didn't even directly deal with the serial com tool itself. Her answer helped me a lot and solve my problem. Thanks to her.

    So, lets not stereotype everyone who asks or posts question(s) about serial communication online as being dumb or someone who assumes everything.

    Oh by the way, dbassnett postings are full of great information about serial communication. Thank you.


    Monday, October 15, 2012 3:19 PM