locked
Synchronising Serial Communication RRS feed

  • Question

  • I have a questiong regarding Serial Communication. Im trying to set up a communication between a computer and a machine. Sending data is not a problem because the machine has  already an algorithm to check and validate the data received. But on my side, I have to develop this algorithm to be sure that I get sensefull information.

    I receive a byte that tells me an id of the type of operation, then a byte which tells me the number of following bytes of data to receive and two bytes for a special type of checksum.

    If for some reason I miss one character, my computer can think that one of the data bytes is the size of the data. Then I will read extra bytes as data, that will include operations IDs and size of data from other data packets. It can also happens that it thinks that the size is small, and start reading a new packet before the other one is complete.

    I know that the checksum has the purpose of avoding this problem, but if the supposed size is big, lets say 256 bytes, then I will miss a lot of information during the process before the checksum takes place.

    Which are the best methods to synchronise Serial Communication? I have a handle of my programs which calls a procedure everytime the serial communication has received a byte. Is there anyway to make a handle everytime the serial communication receives a line? that means, a full packet and then a short pause?

    Saturday, May 20, 2006 10:05 AM

Answers

  • Arturo,
    the best thing to do would be to change the protocol to include start and stop symbols, but I guess you don't have a chance to change the protocol. Actually, unless your protocol is much richer than you explained, you might run in situations in which you go out of sync and never recover.

    Back to your question, without knowing what is your baudrate, and what is the size of the "short pause" you would like to detect, it's pretty difficult to find an answer. Anyway, assuming you are running at 9600, and you expect a pause the size of one symbol...

    1) The ReadTimeout resembles what you have in mind, but it won't work. The resolution you have is 1ms, which is not enough since one character takes about 1ms to be transmitted, and that would be too close for comfort.

    2) You could use a System.Diagnostics.Stopwatch. That is fairly high resolution (a few hundred nanoseconds, if I am not much mistaken). The idea would be to start the stopwatch after receiving a byte, and then checking, at the next event, how much time have elapsed. Unfortunately this is not very reliable either: if your computer gets too busy you might detect pauses that are not real, or miss to detect them as characters line up in the FIFO.

    So, method #2 might work as a "hint": if you think you just detected a pause, keep track of where you detected it. If the checksum doesn't match at that point, try to see if there was a packet starting after what you think was a pause.

    Probably not the answer you hoped for, but it's the best I can come up with.
    HTH
    --mc

     

    Sunday, May 21, 2006 5:12 AM

All replies

  • Arturo,
    the best thing to do would be to change the protocol to include start and stop symbols, but I guess you don't have a chance to change the protocol. Actually, unless your protocol is much richer than you explained, you might run in situations in which you go out of sync and never recover.

    Back to your question, without knowing what is your baudrate, and what is the size of the "short pause" you would like to detect, it's pretty difficult to find an answer. Anyway, assuming you are running at 9600, and you expect a pause the size of one symbol...

    1) The ReadTimeout resembles what you have in mind, but it won't work. The resolution you have is 1ms, which is not enough since one character takes about 1ms to be transmitted, and that would be too close for comfort.

    2) You could use a System.Diagnostics.Stopwatch. That is fairly high resolution (a few hundred nanoseconds, if I am not much mistaken). The idea would be to start the stopwatch after receiving a byte, and then checking, at the next event, how much time have elapsed. Unfortunately this is not very reliable either: if your computer gets too busy you might detect pauses that are not real, or miss to detect them as characters line up in the FIFO.

    So, method #2 might work as a "hint": if you think you just detected a pause, keep track of where you detected it. If the checksum doesn't match at that point, try to see if there was a packet starting after what you think was a pause.

    Probably not the answer you hoped for, but it's the best I can come up with.
    HTH
    --mc

     

    Sunday, May 21, 2006 5:12 AM
  • Hi,

    I've been working on the same problem for a while now.  I don't have a solution to your problem but I have learned this.  If you are using the dotNET Framework 2.0 (Visual Studio 2005) you can use the System.IO.Ports.SerialPort Classs to access serial ports.  This class comes with a "DataReceived" Event which detects any activity on the serial port.  What's been tricky for me is that this event is raised on a separate background thread from which you may not access any form controls.  This link helped me figure out how to get my data to another thread from which I can access form controls.

    http://msdn2.microsoft.com/en-us/library/ms171728.aspx

    Now the problem I am having is synchronizing the 2 threads.  Sometimes I pull the same data off my buffer twice and sometimes the data gets overwritten before I get a chance to get it.  I think one solution is to use the thread.join method to force one thread to wait for the other to finish but I'm still working on that.  The EventHandler that issues the SerialDataReceivedEventHandler doesn't have a join method.  I could also try a Systems.Collections.Generic.Queue which is a FIFO. 

    Thursday, August 3, 2006 3:44 PM
  • Hi I write many cnc and machine sofware interfaces which often have to communicate via RS232 sometimes by RF which is very tricky so I can help you somewhat here.

    Any type of serial communication is tricky, especially when the programmer at the machine end has been a little lazy as in your scenario whereby they do not send special beginning and end characters with the message packet, so if you cannot change this we have to analyse the packet for some unique and identifiable bytes that can show where we are in the packet and avoid loss of synch.

    So you say the first byte is always an operation ID, is the value of this byte the same each time or has it a range of values?.

    Scenario 1. The Operation ID is always the same:-

    All you have to do is test each byte in the read buffer one at a time until you hit a match to this value. Then begin a validation test by reading the rest of the packet against the protocol you mentioned, upon reaching the checksum validate this test packet as authentic or false. If Authentic then DELETE all the bytes before the authentic packet as well as all the bytes in the packet, this will help to keep the buffer memory under control and avoid any overflow conditions as well as ensuring your search loop starts from pretty much the beginning of the buffer again.....

    If for example in the above test, we decode the packet and it does NOT validate against the checksum then this means you were not at the beginning of a valid packet and the start byte was just a phantom, so do it all again but this time starting from AFTER the occurance of the phantom start byte. Delete all the bytes up to and including the phantom byte to keep the read buffer in check.

    Scenario 2. The start byte is not a constant but has a limited range of values....

    In this instance we need to search the buffer as above but this time checking for each permitted value of the start byte, if we get a match then as above check the rest of the packet, valdate it as good or bad. same as above but remembering to check against all the valid ID values. Usually machine communications are pretty simple affairs and control ID's have maybe less than 10 or so possible values so the search loop wont be too slow, however if the range is 0-255 then it has its work cut out a little.

    Scenario 3. The Operation ID byte can be any value:- i.e 0 - 255

    This is the trickiest kind of packet to receive and keep synchronised, and its pretty futile trying so assume there will be synch loss at some point and write recovery code to deal with it. Upon receiving and decoding a bad packet (i.e the checksum doesn't add up, assume synch has been lost and branch to a routine that tests each byte as if it were a Control ID, try to validate the rest of the packet as normal. If it doesn't validate then you are still out of synch, advance one byte in the buffer and try again. If this routine is fast enough it will eventually re-synchronise back into reading valid packets again. The time to re-synch will of course depend on the speed of the incoming data, the speed of your machine, the efficiency of the routine and the number of bytes in the test packet as well as obvious other mathmatical and probability delays, but given the speed of modern pc's and the relatively slow serial speed (usually 9600 baud or less in many cases) this should be the blink of ane eye. Once in back in synch, delete all the bytes in the buffer before and including this latest good packet again to keep the buffer nice and neat.

    hope this helps.

    UkMadprof

     

    Tuesday, August 8, 2006 2:10 PM
  •  

    Thanks for your post.  The data I'm receiving is setup in the standard RIF format which defines a group of bytes as a chunk.  The first 4 bytes are the ChunkID, the next 4 bytes hold the value of the number of remaining bytes after these first 8 have been read.  I used your scenario 1 with the chunk ID but still had problems because the 2 threads were not sychronized.  So I did away with the SerialPortsDataReceivedEventHandler and replaced it with an ordinary thread.  Then after starting my ordinary thread I used the thread.join method to force that thread to wait until the other is done.  That helped but didn't fix everything.  Then I realized that my SerialPort Input buffer was filling up and then writing over bytes which had not yet been read.  Increasing the SerialPort input buffer size not only fixed the problem, it eliminated the need to search for the ChunkID.  I ran the program for several hours without checking any bytes and it worked flawlessly without losing or adding a single byte.

    I did stumble upon a nice way to implement your scenario, however.  There was a byte just a few bytes ahead of the bytes that I wanted to read that remained the same all the time.  I would load an byteArray with bytes begining a few bytes before that marker and a few bytes beyond the bytes I wanted.  Then I checked the location of my marker byte in that array and could make adjustments to compensate for as many a 3 extra or 3 lost bytes.  You could compensate for more than that but that would become impractical at some point.  Anyway based on the location of my marker byte I could define where in the array to start reading my bytes that I needed.  Then I would modify the number of bytes that I read in the following loop to account for the extra or missing bytes from the current loop.  This methond eliminates data getting out of sync over any length of time as long as no more than 3 bytes are missing or added.

    The reason that this worked for me is because I was only interested in 4 of 280 bytes that I recieved in each loop.  And the marker byte I used was only a 4 ahead of the 4 bytes I wanted.  So as long as any missing or extra bytes occurred outside of the range from my marker bytes to the end of my data I was fine.  For me thats (1+4+4)/280 or about 3% chance of an extra or missing byte corrupting my data of interest.  Heres the code.  Each loop had 280 bytes.  Byte number 36 was always (&H10) and the bytes I wanted were 40 thru 44, all neighboring bytes were always 0.

    Private processDataThread As New Thread(AddressOf ThreadProcSafe)

    Private lngMyData As Long
        Private FirstByteReceived As Boolean = False

    Private fineTuningMarker As Integer = 3

    Private MainBuffer(279) As Byte 'each loop has exctly 280 bytes

       

    Private Sub SerialPort1_DataReceived(ByVal sender As Object, _
                ByVal e As System.IO.Ports.SerialDataReceivedEventArgs) _
                           Handles SerialPort1.DataReceived

            Try
                'read bytes preceeding marker byte minus 3
                ' (first time only)
                If FirstByteReceived Then
                    'verify that 280 bytes are available
                    Do While SerialPort1.BytesToRead < 280
                        Thread.Sleep(1)
                    Loop
                    'read 280 bytes (plus or minus the extra
                    ' or missing bytes from the previous loop)
                    SerialPort1.Read(MainBuffer, 0, 277 + fineTuningMarker)
                    'find the location of the Marker byte (&H10)
                    'in the byte array (should be 3rd element)
                    fineTuningMarker = Array.IndexOf(MainBuffer, CByte(&H10))
                    'get the 4 data bytes begining 4 bytes after the marker byte
                    lngMyData = BitConverter.ToInt64(MainBuffer, fineTuningMarker + 4)
                    'start other thread to process the 4 bytes
                    processDataThread = New Thread(AddressOf ThreadProcSafe)
                    processDataThread.Start()
                    processDataThread.Join()
                Else
                    'read bytes preceeding data bytes (first time only)
                    Dim junk(43) As Byte
                    FirstByteReceived = True
                    'verify that 33 bytes are available
                    Do While SerialPort1.BytesToRead < 33
                        Thread.Sleep(1)
                    Loop
                    'discard these bytes
                    SerialPort1.Read(junk, 0, 33)
                End If

            Catch ex As Exception
                MessageBox.Show("Trouble in SerialPort1_DataReceived  " & ex.message)
            End Try

        End Sub

    'from the sub UpdateDisplay (not shown) I can safely access form controls

     Private Sub ThreadProcSafe()
            Try
                Me.UpdateDisplay()
            Catch ex As Exception
                MessageBox.Show("Trouble in ThreadProcSafe " & ex.Message)
            End Try

        End Sub

    Tuesday, August 8, 2006 5:40 PM