none
How does SerialPort handle DataReceived?

    Question

  • Hello,

    I connect my DataReceivedHandler to my SerialPort by
       mySerialPort.DataReceived += myDataReceivedHandler;

     Each time when mySerialPort receives a character then function
     myDataReceivedHandler will be executed.

     What happens if myDataReceivedHandler is busy executing and
     mySerialPort receives another character?

     Will mySerialPort somehow interrupt the execution of
     myDataReceivedHandler and re-start executing
     myDataReceivedHandler or will the SerialPort.DataReceived
     event not happen or will the SerialPort.DataReceived event
     been executed after the execution of myDataReceivedHandler
     has finished?

    Regards,

    Henk

    Thursday, March 22, 2007 10:10 AM

Answers

  • I did some digging through the SerialPort class and its various hidden helper classes.  It doesn't work at all like I expected and I'd assume there are a few ramifications from the way it was implemented.  First off, the SerialPort class fires up a background thread when you Open() the port.  That thread waits for something interesting to happen with the WaitCommEvent() API function.

    When it does, data received in your case, it dips in the ThreadPool and calls QueueUserWorkItem() which allocates a thread pool thread to call your DataReceived event handler.  It then immediately calls WaitCommEvent() again to wait for the next "something interesting".

    After some time and a few context switches later, your DataReceived event will start running.  Things get interesting of course when before that happens after another byte is received.  We're getting to your question and not at all an unlikely scenario at high baudrates.  Depending on the rate the WaitCommEvent() signals interesting events, you could get a DataReceived event handler call for each individual byte received by the port.  That's what I've seen and that explains other threads commenting on how CPU usage shooting way up when the program runs in the debugger.  It does stuff for each thread started by the CLR.

    The call to your DataReceived event may well be delayed, depending on system load.  By the time it runs and you use the BytesToRead property, you may be seeing bytes received for which SerialPort has already queued new thread pool worker items.  That means that you may well see zero bytes (because they were read by the previous invocation of the event).  Also, the the number of bytes may change right after you call it in the middle of your DataReceived event's execution when another byte was received a few microseconds later.

    Another big question for me is how calls to a delegate made through ThreadPool.QueueUserWorkItem() get serialized.  From what I see in the docs, each "user work item" (DataReceived event call) is made on a separate thread.  Which means that, technically, multiple invocations of your DataReceived event could be running at the same time.  The "Queue" prefix to the name suggests that's not how it works but I don't see anything in the docs that specifically states it works that way and that delegate calls get serialized by the thread pool.  If anybody knows about this, I'd like their feedback.

    Summarizing, it sounds to me you should make no assumptions at all in your DataReceived event.  Don't assume anything was received at all, lock your buffers.  Frankly, this is not the way I would have done it (and did) and I have to assume a lot of programs get by this due to the modest data receive rates supported by serial ports.

    A key developer at MSFT that handles the SerialPort class is Kim Hamilton.  I'll try to invoke my "powers" as a MVP to get him/her to respond.  I can't promise anything, I've had no luck before.

    Thursday, March 22, 2007 9:03 PM
    Moderator

All replies

  • Hi,

    Please ignore this post,if i didn't get ur question correctly.

    better use threading in such cases(if applicable).

    When ever u receive a character from the  port,create a thread,pass the character as it's param,process DataReceivedHandler implementation in thread proc.

    In that we can avoid this type of collisions.

    Thanx,

    Ch.T.Gopi Kumar.

    Thursday, March 22, 2007 2:14 PM
  • 'Ignored' ;-)

    Well, I wonder if somebody has some experience with it and really understands what is going on behind the scene...
     I don't think threads are needed. I did some investo and found out that:
     - when the SerialPort.DataReceived EventHandler is started by a received character and this EventHandler contains an infinite loop then this EventHandler is NOT re-executed when new characters are received by SerialPort.
     - SerialPort.DataReceived does not 'fire' at each received character due to internal processing time needed by the SerialPort object itself. I mean for example (I tested it) when SerialPort receives 12 characters in one burst then it will only 'fire' the SerialPort.DataReceived event 4 times. Therefore the SerialPort.BytesToRead is needed.
      - so SerialPort does not generate 12 times a SerialPort.DataReceived event for each of the 12 received characters.

    Using the SerialPort.BytesToRead function in a loop each time a SerialPort.DataReceived event is signalled will be sufficient to handle ALL received characters.

     (I still find this C# 'event' naming very tricky and confusing...)

    Just my two cents.

    Regards,

    Henk



    Thursday, March 22, 2007 2:49 PM
  • Don't use threading. Event handlers will not fire at the same time if you have a single thread. This goes for ALL events. The event will only fire if your program is doing nothing (running idle in the messageloop). So just use the DataReceived event and don't worry about it.
    Thursday, March 22, 2007 3:53 PM
  • I did some digging through the SerialPort class and its various hidden helper classes.  It doesn't work at all like I expected and I'd assume there are a few ramifications from the way it was implemented.  First off, the SerialPort class fires up a background thread when you Open() the port.  That thread waits for something interesting to happen with the WaitCommEvent() API function.

    When it does, data received in your case, it dips in the ThreadPool and calls QueueUserWorkItem() which allocates a thread pool thread to call your DataReceived event handler.  It then immediately calls WaitCommEvent() again to wait for the next "something interesting".

    After some time and a few context switches later, your DataReceived event will start running.  Things get interesting of course when before that happens after another byte is received.  We're getting to your question and not at all an unlikely scenario at high baudrates.  Depending on the rate the WaitCommEvent() signals interesting events, you could get a DataReceived event handler call for each individual byte received by the port.  That's what I've seen and that explains other threads commenting on how CPU usage shooting way up when the program runs in the debugger.  It does stuff for each thread started by the CLR.

    The call to your DataReceived event may well be delayed, depending on system load.  By the time it runs and you use the BytesToRead property, you may be seeing bytes received for which SerialPort has already queued new thread pool worker items.  That means that you may well see zero bytes (because they were read by the previous invocation of the event).  Also, the the number of bytes may change right after you call it in the middle of your DataReceived event's execution when another byte was received a few microseconds later.

    Another big question for me is how calls to a delegate made through ThreadPool.QueueUserWorkItem() get serialized.  From what I see in the docs, each "user work item" (DataReceived event call) is made on a separate thread.  Which means that, technically, multiple invocations of your DataReceived event could be running at the same time.  The "Queue" prefix to the name suggests that's not how it works but I don't see anything in the docs that specifically states it works that way and that delegate calls get serialized by the thread pool.  If anybody knows about this, I'd like their feedback.

    Summarizing, it sounds to me you should make no assumptions at all in your DataReceived event.  Don't assume anything was received at all, lock your buffers.  Frankly, this is not the way I would have done it (and did) and I have to assume a lot of programs get by this due to the modest data receive rates supported by serial ports.

    A key developer at MSFT that handles the SerialPort class is Kim Hamilton.  I'll try to invoke my "powers" as a MVP to get him/her to respond.  I can't promise anything, I've had no luck before.

    Thursday, March 22, 2007 9:03 PM
    Moderator
  • nobugz

    Thanks for the explanation. I have asked excactly the same questions before in both the VB forum and the Networking and Communication forum, but without any answer.

    When you talk to Kim Hamilton, please mention the problem that the wide of the receive and transmit buffers do not match (modern) UART's, which makes it impossible with a precise break, error and 9th bit detection. The receiver FIFO is 11-bit wide plus the overrun flag in all UART's and the newest types like 16C950 also has a 9 bit transmitter FIFO to be able to support 9th bit communication. The stream object of SerialPort is as far as I know already 16 bit wide, so why not utilize 9 bits for the transmitter and at least 12 bits for the receiver? We also miss a TransmitterSerialRegisterEmpty event very much to be able to control the modem signals and transmit a break condition.

    PS. I use the serial port up to 460.8 kbit/s (and even 921.6 kbit/s at short runs) with a 16C950 UART, the DataReceived event, a BytesToRead loop and BeginInvoke to pass the telegrams to the UI thread and it works fine - even in the debugger - except for break detection, which only works up to approximately 600 bit/s due to the way it is implemented :-(

    EDIT

    WaitCommEvent can probably only come from one place - the interrupt system in the UART driver, so if more bytes arrive before the event handler gets up and running, I presume that the UART will just buffer these bytes up in its internal FIFO and will not fire the event once more. Therefore, the very interesting part is which routine reads from the UART FIFO and puts the data into the receive buffer of SerialPort or from which thread is it done, because when the UART FIFO gets below the trigger point the event may fire again?

    NEW EDIT

    As I see it, the "Queue" in QueueUserWorkItem() only refers to the fact that if no thread is available in the thread pool, the delegate will be put in queue until a thread becomes available. 

    Friday, March 23, 2007 10:58 AM
  • Better late than newer.

     

    I have now done some tests to find out when the DataReceived event fires, at it seems that it fires if the number of characters in the receive buffer is greater than or equal to the threshold level and the event handler for the DataReceived event is not busy! As long as the event handler for the DataReceived event is running, DataReceived will not fire again, so there will never be more threads and event handlers in parallel.

     

    Previously, I thought that DataReceived was armed when the number of characters in the receive buffer came below the threshold level, but this is not true. It is (probably) armed when the event handler returns.

    Monday, May 07, 2007 3:57 PM
  • This is right -- your event handler won't be called by two separate threads in parallel. Here's a more long-winded explanation which pulls together some questions from the thread.

     

    SerialPort has a background thread that's waiting for events (via WaitCommEvent). Whenever an event arrives, it queues a threadpool work item that _may_ result in a call to your event handler. Let's focus on one of these threadpool threads. It tries to take a lock (quick reason: this exists to synchronize event raising with closing; for more details see the end) and once it gets the lock it checks whether the number of bytes available to read is above the threshold. If so, it calls your handler.

     

    So this lock is the reason your handler won't be called in separate threadpool threads at the same time.

     

    Note that if we didn't take the lock, it would force apps to handle certain race conditions. Here's the scenario: suppose you have a straightforward handler that checks the number of bytes available and then performs a read requesting that number of bytes. Also suppose there's a burst of data such that our background thread that's waiting for events via WaitCommEvent detects data received events in rapid succession and then doesn't get any more data for a very long time.

     

    If we didn't take the lock and called your handler on multiple threadpool threads, this could happen:

    Threadpool thread 1 is active. This handler checks the SerialPort for the number of bytes available. This will be the total number of bytes available on the serial port (incorporating both events). Let's say this is N.

    Threadpool thread 2 is active. This handler is also checking the num bytes available, so it also gets told there are N bytes available.

    Threadpool thread 1 is active. It does a read for N bytes and gets them.

    Threadpool thread 2 is active. It does a read for N bytes and waits forever...or until the timeout you specified.

     

    Note that this would force your app to take locks to handle this or rely very heavily on timeouts. Smile

     

    Let's focus on the question of what your handler should be doing, now that you know it won't be called on separate threads. It makes things easy on your apps if you check the number of bytes available and read them from the serial port synchronously in your handler. Put differently -- if every time you received a data received event, you start a new thread to read the data from the serialport, it would free up your data received handler to be called again, but you'd have to deal with the same race condition described above.

     

    At the same time, you don't want to be doing too much in your data received handler. A guideline is -- only do what you need to pull the immediate data off the serial port and then start a new thread to do the rest. You don't want to block your handler on expensive things like making a database call with the data or writing to a file.

     

    On a related note, you don't want to be updating your GUI synchronously in the handler. This can result in a nasty deadlock condition described in this thread. Note that this has a quick workaround of using Control.BeginInvoke instead of Control.Invoke.

     

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1591441&SiteID=1&mode=1

     

    Finally, about the lock. The primary reason for it was to coordinate raising events with closing the serial port. In particular, it was a design point to not raise stale events while a close is in process. It leads to interesting behavior, described in this thread and the one linked to above.

     

    Saturday, May 12, 2007 6:09 AM
    Moderator
  • Thanks a lot for your explanation. Tests are good, but knowing for sure is many times better and the way DataReceived behaves is absolute vital for all event driven serial port applications.

     

    However, there are still many questions to be answered. For example, it is extremely important to know if any buffering takes place in case something is not ready. What I actually need is a precise description of what goes on from data enter the UART until they are available on the output of the SerialPort object.

     

    1) According to your explanation, SerialPort graps a threadpool thread when WaitCommEvent shows an event. Does this happen for any event including changes in modem control signals etc. or just when data are received?

     

    2) It then tries to take a lock and once it gets the lock it checks whether the number of bytes available to read is above the threshold. If so, it calls your handler, but what happens if the number of bytes is not above the threshold?

     

    3) What happens if the event handler is already running so that serial port cannot get the lock?

     

    4) How often will the UART driver activate WaitCommEvent when it receives data? I would expect than every time the number of data has been below the threshold level and then reaches the threshold level (again), the UART driver fires an event.

     

    5) How does SerialPort handle a change in the modem control input signals like e.g. CTS? I have always wondered why it is not necessary to use Invoke or BeginInvoke to display these signals - e.g. by changing a background color of a control. I would expect an illegal cross thread call, but you don't get such an error.

     

    6) Why should you return from the event handler as fast as possible as you write? A typical serial port application receives telegrams with a protocol. The simplest one is just to terminate each telegram with a given character like LineFeed. You can then use ReadLine to receive the whole telegram, but at slow transmission speeds, the event handler will be running long time before the entire telegram has been received. In fact, I find the event handler a very good place to wait for bytes because it uses its own thread and will therefore not block the UI thread, and because DataReceived will not fire before the event handler returns, this should not course any problems. Are any events buffered up, which may course trouble if you wait in the event handler?

     

    7) According to the help files, there are two buffers - one in the UART driver and one in the SerialPort object, and BytesToRead shows the sum of the characters in the two buffers. The buffer in the UART driver is the one you set with ReadBufferSize, but how big is the buffer in the SerialPort object?

     

    8) From which thread are bytes transferred from the UART driver buffer to the buffer in the SerialPort object?

     

    9) What happens if you select a ReadBufferSize, which is bigger then the UART driver can handle (many drivers can only handle 32Kbyte)?

     

    10) As far as I know, thread pool threads and the background worker threads have lower priority than the UI thread. Does this mean that all actions on the serial port have lower priority than the UI and therefore must wait for the UI? In other words, why use BeginInvoke to return fast if all jobs on the UI thread have higher priority and therefore anyway are activated first?

     

    11) All PC UART's (16C550 to 16C950) have an 11-bit receiver FIFO to be able to give a precise synchronization between data, break conditions and errors. This is vital if you want to separate telegrams by means of break states like I want, or want to use modern 9th bit communication and perhaps automatically unit addressing. However, some fool has been so "clever" as to put an 8-bit wide buffer on top of this 11-bit FIFO and has therefore thrown all this vital information away. Break is not even latched, so above approximately 600 bit/s, the break state may be gone when the event handler gets up and running. Perhaps this bug is a left from the past and a part of the way UART drivers are made, but it is certainly not very clever. Do you know a workaround for this? I have been working on this problem for several months.

     

    12) Is there any way to get a TransmitterSerialRegisterEmpty event? This is vital to be able to control the modem signals and generate a break condition - especially in case of advanced UART's (16C650 to 16C950) with automatic CTS flow control where it is impossible to estimate when the transmitter FIFO and the transmitter serial register become empty.

    Tuesday, May 15, 2007 7:58 AM
  • Responses inline...

     

    1) According to your explanation, SerialPort graps a threadpool thread when WaitCommEvent shows an event. Does this happen for any event including changes in modem control signals etc. or just when data are received?

     

    This applies to data received, pin changed, and error events

     

    2) It then tries to take a lock and once it gets the lock it checks whether the number of bytes available to read is above the threshold. If so, it calls your handler, but what happens if the number of bytes is not above the threshold?

     

    That thread is then finished because there's nothing to do. But the event loop thread is still waiting for more events.

     

    3) What happens if the event handler is already running so that serial port cannot get the lock?

     

    By lock, I'm referring to the C# lock {} syntax, which is syntactic sugar for Monitor.Enter. If you need more info on how that works, please read this: http://msdn2.microsoft.com/en-us/library/system.threading.monitor.enter(VS.80).aspx

     

    4) How often will the UART driver activate WaitCommEvent when it receives data? I would expect than every time the number of data has been below the threshold level and then reaches the threshold level (again), the UART driver fires an event.

     

    The received bytes threshold is managed on our side; our event loop runner is waiting in WaitCommEvent, which waits according to the event mask (it's pretty straightforward to see how these map to our properties) http://msdn2.microsoft.com/en-us/library/aa363479.aspx

     

    Then we're checking if the amount received is greater than the threshold.

     

    5) How does SerialPort handle a change in the modem control input signals like e.g. CTS? I have always wondered why it is not necessary to use Invoke or BeginInvoke to display these signals - e.g. by changing a background color of a control. I would expect an illegal cross thread call, but you don't get such an error.

     

    6) Why should you return from the event handler as fast as possible as you write? A typical serial port application receives telegrams with a protocol. The simplest one is just to terminate each telegram with a given character like LineFeed. You can then use ReadLine to receive the whole telegram, but at slow transmission speeds, the event handler will be running long time before the entire telegram has been received. In fact, I find the event handler a very good place to wait for bytes because it uses its own thread and will therefore not block the UI thread, and because DataReceived will not fire before the event handler returns, this should not course any problems. Are any events buffered up, which may course trouble if you wait in the event handler?

     

    Great, there's no inconsistency between what you're doing and the guideline I described. What you want to avoid is situations where your event handler is adding significant overhead beyond reading the data. The example I mentioned is if your event handler then opened a database connection, wrote the data there, formatted it, put it on the UI, etc. The concern is that you would start to lose data if you're adding too much overhead. However, note that even if you did all of this (database connection, etc...lots of overhead) you probably still wouldn't lose data unless data is arriving very rapidly. These details depend on the context, but it's a good practice to not add too much unnecessary overhead in the event handler.

     

    7) According to the help files, there are two buffers - one in the UART driver and one in the SerialPort object, and BytesToRead shows the sum of the characters in the two buffers. The buffer in the UART driver is the one you set with ReadBufferSize, but how big is the buffer in the SerialPort object?

     

    1024

     

    8) From which thread are bytes transferred from the UART driver buffer to the buffer in the SerialPort object?

     

    I should clarify that the transfer doesn't happen until you call Read. In other words, there's no transfer beforehand in preparation.

     

    9) What happens if you select a ReadBufferSize, which is bigger then the UART driver can handle (many drivers can only handle 32Kbyte)?

     

    This is just a call to the windows api SetupComm, which you can see doc'd here:

    http://msdn2.microsoft.com/en-us/library/aa363439.aspx

     

    10) As far as I know, thread pool threads and the background worker threads have lower priority than the UI thread. Does this mean that all actions on the serial port have lower priority than the UI and therefore must wait for the UI? In other words, why use BeginInvoke to return fast if all jobs on the UI thread have higher priority and therefore anyway are activated first?

     

    Using BeginInvoke means the call is asynchronous, which only means the event handler is done faster because it doesn't have to wait for the call to complete.

     

    11) All PC UART's (16C550 to 16C950) have an 11-bit receiver FIFO to be able to give a precise synchronization between data, break conditions and errors. This is vital if you want to separate telegrams by means of break states like I want, or want to use modern 9th bit communication and perhaps automatically unit addressing. However, some fool has been so "clever" as to put an 8-bit wide buffer on top of this 11-bit FIFO and has therefore thrown all this vital information away. Break is not even latched, so above approximately 600 bit/s, the break state may be gone when the event handler gets up and running. Perhaps this bug is a left from the past and a part of the way UART drivers are made, but it is certainly not very clever. Do you know a workaround for this? I have been working on this problem for several months.

     

    Sorry, I haven't run into this before. Let's reopen to the wider distribution because maybe someone else has run into thisi before

     

    12) Is there any way to get a TransmitterSerialRegisterEmpty event? This is vital to be able to control the modem signals and generate a break condition - especially in case of advanced UART's (16C650 to 16C950) with automatic CTS flow control where it is impossible to estimate when the transmitter FIFO and the transmitter serial register become empty.

     

     

    Friday, May 18, 2007 10:24 AM
    Moderator
  • Thanks again for your reply. I have been trying to dig down into the serial port for a very long time, but so far without success, so your answers are very helpful indeed. However, I still have some follow up questions:

     

    ------------------

     

    3) What happens if the event handler is already running so that serial port cannot get the lock?

     

    By lock, I'm referring to the C# lock {} syntax, which is syntactic sugar for Monitor.Enter. If you need more info on how that works, please read this: http://msdn2.microsoft.com/en-us/library/system.threading.monitor.enter(VS.80).aspx

     

    3A) Which object (ObjectToRequest) do you take the lock on - Monitor.Enter(ObjectToRequest)?

     

    3B) Will the lock block all events, that is, will a thread which e.g. handles the DataReceived event also block for PinChanged and ErrorReceived event?

     

    -----------------

     

    4) How often will the UART driver activate WaitCommEvent when it receives data? I would expect than every time the number of data has been below the threshold level and then reaches the threshold level (again), the UART driver fires an event.

     

    The received bytes threshold is managed on our side; our event loop runner is waiting in WaitCommEvent, which waits according to the event mask (it's pretty straightforward to see how these map to our properties) http://msdn2.microsoft.com/en-us/library/aa363479.aspx

     

    Then we're checking if the amount received is greater than the threshold.

     

    4A) Do you test the threshold level before you grap a thread from the thread pool for the DataReceived event so that if the threshold limit is not reached, no thread will be grapped, or is it the thread, which does the test, as you write in one of your answers:

    "That thread is then finished because there's nothing to do. But the event loop thread is still waiting for more events." ?

     

    4B) I presume that a call of WaitCommEvent clears the event flags before it returns (of course it don't clear the return value), so that if you call WaitCommEvents again immediately after, the new call will not return before the UART driver has set one of the flags again. Is that true (unfortunately, the help file for WaitCommEvent does not describe this)?

     

    ----------------

     

    8) From which thread are bytes transferred from the UART driver buffer to the buffer in the SerialPort object?

     

    I should clarify that the transfer doesn't happen until you call Read. In other words, there's no transfer beforehand in preparation.

     

    8A) How many bytes are transferred? I could imagine, that if you e.g. call ReadByte, one byte is transferred, and if you call ReadLine, an entire line is transferred and so on, but is this true?

     

    8B) If yes, what happens if the telegram length is bigger than 1024 bytes (the size of the buffer in the serial port object) or you e.g. call ReadExisting and there are more than 1024 bytes in the UART driver buffer?

     

    8C) What is the buffer in the serial port object actually used for?

     

    ----------------------

     

    5) How does SerialPort handle a change in the modem control input signals like e.g. CTS? I have always wondered why it is not necessary to use Invoke or BeginInvoke to display these signals - e.g. by changing a background color of a control. I would expect an illegal cross thread call, but you don't get such an error.

     

    5A) I can see from your answers that PinChanged is handled in a thread pool thread like DataReceived, but I still wonder why you can write directly to a control generated on the UI thread from the event handler without getting an illegal cross thread error. For example, I have the following event handler to display the state of the modem signal lines:

     

        ' This subroutine updates the modem control signal lamps

     

        Private Sub ModemLamps(ByVal sender As Object, ByVal e As SerialPinChangedEventArgs) Handles COMPort.PinChanged

            If COMPort.DsrHolding Then

                DSRLamp.BackColor = Color.LightGreen

            Else

                DSRLamp.BackColor = Color.Gray

            End If

            If COMPort.CtsHolding Then

                CTSLamp.BackColor = Color.LightGreen

            Else

                CTSLamp.BackColor = Color.Gray

            End If

        End Sub

     

    DSRLamp and CTSLamp are just buttons. This works fine and does not generate any compile error as I would expect it to do.

     

    ------------------

     

    10) As far as I know, thread pool threads and the background worker threads have lower priority than the UI thread. Does this mean that all actions on the serial port have lower priority than the UI and therefore must wait for the UI? In other words, why use BeginInvoke to return fast if all jobs on the UI thread have higher priority and therefore anyway are activated first?

     

    Using BeginInvoke means the call is asynchronous, which only means the event handler is done faster because it doesn't have to wait for the call to complete.

     

     I know that BeginInvoke returns as soon as the message (delegate) has been appended to the message queue, but my point is that if the UI thread has higher priority than the background thread used for WaitCommEvent and the thread pool threads used for the event handlers then you may as well use Invoke instead of BeginInvoke because the serial port anyway has to wait until all jobs on the UI thread are finished. The benefit of BeginInvoke is not to wait for the UI thread, but a lower thread priority may destroy that benefit. Even if the event handler returns immediately after calling BeginInvoke it has no practical meaning if neither the background thread nor a new thread pool thread for the next DataReceived event can get up and running before the invoked method on the UI thread and any other UI job have finished their jobs.

     

    10A) Therefore, what priority are you using for the background thread and the thread pool threads compared to the priority of the UI thread?

     

    --------------------

     

    11) All PC UART's (16C550 to 16C950) have an 11-bit receiver FIFO to be able to give a precise synchronization between data, break conditions and errors. This is vital if you want to separate telegrams by means of break states like I want, or want to use modern 9th bit communication and perhaps automatically unit addressing. However, some fool has been so "clever" as to put an 8-bit wide buffer on top of this 11-bit FIFO and has therefore thrown all this vital information away. Break is not even latched, so above approximately 600 bit/s, the break state may be gone when the event handler gets up and running. Perhaps this bug is a left from the past and a part of the way UART drivers are made, but it is certainly not very clever. Do you know a workaround for this? I have been working on this problem for several months.

     

    Sorry, I haven't run into this before. Let's reopen to the wider distribution because maybe someone else has run into thisi before

     

    It is a very serious problem - not only for me, but for everybody, who wants to use modern 9th bit communication where the parity bit is redefined to an address bit. Modern UART's like 16C950 from Oxford Semiconductor are able to compare the address automatically, which is extremely useful on multidrop lines, and some UART's like 16C950 even have a 9-bit wide transmitter FIFO so that the address byte with the 9th bit set can be transmitted automatically together with all data (burst).

     

    The only good solution is to make all buffers including the one in the UART driver 16 bit wide and include Break, Framing error, Parity error/9th bit and OverRun and perhaps the modem input signals on the receiver, and the 9th bit and break and perhaps the modem output signals on the transmitter. I would also use the most significant bit of the transmitter word to distinguish between data and address of the internal UART registers. No matter what you do, you will never be able to follow the technical development of UART's, but if the internal registers was easy accessible, it would be easy to utilize even the most advanced UART now and in the future.

     

    ------------------

    12) Is there any way to get a TransmitterSerialRegisterEmpty event? This is vital to be able to control the modem signals and generate a break condition - especially in case of advanced UART's (16C650 to 16C950) with automatic CTS flow control where it is impossible to estimate when the transmitter FIFO and the transmitter serial register become empty.

     

    12A) Why have you not utilized the EV_TXEMPTY event in SetCommEvent and WaitCommEvent?

     

    --------------------

     

    Sorry for the long row of questions. I would really appreciate if you can get the time to answer them. The serial port is not very well documented and there is virtually no available below-the-hood information, which makes it very difficult to determine the optimized way of utilizing SerialPort. Besides, the help files are almost impossible to understand for beginners. Therefore, I have tried to gather all the information I can about the serial port and has rewritten it in a language, which even beginners should have a chance of understanding. If you have more time, you can read this description at the URL: http://www.innovatic.dk/knowledg/SerialCOM/SerialCOM.htm. Any comment, suggestions and additions - even corrections to my english - are very welcome. I have only one intention - to make this the best and most technical correct description available anywhere, but without your background knowledge I cannot do that.

     

    Best regards

    Innovatic

    Carsten Kanstrup

     

     

    Monday, May 21, 2007 8:34 AM
  • Hi Carsten,

    I've answered inline below.

     

    Your questions have helped point out some common issues users may like to know about, in terms of synchronization, what gets buffered, etc. So I'm going to write up a blog over the next couple of weeks summarizing the issues we've discussed, focusing on scenario-driven implications for SerialPort users. Hopefully this will help people understand the practical implications in terms of synchronization issues, etc. I'lll check out that link you sent too and try to make sure I've addressed those questions.

     

    Also, I'd like to mention a tool you may be interested in, along with a caveat. Lutz Roeder's .NET Reflector is a useful tool which will let you see the decompiled C# code yourself and this would be a quick way to see internal implementation details. Some of our advanced users like to use Reflector out of curiousity. However, we certainly don't want to force people to use this because it may indicate usability issues of a class (and SerialPort certainly seems to have some of those!). So in other words, I just want to point you to that tool as a quick way to unblock you, but please do let us know about any issues because it will help us improve the design and documentation in the future.

     

    Thanks,

    Kim

     

     

    Thanks again for your reply. I have been trying to dig down into the serial port for a very long time, but so far without success, so your answers are very helpful indeed. However, I still have some follow up questions:

     

    ------------------

     

    3) What happens if the event handler is already running so that serial port cannot get the lock?

     

    By lock, I'm referring to the C# lock {} syntax, which is syntactic sugar for Monitor.Enter. If you need more info on how that works, please read this: http://msdn2.microsoft.com/en-us/library/system.threading.monitor.enter(VS.80).aspx

     

    3A) Which object (ObjectToRequest) do you take the lock on - Monitor.Enter(ObjectToRequest)?

      An instance of SerialPort contains a single instance of a SerialStream (an internal class which is exposed by the BaseStream property. It's a 1-1 relationship.

    3B) Will the lock block all events, that is, will a thread which e.g. handles the DataReceived event also block for PinChanged and ErrorReceived event?

     Yes, and it locks on the same SerialStream object for all events

    -----------------

     

    4) How often will the UART driver activate WaitCommEvent when it receives data? I would expect than every time the number of data has been below the threshold level and then reaches the threshold level (again), the UART driver fires an event.

     

    The received bytes threshold is managed on our side; our event loop runner is waiting in WaitCommEvent, which waits according to the event mask (it's pretty straightforward to see how these map to our properties) http://msdn2.microsoft.com/en-us/library/aa363479.aspx

     

    Then we're checking if the amount received is greater than the threshold.

     

    4A) Do you test the threshold level before you grap a thread from the thread pool for the DataReceived event so that if the threshold limit is not reached, no thread will be grapped, or is it the thread, which does the test, as you write in one of your answers:

    "That thread is then finished because there's nothing to do. But the event loop thread is still waiting for more events." ?

     The thread does the test, so it's tested after grabbing a threadpool thread

     

    4B) I presume that a call of WaitCommEvent clears the event flags before it returns (of course it don't clear the return value), so that if you call WaitCommEvents again immediately after, the new call will not return before the UART driver has set one of the flags again. Is that true (unfortunately, the help file for WaitCommEvent does not describe this)?

     This is correct.

    ----------------

     

    8) From which thread are bytes transferred from the UART driver buffer to the buffer in the SerialPort object?

     

    I should clarify that the transfer doesn't happen until you call Read. In other words, there's no transfer beforehand in preparation.

     

    8A) How many bytes are transferred? I could imagine, that if you e.g. call ReadByte, one byte is transferred, and if you call ReadLine, an entire line is transferred and so on, but is this true?

     

    8B) If yes, what happens if the telegram length is bigger than 1024 bytes (the size of the buffer in the serial port object) or you e.g. call ReadExisting and there are more than 1024 bytes in the UART driver buffer?

     

    8C) What is the buffer in the serial port object actually used for?

     

    I'll answer A,B,and C together. This is an important question I want to address in the blog because it's probably confusing to many of our users. Bytes are only put into the internal SerialPort buffer when using the methods dealing with encodings (i.e. chars, strings, not bytes). When reading bytes, some bytes may be taken out of the buffer, but not put into it. So if you worked only with bytes, the internal buffer would never get used.

     

    If you're wondering why this is, note that we prefer to put data directly into the buffer you specify for performance. But we have to special case encoding issues to deal with special cases there, such as some encodings being variable kength.

     

    The size of this buffer is 1024.

     

    In general our reads work like this:: try to get data out of that internal buffer if it exists. If none there or we didn't read enough, then we proceed to read more. If you're calling a byte-related method and you need more bytes, the bytes are put directly into your buffer, i.e. the buffer you passed into Read(byte[],...). If you're calling an encoding-related method such as ReadChars, ReadLine or ReadTo, then the bytes are first put into the internal buffer (length 1024) and then the char-encoded data are put into your buffer (or returned as string, etc, depending on the method).

     

    ReadExisting is a special case because it creates a new buffer, with size = number of bytes available. So all are read out and returned to you as a string. In other words, to answer 8b, you'll get all of the available bytes returned as a string. Note that due to encoding issues, I need to qualify this statement a bit more. If you were using a unicode encoding and only a partial part of a character were given (say a surrogate), tben the assumption is that the next incoming byte would form the complete character, so this byte would be buffered. This probably happens rarely but I wanted to mention it as a possibility.

     

    ReadLine is just ReadTo, where the string read to is the new line string you specify in the NewLine property. The detault is \n.

     

    Some additional info:

     

    // Read Design pattern:

    // : ReadChar() returns the first available full char if found before, throws TimeoutExc if timeout.

    // : Read(byte[] buffer..., int count) returns all data available before read timeout expires, up to *count* bytes

    // : Read(char[] buffer..., int count) returns all data available before read timeout expires, up to *count* chars.

    // : Note, this does not return "half-characters".

    // : ReadByte() is the binary analogue of the first one.

    // : ReadAvailable(): returns all full characters which are IMMEDIATELY available.

    ----------------------

     

    5) How does SerialPort handle a change in the modem control input signals like e.g. CTS? I have always wondered why it is not necessary to use Invoke or BeginInvoke to display these signals - e.g. by changing a background color of a control. I would expect an illegal cross thread call, but you don't get such an error.

     

    5A) I can see from your answers that PinChanged is handled in a thread pool thread like DataReceived, but I still wonder why you can write directly to a control generated on the UI thread from the event handler without getting an illegal cross thread error. For example, I have the following event handler to display the state of the modem signal lines:

     

        ' This subroutine updates the modem control signal lamps

     

        Private Sub ModemLamps(ByVal sender As Object, ByVal e As SerialPinChangedEventArgs) Handles COMPort.PinChanged

            If COMPort.DsrHolding Then

                DSRLamp.BackColor = Color.LightGreen

            Else

                DSRLamp.BackColor = Color.Gray

            End If

            If COMPort.CtsHolding Then

                CTSLamp.BackColor = Color.LightGreen

            Else

                CTSLamp.BackColor = Color.Gray

            End If

        End Sub

     

    DSRLamp and CTSLamp are just buttons. This works fine and does not generate any compile error as I would expect it to do.

     

    I'm not positive since this relates to the WebControl class, but I'm guessing that you might want to check out the code-behind and see if something like this is going on for BackColor. If it's still not obvious, I'd suggest starting a new thread on this question to get the attention of someone more familiar with the UI world.

    http://infosysblogs.com/microsoft/2006/10/

    ------------------

     

    10) As far as I know, thread pool threads and the background worker threads have lower priority than the UI thread. Does this mean that all actions on the serial port have lower priority than the UI and therefore must wait for the UI? In other words, why use BeginInvoke to return fast if all jobs on the UI thread have higher priority and therefore anyway are activated first?

     

    Using BeginInvoke means the call is asynchronous, which only means the event handler is done faster because it doesn't have to wait for the call to complete.

     

     I know that BeginInvoke returns as soon as the message (delegate) has been appended to the message queue, but my point is that if the UI thread has higher priority than the background thread used for WaitCommEvent and the thread pool threads used for the event handlers then you may as well use Invoke instead of BeginInvoke because the serial port anyway has to wait until all jobs on the UI thread are finished. The benefit of BeginInvoke is not to wait for the UI thread, but a lower thread priority may destroy that benefit. Even if the event handler returns immediately after calling BeginInvoke it has no practical meaning if neither the background thread nor a new thread pool thread for the next DataReceived event can get up and running before the invoked method on the UI thread and any other UI job have finished their jobs.

     

    10A) Therefore, what priority are you using for the background thread and the thread pool threads compared to the priority of the UI thread?

      Using the detault priority

    --------------------

     

    11) All PC UART's (16C550 to 16C950) have an 11-bit receiver FIFO to be able to give a precise synchronization between data, break conditions and errors. This is vital if you want to separate telegrams by means of break states like I want, or want to use modern 9th bit communication and perhaps automatically unit addressing. However, some fool has been so "clever" as to put an 8-bit wide buffer on top of this 11-bit FIFO and has therefore thrown all this vital information away. Break is not even latched, so above approximately 600 bit/s, the break state may be gone when the event handler gets up and running. Perhaps this bug is a left from the past and a part of the way UART drivers are made, but it is certainly not very clever. Do you know a workaround for this? I have been working on this problem for several months.

     

    Sorry, I haven't run into this before. Let's reopen to the wider distribution because maybe someone else has run into thisi before

     

    It is a very serious problem - not only for me, but for everybody, who wants to use modern 9th bit communication where the parity bit is redefined to an address bit. Modern UART's like 16C950 from Oxford Semiconductor are able to compare the address automatically, which is extremely useful on multidrop lines, and some UART's like 16C950 even have a 9-bit wide transmitter FIFO so that the address byte with the 9th bit set can be transmitted automatically together with all data (burst).

     

    The only good solution is to make all buffers including the one in the UART driver 16 bit wide and include Break, Framing error, Parity error/9th bit and OverRun and perhaps the modem input signals on the receiver, and the 9th bit and break and perhaps the modem output signals on the transmitter. I would also use the most significant bit of the transmitter word to distinguish between data and address of the internal UART registers. No matter what you do, you will never be able to follow the technical development of UART's, but if the internal registers was easy accessible, it would be easy to utilize even the most advanced UART now and in the future.

     

    ------------------

    12) Is there any way to get a TransmitterSerialRegisterEmpty event? This is vital to be able to control the modem signals and generate a break condition - especially in case of advanced UART's (16C650 to 16C950) with automatic CTS flow control where it is impossible to estimate when the transmitter FIFO and the transmitter serial register become empty.

     

    12A) Why have you not utilized the EV_TXEMPTY event in SetCommEvent and WaitCommEvent?

       I'm not sure this actually gives what you want; see the win32 documentation for this event:

    http://msdn2.microsoft.com/en-us/library/ms810467.aspx

    "The last character in the output buffer was sent to the serial port device. If a hardware buffer is used, this flag only indicates that all data has been sent to the hardware. There is no way to detect when the hardware buffer is empty without talking directly to the hardware with a device driver."

    --------------------

     

    Sorry for the long row of questions. I would really appreciate if you can get the time to answer them. The serial port is not very well documented and there is virtually no available below-the-hood information, which makes it very difficult to determine the optimized way of utilizing SerialPort. Besides, the help files are almost impossible to understand for beginners. Therefore, I have tried to gather all the information I can about the serial port and has rewritten it in a language, which even beginners should have a chance of understanding. If you have more time, you can read this description at the URL: http://www.innovatic.dk/knowledg/SerialCOM/SerialCOM.htm. Any comment, suggestions and additions - even corrections to my english - are very welcome. I have only one intention - to make this the best and most technical correct description available anywhere, but without your background knowledge I cannot do that.

     

    Saturday, June 09, 2007 11:46 PM
    Moderator
  • Kim (or anyone else),

     

    I'm writing an app where upon receiving a message formatted a particular way, I need to generate and send an ACK message within 2ms. Is this fast of a response time even possible to do using the SerialPort object, and if so, how would you suggest implementing this? If not, is there something other than the SerialPort object you would recommend? I would like to keep my solution to this problem as high-level as I can and most of all want to avoid using drivers if at all possible.

     

    Thanks,

    Dan

    Thursday, June 14, 2007 3:21 PM
  • You can't.  While average response time would probably measure around a couple of msec, worst case response time is much worse than that, hundreds of milliseconds on a heavily loaded system.  In a properly designed protocol, this should not be necessary.  If the requirement is firm, you'll need to write your own serial port driver so you can generate responses in the hardware interrupt handler.
    Thursday, June 14, 2007 3:53 PM
    Moderator
  • There is a problem with SerialPort class in .net
    .encoding is used always - there is no way at the moment to use it in binary mode.
    What happens is that in my case sequence 1B 83 is always replaced by 1B 3F
    while 1B 82 is Ok - given from SerialPort by 1B 82.
    This is very annoying!
    I have a code made in VB6 and in the new tools (.net) I had to use SerialPort class Sad

    Any suggestion on how to build encode class for serial port with the logic:
    ByteOut = ByteIn ?
    This can help a lot.

    I see what the driver receives by using PortMon from sysinternals.
    I tried using
    1. SerialPort.Read(buf,0,Len(SerialPort.BytesToRead))
    2. SerialPort.ReadByte

    And all of them gave the same result.
    Help please.

    Ted
    Tuesday, June 26, 2007 6:10 AM
  • Try:

    SerialPort.Encoding = System.Text.Encoding.GetEncoding(28591)

    or

    SerialPort.Encoding = System.Text.Encoding.GetEncoding(1252)

     

    PS: This answer is also send directly to you as a response to your e-mail

     

    Your question has nothing to do with the DataReceived event. In the future, make a new thread for a new subject.

    Tuesday, June 26, 2007 10:21 AM
  • Hi,

    False Alarm! Smile

    Problem is not with SerialPort
    but with new way Chr() works

    before the program I try to modify was widely using String instead of Byte array as it is handy
    to pass to the functions and already can carry concatenations
    so in VB6 (b =asc(chr(b))) is TRUE
    now it can be FALSE
    as all strings are now 2byte per symbol in unicode

    so I'm modifying all parts to use byte array and index for size of data inside
    sW = "" becomes now abW_Len = 0

    Ted
    Wednesday, June 27, 2007 10:32 AM
  • The SerialPort event firing mechanism doesn't require a message loop.  The event will fire when data receives regardless of message pumping.  There are published rules about other events firing at the same time, but nothing here is dependent on the message loop.
    In Christ, Aaron
    Monday, December 12, 2011 2:33 PM