Wince 6.0 R3 OMAP3530 Serial Event Latency RRS feed

  • Question

  • We are developing a medical device using WinCE 6.0 R3 on an OMAP 3530 processor.  I am responsible for reducing the serial comm latency.  I have done some tuning on the driver that was providied by our BSP vendor (Adeneo).  During testing I am seeing behavior that I can't explain.  My BSP is a built in the release configuration.  I am measuring timing by toggling GPIO pins and monitoring the results using a logic analyzer.

    I am monitoring the serial data and the driver code that runs in response to the serial event.  I am trying to understand why the following tests produce radically different results:

    1. I run our application, compiled in debug configuration, from Visual Studio.  This results in an average latency of 47usec from the end of a 1 byte message to the serial event processing.

    2. I run the same application and BSP as above except I start the application from the device.  In this case the device is not connected to a PC.  This results in an average latency of 175usec with a large variation.

    The above is very repeatable.  Has anyone ever seen anything similar to this?

    Saturday, September 22, 2012 12:34 AM

All replies

  • It's possible that the installation configuration created when you debug is different than your #2 test. That is, let's say that your application uses a particular DLL (or maybe it's just the EXE itself). Maybe one configuration uses this DLL/EXE from your application folder in RAM filesystem. Maybe, on the other hand, the second configuration uses a copy of the DLL/EXE stored in ROM filesystem (part of nk.bin/nk.nb0). If execution of code is slower from one or the other, that would explain the differing average.

    Can you confirm that you run the test from inside the Visual Studio debugger, then immediately run test #2 from the same target location and get the same results? If so, start looking at what is loaded and when. Perhaps the debugger loads the DLLs into RAM for execution at the beginning and completely while normal operation demand loads something (and unloads it frequently).

    You should also check for any blocking points in your code (any time a thread waits). Those are chances for assumptions about the state of things to be wrong affecting timing (maybe you assume that event #1 is always set when you get to a WaitForSingleObject() call and that happens to be true for debug, but is not true for normal execution, so there's a delay completing WaitForSingleObject).

    Paul T.

    Monday, September 24, 2012 3:12 PM
  • Hi Paul,

    There is one additional process started on the target when the application is launched with the debugger.  It is edm.exe.  I did the following experiment.

    1. Start our application using the debugger.
    2. Launch remote process viewer.
    3. Monitor the interrrupt latency using the analyzer.  It is steady at 50usec (+/-5usec).
    4. Terminate edm.exe.  This causes the debugger to disconnect.  The application continues to run.
    5. Observe that the interrupt latency.  It is now much more variable.  The times range from 50usec to 400 usec.

    I tried this several times and I get the same results.

    How can edm.exe change the behavior?

    Ed Zucker

    Monday, September 24, 2012 5:30 PM
  • EDM *is* the debugger.

    It sounds to me like demand loading of pages is occurring sometimes when interrupts are handled or you have your IST priority set too low, causing it to compete for processor time with normal priority threads in the system. It's not clear from your original message whether you are using a stock serial driver or layering something on top of that.

    You should check the priorities of any threads that are running (and make sure that application threads aren't ahead of interrupt service threads, of course). You should also be sure that interrupt servicing driver code is locked in memory. You might also check whether "other" interrupts are occurring (maybe by toggling another GPIO bit when another interrupt is found in the ISR). Maybe the IST for one of those other interrupts is prioritized.

    Sorry this isn't a prescription that's easy to follow to fix the problem.

    Paul T.

    Tuesday, September 25, 2012 3:55 PM
  • Just to clarify a few things.  I started with the stock serial driver and customized it to fit our needs.  I didn't layer anything on top of it.  The driver is locked in memory by its settings in platform.bib (NK  SHKM).  In the simple experiment I am running I have the serial IST priority set at 81.  The application I am running is a single thread set at 251.

    It seems strange to me that the debugger makes it work better.  Kernel tracker reveals that while edm.exe is running there is a kernel thread called PagePoolTrimThread active at priority 255.  At the point that edm.exe is stopped PagePoolTrimThread becomes blocked.  Also, while edm.exe is running the idle thread is blocked.  Once PagePoolTrimThread is blocked, the idle thread becomes un-blocked.  In kernel tracker, most of the time, the idle thread is running when the serial interrupt occurs.

    The next step has to be to see what the ISR is doing when the serial IRQ comes in.


    Tuesday, September 25, 2012 4:19 PM
  • Good information. I agree with your thoughts. You might set your application thread to normal priority as a test also (plus review what it's doing; if it's never blocking, it will in turn block normal priority threads in the rest of the system). There could be an interaction there (app calls shell which blocks app, etc.) Remember that any ready-to-run higher priority thread will preempt all lower priority threads, regardless of what they're doing.

    Another experiment would be an adjustment to your IST thread priority. Try changing it both higher and lower. If that makes a significant difference, you may be interacting with a driver or the driver manager or something.

    Paul T.

    Tuesday, September 25, 2012 8:19 PM