locked
Touch responsiveness issue: How to resolve jagged (non-smooth) lines?

    Question

  • There appears to be a performance related touch sampling problem with the Windows 8 touch APIs. Below is an example of the output:

    Touch sampling error with accerlerated input.

    The code that produced this result is shown below:

    <Page
        x:Class="GetIntermediatePointsReversed.MainPage"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:local="using:GetIntermediatePointsReversed"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        mc:Ignorable="d">

        <Grid Name="drawContent" Tapped="TapToClear" Background="{StaticResource ApplicationPageBackgroundThemeBrush}" />
    </Page>

    using System.Collections.Generic;
    using System.Diagnostics;
    using System.Linq;
    using Windows.Foundation;
    using Windows.UI;
    using Windows.UI.Input;
    using Windows.UI.Xaml;
    using Windows.UI.Xaml.Controls;
    using Windows.UI.Xaml.Input;
    using Windows.UI.Xaml.Media;
    using Windows.UI.Xaml.Shapes;

    namespace GetIntermediatePointsReversed
    {
        public sealed partial class MainPage : Page
        {
            const uint radius = 26;
            Dictionary<Ellipse, uint> TouchPoints;

            Dictionary<uint, Polyline> LinePoints = new Dictionary<uint, Polyline>();

            public MainPage()
            {
                this.InitializeComponent();

                TouchPoints = new Dictionary<Ellipse, uint>();

                for (int i = 0; i < 40; i++)
                {
                    TouchPoints[new Ellipse()
                    {
                        Width = radius,
                        Height = radius,
                        StrokeThickness = 2,
                        HorizontalAlignment = Windows.UI.Xaml.HorizontalAlignment.Left,
                        VerticalAlignment = Windows.UI.Xaml.VerticalAlignment.Top,
                        Stroke = new SolidColorBrush(Color.FromArgb(255, 0, 0, 0)),
                        Fill = new SolidColorBrush(Color.FromArgb(255, 0, 60, 200)),
                        RenderTransform = new TranslateTransform(){ X = -radius / 2, Y = -radius / 2}
                    }] = unchecked((uint)-1);
                }
            }

            protected override void OnPointerPressed(PointerRoutedEventArgs e)
            {
                uint id = e.Pointer.PointerId;

                if (CapturePointer(e.Pointer))
                {
                    Point tp = e.GetCurrentPoint(this).Position;

                    DownMark(id, tp.X, tp.Y);
                }

                base.OnPointerPressed(e);
            }

            protected override void OnPointerMoved(PointerRoutedEventArgs e)
            {
                uint id = e.Pointer.PointerId;

                var ipt = e.GetIntermediatePoints(this);
                Debug.WriteLine("Count: " + ipt.Count());

                foreach (PointerPoint pointerPoint in ipt.Reverse())
                {
                    Point tp = pointerPoint.Position;
                    MoveMark(id, tp.X, tp.Y);
                }

                base.OnPointerMoved(e);
            }

            protected override void OnPointerReleased(PointerRoutedEventArgs e)
            {
                uint id = e.Pointer.PointerId;

                UpMark(id);
                ReleasePointerCapture(e.Pointer);

                base.OnPointerReleased(e);
            }

            void DownMark(uint Id, double x1, double y1)
            {
                if (!TouchPoints.ContainsValue(Id))
                {
                    var ellipse = TouchPoints.FirstOrDefault(iE => iE.Value == unchecked((uint)-1));

                    if (ellipse.Key != null)
                    {
                        drawContent.Children.Add(ellipse.Key);
                        TouchPoints[ellipse.Key] = Id;

                        ellipse.Key.Margin = new Thickness(x1, y1, 0, 0);
                    }

                    // Initialize starting point of polyline.
                    Polyline polyline = new Polyline
                    {
                        Stroke = this.Resources["ApplicationForegroundThemeBrush"] as Brush,
                        StrokeThickness = 4
                    };
                    polyline.Points.Add(new Point(x1, y1));

                    // Add to draw Content and LinePoints dictionary
                    drawContent.Children.Add(polyline);
                    LinePoints.Add(Id, polyline);
                }
            }

            void MoveMark(uint Id, double x1, double y1)
            {
                if (TouchPoints.ContainsValue(Id))
                {
                    var ellipse = TouchPoints.First(iE => iE.Value == Id);
                    ellipse.Key.Margin = new Thickness(x1, y1, 0, 0);
                    LinePoints[Id].Points.Add(new Point(x1, y1));
                }
            }

            void UpMark(uint Id)
            {
                if (TouchPoints.ContainsValue(Id))
                {
                    LinePoints.Remove(Id);

                    var ellipse = TouchPoints.First(iE => iE.Value == Id);
                    drawContent.Children.Remove(ellipse.Key);
                    TouchPoints[ellipse.Key] = unchecked((uint)-1);
                }
            }

            private void TapToClear(object sender, TappedRoutedEventArgs e)
            {
                if (e.PointerDeviceType == Windows.Devices.Input.PointerDeviceType.Mouse)
                    drawContent.Children.Clear();
            }

        }
    }

    Using very similar code, Windows 7 does not exhibit this problem.  A Windows 8 desktop application that uses the WM_POINTERxxx events produces exactly the same result.  Is there anything that can be done to fix this?  FreshPaint and a couple other store applications appear to have a solution to this issue.

    These apps are able to draw smooth curves as the user is drawing (no obvious post processing of input).

    Why is Windows 8 touch processing inferior to Windows 7?
    • Edited by noemata Wednesday, July 23, 2014 3:27 PM
    Wednesday, July 23, 2014 3:21 PM

All replies

  • There isn't anything the app can do to affect the input. You'll need to apply a smoothing algorithm.

    If you use the InkManager it can smooth the strokes for you and provide a Bezier path.

    --Rob

    Wednesday, July 23, 2014 3:31 PM
    Owner
  • The bigger problem here is that Windows 7 produces smooth curves and does not have artifacts like those of Windows 8.  Worse still, you can use the Windows 7 WM_TOUCHxxx events under Windows 8 in a desktop application and get a smooth curve.  Using the Windows 8 WM_POINTERxxx events produces exactly the same artifacts for your desktop application as seen in this store sample.

    This is clearly a touch API bug is it not?  The Windows 7 touch events should not outperform the Windows 8 touch events, should they?

    Windows 8 touch handling has been advertised as being superior.  When using the Windows 7 WM_POINTERxxx events under Windows 8, you get significantly more touch data points and no artifacts.

    I can accept lower sampling rates, but the artifacts ... looks buggy to my eyes.

    Wednesday, July 23, 2014 3:41 PM
  • One final point.  If I use the RAW HID device touch data in my Windows Store application, I get significantly more touch data, without artifacts.  I was hoping that there might be a way to adjust system touch properties to eliminate the API artifact (bug).

    Having to resort to HID raw data is an extreme solution that tosses out all the benefits of using the touch APIs.

    I think this should be treated as a bug that needs fixing unless there is a reasonable workaround.

    Wednesday, July 23, 2014 3:54 PM
  • Exactly the same code, even faster rate of input, but using a pen this time.  No artifacts and very good sample rate.  Why does the pen device work as expected, and the touch device not?  The same code should yield the same result.  Touch input appears to be broken.

    Fast rate of input with pen device.

    Wednesday, July 23, 2014 5:22 PM
  • I don't reproduce this on my touch screen in Windows 8.1 Update, although I've heard of similar behavior elsewhere before. It's likely an issue with your specific device.

    On my device, long sweeps (where my finger is fairly steady) are smooth, but slow movement (where my finger is less steady) are a bit jittery. The slow jitter effect is exacerbated by the way the test code draws the PolyLines. This is the opposite of what you are describing and matches the physical input.

    A pen being more precise than touch is expected. Fingers are inherently imprecise instruments as a single finger will cover many input points. The same code should yield the same result with the same input, but switching from touch to pen gives very different input.

    I don't believe touch sample rates are generally configurable, but that's off topic for this forum and outside of my area of experience. For help configuring your system please ask in the Windows forums on http://answers.microsoft.com .

    --Rob

    Wednesday, July 23, 2014 8:05 PM
    Owner
  • I agree with your remarks pertaining to the nature of touch input. Fingers are inherently "blobby" and inaccurate for fine details.

    This sample was tested with several different touch devices. Fast, sweeping, curling gestures produce the artifact shown regardless of the touch technology used (Capacitive, LED shadow location, etc.).  I would concede that this is a non-issue were the results consistent across input APIs.  As noted above, if you use the older Windows 7 API under Windows 8, the same device produces markedly better results; no artifacts and much smoother curves.

    Why is the older API better??

    Taking a closer look:

    Touch error close up.

    One can clearly see an offset issue.  The offset is even deterministic in nature and can be corrected somewhat.  The same result will be observed on a Sony, Acer, HP, etc. touch display (just some of my test options) when fast sweeping gestures are made.

    The fact that this problem is only seen in Windows Store applications and Windows Desktop applications that use the newer APIs tends to suggest the problem is an under the hood bug with these APIs.

    Given one can get perfectly smooth curves with significantly better sampling rates using the above code, but reaching directly into the HID data stream proves without a doubt, that the API implementation is problematic.

    If I can do better with the raw HID data, so can Microsoft!  We need APIs that are flawless with basics such as this.

    If I have to write my own API layer just to fix an OS issue, then the OS has failed my needs.


    • Edited by noemata Thursday, July 24, 2014 7:24 PM
    Thursday, July 24, 2014 7:22 PM
  • I found a surprisingly simple workaround to the above problem.  As I had suspected there is another approach using the touch API for getting at the unfettered touch data.  It's unfortunate that it is so hard to find.

    The alternate approach resolves both the artifact issue and appears to improve sampling.  I found this by reverse engineering some of Microsoft's apps.  Clearly, there are folks at Microsoft that know about the problem and how to work around it.

    This is definitely a bug.  Please, please, fix it.

    Here's the result with the "other" API:

    Good touch result.

    In the image shown, I'm winding as fast as I possibly can using touch.  The result is quite respectable!

    Friday, July 25, 2014 12:46 PM
  • Below is a screen shot of a Desktop WPF application using the alternate touch input approach.  Problem fixed on desktop also.

    Note that correcting this issue for WPF applications is not possible using the standard API.  You have to develop your own TouchDevice that uses WM_POINTERxxx messages.  There are none in the wild.

    Fixed desktop touch.

    Friday, July 25, 2014 5:17 PM
  • What is the "other" API you refer to and what are the specific devices you were able to reproduce this on?

    I will report this for investigation in further versions, but having full information will be very helpful. As I mentioned previously, I haven't been able to reproduce this so either it is device specific or we are using different repro steps. An understanding of your alternate approach will help narrow down what could be the issue.

    --Rob

    Monday, July 28, 2014 8:40 PM
    Owner
  • Hi, I noticed with Microsoft's "Remote Desktop" app for WP8.1 (Lumia 925) using fine touch motions, like rolling the tip of your finger to accurately control an emulated mouse cursor on a Desktop, works great! Reminiscent of using the "nipple" on a Thinkpad... 

    However with the "same" app on WinRT8.1 (Lumia 2520), finely controlling the emulated desktop cursor is like trying to draw with an "Etch A Sketch"! The rolling of my fingertip, or fine movements are interrupted with what seems like attempts by the API at pure Horizontal and Vertical movement.

    With my limited testing the same results can be replicated with gently dragging an explorer window in WinRT desktop mode (on the Lumia 2520).

    Tuesday, July 29, 2014 1:40 AM
  • I am reluctant to describe the details of the "other" API because I do not want it to be declared the solution or that the provided code sample works as is "by design".  All of the existing samples provided by Microsoft present code similar to my original posting, so presumably that is the prescribed approach for processing touch input.  Having a consolidated API for input is fantastic.  We just need to make it work cleanly for touch.

    Here are some more screen shots ...

    Sony VAIO model SV11225PDB (actual photo):

    Sony Buggy Physical

    Sony - same as above (screen shot this time):

    Sony Buggy

    .... more in following message due to posting restrictions



    • Edited by noemata Tuesday, July 29, 2014 9:12 AM
    Tuesday, July 29, 2014 8:49 AM
  • ACER LCD Touch Monitor model T232HL (actual photo):

    Acer Buggy Physical

    Acer - same as above (screen shot this time):

    Acer Buggy

    ... more in following message due to posting restrictions
    Tuesday, July 29, 2014 8:51 AM
  • Acer screen shot using "other" API:

    Acer Good

    Notice how striking the improvement is with the Acer touch display.  The rate of the winding/curling motion in both cases was virtually identical.

    If you are working on a particularly fast PC, then you will need to generate the rotating motion rather quickly to duplicate my results, but you will see the same issue.

    On slower systems (Intel Atom based tablets), even a rather slow curling motion will produce this result.

    The key issue is the fact that an artifact is being introduced.  I am not concerned about how perfectly circular the curves are, but that there is an offset being introduced as a result of gesture/inertia post processing.  Presumably, pen and mouse input is not susceptible to this post processing.  This problem exists only with touch input.


    • Edited by noemata Tuesday, July 29, 2014 9:05 AM
    Tuesday, July 29, 2014 8:52 AM
  • In summary, the Windows Store and Desktop touch APIs produce an artifact when fast winding, curling motions are made via touch.  The offset artifact is due to some sort of post processing.  If there was merely a reduced set of data points as a result of the post processing, that would be acceptable.  Given the amount of processing that can be done on what is actually a very slow data rate for touch, Microsoft should strive not to prune data.

    It's also acceptable to say that the "other" way is preferable when the application wants maximum control over the touch data.  Artifacts are inacceptable.

    Tuesday, July 29, 2014 9:02 AM
  • I am reluctant to describe the details of the "other" API because I do not want it to be declared the solution or that the provided code sample works as is "by design".

    Please provide full details on this if you want us to look into it.

    Please don't spam the forum with redundant images. It makes the thread difficult to read without adding any value. If you can provide a clean list of the systems you've reproduced this on that would help. Please include if it is an integrated touch screen (laptop or all-in-one) or if it's an external screen. Since you're suggesting this is CPU dependent the CPU would also be useful.

    In the meantime I will track down a slower system to see if I can reproduce it.

    Also, which version of Windows are you using? There were some changes to the touch stack in the Windows 8.1 Update that may affect this. What is the version number on %windir%\system32\win32k.sys? 

    --Rob


    Monday, August 04, 2014 8:40 PM
    Owner
  • Rob, I'm not clear on why the images provided would be perceived as redundant.  I provided an actual device image and a screen shot for clarity. The actual device images were provided to show how things appeared on the physical device.

    The list of tested systems is as follows:

    Acer model T232HL and T272HL

    Baanto (all models)

    Samsung Win 8 certified laptops with touch (all models)

    Sony duo 11 and 13 (all models)

    Surface Pro 2 and 3 (all models)

    I am not sure what approach you use for testing.  I use a touch injection strategy that is initially derived from human input to get precisely reproducible timing across devices and systems that is representative of real world human usage as a final sanity check.  That way we can eliminate the physical device as being part of the problem and can focus on how the OS/CPU handle input.

    The problems described affect Windows 8.1 update 1 patched to the updates released as of late last week.

    If you employ a touch injection testing strategy, you will discover several other issues with WPF applications that use touch.  The .Net touch API has several severe bugs across all versions of Windows including Windows 8.1 update 1.

    The company I work for also employs robotics for consistent touch testing across devices.

    If you're doing this by hand and you're not a reasonably capable pianist or the like, you won't have the motor/timing skills to do this consistently.

    The "other" API I'm using is PointerPoint.RawPosition.  It has an equivalent when using the WM_POINTERxxx messages.  I'm hoping you will not declare this as the fix for the problem described.  I'm also hoping that someone at Microsoft will start to look at this more seriously.  Not getting this right will really hurt the Windows 8.1 eco system in particular.  Tablets are all about touch.

    I doubt there are too many other developers that have touch injection, robotics and many devices at their disposal for testing.  Take advantage of it!!


    • Edited by noemata Tuesday, August 05, 2014 3:44 PM
    Tuesday, August 05, 2014 3:34 PM
  • Thanks for the clarification. When I return from holiday I'll file a report to investigate incongruities between the raw pointer positions and the reported positions.

    In the future please try to provide a concise posting with full information but without additional irrelevancies that make it difficult to parse what you are talking about.

    A single image is helpful. A dozen images of essentially the same thing just make the thread hard to follow.

    It doesn't matter how elaborate your test environment is if you don't provide actionable reports.

    WPF uses its own touch input system, different from the core OS (or at least it used to: I'm not up to speed on the latest with WPF). There is no ".Net touch API" itself, although there are several ways to hook into touch input from .Net (whether the core pointer stack or WPF's COM stack). The WPF team is aware of several issues with WPF's touch handling.

    --Rob

    Saturday, August 09, 2014 1:17 AM
    Owner
  • Thanks Rob, I look forward to your final resolution on this.
    Thursday, August 14, 2014 8:50 PM
  • Any more word on this?
    Monday, October 06, 2014 1:22 PM
  • This is being investigated for the next version.
    Thursday, October 09, 2014 2:10 AM
    Owner
  • Hi,

    May be it's related to predictive position ? Windows tries to predict next touch position to make your application feel more responsive : when you move your finger very fast, distance between position reported by device is important : predicted position should be less accurate and explain well your problem.

    Using RawPosition solve your issue but you probably have a little bit more delay in UI reactivity.

    RawPosition is probably useful for precise UI interaction like drawing where delay is not so important.

    Thursday, January 15, 2015 11:52 AM
  • Surprisingly, using raw input API's improved performance. Looks like the OS prefilters rather than employing predictive pre-generated data points that then require a correction. Perhaps this issue is a manifistation of gesture processing?
    Thursday, January 15, 2015 3:47 PM
  • I use GetUnpredictedMessagePos to solve the issue.

    http://msdn.microsoft.com/en-us/library/windows/desktop/hh969204(v=vs.85).aspx

    I agree that Microsoft should give us more details regarding Position, RawPosition & UnpredictedPosition.

    • Proposed as answer by pfresnay Monday, January 19, 2015 8:58 PM
    Thursday, January 15, 2015 3:57 PM
  • Very good; I hadn't noticed that API. Neither have some of the Microsoft staffers responding here.
    Thursday, January 15, 2015 7:53 PM