A screen that you can really control with your finger and your eye

Why is it that objects moving around on a touchscreen are not able to accurately follow the finger guiding them? There are at least three reasons for this. Firstly, the exact movement of the finger is not reproduced identically on the screen, such as when an object is being selected, for example. When it comes to scrolling, complex mathematical functions known as “transfer functions” come into play.

Less delay, but side effects

The input signal from the finger then has to pass through the touch pad, the operating system, the graphics card and the screen, resulting in a delay, or “latency”, of a few dozen milliseconds (ms). The problem here is that users start to notice this latency from 10 ms onwards, sometimes even less.

Designers develop “latency compensation” algorithms in order to reduce these delays which can have a negative impact on user performance. These algorithms are effective, but come at the cost of side effects, which increase with the intensity of the compensation: the object being moved might lag behind, for example, or shake, skip, go up and down, move too quickly, change direction, etc. Latency can be compensated partly, but not entirely.

When they launched the National Research Agency (Agence nationale de la recherche) ANR TurboTouch project back in 2014, the researchers from Inria, with support from a team based in Marseille, sought to tackle both “transfer functions” and latency compensation. Five years down the line, what do they have to show for their work? An innovative latency compensation algorithm that has already attracted the attention of a well-known software publisher.

 The user is always right

Three teams worked together on this project: Loki (Inria), specialists in Human-Computer Interaction or HCI; Valse(Inria), specialists in automatic control and signal processing, and the Institut des Sciences du mouvement(Université Aix-Marseille - CNRS).

“This gives us access to expertise in mathematics, HCI and the movement of the human body ”, explains Géry Casiez from Loki. "It was necessary to bring these different cultures together, and we all complement each other really well. A mathematician validates an algorithm using a digital simulation tool, such as Matlab. For HCI specialists, the user is always right! ”  Looking back on their achievements, Géry Casiez pinpoints a number of significant advances. In relation to transfer functions, the project led to the creation of a screen scrolling mode employing the use of force applied to the screen. This is already available.

Measuring latency, the first step towards reducing it

When it comes to pointing at objects, the mathematicians working within the Valse team have developed a dynamic model capable of describing the task in its entirety: “before, if you wanted to make pointing more effective, you first had to create its transfer function ‘blind’ before evaluating it in relation to others ”, explains Denis Efimov from Valse. "With our model, it is now possible to gradually optimise each stage of the process. ”

When it comes to latency, where the challenge is to find the right balance between compensation and its side effects, researchers lacked simple, precise measurement tools. So they decided to create them themselves. They invented a device used to time latency using a vibration sensor placed on the finger and a photodiode to detect the screen's response.“This allows us to continuously measure latency while a user is interacting with the screen”, explains Géry Casiez. 

Another development involves software tools used to detect and count secondary effects. This means that researchers know if the screen is skipping or shaking or if there are any errors with the trajectory, etc, comparing these measurements to what users experience: in the event of a latency compensation algorithm causing a frequent but relatively unobtrusive side effect, it can be used.

The TurboTouch algorithm, a three-stage rocket

What really stands out about TurboTouch, however, is this new latency compensation algorithm, which helps reduce latency by between 32 and 48 ms without users experiencing any irritating side effects. By way of comparison, existing tools are limited to reductions of between 10 and 15 ms. These algorithms all draw on the same principle: to use the movement of the finger to predict the rest of the trajectory so as to be able to carry out the ongoing task with precision; positioning the object being moved in the right location, for example. To refine its predictive capabilities, the TurboTouch algorithm combines three techniques.

The first involves estimating not only the speed at which the finger accelerates, but also any jolts or bumps (the first, second and third derivatives of movement). The algorithm then adjusts this prediction based on the speed of the finger: if the movement is slow, there’s no need to push the calculation too far because the actual position of the object being moved will be easy to predict.

Lastly, a filter developed by Loki in 2012 provided the finishing touches by toning down shaking. “The majority of latency compensation algorithms calculate the difference between the successive positions of the finger and the object being moved ”, explains Rosane Ushirobira from Valse. “Ours only takes into account their positions at that particular moment, t, which is what makes it so simple. It also goes as far as the third derivative, however, thus explaining its precision. ” Lastly, the algorithm performs its calculations in just a few milliseconds, roughly a thousand times quicker than the milliseconds of latency it has to compensate for: it will always have a decent head start.