An Inria innovation to improve smartphone ergonomics

Smartphones, tablets, laptops and touchscreens are all part of our daily lives, and one of the priorities for designers is to make using them as enjoyable as possible. When a tech giant was developing the latest version of its smartphone a few years ago, it chose to increase the refresh rate of the screen, thereby improving the fluidity of the display. But early feedback from users failed to meet their expectations. Web pages weren’t scrolling as fluidly as they should have been – appearing as though the display was shaking - a phenomenon which experts termed “jitter”.

Although they were able to quickly find a solution, they still had to identify what had caused the problem in order to prevent it from affecting future versions and to protect the company’s brand image. This was the challenge that was successfully overcome by two researchers and a PhD student in IT from Loki, a project team that is a joint undertaking involving Inria and the University of Lille through the University of Lille Inria Centre and the CRIStAL laboratory*: Mathieu Nancel, Géry Casiez – who is also a professor at the University of  Lille – and Axel Antoine.

*UMR 9189

The results of their research, published in an article that was well received by the scientific and industrial community, owed nothing to chance, both scientists having studied human-machine interaction for a long time. “Our goal is to make improvements to generic technology in order to develop systems that will allow users to experiment with and utilise digital information in a more intuitive and immersive way”, explains Géry Casiez. This involves technology such as virtual reality headsets, interactive displays and touchscreens.

How did they end up studying the “jitter” issue? Through researchers at the University of Waterloo in Canada whom they had regularly collaborated with, Mathieu Nancel and Géry Casiez were given an opportunity to make a presentation to Google engineers on their work on the problem of latency - the time it takes for a computer system to process and react to user activity. Google was so taken by their approach to the subject that the American tech giant decided to give their research project a “Google Research Award” as part of its Research Scholar Program. This came as a pleasant surprise to the two researchers - only around 15% or so of candidates are accepted, putting them in select company - and more importantly gave them the chance to work together to tackle the issue of “jitter”.

It wasn't long before they identified what was causing the problem: “When Google engineers developed the Pixel 4.0 in 2019 they increased the refresh rate from 60 Hz to 90 Hz, meaning that information displayed on the smartphone screen was updated 1.5 times quicker than with the previous version”, explains Mathieu Nancel. “But at the same time, the frequency of acquisition of touch information (such as when the user scrolls down using their thumb, or touches the screen with their finger to select something) stayed the same: 120 Hz. And this change in the refresh rate relative to the acquisition frequency is what caused the ‘jitter’.” 

To illustrate this phenomenon, imagine a child (the display) walking alongside an adult (the touch acquisition). The adult’s steps are twice as long as the child’s, but if the child walks at twice the frequency of the adult, then they will be in step with each other. And if every two steps the child plants its foot at the same time as the adult, then they will be walking in sync with each other. But if the frequency the child is walking at is not a multiple of that of the adult, then they will fall out of sync. This is precisely what happened with the smartphone screen: synchronicity was possible with a refresh rate of 60 Hz and an acquisition frequency of 120 Hz (double the refresh rate), but not with a refresh rate of 90 Hz. Each time the screen refreshed, the system thought that the last position of the finger had just been captured, but with the 90 Hz - 120 Hz example the latency varied each time.

Clearly the reality was far more complex, but this was the principle: the key to the problem lay in this loss of synchronicity. And the researchers believed  that it could be solved using a mathematical model. “Our model is highly generic, which is what makes it so useful”, explains Géry Casiez. “It can be used to analyse existing systems, to simulate the performance levels of new systems, or even to develop technological solutions before then analysing and demonstrating their effectiveness. This makes testing far easier, while significantly cutting costs in terms of the physical materials needed.” Not to mention unrivalled savings in terms of time and resources.

The model developed by Mathieu Nancel and Géry Casiez and the first conclusions from their study on “jitter” were presented to the UIST conference (User Interface Software and Technology) in 2020. A version of an algorithm employing this mathematical model was released a few months later in the open-source software program Chromium. The work of the three researchers was remarkable both for its quality and its speed: only four years passed between the model being designed and being applied to the products in question.

The researchers found this to be a particularly exciting scientific adventure:

It was a brilliant experience, delivering everything you might dream about when it comes to research work: identifying a problem, developing a model in order to understand it, and producing a tool in order to provide a solution, says Géry Casiez.

As Mathieu Nancel concludes,“We’re rather proud of this project. Our contribution perfectly illustrates the technical expertise required in order to develop the digital products we’re all so familiar with.”  And the two scientists have no intention of resting on their laurels either. They’ve got their sights set on coming up with new applications for their research as they seek to continue to improve the experience of users.

• Géry Casiez: “There shouldn’t be any latency with touchscreens”, Chut! 19/4/2023.
• Lecture “Human capacities and interactive systems” with Gery Casiez at the Collège de France (video), 22/3/2022
• A screen that you can really control with your finger and your eye, Inria, 22/5/2019.