Industry 4.0: are robots and their operators made to collaborate?

Cobotics: making work less strenuous

Industrial robots have been used in factories for decades. But until now they have been limited to simple, repetitive tasks, operating within a secure perimeter into which any human intrusion would trigger an automatic stop. “With AnDy, our focus was on the next generation, the so called industry 4.0 : exoskeletons, cobots and humanoid robots. These are worn by operators or share a space with them, helping them with their tasks”, explains Serena Ivaldi, research scientist in the Larsen project team at Inria Nancy - Grand Est. “The aim is to make work less strenuous through collaboration that is as smooth and as safe as possible”.

But this sort of innovation is no easy feat. Cobots must be capable of adapting to the dynamics of the human body and recognising an intention at the first signs of movement, and must not force operators into non-ergonomic positions. “A good comparison is dancers”, says Francesco Nori, coordinator of the AnDy project and director of the robotics lab at Deepmind (Google). “Cooperation is not always verbal and planned. It is implicit, resulting from a mutual understanding whereby it is possible to guess the next step”. 

A motion tracking suit for recognising movements and postures

Such a target was never going to be met in four years. However, the researchers were able to reach a number of important milestones in the world of industry. “The greatest achievement was the development of a sensorized suit equipped with 17 motion sensors, which can be used to automatically track and determine the movements and posture of an operator:  picking up an object, carrying it, putting it down, turning something, etc.” explains Francesco Nori.

This bodysuit was used to build a database which has already been downloaded more than 10,000 times (see insert). The AnDy project also led to the emergence of a young Italian startup, iFeel: shoes fitted with sensors to measure foot pressure, expanding the field of automated motion analysis.

Automatic ergonomic risk assessment: a breakthrough for industry 4.0

Over and above these innovations, one significant breakthrough for AnDy was the automation of ergonomic risk assessment for specific tasks or workstations, to which Inria made a significant contribution.

“We identified the posture and movements of the operator wearing the Xsens  motion tracking suit”, explains Serena Ivaldi, “and measured stress on joints in the mechanical sense: were the knees, wrists or lumbar belt subject to intense or repeated effort, which can be as harmful as a non-ergonomic position?” The intensity of effort is displayed on a  digital human model in a physical simulation created by the team, whose joints change colour from green to red depending on the amount of stress or ergonomics risk score for each body part.

This fully-automated assessment is quicker, more in-depth and more accurate than the traditional method, which involves an ergonomist making observations, ticking boxes and calculating a score on a paper sheet. It will be used to determine the movements of industrial cobots required to interact with their operators, ensuring optimum assistance, and has already been tested on a number of exoskeletons.

A method already tested on several exoskeletons

The breakthroughs from AnDy proved their worth before the project was even completed.  In early 2020, at the height of the Covid-19 pandemic, Inria used them as part of the ExoTurn project to select and test exoskeletons for helping staff to turn ICU patients during prone positioning manoeuvres multiple times a day. “The passive lumbar support exoskeleton, which was used within the ExoTurn project, reduces the strain off of the back muscles by shifting it to other parts of the body”, explains Serena Ivaldi. “Care had to be taken to ensure no new areas of discomfort were created and that no changes to medical procedures were required.” More than 60 ICU staff tested this device between April 2020 and May 2021.

© Inria / Photo J. Galet

In the same vein, researchers from AnDy collaborated with the German firm Ottobock on evaluating and improving their passive exoskeleton Paexo Shoulder. This is used to assist operators carrying out demanding physical tasks with their arms raised, in sectors such as construction and vehicle repairs.

A cobot capable of guessing what an operator is about to do next

Inria’s other main innovation for AnDy was linked to the cobot’s ability to anticipate, i.e. be able to determine a human user’s intentions and adjust its movements accordingly, making cooperation smoother. Using the AnDyDataset database, the team was able to train machine learning models to recognise the actions of an operator and accurately predict what they would do next by observing what they were currently doing. These models have been proven to be reliable and quick enough to operate in real-time.

The researchers validated their models using a specific experiment, teleoperating a humanoid robot under long delays of one to two seconds between instructions being received and tasks being executed. “Even for simple tasks such as picking an object up or moving it, this delay can be a nuisance for operators. We compensated for this using our models and their capacity for anticipation, giving the individual with the remote control the impression that their commands were in sync with what the robot was doing.” 

© Inria / Photo F. Nussbaumer - Signatures

The Inria team involved in the AnDy project also tested out iCub, its humanoid robot, in situations without any human interaction where fluidity is key. iCub was able to walk in a flexible way, open doors, stoop down to pick objects up off the ground and even try out a few dance moves! Perhaps Francesco Nori’s analogy with the dancers is not so far from reality...