Hector: augmented reality helping surgeons

Allow me to introduce myself. My name is Hector.

I was born within the MIMESIS team, which works in close partnership with the Strasbourg IHU (institute for image-guided surgery), which specialises in the treatment of diseases of the digestive tract. I was designed to supplement the pre-op images (or virtual 3D models generated using these reconstructions) currently available to surgeons and interventional radiologists - images which help them to better plan operations. How do I do this? I provide them with augmented reality technology, which allows them to view the internal anatomy of an organ during an operation, as though it were transparent.

My goal is to supply realistic, real-time information on movements and changes to the shape of organs.

Through my neural network, I combine numerical simulation and artificial intelligence technology in order to learn how to predict potential movements and changes to the shape of organs during operations. I have even been tested in a clinical context with our partners from the hepatology department of Paul-Brousse hospital in Villejuif (Val-de-Marne).

Why is this important? “Regarding the liver, which is what we are currently focusing on, this technology should help surgeons to remove certain tumours while avoiding  contact with the adjacent vascular network”, explains Stéphane Cotin, director of research for the MIMESIS project team, which specialises in real-time simulation and modelling for use in medicine (see inset). “It’s essential for the liver to be able to maintain its vital functions and to regenerate properly after surgery.” 

My technological features

I use three parallel data sources in order to estimate - in real-time and with maximum precision - any changes to the shape of an organ: images from the operating theatre; the three-dimensional model of the organ created prior to the operation; and deformation calculations. These calculations draw on standard deformation models, numerical calculation methods used to quickly solve complex equations, and a type of artificial intelligence trained “to reproduce the deformable behaviour of an organ”, explains Stéphane Cotin. “What this means is that we are now capable of making extremely quick and quite accurate predictions of how an organ will change shape. We are also able to display the updated images in real time, superimposed over those from the operation.” 

This multidisciplinary research draws partly on the SOFA* simulation platform, which was developed within MIMESIS and other Inria teams and which is maintained by a consortium to enable the scientific community to reap the benefits.

My creator and project developer

The man behind my development, Stéphane Cotin, has spent his whole career as a researcher studying modelling and simulation in the medical field. “I spent around ten years in Boston in the USA at a lab with close ties to a hospital: it was then that I developed an interest in how IT could be used in medicine.” Upon his return to France, he contacted Inria Lille, then Strasbourg IHU, where “surgeons were looking into the possibility of using 3D models constructed using medical images while the operation is taking place - as opposed to simply during the diagnostic phase.” Co-founder and current scientific consultant for InSimo, a Strasbourg-based publisher of software for surgical training, Stéphane Cotin is also in charge of the MIMESIS project team and Hector, and remains heavily involved in clinical innovation.

Upcoming challenges

There is still work to be done on both Hector's precision and robustness, the goal being to ensure that predictions made by the system are reliable in order to better inform users of risks or in cases where the quality of a model seems inadequate as a result of a lack of data of sufficient quality. Lastly, as we have seen with medical robotics, which has now established itself in all operating theatres, Hector could conquer a range of market segments in the future. The software’s potential has already been shown during liver surgery, and other possible uses are currently being assessed, including in neurosurgery.

More about

The MIMESIS project team

https://mimesis.inria.fr/

The work carried out by the researchers and engineers who form this team is focused on numerical medical simulation in surgical training and on intraoperative assistance through augmented reality.