UTSK: a toolbox for medical robotics guided by ultrasound imaging

Unlike x-rays, scanners or MRIs, ultrasound probes deliver images in real-time. This is a real advantage when it comes to controlling medical robots using visual servoing.

A member of the robotics team Rainbow, Alexandre Krupa has been working in this field for the past thirteen years. The USTK library contains tools designed by Krupa and his PhD students over the course of their research projects. 
“We’re now looking to make these tools available to a wider pool of people. Other laboratories and R&D centres may wish to use them to prototype or develop their own real-time medical applications. ”

The library is comprised of a number of modules. Firstly: the data structure “This is the heart of USTK ”, explains Fabien Spindler, research engineer . It is used to characterise images. Obviously, formats already exist for the machines used by practitioners. For the purposes of our research, however, we needed a more comprehensive data structure that would enable us to access the pixels of images, as well as all of the geometric parameters of the ultrasound probe. ”

“Prior to becoming a collection of pixels, the data measured by the machine is a wave, a radio frequency signal penetrating organic matter, the density of which may vary. The issue here is that some probes are linear, while others are curvilinear. From a geometrical perspective, the way in which images are formed is intrinsically different. Furthermore, in addition to probes for 2D, there are also probes for 3D. These are often motorised, and they too have their own systems. You could have a motor performing a curvilinear sweep using a convex transducer. ”

There is an additional complication with these curvilinear sweeps. “Spatial resolution is lost ”, explains Alexandre Krupa. “The data arrives in a simple table containing the results for each individual firing line. You then need to reconstruct the information using Cartesian coordinates. In order to do so, you need to know the geometric parameters of the probe: its radius of curvature, the surface area of the transducer, etc. The data structure employed in USTK makes it possible to code the information at each stage, from the radio frequency (RF) signal through to the geometrically rebuilt image. ”

The second module is used to acquire data from a 3D Ultrasonix ultrasound. “That’s the machine that we have, as is the case with many other laboratories ”, explains Fabien Spindler. “This station delivers an open solution that is particularly well-suited to research given that it allows you to access to the device's internal parameters. We are able to acquire RF images, post-formed (i.e. reconstructed) images and intermediate images, all in real-time, without having to wait for the data to be written on to a hard drive. This is of vital importance for anyone looking to develop robotic controls. "

Confidence map

The third module is used to measure image quality. “Ultrasound images contain artefacts”, explains Alexandre Krupa. “When waves are unable to penetrate bone, for example, this results in shadows, preventing us from seeing what lies behind them. There’s information missing, and this creates uncertainty. The algorithm that we have developed supplies a confidence map capable of telling us: I am fully confident about this pixel, it actually contains information. For this or that other pixel, however, I’m slightly less sure. ”
 So what will this map be used for? “To boost reliability. It’s useful, for example, when it comes to augmented reality. Ultrasound imaging is the most commonly used type in a pre-operative context, but it is also the hardest to interpret. You have the shadows we spoke about earlier, but you also have noise. In order to improve reading, one option is to superimpose the mesh model of a pre-operative image obtained using MRI, resulting in higher quality. The confidence map makes it easier to adjust the two images: pixels in the shaded section will not be processed in the same way as pixels in areas where visibility is much better. This ensures that the mesh takes the best possible shape in relation to what we know about the ultrasound information. ”

The fourth module’s role is to visualise the data, while the fifth module is used to store it on a hard disk. The sixth is used to track areas of interest within the images. “On the screen, there will be movement either if the patient moves or if the practitioner moves the probe. With our tool, all you need to do is indicate an area of interest and it will remain at the centre of the image, no matter what happens".

Needle detection

The last module deals with needle detection. “We’ve been working on a support system for guiding the insertion of flexible needles. The idea is to use its flexibility in order to enable it to follow a given trajectory. The user begins by clicking on the target zone. The system will automatically advance the needle using the deformation induced by the resistance it encounters when it comes into contact with tissue. Its edge is bevelled on just one side, meaning that the forces applied are not symmetrical. If you push it, it loses shape, going off in one direction. When you turn the bevel, it then goes off in the other direction. However, in order to do this, real-time image detection is required. We have an algorithm capable of doing this with 2D sequences, and we’re currently working on a 3D version. ”

A virtual probe

Users of USTK will find various different applications built using these modules and supplied for illustrative purposes. “We also supply data sets for users who don’t have their own ultrasound unit. Consider the example of someone looking to test an algorithm capable of automatically positioning a probe in such a way as to obtain the best possible image of an organ: if they don’t have a robot, we offer a virtual probe capable of operating within the ultrasound space. They will be able to move around inside this space, depending on the controls they apply to this virtual probe. They will then be able to set up visual servoing, enabling them to keep an area of interest in the middle of the screen, for example. All of this can be prototyped in a simulation. ”

Beginning in September 2016, Inria funded a Technology Development Initiative (ADT) for the industrialisation process. “That meant we were able to recruit an engineer on a two-year contract ”, explains Fabien Spindler. “A number of research centres have already expressed an interest, including the LIRMM in Montpellier, The Technical University of Munich, John Hopkins University in Baltimore and the IRT b<>com in Rennes. USTK is distributed under a GPL (General Public License), but we can also offer proprietary licenses for manufacturers or laboratories looking to develop commercial applications. The library can be downloaded from our GitHub space. ”