Towards more reproducible research in artificial intelligence for medicine

The reproducibility of the results is an important issue for the reliability of biomedical research. At present, there is a real difficulty in reproducing the results of the studies, notably linked to the data management and processing methods which are not always sufficiently accessible or standardised. This challenge is present in many fields, in particular the development of artificial intelligence tools for medicine.

The ARAMIS team has addressed this problem in the more specific context of medical decision-making support systems and, in particular, support in diagnosing Alzheimer's disease. 

“It is a field in which an enormous amount of work has taken place over the last 10 years or so, with numerous approaches in decision-making support systems. Most of these studies use the same public patient database, ADNI (Alzheimer's Disease Neuroimaging Initiative) and yet it is impossible to compare them with each other”, Olivier Colliot underlines.

Indeed, each study in this field will use a sub-sample of different patients from this database, with criteria that are not always precisely defined. These data then undergo processing that varies from one study to another. The result: it is often difficult to draw conclusions from these studies and their clinical transfer remains poor. 

In an attempt to address this problem, the ICM researchers have developed a set of open source software tools in order to very easily standardise and reproduce the evaluation results of these diagnosis support systems. It involves a completely automated workflow, starting from the database, and to which all that is needed is to add the artificial intelligence algorithm developed, for example to distinguish between a patient with Alzheimer's disease from other pathologies, and which will lead to a decision. 

At present, this software is intended for the analysis of anatomic MRI and PET (positron emission tomography) data - important tools in the diagnosis of Alzheimer's disease. The researchers wish to continue the development of the software platform in order to extend it to other types of data, but also to other pathologies such as Parkinson's disease.

Olivier Colliot concludes : 

 

We want to move towards a better standardisation of the validation studies of medical decision-making support systems and greater reproducibility, and therefore more reliable results that can be transferred in a more efficient way to the clinic

 

 

• Software that is open source and made available by the ICM
• ARAMIS is an Inria team, based at the ICM, jointly with the CNRS, Inserm and Sorbonne University

Sources : Reproducible evaluation of classification methods in Alzheimer's disease: Framework and application to MRI and PET data. Samper-González J, Burgos N, Bottani S, Fontanella S, Lu P, Marcoux A, Routier A, Guillon J, Bacci M, Wen J, Bertrand A, Bertin H, Habert MO, Durrleman S, Evgeniou T, Colliot O; Alzheimer's Disease Neuroimaging Initiative; Australian Imaging Biomarkers and Lifestyle flagship study of ageing. Neuroimage . 2018 Aug 18; DOI:183:504-521.