NARPS Project: the robustness of the scientific method put to the test
Date:
Changed on 20/11/2020
When 70 scientific teams from around the world study the same neuroimaging data and the same hypotheses, they do not necessarily come to the same conclusions. This is the main lesson of the international NARPS* project, recently published in Nature, which concludes that there is a need to make analysis methods more transparent. Two Inria researchers, Camille Maumet and Bertrand Thirion, participated in this open science experiment.
Why did you join the NARPS project?
"The most rigorous article ultimately delivers the researcher's opinion."
Bertrand Thirion, Research Director, head of the Parietal team
Camille Maumet, Inria Rennes – Bretagne-Atlantique : "My work focuses on analytical variability, i.e. the way in which the processing chain of a dataset - algorithms, software, machines, etc. - is modified in order to improve the quality of the data. - influences the final result. NARPS gave me the opportunity to appreciate the importance of this variability on a very large scale".
Bertrand Thirion, Inria Saclay – Île-de-France : "Neuroimaging is at the heart of my laboratory's activity; we develop our own software and NARPS allowed us to compare their results with those of dozens of teams. There have been similar projects in the past, but never on such a scale (see box). It was an opportunity to be seized."
Your opinion on the conclusions of the study?
"In the same study, it is better to avoid multiplying the data processing chains: this weakens the results."
Camille Maumet, Researcher, Empenn team
C.M.: "Our interim results are quite convergent, but our interpretations and final results diverge. This shows that we need to improve certain practices; in particular, collaborative work can help us make our approach more robust".
B.T.: "NARPS reminds us that the most rigorous scientific article ultimately delivers the researcher's opinion and not an absolute truth. Hence the need to also publish its intermediate results, so that others can replicate our experiments".
Does NARPS encourage you to evolve your approach as a researcher?
Dr. Jeremy Hogeveen, a researcher at the University of New Mexico participating in the NARPS project, said: "We can draw two conclusions from this study. First, we need to describe our methods in great detail when we publish articles, so that we can properly understand any inconsistencies between results. Second, to advance neuro-imaging research, it is essential that we and our colleagues around the world begin to adopt the best analytical processes available, in order to obtain stable results from one laboratory to another."
Read the testimonyC.M.: "I practice more and more "preregistration", i.e. the definition of the analysis method before recovering the data. This avoids the temptation to multiply the processing lines in the same study, which weakens the results".
B.T.: "My laboratory is increasingly sharing its data, for example raw images of the brain, with the difficulty of preserving the anonymity of the subjects. I would also like to experiment with cross-analysis between two teams. Each would evaluate the work of the other, which would greatly limit the risk of bias or error due to the complexity of our statistical methods".
To know more
* The NARPS method: bets and MRI capture of brain activity
The MRI images used for NARPS were taken on 108 volunteer subjects. Upon arrival at the laboratory they were offered a small amount of money and were offered bets with a win and loss of the same amount, or a win that was twice as high as the loss. With each decision (whether to bet or not), the MRI machine in which they were placed captured their brain activity.
The NARPS organizers had also defined nine brain zones, and asked the 70 teams to judge whether the activity in each of them changed as a result of the decisions they made. The researchers obtained fairly convergent intermediate results. On the other hand, 20 to 40 per cent of the teams arrived at significantly different conclusions from the group for the activity of five brain zones.
Research Teams Reach Different Results From Same Brain-Scan Data, The Scientist, 20/05/2020