Joelle Pineau has been leading an effort for eradicating reproducibility crisis in AI research with encouraging researchers to open the core, running the reproducibility challenge and introducing checklist for scientists during the major AI conference held from December 8 to 14.
This year, Conference on Neural Information Processing Systems (NeurIPS) has attracted 13,000 researchers to Vancouver from all over the world, and Joelle Pineau hold a new role dedicated to reproducibility.
In the interview to Nature, she reported that scientists have been responding positively to the initiative, since reproducibility is of crucial importance to future development of the field. For example, efforts lead to increase in papers with open code: this year 75% of paper opted to include a link to the code in their submission papers.
The reproducibility challenge, a competition where scientists recreating each other’s research outcomes, has reportedly attracted 173 participants from 73 institutions. Challenge reports are being shared in OpenReview, and some of the best works will be published in the journal ReScience C, shared Pineau.
Joelle Pineau also shared that the motivation behind her efforts is a result of her own frustration in trying to reproduce some other scientists’ results and scientific community support. In her words, the main challenge is that scientific papers do not always provide all necessary details, or even worse, provide misleading information, so the reported results look more impressive.
Many in the scientific community, including Pineau, share an opinion that now is exactly the right time to establish methodological norms in the industry due to the rise of the scientific publications in the field. In the annual AI Index 2019 report researchers from Stanford University have reported over 300% growth in the volume of peer-reviewed research articles on AI technology this decade, and over 800% increase in some specialised conference attendance rates in a period of 5 years.
Photo credit: Facebook