The data and code that underpin a research study can tell us what was actually done, and how. In a medical study, for example, how were patients selected for inclusion? How were missing values handled? How were the results evaluated?
Access to a study's data and code help researchers to answer these questions, to examine the methods, and to build on the work.
“Studies involve many choices about how to prepare data, develop models, and evaluate performance. These are sometimes called ‘researcher degrees of freedom,’” says Tom Pollard, a research scientist at MIT's Laboratory for Computational Physiology and Computer Science and Artificial Intelligence Laboratory. “The choices we make affect the results, and a methods section in a paper can only capture so much detail.”
While sharing code can be a first step, in order for code to be understood and reused, it must be sufficiently structured and described. The problem is that, right now, there is little guidance for researchers on what constitutes a well-organized, well-described collection of code.
This is where an international team led by researchers at MIT and Dana-Farber Cancer Institute sought to help, by examining how code is currently shared in clinical prediction studies. Such studies develop or evaluate models to estimate patient outcomes, such as the risk of developing a disease. Because these models inform decisions about patient care, researchers need to be able to scrutinize how they were built and tested.
Using large language models (LLMs), the researchers examined nearly 4,000 papers to identify statements about shared code.
Source link







