Reproducibility in Ecological Research

How can we improve the reproducibility of ecological experiments? In a paper published this month in Nature Ecology and Evolution, an international team led by Alex Milcu that included HF Senior Ecologist Aaron Ellison showed that a deliberate introduction of controlled systematic variability (CSV) in experimental designs may increase the reproducibility of experiments done at multiple sites or in multiple laboratories.

According to the researchers, one would expect that if one group tried to reproduce the experimental results of another group, it would be critical to reproduce each methodological step, using, for example, identical seed sources, soils, and reagents.

But Milcu's team tested the controversial hypothesis that such stringent levels of environmental and biotic standardization actually reduce reproducibility by amplifying the impacts of laboratory-specific environmental factors not accounted for in study designs. This hypothesis suggests that deliberately introducing “controlled systematic variability” (CSV) into an experiment should increase the likelihood that the experiment can be reproduced.

To test this hypothesis, 14 European laboratories ran a simple microcosm experiment using grass (Brachypodium distachyon) monocultures and grass + legume (Medicago truncatula) mixtures. Each laboratory introduced environmental, genotypic, or both environmental and genotypic CSV in either growth chambers or glasshouses.

Introducing genotypic CSV increased reproducibility in growth chambers, which have stringent environmental controls, but not in glasshouses, which do not. In contrast, introducing environmental CSV had little effect on reproducibility in either growth chambers or glasshouses.

Researchers and pundits have suggested multiple causes for the ‘reproducibility crisis’ in science, and ecologists have suggested that reproducibility in ecological studies may be particularly difficult. This study's results show that the deliberate introduction of known, quantified genetic variability may be a simple way to increase the reproducibility of ecological studies done in stringently controlled environmental conditions.

• Read the full article.
• For more reading about reproducibility in ecological research: A. Shavit & A. M. Ellison (2017). Stepping in the Same River Twice: Replication in Biological Research. Yale University Press, New Haven, Connecticut.