OverviewTeaching: 5 min Exercises: 0 minQuestions
Can you cite some examples where issures related to reproducibility have been documented to be a problem?Objectives
Show some of the evidence that reproducibility is causing problems in neuroscience.
Image from Prinz:
Insert here some words of description related to the Prinz figure…
Re-executability (publication-level replication). The exact same data, operated on by the exact same analysis should yield the exact same result. This is currently a problem since publications, in order to maintain readability, do not typically provide complete specification of the analysis method or access to the exact data.
Generalizability: We can divide generalizability into three variations:
Generalization Variation 1: Exact Same Data + Nominally ‘Similar’ Analyses should yield a ‘Similar’ Result (i.e. FreeSurfer subcortical volumes compared to FSL FIRST)
Generalization Variation 2: Nominally ‘Similar’ Data + Exact Same Analysis should yield a ‘Similar’ Result (i.e. my kids with autism compared to your kids with autism)
Generalized Reproducibility: Nominally ‘Similar’ Data + Nominally ‘Similar’ Analyses should yield a ‘Similar’ Result
Because we do not really characterize data, analysis and results very precisely in the current literature, ‘Similar’ has lots of wiggle room for interpretation (both to enhance similarity and discount differences).
Ir-reproducibility costs society dearly