ReproNim Statistics Module: Reference

Key Points

Module overview
  • Reproducible analysis is strongly impacted by statistical analyses.

  • Reproducible research requires understanding the notions of sampling, testing, power, model selection.

Statistical basis for neuroimaging analyses: the basics
  • Be familiar with the concept of sampling

  • Know what we call a distribution, a p-value, a confidence interval

  • Have some knowledge of Bayesian statistics and model comparison

  • This is in line with our overall goal of making science (including scientific training) more open.

Effect size and variation of effect sizes in brain imaging
  • Effect sizes come in many forms

  • Significance is not relevance

  • Difference between the raw effect size and the cohen’s d effect size

  • How can the effect size vary? Why is it important to know about this?

  • Effect sizes are under reported, not well understood, and are crucial for our scientific understanding. Let’s fix this.

P-values and their issues
  • A p-value does not give you an idea of the importance of the result

  • A p-value should always be complemented by other information (effect size, confidence interval)

Statistical power in neuroimaging and statistical reproducibility
  • The lack of power is much more problematic that it seems at first sight. - It would usually lead to wasted resources - If an under powered study yields some significant effects, these are likely to be overestimated - If an under powered study yields some significant effects, these are less likely to replicate

The positive Predictive Value
  • A significant (say at the 0.05 level) may have a low chance of replication.

  • The PPV estimates the probability that the alternative hypothesis \(H_A\) is true given that the test is significant at some \(\alpha\) level.

  • This probability depends on several factors such as power \(\beta\), \(\alpha\) level, but also the prior chance that \(H_A\) is true.

Cultural and psychological issues
  • A summary of everything so far

FIXME: more reference material.