Pre-registration and Open Science
Routing Summary
This folder covers how to pre-register research, why it works, and the ecosystem that supports it — fleshing out the practice that Forking Paths and Bayesian Approaches, Garden of Forking Paths, and Researcher Degrees of Freedom recommend and motivate. Based on Nosek, Ebersole, DeHaven & Mellor (2018), “The Preregistration Revolution,” PNAS 115(11):2600-2606.
- Need the big picture / where to start? → Pre-registration and Open Science - Overview
- Need the core epistemic argument (confirmatory vs exploratory, HARKing, why NHST assumes prediction)? → Prediction vs Postdiction
- Need the two mechanisms and results-blind peer review? → Pre-registration vs Registered Reports
- Need what a pre-analysis plan contains, or the registries (OSF, AsPredicted, AEA RCT Registry)? → Pre-analysis Plans and the Open Science Ecosystem
- Need what pre-registration does NOT fix, or rebuttals to common objections? → Limits and Objections to Pre-registration
Concept Map
| Concept | Note | Type | Depends On | Key Result |
|---|---|---|---|---|
| Pre-registration practice and ecosystem overview | Pre-registration and Open Science - Overview | overview | Garden of Forking Paths, Researcher Degrees of Freedom | Pre-registration makes transparent which analyses are predictions vs postdictions; culture provides means/motive/opportunity for rigor |
| Confirmatory vs exploratory and why the line blurs | Prediction vs Postdiction | concept | Garden of Forking Paths | NHST/ values valid only for prediction; hindsight bias + HARKing make postdiction masquerade as prediction |
| The two pre-commitment mechanisms | Pre-registration vs Registered Reports | concept | Prediction vs Postdiction | Registered Reports add results-blind peer review (in-principle acceptance), defeating publication bias |
| Pre-analysis plans and the registry ecosystem | Pre-analysis Plans and the Open Science Ecosystem | concept | Researcher Degrees of Freedom, Prediction vs Postdiction | A PAP pins down every analytic choice; OSF/AsPredicted/AEA RCT/ClinicalTrials.gov preserve it; blinding & cross-validation salvage non-ideal cases |
| What pre-registration does and doesn’t fix | Limits and Objections to Pre-registration | concept | Pre-analysis Plans and the Open Science Ecosystem | Reduces but doesn’t eliminate bias; makes poor practice detectable, not impossible; answers 9 common objections |
Notes
- Pre-registration and Open Science - Overview — CONTAINS: definition of pre-registration, the credibility chain, means/motive/opportunity framing, two mechanisms table, ecosystem summary, the honest caveat, idealized research cycle
- Prediction vs Postdiction — CONTAINS: prediction/postdiction definitions and synonyms table, HARKing, hindsight bias, vague predictions, why NHST assumes prediction, the bright-line operational rule, discovery science legitimacy
- Pre-registration vs Registered Reports — CONTAINS: pre-registration vs Registered Report definitions, two-stage results-blind workflow (Stage 1 IPA, Stage 2 execution review), what-each-fixes table, badges/Preregistration Challenge/TOP/legal mandates
- Pre-analysis Plans and the Open Science Ecosystem — CONTAINS: PAP contents checklist, strategies for deviations/assumption violations (sequential prereg, blinding, decision trees, SOPs), preexisting data, longitudinal/cross-validation, templates/replication, full registry list (OSF, AsPredicted, ClinicalTrials.gov, AEA RCT, RIDIE, EGAP, WHO), incentive layer
- Limits and Objections to Pre-registration — CONTAINS: governing “detectable not prevented” principle, three things it doesn’t fix (preregistered bias, program-level multiplicity, narrative cherry-picking), nine objection/response pairs, strong inference, empirical evidence (Kaplan & Irvin, Franco et al.)
Sources
- Nosek et al 2018 - The Preregistration Revolution.pdf — “The Preregistration Revolution” (Nosek, Ebersole, DeHaven & Mellor, 2018, PNAS 115(11):2600-2606)
See Also
- Research Methodology — parent index
- Garden of Forking Paths — the problem pre-registration addresses
- Researcher Degrees of Freedom — the analytic flexibility a pre-analysis plan removes
- Forking Paths and Bayesian Approaches — pre-registration as a complementary frequentist safeguard
- The Experimental Ideal — randomized design as the complement to a pre-specified analysis