Gelman & Loken (2013) — Front Matter, Borges Epigraph and References
Summary
This note records the parts of the paper that are not argument: the title and its literary source, the Borges epigraph, the abstract, the author affiliations and acknowledgments (p. 1), and the full reference list of 31 entries (pp. 15-17). For each reference it records where the paper cites it and which note in this set covers that passage. Three references are listed but never cited in the text: Gelman (2014), Gelman & Tuerlinckx (2000) and Gelman & Weakliem (2009).
Overview
The paper is a 17-page working paper dated 14 Nov 2013. Its full title is “The garden of forking paths: Why multiple comparisons can be a problem, even when there is no ‘fishing expedition’ or ‘p-hacking’ and the research hypothesis was posited ahead of time” (p. 1). The argument itself is summarised in Gelman Loken 2013 - Garden of Forking Paths - Overview.
Main Content
Title and epigraph (p. 1)
Epigraph (p. 1)
“I thought of a labyrinth of labyrinths, of one sinuous spreading labyrinth that would encompass the past and the future … I felt myself to be, for an unknown period of time, an abstract perceiver of the world.” — Borges (1941)
Where the title comes from. The reference list (p. 15) gives the source as Borges, J. L. (1941), El jardín de senderos que se bifurcan (spelled “El jardin” without the accent in the source), Buenos Aires: Sur. The English translation is by D. A. Yates (1964), in Labyrinths: Selected Stories & Other Writings, pp. 19-29, New York: New Directions. The Spanish title translates as “The Garden of Forking Paths”, which is the paper’s title. The image fits the argument: every analysis the researcher could have run, given other data, is a path in the labyrinth, and the p-value has to account for all of them (§4.1, p. 11; see Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons). The same story also names McElreath’s unrelated Garden of Forking Data.
Abstract (p. 1)
The abstract makes three claims. Researcher degrees of freedom can produce a multiple-comparisons problem “even in settings where researchers perform only a single analysis on their data”. The problem arises because there can be many potential comparisons “when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of fishing or examining multiple p-values”. The examples are published papers whose analysis decisions were “theoretically-motivated based on previous literature” but whose details of data selection and analysis “were not pre-specified and, as a result, were contingent on data.”
Authors, affiliations and acknowledgments (p. 1 footnotes)
| Item | Content |
|---|---|
| Andrew Gelman | Department of Statistics, Columbia University, New York |
| Eric Loken | Department of Human Development and Family Studies, Penn State University |
| Thanks | Ed Vul, Howard Wainer, Macartan Humphreys and E. J. Wagenmakers, “for helpful comments” |
| Funding | ”the National Science Foundation for partial support of this work” |
Several of the people thanked are also cited: Vul et al. (2009), Humphreys, Sanchez & Windt (2013) and Wagenmakers et al. (2011).
Reference list (pp. 15-17)
All 31 entries, in the source’s order. “Cited at” gives the pages where the paper cites each work (citations inside the reference list itself are excluded).
| # | Reference | Cited at | Covered in |
|---|---|---|---|
| 1 | Beall, A. T., & Tracy, J. L. (2013). Women are more likely to wear red or pink at peak fertility. Psychological Science. | pp. 8-10, 12-13 | Gelman Loken 2013 - Data Exclusion, Coding and Pooling Forks in the Red-Pink Study |
| 2 | Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. JPSP 100, 407-425. | pp. 5, 12 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies; Gelman Loken 2013 - Three Reactions and the Bayesian Reading of Bem’s 53 Percent |
| 3 | Bem, D. J., Utts, J., & Johnson, W. O. (2011). Must psychologists change the way they analyze their data? JPSP 101, 716-719. | p. 6 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 4 | Bennett, C. M., Baird, A. A., Miller, M. B., & Wolford, G. L. (2009). Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon. Poster, Human Brain Mapping conference. | p. 5 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 5 | Borges, J. L. (1941). El jardin de senderos que se bifurcan. Sur. Trans. D. A. Yates (1964), Labyrinths, 19-29. New Directions. | p. 1 (epigraph) | this note |
| 6 | Box, G. E. P. (1997). Scientific method: The generation of knowledge and quality. Quality Progress (Jan), 47-50. | p. 13 | Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 7 | Button, K. S., Ioannidis, J. P. A., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14, 365-376. | pp. 3, 7, 11 | Gelman Loken 2013 - Four Testing Procedures and the Data-Contingent Decision Variable; Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 8 | de Groot, A. D. (1956). The meaning of the concept of significance in studies of various types. Nederlands Tijdschrift voor de Psychologie 11, 398-409. Trans. 2013 by E. J. Wagenmakers et al. | pp. 2, 15 | Gelman Loken 2013 - Four Testing Procedures and the Data-Contingent Decision Variable; Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 9 | Durante, K., Arsena, A., & Griskevicius, V. (2013). The fluctuating female vote: politics, religion, and the ovulatory cycle. Psychological Science 24, 1007-1016. | pp. 6-7, 9-10, 12 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 10 | Francis, G. (2013). Replication, statistical consistency, and publication bias (with discussion). J. Mathematical Psychology 57, 153-169. | p. 2 | Gelman Loken 2013 - Four Testing Procedures and the Data-Contingent Decision Variable |
| 11 | Gelman, A. (2004). Treatment effects in before-after data. In Applied Bayesian Modeling and Causal Inference from an Incomplete Data Perspective, ch. 18. Wiley. | p. 5 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 12 | Gelman, A. (2013a). Too good to be true. Slate, 24 Jul. | pp. 1, 4 (and probably p. 8, see errata) | Gelman Loken 2013 - Four Testing Procedures and the Data-Contingent Decision Variable; Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 13 | Gelman, A. (2013b). Is it possible to be an ethicist without being mean to people? Chance. | p. 14 | Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 14 | Gelman, A. (2014). The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. Journal of Management. | not cited in text | this note |
| 15 | Gelman, A., Hill, J., & Yajima, M. (2012). Why we (usually) don’t have to worry about multiple comparisons. J. Research on Educational Effectiveness 5, 189-211. | p. 14 | Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons; Partial Pooling as Multiple Comparisons Correction |
| 16 | Gelman, A., & King, G. (1994). Enhancing democracy through legislative redistricting. APSR 88, 541-559. | p. 5 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 17 | Gelman, A., Shor, B., Bafumi, J., & Park, D. (2007). Rich state, poor state, red state, blue state: What’s the matter with Connecticut? QJPS 2, 345-367. | p. 5 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 18 | Gelman, A., & Tuerlinckx, F. (2000). Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics 15, 373-390. | not cited in text | this note; concept in Type S and Type M Errors |
| 19 | Gelman, A., & Weakliem, D. (2009). Of beauty, sex, and power: Statistical challenges in estimating small effects. American Scientist 97, 310-316. | not cited in text | this note |
| 20 | Greenland, S., & Poole, C. (2013). Living with P-values: resurrecting a Bayesian perspective on frequentist statistics. Epidemiology 24, 62-68. | p. 13 | Gelman Loken 2013 - Three Reactions and the Bayesian Reading of Bem’s 53 Percent |
| 21 | Humphreys, M., Sanchez, R., & Windt, P. (2013). Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Political Analysis 21, 1-20. | pp. 3, 13 | Gelman Loken 2013 - Four Testing Procedures and the Data-Contingent Decision Variable; Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 22 | Monogan, J. E. (2013). A case for registering studies of political outcomes: An application in the 2010 House elections. Political Analysis 21, 21-37. | p. 13 | Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 23 | Mullainathan, S., & Washington, E. (2009). Sticking with your vote: cognitive dissonance and political attitudes. AEJ: Applied 1, 86-111. | p. 4 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 24 | Nosek, B. A., Spies, J. R., & Motyl, M. (2013). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7, 615-631. | pp. 11, 14-15 | Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 25 | Petersen, M. B., Sznycer, D., Sell, A., Cosmides, L., & Tooby, J. (2013). The ancestral logic of politics: Upper-body strength regulates men’s assertion of self-interest over economic redistribution. Psychological Science. | pp. 4-5 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
| 26 | Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allow presenting anything as significant. Psychological Science 22, 1359-1366. | pp. 1-2 (once miscited as 2001) | Gelman Loken 2013 - Four Testing Procedures and the Data-Contingent Decision Variable; Simmons Nelson Simonsohn 2011 - False-Positive Psychology - Overview |
| 27 | Tracy, J. L., & Beall, A. T. (2013). Too good does not always mean not true. 30 Jul. UBC Emotion and Self Lab blog. | p. 8 | Gelman Loken 2013 - Data Exclusion, Coding and Pooling Forks in the Red-Pink Study |
| 28 | Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley. | p. 15 | Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 29 | Tukey, J. W. (1980). We need both exploratory and confirmatory. American Statistician 34, 23-25. | p. 13 | Gelman Loken 2013 - Way Forward - Preregistration, Replication and Analyzing All Comparisons |
| 30 | Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition (with discussion). Perspectives on Psychological Science 4, 274-324. | p. 2 | Gelman Loken 2013 - Four Testing Procedures and the Data-Contingent Decision Variable |
| 31 | Wagenmakers, E. J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). JPSP 100, 426-432. | p. 6 | Gelman Loken 2013 - Main Effects vs Interactions Case Studies |
Also mentioned in the text but not in the reference list: Radin (1997), quoted inside the Bem-Utts-Johnson reply (p. 6); womenshealth.gov and babycenter.com (p. 9); Feynman’s path diagrams (p. 11); Frederick Mosteller (p. 15).
The three uncited references
The paper lists these three works but never cites them in the text. The descriptions below come from their titles and from other vault notes, not from the paper.
Gelman (2014), varying treatment effects and unreplicable research ( Journal of Management)
Its title links unreplicable research to treatment effects that vary across people and settings, from a Bayesian perspective. It is a natural companion to §2: when effects vary, a fork such as “main effect or interaction” picks out whichever subgroup pattern the noise favours.
Gelman & Tuerlinckx (2000), Type S error rates ( Computational Statistics)
The source of the Type S (sign) error idea: the probability that a confident claim gets the sign of an effect wrong. It underlies the paper’s remark (§4.1, p. 11) that even the direction of the reported effects might not reappear on replication. See Type S and Type M Errors and Multiple Comparisons - Bayesian Perspective.
Gelman & Weakliem (2009), "Of beauty, sex, and power" ( American Scientist)
The beauty-and-sex-ratio example of estimating small effects with small samples, where significant estimates are badly exaggerated. It is the same low-power setting that §1.2 (p. 3) and §4.4 (pp. 14-15) describe. The vault works through this example in There Is No Safe Haven.
Examples
The Borges epigraph ^quote-gl-borges and the reference table above are the content of this note; the paper’s worked examples are in Gelman Loken 2013 - Main Effects vs Interactions Case Studies and Gelman Loken 2013 - Data Exclusion, Coding and Pooling Forks in the Red-Pink Study.
Connections
- Gelman Loken 2013 - Garden of Forking Paths - Overview — Source Map and errata list
- Garden of Forking Paths — concept note
- Simmons Nelson Simonsohn 2011 - False-Positive Psychology - Overview — reference 26
- Ioannidis 2005 - Why Most Published Findings Are False - Overview — Ioannidis is a co-author of Button et al. (reference 7)
- Pre-registration and Open Science - Overview — references 21-22 and 24 are early preregistration proposals