Ioannidis 2005 - Reference List and Citation Map
Summary
This note transcribes all 37 references of Ioannidis (2005), p. 0701. For each one it records where in the essay the reference is cited and what claim it supports. Use it to look up the bracketed numbers in the other notes of this set. The references fall into five groups: refutation evidence (1-8), the Bayesian/PPV precursors (9-11), design and reporting standards (14-15, 18-25, 35), conflicts of interest and expert opinion (26-28), and high-throughput discovery and multiplicity (4, 8, 17, 29-34, 36-37).
Overview
The essay makes its argument with formulas and backs its empirical premises with citations. The premises are that refutation is common, that conflicts of interest are common, that selective outcome reporting happens even in RCTs, and that the Proteus phenomenon is common in genetics. Mapping each reference to its claim shows which steps rest on outside evidence and which on the model alone. Much of that evidence comes from the author’s own earlier work (refs 1, 5, 7, 8, 19, 29, 30, 36). This matters when weighing the essay’s empirical claims; see Ioannidis 2005 - Remedies, Limitations and Later Critiques.
Main Content
Full reference list (p. 0701) with citation sites
Bibliographic entries are transcribed from the PDF. “Cited for” paraphrases the sentence that cites the reference, with its section.
| # | Reference | Cited at | Cited for |
|---|---|---|---|
| 1 | Ioannidis JP, Haidich AB, Lau J (2001) Any casualties in the clash of randomised and observational evidence? BMJ 322: 879-880. | Introduction, p. 0696 | Refutation in clinical trials and traditional epidemiology [1-3] |
| 2 | Lawlor DA, Davey Smith G, Kundu D, Bruckdorfer KR, Ebrahim S (2004) Those confounded vitamins: What can we learn from the differences between observational versus randomised trial evidence? Lancet 363: 1724-1727. | Introduction | same [1-3] |
| 3 | Vandenbroucke JP (2004) When are observational studies as credible as randomised trials? Lancet 363: 1728-1731. | Introduction | same [1-3] |
| 4 | Michiels S, Koscielny S, Hill C (2005) Prediction of cancer outcome with microarrays: A multiple random validation strategy. Lancet 365: 488-492. | Introduction; Corollary 3 | Refutation in modern molecular research [4,5]; microarray / high-throughput fields [4,8,17] |
| 5 | Ioannidis JPA, Ntzani EE, Trikalinos TA, Contopoulos-Ioannidis DG (2001) Replication validity of genetic association studies. Nat Genet 29: 306-309. | Introduction | Refutation in molecular research [4,5] |
| 6 | Colhoun HM, McKeigue PM, Davey Smith G (2003) Problems of reporting genetic associations with complex outcomes. Lancet 361: 865-872. | Introduction | False findings may be the majority of published claims [6-8] |
| 7 | Ioannidis JP (2003) Genetic associations: False or true? Trends Mol Med 9: 135-138. | Introduction; Corollary 2 | [6-8]; genetic risk factors with relative risks 1.1-1.5 |
| 8 | Ioannidis JPA (2005) Microarrays and molecular research: Noise discovery? Lancet 365: 454-455. | Introduction; Corollary 3 | [6-8]; high-throughput discovery [4,8,17] |
| 9 | Sterne JA, Davey Smith G (2001) Sifting the evidence—What’s wrong with significance tests. BMJ 322: 226-231. | Framework, p. 0696 | Methodologists who attribute nonreplication to claims based on one significant study [9-11] |
| 10 | Wacholder S, Chanock S, Garcia-Closas M, El ghormli L, Rothman N (2004) Assessing the probability that a positive report is false: An approach for molecular epidemiology studies. J Natl Cancer Inst 96: 434-442. | Framework (three sites); Remedies, p. 0701 | [9-11]; PPV depends on prior, power and [10,11]; PPV is the complement of the false positive report probability [10]; understanding the range of [10] |
| 11 | Risch NJ (2000) Searching for genetic determinants in the new millennium. Nature 405: 847-856. | Framework | [9-11]; [10,11] |
| 12 | Kelsey JL, Whittemore AS, Evans AS, Thompson WD (1996) Methods in observational epidemiology, 2nd ed. New York: Oxford U Press. 432 p. | Bias (reverse bias), p. 0697 | Large measurement error loses true relationships in noise |
| 13 | Topol EJ (2004) Failing the public health—Rofecoxib, Merck, and the FDA. N Engl J Med 351: 1707-1709. | Bias (reverse bias) | Conflicts of interest that “bury” significant findings |
| 14 | Yusuf S, Collins R, Peto R (1984) Why do we need some large, simple randomized trials? Stat Med 3: 409-422. | Corollary 1, p. 0697 | Large cardiology RCTs (several thousand subjects) |
| 15 | Altman DG, Royston P (2000) What do we mean by validating a prognostic model? Stat Med 19: 453-473. | Corollary 1 | Molecular-predictor research with 100-fold smaller samples |
| 16 | Taubes G (1995) Epidemiology faces its limits. Science 269: 164-169. | Corollary 2, pp. 0697-0698 | Modern epidemiology is increasingly obliged to target smaller effect sizes |
| 17 | Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, et al. (1999) Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science 286: 531-537. | Corollary 3 | High-throughput discovery-oriented research [4,8,17] |
| 18 | Moher D, Schulz KF, Altman DG (2001) The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357: 1191-1194. | Corollary 4, p. 0698 | Standardization efforts for RCTs [18-20] |
| 19 | Ioannidis JP, Evans SJ, Gotzsche PC, O’Neill RT, Altman DG, et al. (2004) Better reporting of harms in randomized trials: An extension of the CONSORT statement. Ann Intern Med 141: 781-788. | Corollary 4 | [18-20] |
| 20 | International Conference on Harmonisation E9 Expert Working Group (1999) ICH Harmonised Tripartite Guideline. Statistical principles for clinical trials. Stat Med 18: 1905-1942. | Corollary 4 | [18-20] |
| 21 | Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, et al. (1999) Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 354: 1896-1900. | Corollary 4 | Standardization for meta-analyses [21,22] |
| 22 | Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, et al. (2000) Meta-analysis of observational studies in epidemiology: A proposal for reporting. Meta-analysis of Observational Studies in Epidemiology (MOOSE) group. JAMA 283: 2008-2012. | Corollary 4 | [21,22] |
| 23 | Marshall M, Lockwood A, Bradley C, Adams C, Joy C, et al. (2000) Unpublished rating scales: A major source of bias in randomised controlled trials of treatments for schizophrenia. Br J Psychiatry 176: 249-252. | Corollary 4 | Multifarious outcomes (schizophrenia scales) versus unequivocal outcomes (death) |
| 24 | Altman DG, Goodman SN (1994) Transfer of technology from statistical journals to the biomedical literature. Past trends and future predictions. JAMA 272: 129-132. | Corollary 4 | Stereotyped analysis methods (Kaplan-Meier, log-rank) versus experimental methods |
| 25 | Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG (2004) Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291: 2457-2465. | Corollary 4 | Selective outcome reporting is common even in RCTs |
| 26 | Krimsky S, Rothenberg LS, Stott P, Kyle G (1998) Scientific journals and their authors’ financial interests: A pilot study. Psychother Psychosom 67: 194-201. | Corollary 5, p. 0698 | Conflicts of interest are very common [26] and inadequately reported [26,27] |
| 27 | Papanikolaou GN, Baltogianni MS, Contopoulos-Ioannidis DG, Haidich AB, Giannakakis IA, et al. (2001) Reporting of conflicts of interest in guidelines of preventive and therapeutic interventions. BMC Med Res Methodol 1: 3. | Corollary 5 | [26,27] |
| 28 | Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC (1992) A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 268: 240-248. | Corollary 5 | Expert opinion is extremely unreliable |
| 29 | Ioannidis JP, Trikalinos TA (2005) Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol 58: 543-549. | Corollary 6, p. 0699 (twice) | Definition of the Proteus phenomenon; evidence that it is very common in molecular genetics |
| 30 | Ntzani EE, Ioannidis JP (2003) Predictive ability of DNA microarrays for cancer outcomes and correlates: An empirical assessment. Lancet 362: 1439-1444. | Most findings false…, p. 0699 | Discovery research: 30,000 genes tested, perhaps 30 true [30,31] |
| 31 | Ransohoff DF (2004) Rules of evidence for cancer molecular-marker discovery and validation. Nat Rev Cancer 4: 309-314. | same | [30,31] |
| 32 | Lindley DV (1957) A statistical paradox. Biometrika 44: 187-192. | Remedies, p. 0700 | Very large studies may find significance for trivial effects [32-34] |
| 33 | Bartlett MS (1957) A comment on D.V. Lindley’s statistical paradox. Biometrika 44: 533-534. | Remedies | [32-34] |
| 34 | Senn SJ (2001) Two cheers for P-values. J Epidemiol Biostat 6: 193-204. | Remedies | [32-34] |
| 35 | De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. (2004) Clinical trial registration: A statement from the International Committee of Medical Journal Editors. N Engl J Med 351: 1250-1251. | Remedies, p. 0701 | Upfront registration of randomized trials |
| 36 | Ioannidis JPA (2005) Contradicted and initially stronger effects in highly cited clinical research. JAMA 294: 218-228. | Remedies | ”I suspect several established ‘classics’ will fail the test” |
| 37 | Hsueh HM, Chen JJ, Kodell RL (2003) Comparison of methods for estimating the number of true null hypotheses in multiplicity testing. J Biopharm Stat 13: 675-689. | Remedies | A large statistical literature on multiple-testing corrections exists, yet the dredging that preceded a report usually cannot be deciphered |
Notes on transcription: hyphenated line breaks in the PDF have been joined (“Kaplan-Meier”, “discovery-oriented”). Ref. 25’s title is transcribed as printed. The PDF has the typo “randomized trails” at the citation site in the text, not in the reference entry.
Citation map by section
| Section | References cited |
|---|---|
| Introduction (p. 0696) | 1-3, 4-5, 6-8 |
| Modeling the Framework (pp. 0696-0697) | 9-11, 10 (FPRP), 10-11 |
| Bias / reverse bias (p. 0697) | 12, 13 |
| Testing by Several Independent Teams | none |
| Corollary 1 | 14, 15 |
| Corollary 2 | 7, 16 |
| Corollary 3 | 4, 8, 17 |
| Corollary 4 | 18-20, 21-22, 23, 24, 25 |
| Corollary 5 | 26, 26-27, 28 |
| Corollary 6 / Proteus | 29 (x2) |
| Box 1 | none |
| Most Research Findings Are False… (Table 4), p. 0699 | 30-31 |
| Prevailing bias (null fields) | none |
| How Can We Improve the Situation? | 32-34, 35, 10, 36, 37 |
Examples
Reading the map
The core mathematical sections (Testing by Several Teams, Box 1, the null-field section) cite nothing. They are the essay’s own derivations. The empirical premises are concentrated in the corollaries. Of the 37 references, 8 (1, 5, 7, 8, 19, 29, 30, 36) are co-authored by Ioannidis. For a model-validation reader, the methodological precursors that matter most are Wacholder et al. [10] (false positive report probability), Lindley [32] (large- significance for trivial effects) and Hsueh et al. [37] (estimating the number of true nulls, i.e. estimating ).
Connections
- Ioannidis 2005 - PPV of Research Findings and the 2x2 Framework: refs 9-11, the FPRP.
- Ioannidis 2005 - Bias Parameter u and Reverse Bias: refs 12-13.
- Ioannidis 2005 - Six Corollaries and the Proteus Phenomenon: refs 7, 14-29.
- Ioannidis 2005 - Table 4 PPV by Study Design: refs 30-31.
- Ioannidis 2005 - Remedies, Limitations and Later Critiques: refs 32-37.
- Multiple Testing Corrections: ref. 37 relates to estimating the null proportion in FDR methods, which equals in Ioannidis’s notation.
- Pre-registration and Open Science - Overview: refs 18-22 and 35 are the reporting-standard and registration lineage.