American Journal of Epidemiology Advance Access originally published online on July 8, 2008
American Journal of Epidemiology 2008 168(4):374-383; doi:10.1093/aje/kwn156
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
PRACTICE OF EPIDEMIOLOGY |
Effect of Formal Statistical Significance on the Credibility of Observational Associations
1 Clinical and Molecular Epidemiology Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece
2 Biomedical Research Institute, Foundation for Research and Technology-Hellas, Ioannina, Greece
3 Department of Medicine, Tufts University School of Medicine, Boston, MA
Correspondence to Dr. John P. A. Ioannidis, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece (e-mail: jioannid{at}cc.uoi.gr).
Received for publication May 22, 2007. Accepted for publication January 4, 2008.
The author evaluated the implications of nominal statistical significance for changing the credibility of null versus alternative hypotheses across a large number of observational associations for which formal statistical significance (p < 0.05) was claimed. Calculation of the Bayes factor (B) under different assumptions was performed on 272 observational associations published in 2004–2005 and a data set of 50 meta-analyses on gene-disease associations (752 studies) for which statistically significant associations had been claimed (p < 0.05). Depending on the formulation of the prior, statistically significant results offered less than strong support to the credibility (B > 0.10) for 54–77% of the 272 epidemiologic associations for diverse risk factors and 44–70% of the 50 associations from genetic meta-analyses. Sometimes nominally statistically significant results even decreased the credibility of the probed association in comparison with what was thought before the study was conducted. Five of six meta-analyses with less than substantial support (B > 0.032) lost their nominal statistical significance in a subsequent (more recent) meta-analysis, while this did not occur in any of seven meta-analyses with decisive support (B < 0.01). In these large data sets of observational associations, formal statistical significance alone failed to increase much the credibility of many postulated associations. Bayes factors may be used routinely to interpret "significant" associations.
Bayes theorem; empirical research; epidemiologic methods; meta-analysis; observation; statistics
Abbreviations: RR, relative risk
Editor's note: An invited commentary on this article appears on page 384, and the author's response appears on page 389.
CiteULike 
Connotea 
Del.icio.us What's this?
Related articles in Am. J. Epidemiol.:
- Invited Commentary: Evidence-based Evaluation of p Values and Bayes Factors
- Hormuzd A. Katki
Am. J. Epidemiol. 2008 168: 384-388.[Abstract] [FREE Full Text] - The Author Responds to "Evaluating p Values and Bayes Factors"
- John P. A. Ioannidis
Am. J. Epidemiol. 2008 168: 389-390.[Extract] [FREE Full Text]
This article has been cited by other articles:
|
|
 |
|
  J. P. A. Ioannidis, E. Y. Loy, R. Poulton, and K. S. Chia Researching Genetic Versus Nongenetic Determinants of Disease: A Comparison and Proposed Unification Science Translational Medicine, November 18, 2009; 1(7): 7ps8 - 7ps8. [Full Text] [PDF]
|
|
|
|
 |
|
 B. Rueda, J. Broen, C. Simeon, R. Hesselstrand, B. Diaz, H. Suarez, N. Ortego-Centeno, G. Riemekasten, V. Fonollosa, M.C. Vonk, et al. The STAT4 gene influences the genetic predisposition to systemic sclerosis phenotype Hum. Mol. Genet., June 1, 2009; 18(11): 2071 - 2077. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. P. A. Ioannidis The Author Responds to "Evaluating p Values and Bayes Factors" Am. J. Epidemiol., August 15, 2008; 168(4): 389 - 390. [Full Text] [PDF]
|
|
|
|
|
|
H. A. Katki Invited Commentary: Evidence-based Evaluation of p Values and Bayes Factors Am. J. Epidemiol., August 15, 2008; 168(4): 384 - 388. [Abstract] [Full Text] [PDF]
|
|