American Journal of Epidemiology Advance Access published online on July 8, 2008
American Journal of Epidemiology, doi:10.1093/aje/kwn156
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Practice of Epidemiology |
Effect of Formal Statistical Significance on the Credibility of Observational Associations
1 Clinical and Molecular Epidemiology Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece
2 Biomedical Research Institute, Foundation for Research and Technology-Hellas, Ioannina, Greece
3 Department of Medicine, Tufts University School of Medicine, Boston, MA
Correspondence to Dr. John P. A. Ioannidis, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece (e-mail: jioannid{at}cc.uoi.gr).
Received for publication May 22, 2007. Accepted for publication January 4, 2008.
The author evaluated the implications of nominal statistical significance for changing the credibility of null versus alternative hypotheses across a large number of observational associations for which formal statistical significance (p < 0.05) was claimed. Calculation of the Bayes factor (B) under different assumptions was performed on 272 observational associations published in 2004–2005 and a data set of 50 meta-analyses on gene-disease associations (752 studies) for which statistically significant associations had been claimed (p < 0.05). Depending on the formulation of the prior, statistically significant results offered less than strong support to the credibility (B > 0.10) for 54–77% of the 272 epidemiologic associations for diverse risk factors and 44–70% of the 50 associations from genetic meta-analyses. Sometimes nominally statistically significant results even decreased the credibility of the probed association in comparison with what was thought before the study was conducted. Five of six meta-analyses with less than substantial support (B > 0.032) lost their nominal statistical significance in a subsequent (more recent) meta-analysis, while this did not occur in any of seven meta-analyses with decisive support (B < 0.01). In these large data sets of observational associations, formal statistical significance alone failed to increase much the credibility of many postulated associations. Bayes factors may be used routinely to interpret "significant" associations.
Bayes theorem; empirical research; epidemiologic methods; meta-analysis; observation; statistics
Abbreviations: RR, relative risk
Editor's note: An invited commentary on this article appears on page 000, and the author's response appears on page 000.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  J. P. A. Ioannidis The Author Responds to "Evaluating p Values and Bayes Factors" Am. J. Epidemiol., August 15, 2008; 168(4): 389 - 390. [Full Text] [PDF]
|
|
|
|
|
|
H. A. Katki Invited Commentary: Evidence-based Evaluation of p Values and Bayes Factors Am. J. Epidemiol., August 15, 2008; 168(4): 384 - 388. [Abstract] [Full Text] [PDF]
|
|