American Journal of Epidemiology Advance Access originally published online on May 25, 2007
American Journal of Epidemiology 2007 166(2):238-239; doi:10.1093/aje/kwm164
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LETTERS TO THE EDITOR |
RE: "VARIABLE SELECTION FOR PROPENSITY SCORE MODELS"
1 Centre for Clinical Epidemiology and Community Studies, Sir Mortimer B. Davis-Jewish General Hospital, McGill University, Montreal, QC H3T 1E2, Canada
2 Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, QC H3A 1A2, Canada
3 Department of Mathematics and Statistics, McGill University, Montreal, QC H3A 2K6, Canada
(e-mail: ian.shrier{at}mcgill.ca)
We recently read the paper by Brookhart et al. (1) on different propensity score (PS) models. We would like to comment on the conclusion that "... standard model-building tools designed to create good predictive models of the exposure will not always lead to optimal PS models, particularly in small studies" (1, p. 1149).
Conceptually, the objective underlying the PS method is to transform data from an observational study so that they approximate what would have been obtained had a randomized controlled trial been conducted, by accounting for differences in the probability of treatment allocation among subjects with different risk factors for disease. We would like to point out that the results of Brookhart et al. (1) have analogies in the analysis of randomized controlled trials that give additional insight into what constitutes an "optimal" PS model.
Brookhart et al. (1) described two simulations and, essentially, compared three classes of models. One class includes only covariates related to exposure, a second class includes covariates related to exposure plus covariates related to outcome, and a third class was similar to the second model but excluded covariates that were not causally related to the outcome. The implicit assumption of the first class of models is that the covariates causally related to outcome are independent of outcome given exposure. This is analogous to the unadjusted effect estimate in a randomized controlled trial.
The second class of models improves precision because there is an adjustment for covariates that are causally related to the outcome and chance related to exposure. The chance association situation occurs in actual randomized controlled trials more often than might be recognized; if n = 500 and the prevalence for each of five binary covariates is 20 percent, there is a 23 percent probability that at least one covariate will show a 7 percent difference between groups (2). In essence, class 2 models are analogous to multiple regression modeling of a randomized controlled trial; using multiple regression as the primary analysis in a randomized controlled trial remains controversial even though it can increase precision (3). Some authors argue that, if an unequal distribution of known confounders exists, there is an expectation that an unequal distribution of unknown confounders exists. Adjusting for only one can create bias. Others argue that if a disease is well understood, any unknown confounders would be weak confounders, and adjusting for imbalances in strong confounders yields the most valid effect estimate. In any actual study (as opposed to simulations), the potential for unmeasured confounding depends on how much is known about the particular disease.
Brookhart et al. (1) concluded that the class 3 models (similar to class 2 but excluding covariates not causally related to the outcome) were "optimal." The mean squared error of class 2 is larger because it includes covariates unrelated or weakly related to either exposure or outcome. The same results would occur in a similar multiple regression analysis of a randomized controlled trial.
We view the authors' results through the prism of model selection for general statistical models. When performing model selection, one must always specifically have in mind the goal of the analysis, whether it is feature selection, parameter estimation, or prediction (4). In multiple regression analysis of a randomized controlled trial, the goal is not to produce the model with the most predictive power; rather, it is to precisely estimate the causal effect of exposure in the presence of unmeasured variables. Similarly, propensity scores are used to improve the quality of causal effect estimates rather than to correctly model the treatment process. This suggests that the second and third approaches of Brookhart et al. (1) are analogous and share the same advantages/disadvantages as exist for multiple regression analysis of randomized controlled trials. Therefore, these approaches may increase bias even though the mean squared error is improved; the mean squared error "optimal" model may not always be the preferred model.
 |
ACKNOWLEDGMENTS |
|---|
Conflict of interest: none declared.
|
NOTES |
|---|
Editor's note: In accordance with Journal policy, Brookhart et al. were asked whether they wanted to respond to this letter, but they chose not to do so.
|
References |
|---|
 TOP References |
|---|
- Brookhart MA, Schneeweiss S, Rothman KJ, et al. Variable selection for propensity score models. Am J Epidemiol (2006) 163:1149–56.
[Abstract/Free Full Text] - Agresti A. Categorical data analysis. (2002) New York, NY: John Wiley & Sons.
- Issues in data analysis. Fundamentals of clinical trials. Friedman ML, Furberg CD, Demets DL, eds. (1998) New York, NY: Springer. 297–8.
- Hastie T, Tibshirani R, Friedman J. Model selection and assessment. In: The elements of statistical learning. (2003) Berlin, Germany: Springer-Verlag. 193–224.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||