American Journal of Epidemiology Advance Access originally published online on January 29, 2008
American Journal of Epidemiology 2008 167(5):517-522; doi:10.1093/aje/kwm357
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
PRACTICE OF EPIDEMIOLOGY |
A Statistical Test for the Equality of Differently Adjusted Incidence Rate Ratios
From the Department of Epidemiology, German Institute of Human Nutrition Potsdam-Rehbrücke, Nuthetal, Germany
Correspondence to Dr. Tobias Pischon, Department of Epidemiology, German Institute of Human Nutrition Potsdam-Rehbrücke, Arthur-Scheunert-Allee 114–116, 14558 Nuthetal, Germany (e-mail: pischon{at}dife.de).
Received for publication July 26, 2006. Accepted for publication January 5, 2007.
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
An incidence rate ratio (IRR) is a meaningful effect measure in epidemiology if it is adjusted for all important confounders. For evaluation of the impact of adjustment, adjusted IRRs should be compared with crude IRRs. The aim of this methodological study was to present a statistical approach for testing the equality of adjusted and crude IRRs and to derive a confidence interval for the ratio of the two IRRs. The method can be extended to compare two differently adjusted IRRs and, thus, to evaluate the effect of additional adjustment. The method runs immediately on existing software. To illustrate the application of this approach, the authors studied adjusted IRRs for two risk factors of type 2 diabetes using data from the European Prospective Investigation into Cancer and Nutrition-Potsdam Study from 2005. The statistical method described may be helpful as an additional tool for analyzing epidemiologic cohort data and for interpreting results obtained from Cox regression models with adjustment for different covariates.
cohort studies; confounding; diabetes mellitus, type 2; epidemiologic methods; proportional hazards models; statistics
Abbreviations: IRR, incidence rate ratio
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
In epidemiologic cohort studies, Cox proportional hazards regression is usually applied without and with adjustment for covariates to derive and present both crude and adjusted incidence rate ratios (IRRs). The difference between crude and adjusted IRRs is commonly described verbally without evaluating its significance. Up to now, a statistical test is missing that tests the equality of crude and adjusted IRRs. Also, no statistical method is available that produces a confidence interval for the ratio of two IRRs.
The comparison of crude and adjusted IRRs is closely related to the identification and selection of confounders. Essentially, two different strategies of confounder selection can be distinguished (1–3). In the first strategy, a variable is selected for control only if it is significantly associated with the outcome at some predetermined significance level. This strategy was often criticized because it does not evaluate the actual degree of confounding produced by the selected variable (2, 4–6). In contrast to testing the model parameter of the added covariate, the second strategy focuses on the degree of confounding. Here, a variable is selected for control if the change in IRR is important, for example, more than 10 percent (6). This approach may be criticized because it does not account for statistical variability and the power of the study. A 10 percent change in IRR may likely be due to random variation if the sample size of the study is low but may be important for large epidemiologic studies. The method presented in this paper allows refining the second strategy by assessing both the significance and the impact of the change in IRR. The p value for the equality test addresses the question of whether the added covariates are confounders. The 95 percent confidence interval around the ratio of the IRRs assesses the impact and precision of the change in IRR. If the lower confidence limit is greater than 1.1 or the upper confidence limit is lower than 1/1.1 = 0.9, the data are consistent with a 10 percent change in the point estimate of the IRR or more with an error of no more than 2.5 percent.
To derive a valid test and to compare the crude and adjusted IRRs, we need a joint model including both parameters. We present such a joint model by using a modified version of the augmented data approach that was originally described by Lunn and McNeil (7) for the analysis of competing risks. This model allows testing the equality of crude and adjusted IRRs. Related problems of comparing estimated coefficients across nested models were considered in contingency tables (8) and linear regression (9).
The objective of this paper is to present a statistical test for the equality of two differently adjusted IRRs and to provide a confidence interval for their ratio. For the purpose of illustration, we applied the statistical method proposed to compare crude and differently adjusted IRRs for two risk factors of type 2 diabetes using data from the European Prospective Investigation into Cancer and Nutrition (EPIC)-Potsdam.
|
DESCRIPTION OF THE METHOD |
|---|
|
TOPABSTRACTINTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
We used a data duplication method that is similar to the one of Lunn and McNeil (7) developed for competing risk analysis. With this method, each individual in a given data set has two records, the first record referring to the crude model and the second one to the adjusted model (table 1). Suppose that the crude model is characterized by
= 0 and the adjusted model by = 1. Multiplication of the covariate Z by will then result in exclusion of the covariate in the crude model and inclusion of the covariate in the adjusted model. In contrast to the covariates, the factor X of interest must be included in both models. To allow for differences in the effect size of X between both models, we must add two product terms of the form X and (1 – )X.
|
We applied Cox regression to the augmented data set. The hazard function
(t) has the form
 |
0(t) is an unspecified baseline hazard function; β1, β2, and 
are unknown parameters; and Z stands for a vector of covariates. Here, β1 is the crude parameter and β2 is the adjusted parameter of the exposure variable of interest. The expression exp{β1} represents the crude IRR, whereas exp{β2} is the adjusted IRR. If β1 is significantly different from β2, crude and adjusted incidence rate ratios differ significantly from each other. Equivalently, we can consider the expression exp{β2 – β1}, which is the ratio of adjusted to crude IRRs and tests whether this ratio is significantly different from 1. Moreover, a 95 percent confidence interval for exp{β2 – β1} can be calculated.
To ensure that the crude and adjusted IRRs based on the augmented data set are identical to those ones obtained when using two separate proportional hazards regression models, we must stratify the Cox regression by model type, that is, = 0 or 1. Moreover, this stratification allows that the baseline hazard functions in crude and adjusted Cox regression have no constant ratio, which avoids a strong proportionality assumption that would be needed in an unstratified analysis. Because of this stratification, the likelihood function for the augmented data can be rewritten as the product of two components treating the survival times for the two models separately.
Because two observations are created for each subject, the correlation matrix of the estimated regression coefficients in the augmented data set is affected by the dependence among the observations. Therefore, we cannot use the standard errors from standard Cox regression for calculating confidence intervals and p values in the joint model. To allow for dependence among multiple event times, we instead used robust standard errors obtained from the robust sandwich estimates for the covariance matrix (10, 11). The SAS, version 9.1.3, statistical package (SAS Institute, Inc., Cary, North Carolina) that we used did not produce sandwich estimates for the present data if the PHREG procedure with the covsandwich (aggregate) statement was applied. Therefore, we programmed the sandwich estimator with SAS/IML. The SAS syntax of the proposed method is given in appendix 1.
The method can be easily extended to test the hypothesis that two differently adjusted IRRs are equal. Again, for each individual, two entries have to be created in a given data set, one for the first and one for the second model. However, because both models can be very different and no model must be nested in the other one, we have to distinguish three types of variables concerning their use as covariates in the models. Characterizing the first model by = 0 and the second model by = 1, variables that are only covariates in the first model must be multiplied by 1 – , whereas variables that are included only in the second model must be multiplied by . A variable that is a covariate in both models has to be accounted for twofold, namely, as product of the variable and and as product of the variable and 1 – . Appendix 2 describes how to implement the generalized method.
|
APPLICATION TO REAL DATA |
|---|
|
TOPABSTRACTINTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
Study population
The study population was the European Prospective Investigation into Cancer and Nutrition cohort in Potsdam consisting of 27,548 subjects, 16,644 women aged 35–65 years and 10,904 men aged 40–65 years at recruitment. The baseline examination took place between 1994 and 1998 and included anthropometric measurements, a self-administered validated food-frequency questionnaire, and a personal interview (12). Follow-up questionnaires have been administered every 2–3 years to update medication and other lifestyle information and to identify incident cases of specific chronic diseases including type 2 diabetes. Cases of incident diabetes were identified from self-reports and current treatment for diabetes, and they were verified by contacting the primary care physician. A total of 849 cases were verified until August 31, 2005. The mean follow-up time was 7 years. Informed consent was obtained from all participants of the study, and approval was given by the Ethical Committee of the State of Brandenburg, Germany.
Variables in the Cox model
We separately used smoking status and hypertension as the exposure of interest and evaluated their association with risk of type 2 diabetes to illustrate our method. Smoking status was defined as ever or never smoker. Hypertension was defined by self-report (yes/no). After crude regression (model 1), we adjusted for age as the continuous variable (model 2) and for age and sex (model 3). In model 4, we additionally adjusted for anthropometric measurements (height and waist circumference, continuously) that were taken by trained personnel in the present study (13). Finally, we additionally adjusted for physical activity, alcohol intake, consumption of coffee, and intakes of whole-grain bread and red meat (all as continuous variables) and also mutually adjusted for smoking or hypertension (model 5). Physical activity was assessed through personal computer-guided interviews in the study center and calculated as total number of hours of sports, biking, and gardening per week averaged over 1 year. Habitual dietary intake was assessed by a validated semiquantitative food frequency questionnaire (14). The time to end of follow-up or incidence of diabetes was used as the primary time variable, noting that our proposed method is also applicable for the more complex counting process style of input.
Results for adjusted incidence rate ratios for type 2 diabetes
We estimated the IRR for type 2 diabetes associated with smoking using the crude model and different multivariable-adjusted models with increasing number of covariates (table 2). In all models, smokers had a significantly greater risk of diabetes than did nonsmokers (p 
0.0004); however, the effect size varied across the models. Adjustment for age only increased the IRR significantly. However, additional adjustment for sex and anthropometric variables resulted in a significantly lower IRR than the crude IRR. Adjustment for other risk factors slightly changed the IRR. The ratio of the fully adjusted to crude IRRs was 0.76 (95 percent confidence interval: 0.71, 0.82), indicating a significant difference in risk estimates due to adjustment. Because the upper confidence limit is lower than 0.9, the data are consistent with a 10 percent change in the point estimate of the IRR or more with an error of no more than 2.5 percent. Obviously, model 4 and model 5 produce similar adjusted IRRs, suggesting that the six additional variables do not make any important contribution to confounding. However, the tests and confidence intervals given in table 2 do not allow comparison between differently adjusted IRRs.
|
Table 3 summarizes the association of hypertension with risk of type 2 diabetes using different sets of covariates. In contrast to table 2, the adjusted IRR in model i is not compared with the crude IRR but rather with the IRR of the preceding model i – 1, i = 2, ..., 5, by applying the generalized method described in appendix 2. Participants who reported a diagnosis of hypertension at baseline were at markedly increased risk of type 2 diabetes. The crude IRR was 3.14 and decreased significantly to 2.55 after adjustment for age. Additional adjustment for sex did not alter the IRR as indicated by a p value of 0.30. However, addition of anthropometric variables to the set of covariates markedly attenuated the IRR to 1.61, and this change of IRR was highly significant (p < 0.0001). Finally, additional adjustment for smoking, physical activity, alcohol intake, consumption of coffee, and intakes of whole-grain bread and red meat did not affect the IRR as seen by a p value of 0.10. Thus, the six additional variables are not needed for adjusting the association of hypertension with risk of type 2 diabetes.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
We have presented a method for testing the equality of crude and adjusted IRRs that is similar to the augmented data approach of Lunn and McNeil successfully applied in competing risk analysis (7, 15–17). The idea of including both crude and adjusted Cox regressions in a joint model allows studying the effect of adjustment and quantifying the impact of confounding. Besides a p value for the equality test, a 95 percent confidence interval for the ratio of adjusted to crude IRRs can be derived.
The method can be generalized to compare two IRRs that were adjusted for different sets of covariates (appendix 2). In this case, Cox regression has to be applied on the two strata of the augmented data set with stratum-specific covariates. The generalized method allows evaluation of whether additional adjustment for a variable does alter the IRR and the study of the alteration of IRR from a simple to a larger model in a sequence of nested models. Application to nonnested models may occur if covariates are used alternatively, for example, adjustment for different sets of anthropometric variables.
The proposed statistical test does not focus on model fit, but rather on the change of the effect parameter due to the addition of covariates. If one is interested in comparing the fit of the crude and the adjusted models, the likelihood ratio test that is based on the deviance as measure of model fit should be applied. However, any significance obtained by the likelihood ratio test does not tell anything about the difference between crude and adjusted IRRs. Crude and adjusted IRRs can be equal, although the model fit of the multivariable model is substantially better than that of the crude model.
Although the considered problem of variation in incidence rate ratios due to changes in adjustment has not been treated up to now, a closely related problem was considered in epidemiologic case-control studies more than 20 years ago (8, 18, 19). The equality of crude and adjusted parameters in contingency tables that may be equivalently defined as collapsibility (20) was tested by a closed-form Wald test (19) or a closed-form conditional test (8). Greenland and Mickey (8) also presented a confidence interval for the degree of noncollapsibility, as measured by the ratio of the crude and adjusted odds ratios. Collapsibility was also assessed in linear regression. Clogg et al. (9) derived an exact test for comparing parameters of a reduced and a full linear regression model under the assumption of normally distributed errors. However, the mathematical tools utilized in our approach are very different from those used in the previous collapsibility studies.
Formally, the proposed significance test can be included in a confounder-selection routine. For example, a backward selection algorithm would start with all candidate variables and find out the variable for which its deletion from the model would result in the smallest change in the IRR of exposure. If the changed IRR is nonsignificantly different from the original IRR on the basis of the proposed test, delete the variable from the model and go to the next step. The algorithm continues until no further deletion of a variable occurs. However, we cannot expect that this selection strategy will select the best possible subset of variables to control. Monte Carlo simulation of several confounder selection criteria revealed that preliminary tests have poor sensitivity if conventional significance levels are used (1). Moreover, a statistical test applied in a specific step of the selection routine is carried out conditionally on the results of the preceding steps and ignores the model uncertainty implicit in the variable selection process (21). Furthermore, the statistical test can be misleading when misclassification is present (22). Because of these methodological difficulties, objections to the use of statistical significance testing in confounder selection and identification are widespread (23, 24).
In summary, the presented statistical approach for testing the equality of crude and adjusted IRRs or of two differently adjusted IRRs may be a helpful additional tool in analyzing epidemiologic cohort data. Attenuation of IRR after adjustment as often observed in epidemiologic research can now be explored whether it is statistically significant or negligible. The approach can be easily implemented by use of standard statistical software packages for fitting Cox proportional hazards regression models.
|
APPENDIX 1 |
|---|
|
TOPABSTRACTINTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
SAS Code for the Proposed Method
This appendix provides the SAS code for data augmentation and Cox regression in the augmented data set. Our data file named example contained an exposure variable X and for the sake of simplicity only one covariate Z. At first, the data set was duplicated, and the two equal data sets were concatenated, that is, combined one after the other into a single data set called total. Then, an indicator variable for the second half called delta was added, and product terms of (1 – delta) and X, delta and X, and delta and Z were defined.
 |
Cox regression was then applied on the augmented data set total that also contained a variable case for the disease status, a primary time variable called time, and a variable for subject identification called subj_ID.
 |
The parameter estimates are output to a data set named Est and the DFBETA statistics for the crude and adjusted exposure variable are output to a data set named Out1. The strata statement achieves a stratified analysis in the crude and adjusted model. Instead of the standard notation for the primary time variable, the (start, stop) notation to describe intervals of time at risk can be used. Then, the DFBETA statistics were summed up for each individual.
 |
The following iml procedure reads the parameter estimates and the DFBETA statistics and computes the robust sandwich covariance matrix. Finally, the Wald chi-square test and the confidence interval are calculated on the basis of the robust variance and covariance estimates.
 |
|
APPENDIX 2 |
|---|
|
TOPABSTRACTINTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
A Generalization of the Method
Instead of comparing crude and adjusted IRRs, we now compare two IRRs adjusted for different sets of covariates. Let X be the exposure variable and Z0, Z1, and Z2 be covariates. We are interested in comparing model 1 and model 2 characterized by the sets of covariates {Z0, Z1} and {Z0, Z2}, respectively. Note that both models include the same covariate Z0, but they differ in the second covariate. Apparently, the three covariates are of different types concerning their inclusion in the two models. Let delta be the indicator variable for model 2 and, therefore, ndelta = (1 – delta) be the indicator variable for model 1. Then, we have to form the following product terms:
 |
Then, the model equation for the PHREG procedure in SAS can be written as
 |
Again, the parameter (deltaX-ndeltaX) is the parameter of interest because it is the logarithm of the ratio of the two IRRs. Instead of the single covariates Z0, Z1, and Z2, each type of covariate can consist of several variables. In applications with a simpler model nested within a larger model, only two types of covariates occur.
|
ACKNOWLEDGMENTS |
|---|
Conflict of interest: none declared.
|
NOTES |
|---|
Editor's note: An invited commentary on this article appears on page 523, and the authors' response is published on page 530.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONDESCRIPTION OF THE METHODAPPLICATION TO REAL DATADISCUSSIONAPPENDIX 1APPENDIX 2References |
|---|
- Mickey RM, Greenland S. The impact of confounder selection criteria on effect estimation. Am J Epidemiol (1989) 129:125–37.
[Abstract/Free Full Text] - Maldonado G, Greenland S. Simulation study of confounder-selection strategies. Am J Epidemiol (1993) 138:923–36.
[Abstract/Free Full Text] - Rothman KJ, Greenland S. Modern epidemiology. (1998) 2nd ed. Philadelphia, PA: Lippincott-Raven.
- Miettinen OS. Stratification by a multivariate confounder score. Am J Epidemiol (1976) 104:609–20.
[Abstract/Free Full Text] - Greenland S, Neutra R. Control of confounding in the assessment of medical technology. Int J Epidemiol (1980) 9:361–7.[Web of Science][Medline]
- Greenland S. Modeling and variable selection in epidemiologic analysis. Am J Public Health (1989) 79:340–9.
[Abstract/Free Full Text] - Lunn M, McNeil D. Applying Cox regression to competing risks. Biometrics (1995) 51:524–32.[CrossRef][Web of Science][Medline]
- Greenland S, Mickey RM. Closed form and dually consistent methods for inference on strict collapsibility in 2 x 2 x K and 2 x J x K tables. Appl Stat (1988) 37:335–43.[CrossRef]
- Clogg CC, Petkova E, Shihadeh ES. Statistical methods for analyzing collapsibility in regression models. J Educ Stat (1992) 17:51–74.[CrossRef]
- Lin DY, Wei LJ. The robust inference for the Cox proportional hazards model. J Am Stat Assoc (1989) 84:1074–8.[CrossRef][Web of Science]
- Wei LJ, Lin DY, Weissfeld L. Regression analysis of multivariate incomplete failure time data by modeling marginal distributions. J Am Stat Assoc (1989) 84:1065–73.[CrossRef][Web of Science]
- Boeing H, Korfmann A, Bergmann MM. Recruitment procedures of EPIC-Germany. European Investigation into Cancer and Nutrition. Ann Nutr Metab (1999) 43:205–15.[CrossRef][Web of Science][Medline]
- Kroke A, Bergmann MM, Lotze G, et al. Measures of quality control in the German component of the EPIC Study. Ann Nutr Metab (1999) 43:216–24.[CrossRef][Web of Science][Medline]
- Kroke A, Klipstein-Grobusch K, Voss S, et al. Validation of a self-administered food-frequency questionnaire administered in the European Prospective Investigation into Cancer and Nutrition (EPIC) Study: comparison of energy, protein, and macronutrient intakes estimated with the doubly labeled water, urinary nitrogen, and repeated 24-h dietary recall methods. Am J Clin Nutr (1999) 70:439–47.
[Abstract/Free Full Text] - Tai BC, Machin D, White I, et al. Competing risks analysis of patients with osteosarcoma: a comparison of four different approaches. Stat Med (2001) 20:661–84.[CrossRef][Web of Science][Medline]
- Rosthoi S, Andersen PK, Abildstrom SZ. SAS macros for estimation of the cumulative incidence functions based on a Cox regression model for competing risks survival data. Comput Methods Programs Biomed (2004) 74:69–75.[CrossRef][Web of Science][Medline]
- Glynn RJ, Rosner B. Comparison of risk factors for the competing risks of coronary heart disease, stroke, and venous thromboembolism. Am J Epidemiol (2005) 162:975–82.
[Abstract/Free Full Text] - Whittemore AS. Collapsing multidimensional contingency tables. J R Stat Soc (B) (1978) 40:328–40.
- Ducharme GR, Lepage Y. Testing collapsibility in contingency tables. J R Stat Soc (B) (1986) 48:197–205.
- Asmussen S, Edwards D. Collapsibility and response variables in contingency tables. Biometrika (1983) 70:567–78.
[Abstract/Free Full Text] - Viallefont V, Raftery AE, Richardson S. Variable selection and Bayesian model averaging in case-control studies. Stat Med (2001) 20:3215–30.[CrossRef][Web of Science][Medline]
- Greenland S, Robins JM. Confounding and misclassification. Am J Epidemiol (1985) 122:495–506.
[Abstract/Free Full Text] - Hernberg S. Significance testing of potential confounders and other properties of study groups—misuse of statistics. Scand J Work Environ Health (1996) 22:315–16.[Web of Science][Medline]
- Nurminen M. On the epidemiologic notion of confounding and confounder identification. Scand J Work Environ Health (1997) 23:64–8.[Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
Related articles in Am. J. Epidemiol.:
- Invited Commentary: Variable Selection versus Shrinkage in the Control of Multiple Confounders
- Sander Greenland
Am. J. Epidemiol. 2008 167: 523-529.[Abstract] [FREE Full Text] - Pischon et al. Respond to "Variable Selection versus Shrinkage in Control of Confounders"
- Tobias Pischon, Matthias B. Schulze, Dagmar Drogan, and Heiner Boeing
Am. J. Epidemiol. 2008 167: 530-531.[Extract] [FREE Full Text]
This article has been cited by other articles:
|
|
 |
|
  F. Imamura, A. H. Lichtenstein, G. E. Dallal, J. B. Meigs, and P. F. Jacques Confounding by Dietary Patterns of the Inverse Association Between Alcohol Consumption and Type 2 Diabetes Risk Am. J. Epidemiol., July 1, 2009; 170(1): 37 - 45. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. Greenland Invited Commentary: Variable Selection versus Shrinkage in the Control of Multiple Confounders Am. J. Epidemiol., March 1, 2008; 167(5): 523 - 529. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. Pischon, M. B. Schulze, D. Drogan, and H. Boeing Pischon et al. Respond to "Variable Selection versus Shrinkage in Control of Confounders" Am. J. Epidemiol., March 1, 2008; 167(5): 530 - 531. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||