American Journal of Epidemiology Advance Access originally published online on July 5, 2007
American Journal of Epidemiology 2007 166(6):656-658; doi:10.1093/aje/kwm173
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Invited Commentary: Fewell and Colleagues—Fuel for Debate
From the Division of Cancer Prevention and Population Sciences, Roswell Park Cancer Institute, Buffalo, NY
Correspondence to Dr. James Marshall, Division of Cancer Prevention and Population Sciences, Roswell Park Cancer Institute, Elm and Carlton Streets, Buffalo, NY 14263 (e-mail: james.marshall{at}roswellpark.org).
Received for publication March 31, 2006. Accepted for publication July 12, 2006.
 |
ABSTRACT |
|---|
 TOP ABSTRACT References |
|---|
Concern over the impact of flawed measurement continues to nag epidemiology. Early studies indicated that the impact of measurement error is benign, leading generally only to attenuation of associations; more recent research has documented that this impact, especially within the setting of multivariate modeling, cannot be expected always to be benign. It can, for example, be a source of unsettling inconsistency. Fewell and colleagues (Am J Epidemiol 2007;166:646–655) show that residual confounding is especially persistent in the presence of multivariate confounding.
Extending our understanding of multiple, imperfectly measured confounders, Fewell et al. (1) have documented that confounding is not eliminated by adjustment for a mélange of possible confounding factors. They have shown that error in the measurement of multiple confounders will cause them to resonate through the focal epidemiologic exposures and bias estimates of their effects. These results confirm that, absent understanding of the extent and structure of measurement error in our study variables and their confounders, we need to very carefully circumscribe our claims that our multivariate analyses eliminate the threat of confounding.
Fewell et al. confirm that confounder misclassification complicates the interpretation of epidemiologic data. Epidemiology has long held to Bross' (2) and Newell's (3) demonstrations that completely random misclassification of a binary exposure reduces test power and biases estimated associations toward the null; their results seemed reassuring insofar as one might be most concerned about false-positive findings, although Goldberg (4) later demonstrated that modest departures from random behavior of measurement errors could generate bias away from the null. All of these investigations came before epidemiologists worried much about multivariate analysis. That exposure-misclassification results do not apply in the case of a mismeasured confounder was demonstrated by Greenland (5) and Greenland and Robins (6). More recently, Fox et al. (7) illustrated the importance of seemingly trivial departures from random behavior of measurement error.
Exploring the problem of multiple confounders in epidemiology, Fewell et al. (1) raise the specter of numerous troublesome scenarios: Several confounders could be measured with varying degrees of error; the degree to which confounders actually confound could vary; combinations of these conditions could be present. In the examples Fewell et al. provide, all of the confounders increase risk, and all are positively related to one another; the result will certainly be different if—and this is possible—some of these positively correlated confounders increase risk while others decrease it. In these examples, the confounders are truly confounders: If they are correlated with risk and with the focal exposure but do not actually have a direct bearing on risk, controlling for them may induce bias. In these examples, all measurement errors are well-behaved: Independently of risk and of one another, they may well be correlated.
One never really knows what process gave rise to data-based results. The most significant limitation of the results Fewell et al. develop is that the focal exposure is measured without error. It is, of course, possible that measures of one exposure will contain no error while measures of other exposures will contain substantial error; whether this is likely might be debated, and it would be difficult for the investigator to justify counting on it. The absence of error in measurement of the focal exposure eliminates the possibility of correlations between that exposure's measurement errors and those of the confounders. Such correlations can have significant effects on what is observed, and those effects are not readily derivable from what is seen in the case of well-behaved error (8).
Fewell et al. qualify their results, noting that those results are based on simulation (1). Asserting that results based on data are always desirable, they recognize that such results may not be available. However, it is worth noting that one never really knows what process generated observational data. The distinction between a set of observations and the process that gave rise to those observations is important. We know and can readily understand the process that produced the data Fewell et al. address.
Fewell et al. constructed a set of confounders correlated at either 0.1, 0.3, or 0.5 with the focal exposure (1). However, if the true correlations of the confounders with the focal exposure are 0.1, 0.3, and 0.5, respectively, the observed correlations of these variables with that exposure will not be 0.1, 0.3, or 0.5. With the intraclass correlation coefficient of each confounder being equal to 0.5, the observed correlations will be 0.07, 0.21, or 0.35. With the exposure measured with the same intraclass correlation coefficient as the confounders, the observed correlations will be 0.05, 0.15, or 0.25 (9). Measurement error substantially attenuates these correlations, suppressing evidence of the degree to which the confounders confound.
Recognizing the limitation imposed by their modeling of no exposure measurement error, Fewell et al. suggested that it would be valuable to study the impact of error in measurement of the exposure as opposed to that of the confounder (1). Some limited data on this problem have been generated; in general, the bias imposed by a mismeasured confounder depends on whether the confounder increases or decreases risk, as well as on whether the confounder is positively or negatively associated with the focal exposure (9). In more complicated settings, such as those with correlated measurement error, these effects can be more varied.
These results underline the persistence of bias in spite of control for additional mismeasured risk factors. The problem is especially acute for multiple orthogonal variables. In Fewell et al.'s table 3 (1), with four imperfectly measured but uncorrelated confounders, for example, the correlations of E and X1 and E and X2 are 0.3; the intraclass correlation coefficients for X1 and X2 are 0.5. The crude odds ratio describing the impact of the exposure is 2.35; the odds ratio adjusted for Z1 is 2.24, and that adjusted for both Z1 and Z2 is 2.12. Control for two important variables barely reduces bias; most of it is preserved. Fewell et al.'s table 4 (1) shows that the problem remains even for correlated variables. In the same setting with correlated variables, as seen in Fewell et al.'s table 4, the crude odds ratio is 1.89, while the odds ratio controlled for both Z1 and Z2 is 1.58; again, most of the confounding imposed by measurement error is preserved.
Fewell et al. state that their results apply most specifically to the situation in which exposure does not affect risk (1). Of course, the interpretation of any apparent risk alteration—miniscule or substantial—requires understanding of the extent and structure of errors underlying exposure and confounder measurement. One can neither establish nor dismiss the importance of exposure to risk, absent information on the degree and structure of measurement error. One can readily construct from a data set—an outcome and series of exposures—using nothing more than measurement error, an infinity of apparent impacts linked to those exposures.
This thoughtful analysis confirms that measurement errors bias the expected value of association measures. In other words, notwithstanding sample size, notwithstanding statistical significance, measurement error is more than a benign influence that leads us to occasionally overlook risk factors. What we are increasingly having to recognize is that measurement error can lead, under a broad array of circumstances, to substantial bias. We cannot abstract from our data the correct conclusions unless we thoroughly understand what we have measured and how well we have measured it. Passing glances at the repeatability of one or two variables out of a series of variables, even if reinforced by claims that these analyses indicate that the data are valid, will not do. Sanguine reassurances that the data are probably "good enough" (10, 11) are likely to contribute little more than false confidence in our results. As we seek to refine epidemiology's contribution, we may need to devote more attention than we have heretofore to the quality of our data, assaying that quality and allocating resources to increase it.
It is, of course, one thing to recognize measurement error as an obstacle to the interpretation of epidemiologic findings and quite another to do something about it. One of the most comprehensive approaches has been provided by Spiegelman et al. (12, 13). A more modest approach, proposed as a ready guide to the interpretation of categorical data, was described by Fox et al. (7).
We need to understand how measurement errors are correlated. If we assume incorrectly either that these errors are uncorrelated or that they are correlated, we run the risk of extracting the wrong conclusions from our data.
It may also be worth remembering, as we struggle with the formidable challenge of examining factors that cannot readily be measured other than very crudely, the importance of triangulation: juxtaposing epidemiologic data against in vitro and in vivo findings, against human experimental findings, and even against ecologic evidence (14). To be sure, probability theory tells us that individual-based analysis will give us the right answer over the long run, if we specify our epidemiologic models correctly, measure our variables accurately, and draw our samples appropriately. We are never sure we have the correct model, are virtually never sure we have measured our variables adequately, and are almost never sure we have drawn our samples appropriately; and significant violations of any of these conditions may mean that bets are off. These biasing effects may tend to cancel one another, and we do not at present have enough information to justify relying on such benign cancellations. In that our job is to help people understand the impact of their experiences and exposures on the probability that they will get sick, not to preserve our approach to information collection, we must recognize the limits of and remain open to all of the information to which we have access.
|
ACKNOWLEDGMENTS |
|---|
The author thanks Dr. Sander Greenland for helpful advice.
Conflict of interest: none declared.
|
References |
|---|
|
TOPABSTRACTReferences |
|---|
- Fewell Z, Davey Smith G, Sterne JA. The impact of residual and unmeasured confounding in epidemiologic studies: a simulation study. Am J Epidemiol (2007) 166:646–55.
[Abstract/Free Full Text] - Bross I. Misclassification in 2 x 2 tables. Biometrics (1954) 10:478–86.[CrossRef][Web of Science]
- Newell D. Errors in the interpretation of errors in epidemiology. Am J Public Health (1962) 14:459–66.
- Goldberg JD. The effects of misclassification on the bias in the difference between two preparations and the relative odds in the four-fold table. J Am Stat Assoc (1975) 70:561–7.[CrossRef][Web of Science]
- Greenland S. The effect of misclassification in the presence of covariates. Am J Epidemiol (1980) 112:564–9.
[Abstract/Free Full Text] - Greenland S, Robins JM. Confounding and misclassification. Am J Epidemiol (1985) 122:495–506.
[Abstract/Free Full Text] - Fox MP, Lash TL, Greenland S. A method to automate probabilistic sensitivity analyses of misclassified binary variables. Int J Epidemiol (2005) 34:1370–6.
[Abstract/Free Full Text] - Marshall JR, Hastrup JL, Ross JS. Mismeasurement and the resonance of strong confounders: correlated errors. Am J Epidemiol (1999) 150:88–96.
[Abstract/Free Full Text] - Marshall JR, Hastrup JL. Mismeasurement and the resonance of strong confounders: uncorrelated errors. Am J Epidemiol (1996) 143:1069–78.
[Abstract/Free Full Text] - Byers T. Food frequency dietary assessment: how bad is good enough? Am J Epidemiol (2001) 154:1087–8.
[Free Full Text] - Willett W, Stampfer M, Chu NF, et al. Assessment of questionnaire validity for measuring total fat intake using plasma lipid levels as criteria. Am J Epidemiol (2001) 154:1107–12.
[Abstract/Free Full Text] - Spiegelman D, Schneeweiss S, McDermott A. Measurement error correction for logistic regression models with an "alloyed gold standard." Am J Epidemiol (1997) 145:184–96.
[Abstract/Free Full Text] - Spiegelman D, Zhao B, Kim J. Correlated errors in biased surrogates: study designs and methods for measurement error correction. Stat Med (2005) 24:1657–82.[CrossRef][Web of Science][Medline]
- Greenland S. Ecologic versus individual-level sources of bias in ecologic estimates of contextual health effects. Int J Epidemiol (2001) 30:1343–50.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
Related articles in Am. J. Epidemiol.:
- The Impact of Residual and Unmeasured Confounding in Epidemiologic Studies: A Simulation Study
- Zoe Fewell, George Davey Smith, and Jonathan A. C. Sterne
Am. J. Epidemiol. 2007 166: 646-655.[Abstract] [FREE Full Text]
This article has been cited by other articles:
|
|
 |
|
  Z. Fewell, G. D. Smith, and J. A. C. Sterne Fewell et al. Respond to "Fuel for Debate" Am. J. Epidemiol., September 15, 2007; 166(6): 659 - 661. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||