American Journal of Epidemiology Advance Access published online on May 13, 2008
American Journal of Epidemiology, doi:10.1093/aje/kwn099
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Practice of Epidemiology |
Adjusting for Covariates in Studies of Diagnostic, Screening, or Prognostic Markers: An Old Concept in a New Setting
1 Division of Public Health Sciences, Fred Hutchinson Cancer Research Center, Seattle, WA
2 Department of Biostatistics, University of Washington, Seattle, WA
Correspondence to Dr. Holly Janes, Division of Public Health Sciences, Fred Hutchinson Cancer Research Center, 1100 Fairview Avenue North, M2-C200, Seattle, WA 98109 (e-mail: hjanes{at}scharp.org)
Received for publication May 4, 2007. Accepted for publication March 21, 2008.
 |
ABSTRACT |
|---|
TOP
ABSTRACT
INTRODUCTIONThe concept of covariate adjustment is well established in therapeutic and etiologic studies. However, it has received little attention in the growing area of medical research devoted to the development of markers for disease diagnosis, screening, or prognosis, where classification accuracy, rather than association, is of primary interest. In this paper, the authors demonstrate the need for covariate adjustment in studies of classification accuracy, discuss methods for adjusting for covariates, and distinguish covariate adjustment from several other related, but fundamentally different, uses for covariates. They draw analogies and contrasts throughout with studies of association.
covariance adjustment; ROC curve; sensitivity; specificity
Abbreviations: AROC, covariate-adjusted ROC curve; PSA, prostate-specific antigen; ROC, receiver operating characteristic
|
INTRODUCTION |
|---|
A popular topic in medical research today is the development of markers to classify subjects as diseased or disease free, as high or low risk, or in terms of treatment response or another future event. These markers may be the results of, for example, genetic or proteomic evaluations, imaging techniques, bacterial culture, or risk factor information. Oftentimes, factors other than disease affect marker observations. For example, levels of prostate-specific antigen (PSA), a biomarker widely used to screen men for prostate cancer, tend to increase with age. Many markers are also affected by aspects of the test procedure, test setting, or test operator; attributes of the specimen collection or storage method (e.g., storage time); or "center effects" in multicenter studies. Although adjustment for covariates is commonplace in therapeutic and etiologic studies, the issue of covariate effects is not well appreciated in the classification setting.
In this paper, we demonstrate the need for covariate adjustment and describe statistical methods that can be used to accomplish it. We also distinguish covariate adjustment from several other related, but fundamentally different, uses for covariates, including matching, prediction, and incremental value. Finally, we provide practical recommendations for determining when and how to adjust for covariates, and we include links to software that can be used to implement these techniques.
|
WHY ADJUST FOR COVARIATES? |
|---|
The classification accuracy of a continuous marker, Y, is its ability to distinguish between two groups defined by a binary outcome. We refer to the groups as "cases" and "controls." Classification accuracy is most commonly quantified by using the receiver operating characteristic (ROC) curve, a plot of the true positive fraction (sensitivity) versus the false positive fraction (1 – specificity) for the set of rules that classify an individual as "test positive" if his or her marker value is above a threshold, c, for all possible thresholds. The ROC curve characterizes the separation between case and control marker distributions. It puts markers on a common scale, thus facilitating comparisons between markers and across studies.
Confounding occurs in evaluating classification accuracy when a covariate, Z, is associated with both the marker and the binary outcome, D. Consider the traditional pooled ROC curve that combines all case observations together and all control observations together regardless of covariate value. The pooled ROC curve describes the ability of the marker to discriminate between cases and controls, and it includes the portion of discriminatory accuracy due to the covariate. Hence, it is biased relative to the covariate-specific ROC curve, the ROC curve for the marker in a population with a fixed covariate value. An example is shown in figure 1, scenario 1. A binary covariate is associated with both the outcome and the marker. For concreteness, suppose that Z is an indicator of study center, where the proportion of cases differs between the two centers. Note that the classification accuracy of the marker is the same in the two centers; it is described by the covariate-specific ROC curve, common to the two centers. In other words, Z (center) is not an effect modifier. We see that the pooled ROC curve for Y lies above the common covariate-specific ROC curve because the center with the most cases also tends to have higher marker levels. Failing to adjust for the covariate (center) leads to an overoptimistic measure of marker performance.
|
The need for covariate adjustment is much more general, however. When evaluating a marker to be used to classify individuals, if marker observations depend on a covariate, the marker should be calibrated to account for this covariate. The covariate-specific ROC curve calibrates the marker with respect to Z, as we will describe. First, recall that the pooled ROC curve describes the accuracy of rules that classify individuals using a "common" threshold, that is, a threshold independent of the subject's covariate value. The consequence of such classification is illustrated in figure 1. Observe that the same threshold applied to groups with different covariate values yields dramatically different operating characteristics (the two points on the common covariate-specific ROC curve) because of the different marker distributions in the two groups. In contrast, the covariate-specific ROC curve describes the accuracy of the marker when covariate-specific thresholds are used for classification. In particular, at a specified false positive fraction = t on the x-axis, the threshold for subjects with covariate value Z = z is the value yielding a false positive fraction of t in the z population. This approach yields constant operating characteristics across covariate populations. At each point on the covariate-specific ROC is the common (false positive fraction, true positive fraction) pair achieved by using covariate-specific thresholds. As an example, consider that age-specific thresholds have been advocated in the PSA setting to maintain low false positive fractions across different age groups (1, 2).
When a covariate affects marker observations but is independent of the binary outcome, the pooled ROC curve will always be attenuated relative to the covariate-specific ROC curve (3). The difference between the two curves reveals the increased accuracy that can be achieved when covariate-specific thresholds are used for classification. Suppose that, in the two-center example above, the proportion of cases is the same at the two centers. Observe in figure 1, scenario 2, that the pooled ROC curve for Y is now attenuated relative to the common covariate-specific ROC curve. This is directly analogous to studying the association between a predictor and an outcome when a covariate is associated with the predictor but not the outcome. The pooled odds ratio is attenuated relative to the covariate-specific odds ratio (4, 5).
Figure 1 illustrates that, when a covariate affects the distribution of marker values among controls, covariate adjustment is necessary to appropriately compare the case and control marker distributions. The covariate-adjusted ROC curve, written AROC, is a measure of covariate-adjusted classification accuracy (6). Conceptually, it is a stratified measure of performance. When the performance of the marker is the same across covariate groups (in other words, the covariate is not an effect modifier, as in figure 1), the AROC is the common covariate-specific ROC curve. It describes the performance of the marker in a population with a fixed covariate value (refer to the solid ROC curve in figure 1B). The AROC is analogous to the adjusted odds ratio in an association study. Figure 2A shows a sample AROC curve. The age-adjusted ROC curve for PSA (solid line), estimated by using data from the Physicians' Health Study (7), describes the ability of PSA to discriminate between prostate cancer cases and controls of the same age.
|
Estimation of the pooled ROC curve involves standardizing case marker observations with respect to the control reference distribution and then calculating the cumulative distribution function of these standardized marker values (3, 8, 9). Estimation of the AROC is identical except that case observations are standardized with respect to the control distribution with the same covariate value as the case (6, 9). Additional details on estimating the AROC, including links to software, are included in the Appendix.
|
THE NEED TO ADJUST FOR COVARIATES WHEN COMPARING MARKERS |
|---|
In paired therapeutic studies (i.e., crossover trials), covariate adjustment is unnecessary for patient-specific characteristics because each patient's outcomes under the two treatments are compared directly. Patient-specific characteristics cannot confound the results. In contrast, covariate adjustment is still necessary in marker studies, even when both markers are measured on each subject. Evaluating classification accuracy involves comparing ROC curves, that is, the separation between case and control marker distributions, rather than comparing the markers themselves. Consider the example shown in figure 3, where Y1 and Y2 are two markers measured on the same set of subjects. Here, Y1 and Y2 have the same covariate-specific performance (ROC curve), but the distribution of Y1 is affected by a binary covariate, Z, say, study site, whereas that of Y2 is not. Observe that the pooled ROC curves incorrectly indicate that Y2 outperforms Y1, because the ROC curve for Y1 is attenuated. A covariate-adjusted analysis is necessary to compare the performances of Y1 and Y2.
|
Markers can be compared with respect to covariate-adjusted classification accuracy using any of the commonly used ROC summary indices. For example, the area under the covariate-adjusted ROC curve (AAUC), the partial area under the covariate-adjusted ROC curve (pAAUC), sensitivity at a fixed specificity, or specificity at a fixed sensitivity can be used as summary measures. Software for estimating and comparing these indices is described in the Appendix.
|
WHY MATCHING IS NOT ENOUGH |
|---|
Matching is a design technique commonly used when there are covariate effects on classification accuracy. Cases are randomly sampled, and controls are matched to the cases with respect to covariates known to be associated with the marker and the binary outcome. Such matching is an attempt to control for confounding by the covariates. For example, in the Physicians' Health Study, cases and controls were matched with respect to age to eliminate the contribution of age to the apparent discriminatory accuracy of PSA (7). However, matching alone does not solve the problem of confounding.
In etiologic studies, it has long been understood that matching does not eliminate confounding. Odds ratios estimated from a matched study must be adjusted for the matching covariates in the analysis (4, 5). Without adjustment, the odds ratios are biased toward unity. The real role of matching is to gain efficiency in estimating these odds ratios (4, 5).
Directly analogous results have been found in the classification setting (10). That is, matching does not eliminate confounding. Rather, it converts the confounded pooled ROC curve for Y into an attenuated ROC curve. Consider the example shown in figure 1, scenario 1, where Z (study center) is associated with both the marker and the outcome. We noted previously that the pooled ROC curve for the marker is overly optimistic. A matched design forces Z to be independent of the outcome in the data (i.e., the same proportion of cases at the two study centers), as in figure 1, scenario 2. Observe that the pooled ROC curve under a matched design is still biased, attenuated toward the 45° line. The covariate-adjusted ROC curve correctly estimates the common covariate-specific ROC curve.
Perhaps an even stronger argument for covariate adjustment in matched studies involves interpretation of the ROC curves. The pooled ROC curve in the matched data describes the ability of the marker to discriminate between cases and controls with the same distribution of Z. This control population is artificially constructed and has no real-world relevance. Therefore, interpretation of the false positive fraction axis is problematic. In contrast, the AROC has direct clinical relevance; it describes the accuracy of the marker in a population with a fixed covariate value.
Figure 2A contrasts the pooled and age-adjusted ROC curves for PSA in the age-matched Physicians' Health Study. Observe that the pooled ROC curve in the matched data is generally lower than the age-adjusted ROC curve because it does not account for the matching.
As in etiologic studies, the real role for matching in studies of classification accuracy is to gain efficiency. Matching has been shown to be a maximally efficient design in many settings (10).
|
OTHER USES FOR COVARIATES: PREDICTION AND INCREMENTAL VALUE |
|---|
Covariate adjustment is commonly confused with other uses for covariates in evaluating classification accuracy. We first consider prediction. The predicted probability is the probability of the outcome (e.g., disease) as a function of marker and covariate information, that is, P(D = 1|Y, Z). It is commonly estimated by using logistic regression, where the outcome is regressed on one or more markers and other covariate information,
 |
We emphasize that the ROC curve for the predicted probability—or, equivalently, the combination score—is different from the covariate-adjusted ROC curve for the marker. The ROC curve for the combination score describes the ability of the combination of marker and covariates to discriminate between cases and controls. Observe that the combination score allows Z to contribute to discrimination and hence may perform well even if Y is a poor classifier, particularly if Z discriminates well. Figure 4 displays two examples where the ROC curve for the combination score is much higher than the AROC. In panel A, Z is a good classifier but Y is not, and the two are relatively uncorrelated. The combination score performs well, but the covariate-adjusted ROC curve for Y, that is, the ROC curve for Y stratified by Z, is low because it relates to the classification performance of Y. The pooled ROC for Y is shown for comparison; it also reflects the poor performance of Y as a classifier. In panel B, both Y and Z are good classifiers that are highly correlated. The combination score performs well, as expected, because it should be at least as good as either marker on its own. However, after adjustment for Z, the ROC curve for Y is low because, within a population where Z is fixed, Y is not a good classifier. The pooled ROC for Y, almost indistinguishable from the ROC for the combination score, is above the AROC because it does not condition on the covariate Z.
|
Another concept commonly confused with covariate adjustment is incremental value. The incremental value of the marker over the covariates is the improvement in classification performance gained by adding the marker to the covariates. It is quantified by comparing the ROC curve for the combination of marker and covariates (11) with the ROC curve for the covariates alone (12). Figure 5 shows two examples demonstrating that the covariate-adjusted performance of a marker is different from its incremental value. In panel A, Y and Z are modestly associated with the outcome and are highly correlated among cases but are relatively uncorrelated among controls. The incremental value of the marker is large, but the covariate-adjusted ROC curve is low. The pooled ROC for Y is the same as that for Z because the two variables have the same marginal classification accuracy. In panel B, Y and Z are strongly associated with the outcome and have low correlation in both cases and controls. The incremental value is small, but the covariate-adjusted ROC curve is high. Again, the pooled ROC curves for Y and Z are the same. These examples point to another contrast between studies of association and studies of classification. In association studies, the contribution of one predictor over and above another is its adjusted effect on the outcome. In studies of classification accuracy, covariate adjustment and incremental value are two different concepts.
|
|
WHEN COVARIATES AFFECT ROC PERFORMANCE |
|---|
Covariates that affect the ROC curve are analogous to effect modifiers in the association setting. The separation between cases and controls varies with covariate value. Common examples are severity of disease and protocols for specimen handling. With such covariates, a separate ROC curve should be estimated for each covariate group. Here, as well, covariate adjustment is often necessary. Estimation of covariate-specific ROC curves proceeds in two stages, as follows: 1) adjust for covariate effects on the marker distribution among controls, and 2) estimate the ROC curve as a function of Z (13–15). Consider the example shown in figure 6, where the accuracy of the marker depends on a binary covariate, Z. For concreteness, suppose again that Z is an indicator of study center. Now, however, differences in test procedures between centers affect marker performance (the separation between the distributions of cases and controls). The covariate-specific ROC curves reflect the fact that Y is much more accurate when Z = 1 than when Z = 0. Marker observations among controls also depend on Z (center), necessitating covariate adjustment, or standardization of case marker observations relative to the appropriate covariate-specific control distribution.
|
The covariate-adjusted ROC curve may still be of interest when the covariate affects discrimination. It turns out that the AROC is a weighted average of the covariate-specific ROC curves, with weights corresponding to the proportion of cases in each covariate group (6). Observe in figure 6B that the AROC for the marker lies in between the two covariate-specific ROC curves. It can be interpreted as the average performance of the marker across the two covariate groups. We see then that the AROC is directly analogous to the Mantel-Haenszel adjusted odds ratio: it is the common covariate-specific ROC curve when Z does not affect ROC performance, and a weighted average more generally. It is useful in small studies when covariate-specific ROC curves cannot be estimated with precision, and it also provides a single summary of covariate-adjusted performance for comparing markers.
Figure 6B also shows the pooled ROC curve for Y for comparison. This curve is not a weighted average of covariate-specific ROC curves; it lies below both covariate-specific ROC curves, reflecting the separation between the pooled case and control distributions.
ROC regression methods can be used to test whether covariates affect the separation between cases and controls (3). In an ROC regression model, the parameters that describe the effect of Z on the ROC curve can be tested for statistical significance. An ROC regression model was fit for PSA using Physicians' Health Study data; the resulting age-specific ROC curves are shown in figure 2B. Observe that there is essentially no variation in the ROC curves across the age groups. The hypothesis test of the equivalence of the ROC curves is not significant (p = 0.98), implying that the AROC is the common age-specific ROC curve for PSA.
|
DISCUSSION AND PRACTICAL RECOMMENDATIONS |
|---|
Rigorous evaluation of new markers being developed for medical classification purposes is essential, and adjustment for covariates is an important component of this evaluation. We advocate adjustment for covariates that affect marker observations among controls. Covariate adjustment is necessary to appropriately compare case and control marker distributions. Covariate adjustment is also important when comparing markers, even under a paired design, because unadjusted comparisons can be biased. In addition, matching does not eliminate the need for covariate adjustment; pooled measures of marker performance in matched studies are generally attenuated.
The final measure of covariate-adjusted classification accuracy will depend on whether the covariates also affect the ROC curve (whether they are effect modifiers). The AROC and its associated summary indices are appropriate when the covariates do not affect ROC performance, as well as in small studies and when comparing markers. If there is heterogeneity in the accuracy of the marker across covariate groups, estimating covariate-specific ROC curves should be the ultimate goal.
In practice, we suggest exploring associations between all measured covariates and the marker among controls. If any associations are apparent, these covariates should be used for adjustment. Alternatively, the AROC could be compared with the pooled ROC to determine whether adjustment makes any difference. In our experience, covariates must be very strongly associated with the marker to substantially affect the ROC curve.
It is common in the epidemiologic literature to distinguish between differences in pooled and covariate-adjusted measures of association due to confounding (when the covariate is associated with both the predictor and the outcome) and noncollapsibility (a property of the measure of association) (16, 17). In our setting, the pooled and covariate-adjusted ROC curves differ under confounding, that is, when the covariate is associated with both the marker and the outcome, and when the covariate is associated with the marker but not the outcome. We distinguish between the two scenarios but advocate covariate adjustment for both. When the covariate does not affect marker observations among controls, the pooled ROC is a weighted average of covariate-specific ROC curves; that is, it is collapsible (3).
The covariate-adjusted ROC curve describes the performance of rules that classify individuals by using covariate-specific thresholds. We have argued that this is entirely appropriate; when marker values depend on a covariate, the marker should be used differently in the different covariate groups. Indeed, when the covariate is independent of the outcome and affects marker values, improved marker performance is achieved. However, if for logistical or other reasons classifications are to be based on a fixed threshold that does not vary according to covariates, the pooled ROC should be used to characterize marker accuracy.
A relatively minor concern is the potential for a loss of efficiency associated with adjusting for covariates that are in fact independent of marker observations among controls but that appear to be associated in a given data set by random chance. Interestingly, matching with respect to covariates prevents this loss of efficiency; the pooled and covariate-adjusted ROC curves are equally efficient under a matched design (10).
Covariate adjustment is appropriate for covariates whose associations with the marker and the outcome are considered, in some sense, a nuisance. Covariates considered markers in their own right should be allowed to contribute to discrimination and should be combined with the marker in the predicted probability of the outcome. For example, in the prostate cancer example, age and PSA might be combined by using logistic regression. One could ask whether the combination score performs better than age alone, that is, the incremental value of PSA. We have emphasized that this is different from the age-adjusted performance of PSA. If incremental value is a primary question of interest, matching with respect to age should be avoided because the matched design does not allow assessment of incremental value (10).
|
APPENDIX |
|---|
We have developed Stata programs for estimating, plotting, and comparing covariate-adjusted ROC curves. These programs can be found at the Diagnostics and Biomarkers Statistical Center (DABS) website—http://www.fhcrc.org/science/labs/pepe/dabs. Two recent papers describe the programs in detail (18, 19).
Estimation of the AROC involves standardizing case observations with respect to the appropriate covariate-specific control distribution. A standardized case observation is called its "placement value" (3, 8, 9, 20, 21). The cumulative distribution of the case placement values is the AROC. Hence, estimation of the AROC can be divided into two steps: 1) calculate the placement values, and 2) estimate their cumulative distribution function.
To estimate the placement values, one must first decide how to model the covariates. Under existing approaches, this step can be performed by stratifying on the covariates or by assuming that they act linearly on control marker observations. Next, one must decide how to calculate the corresponding placement values, either empirically or assuming a normal model for control marker observations. Finally, the cumulative distribution function of the placement values can be calculated empirically or can be based on the assumption of a binormal or bilogistic ROC curve (22–27).
The AAUC and pAAUC can also be viewed as functions of case placement values (20, 28). We estimate the AAUC as 1 minus the sample mean of the case placement values and the pAAUC as the sample mean of the "restricted" case placement values. Another interesting summary measure is the estimated true positive fraction at a fixed false positive fraction. Inference about these summary indices is accomplished by using bootstrapping. Clustered data can be accommodated by bootstrapping the clusters.
The Stata programs described here can also be used to estimate and compare pooled ROC curves (i.e., without covariate adjustment). Another program on the DABS website can be used to implement ROC regression.
|
ACKNOWLEDGMENTS |
|---|
The authors thank Gary Longton for creating the figures.
Conflict of interest: none declared.
|
References |
|---|
- Oesterling JE, Cooner WH, Jacobsen SJ, et al. Influence of patient age on the serum PSA concentration. An important clinical observation. Urol Clin North Am (1993) 20:671–80.
- Oesterling JE, Jacobsen SJ, Cooner WH. The use of age-specific reference ranges for serum prostate specific antigen in men 60 years old or older. J Urol (1995) 153:1160–3.[CrossRef][Web of Science][Medline]
- Pepe MS. The statistical evaluation of medical tests for classification and prediction. (2003) New York, NY: Oxford University Press.
- Ahrens W, Pigeot I. Handbook of epidemiology. (2004) Berlin, Germany: Springer-Verlag.
- Rothman KJ, Greenland S. Modern epidemiology. 2nd ed. (1998) Philadelphia, PA: Lippincott-Raven.
- Janes H, Pepe MS. Adjusting for covariate effects on classification accuracy using the covariate-adjusted ROC curve. Biometrika (in press).
- Gann PH, Hennekens CH, Stampfer MJ. A prospective evaluation of plasma prostate-specific antigen for detection of prostatic cancer. JAMA (1995) 273:289–94.
[Abstract/Free Full Text] - Pepe MS, Longton G. Standardizing diagnostic markers to evaluate and compare their performance. Epidemiology (2005) 16:598–603.[CrossRef][Web of Science][Medline]
- Huang Y, Pepe MS. Biomarker evaluation using the controls as a reference population. UW biostatistics working paper series, working report 306, 2007. (http://www.bepress.com/uwbiostat/paper306).
- Janes H, Pepe MS. Matching in studies of classification accuracy: implications for analysis, efficiency, and assessment of incremental value. Biometrics (2008) 64:1–9.[CrossRef][Web of Science][Medline]
- McIntosh MW, Pepe MS. Combining several screening tests: optimality of the risk score. Biometrics (2002) 58:657–64.[CrossRef][Web of Science][Medline]
- Pepe MS, Janes H, Longton G, et al. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol (2004) 159:882–90.
[Abstract/Free Full Text] - Pepe MS. An interpretation for the ROC curve and inference using GLM procedures. Biometrics (2000) 56:352–9.[CrossRef][Web of Science][Medline]
- Alonzo TA, Pepe MS. Distribution-free ROC analysis using binary regression techniques. Biostatistics (2002) 3:421–32.[Abstract]
- Cai T, Pepe MS. Semi-parametric ROC analysis to evaluate biomarkers for disease. J Am Stat Assoc (2002) 97:1099–107.[CrossRef][Web of Science]
- Greenland S, Robins JM, Pearl J. Confounding and collapsibility in causal inference. Stat Sci (1999) 14:29–46.[CrossRef][Web of Science]
- Greenland S, Mortgenstern H. Confounding in health research. Annu Rev Public Health (2001) 22:189–212.[CrossRef][Web of Science][Medline]
- Pepe MS, Longton G, Janes H. Estimation and comparison of receiver operating characteristic curves. UW biostatistics working paper series, working paper 323, 2008. (http://www.bepress.com/uwbiostat/paper323).
- Janes H, Longton G, Pepe MS. Accommodating covariates in ROC analysis. UW biostatistics working paper series, working paper 322, 2008. (http://www.bepress.com/uwbiostat/paper322).
- Hanley JA, Hajian-Tilaki KO. Sampling variability of non-parametric estimates of the areas under receiver operating characteristic curves: an update. Acad Radiol (1997) 4:49–58.[CrossRef][Web of Science][Medline]
- Pepe MS, Cai T. The analysis of placement values for evaluating discriminatory measures. Biometrics (2004) 60:528–35.[CrossRef][Web of Science][Medline]
- Swets JA. Indices of discrimination or diagnostic accuracy: their ROCs and implied methods. Psychol Bull (1986) 99:100–17.[CrossRef][Web of Science][Medline]
- Swets JA, Pickett RM. Evaluation of diagnostic systems: methods from signal detection theory. (1982) New York, NY: Academic Press.
- Hanley JA. The use of the ‘binormal’ model for parametric ROC analysis of quantitative diagnostic tests. Stat Med (1996) 15:1575–85.[CrossRef][Web of Science][Medline]
- Hanley JA. The robustness of the ‘binormal’ assumptions used in fitting ROC curves. Med Decis Making (1988) 8:197–203.
[Abstract/Free Full Text] - Dorfman DD, Alf E. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data. J Math Psychol (1969) 6:487–96.[CrossRef][Web of Science]
- Ogilvie JC, Creelman CD. Maximum-likelihood estimation of receiver operating characteristic curve parameters. J Math Psychol (1968) 5:377–91.[CrossRef][Web of Science]
- Dodd L, Pepe MS. Partial AUC estimation and regression. Biometrics (2003) 59:614–23.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  T. Neogi, D. Felson, J. Niu, M. Nevitt, C. E Lewis, P. Aliabadi, B. Sack, J. Torner, L. Bradley, and Y. Zhang Association between radiographic features of knee osteoarthritis and pain: results from two cohort studies BMJ, August 21, 2009; 339(aug21_1): b2844 - b2844. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. G. Baker Improving the Biomarker Pipeline to Develop and Evaluate Cancer Screening Tests J Natl Cancer Inst, August 19, 2009; 101(16): 1116 - 1119. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
V. Melotte, M. H. F. M. Lentjes, S. M. van den Bosch, D. M. E. I. Hellebrekers, J. P. J. de Hoon, K. A. D. Wouters, K. L. J. Daenen, I. E. J. M. Partouns-Hendriks, F. Stessels, J. Louwagie, et al. N-Myc Downstream-Regulated Gene 4 (NDRG4): A Candidate Tumor Suppressor Gene and Potential Biomarker for Colorectal Cancer J Natl Cancer Inst, July 1, 2009; 101(13): 916 - 927. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. C. Glockner, M. Dhir, J. M. Yi, K. E. McGarvey, L. Van Neste, J. Louwagie, T. A. Chan, W. Kleeberger, A. P. de Bruine, K. M. Smits, et al. Methylation of TFPI2 in Stool DNA: A Potential Novel Biomarker for the Detection of Colorectal Cancer Cancer Res., June 1, 2009; 69(11): 4691 - 4699. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 Y. Huang and M. S. Pepe Biomarker evaluation and comparison using the controls as a reference population Biostat., April 1, 2009; 10(2): 228 - 244. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K. Soreide Receiver-operating characteristic curve analysis in diagnostic, prognostic and predictive biomarker research J. Clin. Pathol., January 1, 2009; 62(1): 1 - 5. [Full Text] [PDF]
|
|
|
|
|
|
M. S. Pepe, Z. Feng, H. Janes, P. M. Bossuyt, and J. D. Potter Pivotal Evaluation of the Accuracy of a Biomarker Used for Classification or Prediction: Standards for Study Design J Natl Cancer Inst, October 15, 2008; 100(20): 1432 - 1438. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||