American Journal of Epidemiology Advance Access originally published online on September 12, 2006
American Journal of Epidemiology 2006 164(10):1019-1025; doi:10.1093/aje/kwj310
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Practice of Epidemiology |
Address-based versus Random-Digit-Dial Surveys: Comparison of Key Health and Risk Indicators
1 National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention, Atlanta, GA
2 Abt Associates Inc., Cambridge, MA
3 Baruch College, City University of New York, New York, NY
Correspondence to Dr. Michael Link, National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention, 4770 Buford Highway, Mailstop K-66, Atlanta, GA 30351-3717 (e-mail: mlink{at}cdc.gov).
Received for publication February 9, 2006. Accepted for publication April 12, 2006.
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Use of random-digit dialing (RDD) for conducting health surveys is increasingly problematic because of declining participation rates and eroding frame coverage. Alternative survey modes and sampling frames may improve response rates and increase the validity of survey estimates. In a 2005 pilot study conducted in six states as part of the Behavioral Risk Factor Surveillance System, the authors administered a mail survey to selected household members sampled from addresses in a US Postal Service database. The authors compared estimates based on data from the completed mail surveys (n = 3,010) with those from the Behavioral Risk Factor Surveillance System telephone surveys (n = 18,780). The mail survey data appeared reasonably complete, and estimates based on data from the two survey modes were largely equivalent. Differences found, such as differences in the estimated prevalences of binge drinking (mail = 20.3%, telephone = 13.1%) or behaviors linked to human immunodeficiency virus transmission (mail = 7.1%, telephone = 4.2%), were consistent with previous research showing that, for questions about sensitive behaviors, self-administered surveys generally produce higher estimates than interviewer-administered surveys. The mail survey also provided access to cell-phone-only households and households without telephones, which cannot be reached by means of standard RDD surveys.
data collection; epidemiologic methods; population surveillance; postal service; sampling studies; telephone
Abbreviations: BRFSS, Behavioral Risk Factor Surveillance System; DSF, Delivery Sequence File; HIV, human immunodeficiency virus; RDD, random-digit dial
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
In response to concerns about increases in telephone survey nonresponse and eroding coverage of random-digit-dial (RDD) telephone sampling frames, researchers are testing new survey modes (or combinations of modes) and alternative frames. We examined survey data from one of the first efforts to use an address-based frame to conduct mail surveys of a probability sample of adults in six states and compared the results of this effort with those of an RDD telephone version of the same survey.
The growth of database technology has facilitated the development of large, computerized address databases, which may provide an inexpensive alternative to RDD for drawing household samples. One such database, the US Postal Service Delivery Sequence File (DSF), contains all delivery-point addresses served by the US Postal Service (1
). Initial evaluations of the DSF as a means of reducing the cost of enumerating urban households in area probability surveys have produced promising results (2–4), including the finding that the DSF covers approximately 97 percent of all US households (2). DSF coverage varies significantly, however, with less coverage in rural areas and lower-income areas. When comparing county-level household counts from the DSF with those from the 2000 US Census for the six states participating in the pilot study, we found that the DSF counts were at least 10 percent lower than the Census counts in nearly 90 percent of the counties where 25 percent or fewer adults lived in an urbanized area (5). Despite these limitations, the DSF appears to be a viable sampling frame for household-based surveys, providing access to households without landline telephones that are not accessible by most RDD surveys.
Using one of the world's largest RDD telephone surveys, the Behavioral Risk Factor Surveillance System (BRFSS) collects uniform, state-specific data on preventive health practices and risk behaviors linked to morbidity and death among noninstitutionalized US adults aged 18 years or older. The median state-level response rate for the BRFSS survey is approximately 50 percent (6). During 2005, we conducted a six-state pilot study of a mailed version of the BRFSS questionnaire using the DSF as a sampling frame and compared the results of this survey with those obtained from the standard RDD BRFSS questionnaire.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
The six states participating in the mail survey were California, Illinois, New Jersey, North Carolina, Texas, and Washington. These states were selected because five of them (North Carolina being the exception) had annual BRFSS response rates below 50 percent and because they collectively represented the US population well, both geographically and in terms of their racial and ethnic mix.
DSF mail survey data collection
The sample designs for the telephone BRFSS surveys are based on state-specific samples of telephone numbers. For the mail survey pilot study, however, the DSF sample frame was based on "deliverable addresses," which included post office boxes as well as residential addresses. To ensure that coverage was as complete as possible, we included seasonal addresses, vacant addresses, throwback units (locations with residents who prefer to pick up their mail at the local post office), and drop-point addresses (locations where mail is dropped off for residents to pick up, such as a general store in a rural area or a trailer park office).
The DSF frame was first stratified by the Federal Information Processing Standards code within each state. We then drew a systematic random sample of 1,680 addresses from each state, for a total of 10,080 addresses.
We used the following three techniques for selecting respondents within a household and randomly selected an equal number of addresses to be subject to each technique: 1) any adult in the household (a nonprobability approach hypothesized to have the lowest associated respondent burden); 2) the adult with the next upcoming birthday (a selection method used widely in RDD surveys); and 3) every adult in the household (each household received three questionnaires along with a toll-free number to request additional questionnaires).
Except for minor wording changes, the mail survey replicated the 75 questions on the 2005 BRFSS core questionnaire. Survey packets included a cover letter and questionnaire booklet. Instructions for requesting a Spanish-language version of the questionnaire were included on the cover in Spanish. Data collection ran from March 15 through May 15, 2005, for California, Illinois, North Carolina, Texas, and Washington and from April 1 through May 30, 2005, for New Jersey. The mail survey data collection procedures were approved by the institutional review boards of Abt Associates Inc. (Cambridge, Massachusetts) and each of the six states.
We weighted the mail survey data to account for the probability of selecting a particular household and to account for the type of within-household respondent selection method used. Next, to adjust for overrepresentation of some sex and age groups, we poststratified the data according to 2005 population totals for 13 age-by-sex categories (males aged 18–24 years were combined with males aged 25–34 years, because of the small size of the younger age group). Finally, we ratio-adjusted the data so that the sum of the weights in each state equaled the average of the total adult population across the six states, giving each state an "equal" contribution to the combined estimates—thus preventing these combined estimates from being dominated by the most populous states.
RDD telephone survey data collection
The mail surveys were conducted in parallel with the monthly RDD data collection in the six participating states. We used telephone survey data for March, April, and May 2005 and weighted the data to adjust for the state-specific sampling designs, poststratified using the same sex and age categories as those specified for the mail survey data, and ratio-adjusted them so that the sum of the final weights in each state equaled the average of the adult population totals across the six states. Details on the BRFSS questionnaire and methods used have been previously published (7) and are also available at the BRFSS website (http://www.cdc.gov/brfss).
Analysis
We used the self-reports of survey participants to assess the prevalences of four health conditions (asthma, diabetes, high blood pressure, and obesity) and four risk behaviors (smoking, binge drinking, human immunodeficiency virus (HIV) testing, and HIV risk behaviors). Asthma, diabetes, and high blood pressure were assessed by asking participants, "Have you ever been told by a doctor, nurse, or other health professional that you have [condition]?" Obesity was assessed on the basis of respondents' body mass index (weight (kg)/height (m)2), which was calculated from their self-reported height and weight; respondents were classified as obese if their body mass index was 
30. Respondents were classified as current smokers if they reported currently smoking every day or on some days, and they were classified as binge drinkers if they reported having consumed five or more drinks at least once during the preceding 30 days. They were considered to have been tested for HIV if they responded "yes" to the question, "Have you ever been tested for HIV?"; and they were considered to have engaged in behaviors linked to HIV transmission if they indicated that they had, within the previous year, used intravenous drugs, been treated for a sexually transmitted or venereal disease, given or received money or drugs in exchange for sex, or engaged in anal sex without using a condom.
The analysis was conducted in three parts:
First, we compared the proportions of item nonresponse—that is, the proportion of respondents who answered "don't know" to a particular survey question or simply did not answer that question—for the mail survey with those for the RDD survey. Missing values for survey items can reduce overall data quality. If use of particular survey modes leads to an increase in the level of item nonresponse, the estimates produced may be less precise because of the reduced sample size and may be more likely to be subject to item nonresponse bias, particularly when persons who do not answer the question differ substantially from those who respond (8). For mail survey participants, "don't know" was an explicit option for the questions about asthma, diabetes, and high blood pressure but not for the questions about current smoking, binge drinking, obesity, HIV testing, and engaging in behaviors closely linked to HIV transmission. Blank or unanswered questions were coded as refusals to answer, except where the skip logic indicated that a response was not necessary. In the telephone version, interviewers had the option of indicating both "don't know" and "refused" as participants' responses to nearly every question, but they were trained to select these responses only if the respondent volunteered them.
Second, we compared survey estimates derived from the two sets of survey data using two-way contingency tables and logistic regression to adjust for various respondent characteristics.
Third, limiting the analysis to the mail survey data, we compared estimates for households with a cell phone only with those for households with a landline phone. Households with no telephone service were excluded from this analysis because of the small number of participants in this group (n = 27).
We conducted all analyses using SPSS, version 13.0, with the Complex Samples module (SPSS, Inc., Chicago, Illinois). We used 95 percent confidence intervals to determine whether differences were statistically significant (9).
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
We received a total of 3,010 completed mail surveys from the six states. Overall and state-specific response rates for the surveys, which we calculated using American Association for Public Opinion Research (Lenexa, Kansas) response rate formula number 4, are provided in table 1 (10
). For all states except North Carolina, response rates were modestly higher in the mail survey than the telephone survey. For the mail survey, overall response rates varied by the type of within-household selection method used (35.4 percent for the "any-adult" method, 33.2 percent for the "next-birthday" method, and 28.0 percent for the "all-adults" method). The response rate for the "all-adults" method was calculated by multiplying the percentage of households that returned at least one completed questionnaire (32.9 percent) by the percentage of all adults in those households who returned a completed questionnaire (85.1 percent). More detailed analyses of differences by within-household selection method are available elsewhere (11).
|
The distributions of several demographic characteristics among mail survey respondents differed significantly from those among telephone survey respondents, including the percentage of respondents who were non-Hispanic White (mail = 76.1 percent, telephone = 68.5 percent; p < 0.001), the percentage with at least some college education (mail = 71.8 percent, telephone = 59.7 percent; p < 0.001), the percentage with an annual family income of $50,000 or more (mail = 48.6 percent, telephone = 45.5 percent; p < 0.01), the percentage having one or more children in the household (mail = 39.0 percent, telephone = 43.2 percent; p < 0.001), and the percentage having three or more adults in the household (mail = 21.2 percent, telephone = 27.1 percent; p < 0.001).
Item nonresponse
The proportion of nonresponse on completed survey questionnaires was significantly higher in the DSF mail survey than in the RDD telephone survey for questions about asthma, diabetes, high blood pressure, and current smoking; significantly lower for questions about obesity and HIV testing; and not significantly different for questions about binge drinking and HIV risk behaviors (table 2).
|
We also examined the level of missing data for several demographic variables commonly used in epidemiologic studies (table 2). In both surveys, the percentage of questionnaires with missing data was by far the highest for the question about income, though significantly higher for the telephone survey (13.0 percent vs. 6.9 percent). Other than the 2.1 percent proportion of missing data on the race/ethnicity question for the telephone survey, less than 1 percent of the data were missing.
Prevalence estimates by survey type
We found that the mail survey produced significantly higher prevalence estimates than the telephone survey for binge drinking, high blood pressure, and behaviors associated with HIV transmission but that the telephone survey produced higher estimates for HIV testing (table 3). These differences persisted even after we used logistic regression to adjust for other potential confounders, including respondents' state of residence, sex, race/ethnicity, age, education, and health-care coverage. Additionally, the odds of a respondent's being obese were higher among those responding to the mail survey once the logistic model was adjusted for other potentially confounding effects.
|
Effect of type of telephone access in household
Among the mail survey households, 13.7 percent had a landline telephone only, 79.4 percent had both a landline phone and a cellular phone, 6.0 percent had a cellular phone only, and 0.9 percent had no telephone of any type. Excluding the small number of cases with no telephone access (n = 27), we found that, compared with adults who lived in households with only a landline phone, those who lived in cell-phone-only households had four times' greater odds of having engaged in behaviors linked to HIV transmission and those who lived in a household with both a landline phone and a cell phone were significantly less likely to be current smokers or obese (table 4).
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
With few exceptions, responses obtained from the DSF-based mail survey appeared to be of similar quality as those from the RDD telephone survey. Significantly more data were missing in the mail survey than in the telephone survey for four of the eight health indicators and three of the five demographic variables, though the proportion of questionnaires with missing data was relatively low. Missing data were more of a problem in the RDD survey, with the proportion of item nonresponse exceeding 5 percent for three of the questions (HIV testing, obesity, and income). The most interesting differences were substantially lower nonresponse proportions for obesity and income in the mail survey. Even for the three questions (on asthma, diabetes, and high blood pressure) for which "don't know" was a response option on the mail survey, the proportions of missing data were still below 2.5 percent. These low proportions of nonresponse may be due to the fact that these were relatively straightforward questions concerning personal health issues about which most adults are knowledgeable.
The two surveys also produced similar prevalence estimates for four of the eight indicators. The higher estimates produced by the mail survey for binge drinking and engaging in behaviors linked to HIV transmission are in line with those from other studies showing respondents to be more likely to give socially desirable responses and less likely to report accurately about sensitive or stigmatized behaviors in interviewer-administered surveys than in self-administered surveys (12–15).
In both surveys, we found a relatively high percentage of questionnaires with missing data on height and/or weight, both of which are necessary to calculate respondents' body mass index and thus their obesity status. This appears to indicate a high level of reluctance among respondents to provide the height and/or weight information necessary to determine obesity. The substantially higher proportion of missing body mass index data on the telephone survey (8.0 percent vs. 3.4 percent on the mail survey) may have contributed to the higher obesity estimate on the mail survey, if one assumes that obesity rates are likely to be higher among nonrespondents than among respondents to height and weight questions.
The DSF mail survey was successful in reaching households with only cell phones and, to a smaller degree, households without telephones. In the mail survey, 6.0 percent of respondents indicated that they lived in households with only cell phones, which was similar to the 5.5 percent who reported living in such households from a national face-to-face survey conducted during the last 6 months of 2004 (16). The percentage of adults living in households with no telephones was somewhat smaller, however, in the DSF mail survey (0.9 percent) than in the face-to-face survey (2.4 percent). Both types of households are missed or excluded by traditional RDD surveys. Although the prevalence estimates did not differ significantly across households with different types of telephone access for most of the health indicators examined, the estimate for HIV risk behaviors was substantially higher among persons in cell-phone-only households, which suggests that people living in homes with cell phones only may differ in some important respects from those living in other types of households.
The results of this study should be viewed in the context of some limitations. First, the DSF frame did not provide universal coverage of all households, particularly in more rural and lower-income areas (5). Second, the high levels of nonresponse (low response rates) to both the mail and telephone surveys may have biased either or both sets of estimates. Third, in the absence of more direct measures (such as patient records or medical tests) or state-based health measures, we could not determine which of the two sets of survey results was more accurate. Fourth, because the study was conducted in only six states, the participants may not have been representative of either the US population as a whole or other subpopulations not represented here.
Although more research and refinement in DSF-based survey designs will be necessary before they truly rival current RDD designs, the findings provided here are promising. The data produced by the pilot mail survey appeared to be reasonably complete, and the estimates derived were largely equivalent to those produced with the use of current telephone survey methods. The DSF frame provided access to households without landline phones, which obviously cannot be reached by means of standard RDD telephone surveys. Surveys for which the DSF serves as a sampling frame could perhaps be improved through the use of a "mixed-mode" or "dual-frame" approach. A mixed-mode approach involving a mail survey with telephone follow-up of nonrespondents may be the optimal design, given that approximately 70 percent of the addresses of participants in this pilot could be matched to telephone numbers. As household access to high-speed Internet service increases, Web surveys could also be incorporated into such mixed-mode survey designs. As part of a dual-frame approach, the DSF and RDD frames could be used in complementary fashion, with the RDD frame being used primarily in more rural areas where DSF coverage is poorer and the DSF frame being used primarily in more urban areas to obtain access to households without landline telephones. Therefore, investigators should continue to examine DSF-based approaches as possible future alternatives or complements to RDD surveys.
|
ACKNOWLEDGMENTS |
|---|
The authors thank the following Behavioral Risk Factor Surveillance System state coordinators for their critical assistance in the design and execution of this study: Dr. Bonnie Davis (California), Dr. Kenneth O'Dowd (New Jersey), Dr. Ziya Gizlice (North Carolina), Jimmy Blanton (Texas), and Dr. Katrina Wynkoop Simmons (Washington). The authors also thank Pamela Giambo and Dr. Sowmya Rao, formerly of Abt Associates Inc., for their assistance with this project.
The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention or Abt Associates Inc.
Conflict of interest: none declared.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
- US Postal Service. Delivery Sequence File. Washington, DC: US Postal Service, 2005. (http://www.usps.com/ncsc/addressservices/addressqualityservices/deliverysequence.htm). (Accessed April 12, 2005).
- Iannacchione V, Staab J, Redden D. Evaluating the use of residential mailing addresses in a metropolitan household survey. Public Opin Q 2003;76:202–10.
- Staab J, Iannacchione V. Evaluating the use of residential mailing addresses in a national household survey. In: Proceedings of the American Statistical Association, Survey Methodology Section. (CD-ROM). Alexandria, VA: American Statistical Association, 2004.
- O'Muircheartaigh C, Eckman S, Weiss C. Traditional and enhanced field listing for probability sampling. In: Proceedings of the American Statistical Association, Survey Methodology Section. (CD-ROM). Alexandria, VA: American Statistical Association, 2004.
- Link M, Battaglia M, Giambo P, et al. Assessment of address frame replacements for RDD sampling frames. In: Proceedings of the American Statistical Association, Survey Methodology Section. (CD-ROM). Alexandria, VA: American Statistical Association, 2005.
- Centers for Disease Control and Prevention. Summary data quality reports, 1998–2004. Atlanta, GA: Centers for Disease Control and Prevention, 2005. (http://www.cdc.gov/brfss/technical_infodata/quality.htm). (Accessed January 31, 2006).
- Mokdad A, Stroup D, Giles W. Public health surveillance for behavioral risk factors in a changing environment. Recommendations from the Behavioral Risk Factor Surveillance Team. MMWR Recomm Rep 2003;52:1–12.[Medline]
- Mason R, Lesser V, Traugott MW. Effect of item nonresponse on nonresponse error and inference. In: Groves RM, Dillman DA, Eltinge JL, et al, eds. Survey nonresponse. New York, NY: John Wiley and Sons, Inc, 2002:149–62.
- SPSS, Inc. SPSS complex samples 13.0. Chicago, IL: SPSS, Inc, 2004.
- American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 3rd ed. Ann Arbor, MI: American Association for Public Opinion Research, 2004. (http://www.aapor.org/pdfs/standarddefs_3.1.pdf). (Accessed January 31, 2006).
- Battaglia M, Link M, Frankel M, et al. An evaluation of respondent selection methods for household mail surveys. In: Proceedings of the American Statistical Association, Survey Methodology Section. (CD-ROM). Alexandria, VA: American Statistical Association, 2005.
- Link M, Mokdad A. Use of alternative modes for health surveillance surveys: results from a web/mail/telephone experiment. Epidemiology 2005;16:701–4.[CrossRef][Web of Science][Medline]
- Turner CF, Ku L, Rogers SM. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 1998;280:867–73.
[Abstract/Free Full Text] - Dillman D, Sangster R, Tanari J, et al. Understanding differences in people's answers to telephone and mail surveys. In: Braverman MT, Slater JK, eds. Advances in survey research. (New directions for evaluation, no. 70). San Francisco, CA: Jossey-Bass Publishers, 1996:45–62.
- Aquilino WS. Interview mode effects in surveys of drug and alcohol use: a field experiment. Public Opin Q 1994;58:210–40.
[Abstract/Free Full Text] - Blumberg S, Luke J, Cynamon M. Telephone coverage and health survey estimates: evaluating the need for concern about wireless substitution. Am J Public Health 2006;96:926–31.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  S. J. Blumberg and J. V. Luke Reevaluating the Need for Concern Regarding Noncoverage Bias in Landline Surveys Am J Public Health, October 1, 2009; 99(10): 1806 - 1810. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. P. Battaglia, M. W. Link, M. R. Frankel, L. Osborn, and A. H. Mokdad An Evaluation of Respondent Selection Methods for Household Mail Surveys Public Opin Q, September 1, 2008; 72(3): 459 - 469. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. W. Link, M. P. Battaglia, M. R. Frankel, L. Osborn, and A. H. Mokdad A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys Public Opin Q, March 1, 2008; 72(1): 6 - 27. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||