American Journal of Epidemiology Advance Access originally published online on August 28, 2007
American Journal of Epidemiology 2007 166(11):1345-1354; doi:10.1093/aje/kwm212
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
PRACTICE OF EPIDEMIOLOGY |
When Epidemiology Meets the Internet: Web-based Surveys in the Millennium Cohort Study
1 Department of Defense Center for Deployment Health Research, Naval Health Research Center, San Diego, CA
2 College of Public Health, University of Iowa, Iowa City, IA
Correspondence to Dr. Besa Smith, Department of Defense Center for Deployment Health Research, Naval Health Research Center, P.O. Box 85122, San Diego, CA 92186-5122 (e-mail: Besa.Smith{at}med.navy.mil).
Received for publication March 1, 2007. Accepted for publication June 19, 2007.
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Almost 60% of American households were connected to the Internet in 2001, when the Millennium Cohort Study, the largest longitudinal study ever undertaken by the Department of Defense, was launched. To facilitate survey completion, increase data integrity, and encourage cohort retention while maintaining the highest standards of participant privacy, an online questionnaire was made available on the World Wide Web in addition to a traditional paper questionnaire sent via US mail. Over 50% of 77,047 participants chose to enroll in the study via the Web, affording substantial cost savings to the project. Using multivariable logistic regression, the authors compared the demographic and health characteristics of Web responders with those of paper responders. Web responders were slightly more likely to be male, to be younger, to have a high school diploma or college degree, and to work in information technology or another technical occupation. Web responders were more likely to be obese and to smoke more cigarettes and were less likely to be problem alcohol drinkers and to report occupational exposures. Question completion rates were 98.3%, on average, for both Web and paper responders. Web responders provided more complete contact information, including their e-mail addresses. These results demonstrate the value of survey research conducted over the Internet in concert with traditional mail survey strategies.
data collection; epidemiologic methods; health surveys; Internet; longitudinal studies; military personnel; questionnaires
Abbreviations: CI, confidence interval; OR, odds ratio
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Researchers interested in data collection through survey implementation have exciting new and unique opportunities to conduct research using the World Wide Web. This new medium for data collection has become available through an explosion in both the growth of the Web and the use of personal computers. Almost 60 percent of US households were connected to the Internet in 2001 (1), with this number jumping to 75 percent by 2004 (2). In light of this growth, it is evident that researchers will begin to use and investigate the Web as a tool, learning the benefits as well as pitfalls of using the Web for data collection. It is highly anticipated that these methods will become a more viable way of implementing future surveys, providing a number of advantages over traditional methods, including convenience for the participant, potentially large cost savings for the researcher, efficiency in data collection, higher data quality, a degree of perceived anonymity for the participant, and the ability to increase response rates (3–9).
A cautious tone is apparent in the literature, however, when addressing survey implementation on the Web. There are many unknowns regarding survey construction, implementation, and Web security, prompting some authors to express concerns associated with Web-based surveys (3, 6, 8, 10). These concerns include sampling problems, lack of participant access to computers with Internet connections, Internet privacy concerns, and response inconsistency across different media. Still, others maintain that the bias associated with collecting information over the Web is no greater than that introduced by traditional paper methods (11). More research is necessary to better quantify the risks and benefits of conducting survey research over the Internet.
In 2001, the largest Department of Defense longitudinal study to date was launched. The Millennium Cohort Study, a prospective study of more than 100,000 military personnel, will survey participants every 3 years over a 21-year period (12, 13). Originally, the cohort was to receive the survey in traditional paper form via the US Postal Service. However, the initial steering committee and focus groups encouraged investigators to additionally offer submission of the survey via the Web. A mixed-mode strategy was developed to facilitate survey completion and increase data integrity, while continuing to maintain the highest level of participant privacy.
The combination of Web and paper data submission presented an opportunity to investigate and compare demographic, occupational, and self-reported exposure and health characteristics between participants choosing Web survey submission and those choosing paper survey submission. Our objective in this analysis was to evaluate the use of a mixed-mode survey approach in a large 21-year prospective study. We tested the hypothesis that there were no differences in demographic, military, and occupational characteristics or in self-reported behavioral, health, and exposure characteristics between Web and paper responders.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Study population
The methods used for the Millennium Cohort Study have been described in detail elsewhere (13). In brief, persons invited to participate in the Millennium Cohort Study were derived from a stratified random sample of US military personnel representing approximately 11.3 percent of the 2.2 million men and women on active service rosters as of October 1, 2000. To ensure adequate power in small subgroups, Reserve and National Guard members, women, and past deployers were oversampled. Using a modified Dillman approach (14, 15), Web and US mail-based enrollment began in July 2001. All invitees were contacted by traditional US mail with postcard reminders and survey instruments. For those from whom an e-mail address could be obtained, an e-mail invitation was also sent. Participants were allowed to submit a survey by paper or online via the Web site, according to their preference. All correspondence, paper or electronic, invited potential respondents to complete the survey via the Web site. Because of the potential to increase data quality and reduce study costs, a nominal cost-savings initiative was offered to persons who completed the survey online. Enrollment ended on June 30, 2003, with 77,047 consenting participants (a 37 percent response rate). Previous analyses have demonstrated that Millennium Cohort Study participants well represent the US military, as measured by demographic and health characteristics and reliable health and exposure reporting (13, 16–19).
Demographic, occupational, and military-specific data included date of birth, marital status, gender, race/ethnicity, occupation, service branch (Army, Navy, Coast Guard, Air Force, or Marine Corps), service component (active duty or Reserve/National Guard), highest educational level, pay grade, and past deployment to Southwest Asia, Bosnia, or Kosovo (between January 1, 1998 and September 1, 2000). For this study, missing demographic data for marital status, occupation, education, and pay grade were supplemented with self-reported data from the survey when possible.
Survey instrument
The Millennium Cohort Study questionnaire consists of more than 450 questions regarding diagnosed medical conditions, reported symptoms, psychosocial history, physical status, functional status, problem alcohol use, tobacco use, occupation, and basic demographic and contact data (12, 13). Contact information was obtained to assist in participant tracking in this 21-year longitudinal effort. Standardized scoring instruments were used for established reliability and validity and for the ability to compare results with those from other populations. They included the PRIME-MD Patient Health Questionnaire (20–22), the Medical Outcomes Study 36-Item Short Form Health Survey for Veterans (23), a Department of Veterans Affairs Gulf War survey (24, 25), the Posttraumatic Stress Disorder Symptom Checklist–Civilian Version (26, 27), and the CAGE questionnaire for the detection of problem drinking behavior (28). Additionally, free-text fields were available to allow participants to report conditions, problems, concerns, and exposures not listed on the survey. The paper survey was created, scanned, and verified using mark-sense TeleForm Elite software (Cardiff Software, Inc., Vista, California).
Web site
The Web site was originally designed as a place for participants to learn more about the study and to obtain research findings as they became available. Informational Web pages, including useful links, study contact information, documentation of annual study review processes, and displays of signed endorsements from leaders in the military community were all important components included to establish a personal relationship with each and every participant and to emphasize the legitimacy and need for the study. During pilot testing, however, the importance of including the opportunity to submit the questionnaire via the Internet became apparent.
Web metrics
Web-based submission also presented us with information not available in traditional paper survey research. Time stamps allowed establishment of the exact dates and times at which participants began and completed their surveys; therefore, the average time to completion of the questionnaire could be calculated in comparison with estimates initially made on the basis of focus groups. Time stamps were also logged to reflect time of modification for persons who elected to save their responses and return later to complete the survey. Additionally, Web metrics allowed for the exploration of individual page submissions to identify lengthy or difficult-to-answer questions for future survey redesign. Finally, by evaluating time stamps for days of the week and times of the day in which participants most frequently submitted their online surveys, we were able to develop optimal mailing schedules to disseminate invitations.
Statistical analyses of Web responders versus paper responders
Initial investigation of population characteristics included univariate analyses with chi-square tests of association to assess significant differences in the demographic, military, behavioral, and health characteristics of persons submitting the survey via the Web when compared with those submitting paper surveys. We conducted a multivariable exploratory model analysis to assess multicollinearity, significant associations, and possible confounding, while simultaneously adjusting for all other covariates in the model. We used multivariable logistic regression to compare the differences in adjusted odds of Web submission while controlling for possible confounders.
A graphical investigation of question completion rates was conducted for assessment of differences in mode of survey submission and differences in fatigue between the Web and paper surveys. All analyses were completed using SAS software (version 9.1; SAS Institute, Inc., Cary, North Carolina).
This research was approved by the institutional review board at the Naval Health Research Center and was conducted in compliance with all applicable federal regulations governing the protection of human subjects in research.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Of the 77,047 persons who responded during 2001–2003, 42,168 (54.7 percent) completed the questionnaire online, while 34,879 (45.3 percent) completed a paper questionnaire. Demographic data were missing for 87 persons, leaving 76,960 (99.9 percent) for further statistical analyses. Table 1 shows proportional differences by method of response, as well as adjusted odds ratios and 95 percent confidence intervals calculated by multivariable logistic regression. After adjustment for age, education, marital status, race/ethnicity, past deployment history, military rank, service component, service branch, and occupation, men were 1.4 times more likely to complete the survey online than were women (odds ratio (OR) = 1.37, 95 percent confidence interval (CI): 1.32, 1.42). Participants in middle age ranges and middle educational levels were more likely to fill out the questionnaire on the Web. Web responders were also more likely to be married (OR = 1.18, 95 percent CI: 1.14, 1.22), to be on active duty (OR = 1.73, 95 percent CI: 1.67, 1.79), and to have an occupation related to information technology or another technical specialty. Finally, Army (OR = 1.39, 95 percent CI: 1.33, 1.45) and Air Force (OR = 1.39, 95 percent CI: 1.33, 1.46) personnel were individually 1.4 times more likely to respond using the Web than were Navy and Coast Guard personnel.
|
by mode of survey response (World Wide Web or mailed questionnaire), Millennium Cohort Study, 2001–2003Potentially traumatic exposures, including knowledge of or witnessing physical abuse, dead and/or decomposing bodies, prisoners of war/refugees, chemical or biological warfare agents, other medical countermeasures for exposure to a chemical or biological warfare agent, and alarms necessitating the wearing of protective gear, were self-reported significantly less often by Web responders than by paper responders, after adjustment for sex, age, education, marital status, race/ethnicity, deployment status, military pay grade, service component, service branch, and occupation (table 2). Only anthrax vaccination was reported more often among Web responders (OR = 1.05, 95 percent CI: 1.02, 1.09). Web responders also self-reported less exposure to occupational hazards requiring protective equipment and less routine contact with paint/solvents, microwaves, and pesticides.
|
Self-reported behavioral health characteristics are displayed in table 3 by mode of survey response. After adjustment for sex, age, education, marital status, race/ethnicity, deployment history, military rank, service component, service branch, and occupation, Web responders were more likely to self-report their health as fair, good, or very good, were 1.2 times more likely to be overweight (OR = 1.17, 95 percent CI: 1.13, 1.21) or obese (OR = 1.20, 95 percent CI: 1.14, 1.26), and were less likely to be classified as problem drinkers (OR = 0.81, 95 percent CI: 0.78, 0.84). Web responders were less likely to report smoking cigars (OR = 0.88, 95 percent CI: 0.85, 0.92) but slightly more likely to report smoking cigarettes (OR = 1.08, 95 percent CI: 1.04, 1.10). Among participants who reported smoking at least 100 cigarettes in their lifetime, Web responders also had a slightly higher mean number of pack-years than paper responders (5.1 and 4.9, respectively; p = 0.04).
|
Figure 1 shows the overall completion percentages for individual questions, which were found to be similar by mode of response, although some questions were found to have a higher percentage of completion by Web responders. Of the 42,127 Web participants, the average completion percentage for all survey questions was 98.3, with a maximum of 100 and a minimum of 87.6. Among Web responders, 87 percent provided a personal e-mail address, 68 percent provided a personal telephone number, and 51 percent provided a phone number for an alternative contact. Of the 34,833 paper survey participants, the average completion percentage was 98.3, with a maximum of 99.7 and a minimum of 79.0. Among paper responders, 32 percent provided a personal e-mail address, 36 percent provided a personal phone number, and 26 percent provided a phone number for an alternative contact.
|
There were 48,958 persons who logged onto the Web survey at some point during enrollment and, of those, 87.3 percent (n = 42,747) submitted their survey via the Web. Of the 6,211 who logged on but did not submit a survey, 814 (13 percent) ultimately submitted a paper copy of the survey. The average number of days from initial Web log-on to survey submission was 7 (standard deviation, 40.6); however, 89 percent of Web participants completed the survey the same day they initially logged on. Among those who submitted a survey the same day that they logged on, the median amount of time it took to complete the survey was 29 minutes (interquartile range, 16.0).
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
Cohort study investigators often rely heavily on survey data to evaluate exposures and outcomes at multiple points in time. Sophisticated methods of manual and electronic data collection in such studies continue to evolve. The dramatic advances in the development of Web-based data collection methods are an important part of this evolution. In this study, the benefits and feasibility of employing both Web- and paper-based data collection techniques were investigated. In this report, we have described the consistency of question reporting as well as a lack of survey-length fatigue bias among both paper and Web responders. Further demographic, behavioral, and health characteristics that distinguished Web and paper responders in a large population-based cohort have been reported and discussed.
The hybrid data submission system, using both postal and Internet capabilities, provided more opportunities for participants to take the survey and easier access for those who may have been on deployment or stationed abroad. Offering the survey both via the Web and on paper provided respondents with a choice as to how to participate, appealing both to those who may have been concerned about informational privacy on the Internet and to those concerned about sending personal information by US mail. The majority of persons who went to the survey on the Internet completed the survey and submitted it online (87 percent). Web responders represented more than half of the cohort (55 percent) when enrollment closed. In the current study, the median amount of time it took Web responders to complete the questionnaire online (29 minutes) accurately reflected the 30-minute estimate included in the instructions of the questionnaire based on focus group testing.
There are benefits and drawbacks to both means of submission. Online survey submission included electronic consent as part of the Web site log-on process. This allowed for easy documentation of consent and mitigated the need for subsequent participant contact, as is necessary when obtaining missing consent forms for paper responders. Data quality can be enhanced by electronic skip patterns that automatically skip irrelevant questions on the Web survey. However, these techniques rely on more advanced computer resources for the responder, and their absence may preclude some persons from accessing the electronic questionnaire. Paper surveys cannot prevent a respondent from entering multiple responses to a single-response question. This can actually provide researchers with unsolicited yet useful additional information. However, deciding on the most appropriate answer when a participant selects multiple answers can be complex and burdensome, quite often resulting in a loss of data. Certain programming techniques in Web-based questionnaires can permit only one answer to be selected, thereby preventing misinterpretation of multiple answers and reducing missing data. Electronic free-text fields offer keystroke survey responses instead of the handwritten responses given on paper surveys, which can be difficult to decipher and costly to transfer to a database. Finally, though one might hypothesize that individual question completion rates and survey fatigue might differ by mode of response, we found comparable results (figure 1).
The cost-effectiveness of Web submission can be measured in terms of data quality as well as direct financial savings. In the Millennium Cohort Study, there were significant initial costs of establishing the Web capability, such as purchasing identical servers to run simultaneously in the event of a primary failure. Gaining team expertise in the form of hiring and training for Web site and database construction, data transfer techniques, and implementation, which included significant understanding of security certificates, was necessary for the online survey. After the initial start-up costs, money for sustaining team expertise and periodic Web site maintenance and security upgrades was budgeted as an ongoing expense, even for non-survey-cycle years. However, these costs are much lower than the labor-intensive costs of compiling, scanning, verifying, sorting, and filing thousands of paper submissions. Additional costs associated with mailed surveys include the logistic and financial challenges associated with securely storing thousands of surveys and consent forms as required by human subjects guidelines. Unforeseen expenses included costs involved in remailing potential participants when several thousand surveys were returned without signed consent statements. These extra costs associated with mailed surveys, in addition to the significant mailing costs of outgoing and return postage for survey packets, resulted in substantial cost savings associated with Web-based submission. For this reason, a modest incentive (a T-shirt or a 60-minute phone card valued at approximately $5.00) was offered to persons choosing to complete the questionnaire online. Even after factoring in the free gift, it was conservatively estimated that each participant who elected to complete the questionnaire online rather than by paper saved the study approximately $50.00. The cost savings to the project, to date, have been estimated to be as high as $2 million. Given that the current project is expected to continue for more than 20 years and the costs of Web surveys and traditional mail surveys are not static, the full savings of using Web-based technology have yet to be realized.
The finding that age influenced the mode of submission has been reported previously (29). In this case, persons in the birth cohort of 1960–1969 and subsequent cohorts were more likely to use the Web than older participants, and this may reflect greater technological savvy among recent generations. Persons with high school diplomas or college degrees were more likely to use the Web than those with lower or higher educational levels. Age and education, however, appeared to be independent in multivariable adjusted models. The paradoxical education relation (i.e., the middle-educated differing from persons with lower or higher educational levels) has been suggested in other studies of Web use (30, 31).
The finding that men enroll via the Web more often than women is interesting and has been noted previously (32); however, this may be a population-specific occurrence, as another study has suggested that women enroll via the Web with higher frequency (11). Married and previously married participants used the Web more often than unmarried participants, suggesting that computer use may be related to family environment. The increased adjusted odds of Web response among persons in occupations related to information technology and other technical specialties may have been due to increased computer access and/or familiarity and comfort with computer use. Additionally, less Web response among Navy and Coast Guard members may have reflected differential access to computers while at sea.
Of particular concern to researchers who base exposure measurement on self-reports was the relatively consistent lower reporting of occupational exposures among Web responders. Because these measures are adjusted for the differences in population composition, it is difficult to surmise why Web responders were at lower odds for these self-reported exposures, with the exception of having received the anthrax vaccine. With the growing concerns about compromised electronic data, as well as identity theft, some people may feel that their data are not as secure via submission on the Web. While the Millennium Cohort Study team has taken extraordinary care to ensure the protection of these data and to convey our dedication to the privacy protection to our members, feelings of insecurity may persist. More research into these findings is warranted.
The finding that Web responders were more likely to report a weight in the overweight or obese range is interesting. This may reflect subtle occupational differences, such as being employed in a sedentary work environment. However, all military members are required to meet standards for weight and physical fitness, regardless of occupation. Outside of work, these findings may indicate a more inactive personal lifestyle among Web responders, with more individual time being allocated to computer use. Interpretation of health outcome results may need to be met with caution in research where only one survey mode, traditional paper or the Web, is utilized. Given that excess weight is so strongly associated with many long-term health problems, this finding is important and may highlight the need to offer both traditional paper surveys and Web surveys in any study of chronic health outcomes.
Web responders were significantly less likely to be classified as problem drinkers, but they were significantly more likely to report smoking cigarettes. These were surprising results given that tobacco and alcohol use are often closely associated (33, 34). Although differences in these behaviors were small, the finding may be consistent with other characteristics of technologically savvy persons and underscores the importance of exploring all differences between study participants who opt for different modes of participation.
Limitations of this study should be noted. The rate of response to the Millennium Cohort Study enrollment invitation was 37 percent, and therefore participants may not be representative of the US military in general (13). However, investigation of possible biases suggested a representative sample of military personnel as measured by demographic and health characteristics and reliable health and exposure reporting (13, 16–19). With the robust sample size and the unique and abundant characteristics available for study, these data suggest subpopulations that may be more or less inclined to respond to a Web-based survey when both Web and traditional paper modes are offered.
In summary, the Millennium Cohort Study has demonstrated successful implementation of multimodal survey data collection using the Web in concert with postal mailed questionnaires. Web response was associated with more complete data and marked cost savings at a minimal risk of enrolling a nonrepresentative group. Web responders were very comparable to paper responders with regard to most demographic and health metrics, but subtle differences were observed. These differences may be consistent with important health challenges, such as obesity, that distinguish a growing generation of computer users. Such differences may become less distinct as Internet use becomes more global over time. Only a large, prospective study like the 21-year Millennium Cohort Study will be able to fully assess changing demographic and health characteristics over time. In the meantime, it remains important to offer multiple modes of participation when a diverse population is sought, and equally important to understand response biases and differences among cohort study participants.
|
ACKNOWLEDGMENTS |
|---|
This article is report 07-06, supported by the Department of Defense, under work unit no. 60002. This work was also supported by the Henry M. Jackson Foundation for the Advancement of Military Medicine (Rockville, Maryland).
The authors thank Scott L. Seggerman from the Management Information Division of the Defense Manpower Data Center (Seaside, California) and Dr. Karl E. Friedl from the US Army Medical Research and Materiel Command (Fort Detrick, Maryland). Additionally, they thank Lacy Farnell, Gia Gumbs, Isabel Jacobson, Cynthia Leard, Travis Leleu, Robert Reed, Steven Spiegel, Damika Webb, Kari Welch, and James Whitmer from the Department of Defense Center for Deployment Health Research (San Diego, California) and Michelle Stoia from the Naval Health Research Center (San Diego, California).
In addition to the authors, the Millennium Cohort Study Team comprises Drs. Paul J. Amoroso, Edward J. Boyko, Gary D. Gackstetter, Tomoko I. Hooper, James R. Riddle, and Timothy S. Wells.
The views expressed in this article are those of the authors and do not reflect the official policy or position of the Department of the Navy, the Department of the Army, the Department of the Air Force, the Department of Defense, the Department of Veterans Affairs, or the US government. This research was conducted in compliance with all applicable federal regulations governing the protection of human subjects in research (protocol NHRC.2000.007).
Conflict of interest: none declared.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONReferences |
|---|
- Nielsen//NetRatings. Internet penetration reaches 60 percent in the U.S. according to Nielsen//NetRatings (2001) New York, NY: NetRatings, Inc. February 28. (http://www.nielsen-netratings.com/pr/pr_010228.pdf).
- Nielsen//NetRatings. Three out of four Americans have access to the Internet, according to Nielsen//NetRatings (2004) New York, NY: NetRatings, Inc. March 18. (http://www.nielsen-netratings.com/pr/pr_040318.pdf).
- Best SJ, Krueger B, Hubbard C, et al. An assessment of the generalizability of Internet surveys. Soc Sci Comput Rev (2001) 19:131–45.
[Abstract/Free Full Text] - Hewson CM, Laurent D, Vogel CM. Proper methodologies for psychological and sociological studies conducted via the Internet. Behav Res Methods Instrum Comput (1996) 28:186–91.[Web of Science]
- Krantz JH, Ballard J, Scher J. Comparing the results of laboratory and World Wide Web samples of the determinants of female attractiveness. Behav Res Methods Instrum Comput (1997) 29:264–9.[Web of Science]
- Schaefer DR, Dillman DA. Development of standard e-mail methodology: results of an experiment. Public Opin Q (1998) 62:378–97.[CrossRef][Web of Science]
- Schmidt WC. World-Wide Web survey research: benefits, potential problems, and solutions. Behav Res Methods Instrum Comput (1997) 29:274–9.[Web of Science]
- Smith MA, Leigh B. Virtual subjects: using the Internet as an alternative source of subjects and research environment. Behav Res Methods Instrum Comput (1997) 29:496–505.[Web of Science]
- Shettle C, Mooney G. Monetary incentives in U.S. government surveys. J Off Stat (1999) 15:231–50.
- Swoboda WJ, Mühlberger N, Weitkunat R, et al. Internet surveys by direct mailing. Soc Sci Comput Rev (1997) 15:242–55.
[Abstract/Free Full Text] - Ekman A, Dickman PW, Klint A, et al. Feasibility of using web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol (2006) 21:103–11.[CrossRef][Web of Science][Medline]
- Gray GC, Chesbrough KB, Ryan MA, et al. The Millennium Cohort Study: a 21-year prospective cohort study of 140,000 military personnel. Mil Med (2002) 167:483–8.[Web of Science][Medline]
- Ryan MA, Smith TC, Smith B, et al. Millennium Cohort: enrollment begins a 21-year contribution to understanding the impact of military service. J Clin Epidemiol (2007) 60:181–91.[CrossRef][Web of Science][Medline]
- Dillman DA. Mail and telephone surveys: the total design method (1978) New York, NY: John Wiley and Sons, Inc.
- Dillman DA. Mail and internet surveys: the tailored design method (2000) New York, NY: John Wiley and Sons, Inc.
- Chretien JP, Chu LK, Smith TC, et al. Demographic and occupational predictors of early response to a mailed invitation to enroll in a longitudinal health study. BMC Med Res Methodol (2007) 7:6. (Electronic article).[CrossRef][Medline]
- Smith B, Leard CA, Smith TC, et al. Anthrax vaccination in the Millennium Cohort: validation and measures of health. Am J Prev Med (2007) 32:347–53.[CrossRef][Web of Science][Medline]
- Smith TC, Jacobson IG, Smith B, et al. The occupational role of women in military service: validation of occupation and prevalence of exposures in the Millennium Cohort Study. Int J Environ Health Res (2007) 17:271–84.[CrossRef][Medline]
- Smith TC, Smith B, Jacobson IG, et al. Reliability of standard health assessment instruments in a large, population-based cohort study. Ann Epidemiol (2007) 17:525–32.[CrossRef][Medline]
- Spitzer RL, Williams JB, Kroenke K, et al. Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 Study. JAMA (1994) 272:1749–56.
[Abstract/Free Full Text] - Spitzer RL, Kroenke K, Williams JB, et al. Validation and utility of a self-report version of PRIME-MD: The PHQ Primary Care Study. JAMA (1999) 282:1737–44.
[Abstract/Free Full Text] - Spitzer RL, Williams JB, Kroenke K, et al. Validity and utility of the PRIME-MD patient health questionnaire in assessment of 3000 obstetric-gynecologic patients: the PRIME-MD Patient Health Questionnaire Obstetrics-Gynecology Study. Am J Obstet Gynecol (2000) 183:759–69.[CrossRef][Web of Science][Medline]
- Ware N, Kleinman A. Culture and somatic experience: the social course of illness in neurasthenia and chronic fatigue syndrome. Psychosom Med (1992) 54:546–60.
[Abstract/Free Full Text] - Gray GC, Reed RJ, Kaiser KS, et al. The Seabee Health Study: self-reported multi-symptom conditions are common and strongly associated among Gulf War veterans. Am J Epidemiol (2002) 155:1033–44.
[Abstract/Free Full Text] - Kang HK, Mahan CM, Lee KY, et al. Illnesses among United States veterans of the Gulf War: a population-based survey of 30,000 veterans. J Occup Environ Med (2000) 42:491–501.[Web of Science][Medline]
- Weathers F, Huska J, Keane T. The PTSD Checklist Military Version (PCL-M) (1991) Boston, MA: National Center for PTSD.
- Lang AJ, Laffaye C, Satz LE, et al. Sensitivity and specificity of the PTSD checklist in detecting PTSD in female veterans in primary care. J Trauma Stress (2003) 16:257–64.[CrossRef][Web of Science][Medline]
- Ewing JA, Detecting alcoholism. The CAGE questionnaire. In: JAMA (1984) 252:1905–7.
[Abstract/Free Full Text] - Morrell RW, Mayhorn CB, Bennett J. A survey of World Wide Web use in middle-aged and older adults. Hum Factors (2000) 42:175–82.[CrossRef][Web of Science][Medline]
- Leece P, Bhandari M, Sprague S, et al. Internet versus mailed questionnaires: a controlled comparison. J Med Internet Res (2004) 6:e39. (Electronic article).[Medline]
- McCabe SE. Comparison of web and mail surveys in collecting illicit drug use data: a randomized experiment. J Drug Educ (2004) 34:61–72.[CrossRef][Web of Science][Medline]
- McCabe SE, Diez A, Boyd CJ, et al. Comparing web and mail responses in a mixed mode survey in college alcohol use research. Addict Behav (2006) 31:1619–27.[CrossRef][Web of Science][Medline]
- Jensen MK, Sorensen TI, Andersen AT, et al. A prospective study of the association between smoking and later alcohol drinking in the general population. Addiction (2003) 98:355–63.[CrossRef][Web of Science][Medline]
- Amit Z, Weiss S, Smith BR, et al, Use of caffeine-based products and tobacco in relation to the consumption of alcohol. An examination of putative relationships in a group of alcoholics in Israel. In: Eur Addict Res (2004) 10:22–8.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  N. S. Granado, T. C. Smith, G. M. Swanson, R. B. Harris, E. Shahar, B. Smith, E. J. Boyko, T. S. Wells, M. A.K. Ryan, and for the Millennium Cohort Study Team Newly Reported Hypertension After Military Combat Deployment in a Large Population-Based Study Hypertension, November 1, 2009; 54(5): 966 - 973. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. Smith, C. A. Wong, T. C. Smith, E. J. Boyko, G. D. Gackstetter, and Margaret A. K. Ryan for the Millennium Cohort Stud Newly Reported Respiratory Symptoms and Conditions Among Military Personnel Deployed to Iraq and Afghanistan: A Prospective Population-based Study Am. J. Epidemiol., October 22, 2009; (2009) kwp287v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. A LeardMann, T. C Smith, B. Smith, T. S Wells, M. A K Ryan, and for the Millennium Cohort Study Team Baseline self reported functional health and vulnerability to post-traumatic stress disorder after combat deployment: prospective US military cohort study BMJ, April 16, 2009; 338(apr16_2): b1273 - b1273. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
I. G. Jacobson, T. C. Smith, B. Smith, P. K. Keel, P. J. Amoroso, T. S. Wells, G. P. Bathalon, E. J. Boyko, M. A. K. Ryan, and for the Millennium Cohort Study Team Disordered Eating and Weight Changes After Deployment: Longitudinal Assessment of a Large US Military Cohort Am. J. Epidemiol., February 15, 2009; 169(4): 415 - 427. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. Dolnicar, C. Laesser, and K. Matus Online Versus Paper: Format Effects in Tourism Surveys Journal of Travel Research, February 1, 2009; 47(3): 295 - 316. [Abstract] [PDF]
|
|
|
|
 |
|
 M. K. Nock, G. Borges, E. J. Bromet, C. B. Cha, R. C. Kessler, and S. Lee Suicide and Suicidal Behavior Epidemiol. Rev., November 1, 2008; 30(1): 133 - 154. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 I. G. Jacobson, M. A. K. Ryan, T. I. Hooper, T. C. Smith, P. J. Amoroso, E. J. Boyko, G. D. Gackstetter, T. S. Wells, and N. S. Bell Alcohol Use and Alcohol-Related Problems Before and After Military Combat Deployment JAMA, August 13, 2008; 300(6): 663 - 675. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. C Smith, M. A K Ryan, D. L Wingard, D. J Slymen, J. F Sallis, D. Kritz-Silverstein, and for the Millennium Cohort Study Team New onset and persistent symptoms of post-traumatic stress disorder self reported after deployment and combat exposures: prospective population based US military cohort study BMJ, February 16, 2008; 336(7640): 366 - 371. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||