American Journal of Epidemiology Advance Access originally published online on October 13, 2006
American Journal of Epidemiology 2006 164(12):1145-1149; doi:10.1093/aje/kwj348
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Invited Commentary: The Art of Making Questionnaires Better
From the Department of Epidemiology, University of California Los Angeles School of Public Health, Los Angeles, CA
Correspondence to Professor Jørn Olsen, Department of Epidemiology, UCLA School of Public Health, Box 951772, Los Angeles, CA 90095-1772 (e-mail: jo{at}ucla.edu).
Received for publication August 8, 2006. Accepted for publication September 6, 2006.
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONHISTORY OF QUESTIONNAIRES AND...QUESTIONNAIRE USE IN CURRENT...MAKING QUESTIONNAIRES BETTERReferences |
|---|
A paper by Schilling et al. (Am J Epidemiol 2006;164:1141–4) addresses a crucial issue for epidemiologists: limited peer access to questionnaires. This limited access hampers the ability to evaluate and improve the questions used by investigators and, in turn, the quality of some of the self-reported data. The authors of this commentary analyzed recent publications in core epidemiology journals, finding that self-reported data were used in 64% of articles, but key questions were seldom printed in the article (9%), and open access to complete questionnaires (16%) was rarely provided. Only 47% of articles even discussed validation; of these, only 67% actually validated questions used in the study. The authors join Schilling et al. in making recommendations to improve questionnaire access and collaboration. A first step, proposed before, involves investigators posting their questionnaires on a website concurrently with publication of their article. Journal editors should require online access to full questionnaires for published articles and inclusion of key questions within the article when possible. Funding agencies should take the lead in increasing access and collaboration by developing a searchable database.
data collection; epidemiologic methods; questionnaires; research design
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONHISTORY OF QUESTIONNAIRES AND...QUESTIONNAIRE USE IN CURRENT...MAKING QUESTIONNAIRES BETTERReferences |
|---|
Schilling et al.'s paper in this issue of the Journal (1) is most welcome, and the problems they address are unfortunately unsolved and important. Questionnaires generate data for research, and we should be as concerned about the quality of these data as laboratory investigators are about the quality of the data they generate. That is unfortunately not the case, despite a much longer history of investigation using questionnaires than of laboratory research.
|
HISTORY OF QUESTIONNAIRES AND QUESTIONNAIRE DESIGN |
|---|
|
TOPABSTRACTINTRODUCTIONHISTORY OF QUESTIONNAIRES AND...QUESTIONNAIRE USE IN CURRENT...MAKING QUESTIONNAIRES BETTERReferences |
|---|
Modern epidemiology's use of questionnaires is rooted in many traditions. The use of surveys is mentioned in the Bible, with a census described in the Book of Numbers after Moses ascended Mount Sinai and can be traced to the Egyptian and Roman empires, which used periodic censuses (2). Survey research aimed at systematic data gathering, health, and social reform likely began with John Howard's 1775 investigations into prison conditions and prisoner complaints in England and Wales (3, 4). Research conducted by life insurance companies linking self-reported medical history, such as previous asthma, and lifestyle factors to mortality in an effort to refine underwriting began in the early 20th century, with publications such as the Actuarial Society of America's Specialized Mortality Investigation (5). Early evidence of questionnaire use also exists in the investigation of food-borne illness. In 1902, H. Timbrell Bullstrode, a British inspector, used a rudimentary food questionnaire to identify oysters as the transmitter of typhoid fever at a banquet (6).
Epidemiologists should know that the way in which questions are formulated and asked may have a profound effect on whether and how they are answered (7). Significant variations in data collected may arise from differing modes of questionnaire administration (8–13) and design of individual questions (14–17). Even factual information may depend heavily on how the question is formulated, and layout choices such as font size or colors can affect questionnaire responses (18, 19). Substantial literature exists to guide epidemiologists and social science researchers in optimizing questionnaire design to develop effective questionnaires (20–29), beginning more than 50 years ago with Stanley Payne's classic The Art of Asking Questions (30), and some high-quality questionnaires have been developed. However, these questionnaires cover only a fraction of the data we need in most studies. Many investigators still build their questionnaires from scratch or use existing questions without access to the experiences from past use.
|
QUESTIONNAIRE USE IN CURRENT EPIDEMIOLOGIC RESEARCH |
|---|
|
TOPABSTRACTINTRODUCTIONHISTORY OF QUESTIONNAIRES AND...QUESTIONNAIRE USE IN CURRENT...MAKING QUESTIONNAIRES BETTERReferences |
|---|
We conducted an exercise similar to Schilling et al.'s (1), but we used epidemiologic journals as our source of data. In the following paragraphs, we present our main findings because they expand on Schilling et al.'s paper and have not been published before. We reviewed all original articles, not from clinical journals but from Epidemiology, the European Journal of Epidemiology, the International Journal of Epidemiology (published from July 2005 to October 2005), and the American Journal of Epidemiology (published from July 2005 to August 2005). These core epidemiology journals were selected because they should be more concerned with epidemiologic rigor than clinical journals are. One hundred and eleven articles were reviewed to analyze whether an interview or questionnaire was used, characteristics of the questionnaire, questionnaire availability, and validation considerations.
Seventy-one (64 percent) of the articles used a questionnaire, and characteristics of these articles are shown in table 1. Self-report was used more frequently for exposure (61 percent) than for outcome (34 percent) and was used in nearly all articles for covariate assessment (97 percent). All three major observational study design types—cohort, case-control, and cross-sectional—used questionnaires frequently.
|
Interviewing was most common (72 percent), but self-administered questionnaires (31 percent) were also frequently used (table 2). Our analysis indicated that computer-administered questionnaires do not appear to have caught on yet to a large degree with epidemiologists. Unfortunately, 13 percent of the articles did not indicate the questionnaire type within the article, demonstrating that questionnaire issues are frequently not of central methodological concern for investigators, editors, or peer reviewers.
|
Only 28 percent of investigators acknowledged using questions from previous instruments they identified by name and with a reference (table 3). In addition, only 21 percent indicated that they had piloted or pretested the questionnaire before conducting the study.
|
Many investigators (n = 13, or 18 percent) used the word "standard" or "standardized" when describing their questionnaire or interview process. Because this word does not have a consistent definition and was seldom or ever elaborated on within the article, it seems to have evolved as a code to placate reviewers and assure without proof the scientific validity of the investigation. In the future, journal editors should challenge authors using these words to more fully explain in the article their meaning and relevance to the study.
Developing standard questions (and by this we mean questions that have been validated against a "gold standard" measure, reproduced, and used universally throughout the field, similar to laboratory research techniques) for prominent lifestyle risk factors is clearly important. For example, many of the studies evaluated used smoking (62 percent) and alcohol consumption (32 percent) as exposures, outcomes, or covariates, likely providing different questions and response options.
The ability to scrutinize the questions is crucial for the epidemiologic community, as is the ability to capitalize on each other's experiences and learn from other investigators' mistakes. As shown in table 4, only six articles (8 percent) printed any complete questions within the article, and only 20 (28 percent) printed questions within a referenced article. Of articles in which the key exposure was determined by self-report, only four (9 percent) published any complete exposure question. A larger percentage (21 percent) printed a question for key outcome when it was determined by self-report. No questionnaires were available via journal websites. In only 15 percent (11 articles) was a complete study questionnaire available in a published article or monograph or on the study website, typically for large, ongoing studies. For the nine articles with questionnaires available online, none listed the website address as a reference within the article.
|
We also examined whether investigators discussed questionnaire validation in their article or references, and 46 percent (33) did. For these, the authors validated some aspect of the questionnaire themselves in 12 articles (36 percent), and six (18 percent) of these validation studies were published. The authors referenced validation studies published previously by other authors in 88 percent of articles. Only eight articles included both validation by the author and previous validation. Furthermore, in only 22 studies (67 percent) were the specific questions used by the investigators validated; in the remainder, the use of self-report in general to assess a specific exposure, outcome, or covariate was evaluated. Validation is clearly not a central concern when epidemiologists use questionnaires.
Although questionnaires are frequently used in epidemiology and it is well known that the design of the questionnaire is crucially important to study results, questionnaire data are not treated with the care they should be. Their use is not well documented, and there seems to be limited interest in validation, standardization, and using past experience to improve new questionnaires. We find this situation unacceptable and quite depressing.
|
MAKING QUESTIONNAIRES BETTER |
|---|
|
TOPABSTRACTINTRODUCTIONHISTORY OF QUESTIONNAIRES AND...QUESTIONNAIRE USE IN CURRENT...MAKING QUESTIONNAIRES BETTERReferences |
|---|
Calls for improving self-reported data collection in epidemiologic research have appeared sporadically in the literature for more than 25 years. Leon Gordis' observation in a little-noticed 1979 article, "Assuring the Quality of Questionnaire Data in Epidemiologic Research," is still remarkably pertinent:
A major challenge to epidemiologic research today is to assure the quality of the "raw data" which are generated for analysis in our studies. A serious potential hazard confronting us is that, as epidemiologists, we may become so enamored of increasingly sophisticated statistical techniques and data processing capabilities that we may pay inadequate attention to the quality of the data obtained in our investigations—data which, in fact, become the raw material for these statistical and data processing methods. For no scientific discipline can be any better than the quality of its raw data. Improvements in study design or in analytic techniques cannot compensate for data of questionable quality generated by epidemiologic investigations.Unfortunately, we have often paid too little attention to the quality of epidemiologic data, and particularly those obtained through interviews and questionnaires (31, p. 21).
In 1999, the International Epidemiological Association expressed concern over the lack of focus on questionnaires and proposed establishment of a committee or clearinghouse by a group such as the World Health Organization to begin developing standardized instruments (32). Allen Wilcox responded in 2000, agreeing that questionnaire design was neglected but suggesting a different solution. He proposed that, as a first step, questionnaires be made available online on the author's website so that peers could easily access and critique them (33). Unfortunately, until Schilling et al.'s article (1), discussion of these suggestions had not continued in the literature.
A clear first step in improving questionnaire access is adopting Wilcox's (33) and Schilling et al.'s (1) suggestion and encouraging or requiring investigators to post their questionnaires online on their own or the journal's website concurrent with publication. Leading scientific journals such as Science and Nature have firm policies requiring deposition of raw biologic data into public repositories at the time of publication. Editors of core epidemiology journals should have the same requirements for questionnaires. In fact, questionnaires are likely substantially less problematic to make available. Key questions from the questionnaires should be published in the paper if possible. Recommendations are presented in table 5.
|
We also agree with Schilling et al. (1) that a searchable questionnaire database needs to be established. Funding agencies and journal editors should require that questionnaires used in large studies be available and easily identified, even at the question or item level. In the database, a question may be linked to previous studies that have used it, to published validation studies, to translations if they have been done, and to other relevant factors.
The major funding agencies should take the lead in developing comprehensive, next-generation solutions for questionnaire access, including searchable databases. These organizations have the necessary resources, continuity, and stability to accomplish this task, and they have a major interest in improved data collection.
It may also be wise to think about devoting an online journal to questionnaire development, and it could be justified if forces are combined for all scientific disciplines using questionnaires, such as sociology, demography, medicine, and psychology. This journal could serve as a central repository for the questionnaires from articles published in other journals and could feature methodological research and commentary papers such as this one. This journal could also serve as a forum for the exchange of ideas and experiences using questionnaires in the best spirit of the Internet.
We epidemiologists ask questions very frequently as part of our research. The process demands significant time from respondents, and we have an ethical obligation to perform our work as well as possible. The results we report and on which health and policy decisions are made often rely heavily or totally on the answers respondents give to these questions. Therefore, we need to take great care in designing, standardizing, and critiquing these questions as a central part of the scientific process. Schilling et al. (1) are to be thanked for reminding us of this important part of epidemiologic studies.
|
ACKNOWLEDGMENTS |
|---|
More details on the findings reported in this commentary are available by contacting Monica Encinias at monniee{at}ucla.edu.
Conflict of interest: none declared.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONHISTORY OF QUESTIONNAIRES AND...QUESTIONNAIRE USE IN CURRENT...MAKING QUESTIONNAIRES BETTERReferences |
|---|
- Schilling LM, Kozak K, Lundahl K, et al. (2006) Inaccessible novel questionnaires in published medical research: hidden methods, hidden costs. Am J Epidemiol 164:1141–4.
[Abstract/Free Full Text] - Warwick DP and Lininger CA. (1975) The sample survey: theory and practice. (McGraw-Hill, New York, NY).
- Howard J. (1977) The state of the prisons. Bicentennial ed(Abingdon, Nashville, TN).
- Young PV and Schmid CF. (1939) Scientific social surveys and research: an introduction to the background, content, methods, and analysis of social studies. (Prentice-Hall, New York, NY).
- Singer RB. (1985) Examples of early mortality follow-up studies. Natl Cancer Inst Monogr 67:15–21.[Medline]
- Morabia A and Hardy A. (2005) The pioneering use of a questionnaire to investigate a food borne disease outbreak in early 20th century Britain. J Epidemiol Community Health 59:94–9.
[Abstract/Free Full Text] - Bowling A. (2005) Mode of questionnaire administration can have serious effects on data quality. J Public Health (Oxf) 27:281–91.
- Tomlin ME, Pinney S, Buncher CR, et al. (1998) The effect of the mode of questionnaire administration on workers' responses to cigarette smoking questions. (Abstract). Am J Epidemiol 147:S85.
- Vuillemin A, Oppert JM, Guillemin F, et al. (2000) Self-administered questionnaire compared with interview to assess past-year physical activity. Med Sci Sports Exerc 32:1119–24.
- Frost NA, Sparrow JM, Hopper CD, et al. (2001) Reliability of the VCM1 Questionnaire when administered by post and by telephone. Ophthalmic Epidemiol 8:1–11.[CrossRef][Medline]
- Kaplan CP, Hilton JF, Park-Tanjasiri S, et al. (2001) The effect of data collection mode on smoking attitudes and behavior in young African American and Latina women. Face-to-face interview versus self-administered questionnaires. Eval Rev 25:454–73.
[Abstract/Free Full Text] - Midanik LT and Greenfield TK. (2003) Telephone versus in-person interviews for alcohol use: results of the 2000 National Alcohol Survey. Drug Alcohol Depend 72:209–14.[CrossRef][Web of Science][Medline]
- Smeeth L, Fletcher AE, Stirling S, et al. (2001) Randomised comparison of three methods of administering a screening questionnaire to elderly people: findings from the MRC trial of the assessment and management of older people in the community. BMJ 323:1403–7.
[Abstract/Free Full Text] - Schwarz N and Hippler HJ. (1995) Subsequent questions may influence answers to preceding questions in mail surveys. Public Opin Q 59:93–7.
[Abstract/Free Full Text] - Couper MP, Tourangeau R, Conrad FG, et al. (2004) What they see is what we get: response options for web surveys. Soc Sci Computer Rev 22:111–27.[Abstract]
- Knauper B and Schwarz N. (2004) Why your research may be out of order. Psychologist 17:28–31.
- Smyth JD, Dillman DA, Christian LM, et al. (2006) Comparing check-all and forced-choice question formats in Web surveys. Public Opin Q 70:66–77.
[Abstract/Free Full Text] - Christian LM and Dillman DA. (2004) The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opin Q 68:57–80.
[Abstract/Free Full Text] - Proceedings of the American Statistical Association, Survey Research Methods Section Smith TW. (1995) Little things matter: a sampler of how differences in questionnaire format can affect survey responses. , Alexandria, VA1046–51 (http://www.amstat.org/sections/SRMS/Proceedings/papers/1995_182.pdf).
- Boynton PM. (2004) Administering, analysing, and reporting your questionnaire. BMJ 328:1372–5.
[Free Full Text] - Questions about questions: inquiries into the cognitive bases of surveys. (1992) (Russell Sage FoundationIn Tanur JM (Ed.). , New York, NY).
- Bradburn NM and Sudman S. (1979) Improving interview method and questionnaire design. 1st ed (Jossey-Bass, San Francisco, CA) (Jossey-Bass social and behavioral science series).
- Measurement errors in surveys. (1991) (WileyIn Biemer PP, Groves RM, Lyber LE (Eds.), et al. , New York, NY).
- Converse JM and Presser S. (1986) Survey questions: handcrafting the standardized questionnaire. (Sage, Beverly Hills, CA) (Sage university papers series. Quantitative applications in the social sciences, no. 07-063).
- Fink A, Bourque LB, Fielder EP, et al. (2003) The survey kit. 2nd ed (Sage Publications, Thousand Oaks, CA).
- Aday LA. (1996) Designing and conducting health surveys: a comprehensive guide. 2nd ed (Jossey-Bass Publishers, San Francisco, CA).
- Groves RM. (1989) Survey errors and survey costs. (Wiley, New York, NY).
- Boynton PM and Greenhalgh T. (2004) Selecting, designing, and developing your questionnaire. BMJ 328:1312–15.
[Free Full Text] - Boynton PM, Wood GW, Greenhalgh T. (2004) Reaching beyond the white middle classes. BMJ 328:1433–6.
[Free Full Text] - Payne SL. (1951) The art of asking questions. (Princeton University Press, Princeton, NJ) (Studies in public opinion, 3).
- Gordis L. (1979) Assuring the quality of questionnaire data in epidemiologic research. Am J Epidemiol 109:21–4.
[Free Full Text] - Olsen J. (1998) Epidemiology deserves better questionnaires. IEA European Questionnaire Group. International Epidemiological Association. Int J Epidemiol 27:935.
- Wilcox AJ. (1999) The quest for better questionnaires. Am J Epidemiol 150:1261–2.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
Related articles in Am. J. Epidemiol.:
- Inaccessible Novel Questionnaires in Published Medical Research: Hidden Methods, Hidden Costs
- Lisa M. Schilling, Katarzyna Kozak, Kristy Lundahl, and Robert P. Dellavalle
Am. J. Epidemiol. 2006 164: 1141-1144.[Abstract] [FREE Full Text]
This article has been cited by other articles:
|
|
 |
|
  K. E.A. Burns MD MSc, M. Duffett BScPharm, M. E. Kho PT MSc, M. O. Meade MD MSc, N. K.J. Adhikari MDCM MSc, T. Sinuff MD PhD, D. J. Cook MD MSc, and for the ACCADEMY Group A guide for the design and conduct of self-administered surveys of clinicians Can. Med. Assoc. J., July 29, 2008; 179(3): 245 - 252. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||