The following article is posted here courtesy of the author, Andy Peytchev, RTI International:

The Allure of Relatively Inexpensive Web Surveys

Andy Peytchev

“…there are no national statistics unless millions of people tick boxes, fill in forms, respond to queries, and otherwise cooperate when asked to give information about themselves and their households.”

- Kenneth Prewitt, Professor of Public Affairs at Columbia University and former Director of the U.S. Census Bureau

There are different ways to devise and implement web surveys, but the attraction of this inexpensive mode of data collection often leads to poorly designed surveys that do more harm than good to the survey field. Social science has benefited substantially, if not greatly, from the implementation of web surveys. Inarguably, one of the factors driving the use of web surveys is the ability to deploy fast and inexpensive data collection. Indeed, software are freely available at little or no cost to develop an instrument, collect responses, and tabulate results. The ability to create inexpensive surveys and easily send invitations to a large number of potential respondents, however, has contributed to a great volume of web surveys, many of which could not be classified as “scientific” – such as not making inferences to a target population and not using rigorous methods to curb different sources of survey error.

The proliferation of web surveys shares similarities with the introduction of other modes, although the adoption has been at a much faster pace. When introduced, telephone surveys enjoyed high response rates, at a time when receiving a long-distance call was not common. Today, it is usual to receive a call from an interviewer at dinner time – whether for a survey or a marketing offer. This leads to another issue, in addition to the increased number of requests – the difficulty for individuals to distinguish between the voluminous marketing requests and the relatively few requests for scientific surveys. Indeed, in order to gain cooperation, some surveys have to begin with a statement that the interviewer is not selling anything. While such development has happened over the course of decades for telephone surveys, the same seems to be happening in years for web surveys.

Scientific web surveys need to be able to differentiate themselves from other survey requests in order to achieve the necessary degree of cooperation. While poorly designed surveys can impact the quality of the estimates they produce, a greater threat arises when they affect the likelihood of participation in well-designed surveys, or the information respondents are willing to provide. There is evidence that multiple requests for web survey completions can lead to lower cooperation to subsequent requests in non-panel populations (Peytchev, 2007). This has been a major discussion topic at conferences on web panels, the latest being organized by CASRO and scheduled to be held in February, 2009. However, even among scientific surveys, more effort needs to be expended on careful design – the effort in constructing the survey perceived by the respondents can affect responses (Yan, 2005).

In fact, poorly designed web surveys can lead to greater unit nonresponse, breakoff, item nonresponse, and deteriorated measurement properties of survey variables. A challenge to the field is defining poor and good survey designs. Systematic research is being conducted on visual design in web surveys (e.g., Tourangeau, Couper and Conrad, 2007). Much can also be borrowed from research on other modes of data collection – specifically to ascertain the legitimacy of the survey request, curb confidentiality concerns, etc.

The relatively low variable cost of web surveys can be a detriment as everyone can design and launch a web survey, but it also aids in the ability to identify sources of error and improve designs through experimentation. Web surveys provide for relatively easy implementation of experiments, but they also provide the opportunity to collect a vast array of paradata – data on how the survey data collection was performed. Among others, these include timing, navigation, changing responses, mouse movements. Tools have been developed to facilitate the collection of paradata in web surveys – some readily available in the survey software, others developed by researchers (e.g., https://perswww.kuleuven.be/~u0034437/public/csp.htm).

The allure of inexpensive surveys that fail to incorporate extant research and make compromises in the design and implementation is to some degree inevitable. Some legislative changes are helping ensure attention to some aspects of web survey design, at least for federally funded research. Section 508 of the Rehabilitation Act aims to ensure that, as in the context of web surveys, individuals with disabilities need to be also able to access the instrument and provide information. There are many usability issues to be considered (see Matulewicz and Coburn, 2008) and many low cost survey software packages may not be able to render surveys for the 8% of the population with some disability - half of whom have vision related disabilities.

Both past and present directors of the U.S. Census Bureau have underscored the need to explore alternative sources of information, such as administrative records and other survey data. This may be particularly pertinent to web survey data collection. For example, an e-mail or IP address can be linked to a variety of data. The utility and legal issues related to the use of such data remain to be explored. While their use can further complicate the design of a web survey, likely increasing cost, it poses vast potential to reduce the burden placed on respondents – possibly improving cooperation and the responses provided in a particular survey.

There could not be web surveys without the cooperation of individuals and yet their cooperation is declining. A reasonable hope is that well-designed web surveys accessible for as many of the sample members would not only encourage participation in a particular study and improve the data being collected, but will also improve the image of scientific web surveys .

References

Matulewicz, H. and J. Coburn (2008). "Universal Design for Web Surveys: Practical Guidelines." Survey Practice(November).

Peytchev, A. A. (2007). Participation Decisions and Measurement Error in Web Surveys. Doctoral dissertation, University of Michigan.

Tourangeau, R., M. P. Couper and F. Conrad (2007). "Color, Labels, and Interpretive Heuristics for Response Scales." Public Opinion Quarterly 71(1): 91 - 112.

Yan, T. (2005). Gricean Effects in Self-Administered Surveys. Doctoral dissertation, University of Maryland.