Using norm-based appeals to increase response rates in evaluation research: A field experiment

A field experiment was conducted to test the effectiveness of norm-based persuasive messages for increasing response rates in online survey research. Participants in an interdisciplinary conference were asked to complete two successive postconference surveys and randomly assigned to one of two groups at each time point. The experimental group received a message asking them to complete an online survey that highlighted a descriptive social norm indicating typical response rates among attendees of prior similar conferences. The control group received a generic request to complete the online survey without any norm-based appeals. As expected, participants receiving a message highlighting the descriptive social norm, when asked to complete an online survey were more likely to comply with the request as compared to those who received the generic message without normative information. Participants receiving a double dose of the descriptive norm treatment at both time points were more likely to complete the second online survey than those who did not receive the descriptive norm intervention at either time point. Implications for future research on evaluation are discussed.

component studies comprise one category of research that entails gathering information about specific evaluation processes with the goal of improving, modifying, or discontinuing certain practices and procedures. This article reports the results of a practice component study designed to test the effects of descriptive social norm-based messages on the response rates associated with online surveys. The study was performed as part of the overall evaluation program for a multi-year initiative to promote interdisciplinary research. The principal aim of the initiative is to encourage interdisciplinary scholarship at the intersection of science, engineering, and medicine. Each year, this program sponsors an interdisciplinary conference focusing on a cutting-edge scientific topic that brings together scholars and practitioners from several different fields.
A fundamental goal of the evaluation program for this initiative is to gauge the effectiveness of specific elements of the annual conference in promoting changes in participants' interdisciplinary attitudes and behaviors, and the trajectory of their research collaborations following the conference. Immediately after the 2009 conference, attendees were asked to complete a brief online questionnaire evaluating various aspects of the conference such as the usefulness of conference sessions in spurring new research ideas and promoting collaboration among scientists from different fields. Three months after the conference, attendees were asked to complete another brief online questionnaire that further evaluated their interdisciplinary attitudes and behaviors (e.g., networking with colleagues from other fields, applying for interdisciplinary grants).
As a part of this ongoing evaluation program, a field experiment was conducted to assess the influence of norm-based persuasive appeals on the response rates associated with the two online surveys. The following sections review earlier research on increasing response rates in online surveys and the use of social norm-based persuasive messages in promoting pro-environmental behavior. To our knowledge, this experiment is the first to test the effectiveness of social norm-based appeals in bolstering participants' response rates to online surveys in the context of program evaluation research.
Low response rates in web-based surveys are an important concern for evaluators because they increase the likelihood of non-response biases in research (Sheehan, 2001). Developing effective strategies for increasing response rates can help reduce nonresponse biases in survey data and improve the quality of research findings. Several factors have been found to influence response rates in web-based surveys including participants' demographic characteristics, survey length and design, ease and intuitiveness of survey questions, perceived importance of the survey topic to respondents, the number of times participants are contacted, and whether personalized or impersonalized, mail-based or e-mail-based follow-up contacts are implemented (cf., Converse, Wolfe, Huang, & Oswald, 2008;Cook, Heath, & Thompson, 2000;Deutskens, Ruyter, Wetzels, & Oosterweld, 2004;Gosling, Vazire, Srivastava, & John, 2004;Kaplowitz, Hadlock, & Levine, 2004;Kaye & Johnson, 1999;Ritter & Sue, 2007;Sax, Gilmartin, & Bryant, 2003;Sheehan & McMillan, 1999).
Empirical evidence concerning ways to augment response rates in online surveys has focused on three strategies: (a) follow-up messages, (b) response incentives, and (c) salience of the issue being evaluated to survey participants.
Follow-up messages. Generally, response rates for e-mail surveys can be increased by administering follow-up reminder messages to participants (Cook et al., 2000;Deutskens et al., 2004;Dillman, 2000;Kaplowitz et al., 2004;Kittleson, 1997). In some instances, however, a large number of reminder messages may decrease participants' responsiveness to an online survey especially among those who receive the largest number of reminder messages-presumably because they become overloaded with too many e-mail messages and may react against receiving multiple reminders to complete the survey (Kittleson, 1997;Solomon, 2001). Dillman (2000) recommends up to four reminder messages per participant but even single follow-ups have been found to increase response rates considerably. Although empirical evidence regarding the optimal timing of reminder messages is scant, Dillman (2000) suggests sending a follow-up message soon after (i.e., within 1 week) the initial request to complete the survey has been sent.
Response incentives. The influence of monetary and other tangible incentives on response rates in online surveys are mixed. Some studies (e.g., Cook et al., 2000) have found that the use of incentives is associated with lower response rates. Others indicate that small prepaid monetary incentives (Dillman, 2000), such as gift vouchers, lotteries (Deutskens et al., 2004;Ritter & Sue, 2007), or grade incentives in the case of class evaluations (Dommeyer, Baum, Hanna, & Chapman, 2004), are linked to higher response rates. Response incentives also have been found to interact with factors such as length of survey. For instance, lottery incentives are more effective in elevating response rates in long surveys compared to short ones. Generally, small lotteries with a higher chance of winning appear to be most effective in raising response rates in web-based surveys (Deutskens et al., 2004).
Salience of the topic being evaluated to survey participants. Salience, or how important the topic being evaluated is to the survey participant, has been found to be positively correlated with response rates in online surveys (Sheehan & McMillan, 1999). Cook et al. (2000) observed that when the results of an online survey are less relevant to the participants, response rates are lower. Ritter and Sue (2007) recommend increasing the salience of the issue being evaluated when participants are invited to complete an online survey. For example, they suggest reminding participants why their responses to the survey would be valuable and pointing out to them, when applicable, that their participation may contribute to future improvements in a particular system or organization.
Social norm-based appeals. Other researchers recommend using social norm-based appeals, especially those that invoke the norms of social cooperation and social responsibility, to increase response rates in online surveys (Ritter & Sue, 2007) and mail-based (Groves, Cialdini, & Couper, 1992;Mowen & Cialdini, 1980). For instance, as an appeal for social cooperation, some suggest presenting the invitation to complete a survey as a request for help (e.g., ''Your participation will really help us out''). However, there are no prior studies that have examined the influence of descriptive social norm-based messages (cf., Cialdini, 2003) on survey response rates.

Descriptive Norms and Proenvironmental Behavior
A large body of research in social psychology has investigated normative strategies to promote attitude and behavior change. Norms are a set of beliefs about approved or disapproved behaviors or about how other people are likely to act in a given situation (Cialdini, Reno, & Kallgren, 1990). Cialdini et al. (1990) have distinguished between descriptive norms and injunctive social norms. Descriptive norms are beliefs about what constitutes ''typical behavior'' within a given situation, such as beliefs about how most people would behave in that situation. Injunctive norms, on the other hand, are beliefs about which behaviors are generally approved or disapproved in a given situation among the members of a particular community or group. Injunctive norms include the importance of abiding by certain values such as helping others in need (social responsibility) or in return for favors that others have bestowed previously on oneself (social reciprocity).
Both injunctive and descriptive social norms are powerful motivators of human behavior. People tend to do what is approved as well as what is popular (Cialdini, 2003). Field experiments in the area of resource conservation and proenvironmental behavior have found that descriptive social norm-based messages are effective in promoting conservation and environmentally protective behavior (e.g., Bator & Cialdini, 2000;Cialdini et al., 1990;Goldstein, Cialdini, & Griskevicius, 2008;Kallgren, Reno, & Cialdini, 2000;Schultz, 1999). For example, a field experiment that investigated the effectiveness of normative messages in increasing towel reuse in a hotel found that a descriptive social norm-based message placed on the towel rack in each room, indicating that the majority of hotel guests reuse their towels when asked to do so, yielded a significantly higher compliance rate (48%) than was obtained using messages based on the injunctive norms of social cooperation (36% compliance) or environmental protection (38% compliance), alone. The injunctive norm conditions omitted descriptive information about what most hotel guests would do in that situation based on the rates of towel reuse observed at the same facility prior to the experimental study (Goldstein et al., 2008).
A theoretical basis for understanding the persuasive power of socially descriptive norms is provided by Festinger's (1954) theory of social comparison processes, which posits that individuals are strongly motivated to compare their attitudes, abilities, and behaviors with those of others they perceive as similar to themselves (e.g., fellow hotel guests as in the study mentioned earlier; persons who belong to the same occupational, ethnic, religious, or cultural groups as oneself). According to social comparison theory, people's motivation to compare themselves with similar others and to align their own attitudes and behaviors with those of others is strongest when objective physical standards of measurement (such as a scale to ascertain one's height or weight) are absent. Objective physical criteria of accuracy usually are not available to help guide individuals' judgments about intangibles such as the desirability of group norms or the appropriateness of their attitudes, abilities, and behaviors in a given situation, so they rely more heavily on subjective social standards of comparison in those instances.

Present Study
The present study extends this earlier social psychological research to test the influence of descriptive social norm-based persuasive messages on response rates achieved in an online survey. Specifically, we hypothesized that participants receiving a message highlighting a descriptive social norm when being invited to complete an initial online survey would be more likely to complete the survey as compared to participants receiving only a generic message that provides no socially descriptive normative information. We further predicted that participants who received a double dose of the descriptive social norm prompt for each of two successive online surveys would be more likely to complete the second survey than those who did not receive the normative prompt at either time point.
Consistent with earlier studies of response rates in online surveys, e-mail reminder messages were sent to those participants in the study who had not yet completed the survey. Although the study included e-mail reminders, the effects of these were not evaluated in this study. Also, information highlighting the importance of the survey topic and the value of individuals' participation was included in prompts administered to participants in the treatment and control groups. However, the effects of this information were not measured. No monetary or other tangible incentives were provided to participants in this study for completing the online surveys.

Participants
Participants were 164 individuals (71.8% male and 28.2% female) who participated in an interdisciplinary conference in 2009. The age range of the participants was 27-72 years. The average age of the sample was 40.7 years (SD ¼ 9.2). The majority of the sample, 81.1%, was employed at academic institutions and the remaining 18.9% were from other sectors (e.g., business/industry, government, social sector/nonprofit, and media). Participants submitted their applications to attend the conference during winter 2009. The applications were reviewed by the conference steering committee. Participants were selected to attend the fall 2009 conference based on their scholarly/professional interests and accomplishments in various fields related to the conference theme.

Procedure
As a part of the evaluation program to assess the effectiveness of annual interdisciplinary conferences, all attendees of the 2009 meeting were asked to complete an online questionnaire immediately following the conference (Time 1) and then 3 months later (Time 2). At Time 1, participants were randomly assigned to one of two groups-treatment or control. The treatment group (n ¼ 82) received an e-mail message providing them with a link to the online conference evaluation questionnaire. The message received by the treatment group highlighted a descriptive social norm-''Most years, over 75% of conference participants complete the survey. Please join your fellow participants in improving the quality of future conferences by filling out the survey . . . .'' The descriptive social norm conveyed by the explicitly stated 75% response rate was derived from survey response rates obtained in prior conference surveys. Response rates observed in prior postconference surveys from 2005 to 2008 are as follows: 75% (2005-2006); 77% (2006-2007); 76% (2007)(2008); and 68%, (2008)(2009).
Participants in the control group (n ¼ 82) received an e-mail request to complete the survey that omitted the norm-based appeal. Further, in an effort to raise the salience of the survey topic among participants in both the treatment and control groups, the e-mail messages sent to both stated that ''We are committed to continuing what works, improving what is less valuable and trying new ways to make the program as beneficial for you as possible.'' The complete text of the messages received by individuals assigned to the treatment and control groups is provided in Appendix A found online at http://aje.sagepub.com/supplemental. At Time 2, participants were sent a similar e-mail message providing them with a link to another online conference evaluation survey. In all 50% of the participants in the treatment group at Time 1 were randomly assigned to the control group; the other 50% received the same prompt they received at Time 1. Similarly, 50% of the participants in the control group at Time 1 were randomly assigned to the treatment group; the other half was assigned to the same group as they were at Time 1. Table 1 summarizes the research design used in this study. This experimental design afforded a withinsubjects replication of participants' assignment to the treatment and control prompts for two successive surveys-one administered immediately after the conference ended and the other administered 3 months later. Accordingly, statistical analyses assessed the degree of change in participants' compliance with the request to complete each survey depending on whether they received a socially descriptive prompt once, twice, or not at all over the course of the field experiment. It was thus possible to evaluate evidence for a dose-response relationship between the treatment (descriptive normbased persuasive message) and outcome measures (compliance with requests to complete two successive surveys).
At both time points, participants were given 2 weeks to complete the online surveys. For both time points, participants who had not completed the survey 1 week after they were sent the initial request to complete the survey received an e-mail reminder. Because the participants were given just 2 weeks to complete the survey, only one follow-up message was sent in this study. The e-mail reminder included the same norm-based or generic message contained in the initial invitation to complete the online survey, depending on the participant's assignment to either the treatment or control group at Times 1 and 2. The text of the e-mail reminder messages received by participants in the treatment and control groups is provided in Appendix A.

Data Analyses
Once the online surveys (at Time Points 1 and 2) were closed, response rates were computed. At Time 1, 65% (107 respondents) of the conference participants responded to the survey. At Time 2, the response rate was 50% (82 respondents). To evaluate the differences in response rates among the treatment and control groups at both time points, we conducted one-tailed z-tests for proportions to ascertain the z-scores. This statistical test was appropriate because: (a) we sought to compare two proportions from two independent groups to determine whether they were significantly different from each other; (b) nominal level data were available; and (c) the study was designed to test an a priori prediction about the direction of the differences between the treatment and control groups.

Results
As hypothesized, participants who received a message at Time 1 highlighting a descriptive social norm when asked to complete an online survey were more likely to do so as compared to those who received a generic message with no descriptive social norm (z ¼ 1.80, p < .05). The same relationship, however, was not found to be significant at Time 2. Tables 2 and 3 below provide details of the number of participants who responded to the online survey in each group and Time Points 1 and 2, respectively.
Additionally, it was found that participants who received a message highlighting a descriptive social norm when invited to complete successive online surveys at both Time 1 and Time 2 were more likely to complete the second online survey, compared to those who received a generic Note: In the above diagram, R ¼ random assignment; X t1 ¼ assigned to treatment group at Time 1; X c1 ¼ assigned to control group at Time 1; X t1t2 ¼ assigned to treatment group at Time 1 and Time 2; X c1t2 ¼ assigned to control group at Time 1 and treatment group at Time 2; X t1c2 ¼ assigned to treatment group at Time 1 and control group at Time 2; X c1c2 ¼ assigned to control group at Time 1 and Time 2; O 1 ¼ observation at Time 1 (immediately following the conference); O 2 ¼ observation at Time 2 (3 months after the conference).
message with no descriptive social norm at Times 1 and Time 2 (z ¼ 1.65, p < .05). Further, participants who received a descriptive social norm-based message at both Time 1 and Time 2 were found to be marginally significantly more likely to complete the second online survey as compared to those who received a descriptive social norm-based message at Time 1 but a generic message at Time 2 (z ¼ 1.33; p < .10). No significant differences in response rates were found between participants in the mixed-prompt groups (i.e., where participants' experimental conditions were inconsistent across the two time points). Table 4 specifies the number of respondents for each combination of the experimental and control conditions across both time points.

Discussion
A field experimental study was conducted to evaluate the effectiveness of descriptive norm-based persuasive messages as a means of increasing response rates in online surveys. The findings provide evidence for the effectiveness of descriptive norm-based appeals for increasing rates of response to online surveys. A dose-response relationship between the treatment and the outcome measure (compliance with requests to complete the conference surveys) was observed. Specifically, participants who received a double dose of the descriptive norm treatment at both time points were more likely to complete the conference survey at Time 2, especially in comparison with those participants who did not receive the descriptive norm intervention at either time point. The mixed-prompt groups, in which participants' treatment and control statuses were inconsistent between Times 1 and 2, were not found to be significantly different from each other. To our knowledge, this is the first study to investigate the influence of descriptive social normbased messages on response rates of web-based surveys. The findings of this study complement X c1c2 compared to X t1t2 : z ¼ 1.65; p < .05; X t1c2 compared to X t1t2 : z ¼ 1.33; p < .10; X c1t2 compared to X t1c2 : NS; X c1t2 compared to X c1c2 : NS; X t1c2 compared to X c1c2 : NS. and extend existing social psychological research examining the persuasive power of descriptive norm-based appeals on proenvironmental attitudes and behavior (e.g., Bator & Cialdini, 2000;Goldstein et al., 2008;Kallgren et al., 2000). Further, the reported findings suggest the efficacy of descriptive social norm-based messages as a strategy for enhancing online survey response rates in future evaluation studies, thereby extending research in this area (e.g., Dillman, 2000Dillman, , 2007. At the same time, the findings are qualified by certain methodological limitations of this research. First, although our sample of 164 participants was sufficient to afford a field experimental assessment of changes in participants' survey response rates based on their cumulative exposure to socially descriptive normative prompts, it was smaller than the samples reported in some earlier studies of survey response rates (cf., Converse et al., 2008;Kiernan et al., 2005). Although the present study incorporated a repeated-measures within-subjects replication design for evaluating dose-response relationships between the treatment and criterion measures, it will be important to replicate the results of this study with larger and more diverse samples to further characterize the external validity of the reported findings. For instance, because participants were drawn largely from academic institutions and scientific professional organizations, replication studies are needed to assess the generalizability of these findings to nonacademic and nonprofessional populations and settings. Second, it remains to be investigated whether the findings of this study are generalizable to different survey formats. Accordingly, future studies should evaluate the effects of descriptive social norm-based messages on response rates in paper-based surveys as well as in different populations and evaluative contexts.
Despite these limitations, the reported study has some notable strengths and suggests potentially valuable directions for future program evaluation survey studies. First, the findings reveal a straightforward and low-cost strategy for increasing response rates on evaluative questionnaires. As such, the results from this study may prove useful to professionals and researchers who use Internet-based surveys.
Second, to our knowledge this is the first study of its kind to evaluate the influence of descriptive social norm-based messages on online survey response rates. The findings contribute to literature on practice component studies within evaluation research as well as to the social psychology literature on norm-based persuasive messages by demonstrating the efficacy of descriptive norm-based appeals in improving rates of response to online surveys.
Third, the present study provides a foundation for future research aimed at comparing the separate and joint effects of persuasive messages to encourage survey participation that invoke different kinds of social norms. This study focused exclusively on the effectiveness of descriptive social norms in prompting higher rates of response to two successive online surveys. Future investigations could be designed to compare the separate, additive, and/or interactive effects of persuasive appeals to complete surveys that invoke injunctive as well as descriptive social norms. Thus, prompts to complete a survey as well as the reminder messages sent to nonrespondents might incorporate messages that raise the salience of descriptive social norms and/or the norms of social responsibility and social reciprocity. It is plausible that multidimensional persuasive messages incorporating and aligning all three normative appeals (i.e., based on social description as well as the values of social responsibility and reciprocity) might prove to be more powerful in increasing survey response rates than those incorporating singular or unidimensional norm-based requests for participation. As well, the joint influences of norm-based appeals and the provision of various kinds of response incentives on rates of participation in online and printed surveys remain to be examined in future research.
In sum, the reported study afforded a cost-effective and theoretically grounded assessment of evaluation research procedures, especially the design and efficacy of descriptive social norm-based persuasive appeals sent to prospective survey respondents. The results of this research contribute to the literature on evaluation practice and may serve to enhance future program evaluation strategies.