A General Guide to Administering Surveys

1. Consider whether you really need to conduct a survey

You may find that administering a survey is not appropriate for your study; for example, depending on your research question, or quality assurance question, it may make more sense to use interviews, focus groups or focused discussions, such as listening sessions.

If you determine that survey data are the appropriate data source for your study, it is possible that the data have already been collected. UBC engages regularly in survey research with students, alumni, faculty, and staff. The Survey page on the PAIR website provides information about the surveys that we regularly administer. For more information, please contact PAIR.

2. Establish an appropriate timeline

While setting up a web survey and distributing it via email can be a quick process, the preparations one needs to do in advance of survey administration can be fairly time consuming. This process can take several weeks or even months and should involve careful consideration of your target distribution date and the amount of time it will take to accomplish all that needs to be done prior to this date. It is important to give yourself ample time to plan, develop, and test your survey, as well as go through the BREB application if required.

3. Determine the timing of your survey

When choosing a target distribution date, it is also important to consider the other things that are may be happening at the time; for example, Winter term 2 (January/February) is a busy time due to the administration of the Undergraduate Experience Survey, while March is particularly busy for student-administered surveys.

Avoiding conflicts with other survey activity

You should also consider times when it may be difficult to attract responses, such as holidays, and midterm- and final-exam periods. Please consult the survey calendar to determine whether there are any potential conflicts. All surveys that have been approved by the Survey Governance Committee will be added to the survey calendar.

4. Identify your target population

Your choice of the target population of your survey will largely be determined by your research question; for example, if your research question addresses the experience of first-year students, the target population would probably be limited to first-year students (or possibly faculty and staff who work with first-year students). Once you have identified the appropriate target population, another thing to consider is whether it makes more sense to survey the entire population (a census) or survey a portion of your target population (a sample). PAIR can assist you with determining an appropriate sample for your survey; please contact us at pair.team@ubc.ca.

5. Determine the mode of survey

There are advantages to administering a survey online. Web surveys are popular because it is easy to develop them, reach your target population, and download and analyze your data. The UBC Office of Research Ethics provides guidance on using online surveys. Other options include paper surveys, which yield high response rates when you have a captive audience, or interviews/focus groups, which can generate a deeper, richer examination of the subject matter.

6. Design the survey instrument

There is a large amount of literature on how to design a survey. The following are tips and resources you may want to consider using as you design your survey.

Here are a few general survey design tips:

  • Keep in mind that your survey items should align with your research or quality assurance question.
  • Avoid leading questions. Keep your questions neutral and make sure they don’t lead respondents to a particular answer.
  • Avoid double‐barreled questions with categorical response options, such as Likert Scales. Example: “How satisfied were you with the sound and visuals of the presentation?” It is unclear whether you want them to evaluate the sound or the visuals. Split it into two questions.
  • Be mindful of the length of your survey. Shorter surveys tend to generate stronger response rates.
  • Have someone review your survey and read through the questions to identify any potentially confusing terminology or inappropriate response options that might lead to unclear feedback from respondents. What makes sense to you may not make sense to the audience.
  • Always test your survey. Fill it out yourself and ask others to fill it out and provide feedback on the flow of the survey questions, ease of responding to items, and length of time it takes to participate in the survey.

7. IMPORTANT: Include informed consent and evaluate whether you need BREB review

Some surveys will need to be reviewed by the UBC Behavioural Research Ethics Board (BREB). The BREB, which oversees behavioural or social sciences/humanities research, or research that may involve the study of patients or health care providers. For more information about BREB review and whether it applies to your survey, please visit the BREB website.

Whether your survey requires BREB review or not, ALL surveys at UBC are expected to include informed consent language. Prior to the start of your survey (e.g., in a signed consent form, in the body of an email invitation to participate, at the start of the survey, all of the above), you will need to inform your potential participants about the purpose of the survey and how their data will be used. Examples of the information you provide include how the data will be stored, who will have access, whether you can provide anonymity or confidentiality, and how and for what purpose the data will be reported. If you have any questions about whether you are providing an acceptable level of informed consent, please contact PAIR or the current BREB representative.

8. Consider offering incentives to respondents

If you are concerned about your response rate, you may want to consider offering incentives to your respondents. A common approach is to offer a prize draw for gift cards for those who participate in the survey; however, offering incentives will not guarantee a high response rate for your survey. Research has shown that while incentives have a positive effect on response rates, this effect is quite small (Porter & Whitcomb, 2003).

If you do offer an incentive, be sure to follow rules (see section 6.5 of the BREB Guidance Notes). To claim their prize, entrants must complete a mathematical skill-testing question. Also keep in mind that compensating individuals for participating in your study, through either payment or a raffle, may have tax consequences. For additional information, consult PAIR.

Something to keep in mind if you are considering offering incentives is that you will need to gather the contact information of survey respondents. If you intend to make your survey anonymous, you should collect the contact information in a separate survey. Qualtrics has a straightforward way of accomplishing this:

  • Set up a separate survey in Qualtrics to collect the contact information of those who wish to be entered into your prize draw. A title and a text entry form with name and email address is probably all that you need.
  • Select “Launch Survey” and then “Activate your survey to collect responses” and copy the anonymous link.
  • Go to the edit screen of your original survey and select “Survey Options.” In the “Survey Termination” section of the “Survey Options” menu, there is an option to “Redirect to a URL.” When you select this option, paste in the anonymous URL from the new survey you have set up.
  • Now, once your respondents complete and submit the survey, they will be automatically directed to the form to collect their personal information for the prize draw.

9. Administer survey and collect data

Crafting an email inviting your target population to respond

When distributing your survey via email, consider how the recipient will react to receiving it. To encourage the recipient to open the email and click on the link to respond to your survey, it is important to use an informative “Subject” title, and to use careful language in the body of the email that explains the value of responding to the survey and minimizes the burden of the respondent. Information that demonstrates the value of responding include the importance of the survey, how the data will be used, and incentives the respondent might receive as a reward for participating. It is also important to include how long the survey should take to complete and an end-date of when your survey will close.  Minimizing the burden to the respondent includes providing an accurate estimate of the time required to complete the survey, clearly explaining the purpose of the survey.

Distributing your invitation email

In Qualtrics, there are two approaches you can use to administer your survey and collect data. The first approach is to use an anonymous link and distribute the survey via email to your population. The second is to distribute the survey through the Qualtrics system by setting up a Contact List. Each of these approaches has its own advantages and disadvantages.

The advantage to the anonymous link approach is that it is straightforward and easy to use, and you can guarantee anonymity to your survey population. All you have to do is activate the survey, copy the anonymous link, paste it into an email, and send the email. A drawback to this approach is that it is impossible to identify your respondents and, when sending reminders, you are unable to remove any of the individuals in your population who have already responded to the survey.

Using Qualtrics to distribute your survey is a bit more complicated, but there are some real advantages. Using this system, you pre-load the information about your population into a Contact List in Qualtrics. (Qualtrics previously referred to this as a “panel.”) The system can distribute a custom email to each person in your population. By including a name and major in your contact list, you can address each individual by name and include their major in the text of the email. These customized emails provide a personal touch that may encourage the recipient to respond to the survey. Another advantage to this approach is that it might allow you to shorten the length of your survey. If you have relevant background information about your population (e.g., gender identity, race/ethnicity, class year, field of study) you can link the survey response to these data rather than including questions about these characteristics on the survey.

Finally, when sending reminders from the Qualtrics system, it automatically removes individuals who have already responded from the reminder distribution, so they are not contacted again unnecessarily. A disadvantage of this approach is that it requires you to have, at a minimum, the email addresses for all of the individuals in your population. Without this information, your best option is likely to distribute an anonymous link via the UBC listservs or UBC newsletters.

Monitoring your survey responses

Once your survey is distributed, be sure to monitor the responses as they come in.

The main Qualtrics screen (“Projects”) shows a list of your survey projects. Click on your survey to view details about the deployment or access response data. The “Distributions” tab provides an overview of incoming survey responses and completion rate. If you used Qualtrics to distribute your survey, you can click “Emails” for additional details about the survey deployment (e.g. invitations, reminders). The “Data & Analysis” tab provides detail about both the survey responses currently in progress and the recorded survey data (completed responses). The “Reports” tab shows aggregate survey data and provides tools to visualize your aggregate data. You will want to use either “Distributions” or “Data & Analysis” tab in order to determine the number of people who have completed your survey.

Most responses are collected within the first 48 hours of distribution. After your initial distribution, you may want to send a reminder email to your population. Once your responses stop coming in, you should close your survey. This can be accomplished by selecting “Close” in your survey menu on the right side of the screen when you are on the “Projects” screen.

What is a good response rate?

The extent to which a response rate is good depends upon a variety of factors, including the size of the population and whether your respondents are representative of the population you have surveyed. For more complex surveys and those based on samples, there are statistical procedures and response-rate calculators that you can use to make judgments about the number of responses you may need. For simpler surveys, the key question you will want to ask is whether the results are believable and representative of your target population, based on the number of responses you have. For assistance on determining an adequate response rate, contact PAIR.

10. Analyze and report results

For many users, the reports that Qualtrics generates may be adequate for descriptive analyses. Reports can be viewed by selecting the View Reports button on the Projects screen or, when you are working on your project, by selecting Reports at the top of the screen. Each survey has a report, called Default Report that displays the results of the survey in real time. It is easy to manipulate these results and export the report in a variety of file formats (i.e., Word, Excel, PowerPoint, PDF). Visit the Qualtrics reporting information page to learn more.

If you need a more robust analysis, it is easy to export your raw data to Excel or SPSS by selecting the Data & Analysis tab and clicking on the Export & Import button.

 

HELPFUL RESOURCES:

(Note: the above information has been adapted from the survey support page at Bates University IRAP)