Planning and Designing a Survey
Create An Advisory Committee
Many of the steps outlined in this section are best accomplished through collaboration. Consider building an advisory committee for the survey that is composed of interested stakeholders who can participate in the planning and design, and who may also serve as an effective resource in terms of analysis, dissemination and utilization of results.
Planning Your Survey
Prior to designing a survey, address the following questions:
- What are the evaluation objectives of the survey and how will the results will be used?
- Who are the possible stakeholders and collaborators?
- Who has the information you need?
Identify Your Sample
In some cases, you may request information from an entire population, that is, all the members of a group who have the information you are seeking. More often you will proceed by identifying a subset of the group, which will be your sample for the survey. You will need to assemble a list with contact information so you can notify individuals about the survey and request their participation. Your stakeholders on advisory committee can be very helpful in providing contact information for you list.
It is important to consider how your sample may be subject to bias, in that it does not adequately represent the entire population you are interested in. For example, if your survey will assess current practices of vegetable growers, how might your sample be biased if you assemble a contact list from only those growers who attend Extension workshops? Or, if you use email to invite growers to participate in your survey, will your results be limited because the sample excluded growers who do not typically use email?
Not all potential sources of bias can be anticipated or overcome, but they are important to consider in the planning phase where they might be addressed before the survey is conducted. This process can also help you to understand and interpret your results in terms of the characteristics or limitations of your sample.
Create A Timeline
Create a timeline for the entire process that allows for:
- Developing and piloting the survey
- Giving respondents prior notice that their participation is requested
- Providing adequate time for respondents to complete the survey
- Entering and analyzing the data obtained
- Summarizing and sharing results with collaborators prior to any broader dissemination
Select Your Methodology
Surveys can be administered to respondents in a variety of ways:
- Forms distributed and completed at an event, workshop, etc.
Additional considerations related to selecting a survey methodology are available online from Rutgers Cooperative Extension at Choosing A Data Collection Method for Survey Research.
Many of the advantages and challenges associated with conducting a web-based survey are explained at Web-based Surveys.
A critical aspect of planning is to determine when you will administer your survey. There are options that relate primarily to how the timing of the survey coincides with the delivery of a program you wish to assess. For example, you could administer a survey at the conclusion of a program (After Only) that asks individuals whether the information and resources they received changed their attitude towards implementing particular IPM strategies.
There are, however, additional approaches, such as a Pretest–Posttest design (also called Before-After, that involve administering the same survey at two points in time, before and after individuals participate in a program. This is frequently a more desirable way to measure potential changes in an individual's knowledge, practices or decisions since the changes occur over a period of time.
A highbred approach, sometimes referred to as “Retrospective Post” (also called " Retrospective Pre-Post") is administered at the conclusion of a program, or at some later time. This approach asks respondents to rate their level of knowledge or the frequency with which they engage in certain practices both currently, and at a specific prior time, such as before they participated in the program you are assessing. Details about these different approaches are available at: Evaluation Design Common In Measuring Outcomes.
Designing Your Survey
Once you have completed the planning phase, you can commence with designing the survey instrument. A good strategy involves first conducting some interviews with members of the survey population as a precursor to make sure you understand how the population perceives the issues addressed in the survey.
Consider keeping your survey as short and as simple as possible. Every additional question and cognitive task in a survey increases the burden on respondents and potentially reduces the response rate. If you have experience completing surveys you will recognize the challenges in completing long surveys as well as the substantial variation in how survey instruments are constructed and how well specific survey items are written.
Details on many of the practices that should be followed in designing good surveys and good survey questions are available online from Rutgers Cooperative Extension at Step-By-Step Design.
Some quick tips for writing good survey questions can be accessed at Writing Survey Questions for Local Program Evaluations.