Step 3: Determine Evaluation Methods
3 Steps In Evaluation Planning:
- Define evaluation objectives
- Identify measurement indicators
- Determine evaluation methods
Determine Evaluation Methods
Once you have identified what data you need to obtain to document project outcomes, you need to consider how you will collect and analyze the data. Available methods are summarized briefly in the table below. Matching the method of data collection to the specific evaluation objective and target audience is important. The pros and cons of some methods are detailed within specific Modules (e.g., Surveys, see Module 2). Modules are being developed for all of the methods listed below with guidelines on when to engage an evaluation specialist or economist for each method. If the module has not been released yet, you will find helful information in Collecting Evaluation Data: An Overview of Sources and Methods.
|Tests||Provide measurements of awareness, knowledge, attitudes, skills||Pre-then-post tests implemented at meetings via written forms or audience response technology to measure a change in IPM knowledge. For more information|
|Surveys||Provide self-reported data from end-users on attitudes, beliefs, knowledge, skills, behaviors and demographics||IPM or pesticide use surveys implemented by mail, telephone, face-to-face, internet, audience response technology, retrospective self assessments|
|Observations||Obtain data related to evaluation objectives by directly observing people and programs||Document pest management operations in a school setting by "shadowing" pest management contractor and making notes. For more information|
|Interviews||Obtain information from people about things that cannot easily be observed, including actions, beliefs, how decisions are made, motivations for choices, etc. Interviews may be standardized or open-ended, conversational, or implemented using an interview guide approach; may be conducted one-on-one or in a focus group setting.||Interviews with land managers to determine how weed management decisions are made on public lands.|
To listen and gather information from a particular group of stakeholders on a particular issue
A group of pest managers identify factors that affect their adoption or non-adoption of recommended pest thresholds for key pests on melons.
|Cost Effectiveness Analysis||Evaluates which program or policy creates the desired result at the lowest cost||Biological control is compared with conventional and uncontrolled approaches on tomatoes.|
|Cost Benefit Analysis||Compares the economic pros and cons of policies and programs to help decision-makers identify the best or most valuable options to pursue. Particularly useful in IPM adoption studies where one goal is to work with a population of potential adopters and determine whether a new IPM practice makes financial sense for them||Comparison of alternative approaches to fruit fly management.|
|Partial Budget Analysis||Determines the net benefit by only examining the costs and gains that change for a program (e.g., using different pesticides or practices). Widely advocated as a way for growers, facility managers and other decision makers, to keep track of the results of changes they make.||Comparison of integrated crop management with conventional and organic production approaches in several fresh market vegetable crops.|
|Provides information on social interactions in relation to a specific set of questions. Data can be collected through surveys, interviews, observations, documents or other methods.||A survey is used to collect data on where growers obtain information on how to manage pests of strawberries and the information is used to construct a description of their social network.|
|Secondary Data (from existing sources based on previous research or surveys)||May be useful for understanding demographic issues or broad trends across certain populations. Could include pesticide use data or summarized data from IPM surveys, etc.||National Agricultural Statistics Service survey data; Department of Education demographic data on schools; Yield, quality or economic data from food processors or packers; pesticide use data from existing studies or sources (e.g., California Department of Pesticide Regulation pesticide use database).|
|Case Studies||Case studies use triangulation of several data sources to develop in-depth knowledge about, and to compare "cases" (e.g., individuals, communities, organizations). Better for understanding processes within a system than for generalizing across populations.||IPM program processes, function and outcomes are examined and compared across four rural school districts using in-depth interviews, document analysis and observation of program participants.|
It is important to consider your target audience. For example, an online survey will not work if a large portion of your potential target audience do not have email or use the web, for example. The results would be biased. The method must be suited to addressing the evaluation objective and it must also be appropriate for reaching the target audience to facilitate collection of a representative sample.
Mertens, D.M. & A.T. Wilson. 2012. Program Evaluation Theory and Practice: A Comprehensive Guide. Chapter 10, Data Collection Strategies and Indicators. The Guilford Press, NY.
Patton, M. Q. 2002. Qualitative Research and Evaluation Methods (3rd ed.). Sage Publications, Thousand Oaks, CA.