How to Collect It
Data Collection Methods - Consideration In Economics
In the Evaluation Planning module (Module 1) there is a list of many approaches which can be used to gather data in an impact analysis (Table in Step 3). Several of these approaches are suitable for collecting economic data from participants in such studies. That is, an economic analysis is something which can be embedded within a study with wider goals, or which can make use of data collection approaches that apply equally to non-economic methods for assessing impact.
Of the methods listed in the table, surveys, observation, interviews, focus groups and case studies can all easily serve a dual purpose of providing a means to collect the data needed for an economic analysis, while allowing other project objectives, such as efficacy or adoption measurement, to be met. It is difficult to make definitive recommendations about which approach to use for collecting economic data because a wide variety of factors can influence the availability and quality of data and these will change from one set of circumstances to another.
In the absence of definitive statements, here are some issues to consider when planning your approach to economic analysis data collection.
How Familiar With The Information Needed Are The Project Participants?
If you are working with a well-informed group of participants who are used to working on technical issues, you may be able to use a relatively hands-off approach, such as a survey in which people self-report, to gather the data, trusting the participants to produce data of a sufficient quality on their own. If a survey will be suitable for your project, particularly for numerical data it may be worth considering using an online tool for data collection since this will allow the transfer of data to a spreadsheet or statistics package to be automated.
In contrast, with less well-informed user-groups you may have to allocate sufficient human resources to helping people gather the data you need and to carry out the necessary calculations. Under such circumstances, focus groups or interviews may act as suitable means to collect the data.
What Size Of Sample Do You Need To Estimate Impact Meaningfully?
The answer to this question will depend in part on the way you define the measure of impact in your project proposal. For example, if you decide to assess economic impact by estimating the cost-benefit balance for a representative case study from your target population, your sample size may be 1. In this instance, the economic analysis will form only a part of the detailed study of the impacts on the case-study subject.
In contrast if you need to assess the average impact across the population of potential adopters, you will need to use a method which allows you to collect a large number of individual estimates, from which you can derive aggregated summary statistics (the mean and variance of the cost-effectiveness, for example) for the population of interest. In this case a survey, or perhaps focus groups may be suitable approaches. Again, the resources needed to make such approaches effective will depend on the expected quality of data from the target population in the absence of help.
Is There A Wider Policy Issue Which Constrains The Choice Of Approach?
General or specific policy statements may prescribe a relevant scale at which economic impacts should be assessed. The sample size, and hence the most suitable approach to collecting the data will be the result of the scale of interest and level of funding by the policy makers.
Collect As High Resolution Data As Possible
These are only three very general questions which may be relevant to the decision about what size of data sample is needed and who should be invited to give their data up for analysis. Notice that the same generic issue is in play in all three cases: we try to define the scale of interest and the sample size needed at that scale. The approach to data collection then flows from answering those questions. An important point to stress is that where possible you should always collect data at the individual scale. Individual measurements can be aggregated into larger groupings, but the opposite process of deriving individual figures from aggregated responses almost never works in a meaningful way.
Finally, You May Want To Provide A Wider Economic Context For Your Analyses.
Two approaches for gathering the necessary information are either to convene a working group of experts and ask for their best guesses for summary statistics which, collectively, define the context for the IPM impact at issue.
The other approach is to try to estimate a meaningful context from public data bases. Public databases are held, for example, by the EPA, ERS and NASS. Further databases may also exist at the State level.