Fertilizer in malawi impact evaluation methodologies outline intro to impact evaluation. A theory of change should be used in some form in every impact evaluation. Impact evaluation in practice is an essential resource for evaluators, social programs, ministries, and others committed to making decisions using good evidence. Data collection and analysis methods in impact evaluation. Impact evaluation methods april 9, 2012 diva dhar clear regional center j. Expert judgment quantitative physical and mathematical models cumulative impact assessment matrices and interaction diagrams rapid impact assessment matrix riam battelle environmental evaluation system. Choosing an impact evaluation method 187 determining which method to use for a given program 187 how a programs rules of operation can help choose an impact evaluation method 188 a comparison of impact evaluation methods 193 finding the smallest feasible unit of intervention 197 chapter 12. Both qualitative and quantitative methods can be used within the design of an evaluation. Quantitative and qualitative methods in impact evaluation. Notes 3 and 4 set out the elements that need to be in place and the decisions to be taken before initiating an evaluation. This series is designed to introduce you to a number of important concepts that will help you understand economic evaluation and how to incorporate these methods into your programs. Policy evaluation applies evaluation principles and methods to examine the content, implementation or impact. Stockholm international peace research institute sipri vegard iversen.
Data collection and analysis methods in impact evaluation page 2 outputs and desired outcomes and impacts see brief no. The evaluation of community engagement may need both qualitative and quantitative methods because of the diversity of issues addressed e. Choosing designs and methods for impact evaluation 20 3. The largest number of detractors manifested for my organization emphasizes the use of impact evaluation findings. The answers to these questions may be less straightforward than the questions themselves.
Qualitative methods may be employed for formative and process evaluations, e. For an example in process evaluation, please see the norwegian case electricity savings in households n 3 in chapter 8. Experimental evaluation social experiment program is randomly assigned, so that everyone has the same probability of receiving the treatment. Impact evaluation seeks to determine the longer term results that are generated by policy decisions, often through interventions, projects or programmes. Impact evaluation is an assessment of how the intervention being evaluated affects.
Public disclosure authorized public disclosure authorized public disclosure authorized. Quantitative methods and practices makes a valu able contribution in this area by providing, for policy and research audiences, a com. Impact evaluation methods in public finance a brief introduction to randomized evaluations and comparison with other methods may 2015 dina pomeranz harvard university and nber1 abstract recent years have seen a large expansion in the use of rigorous impact evaluation techniques. Impact assessment methods this document outlines several common impact assessment methods, including. Impact evaluations need to account for the counterfactual what would have occurred without the intervention through the use of an experimental or quasiexperimental design using.
Introduction voluntary sector organisations are being urged to provide more convincing evidence of their impact and with good reason, given that funders and commissioners want. Pdf on apr 2, 2017, abhisek mishra and others published methods of impact evaluation. The risk of failing to conclude that your program has an impact even when it does. Types and uses of evaluation in order to plan the evaluation in accord with the most appropriate evaluation method, it is necessary to understand the difference between evaluation types. A primer on geospatial impact evaluation methods, tools, and. Theme one states that use does not happen by accident. Needs assessment process evaluation impact evaluation to address the low literacy levels, the government will deliver free text books to schools to improve passing rates and increase employment prospects of school leavers. Impact evaluation uses empirical techniques to estimate the effects of interventions and their statistical significance, whereas impact assessment includes a broader set of methods, including structural simulations and other approaches that cannot test for statistical significance. Pdf 1mb references cited by explains that the differenceindifferences dd method compares the changes in outcomes over time between a population enrolled in a program the treatment group and a population not enrolled the comparison group. This work is increasingly important as the global development community works to reduce poverty and achieve the 2030 sustainable development goals. Measurement of impact of any developmental programme is a difficult task. Impact evaluation methods hosted by the world bank development research group in kuala lumpur impact evaluation in practice 12 15 november 2018 monthurs 9 a. Impact evaluation provides the causal effect of a programme on the outcome. Module 9 introduction to impact evaluation usaid learning lab.
Combining quantitative formal and qualitative informal survey methods dfid, 2001. The formal literature on impact evaluation methods and practices is large, with a few useful overviews. An impact evaluation analyses the positive or negative, intended or unintended impact of a project, programme, or policy on the target population, and quantifies how large that impact is. Introduction to evaluation research methods knowledge base. This paper highlights the existence of different definitions and interpretations of impact, within a context where attention to impact is increasingly prominent and instrumental in shaping the funding, design, management and evaluation of development programmes. However, care must be taken about generalizing from a specific context. In this impact evaluation, the data collected will focus on understanding the relationship between the programs outputs e. Page 2 outputs and desired outcomes and impacts see brief no. Impacts may be positive or negative, intended or unintended, direct or indirect. Impact evaluations establish the causal effect of a. The only statement that has strong support is that impact evaluation does no harm.
Process based evaluations analyze strengths and weaknesses. Introduction impact evaluation is increasingly recognized as an important part of the evaluation toolkit, resulting in calls for more and better impact evaluation. Outline of principles of impact evalation discussion dra. A primer on geospatial impact evaluation methods, tools. The choice of methods and designs for impact evaluation of policies and.
The four types of analysis that we will discuss in this series are. Theorybased evaluations can be used with individual cases such as a single policy change or with multiple cases, such receiving capacity development support. In phase 1, the evaluation is planned, including obtaining the necessary approvals, identifying experts to be consulted, and beginning the contracting process. This brief summarizes the results of a gender impact evaluation study, entitled an impact evaluation of education, health, and water supply investments by the bolivian social investment fund. Quantitative and qualitative methods in impact evaluation and measuring results. Choosing appropriate designs and methods for impact. Section 3 outlines the methods of dealing with endogeneity. The third theme considers the wider set of institutional incentives and rewards for using impact evaluation. We discuss what an mm impact evaluation design is, what distinguishes it from a quant or qual impact evaluation design and why the approach is helpful for understanding development evaluations. Approaches, tor and methods for impact evaluation interact. In 2012 dfid commissioned a study broadening the range of designs and methods for impact evaluations stern et al.
Impact evaluation methods experimental design presented by. Impact evaluation investment learning platform ilp food. It can be used with any research design that aims to infer causality, it can use a range of qualitative and quantitative data, and provide support for triangulating the data arising from a mixed methods impact evaluation. Note 5 is essential prereading to introduce readers to the different options for impact evaluation. The choice of methods should fit the need for the evaluation, its timeline, and available resources holland et al.
Quasiexperimental methods that involve the creation of a comparison group are most often used when it is not possible to randomize individuals or groups to treatment and control groups. It may also be necessary to use quasiexperimental designs for exante. Evaluation is the activity through which we develop an understanding of the merit, worth, and utility of a policy. Outline of principles of impact evaluation part i key concepts definition impact evaluation is an assessment of how the intervention being evaluated affects outcomes, whether these effects are intended or unintended. Survey data collection for impact evaluation its not as easy as we might think. Types of evaluation centers for disease control and.
Introduction to mixed methods in impact evaluation. It begins with a brief statement of the evaluation problem section 2. Programme evaluation different aspects of evaluation case study example. Choosing appropriate designs and methods for impact evaluation. Additional attention was drawn to the issue recently by a. Themes one and two are closely interrelated and speak to the practical steps that enable use. The answer is economic evaluationa powerful tool that can help with all these situations. Impact evaluation is the main means for empirically testing what actually happens when interventions are implemented. In order to plan the evaluation in accord with the most appropriate evaluation method, it is necessary to understand the difference between evaluation types. How to evaluate the programme world health organization. Pure randomization is rare for antipoverty programs in practice, since randomization. A multimethod impact evaluation of a therapeutic summer.
Impact evaluation investment learning platform ilp. In response to this growing consensus there are a number of international initiatives on impact evaluation. Overview of impact evaluation page 2 an impact evaluation should only be undertaken when its intended use can be clearly identified and when it is likely to be able to produce useful findings, taking into account the availability of resources and the timing of decisions about the programme or policy under investigation. The proper analysis of impact requires a counterfactual of what those outcomes would have been in the absence of the intervention. In parallel to these trends, behavioral economics has increasingly attracted. They can be used in any sector and with any kind of work. Quantitative and qualitative methods in impact evaluation and measuring results 1 1 introduction there has been a renewed interest in impact evaluation1 in recent years amongst development agencies and donors. Quantitative and qualitative methods in impact evaluation and. The three main types of evaluation methods are goal based, process based and outcomes based. In parallel to these trends, behavioral economics has. An impact evaluation relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on causeandeffect analysis. In total, impact evaluation was carried out in 29 cases out of 41.
The success case method scm involves identifying the most and least successful cases in a program and examining them in detail. Goal based evaluations measure if objectives have been achieved. Mar 31, 2017 impact evaluation provides the causal effect of a programme on the outcome. Evaluation is about being open to continuing feedback and adjusting your programs accordingly. Approaches and methods of theory based impact evaluation. In theory, this method is assumption free, but in practice many assumptions are required.
This article reports on a multi method impact evaluation of a therapeutic summer camp program for children and youth with learning disabilities and related psychosocial problems. For more on using a theory of change and logic model to inform an evaluation plan, see. Could lead to policy decisions to eliminate the program, which would be detrimental to beneficiaries and society an impact evaluation is powerful if there is a low risk of not. The study examines the degree to which program objectives were achieved through a precamp, postcamp, fol. While there are a variety of different approaches to evaluation, this. Economistpublicationsdocumentsaustralianindustryreport. Process evaluation, as an emerging area of evaluation research, is generally associated with qualitative research methods. Yet there is a need to put the theory into practice in a handson fashion for practitioners. There are a variety of evaluation designs, and the type of evaluation should match the development level of the program or program activity appropriately. Karthik muralidharan is an associate professor of economics at the university of. Like other impact evaluation designs, mm can be applied at any level, from the evaluation of a project operating in a single village to a multicomponent. This brief summarizes the results of a gender impact evaluation study, entitled an impact evaluation of education, health, and water supply investments by the.
Purpose of impact evaluation impact evaluation serves both objectives of evaluation. This note is a guide to implementing an impact evaluation for youth employment. Using the regression discontinuity design method to evaluate the health. Using qualitative methods to assess impact jean ellis, ncvo charities evaluation services associate july 2015 1. University of manchester yashodhan ghorpade institute of development studies tilman bruck.
Could lead to policy decisions to eliminate the program, which would be detrimental to beneficiaries and society an impact evaluation. This approach to evaluation focuses on the practicalities of defining successful outcomes and success cases brinkerhoff, 2003 and uses some of the processes from theorydriven evaluation to determine the linkages, which may take the form of a logic model, an impact model, or a results map. The impact of the full range of development interventions can be assessed projects, programmes, sector wide support and donor country strategies and macro economic growth and programme aid support using qualitative or quantitative approaches or a mix. This approach was developed by robert brinkerhoff to assess the impact of organisational interventions, such as training and coaching, though the use of scm is not limited to this context. The theory of change should also take into account any unintended positive or negative results. This is always the case for expost impact evaluation designs. Impact evaluation, methods, randomisation, regression discontinuity, propensity score matching. Impact evaluations need to account for the counterfactual what would have occurred without the intervention through the use of an experimental or quasiexperimental design using comparison and treatment groups. Design the intervention collect followup data scope of this presentation impact evaluation project cycle 2. What methods may be used in impact evaluations of humanitarian assistance.
831 931 908 1594 757 660 90 858 72 1050 1204 1340 730 960 990 586 29 499 1168 1510 946 879 1108 822 863 120 211 1494 903 1565 1494 450 642 409 966 1013 291 323 154 405 863