Methods and practices for impact evaluation

Different quantitative and qualitative methods for evaluation are available to answer the key questions that have been developed in this module—as well as other questions that evaluators may find pertinent.

Several studies provide technical tools to carry out evaluations on the impact of public policy. A great deal of content from those studies is applicable to cases that concern us: evaluation of the effectiveness of civil participation policies carried out by SAIs.

Although the bibliography for this topic is very extensive, we have highlighted two documents that will help the evaluator:

Shahidur R. Khandker, Gayatri B. Koolwal, and Hussain A. Samad. 2010. “Handbook on Impact Evaluation. Quantitative Methods and Practices.” Washington, DC: The World Bank.

Click Here

Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2011. “Impact Evaluation in Practice.” Washington, DC: The World Bank.

Click Here

Evaluation Methods

Source: From “Supreme Audit Institutions and Stakeholder Engagement Practices. A Stocktaking Report.”Based on Johnsøn and Soreide 2013.

What happened in Chile (From “Supreme Audit Institutions and Stakeholder Engagement Practices. A Stocktaking Report.”)

“Some SAIs have advanced with process evaluation by systematically gathering information about ongoing activities and the outputs they produce. A good example is the indicators and information gathered by Chile’s CGR through a built-in component of the web portal Contraloría y Ciudadano (see Box 23 below). However, one limitation of these incipient efforts is that indicators have not been measured over long periods of time yet, which makes it difficult to conduct longitudinal comparisons and to assess trends and patterns over time.”

Annex

Key Areas and Excellence Factors for Evaluating the Effectiveness of a SAI

See “Working with Supreme Audit Institutions.” Technical Annex 1: Key Areas and Excellent Factors for Evaluating the Effectiveness of a SAI. Department for International Development (DFID 2005).

See the results and best scores in Questions 123–5 in IBP’s Open Budget Survey:

Source: From Open Budget Survey 2012, International Budget Partnership (IBP), Table 9.

Any comments? Please notify us here.

Bibliography

TPA Initiative – ACIJ (2013): “Audit Institutions in Latin America. Transparency, Citizen Participation and Accountability Indicators”.

OECD (2013): “How can Supreme Audit Institutions Engage External Stakeholders to Enhance Good Governance?”, Concept Note - OECD with Supreme Audit Institutions of Brazil, Chile, South Africa & IDI.

Johnsøn, J. & Søreide, T. (2013): “Methods for learning what works and why in anti-corruption: An introduction to evaluation methods for practitioners”, U4 Issue N°8, Bergen, U4 Anti-Corruption Resource Centre- Chr. Michelsen Institute.

Gertler, P. J. et al. (2011): “Impact Evaluation in Practice”, Washington, DC: World Bank.

Hevia, F. & Vergara-Lope Samana (2011): “How to measure participation”, CIESAS – INDESOL. “Practical Guide For Evaluating Participatory Processes”; Marc Pares, Leonardo Díaz (IGOP/UAB) & Melissa Pomeroy (OIDP/OLDP)

“Supreme Audit Institutions and Stakeholder Engagement Practices. A Stocktaking Report”, Effective Institutions Platform, September 2014.

“Working with Supreme Audit Institutions", Department for International Development (DFID, 2005).

Open Budget Survey 2012, International Budget Partnership (IBP)