Indicators to measure effectiveness

Methodological Notes

Measuring the effectiveness of any public policy is a big challenge that is full of complexities (see the basic formula for impact assessment).

The answer to the basic impact evaluation question—What is the impact or causal eff ect of a program P on an outcome of interest Y?—is given by the basic impact evaluation formula:

α = (Y | P = 1) − (Y | P = 0).

This formula says that the causal impact (α) of a program (P) on an outcome (Y) is the diff erence between the outcome (Y) with the program (in other words, when P = 1) and the same outcome (Y) without the program (that is, when P = 0).

Source: From P. J. Gertler et al. 2011. “Impact Evaluation in Practice.” Washington, DC: World Bank.

Even more difficult is measuring practices of citizen participation that often are transversal to the different policies that are implemented. However, this task is essential to obtain better results.

The methodological aspects of how to carry out those measures can be found in the following resource:

 

The main stages of impact evaluation are described in this chart

Roadmap for implementing an Impact Evaluation

Source: From P. J. Gertler et al. 2011. “Impact Evaluation in Practice.” Washington, DC: World Bank.

Testing Program Alternatives for Monitoring Corruption in Indonesia

Source: From P. J. Gertler et al. 2011. “Impact Evaluation in Practice.” Washington, DC: World Bank.

Following is a series of variables that are particularly important for impact evaluations on citizen participation mechanisms in auditing cycles.

Indicators

Importance: Very high

Even though not all citizen participation mechanisms implemented by SAIs are designed to direct or provide content to the audit reports, in most instances they should result—indirectly or directly—in improving the quality of those reports. To that end, the public could offer suggestions for topics that demand auditing, aspects of auditing that are relevant, information about public policies that are audited, or the actual circumstances in which they find themselves, among others.

Measuring effectiveness in this respect is a difficult task, as it implies comparing the content of audit reports with those more limited ones that were generated without the input from citizen participation mechanisms. However, identifying the contribution that the engagement produced is important to advise and make their value and their forms of continuity public.

If the mechanisms of citizen participation implemented by SAIs do not modify the content of audit reports, a structural weakness will emerge concerning its scheme, which lacks significance and sustainability.

Carrying out quantitative and qualitative evaluations is therefore necessary. The reports that are influenced by the input resulting from participatory mechanisms must be measured (quantitative measurement, horizontal). At the same time, the depth and levels of influence in each must also be evaluated to ascertain the degree of quality improvement from those reports (qualitative measurement, vertical).

  • How many audit reports are influenced directly by the input from the implemented citizen participation mechanisms? How many indirectly?
  • What percentage represents each one in the total universe of audits?
  • Does that percentage reach the previously established quantitative goals? (These questions can be generally posed for all SAIs or in a particular way in terms of a specific topic).
  • Has the quality of audit reports improved since implementing the citizen participation mechanisms? In what way?

Importance: High

As we have seen, just as citizen input can contribute to improving audit reports, it can also be an important contribution to promote compliance with suggestions that they prescribe.

Evaluating this aspect of the implemented citizen participation mechanisms implies measuring the progress in levels of compliance from recommendations and analysing how that variation could have been influenced by the mechanisms implemented.

This task is often difficult, whether it is multi-causal (more than one cause could explains the phenomenon) or counterfactual (ascertaining what the level of compliance from recommendations would have been if the participatory mechanism had not been implemented could be difficult). However, the efforts that can be undertaken to analyse this effectiveness—including the eventual subjective considerations of informed internal evaluators—are significant to assess the success of the mechanism, especially in cases in which that mechanism is oriented toward the stages of monitoring reports.

  • Have the levels of compliance from recommendations formulated by the auditing institution changed since the implementation of citizen participation mechanisms?
  • To what extent has the implementation of those mechanisms been a contributing factor to such change?
  • Can the types of specific contributions by the public that added to the compliance of recommendations be identified?

Importance: Medium

Another way to measure the effectiveness of citizen participation in the SAI’s management is by identifying the change in media coverage on audit results since implementation of the initiative.

As we have seen, citizens and CSOs often provide an important contribution to the SAI administration when valuing and replicating the audit report content, especially in terms of its main findings and recommendations.

As a result, another way to identify the contributions made by citizen participation is evaluating media coverage since its implementation. That evaluation can be based on the quantity of press articles that a report (or a group of reports) generates, the increase in high-quality content in terms of the information that the media have or the depth of analysis, and the relevance and hierarchy that the media assigns to that news.

  • Has the level of media coverage of audit reports changed since the implementation of citizen participation or social accountability mechanisms?
  • Has the quality of press articles that refer to the results of the institution changed?
  • Has the relevance and hierarchy that the media assign to their articles regarding the topic changed?
  • To what extent are the changes attributed—totally or partially—to the implementation of citizen participation mechanisms?

Any comments? Please notify us here.

Bibliography

TPA Initiative – ACIJ (2013): “Audit Institutions in Latin America. Transparency, Citizen Participation and Accountability Indicators”.

OECD (2013): “How can Supreme Audit Institutions Engage External Stakeholders to Enhance Good Governance?”, Concept Note - OECD with Supreme Audit Institutions of Brazil, Chile, South Africa & IDI.

Johnsøn, J. & Søreide, T. (2013): “Methods for learning what works and why in anti-corruption: An introduction to evaluation methods for practitioners”, U4 Issue N°8, Bergen, U4 Anti-Corruption Resource Centre- Chr. Michelsen Institute.

Gertler, P. J. et al. (2011): “Impact Evaluation in Practice”, Washington, DC: World Bank.

Hevia, F. & Vergara-Lope Samana (2011): “How to measure participation”, CIESAS – INDESOL. “Practical Guide For Evaluating Participatory Processes”; Marc Pares, Leonardo Díaz (IGOP/UAB) & Melissa Pomeroy (OIDP/OLDP)

“Supreme Audit Institutions and Stakeholder Engagement Practices. A Stocktaking Report”, Effective Institutions Platform, September 2014.

“Working with Supreme Audit Institutions", Department for International Development (DFID, 2005).

Open Budget Survey 2012, International Budget Partnership (IBP)