Everyone agrees that measuring the impact of the EITI is essential – and that doing this properly is notoriously difficult
The Covid-19 pandemic and associated economic downturn is an unprecedented global shock. It has challenged those working in development to redouble their efforts to promote social justice, equality and sustainability. Through the crisis, the EITI’s goal has been to safely contribute to global and national efforts to respond to the pandemic, while upholding transparency, accountability and multi-stakeholder dialogue. This was the backdrop for the EITI Board meeting in mid-June, where attention once again turned to measuring the EITI’s impact.
There is agreement within the EITI community that understanding, in concrete terms, how the EITI makes a difference is essential to attract and retain support from stakeholders. At the same time, there is also an appreciation that measuring impact accurately and consistently is extremely challenging.
The EITI has also been the subject of dozens of independent evaluations and research projects. A review of 50 such evaluations concluded that the EITI had succeeded in “diffusing the norm of transparency, establishing the EITI Standard and institutionalising transparency practices.” But it also identified a lack of evidence linking EITI adoption to development outcomes.
Connecting the dots
In the development community, the gold standard of impact evaluation has been to develop a clear “theory of change,” which is used to assess the impact of interventions by comparing actual outcomes with the potential outcomes of non-intervention. Impact evaluations of the EITI have taken various approaches. Some have focused specifically on the impact of the EITI on corruption. Others have taken a broader lens, seeking to measure impact on a wider set of governance indicators.
These divergent approaches highlight a fundamental challenge: EITI stakeholders perceive the EITI’s objectives and expected impact differently. For some, the EITI is about preventing conflict, for others it is about economic growth and investment, widening democratic space or improving government accountability and community benefits. Other perspectives reference less tangible objectives, such as improved reputation and trust. Even if there is a widely shared goal to reduce corruption risks, views diverge on which ones to prioritise.
The EITI is implemented within complex national settings, with results expected over the long term. There are also many other factors at play which influence the success of EITI implementation. The EITI’s approach to impact assessment needs to take into account the diversity of implementing country circumstances and the divergent (and sometimes conflicting) expectations of different stakeholders.
The EITI Board has worked for many years to find the right formula for monitoring and evaluation, both at the national and global level. But, with the recent introduction of the 2019 EITI Standard, the time was right to commission research that would update our knowledge base and inform future debate and actions.
The resulting report begins with a review of best practice in other multi-stakeholder initiatives. It offers four important insights from this research.
- Most organisations find that measurement of key outcomes provides greater value than measuring long-term socio-economic impacts. Assessment of impact is best left to independent research.
- There is a move towards “user-centric” measurement and evaluation (M&E) frameworks. This means that these frameworks are increasingly geared towards continuous assessment and improvement.
- The diversity and country ownership of EITI implementation is unique in the multi-stakeholder transparency space. Best practice from other organisations cannot be assumed to be best practice for the EITI.
- There is an opportunity for global leadership in public discourse on the results measurement of complex multi-stakeholder initiatives such as the EITI.
A focus on the needs of the user (in the “user-centric” approach to M&E outlined above) allows for the identification of three categories of evidence needs:
- Evidence to improve implementation on an ongoing basis, to understand “what works” based on early results, and to help stakeholders adapt implementation to changing conditions and assumptions.
- Evidence to justify the EITI’s effectiveness as a platform for improving governance, demonstrating to national and international stakeholders why the EITI matters and deserves their support.
- Evidence to promote the EITI and recruit new countries and supporting companies.
The report recommends that the EITI should develop measurement systems that meet these demands for evidence, promote internal learning and lead to better understanding of results at a country level. It cautions that the latter should be based on a realistic assessment of the constraints posed by different national contexts, rather than a box-ticking approach to measurement motivated by a perceived need to satisfy external stakeholders.
It offers a series of suggestions on how to improve the EITI’s approach. These include the development of country capacity and an increased focus on developing a culture of learning. The report further identifies an opportunity for international leadership in the discourse around outcomes and impact. It advocates an independent evaluation of the impact of the EITI based on these principles.
In response, the EITI Board has agreed a series of measures. Three are particularly noteworthy. First, agreement to revise the guidance to EITI implementing countries on work plan development, reporting on progress and establishing monitoring and evaluation frameworks. Second, the development of a results measurement framework that can be adapted by implementing countries. Finally, agreement to undertake further work on independent evaluation of the EITI. The scope, timing and resourcing will be confirmed when the EITI Board meets in October.
The EITI International Secretariat has created a focal point for further work on these issues. The EITI’s Technical Director Sam Bartlett will lead this work together with the EITI’s Digital Manager, Christina Berger. EITI stakeholders that are interested in further contributing to this work are invited to contact the International Secretariat.