Latest Trends in Independent Evaluation at ADB

Olivier Serrat
2010

The views expressed in this presentation are the views of the author/s and do not necessarily reflect the views or policies of the Asian Development Bank, or its Board of Governors, or the governments they represent. ADB does not guarantee the accuracy of the data included in this presentation and accepts no responsibility for any consequence of their use. The countries listed in this presentation do not imply any view on ADB's part as to sovereignty or independent status or necessarily conform to ADB's terminology.

The Changing Operations Evaluation Landscape

There is new emphasis on aid effectiveness from increased demand for better development results. The aid community faces challenges caused by the crowded policy agenda, e.g., conflict reduction and prevention, fair international trade, environmental agreements, gender, a new international financial architecture, etc., many of which suffer from definitional problems and unclear time horizons, making them intrinsically difficult to evaluate. The aid community finds now often works with new partners in different partnership configurations; multi-agency consortia are becoming more common and agencies are moving beyond government-to-government links.

The Changing Operations Evaluation Landscape

The growing influence of nongovernment and civil society organizations is adding dimensions, often positive, to debates about aid and its delivery. New instruments, e.g., sector-wide approaches, comprehensive development frameworks, the livelihoods approach, etc., are being tried to change the way the aid community works; they require significant shifts in how evaluation is approached. Aid delivery has been shifting to new horizonsfrom the project to the program to the country levels; there is also an array of issues, e.g., climate change, trade, migration, etc., that require regional or global approaches.

Agents of Evaluation in ADB

In ADB, self-evaluation is conducted by the operations departments responsible for designing and implementing country partnership strategies, programs, projects, and technical assistance activities.
Independent evaluation is conducted by the Independent Evaluation Department to systematically evaluate policies, strategies, programs, and projects, e.g., design, implementation, results, and associated business processes, to determine relevance, effectiveness, efficiency, and sustainability, and impact.

Audiences for Evaluation

ADB Operations Departments (monitoring and evaluation, results agenda, applied learning)

Developing Member Countries (use of evaluations, evaluation capacity development, joint evaluations)

ADB Management (use and follow-up of evaluations, influence on ADB directions)

International Evaluation Community (harmonization and joint evaluations)

ADB Board of Directors (oversight)

Independent Evaluation Department (evaluations, monitoring and evaluation expertise, self-assessment)

Purpose of Independent Evaluation

Development Effectivenessto contribute to development results, i.e., maximize the development effectiveness of ADB's operations. Accountability to confirm that ADB is doing the right things, that resources are properly allocated and used, that intended outcomes are realized.
Generating Knowledge/Learning Lessonsto identify or derive lessons to improve the development impact of future policies, strategies, and operations.

Evolution of Independent Evaluation in ADB

1972A unit was established in the Economics Office. 1978A Post-Evaluation Office was set up. 1999An Operations Evaluation Office was formed. 2001An Operations Evaluation Department was constituted. 2004The Evaluation Head reports to the Board of Directors of ADB through its Development Effectiveness Committee. 2009An Independent Evaluation Department was created whose head is to be appointed by the Board.

The Role of the Development Effectiveness Committee

The Development Effectiveness Committee was established in December 2000. The DEC's mandate is to help the Board of Directors of ADB carry out its responsibility of ensuring that the programs and activities of ADB achieve development effectiveness. Development effectiveness is assessed in part through independent evaluation. For the purpose of the DEC's work, development effectiveness is the measure of (i) whether ADB's programs and activities have resulted in the desired outcomes; and (ii) whether these programs and activities have made efficient use of ADB's available resources.

The Role of IED

The core task of IED is the systematic and objective assessment of policies, strategies, programs, and projects, including their design, implementation, results, and associated business processes, to determine their relevance, effectiveness, efficiency, sustainability, and impact.

IED evaluates the development effectiveness of operations. It derives lessons to improve future operations. It contributes to ADB's accountability for results. It provides assistance to the DEC as the focal point for interaction with the Board of Directors. It seeks to improve self-evaluation in ADB.

A Few Influential Evaluations

Building a Results-Based Management Framework, 2003 Promoting Portfolio Performance, 2006 Improving Country Partnership Strategies, 2000s Contributing to Safeguard Policy Update Involuntary Resettlement, 2006 Indigenous Peoples, 2002 Environment, 2006

Issues in Independent Evaluation

The balance between accountability and learning, and between independence and engagement Conflict between closer engagement with developing member countries and Management Independence from the Boardis that advisable? Attribution or contributionnot all evaluations can address attribution issues How far should one move on self-evaluation and validation? How might one improve buy-in? Conduct fewer individual public sector project evaluations? Launch more country, sector, impact, and policy evaluations? Prepare more private sector evaluations?

Issues in Independent Evaluation

How to improve credibilitythrough stronger evidence? How to make better use of evaluation results? Which evaluation methods to apply? Qualitative or quantitative Rigorous or non-rigorous Experimental or non-experimental How to assess impact and sustainability? How to strengthen the accountability of Management? Recommendations must be clear, actionable, and monitorable How about the accountability of IED itself?

Knowledge Management for Independent Evaluation

Evaluation findings only add value when they are used. To be used, evaluation findings must be available when needed by decision makers, in user-friendly formats. A Knowledge Management Unit was established in IED in late 2006. Efforts are being made to improve dissemination of findings. IED invests continuingly in a searchable lessons database system that includes well over 1,000 lessons to date. Efforts to increase external dissemination through www.adb.org/site/evaluation/main, among other platforms, are well received.

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Matching Feedback Vehicles to Accountability and Learning

Improving Evaluation-Based Learning

ADB's strategic frameworks are moving toward greater concentration on country, sector, and thematic areas, and greater decentralization. ADB sees itself as a learning organization where operations evaluation plays a key part, and is given greater institutional independence and greater priority. More knowledge is generated from evaluations using more sophisticated methods. More impact-orientation is needed for quality improvement, conceptual advancement of development, and greater external accountability.

Improving Evaluation-Based Learning

ADB must systematize and institutionalize feedback systems for evaluation-based learning and accountability. Efforts are needed to manage the quickening flood of data and information, network the systems, and cater for upward feedback. ADB must intensify the internalization of lessons from evaluation. Implementation monitoring, that is, implementation of findings and recommendations from evaluation, should be strengthened by means of a Management action record system.

Improving Evaluation-Based Learning

ADB should continue to publicly release proposed evaluation approach papers at an early stage and post the comments of external stakeholders. ADB should continue to recognize the need for greater participation of in-country partners and stakeholders in the feedback process. Moving from project evaluations to a higher-level country, sector, and thematic focus is an important way of increasing potential evaluation impact.

Possible Directions in Evaluation

Assigning resources to broader evaluation studies Harmonizing evaluation standards (Jointly) Evaluating country/sector assistance programs Validating country partnership strategy and project/program completion reports Evaluating impact Developing evaluation capacity Promoting portfolio performance

Possible Directions in Evaluation

Evaluating business processes Disseminating findings and recommendations Promoting monitoring and measuring of outcomes Institutional evaluations , e.g., human resource function, information technology, knowledge management, Asian Development Bank Institute Rigorous impact evaluations

Further Reading
ADB. 2007. Independent Evaluation at the Asian Development Bank. Available: www.adb.org/publications/independentevaluation-asian-development-bank ADB. 2008. Conducting After-Action Reviews and Retrospects. Available: www.adb.org/publications/conducting-after-actionreviews-and-retrospects ADB. 2009. Learning from Evaluation. Available: www.adb.org/publications/learning-evaluation ADB. 2009. Most Significant Change Technique. Available: www.adb.org/publications/most-significant-change-technique ADB. 2011. Learning Histories. Available: www.adb.org/publications/learning-histories

Knowledge Management Center

Olivier Serrat
Principal Knowledge Management Specialist Knowledge Management Center Regional and Sustainable Development Department Asian Development Bank
knowledge@adb.org www.adb.org/knowledge-management www.facebook.com/adbknowledgesolutions www.scribd.com/knowledge_solutions www.twitter.com/adbknowledge