Part 5: Evaluating aid programmes

5.1
In this Part, we set out our expectations for how NZAID:

• evaluates the effectiveness of aid programmes;
• provides resources for evaluation; and
• uses evaluation to improve aid programmes.

5.2
We then present our findings on the extent to which NZAID met those expectations.

Our expectations

5.3
We expected NZAID to:

• evaluate the effectiveness of its aid programmes in terms of how well they are
• implemented, and their outputs and longer-term outcomes;
• have enough resources and people skilled in development evaluation to evaluate the effectiveness of its aid programmes; and
• use evaluations to improve how effectively aid programmes are planned, managed, and delivered by sharing the results and lessons learned internally and externally with its aid partners and others.

Summary of our findings

5.4
NZAID reviews and evaluates the effectiveness of its aid programmes through a range of evaluative activities. It collects, monitors, and analyses data on programme reviews and evaluations carried out throughout NZAID. Although NZAID staff actively take part, most evaluations and reviews are contracted out. The lessons learned from reviews and evaluations are fed back formally and informally into the management and delivery of aid programmes through various mechanisms.

5.5
In our view, NZAID needs to:

• prioritise initiatives and projects within programmes for review or evaluation;
• ensure that evaluations at the end of an initiative or project focus on assessing effectiveness;
• enter relevant information on reviews and evaluations consistently into its Activity Management System (AMS);¹
• ensure that external contractors with specialist evaluation skills and experience on the ACS database are more easily identified;
• assess the performance of contractors after they complete each assignment; and
• ensure that reviews and evaluations of initiatives and projects within programmes focus on lessons learned from how the activity was planned, managed, and delivered and how the activity contributed to wider programme objectives.

Evaluating the effectiveness of aid programmes

5.6
NZAID reviews and evaluates the effectiveness of its aid programmes through a range of evaluation activity. Evaluations and reviews are carried out of:

• programmes;
• initiatives and activities within programmes;
• all activity within a sector (for example, education or health); and
• all activity within and across programmes to address thematic and crosscutting issues (for example, sustainable rural livelihoods or human rights).

5.7
Figure 6 provides more information about the reviews of the Papua New Guinea and Indonesia bilateral programmes.

Figure 6
Reviews of the Papua New Guinea and Indonesia bilateral programmes

5.8
NZAID has an evaluation framework that includes an overarching policy statement and various evaluation practice guidelines and standards. NZAID’s evaluation policy statement provides direction for evaluation activities within NZAID, and is guided by the OECD Development Assistance Committee’s principles for evaluating development assistance and evaluation criteria.

5.9
NZAID programme teams are responsible for ensuring that the aid programmes they manage, and initiatives and projects within those aid programmes, are evaluated and reviewed. However, there is no clear approach throughout NZAID to prioritising initiatives and projects within programmes for evaluation or review, or ensuring that evaluations at the end of an initiative or project focus on assessing effectiveness against objectives and relevant outcome measures.

5.10
In our view, to support consistency and the analysis of evaluation activity carried out throughout aid programmes, NZAID needs to prioritise projects and initiatives across programmes for review or evaluation. The primary focus needs to be on assessing effectiveness against objectives and outcome measures. Unintended outcomes may also need to be considered as part of assessing effectiveness.

5.11
NZAID collects, monitors, and analyses data on reviews and evaluations of programmes and the initiatives and projects within programmes.

5.12
So far, three summary review reports have been prepared – for July 2002 to December 2004, January to December 2005, and January to December 2006. These reports are desk-based studies that summarise findings and draw lessons from reviews and evaluations carried out by NZAID during those periods. However, NZAID staff noted that it was difficult to get an accurate picture of all the evaluation activity carried out throughout NZAID because the information is held within programme teams and not entered consistently in its AMS.

5.13
We reviewed a sample of 19 programme activity authority sheets held in NZAID’s AMS for initiatives and projects within the Papua New Guinea and Indonesia bilateral programmes and the Pacific Regional Health programme. Programme activity authority sheets contain information on each initiative or project within the programme, including specific fields for “monitoring” and “evaluation”, and team leaders or Group Directors use these for review purposes. NZAID staff noted that the “evaluation” field had been added in the AMS recently.

5.14
Although information about reviews and evaluations was contained in some “monitoring” fields, it was not entered consistently. Only two of the 19 programme activity authority sheets contained any detail within the “evaluation” fields that showed when the reviews or evaluations were scheduled to take place.

5.15
In our view, NZAID needs to ensure that all relevant information about aid project or initiative reviews and evaluations is consistently entered into its AMS.

Resourcing evaluations

5.16
NZAID has a small evaluation team within the Strategy, Advisory and Evaluation Group that is responsible for providing advice on evaluation activity throughout NZAID. The evaluation team focuses on providing advice to development programme managers on the terms of reference for, and peer review of, evaluations and reviews.

5.17
Although NZAID staff actively take part, external contractors carry out most evaluations and reviews of programmes, and of initiatives and projects within programmes. NZAID considers that external reviews and evaluations provide a useful level of independence. External contractors often lead collaborative evaluation or review teams, and are appointed under NZAID’s ACS. The ACS focuses on individuals who are chosen for their specific skills and experience, including evaluation. Figure 7 provides more information about the ACS.

Figure 7
NZAID’s Approved Contractor Scheme

5.18
Because of a shortage of suitably experienced evaluators in development internationally, NZAID staff noted that there were few contractors in the ACS database with specialist evaluation skills and experience. Those contractors on the database with evaluation skills and experience were often difficult to find. There is no formal assessment or review of ACS contractors after they have completed an assignment.

5.19
In our view, because external contractors carry out most NZAID evaluations and reviews, NZAID should review the performance of those contractors after they have completed each assignment.

Using evaluation to improve aid programmes

5.20
Lessons learned from reviews and evaluations are fed back formally and informally into the management and delivery of aid programmes through various mechanisms, including:

• NZAID’s Internal Evaluation Committee;
• presentations by evaluation teams; and
• periodic cables to overseas posts.

5.21
NZAID’s guidelines for disseminating and using evaluations notes the value of disseminating evaluations, and NZAID staff have said that evaluation summaries will soon be published on NZAID’s website.

5.22
NZAID has reviewed evaluations carried out across its programmes. Three summary review reports have been prepared (see paragraph 5.12) that summarise findings and draw lessons from reviews and evaluations carried out throughout NZAID.

5.23
The evaluation summary report for January to December 2005 summarised lessons learned from the reviews and evaluations, and identified the strengths and weaknesses of the reviews and evaluations carried out. On the quality of the evaluations carried out, the report noted that the 17 reports considered as part of the review were of “very mixed quality”.

5.24
Common programme issues identified in the report included:

• weaknesses in design;
• inadequate use of participatory approaches;
• lack of gender analysis;
• inadequate attention to issues of sustainability and replication, particularly with setting up parallel structures and systems rather than strengthening existing structures and systems; and
• a lack of focus on poverty.

5.25
The evaluation summary report for January to December 2006 also noted that the quality of the 16 review and evaluation reports that were assessed varied considerably from comprehensive, clear, credible, and useful reports to reports that were too brief, too general, or incomplete.

5.26
We reviewed a sample of 15 review and evaluation reports commissioned by NZAID or wider stakeholder or aid donor groups for aid projects and initiatives within the Papua New Guinea and Indonesia bilateral programmes and the Pacific Regional Health programme. Of the 15 reviews and evaluations, 11 focused on projects and initiatives, and four focused on a particular sector or issue (for example, health and immunisation).

5.27
Collaborative evaluation or review teams led by external contractors carried out most of the reviews and evaluations, and many involved consultation with development partners, stakeholders, and selected local communities.

5.28
The reviews and evaluations varied in their focus and objectives, and it was often not clear in the reports when the reviews or evaluations occurred during the activity cycle. Many review and evaluation objectives included considering increased support in the future, but few focused on or considered lessons from how the initiative or project was planned, managed, or delivered or how the project or initiative contributed to wider programme or NZAID objectives.

5.29
In our view, NZAID needs to ensure that reviews and evaluations of aid projects and initiatives within programmes focus on lessons learned from how the activity was planned, managed, or delivered, and how the activity contributed to wider programme objectives.

1: The Activity Management System is the software used to plan and manage NZAID’s activities and budgets.