Part 4: Accountability through reporting, monitoring, and evaluation
4.1
In this Part, we describe:
- the reporting, monitoring, and evaluation carried out for the Programme of Action as a whole and the Sustainable Cities and Energy workstreams;
- how the Programme of Action contributed to capability building through shared learning forums;
- our expectations and findings; and
- the implications for cross-agency work.
Reporting to Ministers, local government councillors, and the public
4.2
The Programme of Action contains a requirement for reporting on progress towards sustainability. Also, staff were required to report to Ministers on emerging sustainability issues and progress with implementation of the Programme of Action.
4.3
The Programme of Action pointed out that no single agency collected all the data needed for sustainable development reporting and recognised that partnerships and co-ordinated reporting would be needed. It listed local government, sector groups, and central government agencies as partners.
4.4
Although it is not mentioned in the Programme of Action, the United Nations Commission on Sustainable Development expects to receive regular reports on the implementation of commitments agreed at the World Summit.
Our expectations
4.5
We expected Ministers and local government councillors to have been kept informed of progress with implementation of the Programme of Action and advised of emerging sustainability issues.
4.6
We note that the Programme of Action makes no reference to a requirement for reporting to the general public. Nevertheless, as the Programme of Action was both a public document and the Government’s response to an international agreement, we expected that public information about the implementation of the Programme of Action would be available.
What we found – the Programme of Action as a whole
4.7
We were told that reporting to the co-ordinating Minister was mostly informal, and we found few written reports.
4.8
Departments involved in the various workstreams reported on their activities to the Senior Officials Co-ordinating Group, which in turn reported to the Chief Executives Group. Each chief executive also had direct reporting responsibility to their Minister.
4.9
In late 2004, DPMC prepared for publication a mid-term report on progress to implement the Programme of Action, but this was not published. There have been no reports to Parliament about implementation of the Programme of Action. At least one summary report was sent to the United Nations in 2004.
What we found – the Sustainable Cities workstream
4.10
Cabinet received a progress report for the Sustainable Cities workstream in August 2003.
4.11
The Ministry for the Environment and the Ministry of Economic Development prepared regular but separate reports for Sustainable Cities to their responsible Ministers – the Minister for Urban Affairs and the Minister for Industry and Regional Development respectively. The National Programme reported to the Sustainable Cities Senior Officials Group (see Figure 5).
4.12
For the Auckland Programme, the Programme Leaders Group:
- reported to mayoral meetings, chief executive forums, and local government councillors on issues and progress;
- reported to the Auckland Programme’s Combined Steering Group; and
- published progress and evaluation reports on a website.
4.13
In turn, the Auckland Programme’s Combined Steering Group (which included central and local government officials) gave or approved reports to:
- an Auckland Programme’s Senior Officials Group, made up of local government members; and
- the Sustainable Cities Senior Officials Group, made up of central government members.
4.14
These two groups then reported to their chief executives and to their local government councillors. The Auckland Programme established reporting templates and processes, and published various reports such as Success in Sustainability in July 2006.
What we found – the Energy workstream
4.15
Cabinet approved the release of the discussion document Sustainable Energy: Creating a Sustainable Energy System in late 2004. The results of consultation Accountability through reporting, monitoring, and evaluation on this document were reported to Cabinet in July 2005. At the same time, the Cabinet Business Committee agreed that staff would prepare a further report by March 2006.
4.16
The Ministry of Economic Development reported regularly to the Minister of Energy. After the general election in 2005, the Government announced its intention to write a national energy strategy. As a result, the reports scheduled for March 2006 were deferred till June 2006. As the Energy workstream was incorporated into the preparation of a national energy strategy, no further reports on this workstream were prepared.
Our views
4.17
Reporting to Ministers was done both informally and through written reports. Workstream leaders reported to their Ministers in writing regularly, although, in general, reporting was less frequent in the latter half of the Programme of Action.
4.18
Reporting to the public was done through the workstreams, by publishing a variety of information on workstream activities, as shown in Figure 8. We discuss how public information supports shared learning and capability building in paragraphs 4.31-4.47.
Figure 8
Public information provided by workstream leaders and the Department of the Prime Minister and Cabinet
Workstream leaders published a lot of information about their activities. This information included material on websites, consultation documents, and reports – for example:
Project information and updates were given to a variety of interested parties by workstream leaders and DPMC at a number of public presentations, such as conferences, public sector training programmes, and forums of groups with an interest in, or commitment to, sustainability. |
4.19
The regular reporting to local councils in the Auckland Programme provided opportunity for reflection and input by local government councillors and local authority chief executives.
Monitoring and evaluation
4.20
The Programme of Action sought to strengthen the way that central government departments work together. It also sought to have central government work in partnerships with relevant external agencies and, by implication, to work in an open way.
Our expectations
4.21
We expected that:
- there would be evaluations of how the Programme of Action and workstreams were implemented; and
- participating agencies would be involved in such evaluations.
What we found
4.22
The Programme of Action was under review and subject to adjustments by the Chief Executives Group throughout its three-year life. For example, in March 2004, chief executives were surveyed about their views on the implementation of the programme.
4.23
DPMC planned a final independent evaluation of the Programme of Action with the Project Leaders Group. The evaluation sought the views of stakeholders at a preparatory national workshop, and the final report Implications of the Sustainable Development Programme of Action was completed in October 2006.
4.24
Each workstream undertook a separate evaluation, often including stakeholder interviews. Participating agencies were involved in the evaluations as stakeholders. The results of these evaluations were included in the final report commissioned by DPMC.
4.25
The Auckland Programme carried out evaluations of projects as they progressed and at the end of the programme. The Auckland Programme’s Combined Steering Group prepared their evaluation methodology and the final report.
Our views
4.26
Various factors presented a challenge for evaluation and performance monitoring of the Programme of Action. These included the mix of activities, outputs, and outcomes in the Programme of Action, the new ways of working across agencies and with local government, and the difficulties associated with tackling complex policy issues.
4.27
As we have stated in earlier reports,1 evaluations are helped by early planning, which includes consideration of criteria for impact evaluation requirements in the future. In our view, programme planning that establishes short-term, mediumterm, and long-term goals makes it easier to perform the subsequent evaluations that assess whether goals have been achieved.
4.28
The outcomes from many of the projects in the Sustainable Cities and Energy workstreams will become visible only in the longer term – for example, outcomes from the New Zealand Urban Design Protocol project or the Regional Migrant Settlement and Sustainable Communities projects that formed part of the Auckland Programme.
4.29
Monitoring beyond 2006 will be important for assessing the longer-term outcomes to enable progress reporting to Ministers and the public, and to identify emerging issues.
Capability building
4.30
The Programme of Action was intended to help build experience and capability in applying the sustainable development approach to public policy. One important objective of the programme was to identify lessons to be applied in the future.
Our expectations
4.31
To support capability building, we expected to find that lessons learned were shared among workstream participants and stakeholders, as well as being recorded and available for the future.
What we found
4.32
The Quality Practice initiative sought to make all departments familiar with the aims of the Programme of Action, to create tools to support sustainable development thinking and practice in the workstreams, and to promote discussions and forums to share information. These were all intended to support the capability of the sector to work in a way that was consistent with sustainable development objectives (see paragraphs 2.59-2.64). Many of the forums involved departments updating each other on the progress in their workstream.
4.33
There were opportunities to expand participants’ understanding and application of sustainable development processes. These opportunities included meetings with departments, liaison with interest groups, invitations to speak at or attend conferences, staff seminars, and engagements with visiting speakers.
4.34
Substantial work went into preparing a public sustainable development website, but it was discontinued because the project leaders thought it would duplicate other websites that were being developed.
4.35
The Auckland Programme held workshops, set up learning groups, and established a website with links to other organisations.
Our views
4.36
Workstream leaders used a variety of opportunities to increase shared learning about sustainable development processes. These opportunities were also promoted through the Quality Practice initiative.
4.37
Material from presentations and seminars was shared by publication on the Sustainable Development intranet. However, as the intranet is no longer active and access was restricted to workstream participants, this process limited the potential for sharing the information with an even broader audience over time. We consider that the high turnover of staff and lack of documentation further limited the opportunities for ongoing shared learning.
4.38
In our view, shared learning and public sector capability is assisted not only by “learning by doing” and providing information through public presentations, but also through forms of public reporting and information sharing that captures the learning in a permanent and accessible form. Without such mechanisms, learning is likely to be restricted to the participants and the people they are in immediate contact with. Several people told us that the Programme of Action experience had beneficially influenced their way of working and therefore subsequent government programmes.
4.39
Shared ongoing mechanisms for reporting and sharing information would be consistent with open and transparent government, and with the Programme of Action principle of working in partnership and encouraging transparent and participatory processes.
4.40
In our view, given that two reports on the Programme of Action have not been publicly released, the potential envisaged for shared learning to contribute to building capability was not fully realised. Given the similar international initiatives and interest in sustainable development strategies and initiatives, such material could also be of value to others interested in sustainable development nationally and internationally.
4.41
Overseas reports on progress towards World Summit commitments and learning are readily available and often refer to learning from their experience. A 2004 progress report from the Netherlands on its Sustainable Development Action Plan refers to the Learning for Sustainability Programme, which provided support to government departments to put the plan into practice.
4.42
As part of strengthening the delivery of the United Kingdom Government’s Sustainable Development Strategy, sustainable development was intended, in 2005, to become part of the curriculum for the National School of Government.2 These and other reports on progress on World Summit commitments are readily available and are listed in Appendix 1.
Implications for cross-agency work
4.43
The implications for accountability arising out of our findings are that:
- public information supports capability building and accountability; and
- long-term initiatives need ongoing monitoring and evaluation.
Public information supports capability building and accountability
4.44
Public information enables the public, partners, participating agencies, and international agencies to assess progress against programme aims (accountability) and records lessons learned for the future (capability building).
4.45
Shared learning is supported by learning and reporting methods that can be informal or formal but that do need to be available in the public arena. This is important in the highly dynamic environment of the public service, particularly when a programme is seeking changes in the way the public service sets policy.
Long-term initiatives need ongoing monitoring and evaluation
4.46
Many outcomes from cross-agency work will become visible only in the longer term. Assessment of outcomes will be supported through evaluation of social, economic, environmental, and cultural effects. Ongoing monitoring is important. It provides the information for assessing outcomes and reporting results to Ministers and the public, and identifies emerging issues.
1: First Report for 2000: Health, School Boards, and Impact Evaluation, parliamentary paper B.29[00a], and Key Success Factors For Effective Co-ordination and Collaboration Between Public Sector Agencies, 2003.
2: HM Government (2005), The UK Government Sustainable Development Strategy, Department for Environment, Food and Rural Affairs, United Kingdom.
page top