Part 5: How performance was measured and reported

Canterbury Earthquake Recovery Authority: Assessing its effectiveness and efficiency.

5.1
In this Part, we consider:

Summary of our findings

5.2
International experience has shown the importance of transparency and accountability for an effective recovery. A clear and well-articulated performance management framework is one of the most effective means for a public entity to be transparent and accountable for its performance.

5.3
CERA did not measure and externally report on its performance effectively. This has made it difficult to assess how effective and efficient CERA was based on its publicly available performance information. It also meant that CERA could not demonstrate its value for money to Parliament and the public.

5.4
CERA's annual performance reporting forms part of monitoring and evaluating the progress of the recovery. However, it was the primary way that CERA could publicly show how its activities had helped achieve the outcomes of the recovery.

Measures to provide a public account of performance

5.5
Under the Public Finance Act 1989, CERA was required to externally report on its performance in its annual reports, as well as provide performance information in Budget documents (the Estimates) and in its Statement of Intent.

5.6
In our annual audits of CERA, we found that CERA needed to improve the way it measured and reported on its performance. Many of CERA's measures for reporting on the effect of its work were measures of activity (such as reporting the number of meetings) rather than the effect on community well-being or the rebuild of the central city.

5.7
Our recommendations to CERA on improving performance information were to:

  • establish measures for what CERA intended to achieve from its activity;
  • have more specific targets;
  • have more consistency in performance measures so performance could be tracked over time; and
  • improve the links and consistency of performance information between accountability documents (between measures reported in the Annual Report, Statement of Intent, and Budget documents).

5.8
CERA's Audit and Risk Committee also reported to management that CERA needed more reliable and stable performance measures.

5.9
CERA did not have a clear view of what it would achieve by the end of its five-year term. It reported annually on the progress of the recovery, with outcome measures for community well-being, economic environment, and the built environment. However, it did not link its own activity with these outcome measures, so it is hard to tell how effective CERA was in influencing them.

5.10
This meant that CERA was not able to clearly demonstrate its value for money during its term, and some of its key achievements, such as the implementation of the policies for the Red Zones and the demolition programme, were not well communicated to the public.

Assessing effectiveness from the publicly reported information

5.11
Despite the shortcomings in CERA's reported performance information, we have analysed CERA's performance against the performance measures set out in CERA's statement of intent and Budget documents (in the annual estimates of appropriations). Because most of the performance measures changed every year, we have put them into three different groups and analysed CERA's performance in achieving them. The three groups are:

  • Built and natural hazards: consisting of measures that we determined relate to the Red Zones, demolitions, Port Hills, land zoning, and insurance recoveries. For example, complete clearance of all Crown-owned dwellings on the flat lands Red Zones by 31 December 2014.
  • Anchor Projects: consisting of measures that we determined relate to the Anchor Projects. For example, the preparation of business cases to support the development of the Anchor Projects.
  • Other: consisting of measures that did not fit into the two categories above. These related to a range of CERA's work, including monitoring the recovery, psychosocial recovery, and the Minister's satisfaction with policy advice from CERA.

5.12
CERA met about 80% of its performance targets in 2011/12 and 2012/13. However, its performance declined in 2013/14, 2014/15, and 2015/16 when it met about 70% of its performance targets.

5.13
Figure 15 shows CERA's performance in each year according to its own performance measures. In 2011/12, CERA met about 40% of its built and natural hazards performance targets. Achievement of these targets then increased to about 70% in 2012/13 and 2013/14, before declining to about 40% again in 2014/15, and increasing to just over 90% in 2015/16.

5.14
After targets were introduced for the Anchor Projects in 2012/13, CERA met 100% of them. Achievement of these targets then declined to about 60% in 2013/14, before increasing to about 80% in 2014/15, and then declining to 60% in 2015/16.

Figure 15
Achievement of performance targets, by group and year, 2011/12 to 2015/16

Figure 5 - Achievement of performance targets, by group and year, 2011/12 to 2015/16.

5.15
This analysis does not tell the full story of how effective CERA was in supporting the recovery from the Canterbury earthquakes. Each year, we reported that there were limitations in CERA's performance framework, including that the performance measures tended to focus on activities rather than what CERA was trying to achieve from those activities.

5.16
In other instances, some measures reported on how the recovery was progressing (such as community well-being and economic performance of the region). However, the role that CERA had played in influencing these outcomes was not clearly stated.

5.17
CERA regularly changed its performance measures, and did not have a consistent performance framework. It changed its measures in response to its changing role, and the different roles and responsibilities it was taking on. However, CERA was not consistent in how it reported on some of its performance measures, which made it difficult to assess trends in its performance.

5.18
Because of the deficiencies in how CERA measured and reported on its performance, it was not possible to determine how effective it was based on its performance reporting. Our assessment needed to be based on evidence from external reviews of CERA, our audit and performance audit work, and interviews with former CERA staff and stakeholders.

5.19
CERA could have learned from other disaster recovery agencies about how to effectively report on its performance and its value for money. For example, the World Bank reported favourably about how the Queensland Recovery Authority (established to co-ordinate the recovery from the Queensland floods in 2010 and 2011) measured its performance. The Queensland Recovery Authority's performance framework focused on a value-for-money framework that included regular reporting on project and programme objectives, costs, scale, and complexity, as well as progress.73

Lessons for the future

5.20
Good performance reporting would not only have helped with CERA's accountability to the public, it also would have helped CERA to adapt to the different phases of recovery and build and maintain the trust of the community. In our view:

  • An effective performance framework needs to link recovery activities to the desired outcomes. Effective performance reporting gives recovery agencies the opportunity to adjust their processes to achieve the desired outcomes. The World Bank has identified that a performance framework (also called a results framework) needs to harmonise and integrate all the strategic pillars and areas of a given reconstruction programme. The framework would establish a streamlined results chain by focusing on key results, and measuring intermediate outcomes (contributions made to an outcome by a specific set of outputs) rather than outputs. Intermediate outcomes need be carefully designed to capture and to track intended changes as they begin to unfold.

73: World Bank (2011), Queensland: Recovery and Reconstruction in the Aftermath of the 2010/11 Flood Events and Cyclone Yasi, Washington.