Part 9: Improving service performance reporting

9.1
In this Part, we discuss our work to help public entities improve their performance information and reporting. We note how the Auditor-General's revised auditing standard has contributed to improving performance reporting and review our recent work in this area. Our efforts will continue as more public entities will be subject to the revised auditing standard in 2011/12 and 2012/13.

Why performance information and reporting is important

9.2
Performance reporting is an integral part of our parliamentary accountability system, enabling Parliament to hold Ministers and central government entities to account. Parliament is required to make decisions about the funds to be voted through the annual Budget process and to review the performance of the entities that have used those funds.

9.3
Accountability documents such as the annual report and the Statement of Intent (SOI) help ensure that government departments and Crown entities can be held to account by Parliament and the public. Parliament and the public rely on accountability documents to assess the performance of public entities and how effectively they use public funds.

9.4
Performance reports are an essential part of accountability documents. We consider that improving performance information and reporting is crucial in helping to improve the effectiveness and efficiency of public sector entities and to show accountability for their performance. The Auditor-General is keen to see continued improvement in this area.

9.5
If public entities are clear on why they exist, what changes they are seeking to influence, and how their services contribute to those desired changes, then they can build a sound framework for planning, measuring, managing, and reporting their performance. When public entities have good performance systems for running their organisations and making informed decisions, they should more easily produce external accountability reports that are relevant and reliable and in keeping with legislation and generally accepted accounting practice.

Issues about the quality of performance reporting

9.6
In the past several years, we have reported our view that the quality of performance reporting has needed to improve. Until recently, we have observed that in central government, entities have had major deficiencies in their performance reporting, including failures to:

• clearly explain what they are trying to achieve;
• clearly explain the services they are accountable for providing;
• present a framework that illustrates the cause-and-effect relationships and assumptions underpinning the spending of taxpayer funds on public services to achieve public benefit;
• report targets and results through a well-rounded suite of appropriate performance measures; and
• report externally in a way that is integrated with the information they use to manage their business.

9.7
We have continued to work with entities and with central agencies to help ensure that the above issues are addressed.

9.8
Now, after witnessing minimal improvement in the 20 years since the Public Finance Act 1989 was enacted, we have begun to see, in the last two years, government departments and Crown entities improve the quality of their performance reports.

Phasing in our revised auditing standard

9.9
We are phasing in a revised standard for auditing service performance reports. Our revised standard requires our auditors to modify their audit opinion if the performance information in the annual report does not, in their opinion, fairly reflect performance for the year. Under the previous auditing standard, there was very little scope for appointed auditors to modify their audit opinion if the content of the performance report was inadequate.

9.10
Before implementing our revised auditing standard, we have endeavoured to prompt and help public entities raise the standard of their performance reporting through our support initiatives, combined with regular evaluation and feedback to entities from appointed auditors. We provided advance notice about the new auditing standard to allow time and to provide support to entities to upskill and improve their performance information and reporting. During this period, we placed greater emphasis on performance information in the audit, provided clear signals and specific advice to entities about continual improvement in this area, and provided guidance and other support to help them meet the challenge. We have supported public entities by carrying out document reviews, one-on-one conversations, various guidance and advice, seminars, workshops, and other forms of support to help them improve their performance information and reporting.

9.11
To help make the improvement programme manageable and to provide a concentrated effort on our support initiatives, we phased in the application of the standard: 28 larger and more complex departments and Crown entities were audited under the revised standard for 2010/11. We will apply the revised standard in our audits of another group of entities for 2011/12, and the rest of the entities in 2012/13.

Audit reports for 2010/11

9.12
All entities being audited under the revised standard reported sufficiently well in their 2010/11 annual reports that we issued an unmodified audit opinion for each entity. A pleasing result indeed.

9.13
In Figure 12, we list the departments and Crown entities that were audited under the revised auditing standard for 2010/11.

Figure 12
Entities audited under the revised auditing standard for 2010/11

9.14
During the last year, our appointed auditors maintained contact with these entities, offered advice on critical deficiencies, monitored progress, and reviewed draft performance frameworks and accountability documents. Entities have responded well and have made significant improvements to their performance frameworks, choice of performance measures, and presentation of those frameworks and measures in their annual reports.

9.15
One year ago, we estimated that 15 of these entities appeared to be at risk of receiving a modified audit opinion if they did not make significant improvements to their performance reporting in 2011. As improvements appeared to be taking effect in several entities, we were able to refocus our attention during the year from the 15 entities initially identified as problematic to only eight "at risk" entities, which we monitored closely in the period leading up to the production of their annual reports. Since then, entities have made considerable effort to improve performance measurement systems and reports.

Assessing the quality of performance reporting

9.16
As well as attesting to an entity's service performance in their audit reports, our appointed auditors also provide advice to entities on where improvements are needed, together with a grade for service performance information and its associated systems and controls (see Part 6).

9.17
The grade for the service performance aspect can be either "Poor", "Needs improvement", "Good", or "Very good". (See Figure 2, Part 6, for explanations of these grades.)

9.18
There has been a marked improvement overall in the quality of performance reports of those entities being audited under the revised standard. During the last three years, since we have been issuing grades for service performance reporting, the number of these entities graded as "Good" has nearly trebled, and those graded as "Poor" or "Needs improvement" have halved. In the last year alone, more than one-third of these entities have improved their performance reporting grade. Some entities have also told us that they are finding the performance reporting discipline useful for internal management and decision-making purposes.

9.19
Our grade data indicate a clear improvement in the quality of performance reporting, although there remains much room for further improvement by several entities. While the signs of improvement are heartening, it appears that some entities are working on improving their SOIs and annual reports as external reports without, or before, giving attention to the performance objectives relating to their business strategy and operations. We expect public entities to measure and report performance first and foremost for strategic and operational management decision-making purposes, that is, for internal use. Ideally, external accountability documents should be derived from a robust internal performance management system. However, in some instances, the discipline being applied to the external reports is driving greater scrutiny of the entities' internal performance management frameworks, rather than the other way around.

9.20
As part of our audits, we continue to probe into whether the performance measures in the SOI and annual report are relevant to the internal management of the entity.

9.21
Figure 13 shows the spread of grades for 26³² of the original 28 entities (listed in Figure 12) since 2008/09, when we began issuing grades for service performance information and associated systems and controls. The graph tells a clear story: in the three years leading up to applying our revised auditing standard to these government departments and Crown entities:

• no entity has remained in the "Poor" category;
• the number of entities in the "Needs improvement" category has nearly halved;
• the number of entities whose service performance reports are considered "Good" has nearly trebled (and almost doubled in just the last year, with eight³³ entities moving from "Needs improvement" to "Good" in 2011); and
• no entity has yet reached the level of "Very good". (When an entity receives a "Very good" grade, it means that our appointed auditors cannot recommend any further cost-beneficial improvement.)

Figure 13
Trends from 2008/09 to 2010/11 in service performance grades for 26 of the 28 entities that were audited under the revised auditing standard in 2010/11

9.22
It is encouraging to see the extent of the improvement this year, which is possibly understated by the graph because several of those entities whose grades remained the same as last year have improved the standard of their reporting within the same grade band.

9.23
For an overview of the grades of all government departments, see Part 6.

Recent work and future intentions

Previous years

9.24
Since 2007, and in the lead-up to applying the revised auditing standard, we have been helping public entities improve the quality of their performance reporting. Our work programme has included:

• forums with entities, in which we outlined issues and our expectations regarding the preparation of performance reports; and
• intensive evaluation of accountability documents to provide feedback and advice to entities on the quality of performance reports.

9.25
In 2008, we issued a discussion paper, The Auditor-General's observations on the quality of performance reporting,³⁴ to explain the Auditor-General's expectations and to report on our assessment of the current state of performance reporting.

9.26
In 2008/09, after several years of providing feedback and guidance to entities, appointed auditors began issuing grades for service performance information and associated systems and controls to provide a high-level indicator to entities and to Parliament of our assessment of the quality of performance reports. In 2009, we issued another discussion paper, Statements of intent: Examples of reporting practice,³⁵ to share with entities some examples of better practice that we thought might help to improve performance reporting.

9.27
In 2009/10, in partnership with the Treasury, we ran a series of workshops for government departments and Crown entities to:

• identify issues and problems with performance reporting; and
• identify themes that might help us in solving problems and overcoming barriers.

9.28
Along with the Treasury, we also spent time providing one-on-one support and advice to the 28 entities in the first group to which our revised auditing standard was to apply, to help them with their improvement focus ahead of applying the standard.

Initiatives and observations during 2010/11

9.29
During 2010/11, we continued to provide workshops that reinforced our messages, explored further some of the barriers to good performance reporting, and explored suggestions for how those responsible for performance planning, measuring, and reporting could further improve the quality of their performance frameworks, systems, and reports.

9.30
While appointed auditors gave a concentrated effort to advising the larger entities on what was needed to ensure adequate performance reporting in their 2010/11 annual reports, we dedicated other staff resources to providing one-on-one assistance to a second group of entities to which we will apply our revised auditing standard in 2011/12.

9.31
We issued two general publications in 2010/11: Central government: Case studies in reporting forecast performance information³⁶ (February 2011) and Central government: Cost-effectiveness and improving annual reports³⁷ (June 2011) to stimulate thinking and discussion about preparing and using performance information. We also issued specific guidance to the health sector: District Health Boards: Learning from 2010-2013 Statements of Intent.³⁸

9.32
We have received feedback that suggests positive changes are taking place in central government entities regarding the measuring and reporting of performance. There appears to be a greater acceptance than there was a couple of years ago of the need for, and value of, robust reporting of performance, both internally and externally. It appears that greater improvement has taken place in those entities where commitment has come from the top. We have received several comments from chief executives and senior management on the value of performance reports, including comments that improving the performance framework for external accountability documents has helped improve internal performance planning, management, and communication.

9.33
Although public entities appear to more readily accept the need to measure and report their performance in non-financial terms, many advise us that they still find the task difficult. We hope our continuing work programme of support and advice will go some way toward helping these entities to overcome the difficulties in developing robust performance frameworks.

9.34
We also note that government departments and Crown entities are quite keen to explore the possibilities for sector-wide performance reporting, particularly for reporting progress towards the higher-level outcomes. Several sectors have begun developing sector outcomes frameworks that are aimed at providing a common framework for entities within a sector for reporting at the outcome level and reducing duplication of effort. See Part 10 for further discussion of this approach to reporting.

Future intentions

9.35
In 2012, our appointed auditors will concentrate efforts on monitoring the progress of the second group of entities to which we will apply our revised auditing standard as they prepare their 2011/12 annual reports.

9.36
We have begun a series of workshops to support the third group of entities to improve their performance information frameworks, systems, and reports as they prepare their 2012–2015 SOIs, which they will report against in 2012/13. We have also begun a pilot initiative to help this third group of entities establish a community of practice, including a website for sharing documents, providing tools, discussing issues, and providing advice.

9.37
We will follow up these support efforts in 2012/13 as our appointed auditors closely monitor the progress of the third group of entities, ahead of applying the revised auditing standard to their audits in 2013.

32: Two of the entities that we audited under the revised standard were disestablished in 2010/11 and therefore were not issued with grades.

33: One entity moved from "Good" to "Needs improvement".

34: Available on our website (www.oag.govt.nz).

35: Available on our website (www.oag.govt.nz).

36: Available on our website (www.oag.govt.nz).

37: Available on our website (www.oag.govt.nz).

38: Available on our website (www.oag.govt.nz).