Part 8: Statements of Intent and our intentions with performance information

8.1
In this Part, we describe the work we carried out in the last year to report on and improve the quality of service performance information.¹ As in 2006/07,² our work included reviewing the forecast performance information for government departments and selected Crown entities.

8.2
We also describe our intended work on performance information in 2009/10. For example, we intend to assess performance information and associated systems and controls in the 2008/09 audits and report our assessments to Ministers and select committees.

Background

8.3
Non-financial performance reports³ are essential for ensuring that government departments and Crown entities are held accountable to Parliament and the public. In 2008, the Auditor-General reported that, in his view, the overall poor quality of performance reporting by public entities is disappointing and needs to improve significantly.⁴ Improving the quality of information about the performance of public entities should help the public sector to demonstrate its accountability and to continuously improve its effectiveness.

8.4
In his 2008 report, the Auditor-General outlined his view of the issues that contribute to the poor quality of performance reporting and the reasons that the public accountability framework is not promoting improved reporting. These included:

• There are no reporting standards in New Zealand for non-financial performance reports. Therefore, every entity needs to prepare and customise its own framework and the elements (primarily outcomes and outputs) within that framework.
• There is no clear responsibility for professional leadership and oversight. Many parties have an interest in preparing and using performance reports.
• Relationships between outcomes and outputs are often not predictable or understood. These relationships and expectations change over time, with a range of circumstances and events influencing performance.
• Senior managers and governors do not always give external performance reports the attention they merit. Reports are not always aligned to the information governors and managers use to plan and monitor performance. If a public entity does not have a system for collecting and monitoring the range of information used for internal day-to-day management and governance, governors and managers cannot be confident that they are fulfilling their responsibilities.

8.5
The Auditor-General’s report concluded that, in his view, improving performance reporting will require deeper, more sustained, and focused attention. This attention should include:

• the Treasury – in consultation with entities with monitoring responsibilities and others with an interest in public sector management – facilitating the preparation of public sector standards for reporting, including considering how such standards are applied and maintained;
• identifying clearer responsibilities among central agencies, entities with monitoring responsibilities, and others interested in public sector management for co-ordination, leadership, and accountability for preparing and improving performance reporting within the public sector;
• central agencies and those with monitoring responsibilities giving consistent messages about the elements of performance reporting and their application, and considering whole-of-government and sector-level needs for information about outcomes, including how these can be identified, co-ordinated, collected, and reported; and
• public entities considering the identifiable users of external performance reports and their needs, so that public sector accountability regimes can better accommodate both internal and external uses of reported information.

Reviews of Statements of Intent

8.6
Since 2005, our reports on the results of central government audits have noted that the quality of government departments’ Statements of Intent (SOIs)⁵ varies. We have seen only a small improvement in the overall quality of SOIs since 2004/05.

8.7
We have similar concerns about the quality of Crown entities’ information and the lack of clear improvement in the information during 2006-08. However, many Crown entities were required to prepare SOIs under the Crown Entities Act 2004 for the first time in 2006/07. These entities will have been going through a learning process.

8.8
In our report on the results of the 2006/07 audits, we advised that we had decided not to assign grades for the performance aspect within our reporting to Ministers and select committees on the 2007/08 annual audits.⁶ This was the second year in which we did not assign grades on the performance aspect in our reporting. We decided not to assign grades because:

• The results from our 2007/08 reviews of SOIs indicated that entities still had to carry out considerable development work.
• We were reviewing and updating our audit methodology and standards for statement of service performance (SSP) information to ensure that these take account of changes in statutory requirements and other guidance.
• The structural and non-structural changes arising from the Treasury’s Review of Accountability Documents (ROADs) would require additional effort to adjust the presentation of information, particularly for government departments. Therefore, we anticipated that non-structural changes would not receive the level of effort that, in our view, is needed to achieve the improvements.

8.9
The purpose of our reviews, which we carried out for entities’ 2007-10⁷ and 2008-11 SOIs, was to suggest to entities how they could improve their forecast non-financial performance information in future. We commented in our reports to Ministers and select committees on the issues we identified and on the ways our auditors considered performance information should improve. We aligned our expectations on these matters with legislation and the guidance and instructions from the State Services Commission (SSC) and the Treasury.

Overall conclusions from our reviews

8.10
In 2007/08, we reviewed in depth 125 of the 127 SOIs that government departments and Crown entities are required to prepare, where the Auditor-General is required to attest to the SSP in the entity’s annual audit report. Two entities had either not prepared an SOI or had not gained Ministerial approval for their SOI at the time this article was prepared. Of the SOIs we reviewed, 30% were for government departments.

8.11
Our overall findings are similar to those in our report on the results of the 2006/07 audits. Again:

• Many entities’ performance information did not, in our view, set out coherent performance frameworks showing logical links from the information about the medium-term outcomes sought by the entity to the annual outputs (goods and services) delivered by the entity.
• Many SOIs did not have well-specified, relevant performance measures and standards for both the medium-term and SSP information.
• In many instances, there was a lack of robust, best estimate-based standards combined with historical or benchmark information that gives context to the anticipated achievement.

8.12
In our view, performance information should reflect good management practice. It should clearly articulate strategy, link that strategy to operational and other business plans, and be used to monitor the delivery of operational and business planning, and to evaluate the strategy’s effects.

8.13
We have now provided detailed feedback on most government departments’ and Crown entities’ SOIs for two years. In our view, this timeframe should have provided a reasonable opportunity for entities to improve their reporting. We will begin grading and reporting on the quality of performance information in the SOIs during the 2008/09 audits.

Our detailed findings on the 2008-11 Statements of Intent

8.14
Our overall conclusion is that the quality of SOIs has not improved between our reviews of the 2007-10 SOIs and the 2008-11 SOIs. Although there appears to have been deterioration in some areas we assessed, undue emphasis should not be placed on our detailed findings, as performance information is by its nature more subjective than financial information.

8.15
For the 2007-10 SOIs, we were developing our audit methodology and standards. For the 2008-11 SOIs, we were more experienced in assessing the information. We also anticipated that entities would have reviewed our suggestions for improvement on their 2007-10 performance information and incorporated them into their 2008-11 SOIs.

8.16
With the significant ROADs changes being largely implemented as part of the 2008 Budget, the SSC and the Treasury are moving their emphasis to the quality of performance information. Our three agencies are now working on a broader and long-term programme to improve the quality of this information. As a result, we expect to see evidence of improvement in the 2009-12 SOIs.

8.17
There are a handful of areas within our reviews in which the results for Crown entities are more positive than those for government departments. Our assessments may reflect that some government departments have a range of roles and outputs, and may not have a single organisational purpose (by contrast with many Crown entities). This makes a coherent and simple account of their performance intentions more difficult to prepare. ROADs may also have affected the 2008-11 SOIs and led to government departments deferring improvements they may have been planning to make. The Treasury has completed a review of the implementation of ROADs as part of the 2008 Budget Reviews and has issued guidance for the 2009 Budget, including guidance for 2009-12 SOIs. The guidance includes addressing problems in preparing information that were identified through the implementation review.

8.18
Figure 20 sets out our expectations and our assessment of the 2008-11 SOIs we reviewed, and compares these results with those from our reviews of the 2007-10 SOIs.

Figure 20
Our assessment of the 2008-11 Statements of Intent we reviewed

Our expectations	Assessment of 2008-11 SOIs	Assessment of 2007-10 SOIs
Medium-term component of the SOI
Clearly identified outcomes, which provide context for the entity’s role and functions.	About 25% of SOIs had shortcomings in specification of outcomes. Shortcomings varied from the complete absence of outcome information, to information that was difficult to locate or identify, to the substitution of outcome information with internal organisational goals, and to an absence of clarity about the societal change sought.	Over 15% of SOIs had shortcomings in specification of outcomes.
Main measures and standards for outcomes, objectives, or impacts are clearly specified, cover a period of three years, and provide baseline data that places measures and standards in a more meaningful context and allows progress to be tracked.	Nearly 70% of SOIs had missing or unclear main measures or deficiencies in the specification of standards for their main measures. For about 40% of SOIs, we had concerns about the strength or the logic of the relationship between the main measures and standards and what they were intended to measure. As in 2007, many SOIs would benefit by adding baseline data about current state information. In particular, baseline data about the current state of outcomes and intended impacts of outputs is required to allow progress to be tracked over time and achievement evaluated.	Nearly a third of the SOIs had missing or unclear main measures, and another third needed to improve their main measures (in total nearly 70%). Many SOIs would benefit by adding baseline data about the current state of outcomes, objectives, or impacts, and their associated measures.
	There was a small difference in our assessments between SOIs of government departments and Crown entities in this area. We assessed a greater proportion of government departments as having deficiencies in their main measures and standards compared with Crown entities, and as being more likely to have measures and standards that we were not sure measured what they were intended to.
	Over 50% of SOIs could improve the structure of the forecast SSP and its links to the medium-term component of the SOI. As in 2007, weaknesses in the links ranged from minor to more significant – for example, from the need to clarify layout or the use of diagrams to more significant issues that made links difficult to assess, such as a lack of discussion about how outputs contributed to outcomes. For government departments, the separation of some outcome and output information into the Information Supporting the Estimates with the remainder of the information remaining in the SOI appeared to have a slight impact. There was a small negative change from our 2007 reviews in our assessments of SOIs for government department in the link between medium-term and annual information.	Over 50% of SOIs could improve the structure of the forecast SSP and its links to the medium-term component of the SOI. Weaknesses in the links ranged from minor to more significant – for example, from the need to clarify layout or the use of diagrams to more significant issues that made links difficult to assess, such as a lack of discussion about how outputs contributed to outcomes.
Description of the output classes and outputs and how the SSP links to the medium term and makes evident the reasons for the entity’s outputs and the focus of its reporting.
Forecast SSP
Logically aggregated output classes and outputs, with clearly specified outputs that focus on external impacts.	We had queries about the basis for the identification and aggregation of output classes, and noted that outputs were missing, incomplete, or not well specified, to a varying degree, for nearly 30% of SOIs. We assessed a greater proportion of Crown entities as having deficiencies in their output aggregation compared with government departments.	We had queries about the basis for the identification and aggregation of output classes, and noted that outputs were missing, incomplete, or not well specified, to a varying degree, for nearly 40% of SOIs.
Our expectations	Assessment of 2008-11 SOIs	Assessment of 2007-10 SOIs
	While not specifically included within our reviews, we also noted instances where Crown entities’ financial information was not presented using the same groupings as that for performance information or where the expenditure presented in performance information did not total to the expenditure in the financial forecasts.
Clearly specified performance measures and standards that are relevant and balanced, and provide baseline data that places measures and standards in a meaningful context and allows progress to be tracked.	About 75% of the forecast SSPs had shortcomings in the range and coverage of performance measures and the specification of standards. As in 2007, measures of output quality, in particular, need enhancing. Many SOIs would benefit from the addition of baseline data about current and recent achievements for output delivery. We assessed a greater proportion of government departments as having output measures and standards that we were not sure were relevant to or measured the output identified.	About 60% of the forecast SSPs had shortcomings in the range and coverage of performance measures and the specification of standards. Measures of output quality, in particular, need enhancing. Many SOIs would benefit from the addition of baseline data about current and recent achievement for output delivery.

Our intended work on performance information in 2009/10

8.19
Because of our concerns about the current quality of performance information, we have been reviewing and updating our own audit methodology and standards for performance information. Currently, auditors verify the accuracy of entities’ SSP information against the forecast statements.

8.20
In December 2008, we issued a consultation draft of the Auditor-General’s revised auditing standard on performance information (AG-4),⁸ which is available on our website – www.oag.govt.nz. We expect to adopt the revised AG-4 in mid-2009. It will require auditors to report on whether the performance information:

• provides an adequate basis for the assessment of performance; and
• fairly reflects the entity’s performance.

8.21
Our revised AG-4 will be phased into our audit reporting. First we will assign grades for performance information and associated systems and controls in the reports to Ministers and select committees. This is intended to allow the entities and our auditors to prepare and adjust before the revised AG-4 is applied to our audit opinions. Therefore, we intend to:

• assign grades for performance information and associated systems and controls that will include our assessment of the quality of the forecast SOIs under our revised standard beginning with the reports on the 2008/09 and 2009/10 audits; and
• apply the revised standard to 2010/11 audit opinions.

8.22
Ongoing improvement in performance information will require an environment of clear and consistent policy objectives, strong central co-ordination and direction, well-established good management practices, and unwavering willingness to be accountable for results. We expect our intended work on performance information during 2009/10 and beyond, including our work toward greater co-operation and consistency with the Treasury and the SSC, to help provide this environment.

1: Service performance information provides primarily non-financial information that records the output delivery performance of a public entity against specified objectives. This information is usually shown in statements of service performance (or equivalent reports) and is compared with information contained in forecast non-financial performance reports. For ease of reading, we use the term “performance information”.

2: See Central government: Results of the 2006/07 audits for the results of our 2006/07 review.

3 Non-financial performance reports provide primarily non-financial information that records the performance of a public entity against specified objectives. They can encompass a comprehensive range of performance elements (including outcomes, outputs, inputs, and capability), and the information can be presented in various statements.

4: See The Auditor-General’s observations on the quality of performance reporting.

5: As a result of the Review of Accountability Documents for government departments in 2008, this information was contained in the 2008-11 Statements of Intent and the Budget 2008 Information Supporting the Estimates.

6: Central government: Results of the 2006/07 audits, page 41.

7: We reported the results of these reviews in Central government: Results of the 2006/07 audits, page 41.

8: The revised AG-4 is intended to apply to those local authorities, government departments, and Crown entities required to prepare an SOI and SSP under sections 139 and 150 of the Crown Entities Act 2004. This excludes the audit of performance reports of other Crown entities (such as tertiary education institutions and those Crown entities required to prepare and report against a statement of corporate intent) whose performance reporting requirements are governed by other legislation.