Part 5: Statements of intent and our intentions with service performance information

Background

5.01
Our 2006 report on the results of central government audits noted that the quality of departmental 2005/06 Statements of Intent (SOIs) varied. We had observed only incremental change in the overall quality of SOIs since 2004/05.¹ Similarly, in 2007, we observed that there has been little development in departments’ 2006/07 SOIs compared with 2004/05 and 2005/06.²

5.02
We therefore indicated last year that our 2006/07 annual audits would emphasise service performance reporting - in particular, the appropriateness of service performance measures used in the statement of service performance (SSP). To do this, we reviewed 104 2007/08 SOIs for government departments and Crown entities. We provided feedback to entities and commentary on issues identified and on ways the auditors believed service performance information reported to Ministers and select committees should improve. We also provided feedback to central agencies and monitoring departments where relevant.

Conclusions on 2007/08 Statements of Intent

5.03
Overall, we were disappointed that many entities’ service performance information did not, in our view, set out coherent performance frameworks showing logical links from the information about the medium-term outcomes sought by the entity to the annual outputs (goods and services) delivered by the entity. Many SOIs did not have well-specified, relevant performance measures and standards for both the medium-term and SSP information.

5.04
We were concerned about the weak links of the medium-term contextual and strategic information to the forecast SSP. These links should clearly set out the rationale for the outputs and identify key dimensions of service performance for each output. The relevance of performance measures and standards, and subsequent achievements against standards, can be assessed only in the context of the entity’s operating environment and strategic direction. Therefore, a logical link between strategy and service delivery is vital not simply for external accountability but, more importantly, for management evaluation and future service planning.

5.05
We were concerned about the issues of identification and specification noted for both output information and medium-term achievement. In our view, as for financial reporting standards, if the underlying elements of the SSP are not properly identified and treated, the basis of the reporting is undermined for external accountability and for the usefulness and relevance of information for management and business improvement purposes.

5.06
We were also concerned about the lack in many instances of robust, best estimate-based standards combined with historical or benchmark information that gives context to the anticipated achievement.

5.07
In our view, service performance information should reflect good management practice. It should clearly articulate strategy, link strategy to operational and other business plans, monitor the delivery of operational and business planning, and evaluate strategy impacts and results.

5.08
We regard improving the state of information about public sector entities’ performance as crucial not just to demonstrating accountability but to achieving continuous improvement in public sector effectiveness.

5.09
Many Crown entities were required to prepare SOIs under the Crown Entities Act 2004 for the first time in 2006/07, and it is therefore likely that these entities are still going through a learning process. Government departments, on the other hand, have been required to prepare the information currently required by the Public Finance Act 1989 since 2004/05. Before the 2004 Public Finance Act amendments, government departments had to prepare SOIs under Cabinet direction with similar requirements.

5.10
Despite the greater experience of government departments, their 2007/08 SOIs continue to be of variable quality. Improvements in quality since the 2004/05 SOIs have still been only small and incremental. We have also previously reported that, under our annual assessments of information systems and controls in government departments, the Service Performance Information Systems aspect³ consistently had the lowest proportion of "Excellent" or "Good" ratings between 1993/94 and 2005/06 (compared to the other aspects rated).⁴

5.11
Non-financial performance reports are essential documents for ensuring that government departments and Crown entities are held accountable to Parliament and the public. If Parliament is unable to adequately assess entity performance because of the poor quality of performance reporting, then we would expect those entities and the agencies that monitor them to be held accountable for their inadequate reporting. In our view, for the public sector to demonstrate accountability to Parliament, the quality of non-financial performance reporting needs to be significantly improved. It is a significant weakness in improving the effectiveness of the public sector.

5.12
In 2007, the Treasury carried out a review of accountability documents at the Government’s request and in consultation with Parliament, with the objectives of increasing the documents’ usability and reducing duplication. The review:

• has resulted in structural changes (involving changes to the format of documents and the relocation of information from departmental SOIs to the annual Estimates of Appropriation); and
• over time, proposes to improve the quality of service performance information, including through inter-departmental peer review, although to date this work has primarily focused on improving appropriation scope statements.

5.13
We are concerned that an undue focus on structural change in 2007/08 could displace effort that might have been directed to improving the quality of information. We urge both entities and central agencies to pay attention to the quality and the substance of information that appears in both forecast and annual reports as well as to its presentation or form.

5.14
In our view, enduring improvement in performance information will require clear and consistent policy objectives, strong central co-ordination and direction, well-established good management practices, and unwavering willingness to be accountable for results.

5.15
As long as the weaknesses in information persist, parliamentarians can have only limited assurance that the performance information of public entities reflects the purpose and effectiveness of their endeavours. While there are some good examples, these are few and far between.

Our detailed findings on 2007/08 Statements of Intent

5.16
We reviewed in depth 104 2007/08 SOIs, or 81% of the 128 SOIs required to be prepared. We requested and reviewed many of these in draft form. If we did not receive SOIs in enough time to provide feedback on a draft, we reviewed the final SOI. Twenty-eight percent of the SOIs we reviewed were for government departments.

5.17
Figure 3 sets out our findings on the 2007/08 SOIs we reviewed.

5.18
Appendix 1 sets out further information about our expectations leading into our examination of the 2007/08 SOIs and the basis for those expectations.

Figure 3
Our findings on the 2007/08 Statements of Intent we reviewed

Our expectations	Our findings
Medium-term component of the SOI
Clearly identified outcomes, which provide the the context for the entity’s role and functions.	Over 15% of SOIs had shortcomings in specification of outcomes.
Supporting discussion on the entity’s role, functions, strategic priorities, challenges, risks.	About half of SOIs were well presented and "readable". Over 40% included useful discussion and contextual information in the medium-term component of the SOI.
Main measures and standards for outcomes, objectives, or impacts are clearly specified, cover a period of three years, and provide baseline data that places measures and standards in a more meaningful context and allows progress to be tracked.	Nearly a third of the SOIs had missing or unclear main measures, and another third needed to improve their main measures. Many SOIs would benefit by adding baseline data about the current state of outcomes, objectives, or impacts, and their associated measures.
Link	Over 50% of SOIs could improve the structure of the forecast SSP and its links to the medium term component of the SOI. Weaknesses in the links ranged from minor to more significant for example, from suggestions about clarifying layout or the use of diagrams to more significant issues that made links difficult to assess, such as a lack of discussion about how outputs contributed to outcomes.
A coherent structure and integrated contextual information that makes evident, through linking within and between the information in the two components: the reasons for the entity’s outputs; and the focus of its reporting, including the rationale for, and the relationships among, the elements, performance measures, and standards.
Forecast SSP
Logically aggregated output classes/outputs with clearly specified outputs that focus on external impacts.	We had queries about the basis for the identification and aggregation of output classes, and noted that outputs were missing, incomplete, or not well specified, to a varying degree, for nearly 40% of SOIs.
Clearly specified performance measures and standards that are relevant and balanced, and provide baseline data that places measures and standards in a meaningful context and allows progress to be tracked.	About 60% of the forecast SSPs had shortcomings in the range and coverage of performance measures and the specification of standards. Measures of output quality, in particular, need enhancing. Many SOIs would benefit from the addition of baseline data about current and recent achievement for output delivery.

Our intended work on service performance information in 2008/09

5.19
In the coming year, we intend to maintain our focus on service performance information. In our report last year, Central government: Results of the 2005/06 audits, we advised that, from the 2007/08 annual audits, our reporting to Ministers and select committees would include a grading for entities on their service performance aspect. We have reconsidered our intention to assign grades from 2007/08 audits in the light of the following factors:

• The results from our 2007/08 reviews of SOIs indicate that there is still considerable development work to be carried out by entities.
• The structural and non-structural changes arising from the Treasury’s Review of Accountability Documents will require additional effort to adjust the presentation of information, particularly for government departments. Nonstructural changes will not receive the level of effort we know is needed to achieve the improvements needed.
• We have been reviewing and updating our own audit methodology and standards for statement of service performance information to ensure that these also take account of the changes in the statutory requirements.

5.20
Therefore, we have concluded that we will defer grading the service performance aspect in our reporting to Ministers and select committees until the 2008/09 audits. Our 2007/08 audit reports to Ministers and select committees will continue to provide only commentary.

5.21
We will once again carry out a concentrated review of finalised 2008/09 forecast non-financial performance information during 2007/08 audits to provide suggestions to entities on how they can improve their preparation of 2009/10 forecast non-financial performance information. Because of the work programme associated with the structural changes from the Review of Accountability Documents, we intend to review final forecast non-financial performance information presented to Parliament (rather than the draft information as we did for many 2007/08 SOIs). Entities seeking feedback on draft performance information from their auditor will be provided with comments on the extent of their improvement in addressing issues noted on their 2007/08 SOIs. Our review of final information will provide more extensive feedback.

5.22
We will consider over the 2008/09 year, and in tandem with the efforts of central agencies as they implement the Review of Accountability Documents, how we can support greater improvement in the preparation and disclosure of performance information. This consideration will relate to both the audit work we do and the way in which we report this work to entities, Parliament, and others.

1: Central government: Results of the 2004-05 audits, parliamentary paper B.29[06a], page 74.

2: Central government: Results of the 2005/06 audits, parliamentary paper B.29[07a], page 79.

3: “Service performance information systems” are the systems to record service performance (non-financial) data, and the internal controls (manual and computer-based) to ensure that data is complete and accurate.

4: Central Government: Results of the 2005/06 audits, parliamentary paper B.29[07a], “Part 2 - Government departments - results of the 2005/06 audits”.