Appendix 2: What auditors considered when grading service performance information
Central government: Case studies in reporting forecast performance information.
We produced the following guidance for public entities to help them understand the approach the auditor would take to reviewing forecast non-financial performance information reports. This review work helps the auditor form a view about the appropriateness of the forecast SSP as a basis for reporting.
Although this guidance aims to describe the aspects that we expect to see reflected in the accountability documents, we recognise that public entities and their auditors may identify other aspects that need further consideration.
| Matters to consider | Why important? |
|---|---|
| 1. Performance framework/story | |
| a) The document should provide the reader with a clear picture of what the public entity is trying to achieve and how it considers it contributes to this. | For the document to be used to hold a public entity to account, it is important that the reader can understand what the public entity is trying to achieve and how its activities contribute to this. |
| b) Outcomes should be identified at an appropriate level. Where high-level outcomes are identified, these should be supported by lower-level outcomes and impacts as appropriate. | Outcomes need to be specified at a level that the public entity can reasonably aim to contribute to or influence. |
| c) The document should provide the reader with a clear picture of the outputs (services) that the public entity is accountable for. | The outputs of the public entity help the reader to understand what the public entity actually delivers through the application of its resources. |
| d) The relationship between the outputs of the public entity, the impact it is intended to have, and the outcomes that it is seeking to contribute to or influence should be clearly explained to the reader. This may require a number of layers in the intervention logic to be stated. | Clearly explaining the intervention logic helps the reader to understand how the public entity considers that the goods and services it is accountable for will ultimately result in improved outcomes. |
| e) The SOI and the forecast SSP/Information Supporting the Estimates should reflect the public entity's performance management arrangements. | To fairly reflect the public entity's service performance for the year, the SSP must be consistent with the internal performance management arrangements. Given that the SSP also needs to be consistent with the forecast SSP, it follows that the Forecast SSP and SOI should also reflect a public entity's performance management arrangements. |
| 2. Outcomes | |
| a) Outcomes, which should relate to a state or condition of society, the economy, or the environment, and including a change in that state or condition, should be identified in the document. | The SOI is required to identify the outcomes that the public entity is aiming to contribute to or influence. |
| b) Outcomes should not include objectives for organisational improvement. | It is clearly important for public entities to look to improve their organisational capacity and capability and to specify objectives for these. However, "outcomes" should relate to the public entity's purpose and strategy – that is, the end to which any organisational improvement is to be used. |
| c) Outcomes should be immediately clear to the reader. | The outcomes should not be "lost" in the document among priorities, goals, strategic aims, objectives, and other material. |
| d) Outcomes should be phrased appropriately. | Outcomes should relate to a state or condition of society, the economy, or the environment; and include a change in that state or condition. |
| e) Outcomes should be supported by measures. | Without measures, it will be difficult for the public entity and reader to know that progress is being made toward the outcomes. |
| f) Targets for outcome measures (covering the full period of the SOI) should be included in the document. | Clear targets for outcomes, which must cover the full period to which the SOI relates, will allow the public entity and reader to understand what the public entity is seeking to achieve and reach a view about whether enough progress is being made. |
| g) The inclusion of current/historical performance for outcome (main) measures provides helpful context to the reader about outcomes. | Current/historical performance information provides useful context, which can be used to help understand targets and future performance. |
| h) The inclusion of comparative data from other organisations and/or countries provides helpful context to the reader about outcomes. | Comparative information from other organisations or countries helps the reader to understand the level of outcomes which the public entity is seeking to influence. |
| 3. Output classes | |
| a) The output classes used for financial and non-financial information should be consistent, or a reconciliation of the two sets of information should be provided. | For the reader to reach a view about a public entity's planned performance, it needs to be possible to reconcile the non-financial performance forecasts with the financial forecasts. |
| b) The output classes referred to in the medium-term component of the document should be consistent with those used in the forecast SSP to group outputs. | Consistent use of output classes allows the public entity to more easily set out its performance framework and to demonstrate the links between outputs, impacts, and outcomes. |
| c) Output classes, and the number of them, should be appropriate to the public entity. | The use of output classes allows the public entity to aggregate its services in such a way that the reader is provided with enough information about the activities of the whole public entity without being burdened with too much detail. |
| 4. Outputs | |
| a) Outputs, which should relate to the goods and services provided to third parties that the public entity is accountable for, should be included in the document. | The forecast SSP is required to include the outputs that the public entity is accountable for delivering. |
| b) The outputs included in the document should cover all "significant" services of the public entity and a significant proportion of the public entity's budget. | The SOI should provide the reader with a comprehensive picture of a public entity's activities. To do this, it is important that all of the major goods and services provided are covered by outputs in the forecast SSP. This can be demonstrated by ensuring that the outputs represented in the forecast SSP cover a significant proportion of the public entity's budget. |
| c) Output performance measures should not relate to internal processes, events, milestones, and other deliverables. | Internal processes, events, milestones, and other deliverables represent important information that may need to be communicated to the reader. However, they are not outputs. |
| d) The outputs included in the document should include those goods and services that are contracted out by the public entity. These should relate to the end service provided and not to the contract management process. | Outputs and the associated measures of output performance should cover those end services for which the public entity is accountable, even if does not deliver them directly. |
| e) Performance measures should cover enough dimensions of performance. | Measures for each output should cover an appropriate range of dimensions (such as quality, quantity, and timeliness) of an output so that a balanced picture of performance can be obtained. |
| f) As a minimum, targets for output measures should be included for the applicable financial year. | Without clear targets being set and included in the forecast SSP, it is difficult for the reader to form a view about planned performance, and for actual performance to be compared to forecast. The inclusion of targets for more than one year helps the reader to understand planned changes in the services provided over time. |
| g) Targets included in the document should be reasonable and represent best estimates. | Targets should reflect the priorities of the public entity, its resources, choices, and historical performance. |
| h) Where the documents contain demand-driven measures, these need to be clearly identified as such and the reason for their inclusion needs to be explained. | Demand-driven measures provide useful information and context in relation to a public entity's performance. However, they are not under the control of the public entity. They need to be clearly identified as such and supported by "true" outputs. |
| i) Management commentary should help to explain how and why targets have been set at a particular level. | Without appropriate explanation, it is difficult for the reader to understand why particular targets have been set at particular levels. This is especially true when targets are set at the same or a lower level than in previous years, although increased targets may also require explanation. |
| j) Performance measures and targets should be supported by current/historical levels of performance, where this is available, or by an explanation of when it will be available. | Current/historical performance information provides the context for future performance targets. |
| k) The inclusion of comparative data from other organisations or countries provides helpful context to the reader about outputs. | Comparative information helps the reader to understand the level of output performance that the public entity is seeking to achieve. |
| l) It should be clear to the reader which output performance measures will be reported against in the annual report/SSP. | The annual report/SSP should reflect the content of the SOI/forecast SSP. |
| 5. Other document considerations | |
| a) The document should be easy to read, concise, and include diagrams where this will aid the reader's understanding. | It is important that, as a key accountability document to Parliament and the public, the SOI is clearly written and accessible. |
| b) The document should contain information about how the public entity will ensure that it has enough capacity and resources to deliver its outputs and achieve the outcomes sought. | It is important that the reader can see that the public entity's plans in terms of its non-financial performance are supported by the resources, capacity, and capability it has available. |
| c) The document should provide enough information for the reader to assess cost-effectiveness. Enough information could include cost-effectiveness measures for major outputs or logical demonstration of linkages between expenditures, outputs, intermediate outcomes, and end outcomes, with an emphasis on the attribution of outcomes to spending. | The SOI is required to provide information on the cost-effectiveness of its interventions. |
| d) There should be consistency between the SOI and the Information Supporting the Estimates (this applies only to government departments). | The use of different terminology and the inclusion of different outcomes/impacts in the SOI and the Information Supporting the Estimates can make it extremely difficult for the reader to understand the relationship between financial appropriations, the delivery of outputs, and the achievement of outcomes. |
| f) The SOI and the forecast SSP/Information Supporting the Estimates should be "stand alone" documents. | The SOI should provide a concise overview of a public entity's planned activities, how these will contribute to outcomes, and the measures and targets that will be used to demonstrate progress, without the need to refer to other documents. |
| g) The SOI and the forecast SSP/Information Supporting the Estimates should contain information about all significant aspects of the public entity, including recent/planned developments. | The accountability documents should provide the reader with a high-level overview of the public entity and its activities. |
| 6. Supporting Systems and Controls | |
| a) The forecast performance information in the SOI and Forecast SSP/Information Supporting the Estimates should be supported by robust systems and controls that are integrated into the public entity's overall performance management arrangements. | The absence of robust systems and controls that can be relied on to produce forecast and historical performance information is likely to: reduce the ability of the public entity to produce it (and therefore monitor performance) regularly; and reduce the reliability of the information produced. |