Part 6: Conceptual framework for reporting non-financial performance information

The Auditor-General's observations on the quality of performance reporting.

6.1
In this Part, we set out our conceptual framework, based on GAAP and legislative requirements, that informs our approach to auditing performance reports. We expect public entities' performance reports to be consistent with our conceptual framework.

Purpose of the conceptual framework

6.2
During the past two years, our expectations in reviewing forecast information have been customised to the various sectors we audit. As a result of this work, we have prepared a conceptual framework that we intend to use in auditing performance reports. There are variations within different sets of legislation in wording and time requirements for different types of entities. Our reason for developing a conceptual framework is to capture the essence of the principles and underlying definitions that are common to all affected entities.

6.3
This conceptual framework therefore pulls together performance reporting concepts from GAAP, legislation, and other guidance into one unified and consistent framework. We have sourced concepts from material developed for the central government, local government, and not-for-profit sectors, and we intend the framework to be interpreted for, and applied equally to, those sectors.

6.4
The framework's primary function is to provide a consistent basis for describing the expectations placed on those responsible for preparing performance reports on which we are required to issue an audit opinion.

What best practice in accountability reporting would look like

6.5
This paper discusses outcome and output reporting. However, best practice in external accountability performance reporting would reflect:

  • a comprehensive model of performance;
  • a properly set direction;
  • good measurement systems; and
  • accessible reporting of significant information while meeting cost-benefit tests.

6.6
Good performance reporting also takes a broader view than individual entities. New Zealand's current performance reporting framework is entity-based. An ideal framework would take account of and report not only entity performance, but also sectoral, cross-sectoral, and whole-of-government performance.

A comprehensive model

6.7
The best external accountability reporting draws selectively from a comprehensive set of performance elements to identify, measure, and report those performance elements of interest to stakeholders. Compliance with statutory performance reporting requirements is essential, but statutes do not limit or prohibit enhanced reporting. Figure 4 (from our 2001 report Reporting Public Sector Performance) illustrates the elements of a comprehensive model of performance.

6.8
Reflecting a comprehensive model in external accountability reporting also requires a range of specific capabilities. Organisational capability can involve:

  • understanding operations and stakeholder information requirements, to identify which elements of performance should be reported and which performance indicators or measures will be relevant to report on those elements;
  • technical knowledge, to identify needed data or design data collection systems;
  • capacity (of people or systems), to collect the necessary data; and
  • data collection over a period of time, and specific retention of data during change (such as entity restructuring).

6.9
Building organisational capability in these ways has potential benefits for both external reporting and internal management. There is also potential benefit for government departments that have responsibilities for monitoring other agencies and for policy ministries that have lead roles in sector development.

6.10
Elected representatives have an important role to play in creating demand for the use of broader models. In our view, members of Parliament and local authority councillors should consistently encourage general use of comprehensive models of performance. In particular, elected representatives can insist on best practice in applying the models, such as requiring well-specified links between outputs and outcomes.

A properly set direction

6.11
Performance reporting needs to start from the direction set for the entity and, increasingly, for a sector or across sectors. Setting direction:

  • requires inclusive consultation processes that cater for suitable communities of interest; and
  • provides guidance on what data to gather and what to ignore.

Figure 4
Elements of a comprehensive model of performance reporting

Figure 4: Elements of a comprehensive model of performance reporting.

Good measurement systems

6.12
Performance reporting needs to be based on good performance measurement systems that collect relevant and reliable data. Good quality measurement and data collection provides an essential resource to generate performance information from. Good performance measurement needs:

  • the capability to produce appropriate sets of information for a variety of stakeholders; and
  • an enhanced set of data to use to answer as yet unspecified future questions.

Objectives of performance reporting

6.13
Performance reports, including SSPs, help users to:

  • assess a reporting entity's service performance and achievements;
  • assess the reporting entity's compliance with legislation, regulation, and contractual arrangements as they relate to the assessment of its service performance; and
  • make decisions about providing resources to, or doing business with, the reporting entity.

6.14
As with financial performance reports, non-financial statements fulfil an accountability role and a decision-making role. They can also be used as a framework to help public entities prepare policies about which outputs are likely to lead to which outcomes.

Presentation of performance reports

6.15
Performance reports contribute to the objectives of general purpose financial reporting by providing:

  • narrative, data, and statistics on the reporting entity's performance in supplying goods and services; and
  • information on the effects on stakeholders, society, or the community of the entity's existence and operations.

6.16
Accountability requires that reports:

  • identify objectives and targets normally established by formal processes; and
  • measure actual achievements against those objectives.

6.17
The SSP (or forecast SSP) may be supplemented by other non-financial statements containing contextual or background information, such as:

  • aspects of the direction-setting process, assessment of risks, and trade-offs/ choices made and why;
  • forecast and actual outcomes, intermediate outcomes, or impacts;
  • unintended outcomes;
  • ways to address unintended costs or benefits;
  • information on the resources used to achieve performance compared with the forecast resources;
  • any changes to plans, and consequent effects on results; and
  • whether performance measures need to be changed and how stakeholders will be consulted as part of re-setting direction.

Elements of non-financial performance

6.18
The elements of non-financial performance may be classified as:

  • results – outcomes or achievements and consequences for the community:
  • interactions with the public – processes (including systems, operations, and behaviour) and outputs (delivery of goods and services); and
  • costs – inputs and decline in public entity capability.

6.19
The non-financial elements most commonly included in financial reports are those directly related to non-financial performance – inputs, outputs, and outcomes – which are defined in various current and historical statements of GAAP, legislation, and a wide range of published material on organisational goal setting, planning, and strategy. Although these sources vary in their degree of authority and in their nuances of definition, they are in broad agreement on the meaning of these terms (see the Glossary in Appendix 1).

6.20
Reporting entities often find it useful to introduce other reporting elements to help bridge the relationship between outputs and outcomes. This is particularly useful when the cause and effect relationship between outputs and outcomes is relatively loose or remote. These other elements are typically lower-level outcomes that describe or measure effects that are more directly attributable to output performance. They are usually described as immediate outcomes, intermediate outcomes, or impacts1 and form part of an outcomes hierarchy that logically lays out the links between outputs, the intermediate-level elements (lower-level outcomes), and the higher-level outcomes.

6.21
The SSP is concerned with reporting on output (goods and services) performance and the cost of producing or delivering them. Other aspects often reported on – such as management systems, internal outputs, processes, activities,2 efforts, and detailed information on inputs – should not be confused with outputs, should not be described or reported as outputs, and generally should not be emphasised in the SSP. Their measurement may be important for internal management purposes, but, for general purpose reporting, the external users of SSPs need information about results rather than efforts. Hence, the SSP focuses on output delivery, with reporting against outcomes (that is, the effects of output delivery) being in a separate performance statement outside the SSP, as explained in the following section.

Content of performance reports

6.22
The objectives of performance reporting are more commonly achieved by publishing:

  • a medium-term, outcome-oriented statement of intended achievements;
  • an annual, output-oriented forecast SSP; and
  • an annual report, incorporating the SSP.

6.23
The first two statements are forecasts, and the third statement is historical.

6.24
The SSP in the annual report should report actual results against the forecast performance measures and targets outlined in the forecast SSP for any one year. But an effective performance reporting framework is broader than the SSP (or forecast SSP).

6.25
The medium-term statement of intended achievements tends to focus on forecast information (that is, intended outcome achievement) for the prescribed number of years. Ideally, it will also provide historical actual results so that readers can track progress towards intended outcomes.

6.26
Both the outcome-oriented and the output-oriented statements should be supplemented by other non-financial information giving context to those statements. Explanations of the reporting entity's role and functions, legislative or constitutional mandates, objectives, and strategies for achieving those objectives will provide a necessary context for readers to make sense of the service statements. Information on resources, capability, and risks also provides valuable background information to help readers understand and assess how well the reporting entity is delivering its services and making progress towards achieving its outcomes.

6.27
The outcomes information and the broader contextual information are necessary for readers to draw conclusions about the rationale for, and the appropriateness of, the elements (outputs and output classes), performance measures, targets, and results reported in the SSP (or forecast SSP).

6.28
A sound performance reporting framework will present this information in such a way that the reader will be able to readily identify the links between the reporting entity's function, strategy, and performance goals. It will specifically identify the links between outputs and the outcomes they contribute to, making clear the rationale for doing the activities and producing the outputs.

Medium-term, outcome-oriented statement of intended achievements

6.29
The period covered by the medium-term, outcome-oriented statement will depend on the reporting entity's requirements – for example, requirements in keeping with legislation, its constitution, or the rules governing its accountability requirements. The statement will usually relate to a minimum of three future financial years. Although its content will also often be determined by specific legislation or other prescription, it will typically provide a succinct, strategically oriented description and explanation of:

  • what the entity does (that is, its purpose, functions, and activities);
  • the specific impacts, outcomes, and objectives the entity seeks to achieve;
  • how the entity intends to achieve these, including its operating intentions;
  • measures and targets that indicate the progress made;3
  • the cost-effectiveness of the entity's interventions;
  • the challenges the entity faces, including references to organisational health and capability, as well as to risks and how these are to be addressed; and
  • any other matters necessary for the reader to understand the entity's operating intentions.

6.30
Entities should reconcile long-term outcomes specification with short-term performance reporting by reporting against intermediate outcomes or impacts.

6.31
The primary focus of the medium-term statement of intended achievements is outcomes. To identify and properly specify outcomes in non-financial statements, the following principles (or recognition criteria) should be applied to each outcome:

  • Progress or achievement can be influenced by the outputs of the entity.
  • The outcome describes a specific impact on society, the community, or a target group.
  • The outcome is believed to be of value to society, the community, or the target group.
  • The outcome provides a clear statement of what the “purchaser” of the outputs is seeking to achieve.
  • The outcome is part of a hierarchy of outcomes, and the hierarchy structure should be clear.
  • If the outcome is aimed at specific people, those people should be clearly identifiable or identified.
  • Progress or achievement of the outcome can be assessed within the specified time.

6.32
Information on the specific links between desired outcomes and outputs is vital, as is the continuing need to evaluate the extent to which current outputs are contributing to those outcomes. There should therefore be a clear and credible relationship between the medium-term, outcome-oriented statement of intended achievements and the forecast SSP. If the forecast SSP indicates significant resources are directed to a specific output class, then that output class should feature strongly within the medium-term, outcome-oriented statement of intended achievements.

Annual, output-oriented statement of forecast service performance

6.33
The content of the output-oriented forecast SSP will also often be determined by specific legislation or other prescription. It should provide detailed performance information on intended output delivery that the entity must report against as part of its annual report. Details will typically include:

  • a description of each class of output the entity proposes to supply during the financial year;
  • the outcomes to which each output is intended to contribute (if practical and appropriate);
  • performance measures and forecast targets of output delivery for each class of output;
  • the expected revenue to be earned, and proposed output expenses to be incurred, for each output or class of outputs; and
  • any other matter necessary for the forecast SSP to be prepared in keeping with GAAP.

6.34
To be useful for accountability and decision-making, the basis for aggregating outputs must be specified in the forecast SSP.

6.35
Inputs, outcomes, management systems, internal outputs, processes, activities, and efforts are not outputs of the entity in this context. They should therefore not be described or reported in service performance reports as outputs. However, information on the links between desired outcomes and outputs is vital, as is the continuing need to evaluate the extent to which current outputs contribute to those outcomes. Explanation of the rationale for a particular mix of outputs is useful contextual information for both forecast and historical reporting, and the links should be made as transparent as possible.

6.36
The focus of the forecast SSP is outputs. To identify and properly specify outputs in the forecast SSP, the following principles (or recognition criteria) apply:

  • distinctiveness – they are distinct products or services;
  • externality – they have an external focus (that is, they are produced for third parties or persons or entities outside the reporting entity) and they reflect the purchaser's interests and priorities; and
  • controllability – they are controllable by the reporting entity and, as such, their specifications or performance dimensions reflect the reporting entity's extent of control.

Statement of service performance in the annual report

6.37
The function of the SSP in the annual report is to report actual results against the proposed levels of service delivery signalled in the forecast SSP. It should include:

  • a description of each class of output that the entity has supplied during the financial year;
  • the outcomes to which each output was intended to contribute (if practical and appropriate);
  • the levels of delivery performance achieved, compared with the targets included in the forecast SSP, for each output or class of outputs;
  • the actual revenue earned and output expenses incurred, compared with the expected revenue and proposed output expenses included in the forecast SSP, for each output or class of outputs; and
  • any other matter necessary for the SSP to be prepared in keeping with GAAP, including any other measures and targets needed to assess the entity's performance for the year.

6.38
We recommend disclosing cost allocation policies and providing costing information below the aggregated output level.

6.39
If information on a particular variance between forecast and actual performance could affect the users' assessment, then that information should be reported, together with the reason for the variance. The following should also be reported:

  • long-term output work-in-progress;
  • changes to specified output performance; and
  • full disclosure of changes in objectives during the year.

6.40
To provide context, the SSP should identify the outcomes that the outputs have contributed to. Explanation of the rationale for a particular mix of outputs is useful contextual information for both forecast and historical reporting. The output-to-outcome links should be made as transparent as possible.

6.41
It is also desirable to regularly report information on the achievement of outcomes (actual results against intended results). This information could be reported within the annual report, but outside the SSP. In most cases, the frequency and location of outcome reporting will be determined by specific legislative or constitutional requirements.4

Performance measures and performance targets

6.42
Performance measures are used to measure outcome or output achievement against performance targets or objectives. Performance measures specify one or more of the “dimensions” of performance.5

6.43
In selecting performance measures to report, entities should consider the characteristics of performance that:

  • are of greatest importance to stakeholders;
  • reflect the financial significance of the activity; and
  • reflect both the objectives for carrying out the activity and any (external or internal) risks needed to be managed in achieving those objectives.

6.44
For dimensions to be relevant to the assessment of an entity's performance, the entity must be able to control to a significant extent the dimension being measured. To the extent that there is uncertainty about an entity's ability to control a dimension of its performance, the entity should provide enough information on the risks to its service delivery.

6.45
Performance targets describe the specific levels of performance to be delivered or achieved. They may be expressed as (but are not limited to) absolute numbers, percentages, ratios, point estimates, or ranges. Performance targets should be realistic and based on best estimates.

Qualitative characteristics

6.46
The quality of the information provided determines the usefulness of performance reports to users. The qualitative characteristics for financial reports are also important for performance reports.

6.47
All of the important qualities that make performance reports meaningful and credible, and therefore useful, can be summed up under four qualitative characteristics:

  • relevance;
  • reliability;
  • understandability; and
  • comparability.

6.48
These four qualities should apply, as appropriate, to:

  • the reported elements (primarily the outcomes and outputs);
  • the performance measures selected; and
  • the performance targets set (and actual performance levels achieved).

6.49
Each concept includes several sub-concepts, which are derived either directly from the legislation and standards or from other guidance and work in the area of service performance information.

Relevance

6.50
To be relevant, the reports should:

  • be presented within the context of the entity's strategic objectives, past performance, and current environment (including government themes, as appropriate, and other themes and considerations, such as sustainable development);
  • show clear and logical links between entity-level objectives (and themes), outcomes, outputs, performance measures, and performance targets (so that the rationale for the selection of elements, measures, and targets is evident);
  • meet the information requirements of stakeholders (including by reporting different levels and layers of information) and be useful for decision-making, as appropriate; and
  • be clearly linked to the financial information, including significant areas of planned expenditure.

Reliability

6.51
To be reliable, the reports should be faithfully represented and supportable, in that they:

  • are measurable;
  • represent what they purport to, or are expected to, represent (that is, they represent the substance of transactions and events);
  • include elements and performance measures that informed readers are likely to have chosen; and
  • are free from material error (that is, they are accurate or capable of having their accuracy determined within an acceptable range of precision or certainty).

6.52
To be reliable, the reports should be neutral – that is, free from bias in the selection, measurement, and disclosure of the elements, performance measures, and performance targets.

6.53
To be reliable, the reports should also be complete and balanced. They need to:

  • be comprehensive enough, aggregated where appropriate, and reasonable (with the basis for aggregation clearly specified); and
  • cover the significant activities and all important aspects (including identifying the important dimensions of performance), and give them suitable emphasis, to fairly reflect their significance to the entity's performance.

6.54
Completeness is subject to considerations of economy, succinctness, materiality, feasibility, and the cost of reporting. An element or performance measure is material if its omission or mis-statement would affect users' perceptions or assessments of the reporting entity's performance or their decision-making.

Understandability

6.55
To be understandable, the reports should:

  • have a clear format and layout;
  • be presented in a way that engages the reader – for example, by creating visual interest through the use of charts, tables, and symbols;
  • classify reported items clearly and logically;
  • be coherent, with easy-to-follow links between the different parts;
  • be presented within the context of the reporting entity's strategic objectives, past performance, and current environment (including government themes, as appropriate, and other themes and considerations, such as sustainable development);
  • show clear, logical, and easy-to-follow links between entity-level objectives, outcomes, outputs, performance measures, and performance targets;
  • be clear and concise in their content; and
  • be easy to read, expressed in plain English, and use words and terms suitable for users (with adequate explanations of acronyms, jargon, and technical terms).

Comparability

6.56
To be comparable, the reports should:

  • be consistent in their format, layout, and in the way information is classified;
  • be consistent in the selection, measurement, and disclosure of elements and related information;
  • allow users to identify similarities and differences and to track progress:
    • most importantly, of actual performance against forecast performance, and
    • across different reporting periods, and to identify trends; and
  • allow users, where appropriate and practicable, to identify similarities and differences among different entities.

Constraints on qualitative characteristics

6.57
We acknowledge that there are some constraints on producing performance reports that have the required qualitative characteristics. These constraints include:

  • Timeliness – To be useful, reports must be produced within a certain time frame, and the data must be timely for assessment and/or decision-making.
  • Balance between benefit and cost – There are limits to the extent to which more or better information justifies its cost in terms of benefits to the user.
  • Balance between qualitative characteristics (that is, trade-offs between qualitative characteristics) – Some qualities may need to be partially sacrificed to enhance other qualities (for example, sacrificing comparable data when introducing new more relevant data).
  • Materiality (that is, the extent to which omission or mis-statement would affect users' perceptions or assessments of the reporting entity's performance or their decision-making) – The identification of subject matter and the volume of information needs to be subject to decisions about its significance.

1: “Impacts” is separately defined in the Public Finance Act 1989 (see the Glossary in Appendix 1).

2: The Local Government Act 2002 uses the term “activity” to refer to goods and services. However, to avoid confusion, we use only the term “output” to refer to goods and services, and we use the word “activity” according to its common and wider meaning.

3: Progress can be demonstrated by measures of historical performance or achievement, trend information, future targets, baseline data, or other benchmark data.

4: For example, the Local Government Act 2002 (clause 15 of Schedule 10) requires the annual report to include the results of any measurement carried out during the year towards the achievement of outcomes, as well as any identified effects that any activity within the group of activities (that is, output classes) has had on the well-being of the community.

5: Including, but not limited to, quantity, quality, timeliness, location, cost, and reliability.

page top