Part 5: Specifying and presenting the outcomes framework

5.1
The SOI must identify the outcomes the entity aims to contribute to or influence, at least in the medium term. This is important because the outcomes explain why the entity exists, whereas the outputs explain how the entity intends to intervene to help bring about the outcomes. Therefore, if appropriate, the outcomes should be identified at various levels (for example, impacts, low-level intermediate or intervening outcomes, high-level outcomes), with a clear depiction of the cause-and-effect relationships between them.

5.2
It is important that the outcome statements are accurate and well-phrased; they should capture and describe the desired effects. An outcome statement should refer to the state or condition of society, the community, economy, or environment. It should also include a statement about the desired change in that state or condition.¹ It is therefore important that the outcome statement be “dynamic” – that is, it describes the direction of the change being sought and preferably the targeted or desired level of change.

5.3
The SOI should capture those impacts and outcomes that are priorities for the entity. It should be clear to the user why the reported impacts and outcomes are priorities.

5.4
The Retirement Commission identifies two high-level goals, which are then split out into four stated outcomes (Example 4). The first of these goals, and the first of the four stated outcomes, come closer to what we would expect to see than most other outcome statements we reviewed. This is because they express something about a desired state of society as a whole.

5.5
In its SOI, the Retirement Commission expresses outcomes in static rather than dynamic terms, although, within its performance measures and targets (Example 5), the entity does point out the expected change. It is necessary to specify the desired change of direction within the outcome statement to identify precisely what the entity is seeking to achieve. This makes it easier to then identify appropriate performance measures. We consider the Retirement Commission could improve its outcome statements (and meet the statutory definition of “outcomes”) by describing the specific improvement sought in the state or condition of society.

5.6
The Retirement Commission’s usage data (use of “Sorted” resources, visits to the website, and calculations made on the website) are all impact measures – in that they measure the direct effects, on people, of the service provided. This is their primary value. Impact measures are also useful because they have the potential to imply something about the quality of the service.

5.7
By contrast, stating the number of booklets distributed (an output measure) is of limited value because there is no sense of their quality or their impact on users. To be really useful, both the impact and output measures shown in this example need to be augmented by measures that capture their actual quality (in the case of the booklets) and inferred quality (in the case of the impact measures).

5.8
The MED’s outcome statements describe the desired changed state, although the desired change is stated simply as “improving” (Example 6). It is not always clear from these statements what “improving” actually means (that is, more, or better). More information is needed to explain what sorts of improvements are desired, how each improvement will be measured (performance measures), and how much improvement is required (performance targets). (The MED discusses what it will measure in its “How we will demonstrate success” sections.)

5.9
One of the major challenges in reporting outcomes is deciding on the most appropriate level or levels at which they should be reported. Reporting outcomes at only one level usually provides only an incomplete performance story. A more informative story is one that reports a hierarchy of outcomes, demonstrating how lower level impacts of service delivery feed through a cause-and-effect chain to higher level outcomes and Government priorities. Therefore, it is helpful to include intermediate outcomes or impacts (as defined in legislation)² to clearly articulate the entity’s assumptions about the intervention logic they use (that is, the theory behind how courses of action resulting in outputs cause changes to the state of individuals and society). On one page, the Ministry of Agriculture and Forestry (MAF) illustrates its hierarchy of government priorities, high-level outcomes, intermediate outcomes, output appropriations (called Votes), and inputs/processes (for example, capability and collaboration) (Example 7). Many of MAF’s intermediate outcome statements are dynamic, describing the change direction sought.

5.10
The Charities Commission presents a succinct, two-layer outcomes framework linked to outputs (Example 8). The framework clearly lays out the general cause-and-effect chain between intermediate outcomes and the single “overall outcome”. The cause-and-effect relationship between the outputs and intermediate outcomes is also clear, although the diagram relates the bundle of outputs to the bundle of intermediate outcomes (that is, it does not show which specific outputs relate to which specific intermediate outcomes).

5.11
Another notable feature of this example is that the intermediate outcome statements give a broad indication of the changed state desired.

5.12
However, we consider that some of the intermediate outcome statements would be improved if they were not so complex: the second and third intermediate outcome statements contain more than one outcome-type statement. We think it would be better for each outcome statement to be simple and refer to only one outcome. For example, the second intermediate outcome statement refers to: (1) the sector being well advised; (2) improved governance and management; and (3) more effective use of resources. Therefore, the specific focus of the second intermediate outcome is unclear.

5.13
A clear expression of outcomes is also important for letting the user know the important effects the entity measures and monitors. For the Charities Commission’s second intermediate outcome, it is not clear which of the three effects is most important. It may be that all three outcomes are vital, and there may even be a cause-and-effect chain involving all three of these effects. If so, then an alternative approach might be to express this intermediate outcome as three separate outcome statements, one leading to the other, and to depict this in a multi-layered diagram of impacts and consequent outcomes.

5.14
Te Māngai Pāho’s framework sets out desired outcomes in a hierarchy of elements, from its vision statement (high level) down to capability initiatives (low level) (Example 9). The intermediate outcome statements could be improved by the agency asserting the desired change in direction and level, in the state of society, which would result from their outputs.

5.15
The outputs, and the major and intermediate outcomes, are set out in separate layers. If there are specific relationships between individual outcomes and individual output classes, then those relationships are not clear from this diagram.

5.16
It is common practice to include something about capabilities, inputs, or resources in diagrams like this, and to show them feeding into the outputs. Some entities overlay the input-output-outcome model with a depiction of strategic goals, which relate to anything ranging from input/capabilities to outputs and outcomes. Te Māngai Pāho has inserted a “Strategies” layer between outcomes and outputs; the individual elements of this layer refer variously to inputs, outputs, impacts, and outcomes. Another approach is to have the most important strategic priorities spanning the various layers.

5.17
The New Zealand Customs Service (Customs) has several outcomes-type layers in their strategic framework diagram (Example 10). The diagram identifies different elements, including immediate results, impacts, and outcomes stemming from its output delivery. We consider this to be a good example, because of its structure, orderliness, and general clarity. The elements are shown in layers, which is a useful way of showing the performance framework, including the hierarchy of impacts, outcomes, and inputs. As with the previous example, if there are any specific links between output classes and their outcomes, then they are not explicit.

5.18
Customs’ diagram includes an “immediate results” column, which provides additional and relevant information to the other columns. The immediate results column appears to contain a mix of different elements of performance. Some of the statements in the column relate to the intended effects of Customs’ operations, whereas others refer to the quality of the services (or inputs/processes relating to the services) themselves.

5.19
We consider that improvements could be made to the way outcomes are expressed, because they fall short of being complete statements of the desired change to the state of the community or society. Further, we would consider “relationships” as an enabler of service delivery rather than an outcome of service delivery, as defined under legislation.

1: As defined in section 2 of the Public Finance Act.

2: The legislative definition is included in the Glossary.