Do your measures measure up?

There’s a need for better reporting on how the public sector is performing.

Performance reporting by public organisations is crucial to enabling the public and Parliament to determine whether public money is being spent appropriately. When done well, this reporting clearly explains how well the public sector has used funding to deliver services, achieve outcomes, and ultimately improve the lives of New Zealanders.

We have found through our work that it has long been unclear to the public and Parliament what outcomes are being achieved with public spending – about $160 billion in 2022/23. To maintain trust and confidence in government, we all need to understand what value we are getting from our taxes, and what difference it is making to our lives.

In March 2023, the Auditor-General wrote to chief executives of public organisations, noting these concerns and asking chief executives to significantly improve the quality of their performance reporting. They were asked to develop measures for the 2023 Budget that would provide meaningful information on how spending is contributing to outcomes. This Budget was also an opportunity to improve their performance information to meet a new financial standard on service performance reporting.

After this, we focused on the quality of performance reporting in the advice we gave to Parliament about the Government’s 2023/24 spending plan. This plan is set out in the Budget and examined by select committees through the Estimates of Appropriations process.

The Estimates process is important because it is the mechanism through which Parliament can scrutinise Ministers and organisations before it approves Budgets. But the Estimates are just one component of an overall performance reporting framework. It should be clear to readers where the other meaningful performance information can be found.

For the 2023 Budget, we focused on spending by public organisations in six key Votes, totalling over $101 billion in annual spending. These were:

Vote Health;
Vote Business, Science and Innovation;
Vote Social Development;
Vote Education;
Vote Transport; and
Vote Agriculture, Biosecurity, Fisheries and Food Safety.

We chose these because of their scale and the important role they play in achieving critical outcomes for New Zealanders.

We found some improvements to performance information, with some examples of good reporting. However, we also found three weaknesses that urgently need addressing:

Measures that are not meaningful or comprehensive.
Gaps in measuring what difference is being made.
Poor measures for assessing the stewardship, oversight, and monitoring functions of departments.

We share some examples of these below. We also share examples of where organisations are getting it right. These are based on briefings we prepared for select committees for the 2023/24 Estimates, and are publicly available via our website and www.parliament.nz.

The matters we identified and set out here are often not limited to the organisations we have highlighted. As public organisations plan their performance measures for this financial year and beyond, we encourage them to develop meaningful measures that tell the public and Parliament a clear, compelling story about whether their spending is achieving better outcomes for New Zealanders. The examples below can help.

Measures need to be meaningful and comprehensive

Performance measures should give a comprehensive assessment of how well an organisation is delivering its services and making a difference for New Zealanders. Measures should give the public and Parliament clear and easy-to-understand information about how well an organisation is performing.

Measures should focus on what is important to those who use government services and cover all the organisation’s functions and operations.

Across the Votes, we found many appropriations with measures that are focused on process, inward-looking, and not meaningful to the public and Parliament.

For example, some of the measures include:

the percentage of funding applications processed and approved;
funding being drawn down and utilised for the purposes and on the terms agreed to by Cabinet;
the percentage of forecast investment allocated each year; and
the percentage of contracts monitored against their milestones.

These kinds of measures simply describe the activities and busyness of the organisation. They do not give a sense of how well the organisation is providing services and what difference is being made through these activities.

We also found measures that did not comprehensively cover performance. For example:

The Budget includes a small number of performance measures relevant to mental health (such as the number of people who have access through the Access and Choice programme), but the information is not a sufficient assessment of what is intended to be achieved through funding for mental health. For example, it is not clear what the actual targets are for coverage and access to tailored services for different groups, including Māori, Pacific, and youth.
The Ministry of Education is funded just over $16 million for support and resources for parents and the community. However, the related measure (the percentage of families receiving targeted support who consider they are more confident to support their children’s learning) is limited to parents in a targeted 10-week parent education programme. It is hard to draw conclusions on the effectiveness of the Ministry’s overall support to parents and the community based on measuring the performance of a single programme.
Most of the Ministry of Social Development’s work to administer the benefits system, improve employment outcomes, and provide or enable community services is funded through two pools of money that they are allocated in the Budget. Although there are some meaningful performance measures for this money, some key services have dimensions of performance that are not measured. For example, the purpose of the category Administering Income Support ($465 million in 2023/24) includes assessing, paying, and reviewing entitlements; collecting balances owed by clients; and administering international social security agreements. There are two measures in total, covering timeliness and accuracy of the assessment of benefits, but none for other dimensions of performance such as the reliability of payments or the client experience of the service. We also note that, although included in the description of outputs, there are no measures in this appropriation for collecting balances owed or administering international agreements, although there are measures related to balances owed in another appropriation.

There are gaps in measuring what difference is being made

Many of us want to know if, and how, services and activities delivered by the public sector are leading to better outcomes for New Zealanders.

For many appropriations, public organisations are required to set out what difference they seek to make (that is, the impact or outcomes they aim to achieve) and how this will be assessed. Having meaningful ways of assessing whether they are making this difference is critical to supporting trust and confidence in the public sector.

We found that measures that are meant to set out what is intended to be achieved often contained gaps.

The wider performance reporting framework for spending under Vote Transport has meaningful and appropriate information about what value is being achieved. For example, the transport sector’s outcomes framework is supported by 37 indicators to measure and report on progress against outcomes. However, performance measures for several new initiatives have not yet been developed. The measure associated with the $275 million in new road funding in response to the January 2023 floods and Cyclone Gabrielle are focused on the administration of funds. The $55 million allocated for improving resilience of the roading network, in response to climate-related weather events, is also administrative. Because outputs for both initiatives were in the early stages of development at the time of the Budget, we were advised that the performance measures for both will be revisited through the 2023/24 Supplementary Estimates Process.

The Ministry for Primary Industries has budgeted $365.2 million for biosecurity in 2023/24. The performance measures for these appropriations focus on the Ministry’s activities to prevent, monitor, and manage biosecurity risks. One, for example, measures the “percentage of international air passengers that comply with biosecurity requirements by the time they leave the Biosecurity Control Area at the airport.” The measures, while helpful, focus more on the Ministry’s activities rather than the impact of these activities and whether they resulted in better protection from biosecurity risks.

Conversely, performance information for the fisheries and food safety appropriations generally provide a meaningful assessment of how well the Ministry is delivering its services and achieving better outcomes. The performance measures in these two areas include a mix of measures that assess how well the Ministry is delivering its services and the outcomes it intends to achieve — for example, the rate of a common food-borne illness per 100,000 people, and the percentage of scientifically evaluated fish stocks with no sustainability issues.

In Vote Education, performance measures for the primary and secondary education system focus largely on inputs and participation rates. It is unclear how they assess the quality of education the system is delivering.

The funds administered by the Ministry of Business, Innovation and Employment under the Regional Development portfolio mostly involve multiple Votes and appropriations. These are administered by Kānoa – Regional Economic Development & Investment Unit as well as other agencies. Consequently, there are different levels of reporting across the different funds that are mostly managed by Kānoa, with some fund reporting being managed by other agencies. In addition to different project deliverables and milestones, there are different ways the outcomes and benefits of projects are captured in the reporting across the different funds. We note that across the Vote, the public and Parliament can find it challenging to understand what has been achieved with the funding and how the funding programmes are contributing to the intended outcomes, such as lifting regional productivity potential.

Measures that assess the stewardship, oversight, and monitoring functions of departments must improve

New Zealand faces a set of increasingly complex and long-term challenges, and the public sector is looking to work closer together to solve these problems.

Several of the organisations we focused on are lead departments – or “stewards” – of different sectors. These departments have a leadership role. They provide advice, monitor, and report on what outcomes are being achieved across the systems they oversee.

Performance measures and reporting for these functions and appropriations were often inward-looking. This reporting is not particularly meaningful to the public or Parliament.

Performance measures for the Ministry of Education’s oversight of the education system include measuring timeliness of responses to Parliamentary questions and the number of webpage visits/downloads of new research reports. They do not provide a meaningful assessment of how well the Ministry is providing effective oversight of the education system.

The Ministry of Social Development has an important leadership role in the social sector as a steward of benefit and employment support functions. Appropriations that relate to stewardship include Policy Advice ($22.26 million in 2023/24), Data, Analytics and Evidence Services ($44.65 million in 2023/24), and Planning, Correspondence and Monitoring ($6.27 million in 2023/24). In our view, the performance measures for these appropriations are inward-looking – for example, measuring the quality and timeliness of policy advice and other services to Ministers. They do not provide a meaningful assessment of how well the Ministry is providing effective oversight of the system. The Ministry provides a substantial amount of useful information on its website and in publications. We think there are opportunities to incorporate these activities more formally into the performance framework.

The performance information for the Ministry of Transport’s stewardship role, including oversight of the Vote, does not enable an assessment of how well the Ministry is addressing major funding and financing risks faced by the sector now and into the future, such as with the National Land Transport Programme.

Our conclusion

Overall, we found that the public and Parliament are not given the performance information they need to effectively scrutinise how public money will be spent. It is difficult for the public to use this information to understand what their taxes are being spent on. This needs to change.

Based on our observations, changes to Standing Orders for the 2023-26 Parliamentary term will facilitate more and deeper scrutiny of public organisations. The quality of their performance measures and reporting will need to greatly improve to give Parliament the information it needs to hold them to account.

We urge public organisations to develop meaningful, comprehensive, and relevant performance measures for their appropriations. Parliament can then use these measures to effectively scrutinise proposed public expenditure during the Estimates process, and both the public and Parliament can use them to hold the public sector to account for its performance.

The Office has good practice resources dedicated to improving performance reporting. These include guidance on setting performance reporting frameworks, the essential qualities of performance reporting, and what makes for good performance indicators.