Appendix 4: Performance indicators, data reliability, and usefulness
In this Appendix, we discuss NZGP's performance indicators in more detail, their reliability, and their usefulness in guiding further improvements.
Cost savings
The performance indicator for assessing costs savings is: "Savings target across public sector public organisations identified through the Government Procurement Reform agenda for the financial year".
The indicator reports on the estimated annual savings from the all-of-government contracts. Figure 4 shows MBIE's data for 2016-19, which reports that the targets were exceeded.
Figure 4
Estimated cost savings from all-of-government contracts for the years ended 30 June, 2016 to 2019
The table shows actual estimated savings ranging between $128 million and $186 million during the four years.
| Year ended 30 June | Savings target across public sector public organisations identified through the Government Procurement Reform agenda for the financial year | |
|---|---|---|
| Target | Actual | |
| 2016 | $88 million | $128 million |
| 2017 | $90 million | $152 million |
| 2018 | $90 million | $146 million |
| 2019 | $90 million | $186 million |
Source: Ministry of Business, Innovation and Employment.
In 2012, Cabinet recognised that there would be scepticism about the extent of the savings because of problems with the data used to estimate savings. We found that public organisations and suppliers are sceptical about the extent of the savings.
Issues with the reliability of the procurement data used to estimate savings and the methodology used to estimate savings contribute to scepticism about the reliability of the estimates. Through our annual audits of MBIE, we have examined both of these issues.
MBIE relies on supplier-reported information about spending through the all-of-government contracts. This is not ideal and risks overstating the savings that all-of-government contracts achieve. Estimates of savings would be more reliable if NZGP had direct access to public organisations' information.
The process for collecting the information is also prone to error. At our request (through the annual audit), MBIE included a note on the reliability of the estimated savings in its annual reports:
The Ministry relies on information provided by suppliers to calculate savings. The Ministry reviews information provided for reasonableness, but cannot confirm the accuracy of the information.
NZGP uses the data to estimate savings using standard published methods for goods and services.24 These methods have their limitations, but NZGP calculates estimates consistently from year to year, and the result is based on the best available information.
NZGP periodically commissions a firm to review spending on selected all-of-government contracts for a small sample of public organisations. The work involves trying to match three sets of information:
- the amount that public organisations' records show has been paid to a supplier;
- the amount that suppliers' records show they have been paid by public organisations; and
- the amount that suppliers report to NZGP as having been spent by the selected organisations.
In most instances, the data cannot be reconciled easily. In the reports we saw, the amounts reported to NZGP compared with the other amounts had differences ranging from a few hundred dollars to hundreds of thousands of dollars. Mostly, suppliers' reports to NZGP over-reported spending, but not always, and the reasons for the differences could not be easily explained.
Public organisation satisfaction with all-of-government contracts
The indicator for assessing public organisations' satisfaction with the value that all-of-government contracts deliver is "Percentage of participating public organisations satisfied or very satisfied" (see Figure 5).
Figure 5
Percentage of public organisations satisfied or very satisfied with all-of-government contracts, 2014 to 2019
The table shows that the target has been achieved since it was reduced from 75% to 70% in 2016.
| Year ended 30 June | Target % | Actual % public organisations satisfied or very satisfied | Variance from target % |
|---|---|---|---|
| 2014 | 75 | 69 | - 6 |
| 2015 | 75 | 64 | -11 |
| 2016 | 70 | 74 | +4 |
| 2017 | 70 | 75 | +5 |
| 2018 | 70 | 71 | +1 |
| 2019 | 70 | 71 | +1 |
Note: In 2018/19, the survey was sent to 859 participating public organisations. NZGP received 223 valid responses – a response rate of 26%.
Source: Ministry of Business, Innovation and Employment.
For the responses that are received, the data reported is reliable but is not useful because it is a peak indicator. It does not show what aspects of the all-of-government contracts are satisfactory and not satisfactory. However, the detailed survey results are available online at procurement.govt.nz.
Business survey of suppliers
The indicator for assessing improvements in procurement capability is "Year on year improvement in business feedback about the quality of government procurement practice from the annual government procurement business survey".
The business survey started in 2013/14 and asks for suppliers' comments on their experiences of government procurement generally. The survey measures improvements in procurement capability because it assesses public organisations' quality of procurement.
MBIE reports whether the indicator is "achieved" or "not achieved". The indicator was achieved from 2016 to 2019. The indicator is not useful because it does not indicate the extent of change or the baseline that change is measured from. However, the detailed survey results are available online at procurement.govt.nz.
24: Ministry of Business, Innovation and Employment (2018), Standard reporting methodologies: All-of-government contracts, at procurement.govt.nz.