Part 5: Measuring progress towards Primary Growth Partnership objectives

5.1
In this Part, we discuss:

• long-term benefits and results;
• how the Ministry monitors the progress of PGP programmes;
• the Ministry's intention for measuring PGP's achievements as a whole; and
• how the Ministry shares results with the public.

Our expectations

5.2
We expected that the Ministry would use good information to govern programmes effectively. Effective governance includes monitoring whether programmes are being delivered, using information to achieve programme goals, and reporting results in an open and transparent way. We also expected the Ministry to have systems to evaluate programme results.

5.3
We expected that the Ministry would be able to monitor and report progress on programmes and on PGP as a whole. We also expected the Ministry to have tools that would enable it to:

• assess how well investments were tracking;
• manage risk;
• view milestones and results; and
• report in an open and transparent way on the progress of each programme and PGP as a whole.

5.4
To assess these aspects, we reviewed strategic planning and risk documents, quarterly reports, IAP minutes, and Ministry accountability documents. We also talked with Ministry staff. We were given documents and presentations about new tools the Ministry is preparing.

Summary of findings

5.5
To date, the Ministry has focused its monitoring of PGP on individual programmes. Public reporting of results started late and, in our view, has been unsuitable because reports are inconsistently presented and sometimes too technical, which makes them hard to understand.

5.6
The Ministry has not yet been able to show what PGP as a whole has achieved. Since late 2013, the Ministry has been adopting a broader approach to managing PGP by introducing portfolio management to better demonstrate the value over time of PGP investment. The Ministry expects the portfolio management system to be operating by mid-2015.

5.7
It is too soon to observe any economic benefits of the six programmes we reviewed. However, results are being achieved that indicate progress towards the expected long-term benefits.

5.8
At the programme level, progress is being monitored and measured. Quarterly reports on the respective programmes provide a comprehensive view of programme activities and progress. These reports enable PSGs to track the progress of programmes and to carry out their governance role effectively.

5.9
The Ministry began public reporting about the progress and achievements of individual programmes in 2013. This was well into the life of some of the programmes and should have started earlier to provide transparency. Public reporting needs to be simpler and more readily understandable to appropriately inform members of the public about the performance of PGP programmes. The Ministry has not yet reported to the public about the achievements of PGP as a whole.

Long-term benefits and results

5.10
The six programmes we reviewed are multi-year and have long-term goals. One programme will be completed in 2015. The remainder will be completed before or by 2019. Even when all programmes have been completed, and assuming that all of the programme goals will be achieved, there will need to be time for the long-term benefits of the programmes to affect the economy.

5.11
The business cases of the six programmes showed that a range of economic benefits were expected to be achieved by 2019 and beyond.

5.12
In 2014, the Ministry asked the New Zealand Institute of Economic Research (NZIER) to investigate PGP's economic effects and analyse potential growth in exports and gross domestic product and other economic measures. The report shows that PGP could add up to $6.4 billion to New Zealand's GDP from 2025, with the possibility of a further $4.7 billion if all the research and development is successful, the aspirational stretch of programmes is achieved, and the innovations are taken up widely. The NZIER found that the estimated effect on GDP has a high benefit-cost ratio, which it said indicates that PGP is a worthwhile investment of government funds.

5.13
It is too soon to observe the economic benefits of PGP programmes, and it will be at least five to 10 years before we see the extent to which New Zealand's primary industries achieve the anticipated economic benefits.

Results indicate progress towards long-term goals

5.14
Each PGP programme is made up of several different projects that together contribute to the programme's overall goals. We assessed whether projects in all six programmes were on track.

5.15
Most projects in the six programmes were on track. However, the status of one project we reviewed was not on track and the industry partner had decided to remove this project from the programme.

5.16
Results are being achieved that indicate progress towards long-term goals. For example, the FoodPlus programme has reported that product development work is progressing and that some new products will be commercialised in the next year. Figure 4 following paragraph 2.24 provides more examples of the results of the six programmes.

Monitoring of programmes

Monitoring information provides a comprehensive view of programme activities and progress

5.17
Industry partners are required to report to their respective PSGs about their programmes, including activities carried out and progress towards achieving results. Effective programme monitoring and reporting allows the Ministry, the IAP, and PSGs to manage risks, provide appropriate oversight of programme activities, and monitor whether programmes are being delivered successfully.

5.18
Quarterly reports that industry partners submit to their respective PSGs are usually in-depth and highly detailed. They provide a comprehensive view of programme activities and progress. Quarterly reports include information about programme progress, human resourcing, communications, financial management, and work the industry partners have done with related parties to build relationships and keep them informed.

Monitoring information allows PSGs to track the progress of programmes and supports PSGs to carry out their governance role effectively

5.19
For the six programmes, PSG minutes showed that monitoring data was discussed and used to keep the Ministry informed, monitor progress, and help to make decisions about managing programmes.

5.20
For example, quarterly reports inform PSG decisions about whether to approve communications about programmes. PSGs also use information taken from quarterly reports to inform their decisions about whether to approve invoices for submission to the Ministry for programme activities.

5.21
Another example of how PSGs use monitoring information is a PSG approving additional programme activities. The Seeds PSG endorsed extra pasture trials to get a clearer understanding of the effectiveness of seed additives. The PSG acknowledged the likely increased expenditure by the industry partner.

5.22
Our 2013/14 annual audit of the Ministry involved auditing a sample of the Ministry's monitoring, control, and governance arrangements for PGP. The audit noted no concerns about the quality or regularity of reporting to relevant PSGs. It noted evidence that PSG members regularly questioned and challenged reported expectations.

5.23
However, during that audit, we found an example where the quality of reporting to a PSG had been inadequate for some time. In 2012, the Ministry raised concerns with one of its industry partners about the quality of reports provided to the respective PSG. In particular, the Ministry raised concerns that not enough details of programme activities were reported, meaning that it was unable to adequately monitor progress towards the programme's original long-term objectives.

5.24
In response, the industry partner prepared a revised reporting template. In 2013, the Ministry noted that the most recent quarterly report had more clearly showed how the programme's activities linked to its long-term objectives.

5.25
In our view, the emphasis the Ministry places on monitoring and reporting allows the PSGs to track the progress of programmes and to carry out their governance role effectively.

Measuring achievement as a whole

The Ministry has not yet been able to show what PGP as a whole has achieved

5.26
The Ministry has taken a programme-by-programme approach to measuring the achievements of PGP, although it began introducing a more strategic approach in late 2013 (see paragraph 5.30). In our view, this programme-by-programme approach has contributed to the Ministry not yet being able to show what PGP as a whole has achieved.

5.27
Our review of individual programme monitoring information, such as quarterly reports, showed that the Ministry has access to the information it needs to measure the progress of PGP as a whole. This includes information about:

• how well investments are tracking;
• money spent on programme activities;
• progress towards milestones and results; and
• risks to programmes and partners.

5.28
However, the respective industry partners prepare the quarterly reports for each programme. Quarterly reports are presented in a range of different formats and reflect the terminology and style of the industry partner that prepared the report. Because of this, the Ministry receives markedly different quarterly reports from programme to programme, making it difficult to compare programmes.

5.29
By focusing on programmes in isolation, the Ministry is unable to generate an overall view of all programmes. It has not yet been able to show what PGP has achieved as a whole, acknowledging the long-term nature of PGP. The Ministry is building systems to enable it to do this.

A more strategic approach to managing PGP

5.30
Since late 2013, the Ministry has been introducing a more strategic approach to managing PGP. This is meant to help the Ministry manage, measure, and report on the PGP portfolio. The Ministry expects that the work it has under way will enable it to track progress towards portfolio-level outcomes and communicate the combined benefits at a portfolio level using categories such as new products, new technologies, and job creation. The Ministry expects to have the portfolio management system operating by mid-2015 and will refine it over time, including integrating it with wider Ministry investment management systems.

5.31
So far, for example, the Ministry has created and is using three portfolio risk registers that consolidate risks from all programmes in three categories:

• risks to programmes;
• risks to industry partners; and
• risks to the portfolio.

5.32
The registers show potential risks, the effects and consequences that they could have, and planned actions to mitigate the risks. By grouping risks in this way, the Ministry is able to consider risks to PGP as a whole as well as to the respective programmes and partners.

5.33
The Ministry also showed us a new tool called a "Strategic Journey Map" (the map). The Ministry told us that this tool would provide an overall view of programmes, bringing together information about the milestones, deliverables, and results from all programmes within PGP. We were told that the map would help the Ministry to track, evaluate, and communicate what the programmes are achieving.

5.34
The Ministry is confident that the map will allow it to track progress and communicate results to the public. The Ministry expects the map to be in place by June 2015. The intent of the map is promising.

5.35
However, the Ministry should have considered making these improvements sooner.

5.36
Because of the potential of PGP to increase the economic and sustainable growth of primary industries, the Ministry must prioritise strengthening the strategic management of PGP.

Sharing results with the public

5.37
In July 2013, the Minister for Primary Industries wanted to show PGP's value by increasing its transparency and accountability. One Ministry response to this was to share information publicly about the progress of programmes and what they are achieving. The Ministry did this by providing public reports on its website.

5.38
From a transparency perspective, the move to report publicly is beneficial. However, for some programmes that started in 2010, it was three years before information about them was shared with the public.

5.39
We reviewed how well the Ministry shares information with the public. Public reports on the Ministry's website contain a range of material, including:

• financial expenditure information;
• results from product and prototype testing;
• information about marketing and communication work;
• health and safety information; and
• information about the progress of programmes.

5.40
The information the Ministry currently reports publicly is prepared by the relevant industry partner based on quarterly reports to the respective PSGs. In our view, these reports are unsuitable for the public because they have been prepared for people with specialist knowledge of the relevant programmes. Public reporting needs to be simpler and more readily understandable to appropriately inform members of the public about the performance of PGP programmes.

5.41
The Ministry has not yet reported publicly about the achievements of PGP as a whole. The Ministry expects that the portfolio management work it has under way will enable it to do this over time.

page top