Part 5: Monitoring and reporting
5.1
In this Part, we discuss:
- how well public organisations are providing information that supports broader public accountability for each initiative;
- how well public organisations are reporting the outputs, impacts, and outcomes delivered for each initiative;
- the progress that each initiative has reported to date; and
- how reporting could be improved.
5.2
Performance reporting is the reporting that public organisations do to explain how they have spent public money and what they have achieved. The information public organisations include in performance reporting can be used to hold them to account, in particular for the difference that they have made in terms of improved outcomes for communities.
5.3
Performance reporting, including reporting from providers, is also an important management control. Public money was invested in these initiatives for a specific purpose. In our view, how they are progressing should be the focus of reporting to Parliament, the public more generally, and the specific groups the initiatives are there to benefit.
5.4
We wanted to understand how well positioned public organisations are to monitor and report on what the initiatives have achieved. We expected to see:
- regular and meaningful reporting of achievement towards the outputs, impacts, and outcomes sought, using formats appropriate for different stakeholders; and
- reporting that shows what has been spent, what has been achieved, and, when applicable, why results differ from targets.
Summary of findings
5.5
Although we heard that the four initiatives we looked at have made a positive difference, we did not see this adequately reflected in performance reporting. The public organisations involved have taken some steps to capture what has been achieved by each initiative but, currently, reporting does not capture the full range of information relevant to being held accountable for the public money invested.
5.6
These concerns are not unique to these initiatives. Improvement is needed across the public sector in the way that public organisations report on their performance. We encourage the public organisations responsible for these initiatives to provide better information about what has been achieved by using performance information they already hold, or information they plan to collect, such as from planned evaluations.
5.7
We were encouraged by some of the reporting practices that we saw. In an effort to produce a richer picture of performance, we saw public organisations:
- using videos to record rangatahi describing their experiences of being part of He Poutama Rangatahi in their own voice; and
- giving iwi the option to report verbally on progress of their Whānau Engagement contract.
5.8
We encourage innovation in reporting practices where they provide meaningful information in an accessible way.
Regular reporting requirements should focus more on impacts and outcomes
5.9
Most of the reporting requirements for iwi, other providers, and MABx clusters are focused on what activity there has been during the reporting period. This type of reporting is useful as a management control because it shows whether providers are meeting the terms of their contracts or funding agreements. However, it does not show the full range of impacts and outcomes that participants have experienced or that are ultimately sought from the funding.
5.10
The public organisations involved are starting to introduce different ways of reporting, such as verbal reporting, to capture richer information. We encourage public organisations to continue making improvements to reporting that allows them to fully understand what is being achieved through these initiatives, and what barriers iwi, providers, and landowners face.
He Poutama Rangatahi reporting from providers includes some richer information
5.11
Funding agreements between the Ministry of Social Development and He Poutama Rangatahi providers include a mixture of the detailed outputs, impacts, and outcomes that will be measured for each provider. He Poutama Rangatahi providers report monthly to the Ministry on these.
5.12
Some measures are based on participation – such as the number of rangatahi that are enrolled, the number that complete training, and the number that transition into education, training, or employment.
5.13
Other measures are more specific to individual providers and the programme they offer. Those include measures such as the qualifications, cultural skills, and work readiness skills rangatahi have gained.
5.14
Some providers felt that the monthly reporting is of limited value because it typically takes longer than one month for many rangatahi to show significant change. This does not mean rangatahi are not making progress, but based on the measures in the reporting it can appear that there has been no change since the previous month.
5.15
The Ministry for Social Development does collect some richer information from providers. In addition to the monthly reports, providers must complete a quarterly narrative report and six-monthly case studies. These reports have more detail about progress as well as the challenges that providers and participants have faced and how they have worked to overcome them. However, providers told us that completing these reports can be time-consuming.
5.16
Two He Poutama Rangatahi providers told us that video reports, where rangatahi tell their own story in their own voice, can offer more powerful storytelling about what individual rangatahi have achieved and the challenges they have had to overcome. We saw examples of this but providers told us they do not currently have much capacity to produce video reports. The Ministry of Social Development also told us that video reports by themselves would be difficult to store and to incorporate into more summarised internal reporting.
5.17
Providers told us that the reports they are required to produce do not provide a complete picture of what has been achieved. One reason for this is that employment is not immediately realistic for all participants. Some rangatahi have complex needs and will take longer to be ready for employment. For those rangatahi a good outcome is when they have engaged with the programme, built trust with the providers, and developed skills and confidence.
5.18
Another reason is that He Poutama Rangatahi programmes can provide a level of care that does not fit neatly into set hours. This support can continue after rangatahi have completed the programme. Providers told us their door is always open. Rangatahi can need support for navigating the system (such as help completing Inland Revenue forms) or if they lack confidence dealing with authority. Sometimes providers support rangatahi into one job and again later when they move into another. As a result, providers told us that they often do more than what is funded in their contract or included in reporting to the Ministry of Social Development.
5.19
Current reporting does not capture barriers to He Poutama Rangatahi achieving its outcomes. Providers told us that some He Poutama Rangatahi participants need more specialist support (such as support for issues with drugs and alcohol), but providers do not always have the knowledge or connections to access this support. There is often a lack of capacity for delivery of these types of services even if they have funding. We heard that it could be beneficial to have a social worker or navigator attached to each He Poutama Rangatahi programme to help rangatahi access other types of support where needed.
5.20
The success of He Poutama Rangatahi can also be affected by the availability of appropriate and sustainable employment for rangatahi who finish the programme. In some cases the local labour market does not have the right opportunities. For example, in a region such as Hawke's Bay where many jobs are in the horticultural industry, employment opportunities are often for seasonal rather than permanent jobs.
The Ministry for Primary Industries uses evaluations to gain a deeper understanding of what each cluster has achieved
5.21
Currently MABx clusters report monthly to the Ministry for Primary Industries on their activity and progress towards agreed milestones. The Ministry told us that the monthly reports from clusters do not easily show the status of each cluster against their agreed outcomes. The Ministry also told us that, until recently, it has been difficult to measure the overall status of MABx across all clusters because it was too early to track trends.
5.22
Cluster facilitators produce quarterly narrative reports for clusters as part of the framework and to provide richer information about what is being achieved. End-of-cluster evaluations are used to take a deeper look at what each cluster has achieved and what has been learned.
5.23
The reports that MABx clusters produce for the Ministry for Primary Industries do not show additional outcomes being achieved by each cluster. For example, one landowner talked about being able to kōrero and build connections with people they would not usually meet. This led to an investment opportunity not directly related to that cluster.
5.24
In another cluster, landowners saw their community come together and work towards a common purpose. We heard that the community learned and grew together in a way that is difficult to describe or measure. They provided learning opportunities for mokopuna who became involved with weather monitoring and crop trials that were part of the cluster's research. More specifically, cluster members learnt about land use and were able to pass this knowledge on to their whānau. These were outcomes that mattered to the cluster members.
5.25
Current reporting also omits some challenges that clusters can face in achieving their intended outcomes, especially after they have completed their MABx projects. This is because MABx was set up to cater to Māori landowners but other schemes can be difficult for Māori to access. We were told that clusters can find it hard to move from the Māori-focused approach that MABx uses to a system that does not specifically cater for Māori land. One person described the gap between these approaches as a "valley of death".
5.26
After completing MABx, clusters might need to apply for further funding and investment to allow them to implement the ideas or proposals they developed through MABx. Raising funds to invest has always been a challenge for Māori-owned farms and land trusts due to their complex ownership. This can be even harder with a cluster, which is not a legal entity and therefore cannot enter into agreements for investment.
Reporting from providers for Te Ahu o te Reo Māori could include more about impacts and outcomes
5.27
Providers for Te Ahu o te Reo Māori report quarterly to the Ministry of Education, mainly on the number of participants taking part. Providers felt that the Ministry was not measuring the impacts on the experience of ākonga Māori hearing more of their language and culture in their schools. One provider also noted that the Ministry does not measure the quality of te reo Māori that teachers and other school staff are using after they have completed the training.
5.28
To attempt to address this, we understand that reporting from providers now includes comments and feedback from participants. Through this reporting, and from its regular engagement with school leadership and teachers, the Ministry has heard that teachers and staff are increasing their confidence in using te reo Māori in the classroom. As well as learning te reo Māori, participants have lifted their cultural competency and developed their relationships with their students' whānau. Some teachers have said that Te Ahu o te Reo Māori is the best professional development they have received. We understand that Te Ahu o te Reo Māori is currently oversubscribed.
5.29
We also heard that some participants had experienced additional benefits. For example, some Māori teachers had strengthened their connection with their culture through their participation in Te Ahu o te Reo Māori. Others had been able to learn more about the history of their local area because this was incorporated into the local aspect of Te Ahu o te Reo Māori by their provider.
5.30
The Ministry of Education reports quarterly to the Associate Minister of Education (Māori Education) to provide an update on the Supporting Māori Learners and Te Reo Māori Post COVID-19 package. These reports include more details about each initiative in the package, including Te Ahu o te Reo Māori. In our view, the additional reporting from providers, and to the Associate Minister, is a step in the right direction. We encourage the Ministry to continue to incorporate more information about the impacts and outcomes achieved from Te Ahu o te Reo Māori in its regular reporting.
The Ministry of Education uses verbal reporting to capture what has been achieved through Whānau Engagement
5.31
Reporting requirements about how and when each iwi will report to the Ministry of Education about Whānau Engagement have been jointly agreed by the Ministry and each iwi. These requirements are set out in individual funding agreements. The funding agreements also include the outputs, impacts, and outcomes sought by each iwi as well as customised milestones and deliverables.
5.32
The Ministry of Education has offered iwi the option to report verbally on their progress with Whānau Engagement. Both Ministry of Education staff and iwi we spoke to are supportive of verbal reporting where it has been used. As well as reducing the reporting burden on iwi, a verbal report is seen as a more powerful way of telling the story of the journey whānau have been on and can capture a broader range of impacts and outcomes being achieved. When verbal reporting is used, either the Ministry or the iwi will also prepare a written record of what has been reported.
5.33
Representatives of one iwi told us that, through Whānau Engagement, an additional outcome they have benefitted from has been making new connections with whānau, some of whom felt like they had been heard for the first time. Whānau Engagement has also provided an opportunity for iwi to become more connected with local schools and have more influence in the local education system.
5.34
The story-telling that can come from verbal reporting can be an effective way of describing barriers to achieving outputs, impacts, and outcomes. One barrier we were told about was that Whānau Engagement could be limited by how well the school system supports Māori tamariki. As noted in Part 3, the Ministry of Education recognises that the successful delivery of its initiatives for supporting Māori learners requires the education sector to work differently. Even if Whānau Engagement succeeds in getting tamariki to attend school, any benefits will be limited if the wider school system is not supporting good educational outcomes for the tamariki.
5.35
Many tamariki also face broader issues that affect school attendance and engagement and which are outside the scope of Whānau Engagement. Iwi in rural areas told us they had added barriers including limited numbers of service providers and, for whānau, rising fuel prices making it harder to get to school. We also heard that iwi in rural areas sometimes missed out on Government funding for other initiatives, which in their experience often targeted larger iwi or communities. In some cases, iwi did not feel the funding they were receiving for Whānau Engagement was enough to meet the need in their communities. Where available, other services they provide were picking up what Whānau Engagement does not cover, such as Whānau Ora services.
Public organisations need to strengthen their accountability to the public for these initiatives
Public reporting does not show how much has been spent on each initiative
5.36
Public reporting on where public money has been spent is an important aspect of public accountability. Of the four initiatives we looked at, only one has had its budgeted and actual spending disclosed in public accountability documents.12
5.37
He Poutama Rangatahi has been treated as a stand-alone initiative in Budget documents, with its own appropriation. This means that it is possible to see how much funding was allocated to He Poutama Rangatahi and how much was spent. For the 2021/22 financial year, the Ministry of Social Development's annual report shows that about $44 million was appropriated but only about $20 million was spent. The Ministry explained that this underspend was due to delays in signing up new contracts because of the impact of Covid-19 on providers, as well as an earlier underspend transferred from the Ministry of Business, Innovation and Employment. However, the annual report does not include an explanation for this difference and in our view could provide more transparency about this.
5.38
The other three initiatives have less transparency. They have each been funded as part of a larger package and it is not possible to know from public documents how much funding has been allocated to and spent for each individual initiative – or even how much was spent on the larger package that each initiative was part of.
5.39
Other work by this Office suggests that this is a systemic problem across public sector initiatives and is not particular to the initiatives we looked at for this audit. Better reporting of outcomes is critical for providing transparency and accountability about how public funds are being spent. It would also help the government and public organisations make good decisions about where to invest public funds.
Public reporting does not reflect the positive impact of He Poutama Rangatahi
5.40
We heard many accounts of the positive impact He Poutama Rangatahi is having on young people. The Ministry of Social Development also has evidence that He Poutama Rangatahi is more successful than other ways of supporting young people into education, training, or employment. However, this success is not reflected in internal reporting or reporting to the public.
5.41
Internal reporting by the Ministry of Social Development is focused on spending, the number of contracts with providers, the number of enrolments, and the number of rangatahi who have moved into education, employment, or training. In our view, this is useful as management information but provides limited information about what is being achieved.
5.42
The Ministry of Social Development's data shows that 57.5% of rangatahi who have enrolled in a He Poutama Rangatahi programme and for whom there is a known outcome have transitioned into employment, education, or training. This is a much better result than has been achieved by other interventions targeting the same issues.13
5.43
The Ministry of Social Development estimates that after all currently enrolled rangatahi have completed their He Poutama Rangatahi programme, as many as 70% will have moved into employment, education, or training. The Ministry estimates that up to 2000 rangatahi are still enrolled in a programme, including those receiving ongoing support, and who are not yet ready to transition.
5.44
The performance measure that the Ministry of Social Development reports publicly, however, is about the number of young people enrolled in programmes funded by He Poutama Rangatahi and not the results that are being achieved. This measure, and its expected and actual results for the last two financial years, is shown in Figure 3.
Figure 3
Publicly reported performance measures for He Poutama Rangatahi
| Performance measure – Number of young people supported onto education, training, or employment pathways | 2021/22 | 2022/23 |
|---|---|---|
| Expected performance (in the annual Budget) | 2000 | 2000 |
| Actual performance (in the Ministry of Social Development's annual report) | 2174 | n/a |
5.45
Figure 3 shows that the Ministry of Social Development engaged more young people than expected in programmes funded by He Poutama Rangatahi, but these numbers alone do not show the impact on the young people involved.
5.46
In its annual report for the year ended 30 June 2022, the Ministry of Social Development did provide some additional information about what has been achieved with He Poutama Rangatahi. This included publishing that around 1200 young people enrolled in He Poutama Rangatahi subsequently moved into education, employment, or training during the year. However, this additional measure only gives limited information because the Ministry has not published any comparative or expected results.
5.47
Qualitative information about impacts and outcomes can also support accountability. The Ministry of Social Development also included a narrative about He Poutama Rangatahi in its latest annual report. This provided more contextual information, but was focused on describing the support that is offered by one provider that offers a career development programme rather than describing the outcomes experienced by participants.
The Ministry for Primary Industries has developed a reporting framework for the Māori Agribusiness Extension Programme
5.48
The Ministry for Primary Industries has designed a Results Monitoring Framework to monitor and evaluate MABx against the short-, medium-, and long-term outcomes sought (outlined in Part 2). The Ministry has also applied this framework at a cluster level for regular reporting about cluster performance but for most clusters it is too early to measure the medium- and long-term outcomes.
5.49
The Ministry for Primary Industries' annual reporting on MABx is focused on the number of clusters set up each year. This measure is shown in Figure 4.
Figure 4
Publicly reported performance measures for the Māori Agribusiness Extension Programme
| Performance measure – Number of new clusters | 2019/20 | 2020/21 | 2021/22 | 2022/23 |
|---|---|---|---|---|
| Expected performance (in the annual Budget) | Minimum 2 | Minimum 6 | Minimum 6 | Minimum 6 |
| Actual performance (in the Ministry for Primary Industries' annual report) | 6 | 23 | 7 | n/a |
5.50
From Figure 4 we can see that the Ministry for Primary Industries has exceeded its expected performance for setting up new clusters every year so far. However, this information does not show what progress those clusters have made or what outcomes they have achieved.
5.51
The Ministry for Primary Industries has included qualitative information about MABx in its annual reports. This has included case studies of two clusters and provides some insights about the impacts experienced by some clusters. The Ministry's latest annual report also included an explanation of how MABx is helping the Ministry make progress in its key result area of Strengthening relationships with Māori. As more clusters reach the end of their MABx funding, we encourage the Ministry to include more information in its annual reports and consider other ways of reporting publicly about the outputs, impacts, and outcomes those clusters have achieved.
The Ministry of Education has not measured the impacts and outcomes for Te Ahu o te Reo Māori
5.52
The public reporting for Te Ahu o te Reo Māori not only does not reflect the difference being made, but also does not give an accurate indication of the performance that is being measured. The main measure the Ministry has reported on is the number of people who participated in Te Ahu o te Reo Māori each year.
5.53
Staff from the Ministry of Education acknowledged to us that its current measures are not adequate to capture the full impact of this initiative. Some providers we spoke to agreed. Providers recognised that it is hard to measure impact. However, some had anecdotal evidence that the training was having a wider impact, as noted above, and thought the Ministry should put more effort into capturing this.
5.54
The performance measure reported on publicly for Te Ahu o te Reo Māori is to maintain or improve the number of people who participated during the year. In its annual report for the 2021/22 financial year, the Ministry of Education states the number of people who participated for 2020/21 (1054) and for 2021/22 (6190) and uses this increase as a basis to state that it has achieved its expected performance.
5.55
This does not accurately reflect that 2020/21 was a pilot stage, and the large increase in 2021/22 was due to the initiative expanding from four regions to the whole country.
Performance of Whānau Engagement has not been publicly reported
5.56
We have not seen any publicly available information about performance expectations or what the Ministry of Education has achieved through Whānau Engagement. This is a small initiative in the context of public spending on education and we would not necessarily expect to see it reported on separately. Nevertheless, Whānau Engagement might be seen as a significant initiative, particularly by iwi, because of the way that the Ministry has engaged with iwi. We heard positive feedback from iwi about this.
5.57
The Ministry of Education's quarterly reports to the Associate Minister (see paragraph 5.30) include more detail about what is being done with the Whānau Engagement funding and some of the effects it is having on ākonga and whānau. One impact mentioned is improved attendance rates for ākonga who are being supported by iwi. In our view, the Ministry should consider whether the information in these reports could be made available to the public.
| Recommendation 3 |
|---|
We recommend that the Ministry of Social Development, the Ministry for Primary Industries, and the Ministry of Education improve the way they measure and publicly report on these initiatives so there is:
|
Evaluating these initiatives will help to identify the full range of impacts and outcomes being achieved
5.58
Reporting by providers is a management control used to ensure that providers are meeting the terms of their contract or funding agreement. This reporting does not produce information about what is being achieved across each initiative as a whole, progress towards longer-term outcomes, or how each initiative is contributing to more strategic outcomes, such as outcomes at an organisation or sector level.
5.59
As discussed in Part 3, the three public organisations each evaluated or reviewed the initiatives at an early stage. These evaluations were done to see how well the design was working on a small scale before expanding the initiatives.
5.60
Now that the initiatives have been in place for some time, the responsible public organisations need to carry out further work, which might include evaluations to see how well each initiative is achieving its intended impacts and outcomes and to identify whether any changes are needed. This should be proportionate to the investment made.
5.61
The Ministry of Social Development is carrying out a full evaluation of He Poutama Rangatahi. This will look at how well He Poutama Rangatahi is achieving broader outcomes such as sustained employment and improved social outcomes.
5.62
The Ministry for Primary Industries will include its approach for measuring achievement of medium- and longer-term outcomes in its Results Monitoring Framework for MABx. We understand that the Ministry is planning to carry out a review of MABx, including its processes and whether some of the most advanced clusters are on track to achieve outcomes. This should be complete by mid-2023. The Ministry also has a longer-term evaluation plan for MABx.
5.63
The Ministry of Education is currently working through an evaluation process for the initiatives from its Supporting Māori Learners and Te Reo Māori Post COVID-19 funding package, which includes Te Ahu o te Reo Māori. We understand that the Ministry has engaged a Māori language education and research consultancy to design and conduct an evaluation of Te Ahu o te Reo Māori. The first phase was completed in December 2022. The Ministry expects that the evaluation findings will provide evidence of the impact of Te Ahu o te Reo Māori on students and student outcomes. We encourage the Ministry to ensure that it includes an assessment of the broader outcomes being achieved over time and as more people complete the training and can apply it in their schools.
5.64
We have not seen any plans for how the Ministry of Education intends to measure how well it is achieving its intended impacts and outcomes for Whānau Engagement. In our view, there would be a benefit in the Ministry of Education evaluating Whānau Engagement, particularly to look at the effectiveness of the way that the Ministry engaged with Māori for this initiative.
5.65
We encourage the Ministry of Social Development, the Ministry for Primary Industries, and the Ministry of Education to each publish the evaluation reports about these initiatives after those evaluations are complete.
12: For example, the Estimates of Appropriation that are part of the Government's annual Budget and the annual report for the relevant public organisation.
13: An early evaluation of He Poutama Rangatahi analysed Ministry of Social Development data as a comparator, and estimated that over a year, 16-23% of rangatahi who were not part of He Poutama Rangatahi moved off a benefit into education, employment or training. This figure includes rangatahi who received other forms of assistance from Government agencies, and those who received no assistance.