Part 4: Aims of the Initiatives
4.1
In this Part, we discuss what the Initiatives are intended to achieve and Te Puni Kōkiri's approach to measuring performance and progress towards the aims. At the end of the Part, we make some observations.
Performance measurement approach
Whānau Ora Outcomes Framework
4.2
Figure 2 shows the outcomes framework that Te Puni Kōkiri prepared. Te Puni Kōkiri considers that this framework describes the high-level outcomes for the first phase of the Initiatives. Te Puni Kōkiri told us that two outcomes in the framework represent the taskforce's six whānau outcome goals (see Figure 1). The outcomes are:
- Individuals experience positive cultural, social, and economic outcomes.
- Whānau experience positive cultural, social, and economic outcomes.
4.3
A simplified version of the framework was included in guidance given to the provider collectives on how to prepare their project plans to deliver whānau-centred services (dated January 2012).8 Neither version was published or widely shared. Te Puni Kōkiri told us that it did not prepare the framework to publicise the aims of the Initiatives. Te Puni Kōkiri told us that it based its performance measurement approach on the outcomes framework.
4.4
Te Puni Kōkiri has now prepared a new Whānau Ora Results Commissioning Framework for measuring the commissioning agencies' effectiveness. Te Puni Kōkiri told us that the commissioning agencies are to report on six whānau outcome goals, which are the same as the ones in the taskforce's framework (see Figure 1).
4.5
The commissioning agencies will choose their own short-term, medium-term, and long-term measures to report against each goal. Te Puni Kōkiri told us that it wants a more balanced mix of measures for whānau and government priorities. It expects that the medium-term and long-term measures will tend to align with government priorities for educational and economic achievement.
Measures of provider collectives' performance
4.6
Te Puni Kōkiri picked two topics (infant immunisation and early childhood education) to report on the provider collectives' performance from 2011/12 in its annual reports to Parliament. Te Puni Kōkiri told us that it picked the measures to show how the provider collectives had contributed to the Government's Better Public Service targets.9
Figure 2
Whānau Ora Outcomes Framework, 2010
| Whānau | Whānau Ora | |
| Whānau are self-managing and empowered | ||
| Individuals experience positive cultural, social, and economic outcomes | Whānau experience positive cultural, social, and economic outcomes | |
| Whānau have the collective capacity and strength to pursue their aspirations | ||
| Whānau have goals and plans and are skilled at accessing social support | ||
| Providers are effective in delivering Whānau Ora | Whānau are strengthened and take responsibility for their well-being and development | |
| Holistic service design leads to measurable whānau ora gains | Strengths-based service delivery leads to measurable whānau ora gains | |
| Whānau Ora navigators support whānau to develop plans and access appropriate services | ||
| Whānau Ora navigators engage eligible whānau and identify their needs | ||
| Providers have adequate service delivery capability | ||
| Government agencies are effective in designing and implementing Whānau Ora | Effective contracting and reporting processes enable effective whānau ora service provision | |
| Whānau ora provider selection process is efficient and results in the engagement of high-quality providers | ||
| Quality agency and whānau ora provider relationships | ||
| Quality inter-agency collaboration | ||
| Whānau Ora Governance Group and Regional Leadership Groups provide strong leadership to overall initiative | ||
| Whānau Ora approach is well designed | ||
| Effective and appropriate policy settings | ||
Source: Te Puni Kōkiri.
4.7
Te Puni Kōkiri used two versions of each measure, and both versions were reported on for 2012/13 in different reports (see Figure 3). Only measure 3 (which was replaced) matched the relevant Better Public Service target.10
Figure 3
Measures of provider collectives' performance, 2011/12 to 2013/14
| Measures | 2011/12 | 2012/13 | 2013/14 |
|---|---|---|---|
| Early childhood education | |||
| 1. The percentage of whānau with children enrolled in early childhood education | 67% of whānau indicated that they had improvements in attendance at early childhood education (national rate: 95.7%)^ | 67% of whānau engaged in the Initiatives noted improvements in their attendance in early childhood education^ | Replaced by indicator 2 |
| 2. The percentage of engaged whānau supported to access early childhood education | - | 95.7%* | 95.9%* |
| Immunisation | |||
| 3. The percentage of engaged whānau with children whose vaccinations are up to date. (The measure is the eight-month immunisation rate from a sample of general practices and primary health organisations that are members of a provider collective.) | - | 85%^ | Replaced by indicator 4 |
| 4. The percentage of engaged whānau supported to access immunisation | - | 85%* | 83.5%* |
Notes: Data marked with ^ was reported in Te Puni Kōkiri's annual reports for the years ended 30 June 2012 and 2013 (see www.tpk.govt.nz).
Data marked with * was reported in Vote Maori Affairs Report in relation to the Whānau Ora-based Service Development Multi-class output appropriations (non-departmental appropriations) for the years ended 30 June 2012, 2013, and 2014 (see www.treasury.govt.nz).
4.8
It is not clear whether the results reported for the early childhood education measure related to increased enrolment rates, improved attendance rates for children who were already enrolled, or both.
Regional plans and population indicators
4.9
In the last year of their original terms (2012/13), the 10 regional groups had to prepare plans and report quarterly to Te Puni Kōkiri on the implementation of those plans.11 Te Puni Kōkiri told us that the regional groups were to identify priorities for co-ordinating and investing in activities that would help improve whānau outcomes.
4.10
In the second half of that year, Te Puni Kōkiri required each group to prepare a population outcomes framework for their region and identify matching indicators. This involved bringing members from the regional groups together for training in a model called Results-Based Accountability.12
4.11
Te Puni Kōkiri expected that the frameworks would enable the regional groups to:
- identify priorities for whānau in their region;
- set priorities; and
- check what progress was being made for whānau in their region.
4.12
Each of the regional groups identified three or four population outcomes.13 Half of the regional groups identified one or more indicators for each outcome before the groups were disbanded.14 Most of the indicators were measures about individuals (such as low-birthweight babies or family violence incidents) that could be reported on easily because the data is readily available from official sources.
4.13
Te Tairāwhiti regional group's outcomes framework was prepared first and used to trial the Results-Based Accountability method. Te Tairāwhiti's framework had more detail than the other regions' frameworks. It included information about why the framework was important, how it should be used, and how people and organisations in all sectors should work together to achieve the agreed outcomes. The Appendix summarises Te Tairāwhiti group's framework and shows which parts of the framework the other regional groups completed.
4.14
Te Puni Kōkiri told us that the regional groups were expected to monitor their indicators. In reality, it would have been difficult for the groups to notice changes during their terms because some data is collected annually (such as students' achievement rates) or less often (such as changes in life expectancy, which is collected during the census). Data for other indicators, such as for employment rates, is collected and reported quarterly.
4.15
It is not clear whether the regional frameworks and indicators (where they exist) have any role in the second phase of the Initiatives or will be taken up by anyone else.
Reporting on progress towards the aims of the Initiatives
4.16
Te Puni Kōkiri collected and collects information on performance through written reports sent by:
- regional groups, while they existed;
- whānau; and
- provider collectives and research collectives (groups of researchers).
4.17
Te Puni Kōkiri uses the information, as well as the results of any other evaluation activities, to provide regular reports on the Initiatives to the Minister for Whānau Ora.
4.18
Te Puni Kōkiri reports on the Initiatives to Parliament and the public in several ways. Information is published on its website (such as in occasional reports, fact sheets, and newsletters) and in annual reports. It also helps to draft reports that the Minister presents to Parliament. However, these separate reports do not give a comprehensive report on progress towards the Whānau Ora Outcomes Framework.
Our observations
4.19
Government agencies need to be able to explain what results are expected – or hoped for – and achieved from spending public funds. We could not get a consistent explanation of the aims of the Initiatives from the joint agencies or other people that we spoke to.15
4.20
In our view, Te Puni Kōkiri's failure to prepare an outcomes framework with SMART16 aims, and publicise it, has contributed to the range of explanations and opinions we were given about what the Initiatives were intended to achieve and about how effective they have been.
4.21
Te Puni Kōkiri's measures for reporting on provider collectives' performance in its annual reports (about immunisation and early childhood education) are inconsistent and confusing. They do not match the Better Public Service targets, and we found it difficult to relate them to the Whānau Ora Outcomes Framework.
4.22
The regional plans were prepared too late to be useful. It is likely that some regional groups did not select matching indicators because they were going to be disbanded. Even if there had been time to detect changes in some of the indicators, it would not have been feasible to credit any improvements to the provider collectives' actions.
4.23
Te Puni Kōkiri and the regional groups chose measures and indicators that were measures of individual, rather than whānau, achievement. In 2011, Professor Sir Mason Durie had proposed that targets for whānau achievement could be defined and set. Those targets could be, for example, that by 2015:
- 60% of whānau are financially literate;
- 75% of whānau are health literate;
- 60% of whānau are e-literate;
- 80% of whānau are succeeding in programmes of learning;
- 60% of whānau are fluent speakers of Māori; and
- 40% of whānau are "estate" literate.17
4.24
In June 2016, the last Programmes of Action will be completed. At that time, Te Puni Kōkiri will have a full set of reports – from provider collectives, whānau, and research collectives – that it can use to report on the first phase of the Initiatives.
4.25
There should have been clearly understood aims, roles, and responsibilities, which should have led to clear accountability and good reporting. So far, the situation has been unclear and confusing to many of the public entities and whānau. This is unacceptable.
8: Te Puni Kōkiri (2012), Whānau Ora: A guide to the development of a Whānau Ora approach.
9: Information on the targets is available at www.ssc.govt.nz.
10. The relevant Better Public Service targets were:
- By December 2014, 95% of eight-month olds will be fully immunised and this rate will be maintained through to June 2017.
- In 2016, 98% of children starting school will have participated in quality early childhood education.
11: Te Puni Kōkiri reported on this in the Statement of Service Performance in its 2012/13 annual report.
12: In the Results-Based Accountability model, there are two types of accountability (population and performance) and three types of performance measures (How much did we do? How well did we do it? Is anyone better off?). For more information, see www.raguide.org.
13: Examples of outcomes were culturally connected, aroha, self-determining, guardians, waiora, manaakitanga, secure, economically independent, healthy, and safe. Each regional group defined what it meant by each outcome, so outcomes that were common between the regional groups did not necessarily have the same definitions.
14: Examples of indicators were employment/unemployment rates, life expectancy, recorded family violence offences, te reo Māori speaking rates, and participation in local decision-making bodies.
15: One provider's chief executive told us that the joint agencies took different approaches to discussing the Initiatives, which sometimes made matters confusing or more complex for providers. He explained that health sector agencies wanted providers to focus on people with high health needs and chronic health issues who could have services "wrapped around" them. He said that Te Puni Kōkiri's officials focused on realising Māori potential, and the Ministry of Social Development focused on how providers could show that they were making a difference in delivering contracted services (that is, Results-Based Accountability).
16: By this, we mean that the framework is specific, measurable, assignable, realistic, and time related.
17: The example targets were published at www.tpk.govt.nz.