Part 5: Using information to assess outcomes

5.1
In this Part, we cover:

• the information that the Ministry and DHBs hold and how they use it;
• efforts under way to improve how information is used; and
• how information can be used better to understand people's experiences and improve services.

Summary of our findings

5.2
The Ministry and DHBs have a lot of data about mental health services. The Ministry uses the data to report mainly on what services are provided and who is providing them. DHBs and other providers use the data to understand service performance through a set of indicators that they have been developing since 2006.

5.3
Both the Ministry and DHBs are seeking to improve how they use information. The Ministry has started to collect information about outcomes for people, and is intending to collect data and improve reporting on the use and quality of discharge planning from 2017/18. DHBs are working to use indicators more effectively to inform improvements to services and outcomes for people.

5.4
There is more for the Ministry and DHBs to do to make better use of information to understand what influences outcomes for people, including the effectiveness of discharge planning, and improve their services. More work is needed to:

• establish and use solid outcomes measures;
• systematically gather and use feedback from people using mental health services and those supporting them;
• build capability to use data and information; and
• address some lack of trust and confidence in the quality of the available data.

5.5
In our view, the Ministry and DHBs can gain a greater understanding of how to improve services for people by understanding the patterns and trends in people's experience of services. We show some examples in this Part. We also introduce the concept of viewing a person's contact with mental health services as a timeline of interactions. This is a concept developed by people working in the health sector, which we refined to highlight its potential uses. We have shared this with people working in the health sector.

Using data about mental health services to report on and understand service performance

5.6
The Ministry collects mental health data from DHBs and non-governmental organisations providing mental health services. Information about consumer satisfaction with mental health services is also collected by the Ministry and DHBs.

Collection and use of data about provision of mental health services

5.7
Data about specialist mental health services are collected by the Ministry in the Programme for the Integration of Mental Health Data (PRIMHD) system. The Ministry uses PRIMHD to report mainly on what services are provided and who is providing them.

5.8
The Ministry also collects information through the DHB non-financial monitoring framework. For 2016/17, the Ministry's DHB non-financial reporting framework has three performance measures about mental health services. One of them, Improving mental health services using transition (discharge) planning and employment, is about discharge planning.

5.9
DHBs and other providers use PRIMHD data to understand service performance through the New Zealand Mental Health and Addiction Services Key Performance Indicator Programme (the KPI Programme). The KPI Programme is a provider-led initiative that began in 2006. It is primarily a benchmarking forum whose purpose is to systematically analyse and use service and outcome data to inform service development and improve the outcomes for people using mental health and addiction services and their families.

5.10
Through the KPI Programme, DHBs have designed a framework of key performance indicators and associated stretch targets for adult mental health and addiction services⁵ that represent good performance. The framework includes just over 60 indicators. Results were published in May 2016, covering the three years from 2012/13 to 2014/15. Selected results are also available on the KPI Programme website⁶ for 2015/16 and 2016/17 (year to date).

5.11
DHBs also have access to an interactive web-based tool that allows them to examine their own KPI Programme results. Currently 12 of the 60 indicators can be examined using the tool.

5.12
Figure 6 shows the results of six of the KPI Programme indicators relating to the discharge of people with mental health problems from acute inpatient units and follow-up contact with them by community mental health services. For each KPI we show, for the three years from 2012/13 to 2014/15, the weighted average value and the highest and lowest value for all DHBs. Our observations from these results are that:

• the average performance of DHBs against the indicators has remained reasonably static in the three years, and the performance of DHBs has varied; and
• the average performance of DHBs did not meet the targets for four of the indicators (see the darker shading in Figure 6), and is well below the targets for percentage of people followed up within seven days of a discharge and the percentage of discharges with qualifying Health of the Nation Outcome Score (HoNOS) assessments (see paragraphs 5.23-5.24).

5.13
Participants from DHBs, NGOs, and their "strategic partners" are involved in benchmarking forums twice a year with the aim of understanding variations in performance, and learning from each other about service improvements and practices to improve outcomes for people using mental health services. For the last 12 months, the KPI Programme has changed its approach to focus on one indicator for all DHBs (and two indicators at a sub-national level, focusing on northern and southern priorities). This approach is intended to increase collective learning on how to improve performance.

5.14
At the DHBs we visited, we heard examples of how information, such as some of the indicators from the KPI Programme and case files of people admitted to an inpatient unit, was analysed to identify trends and service improvements. We also heard examples of how people's progress, such as length of stay as an inpatient and contact with community mental health services, was monitored.

Satisfaction of people using mental health services

5.15
The Ministry collects and publishes consumer satisfaction information. Since 2006/07, DHBs have been carrying out an annual national mental health consumer satisfaction survey. Survey participants have all received specialist mental health services. In 2014/15, 14 of the 20 DHBs participated in the survey. The Office of the Director of Mental Health reported that 82% of respondents either agreed or agreed strongly with the statement "overall I am satisfied with the services I received".

5.16
In our view, the results that are published do not contain enough information to give a reliable indication of the satisfaction levels of people using mental health services. The response rate for the survey was not provided and there was no breakdown by DHB.

Figure 6
Summary of district health boards' results against six key performance indicators, 2012/13-2014/15

	Year	Weighted average (all DHBs)	Lowest (all DHBs)	Highest (all DHBs)
Average length of acute inpatient stay (Target: 14-21 days)	2012/13	18.2	12.1	29.6
	2013/14	18.1	11.4	27.9
	2014/15	17.4	12.7	28.3
	Average	17.9	-	-
Percentage of discharges with qualifying Health of the Nation Outcome Score assessments (Target: 75-100%)	2012/13	57.9	12.2	90.4
	2013/14	58.1	12.8	94.3
	2014/15	59.0	28.9	94.3
	Average	58.3	-	-
Percentage of discharges for which community mental health contact is recorded in the seven days after discharge (Target: 90-100%)	2012/13	65.6	50.8	85.5
	2013/14	62.5	38.4	80.1
	2014/15	64.1	39.5	80.8
	Average	64.1	-	-
Percentage of discharges re-admitted to acute inpatient unit within 28 days of discharge (Target: 0-10%)	2012/13	15.6	6.3	28.4
	2013/14	14.7	6.4	21.4
	2014/15	15.9	8.7	32.4
	Average	15.4	-	-
Number of community treatment days provided for each person each quarter (Target: 10-20 days)	2012/13	8.5	6.0	11.2
	2013/14	8.3	6.0	10.6
	2014/15	8.1	5.6	12.2
	Average	8.3	-	-
Percentage of community service-user related time with client participation (telephone or face-to-face) (Target: 80-90 %)	2012/13	86.2	72.9	96.8
	2013/14	89.0	73.5	97.9
	2014/15	89.7	68.7	98.6
	Average	88.3	-	-

Source: www.mhakpi.health.nz.
Notes: We have not included data from Lakes DHB because we were told these were incorrect.
One person can have multiple discharges, because each time they are discharged is counted separately.

Efforts to improve how information is used

5.17
In our view, the Ministry is starting to collect and report more useful information. DHBs and other service providers, supported through the KPI Programme, have been looking at how they can improve their use of information over time. There is more for the Ministry and DHBs to do to understand what influences outcomes for people, including the effectiveness of discharge planning so that they can improve their services.

The Ministry is starting to collect and report more useful information

5.18
From 1 July 2016, the Ministry has started to collect data from some DHBs, in a Supplementary Consumer Records Collection to PRIMHD, on selected social outcomes indicators for people receiving services for mental health and whether they have a wellness plan in place. The social outcome indicators are accommodation status, employment status, and education and training status.

5.19
For 2017/18, the Ministry is modifying the discharge planning measure in the DHB non-financial reporting framework to include all age groups and with an expectation that 95% of people have a transition plan⁷ at discharge and 95% of those who have been in the service for a year or more will have a wellness plan.⁸ The measure will also expect DHBs to carry out file audits to determine the quality of the plans and report the results. The Ministry is introducing file audits in response to some of our findings from this audit. We support changing performance measures when doing so makes them more meaningful.

Making better use of information is an ongoing focus for the KPI Programme

5.20
In our view, the mental health sector has made progress in using information to improve service performance through the KPI Programme. The KPI Programme Strategic Plan 2015-2020 outlines how developing how the indicators are used is expected to better inform improvements to services and outcomes for people. The strategic plan focuses on three priorities:

• governance and leadership in the use of information to drive improvement, including through advocating for sector-wide improvement and engaging people using mental health services and their families;
• collaborative learning and performance improvement, including through focusing on improving a person's experience and sharing lessons and experiences across all those involved in a person's continuum of care; and
• increasing data capability to help the sector in improving the range and quality of data and information available for decision-making.

There is more to do to make better use of information

5.21
In our view, the Ministry and DHBs need to make better use of information to understand what influences outcomes for people, including the effectiveness of discharge planning, and make service improvements. More work is needed to:

• establish and use solid outcomes measures;
• systematically gather and use feedback from people using mental health services and those supporting them;
• build capability to use data and information; and
• address some lack of trust and confidence in the quality of the available data.

Solid outcomes measures need to be established and used

5.22
We acknowledge that data for some social outcome indicators have started to be collected recently, but in our view further work is needed to establish and use solid outcome measures and create a framework to demonstrate how activities such as discharge planning contribute to outcomes.

5.23
The sector uses the HoNOS outcomes tool,⁹ completed by clinicians, to assess the health and social functioning of adults with severe mental health problems. HoNOS improvements after admission to an acute inpatient unit are used as one of the qualifying criteria for discharging a person from the unit.

5.24
HoNOS has supporters and detractors. At one of the DHBs we visited, we were told that HoNOS did not contribute useful information to service improvements. We were told that it was not used consistently and was seen as a "tick box exercise", with poor reliability and a lack of training in how to use it. Some staff said they would like to use it better and that steps were being taken to improve its use. Others said they would like to have it scrapped. At another DHB, some staff were not committed to using HoNOS because it was not seen as an accurate representation of the treatment provided.

5.25
The Ministry told us that it is moving to outcomes-based commissioning for mental health services. The Mental Health and Addiction Commissioning Framework published in August 2016 provides guidance and direction for those who are responsible for commissioning care to improve outcomes for people with mental health and addiction issues. We support an outcomes-based approach to improving mental health initiatives.

Better collection and use of feedback is needed

5.26
DHBs seek feedback from people in different ways, but make limited use of this feedback to improve services. In particular, the views of people who have used mental health services and those supporting them, such as their family and GP, about their experiences of discharge planning and its effects are not systematically collected and considered. This means that the Ministry and DHBs do not have a good understanding of how people are involved in discharge planning and how well discharge planning is supporting better outcomes.

5.27
Those DHBs that provided us with information about how they seek feedback said that they used means such as surveys and "service user" forums. At one of the DHBs we visited, we heard from advocates that a Whānau hui gave families the opportunity to provide feedback about inpatient unit services. At another, we heard of opportunities for people who had used mental health services to provide feedback and saw an example of a change made as a result of feedback.

5.28
However, generally we found that systems to collect and use feedback were underdeveloped. Most DHBs relied on complaints, or the absence of complaints, as a measure of satisfaction.

5.29
Because people receiving treatment can be reluctant to complain about the staff treating them, some DHBs have begun to use anonymised real-time feedback by giving people access to an electronic tablet running a feedback application.

5.30
We also heard from patient advisors and family advisors that DHBs did not make the best use of them as a resource to bring a user perspective. Nearly every advisor we spoke to or who responded to our survey felt that their involvement was tokenistic. One DHB had not had a patient advisor for nearly two years.

Barriers to using data and information to improve services

5.31
Increasing the capability to use data and information is one of the priorities identified in the KPI Programme Strategic Plan 2015-2020. For staff in the DHBs we visited, capacity and systems make the effective use of information difficult.

5.32
Overall, there was limited use of data and information to inform service improvements at the three DHBs we visited. At one DHB we visited, we were told that there was limited evaluation of what mental health interventions work well because staff lacked the tools, time, and expertise. We also heard that cross-sector evaluation of what works was weak. At another DHB, we were told that there was a lack of capacity to use data to effectively monitor service delivery and make changes.

5.33
During our visit to one DHB, staff said that information systems did not support accurate data collection and that information was held in different places, making it hard for staff to gather data and get a complete picture of outcomes for people using services. Staff said that there was no way of tracking a person's care between services. At another DHB, we were told that information was hard to find. At another, we were told that there was no one place to check whether discharge plans were working.

Addressing a lack of trust and confidence in the quality of the available data

5.34
We heard a range of frustrations – and varying degrees of confidence or mistrust – about the reliability of the available data from the people throughout the sector that we talked to about how information could be used.

5.35
The Ministry and DHBs use data definitions and data quality checking to control the quality of the data in PRIMHD. However, these do not prevent discrepancies occurring. We also heard that some people felt what the KPI Programme indicators showed was disconnected from the reality they experienced.

5.36
For some, these issues devalued the data and indicators, and how they could be used.

Using information better to improve services and understanding

5.37
In our view, looking at the patterns and trends in people's experience of services will provide a greater understanding of how to improve services. In the remainder of this Part, we show this by looking at the data for two of the indicators from the KPI Programme and what these reveal. We also introduce work we have shared with the health sector on viewing a person's contact with mental health services as a timeline of interactions. Our work was based on the innovative thinking of an individual working in the mental health sector.

Patterns and trends in people's experiences

5.38
We looked at two indicators from the KPI Programme and analysed the data. The indicators related to:

• follow-up contact with people after their discharge from inpatient units; and
• people's re-admission to inpatient units.

5.39
In our view, the patterns and trends we highlight could be useful for the Ministry and DHBs in considering how to improve services.

Patterns and trends in people's experiences of re-admission to inpatient units

5.40
The indicator for re-admission shows the overall percentage of re-admissions to acute inpatient units within 28 days of discharge. We looked at the distribution of all re-admissions that occurred within 28 days of discharge from an inpatient unit, for the period 2011/12-2014/15. Figure 7 shows that almost half of all re-admissions occurred within nine days of discharge, and three-quarters within 17 days. In other words, most re-admissions occurred well before 28 days had passed.

5.41
Figure 7 excludes people who were re-admitted on the same day that they were discharged. This often happens when people given day leave from the inpatient unit get counted as a discharge and re-admission on the same day.

Figure 7
Profile of when re-admissions occurred for re-admissions between 1 and 28 days after discharge, 2011/12-2014/15

Source: Our analysis of Ministry of Health data.

5.42
We also looked at the profile of re-admissions by three-month periods for small, medium, and large DHBs and found that:

• the spread of rates is erratic from one quarter to the next;
• the rates for some DHBs (outliers) is as high as 100% in a particular quarter (meaning every single inpatient stay in that quarter would have been a re-admission); and
• there is no clear pattern of an ongoing decrease (or increase) in re-admission rates.

5.43
Small DHBs (which have the lowest numbers of people using acute inpatient services) are more prone to erratic swings in their re-admission rates than medium or large DHBs. Medium and large DHBs display progressively tighter distributions of re-admission rates from one quarter to the next, with less wide-ranging outliers.

5.44
Focusing in on the cohort of people described in paragraph 1.13 and needing acute mental health services, we looked at re-admissions for each person. Figure 8 shows re-admissions for each person treated in five small DHBs.

Figure 8
Distribution of re-admissions and all inpatient unit stays for people at five small district health boards

Source: Our analysis of Ministry of Health data.

5.45
The larger bubbles in the top three-quarters of Figure 8 represent people who have experienced many re-admissions during their inpatient stays. There is a relatively small number of these people. Most people have had fewer inpatient stays and no re-admissions.¹⁰ These are represented by the smaller bubbles at the bottom. We found a similar distribution pattern for all DHBs.

Patterns and trends in people's experiences of follow-up contact

5.46
The indicator for follow-up activity looks at the percentage of people who were contacted by the community mental health team within seven days of their discharge from an inpatient unit. The contact does not need to be in person, but some forms of social media contact are excluded.

5.47
Our analysis showed that follow-up rates for small DHBs display a similar picture to that of re-admission rates: movement is erratic and highly variable over time. Similar to the 28-day re-admission rates, the follow-up KPI displays a tighter distribution as we move from small to medium to large DHB groups. However, there is no clear observable trend for all three DHB groups.

Viewing a person's contact with mental health services as a timeline of interactions

5.48
Building on innovative thinking already happening in the sector, we used the data for the cohort of people described in paragraph 1.13 to construct timelines of people's contact with mental health services. We constructed these timelines for individuals and for groups of people. We took the concept of using visual timelines to understand people's interactions with mental health services, which was developed by people working in the health sector, and refined it to highlight its potential uses.

5.49
Currently, it is not always easy for clinicians to form a quick impression of a person's contact history (the details of which might be bundled together as part of various case notes). Timelines show a single picture of a person's contact history, providing clinicians with an intuitive mechanism for rapidly understanding patterns of contact, and can be adapted to focus on different types of contact, groups of people, or areas of the health service.

5.50
Viewing data from a person's perspective can also:

• help DHB clinicians and administrators to understand who is using their services and plan to meet their needs, including identifying service gaps; and
• enable identification and sharing of good practice between DHBs, and enable services between DHBs to be co-ordinated when a person moves, to help with their continuity of care.

5.51
Alongside this report, we have made more information available on our website (oag.govt.nz) about the concept of people's timelines showing their different types of contact with mental health services, and potential uses of it.

Recommendation 4

We recommend that the Ministry of Health and district health boards quickly make improvements to how they use information to monitor and report on outcomes for people using mental health services.

Recommendation 5

We recommend that the Ministry of Health and district health boards use the information from this monitoring to identify and make service improvements.

---

5: The KPI Programme also has indicators for children and forensic mental health service users. These are outside the scope of our audit.

6: See www.mhakpi.health.nz.

7: A transition plan is equivalent to a discharge plan.

8: A wellness plan is another term for a relapse prevention plan.

9: Developed in the United Kingdom by the Royal College of Psychiatrists' Research Unit between 1993 and 1996.

10: An inpatient stay counts as a re-admission when it occurs within 28 days after the most recent discharge and within the same inpatient unit. This means that, for example, a person can have more than one inpatient stay but no re-admissions.