Part 5: Measuring case management performance

Accident Compensation Corporation: Using a case management approach to rehabilitation.

5.1
In this Part, we set out our assessment of the extent to which ACC uses performance information to improve the effectiveness and efficiency of its case management services, including for those claimants who experience greater than usual barriers when using ACC's services.

Summary of our findings

5.2
ACC has extensive performance information that is used to inform thinking about its future, but ACC has been slow to make specific service design and delivery changes in response to some of the information. ACC needs to be better informed about how well it is meeting its case management expectations.

5.3
ACC has clear performance expectations in terms of the outcomes of case management, the experience of claimants, the behaviour of case managers, and the policies and processes that case managers are to follow, but it needs to be better informed about how well it meets its expectations.

5.4
ACC gives less attention to how its different case management streams perform than how its teams, offices, and regions perform.

5.5
ACC has extensive customer satisfaction and other customer feedback information, including detailed and robust improvement suggestions from a consumer perspective.

5.6
ACC has an accurate perspective of its customers' experiences of ACC.

5.7
ACC has room to further use its extensive performance and customer experience information to improve service delivery and design, particularly for longer-term claimants with complex needs who do not reach the serious injury thresholds.

Setting performance expectations

5.8
ACC has clearly recorded its expectations for the results of case management.

5.9
The overall outcome that ACC is seeking to achieve from case management is more effective rehabilitation of injured people. ACC sees effective rehabilitation services as those that deliver quality and cost-effective services that achieve appropriate outcomes.

5.10
Figures 6 and 7 describe ACC's case management expectations for the organisation, local offices, and case managers.

5.11
The main expectation of case managers is about how well they engage with claimants. However, the main expectations of local offices involve claim duration measures.

5.12
ACC should use a more balanced set of performance expectations and measures at all levels because the actions of case, team, and local office managers all influence cost and customer experience. One ACC staff member described the balance that was required as staff needing to "think like a client but act like an owner".

5.13
A more balanced set of key performance measures might include both a claimant focus and some form of cost indicator as minimum. Some form of complexity indicator could also be a useful measure to include.

5.14
Without a more balanced set of performance measures, there is the risk that single measures could drive unintended behaviour. For example, case managers can perform highly against their performance measure of engaging well with people by concentrating on less complex, shorter-term, or otherwise easier cases. However, doing this could mean that an office performs poorly on its duration measures and that overall costs are higher than they should be.

Figure 6
Expectations for case management at an organisation level

	Organisation-wide expectations
	Functional expectations	Claimant satisfaction expectations	Claimant experience expectations
Sources	Statement of Intent 2013-2016		Code of ACC Claimants' Rights
Expectations	The average duration of weekly compensation days paid in the first 365 days is reduced to 84.6 days. The percentage of National Serious Injury Service goals achieved or exceeded increases to at least 71%.*	At least 68% of claimants are satisfied or very satisfied with the overall quality of service delivery.	Claimants have the right to: be treated with dignity and respect; be treated fairly, and have their views considered; have their culture, values, and beliefs respected; have a support person or persons; be effectively communicated with; be fully informed; have their privacy respected; and complain if they choose to do so.

* In 2012/13, ACC achieved or exceeded 69.9% of National Serious Injury Service goals.

Figure 7
ACC's expectations for case management by local offices and case managers

	Local office and case manager expectations
	Local office manager expectations	Case manager expectations	Customer Engagement Appraisal expectations
Sources	Claims Management Network Performance Objectives 2013-2014		Customer Engagement Appraisal
Expectations	The local office manager: reduces the average duration of weekly compensation days paid in the first 365 days; maintains the percentage of claimants receiving weekly compensation entitlements who have been rehabilitated within 70 days; maintains the percentage of claimants receiving weekly compensation entitlements who have been rehabilitated within 273 days; achieves 365-day net entry target; achieves 912-day net entry target; increases the percentage of National Serious Injury Service goals attained as achieved or exceeded – target 71%; and attains a local office average of "meets rating" or better for Customer Engagement Appraisals.	The case manager: attains an average of "meets rating" or better for Customer Engagement Appraisals; has Individual Rehabilitation Plans for 90% of claims; and completes the return to work indicator.	The case manager: understands claimant circumstances; addresses gaps or unclear information; proactively engages with relevant stakeholders; communicates with the stakeholders regarding entitlements, decisions, complaints, and reviews; sets realistic expectations and communicates responsibilities to relevant stakeholders; involves the claimant in the entitlement and rehabilitation decision-making process; records the agreed rehabilitation planning and decision-making, and whether this is likely to achieve the agreed outcome; implements the agreed rehabilitation/support actions; and monitors that planned rehabilitation/support is on track.

5.15
Care needs to be applied when having duration and other performance metrics at the level of an individual staff member, because of the potential effect of anomalous claims – such as a complex and large claim – on assessing an individual's performance. As one staff member told us, "complexity and duration generally run hand in hand". Such factors need to be taken into account when considering duration and cost metrics.

5.16
ACC has a case complexity indicator that it uses mainly to work out the case management stream that a claim should be assigned to. That indicator, or a similar indicator, could be used more for performance measurement.

5.17
As part of advice to the ACC Board about Shaping Our Future, it has been recognised that ACC's key performance indicators need to be more in line with expected claimant outcomes.

5.18
ACC has clearly recorded the policies and procedures that it expects case managers to follow and the competencies and behaviour that it requires of case managers. The policies, procedures, and required competencies emphasise effective communication and take into account cultural considerations.

Measuring how well expectations become reality

5.19
ACC gives less attention to how its different case management streams perform than how its teams, offices, and regions perform. ACC needs to understand better how well it is achieving the results it expects from case management, including the results achieved by each of the case management streams.

5.20
ACC measures its performance in rehabilitating people in line with the expectations set out in Figures 6 and 7. It measures:

how many people who have returned to work after an injury remain employed after a set amount of time;
claimant satisfaction;
the percentage of individual rehabilitation goals achieved for seriously injured people;
the number of days compensation is paid before people return to work or independence, and;
the number of long-term claimants who are rehabilitated.

5.21
As well as these measures, ACC is moving to having more of an outcome focus expected of its contracted providers of treatment and rehabilitation services. This involves providers being required to understand the outcomes they are achieving for ACC claimants. For example, providers are required to carry out their own claimant satisfaction surveys.

5.22
This set of measures provides an indication of what ACC is achieving for its claimants. However, the measures are not necessarily enough for ACC to know how effectively it is rehabilitating claimants in each of its case management streams.

5.23
The main ways ACC assesses case management performance are:

performance management monitoring and appraisals;
Customer Engagement Appraisals, with performance advisors (four full-time employees) reviewing the appraisals;
claims reviewers (six full-time employees) checking claims;
routine organisational performance reporting – this involves a combination of weekly, monthly, and quarterly reporting of progress against the Claims Management Network's performance objectives;
internal audits; and
external consultancy advice (as required).

5.24
These methods and/or our observations have shown:

inconsistency in practice between case managers, including the extent to which they use ACC's online procedural and policy guidance, variations in discretionary decision-making, and difference in the type and style of interactions with people;
inconsistency in the use and checking of Customer Engagement Appraisals – ACC's main case management quality assurance tool;
that the case management system might not work as well for people with complex needs and long periods of incapacity, who do not reach the threshold for serious injury, as it does for other claimants; and
that, historically, the system has not worked well for people with sensitive claims, but ACC is improving its services for people with sensitive claims as part of its Assist project.

5.25
Each year, ACC measures the level of staff engagement with their work and the organisation (by office). The level of engagement in offices varies. This is in line with the observations we made during our visits to offices. There is merit in ACC looking at the engagement of staff in the case management streams.

5.26
Local offices sometimes do not have the capacity to assign all people with claims of more than 912 days to the RIS stream. It is not clear how widespread this practice is. Because ACC relies on claim duration to identify RIS claimants, it is measuring some claimants as part of the RIS stream when they are being managed in other streams.

Measuring customer and claimant satisfaction

5.27
ACC has extensive information about customer and claimant experience and expectations. It is now using that information to inform future improvements.

Customer satisfaction

5.28
Customer satisfaction "is a valuable performance measure as a complement and cross-check on performance measures that use other types of information".¹⁷

5.29
ACC gets relatively detailed information about customer satisfaction on a quarterly basis. The sample size for this survey was increased to 3000 from the July to September 2013 quarter to allow reporting at a local office level. Information from the quarterly satisfaction surveying is available to all offices through a portal referred to as "The Hive".

5.30
Since 2011, there have been periods with relatively low customer satisfaction compared with ACC's target (see Figure 8).

Figure 8
Customer satisfaction rates, by quarter and against target, 2011 to 2014

Quarter	Target*	Below target	Above target
July – September 2011	70%
October – December 2011
January – March 2012
April – June 2012
July – September 2012	72%
October – December 2012
January – March 2013
April – June 2013
July – September 2013	68%
October – December 2013
January – March 2014
April – June 2014

* Percentage of customers satisfied or very satisfied with the overall quality of ACC's service delivery. Source: ACC.

5.31
In general, there is no significant correlation between demographic groups, and trust and confidence in ACC.

5.32
ACC is concerned about its relatively low overall level of customer satisfaction and has acknowledged that the level needs to improve. ACC's Client Service Optimisation and Shaping Our Future programmes are intended to support improvements in customer satisfaction.

5.33
We have been told that there are unique challenges in Christchurch in terms of understanding what ACC does, because many immigrants with limited knowledge of ACC have arrived in Christchurch to take part in rebuilding work after the earthquakes.

5.34
Another source of information about customer experience is the stakeholder conversations performed as part of the Customer Engagement Appraisal process. However, it was evident from the reviews of the Customer Engagement Appraisals that these conversations had not been regularly carried out. (We looked at a sample of 36 Customer Engagement Appraisal reviews.)

5.35
We spoke with ACC staff, who had mixed views about how useful they found stakeholder engagement as part of the Customer Engagement Appraisal process. Some told us that the process provided useful information – one staff member told us that "speaking with clients is absolute gold" – while others told us that the process did not provide them with information that was not already known.

Claimant satisfaction

5.36
As well as general customer satisfaction data, ACC has significant additional information about people's experience with its services. In summary, this information (mainly from 2013) shows that:

claimants who believe that the outcome of their receiving ACC services (including services that ACC-funded providers deliver) has been favourable to them are relatively satisfied; and
longer-term claimants with ongoing and unresolved problems are relatively dissatisfied.

5.37
Specifically:

RIS claimants are less satisfied than short-term claim claimants or front-end claimants managed in local offices.
SIS claimants have high levels of satisfaction, with some evidence of very good service, but claimants' historical grievances or unresolved problems can affect their perceptions of that service.
People with a treatment injury (an injury resulting from medical or surgical treatment) with straightforward claims that were accepted have higher levels of satisfaction, but there are lower levels of satisfaction for claimants with moderate or complex claims whose claims were declined.
Some claimants who had a medical assessment perceive bias in how ACC chooses providers. Clients have lower levels of trust and confidence in ACC after a medical assessment than before an assessment.
Vocational rehabilitation claimants rated providers of this service positively on a range of service attributes, with claimants who had returned to work and who felt ready to do so rating providers more highly than those who had returned to work but did not feel ready to do so, or those who had not returned to work.

5.38
ACC has carried out research into the experiences of people receiving social rehabilitation and the differences in experience with ACC between people in paid employment and those not in paid employment.

5.39
In 2006, ACC analysed claimant satisfaction data by ethnicity and found little difference in how satisfied Māori and non-Māori claimants were with ACC. ACC has commissioned research on the attitudes and behaviour of Māori accessing ACC's services.

5.40
ACC has carried out or commissioned ethnic-specific studies of injuries among Asian people and the accessibility of ACC services to Pacific people.

Claimant expectations

5.41
In 2012, ACC commissioned a thorough piece of research on claimants' expectations. In short, the research found that:

People whose expectations were met were more likely to be people with relatively minor injuries, people whose treatment and rehabilitation go to plan, and people who get back to their pre-injury functioning quickly.
People whose expectations were not met were more likely to be longer-term claimants with ongoing and unresolved treatment problems.

5.42
The research found that people expect to be:

informed well in advance of any changes to cover;
fully informed and consulted about treatment and rehabilitation decisions; and
proactively informed about, and provided with, appropriate entitlements and supports.

5.43
The research also identified people's perspectives of good case managers. These case managers are:

described as compassionate, respectful, and great communicators;
thought to look after the best interest of the claimant; and
good at establishing trusting relationships with claimants.

5.44
These are very similar expectations to what ACC claimants told us about their expectations of case managers. Appendix 2 lists what people told us that they expected from case managers.

5.45
These expectations are similar to what serious injury claimants told ACC in 2013 about their expectations of ACC. Some general expectations included that ACC:

communicates well;
provides enough and detailed information;
proactively manages cases;
treats claimants as individuals;
focuses on claimants during rehabilitation; and
has professional staff.

5.46
Some expectations more specific to serious injury included that ACC:

involves family and support people;
makes contact when the person is able to think, and process and recall information and conversations; and
uses a buddy or mentoring systems at significant points during the claimant's ACC journey.

5.47
Good communication and good information are threads common to all of these sources of information. ACC says that the most commonly breached aspects of the Code are the right to effective communication and the right to be fully informed.

5.48
ACC has received advice that suggests that socio-economic, ethnic, and geographic groups differ little in their expectations of ACC.

Customer improvement suggestions

5.49
ACC has set up four groups to get a "customer" perspective on how it manages claims. ACC considers a "customer" to be a claimant, a levy payer, or a provider of health or rehabilitation services.

5.50
The four groups are:

Consumer Outlook Group, which focuses on the biggest problems for ACC's consumers;
Older Persons Advisory Group, which focuses on problems for ACC's older claimants;
Serious Injury Advisory Group, which focuses on problems for ACC's seriously injured claimants; and
Advocates and Representatives, which focuses on policy and strategy matters.

5.51
The groups expect that ACC will formally respond to their recommendations. The groups have made recommendations to ACC during 2013 and 2014 on several topics, including:

medical assessments;
claimants managing themselves;
vocational independence;
improving forms, leaflets, and information sheets;
planning for discharges;
disputes and decision-making;
elective services; and
use of codes to record primary health care interventions.

5.52
In our view, the recommendations that the groups made appear to be well informed, thorough, and focused on improvement. It is important that ACC continues to give the recommendations due consideration to show that it is committed to having stronger focus on the needs of customers.¹⁸

5.53
The people we interviewed suggested several ways to improve ACC services, including how it uses case management to deliver those services. Appendix 2 provides details about those suggestions.

Complaints

5.54
Complaints are a source of information about customer satisfaction, experience, and expectations. Our August 2014 report on how ACC manages complaints looks at how ACC uses information about complaints.¹⁹ We found that ACC did not learn from complaints to the extent that we had expected.

Using information to improve case management

5.55
Except for information about complaints, ACC uses information about performance and customer experiences to improve service delivery and design. However, it could use that information more, particularly for people with complex needs who do not reach the serious injury threshold.

5.56
Evidence suggests that ACC is using Customer Engagement Appraisal results as feedback to case managers and managers. However, ACC is not fully using the opportunity that the Customer Engagement Appraisal process offers to routinely collect customers' comments.

5.57
ACC is moving to using more customer satisfaction and experience information to improve the design of its practices. An example of where claimant guidance has been used to inform the development of an initiative is with the claimant self-managing pilot project for people with serious injuries.²⁰

5.58
Evidence about customer satisfaction and trust, as well as other evidence information, has been a main determinant in ACC's Shaping Our Future change programme. An important purpose of that programme is to make ACC's services focus more strongly on customers.

5.59
It is also important to note that introducing new information technology, a critical element of Shaping Our Future, does not automatically bring better results. We see better results when technology gives people the power to work differently. ²¹

5.60
To ensure that ACC makes improvements, and to ensure that there is a return on ACC's substantial investment in consultants' reviews and advice over several years, it is critical that change projects are put into practice well and economically.

5.61
Because of the anticipated scale, complexity, and cost of the projects, putting them into practice successfully will be critical and challenging. ACC has identified that it will need to carefully manage the risks, particularly to Shaping Our Future.

5.62
Figure 9 lists the important change and improvement projects that we have identified as highly relevant to ACC's case management services. ACC needs to manage all of the projects effectively and economically.

Figure 9
Change and improvement projects and their relationship to case management

Projects	Intended contribution to improved case management
Client Service Optimisation	To support case management, better matched to individual claimant need though streaming of claimants; and to improve ACC's internal and external clinical services and reduce fragmentation of those services.
Forms, Letters, and Information Sheets	To improve the readability of ACC's forms, letters, and information sheets that case managers use.
Privacy	To complete implementation of the recommendations from an independent 2012 review, ultimately to protect claimant information.
Assist	To improve service delivery to claimants with sensitive claims.
Academy	To help with the availability to case managers of specialist training in case management.
Shaping Our Future	To improve consistency of customer service delivery and customer experience, better link evidence to rehabilitation pathways for individual claimants, improve administrative practices, and add a more self-service-like service delivery channel.

17: James, O (2010), End user satisfaction as a performance measure for public service, paper for the National Audit Office, University of Exeter, page 3.

18: We have excluded recommendations about government policy from our comments.

19: Accident Compensation Corporation: How it deals with complaints, available on our website at www.oag.govt.nz.

20: This pilot project involves giving a budget to some serious injury clients, which they can directly manage themselves rather than ACC having to approve expenditure on individual items.

21: See our June 2012 report, Realising benefits from six public sector technology projects, available on our website at www.oag.govt.nz.

page top