Part 5: Measuring and reporting complaints

… reduce the number of problems and concerns raised with ACC that can lead to complaints lodged with the Office of the Complaints Investigator. The target is a 10% annual reduction from the forecast year-end baseline (1,700).

ACC's Statement of Intent 2013-2016

5.1
In this Part, we discuss how:

• ACC needs to record complaints better;
• ACC needs to improve reporting further;
• complaints are not handled transparently; and
• the true cost of complaints is unknown.

Summary

5.2
ACC needs robust and useful information to analyse and use to improve services – but the information technology cannot provide detailed analysis of complaints, complaints are under-reported and recorded in disconnected systems, and the data quality is not high.

5.3
Within current data limitations, reporting about complaints is mixed. OCI's reporting to senior managers has improved and now includes some analysis. CSS reports would be more useful if they included analysis as well as activity.

5.4
Being more transparent in showing the public how it deals with complaints could help ACC increase public trust and confidence. ACC has recently adopted a target to reduce complaints. In our view, there is a risk that the target could encourage behaviours that discourage people from complaining or discourage accurate recording of complaints.

5.5
The true cost of complaints to ACC is unknown because data about the cost of complaints is limited. ACC does not routinely analyse the cost of handling complaints.

Complaints could be better recorded

Because of problems with definitions, expressions of dissatisfaction made to local offices are not recorded as complaints. The information technology used cannot provide detailed analysis of recorded complaints and data quality is not high. Without full and accurate records, ACC cannot analyse and use complaints information to improve services.

5.6
Local offices do not record complaints in a way that makes analysis easy. Frontline staff do not define complaints as "complaints", even though they are expressions of dissatisfaction.

5.7
Frontline staff record complaints as "client contacts" in a claims management system called "Eos". The information cannot be easily analysed. Some local offices keep their own stand-alone records of complaints.

5.8
CSS and OCI record complaints using a different part of Eos and the ability to report from this system is limited. More detailed analysis is sometimes possible through a request to the research team, but this can sometimes require data to be manipulated manually.

5.9
Other teams, such as the government services team, privacy team, and IMT, keep separate records about complaints using different software. There is a risk that complaints dealt with by these teams are not included in the publicly reported complaints data. If the complainant has complained to more than one team in ACC, there is a risk that the complaint could be recorded in more than one system.

5.10
The lack of co-ordination of the complaints system means that ACC is not able to identify accurately whether there is any double counting of complaints. Our analysis of ACC data for 2013 shows that teams outside CSS and OCI might have handled up to 500 more complaints.

5.11
We have not audited how good ACC's data quality is. However, high-level analysis using the six dimensions of data quality shows signs of potential weakness, including data extracted from free text fields, inconsistent and multiple recording, and using stand-alone spreadsheets. In some instances, we found it difficult to reconcile ACC data sets and we uncovered weaknesses in the data that the telephone system used by CSS produces.

5.12
Any data that supports public reporting needs greater rigour than operational management data. This includes the new public corporate complaints target in ACC's Statement of Intent 2013-2016. Recording information incompletely and inaccurately could undermine trust and confidence in the publicly reported numbers of complaints used to show progress towards ACC's target.

5.13
ACC is aware of weaknesses in how it records, analyses, and reports complaints and is working on some improvements. We have not audited the improvements so we cannot provide assurance about their effectiveness.

Reporting could be further improved

CSS and OCI reporting of complaints information to senior managers has improved. More can be done, but significant improvements will need better information.

5.14
OCI's monthly reports now include yearly trends, performance against targets, and the number of complaints by section of the Code and by local office. This is a useful start, but is mostly data rather than data and analysis. The local office managers we spoke to held varied views on the usefulness of the OCI reports. Some managers would welcome more detailed analysis.

5.15
More sophisticated analysis, such as relating the number of complaints to local office size and to demographic information would provide more valuable information about complaints and complainants. However, we recognise that significant improvements are unlikely without better information technology systems and better recording of detailed data.

5.16
CSS began reporting about complaints in March 2013. CSS reports are less useful than OCI reports. They report activity, which is currently based on data with variable quality. Limited and occasional analysis has been done in the past, but regular systematic analysis would increase the usefulness of CSS reports and provide a co-ordinated picture throughout OCI and CSS.

Complaints are not handled transparently

ACC could increase public trust and confidence by being more transparent and showing the public how it deals with complaints. ACC has introduced a complaints target but there is a risk that the target could encourage staff to behave in ways that are unlikely to improve complainants' experience.

5.17
Service users often complain because they do not want others to experience the same problems and want to see improvements.¹⁶ Some people do not complain because they think that no change will result.¹⁷ Therefore, providing feedback to complainants about what has changed is an important part of dealing with complaints and increasing people's satisfaction with the process.

Some people felt that the information about making complaints was misleading.

5.18
International experience shows that some private and public entities carry out regular "you said – we did" exercises. These include the Health and Disability Commissioner, Z Energy (the fuel retailer), and the British National Health Service. The output from these exercises can take several forms, such as posters, advertisements, and sections in annual reports or other public documents. Regardless of the form they take, this is a public demonstration of an organisation's commitment to improve services by listening to people (including their complaints) and providing feedback on the changes that it has made as a result.

5.19
ACC could be more transparent in showing how well it deals with complaints. ACC's annual reports have not included any measures of complaints since 2007.

5.20
ACC has a target to reduce the number of complaints. This is detailed in the Statement of Intent 2013-2016. The target is to:

… reduce the number of problems and concerns raised with ACC that can lead to complaints lodged with the Office of the Complaints Investigator. The target is a 10% annual reduction from the forecast year-end baseline (1,700).

5.21
ACC has started to report performance against the complaints target in its quarterly reports, which are on its website. However, in our view, the quarterly reports are difficult to find.

5.22
There is a significant risk that the complaints target could deliver unintended consequences. For example:

• The target could encourage staff to behave in ways that discourage people from complaining.
• Basing the target on an absolute number might not be appropriate without understanding what influences the number of complaints, such as the number of claims that ACC is managing each year.
• The compounding nature of a year-on-year reduction means that early reductions will be easier to achieve than later ones and might encourage staff to focus on dealing with easy complaints – "easy wins" – rather than the most significant complaints.

5.23
In contrast, ACC's privacy target uses a different method to measure performance. A "balanced scorecard" of three different measures is amalgamated to come to an overall conclusion on whether performance has improved. The three measures are:

• the number of privacy breaches;
• governance and staff capability; and
• performance against the 12 information privacy principles.

5.24
ACC told us that the complaints target is unlikely to encourage unwanted behaviour among local office staff because it does not form part of their performance objectives. We note that it is included in the regional area leaders' performance objectives, a staff group with less involvement in the complaints process than, for example, managers in local offices. Logically, a target needs to involve those staff whose behaviour influences the achievement of the target. Without this link, it is difficult to see how the target will be achieved.

5.25
In our view, if ACC's aim is to become an organisation that focuses on customers, then it needs to consider whether a target to reduce complaints is compatible with this aim.

True cost of complaints is not known

Data about the cost of complaints is limited. ACC does not routinely analyse and report the cost of handling complaints.

5.26
Because handling complaints is regarded as part of frontline staff's role, many organisations do not measure the cost of handling complaints. The service recovery approach encourages resolving complaints at the lowest possible tier of the complaints system – the local office. ACC does not categorise these interactions as complaints. This means that ACC does not know the true cost or nature of complaints and is not managing the cost of complaints.

5.27
We have estimated the annual cost and number of complaints that ACC receives. We estimate that ACC received about 20,700 complaints during 2013. Of these, only about 1600 were classified as complaints and recorded. Using this estimate, almost one-third of the total cost of managing complaints lies outside the two core complaints business units, CSS and OCI. Figure 23 shows our estimates of the cumulative number and cost of complaints at each tier of the complaints system.

5.28
As mentioned in paragraph 3.44, if about 200,000 people are not satisfied, the 20,700 represents about one in 10 of all those who are not satisfied. This is consistent with figures from research about other organisations.

5.29
Our estimate shows the importance of resolving complaints at the lowest possible tier. The cumulative cost of resolving a complaint at:

• the local office can be as low as $18;
• CSS is about 24 times more expensive than the local office; and
• OCI is almost 142 times more expensive than the local office.

Figure 23
Estimated number and cost of complaints, 2013

Source: Office of the Auditor-General.

Note: For local offices, we used information from our survey of ACC staff. For CSS, OCI, and review tiers, we used ACC data. We asked local office staff to estimate how long they spent dealing with a typical dissatisfied person. We multiplied this by the lower limit of the pay band for staff dealing with complaints. For CSS and OCI, we used direct staff costs. For review, we used the cost of services provided by Fairways Resolution Limited. Our estimate is likely to be conservative. It does not include indirect costs associated with complaints (such as the costs of preparing information for OCI).

page top