Part 4: Quality of auditing and certification has improved but the effect on the quality of care is less certain

4.1
In this Part, we discuss:

• what the Ministry is doing to assess and continue to improve the quality of certification audits;
• further audit work we have done to assess whether the Ministry's changes to certification arrangements have improved the effectiveness of those arrangements; and
• what the Ministry's and our analyses indicate.

Our overall findings

4.2
There are indications that the quality of the auditing and certification process for rest homes has improved during the last two years, because:

• third-party accreditation has improved DAAs' systems and processes;
• the integrated audit approach has resulted in better communication between the Ministry, DHBs, and DAAs;
• the Ministry, DHBs, and DAAs respond more quickly and effectively when issues in rest homes are identified; and
• the Ministry consistently monitors the quality of DAA audits.

4.3
The effect that these improvements have had on the quality of care delivered to rest home residents is less certain. There is scope for certification and auditing to provide better assurance about the quality of care provided in rest homes, including better assurance to DHB planning and funding managers and rest homes. The Ministry needs to keep shifting the focus of rest home audits towards ensuring that documented policies and procedures result in safe quality care being delivered to residents, as it has started to do by introducing tracer audit methodology.

How the Ministry assesses the ongoing quality of the certification audits

The Ministry has a number of ongoing activities to assess the quality of rest home certification audits.

4.4
The Ministry monitors DAAs by observing auditors on-site, evaluating each audit report completed by DAAs, and asking for feedback from rest homes.

4.5
During 2011, the Ministry reviewed the quarterly benchmark reports. It collected this information to find out whether DAAs' compliance with the best practice criteria had improved (see paragraph 3.24). The evaluation covered the last quarter of 2009, the last quarter of 2010, and the first quarter of 2011.

4.6
At the end of 2011, the Ministry analysed the 108 responses it had received from a survey of rest homes that it carried out between 12 August 2011 and 25 October 2011. The responses covered the five DAAs that operated during that period.

4.7
In June 2011, the Ministry carried out a preliminary evaluation of the effectiveness of accrediting the DAAs. This was to find out how effective third-party accreditation had been in strengthening the certification process. This evaluation was inconclusive, largely because the Ministry had introduced third-party accreditation only at the end of 2010. The evaluation took place in June 2011 to meet one of our 2009 report recommendations. The Ministry has told us that it will carry out another evaluation of DAA accreditation during 2013. This evaluation should provide more useful information.

What we did to find out whether quality has improved

We surveyed rest home residents, their families, and friends, and surveyed DHB planning and funding managers. We analysed complaints to the Health and Disability Commissioner.

4.8
We carried out three pieces of work to find out whether the changes to the certification arrangements had improved the effectiveness of the arrangements to check the standard of services provided by rest homes.

4.9
For the first piece of work, we published a survey on our website. A link to the survey was also published on the Health and Disability Commissioner's website and Aged Concern New Zealand's website. Our questions focused on any quality improvements or concern that rest home residents and their families and friends had noted during the last two years. We received 53 responses to our survey.¹⁸ The responses are not statistically significant and do not necessarily reflect the views of all rest home residents, their families and friends, staff and caregivers. However, they raise important issues and contribute to our overall view.

4.10
Our second piece of work was a questionnaire asking the DHBs' planning and funding managers about their views on the quality of the arrangements for certifying rest homes. In particular, we sought the managers' views on:

• their DHB's involvement in the certification process;
• how audit findings are communicated to the DHB;
• whether the DHB receives enough information to allow it to follow up on audit findings;
• whether auditors are interpreting the Standards consistently;
• whether DHB issues-based audits (see paragraph 4.20) identify issues that should have been identified during the certification audits;
• how satisfied DHBs are that certifying rest homes provides reliable assurance of safety and quality; and
• whether the arrangements for certifying and monitoring rest homes have improved.

4.11
In our third piece of work, we analysed complaints data held by the Health and Disability Commissioner.

What the Ministry's and our analyses indicate

Although there are indications that the quality of the certification process for rest homes has improved during the last two years, some concerns remain. The effect that the improvements have had on the quality of care delivered to residents is less certain.

4.12
We have looked at the results of the Ministry's evaluation and analysis, as well as the results of the work that we did.

Third-party accreditation has provided benefits and improved DAA processes

4.13
The Ministry's evaluation of third-party accreditation included looking at whether accreditation had:

• identified areas for improvement that affect the standard of auditing done by DAAs; and
• resulted in DAAs changing how they operate.

4.14
The Ministry's survey of DAAs (which was carried out as part of its evaluation of third-party accreditation) suggests that third-party accreditation has encouraged DAAs to identify and implement improvements to their processes. In response to the survey, DAAs said that they had reviewed and revised their quality system documentation before their third-party accreditation assessment. Five of the six DAAs said that the third-party accreditation assessment process identified areas for improvement in their audit process, business practices, and management of auditors.

The standard and consistency of audits have improved but they still do not identify all issues

4.15
DAAs were less positive about the effect that third-party accreditation had had on the standard and consistency of auditing. Two of the six DAAs felt that the standard and consistency of auditing had improved with the introduction of third-party accreditation requirements, one was not sure, and the other three did not think that the introduction had had any effect.

4.16
Planning and funding managers in DHBs were more positive. Thirteen of the 18 managers who responded thought that the process for auditing and certifying rest homes had improved significantly during the last two years; three thought that it had improved slightly, one thought it had deteriorated, and one did not answer the question. This may reflect the introduction of the integrated audit approach. Half of the managers thought that DAA audits and auditors have become more consistent in applying the Standards.

4.17
The planning and funding managers thought that communication before and after the certification audits was good. The managers were asked to provide feedback to the DAA on any issues that the DHB would like investigated during the audit, such as complaints that needed to be followed up or concerns with the levels of service provided in the rest homes.

4.18
Most of the DHB managers thought that the findings from the certification and spot audits were sent to them in a timely manner. They were sent or had access to a draft report and have access to the final report.

4.19
Some of the planning and funding managers thought that there is still room for improvement in the audit approach. For example, they thought auditors needed to shift their focus towards higher-risk areas such as individual care plans, infection control, medicines management, falls, nutrition, restraint, workforce development, diversion therapy, and quality assurance.

4.20
We were concerned to note that issues-based audits¹⁹ are still identifying issues that planning and funding managers believe should have been picked up during the certification audits. Four of the seven issues-based audits that DHBs carried out during the year ended 30 June 2011 identified major safety and quality issues that should have been picked up during the certification audits. In another instance, where the issues-based audit was done because of the findings in the certification audit, the issues-based audit found many instances of unsatisfactory patient care that were not picked up in the DAA's certification or spot audits. The manager in this DHB noted that certification audits are more of a systems' and processes' audit rather than an audit of individual care. It is our view that the improvements the Ministry is planning (as discussed in Part 5) should help to address these issues.

The quality of audit reports is still variable

4.21
The Ministry's evaluation of third-party accreditation looked at whether third-party accreditation had improved the standard and consistency of auditing and audit reporting. The evaluation found that no DAAs had achieved full compliance with HealthCERT's audit reporting requirements from year to year. The most common criteria where DAAs were not meeting the expected benchmarks were:

• reports requiring resubmission (five of six DAAs);
• terminology not being explicit (five of six DAAs);
• requirement for additional information to ensure triangulation of evidence (four of six DAAs);²⁰
• HealthCERT advisors needing to request further information more than once (four of six DAAs); and
• audit reports not submitted on time (five of six DAAs).

4.22
The Ministry's analysis of the quarterly benchmarking results found that all DAAs are now achieving 100% compliance for the composition²¹ of the audit teams and all have improved how they quantify and organise evidence. However, all but one DAA continue to submit completed audit reports that require resubmission.

4.23
The Ministry noted that HealthCERT still has to contact DAA auditors for further information to ensure that evidence was triangulated and to ensure that the evidence matched the risk ratings given by the auditors. For some DAAs, these benchmarking results had worsened since the end of 2009.

4.24
The DHB planning and funding managers commented on the quality of the audit reports. Half of the managers considered that the audit reports contained enough information to allow them to follow up on the findings. The other managers thought that the DAAs need to increase the amount of information provided in the audit reports so that the managers did not have to go to the DAA to clarify the findings.

The effect of changes on quality of care is less certain

4.25
Seventeen planning and funding managers were fairly satisfied or very satisfied that certification provides a reliable assurance of safety of care. Eleven managers were either fairly satisfied or very satisfied that certification provides a reliable assurance of quality of care in rest homes (the other six managers were neither satisfied nor dissatisfied). This may indicate that there is scope for certification and auditing to provide better assurance about the quality and safety of care provided in rest homes.

4.26
Figure 4 shows DHB planning and funding managers' responses to questions about safety and quality of care.

Figure 4
Responses from 18 district health board planning and funding managers to our questions about safety and quality of care provided in rest homes

4.27
Nearly half of the respondents to our public online survey noted improvements in the quality of services in rest homes since 2009. These largely related to improved communications between the rest homes, residents, and their families and friends and improved communication among staff. Respondents considered that the way that the rest home dealt with concerns or problems had improved.

4.28
Forty-two of the 53 respondents to our survey still had concerns about the quality of services provided in rest homes. The concerns most often cited were:

• the quality of one-to-one care provided (34 of 42 respondents); and
• the housekeeping, including food (23 of 42 respondents).

Analysis of complaints to the Health and Disability Commissioner is inconclusive

4.29
We analysed the complaints received by the Health and Disability Commissioner between 2006 and 2011. We note that the number of complaints increased from 2006 to 2009 but have declined since 2009. We note that the proportion of complaints that have been upheld or had further action taken has increased. It is difficult to work out the reasons for these changes.

4.30
The Health and Disability Commissioner told us that the volume of complaints did not necessarily reflect the quality of care and that different factors can influence the volume of complaints. For example, when a complainant has used a rest home's internal complaints procedure and been satisfied with the investigation of their complaint, they are less likely to escalate their concerns to the Health and Disability Commissioner.

4.31
We note that some significant changes to rest home certification, such as third-party accreditation, were not introduced until late 2010, so the effects of these changes may not yet have affected the number of complaints to the Health and Disability Commissioner.

More focus needs to be placed on auditing rest home practices

4.32
Although most respondents (87%) to the Ministry's survey of rest homes (see paragraph 4.6) wanted no change to the amount of time that auditors spend on observing actual rest home practices, a notable minority (13%) wanted auditors to increase the time that they spend observing the rest home's actual practices. We also received this feedback in our interviews with rest homes and from DHB planning and funding managers. We support this approach.

4.33
Respondents to our online survey felt that the certification process was generally too paper-based. It was described as an exercise more concerned with compliance than outcomes for residents. Suggestions included spending more time with residents and their families rather than looking at files, and actively observing staff behaviour. These responses support other evidence about getting the balance right between checking on policies and procedures and examining their practical application.

4.34
In our DHB survey, six of the DHBs chose to give us additional comments that they would like to see the current audit processes changed to focus on the quality of care rather than documented processes. One DHB commented that:

The audit does not measure the effectiveness of care delivered – it manages compliance with documented processes to achieve certification. In some instances the audit report has been very positive, but the standard of care delivered has been found to be poor.

4.35
A second commented that:

The quality measures that we currently use such as certification do not reflect the standard of care provided in ARRC. This mismatch can create misunderstandings and disbelief. Now that the integrated audit is embedded in the contracting obligations of health of older people services in ARRC, there is an urgent need to develop quality measures alongside this national infrastructure.

4.36
A third commented that:

The current approach addresses the systems required for the provision of safe services. However, there is minimal assessment of the client experience of delivery of client focused, quality services.

4.37
In Part 5, we discuss how the Ministry might connect clinical and audit information to continuously improve the quality of care provided in rest homes.

18: Of those who responded, more than half (53%) were either caregivers in the rest home or members of staff other than caregivers. Almost a quarter (24.5%) of respondents were family members or friends of residents. The last group of respondents classed themselves as "others". This group included a range of people who visited rest homes, such as registered nurses or advocates.

19: An issues-based audit is carried out by a DHB after identification of immediate and serious concerns with the care being provided within a rest home and/or there has been an unsatisfactory response by the rest home to address serious concerns about that rest home. These audits are usually more targeted than contractual audits and there is no notification to the rest home. The audits are still focused on meeting contractual and Standards' requirements.

20: Triangulation of evidence is a process for gathering reliable evidence by drawing information from three sources – interviews, observation, and documentation.

21: The composition of the audit team must reflect the characteristics of the service and its users – for example, in terms of background and service type. Every audit team must include a person with clinical and technical expertise relevant to the service being audited who holds a relevant annual practising certificate and an auditor with appropriate qualifications and experience as a lead auditor or team leader.