Part 3: Certifying rest homes
3.1
In this Part, we discuss:
- how well the Ministry oversees DAAs;
- the effectiveness of progress reports in encouraging rest homes to provide services safely;
- how well the Ministry uses and shares information about certification; and
- the effectiveness of certification in encouraging rest homes to improve the quality and safety of their services.
Our overall findings
3.2
Since its introduction in October 2002, certification has not provided a consistently adequate level of assurance that rest homes have met all the criteria in the Standards. The Ministry has been aware of weaknesses in auditing by some DAAs since 2004, but these weaknesses were not acted on quickly enough or with enough effect.
3.3
The effectiveness of certification has been compromised by inconsistent auditing by DAAs, which makes it hard for the Ministry to certify rest homes for appropriate periods. Certification is also not as effective as it should be because information has not been shared enough between the different organisations involved.
Overseeing the designated auditing agencies
The Ministry has known since 2004 that the auditing carried out by DAAs is inconsistent and, in some cases, of a poor quality. In our view, the Ministry did not respond to these problems quickly enough or with enough effect.
3.4
It is important that the auditing and reporting by DAAs is consistent and reliable. Poor auditing can mean that aspects of care where rest homes are not meeting the required standards are not identified. This creates a risk that the Ministry could certify rest homes for longer periods than it should, or certify a rest home that should not be certified.
3.5
The period of certification given to a rest home is one of the main ways of managing risks in the care of older people in rest homes. Rest homes that receive poor audit reports are certified for shorter periods, and are subject to more regular certification audits. The length of certification can also be used by DHBs to inform their monitoring activity.
3.6
At the time of our audit, 80% of rest homes were certified for three years, and another 14% for two years. Very few rest homes were certified for more than three years.
3.7
Figure 2 sets out a timeline of actions the Ministry has taken, between 2002 and 2009, to improve certification. It also shows the performance information that has been available to the Ministry.
Figure 2
Timeline of action taken to improve certification, and the performance information available, 2002-2009
| Date | Actions | Performance information |
|---|---|---|
| 2002 | October: All new rest homes must be certified under the Health and Disability Services (Safety) Act 2001. | |
| 2003 | December: DAA Handbook is produced. | |
| 2004 | October: All existing rest homes must be certified under the Health and Disability Services (Safety) Act 2001 by this date. | October: Ministry-commissioned audit of five DAAs with the largest market share identifies significant weaknesses in their performance. |
| 2005 | June: DAA Handbook is revised. September: DAA Handbook is revised. |
November: Ministry-commissioned audit of the other four DAAs concludes that there are serious weaknesses common to most, if not all, DAAs. |
| 2006 | September: Requirement for third-party accreditation is removed as a condition of designation for DAAs. Further designation conditions added by way of notice in the New Zealand Gazette. | |
| 2007 | March: DAA Handbook is revised. July: Ministry is restructured. October: HealthCERT manager resigns. December: HealthCERT prepares new audit tool and report for its unannounced inspections of rest homes. |
July: Discussion paper about a review of the Health and Disability Services (Safety) Act 2001 is published. It raises concerns about conflicts of interest and inconsistency between DAAs, which makes certification decisions difficult. |
| 2008 | HealthCERT is restructured, experiences high staff turnover. January: Risk matrix to determine periods of certification is introduced. More use of information from other agencies is included in certification decisions. May: Ministry starts process to implement new Standards, including a new standard audit reporting template and communication with DAAs about the transition period. July: Training begins for HealthCERT staff who will observe DAA audits. August: Revision of complaints process starts. A DAA is re-designated despite Ministry concerns about its performance. Ministry plans to commission an audit of a DAA with a large market share. September: Revised Standards are notified in the New Zealand Gazette. May, September, October: Cabinet papers raise concerns about risks in the certification process and the performance of some DAAs. October: New team of senior advisors appointed in HealthCERT. HealthCERT begins review of DAA Handbook and prepares tables comparing 2001 and 2008 Standards. November 2008: Revised Standards are introduced. |
July: Belhaven Rest home (certified for two years) is shut down. |
| 2009 | February: Programme of seminars begins, explaining changes to the Standards. February: DAA "Train the Trainer" sessions begin, with the aim of greater auditing consistency. April: Ministry project to improve the effectiveness of rest home audits begins. May: DAA Handbook is revised and published in draft. New Zealand Gazette notice is revised to include a requirement to use the Ministry's reporting template, and to submit audit summaries that the Ministry will publish. Risk matrix is revised, contains a new quantitative element to support HealthCERT advisors. June: Electronic audit report template is introduced, all DAAs are required to use it. June: Ministry begins to publish summaries of audit reports. July: Risk matrix is revised again. August: Ministry works with DHBs to reach agreement that, once DHBs have increased confidence in DAA auditing, certification audits could meet DHB needs for monitoring. September: Pilot project to carry out 23 unannounced audits begins. Risk matrix developed further. October: Enhanced DAA evaluation form is introduced to support monitoring of DAAs. Ministry begins to observe DAA certification audits. Draft DAA Handbook is revised. November: Ministry commissions an audit of a DAA with a large market share. Ministry starts analysis of "partially attained" ratings against medicine management standard. |
September and October: Ministry analysis shows that, although there is evidence that some aspects of the performance of DAAs have improved, the auditing by DAAs is still of a variable quality. |
The Ministry's response to the reports commissioned in 2004 and 2005
3.8
The Ministry has had concerns about the quality and consistency of auditing by DAAs since soon after certification was introduced in late 2002. In 2004 and 2005, the Ministry commissioned reports by an external organisation, The Systems 3 Group Pty Ltd (S3G). S3G looked at how well DAAs were auditing rest homes, and whether the information DAAs provided to the Ministry could be relied on when making decisions about how long rest homes should be certified. A senior staff member within HealthCERT at the time was actively involved in the work carried out by S3G.
3.9
The reports by S3G found serious weaknesses common to all or most DAAs. The weaknesses were in management controls, auditing practice, reporting, and auditor competency management. Figure 3 summarises the findings of the 2005 report.
Figure 3
Summarised findings of The Systems 3 Group report (2005)
| Management controls: DAA management controls were not sufficiently robust to ensure that audit teams delivered valid and reliable reports. Internal audits by DAAs were carried out poorly, with a general lack of follow up of findings. Accreditation audits by external quality assurance organisations had failed to identify irregularities in audits against the Standards. Internal DAA management reviews were not followed up, and did not focus on the work DAAs do in the health and disability services sector. |
| Auditing practice: S3G concluded that "Audit teams do not perform the audit in a manner that delivers the most accurate information upon which a reliable report can be written." Deficiencies included inadequate planning, lack of audit evidence corroboration, inadequate sample size selection, and inadequate time spent on site. |
| Reporting: DAA reports did not have sufficient information for certification decisions. Data reliability was poor, ratings were inconsistent, and evidence did not align with ratings. |
| Auditor competency management: The way DAAs determined and managed the competency of their auditors varied considerably. DAAs were not always using personnel who were competent. There were deficiencies in technical skill, communication, and independence, with a "general failure" to declare conflicts of interest. DAAs were not publicising conflict of interest provisions. There was weak assessment of contract auditors, and variable support and guidance given to contract auditors. |
3.10
The Ministry responded to the recommendations in the reports mainly by modifying the DAA Handbook and adding conditions to the designation of DAAs in the New Zealand Gazette.
3.11
Based on S3G's reports, the Ministry concluded that requiring DAAs to have third-party accreditation was not efficient:
The primary purpose of requiring third party accreditation was to provide HealthCERT with a reasonable level of confidence in individual DAAs' auditing systems and to achieve a reasonable degree of consistency between DAAs. HealthCERT shares S3G's view that third party accreditation is not providing this level of confidence or consistency.
3.12
One of the accreditation bodies disputed S3G's findings about accreditation, but the Ministry kept to its decision and removed third-party accreditation as a condition of designation in 2006. In the Ministry's view, the third-party accreditation bodies should have identified the problems identified by S3G and the Ministry, and should have alerted the Ministry to the non-compliance with accreditation standards and Ministry guidelines.
3.13
A joint Ministry and DAA working group was established in 2006 to address the issues in the S3G reports, but there are few records of meetings of the group and it is not clear what the group has achieved.
3.14
The S3G reports identified some important DAA weaknesses, and the reports could have provided a useful benchmark for the Ministry to measure any improvements in DAAs. There is little evidence that progress in implementing the recommendations, or what effect they were having, was systematically tracked or that the performance of DAAs was measured to see if it was improving in the years after the reports were written. In our view, the Ministry's actions after receiving the reports did not have the necessary urgency or efficacy.
The Ministry's response to concerns raised in 2008
3.15
In September and October 2008, the Minister of Health, responding to concerns about whether all DAAs were functioning as expected in all circumstances, reported to the Cabinet Business Committee. The reports concluded that the safety and quality assurance processes in the aged care sector needed to be strengthened. The Minister of Health proposed a number of changes, including:
- changing the model whereby rest homes choose their DAA;
- introducing unannounced audits;
- that the Ministry begin witnessing audits; and
- that DAAs be subject to independent audits.
3.16
Some of the proposals, such as unannounced audits and independent audits of DAAs, are now being implemented.
3.17
Rest homes continue to choose their DAA. During interviews with us, the operators of multiple rest homes said that this gives them confidence that all their rest homes are being audited consistently.
Analysis and evidence of auditing practices in 2009
3.18
In 2009, there is evidence that the quality of auditing by DAAs continues to vary. However, there are also indications that DAAs are improving some aspects of their work. For example, recent analysis by the Ministry showed that DAAs were meeting the requirements for the skills and experience of their audit teams (see Figure 4).
3.19
Performance in other aspects of auditing remains variable and there is scope for improvement. In May 2009, an operator of many rest homes changed to a different DAA because of the operator's experience of poor auditing.
3.20
When the Ministry analysed DAA audit reports in October 2009, it found variation across a range of indicators (see Figure 4). Of the 44 audit reports analysed, 45% were incomplete.
3.21
The analysis also showed that DAAs need to improve the way they quantify evidence – only 36% of the audit reports indicated how many files the DAAs had reviewed or how many interviews they had conducted. The S3G report in 2004 had highlighted this as an aspect of DAA auditing that needed to improve. Although the sample used for the analysis in 2009 was relatively small, it indicates that progress is still needed.
Figure 4
The Ministry of Health's analysis of audit reports from designated auditing agencies in 2009
| DAA 1 | DAA 2 | DAA 3 | DAA 4 | DAA 5 | DAA 6 | Total | |
|---|---|---|---|---|---|---|---|
| Number of audit reports | 2 | 2 | 3 | 9 | 14 | 14 | 44 |
| Audit team met team composition requirements | 100% | 50% | 100% | 100% | 100% | 79% | 91% |
| All of the audit report was completed | 50% | 50% | 33% | 78% | 50% | 50% | 55% |
| Audit report was re-submitted | 0% | 0% | 33% | 0% | 0% | 29% | 11% |
| Evidence was triangulated | 100% | 50% | 67% | 89% | 64% | 93% | 80% |
| Evidence included resident/relative interviews | 100% | 50% | 67% | 89% | 93% | 57% | 77% |
| Terminology was not ambiguous | 100% | 100% | 100% | 100% | 79% | 79% | 86% |
| Standards matched criterion | 50% | 100% | 33% | 100% | 93% | 93% | 89% |
| Evidence matched attainment rating | 0% | 100% | 100% | 89% | 93% | 93% | 89% |
| Evidence matched risk ratings | 50% | 100% | 67% | 89% | 93% | 100% | 91% |
| Quantification of evidence (e.g. 4 of 5 staff interviewed, 3 files reviewed) | 50% | 50% | 33% | 56% | 29% | 29% | 36% |
| Evidence was written in the present tense | 100% | 100% | 33% | 100% | 71% | 100% | 86% |
| Information requested to match evidence and risk ratings | 50% | 0% | 100% | 11% | 21% | 14% | 23% |
| Information requested for triangulation | 50% | 0% | 0% | 0% | 7% | 7% | 7% |
| DAA responsive to Ministry requests (within 48 hours)* | 100% | 0% | 100% | 100% | 100% | 67% | 34% |
Source: Based on data from the Ministry of Health.
* These percentages can relate to fewer than the total number of audit reports, because the Ministry does not always make requests that DAAs need to respond to.
3.22
The Ministry has evidence of sustained poor auditing practice by one DAA in 2009. In interviews with us, staff within the Ministry and within DHBs raised concerns about the performance of another DAA. In the case of the former, the Ministry has written to the DAA setting out its concerns, which include conducting audits outside time frames described in the DAA Handbook, using auditors who did not have the required qualifications, and reporting positive results – since 2004 – for a rest home that the Ministry has inspected and found to be failing against many of the Standards. The Ministry has recently commissioned a separate audit of this DAA, to be conducted by S3G.
3.23
There are examples (from 2008 and 2009) where DAAs have failed to report or find instances where rest homes have not met the Standards, and where associated serious failures in the care of residents have not been identified (see Figure 5). These failures have later been found by other regulatory bodies. Although the frequency of these events may be low (we found five examples in our file reviews of the certification of 73 rest homes since 2003), they are significant because the failings are serious.
Figure 5
Examples of designated auditing agency failures to identify serious shortcomings in rest homes (2008 and 2009)
| Organisation | Incident |
|---|---|
| District health boards DHBs and the SSAs that carry out audits on their behalf report that, in the course of their monitoring of rest homes, they have found failings that have required immediate action for criteria that had been rated as fully attained by DAA audits. |
In 2008, after serious complaints about the conditions in a rest home (which included a shortage in the supply of oxygen and the maladministration of medicines), a DHB commissioned an issues-based audit. This audit was carried out on the same day that a surveillance audit was carried out by a DAA. Among other issues, the DHB's audit found serious failings with the medication management system. They included a failure to investigate errors and controlled drug counts not adding up. However, the DAA reported that "systems were in place for the safe management of medicines" and the criteria for medicine management were rated as fully attained. |
| Health and Disability Commissioner The Health and Disability Commissioner has substantiated serious complaints about the provision of care in rest homes. Those rest homes have received positive reports by DAAs. |
A complaint was made to the Health and Disability Commissioner in 2008 by a woman unhappy about the care her husband had received in a rest home. The Commissioner discovered serious issues in the standard of care, staff training, communication, behaviour, risk management, and clinical records at the rest home. The matter was referred to the appropriate DHB, and the manager of the rest home resigned. A DAA had carried out a surveillance audit at the rest home five days after the complainant's husband had been admitted in 2008. The surveillance audit report said that the "organisation has established, documented and maintained quality and risk management systems in place." |
| Ministry of Health Ministry inspections have discovered failings not identified in DAA audits. |
In 2009, the Ministry received a complaint from a medical centre about a rest home. The complaint alleged that a resident from the rest home was taken to the medical centre by car after she was found to be in an unresponsive state. At the medical centre, she was diagnosed with hypothermia and a suspected fracture of her femur. The Ministry carried out an unannounced inspection of the rest home. The inspection report contains 21 corrective actions that are needed for the rest home to comply with the Standards. Later in 2009, the Ministry wrote to the chief executive of the DAA that audits the rest home. The Ministry said that it was concerned that, since 2004, audits of the rest home by the DAA had found that it had fully attained all relevant Standards. The audits had included a surveillance audit and a provisional audit carried out nine months before the Ministry's inspection. The Ministry said that the "nature of the partial attainments at the time of the unannounced inspection suggests that at least some of this evidence must have been present at the time of the surveillance and provisional audits." |
| In another case, in 2008, a friend of a rest home resident made a complaint to the Health Consumer Service, alleging that the resident had been attacked by another patient and this was the latest in a string of incidents, including 10 falls in three months, and injuries including broken bones, lacerations, and bruising. A geriatrician who visited the resident was also very concerned about the treatment of this resident. A surveillance audit of the rest home in 2008 had not found any shortcoming in the safety and quality of care being provided to residents. In response to the complaint, the Ministry conducted an unannounced inspection of the rest home in early 2009. The inspection found evidence that substantiated the complaint as well as other failures in the standard of care. These failures included hot water temperatures that were too high and not monitored, a dirty fridge, skin tears that were not recorded, the facility was cold and residents were in summer clothing, hazardous chemicals were not stored appropriately, linen skips were uncovered, there were flies throughout the home, and intravenous fluids and urine testing strips were out of date. The Ministry inspection team concluded that there were 19 failings that needed attention and nine of these presented a high risk to the residents. The complaint was upheld and the manager of the rest home resigned. |
3.24
In our view, the Ministry has to strengthen how it oversees the work of DAAs. There is scope for the Ministry to consult more regularly with DHBs and organisations providing advocacy services for older people, because they often have access to information about the quality of care provided in particular rest homes. There is also scope for the Ministry to observe more DAA audits, and to better benchmark DAA performance.
| Recommendation 1 |
|---|
| We recommend that the Ministry of Health continue to strengthen how it oversees designated auditing agencies. |
3.25
As we noted in Part 2, the Director-General of Health is required by section 33(b) of the Act to designate external agencies if – and only if – the Director-General is satisfied that the external agency has the technical expertise needed, has effective auditing systems in place, and will administer those systems competently. Despite the information available to the Ministry, and the importance of consistent and reliable auditing by DAAs for certification, no DAA has ever had its designation removed, despite evidence of sustained poor performance.
| Recommendation 2 |
|---|
| We recommend that the Ministry of Health cancel the designation of audit agencies that continue to perform poorly. |
Effectiveness of progress reports in encouraging the safe provision of services
Progress reports are not effective enough as a tool to encourage rest homes to safely provide services to their residents. The information that rest homes provide in progress reports is inadequately verified, and some DAAs are behind in supplying progress reports to the Ministry.
Progress reporting is not always effective
3.26
Where the services provided by a rest home do not fully meet the criteria in the Standards, the Ministry usually requires the rest home to submit a written progress report within a certain amount of time. Progress reports are supposed to set out what action the rest home has taken to meet the criteria. Rest homes send the progress reports to their DAA, which forwards the reports to the Ministry.
3.27
Our file reviews showed that DAAs largely rely on rest homes reporting their own progress. They rarely verify progress with visits to the rest home. Some of the progress reports that we saw in our file reviews contained little detail about the progress that had been made, particularly in cases where the progress was reported by the DAA as satisfactory. In the files we reviewed, the Ministry accepted most progress reports and did not require the rest home to take any further action. The Ministry told us that it accepts most progress reports because the actions taken to fully meet the criteria are either complete or under way.
3.28
Progress reporting is not always effective in ensuring that improvements are made in rest homes. In our file reviews, we found that rest homes often fail in the same or closely related criteria. More than half of the rest homes in our sample received recurring "partially attained" ratings in one or more criteria. This indicates that progress reporting is not leading to sustained improvements.
3.29
In one case, a rest home was audited in November 2006 and found to not comply with the criterion that requires rest homes to securely store chemical products. The auditor rated this failure as high risk, and gave the rest home six weeks to fix the problem. When the Ministry issued the rest home with certification in January 2007, it required the rest home to submit a progress report on this issue within one month.
3.30
The rest home provided a report to the DAA claiming that all appropriate action had been taken to ensure that chemical products were securely stored. The DAA submitted this progress report to the Ministry within the month. The progress report did not indicate whether a site visit was made to verify that the problem was fixed.
3.31
A surveillance audit was carried out by the DAA in June 2008, and the problem of insecurely stored chemical products was found to have not been fixed. Another progress report was required. The DAA submitted the required progress report, stating that a lock had been fitted to the chemical storage cupboard in September 2008. The problem with the storage of chemicals in this rest home had continued for almost two years after it was first identified during an audit.
Progress reports are not always timely
3.32
DAAs are behind in submitting progress reports to the Ministry. In July 2009, one DAA had 80 progress reports overdue, dating back to November 2008. The Ministry was following up with several other DAAs that have overdue progress reports.
3.33
In 2009, the Ministry has worked with DAAs to strengthen progress reporting and a new system is being implemented. There will be more on-site verification of progress where there are issues considered to be high risk, and the Ministry will require better verification of the corrective actions that rest homes have reportedly taken.
Sharing and using information about certification
Until recently, information about rest homes has been poorly shared and used. Although we are pleased with the Ministry's recent efforts, we consider that it could do more with the information at its disposal to improve the effectiveness and reliability of certification.
Recent sharing of information with district health boards
3.34
In 2009, the sharing of information between the Ministry and DHBs improved markedly. DHBs have had access (since July) to full certification reports through a password-protected website. The Ministry has worked effectively with DHBs, along with providers of rest homes, on a pilot project to replace surveillance audits with unannounced (or "spot") audits.
3.35
Until 2008, there was little communication from the Ministry with DHBs in the course of making decisions on certification. Communication with DHBs has improved in 2008/09, with five DHBs stating in response to our survey that they are now regularly contacted by the Ministry on certification decisions (see Part 6) and there is greater sharing of audit reports between DHBs and the Ministry.
Better use of information
3.36
The Act allows the Ministry to use other information available to it when making decisions about the length of certification. Our survey of DHBs, our file reviews, and our interviews with senior advisors in the Ministry revealed that, until 2009, the Ministry had not made enough use of the information that DHBs have on rest homes.
3.37
For example, in our file reviews, only 4% of certification audit reports contained evidence that information from DHBs had been used in determining the length of certification. There are signs that this is now improving, especially with the new policy and procedure for recommending periods of certification that was introduced in May 2009, as well as the decision-making matrix introduced in 2008 (which was revised and improved in 2009).
3.38
Complaints systems and processes about rest homes provide a means, not only to right individual wrongs, but also to identify aspects of care that need to be strengthened and improved. Complaints also provide a useful means of assessing the level of risk associated with particular rest homes and they are referred to when the Ministry makes decisions on periods of certification. Complaints about rest homes can be lodged with rest homes, DHBs, the Health and Disability Commissioner, or directly to the Ministry. This could appear complicated to the public. The Ministry has worked with DHBs and the Health and Disability Commissioner to improve the handling of complaints so that it is more co-ordinated, and to make better use of information from complaints to inform decisions about certification.
3.39
After seven years of certification, the Ministry holds a large amount of information on the performance of rest homes, DAAs, and the sector in general. Until 2009, the Ministry had not analysed this information in detail. Recently, with an analysis of attainment ratings in medicine management in November 2009, the Ministry has begun to analyse information more. This will help it to identify common trends or themes and aspects of care in the rest home sector that need improvement. The analysis will also help to measure how effectively certification leads to improvements in the care of elderly people living in rest homes.
3.40
In our view, the Ministry could make more use of information from other organisations when making decisions. In particular, using information from DHBs could help to manage some of the risks that come with using eight different audit agencies. DHBs have knowledge about the performance and risks in rest homes and this information could be compared more regularly with the findings of DAA audit reports.
| Recommendation 3 |
|---|
| We recommend that the Ministry of Health continue to improve its use of auditing and certification information to identify common themes and trends in the rest home sector, and use that knowledge to identify how and where rest home residents are at greatest risk. |
Effectiveness of certification in encouraging rest homes to improve the quality and safety of their services
People working in the sector agree that certification has improved the quality and safety of services provided in rest homes. The rate of improvement appears to have slowed, and the performance of some rest homes has remained largely static since 2006.
3.41
Rest home operators, DAAs, and DHBs agree that the introduction of the Standards and certification have raised the quality and safety of services in rest homes, because they have forced a greater focus on the outcomes for residents. However, there is evidence that the rate of improvement has slowed.
3.42
If the standard of care provided in rest homes is improving, then there should be fewer "partially attained" ratings given by DAAs. In our file reviews, we found that the average number of "partially attained" ratings has remained static since 2006. In our interviews with representatives of the rest home sector, we were told that certification now has less influence over improvements. Our file reviews showed that some rest homes are consistently receiving "partially attained" ratings in the same or closely related criteria. Performance against some criteria – for example, medicine management – has been poor throughout the sector since 2003.
3.43
In our view, the Ministry needs to further analyse all the relevant information it holds to identify trends in the performance of rest homes. This trend information could be used to focus attention on those aspects of care where improvements are needed most.
3.44
Also, it is not clear that certification is still doing what the Act envisaged – encouraging rest homes to continuously improve the quality of their services, and encouraging them to take responsibility for providing those services to their residents safely. Therefore, in our view, the Ministry needs to reconsider whether the existing certification arrangements are the most effective it could use (see paragraphs 5.48-5.53).