Part 5: Recent efforts and plans to improve certification

5.1
In this Part, we discuss:

• a review of the Act in 2007;
• the Ministry's overall work programme to improve certification; and
• other changes the Ministry could make to improve certification.

Our overall findings

5.2
In 2008, the Ministry began to address the issues with certification more thoroughly. In 2009, it has quickened the pace of improvements, working closely with DHBs, DAAs, and rest homes.

5.3
It is too early to judge whether the Ministry's work will lead to more consistent and more reliable certification of rest homes, or whether other, more fundamental, changes are needed to the certification arrangements.

Review of the Health and Disability Services (Safety) Act 2001

The Ministry reviewed the Act in 2007, identifying risks and weaknesses that needed to be addressed.

5.4
In 2007, the Ministry reviewed the Act to improve the processes and enforcement that assure the Director-General of Health that services are safe, and to look at which services the Act should cover. The review's findings included concerns about:

• the quality and consistency of the performance of some DAAs;
• an inherent risk of conflicts of interest in having DAAs auditing their own clients;
• the Act giving the Ministry little control over the quality and number of DAAs;
• the limited range of sanctions available to the Ministry where DAA performance is poor, except cancelling the designation; and
• the processes of gaining, retaining, and losing certification being unclear, and in some cases incoherent.

5.5
After the review, the Cabinet Social Development Committee concluded that the concerns identified in the review could be managed by the Ministry without changing the Act. Some changes would be needed, to make clear that auditing expertise and knowledge of the service sector (in this case, rest homes) is required.

Work programme to improve certification

The Ministry's work programme to improve certification is extensive and well managed. It is too soon to determine whether the various projects will produce the desired results.

5.6
In 2008, the Ministry prepared a wide-ranging and extensive work programme to improve certification. Within that work programme, it has a project to improve the effectiveness and efficiency of the certification of rest homes, which includes:

• improving the quality of DAA auditing by again requiring third-party accreditation;
• publishing summaries of DAA audits on the Ministry‘s website;
• introducing unannounced surveillance audits;
• improving complaints management;
• improving information sharing throughout the sector; and
• reducing multiple audits and removing duplication.

5.7
After a period of high staff turnover in the HealthCERT team, the Ministry has recruited staff with experience in the care of older people and from the wider health care sector. There are now five senior advisors, supported by two analysts and two co-ordinators. The team is led by a team leader and the manager of the Ministry's Quality and Safety team.

5.8
The Ministry has begun to analyse the information it holds on rest homes and DAAs, and has commissioned research to determine more clearly the type of expertise and skill that should be required of auditors. Recently, the Ministry has started to:

• introduce a programme of observing audits by DAAs;
• benchmark the performance of DAAs;
• provide more education sessions to DAAs; and
• evaluate more thoroughly the compliance of DAAs with the DAA Handbook.

Unannounced surveillance audits

5.9
As noted in Part 2, audits for certification are arranged between the rest home and the DAA. This means that the rest home has time to prepare for the audit and knows when it is going to take place. There is a risk that what auditors see on the day of an audit may not necessarily reflect the standard of services normally provided by that rest home.

5.10
To manage this risk, the Ministry is working closely with DAAs, rest homes, and DHBs to introduce a different approach to surveillance audits. Surveillance audits will be unannounced and will focus largely on the care that is provided to the rest home residents. Unannounced surveillance audits are being piloted in 12 DHBs and will be introduced nationally in 2010. Certification audits will continue to be pre-arranged.

Plans to re-introduce third-party accreditation

5.11
When the Ministry has focused on rest homes, its staff have had to attend to the inconsistencies in DAA auditing and the poor performance of some DAAs. This reduces the Ministry's ability to focus on the quality and safety of the services provided in rest homes.

5.12
The Ministry proposes that the Act be amended in 2010 to include a requirement that all DAAs be accredited by a Ministry-approved body. Although the Ministry removed the accreditation requirement in 2006, six of the eight DAAs remain accredited by an accreditation body. The current arrangements check general auditing systems and do not focus specifically on health and disability auditing.

5.13
The Ministry has worked with two accreditation organisations, the Joint Accreditation System of Australia and New Zealand (JAS-ANZ) and the International Society for Quality in Health Care (ISQua), to prepare a health and disability component that DAAs will have to comply with to be designated. This means that they will be subject to more, and more regular, reviews. The reviews will check that the auditing of DAAs meets international standards of good practice and that the DAAs are well-managed organisations.

5.14
If a DAA passes the accreditation review, it will be accredited for four years and will be subject to regular monitoring, progress reporting, and surveillance auditing. The frequency of monitoring and auditing will depend on the level of confidence the accreditation body has in the DAA, but will include at least a two-yearly surveillance audit (including witnessed audits) and annual reporting requirements.

5.15
The Ministry expects third-party accreditation to help to reduce the amount of time it spends on overseeing DAAs. The accreditation organisations will be responsible for monitoring the quality and consistency of DAA's auditing and their management systems.

5.16
The Ministry also expects an improved system of third-party accreditation to provide more stringent checks of DAAs (see Figure 7). Better communication between the Ministry and DAAs is expected to help all parties to be clear about the expected standard of auditing. The improved communication between DHBs and the Ministry is also expected to enhance decision-making about periods of certification.

Figure 7
What third-party accreditation is expected to mean for designated auditing agencies

Source: Ministry of Health.

5.17
We note that, in 2006, the Ministry decided that third-party accreditation was ineffective. It remains to be seen whether the new third-party accreditation arrangement will improve the consistency and quality of DAA auditing. In our view, the Ministry needs to gather and evaluate information as soon as possible about the effectiveness (or otherwise) of third-party accreditation.

Our view of the Ministry's work programme

5.18
The Ministry's wider work programme has made good progress. There is better communication between the Ministry, DAAs, and DHBs, and plans for third-party accreditation are well advanced. The Ministry has also been monitoring more closely those DAAs where performance is known to be at risk. In one instance, it has commissioned an external audit of a DAA's management systems and auditing practice.

5.19
The improvement project has been well managed. The rest home sector has been consulted through a reference group that includes representatives from rest homes, DHBs, organisations that act and speak on behalf of older people (Age Concern and Grey Power), and DAAs.

5.20
The different streams of work the Ministry has carried out in 2009 are important steps towards improving the quality and consistency in auditing rest homes for certification. However, it is too early to determine how effective the Ministry's improvement programme will be in making certification more effective and reliable.

Other changes the Ministry of Health could make

In our view, the Ministry needs to do more than the measures planned for its project to improve the certification of rest homes. We encourage the Ministry to consider whether the current certification arrangements are the most effective it could use.

5.21
Contracting auditing to private firms is not unique to the health sector. For example, our Office, which is responsible for auditing more than 4000 entities, contracts some of our auditing responsibilities to private audit firms. We use four steps to manage the contracts we have with private audit firms. These steps are:

• appointing auditors;
• setting standards for auditing;
• providing technical advice and assistance; and
• carrying out quality assurance reviews.

5.22
The Ministry has been carrying out each of these four steps, but there is scope for the Ministry to improve how it does so.

Designating the auditors of rest homes

5.23
Only designated auditors can audit rest homes for certification purposes. Audit companies apply to the Ministry for designation and must meet the criteria set out in section 33(b) of the Act. The Ministry conducts desk-based checks to verify that DAAs comply with these criteria when they apply to renew their designation.

5.24
The Ministry has found problems with DAAs that could have been prevented if the systems DAAs are required to have had been properly implemented. No DAA has been refused re-designation, even when there has been evidence of poor performance.

5.25
Rest homes choose which DAA to use for certification audits and there are no requirements for them to change their DAA after a specified period. In our file reviews, 82% of the rest homes had used the same DAA since certification was introduced in 2002.

How designation could be improved

5.26
In our view, the Ministry needs to make more stringent and rigorous checks of DAA systems and processes – including systems to assure the auditor's independence and identify and manage potential conflicts of interest – at the time of re-designation. DAAs will have little incentive to improve unless sustained poor performance or an inadequate regard for proper systems and processes leads to the Ministry cancelling a DAA's designation.

5.27
Requiring rest homes to change the DAA they use after a specified period would help to manage the risk of conflicts of interest and the risks to auditor independence. It would also help the Ministry to identify inconsistencies in how auditors interpret the Standards and in their auditing practices.

Setting standards for auditing

5.28
The Ministry publishes the Standards, which are not specific to rest homes but apply to the whole health and disability sector. The Ministry produces guidance in the form of the DAA Handbook, to help DAAs interpret and apply the Standards when they audit rest homes. DAAs must comply with the DAA Handbook as a condition of their designation.

5.29
The latest version of the DAA Handbook has been in draft form since May 2009. It requires that audits be carried out in keeping with AS/NZS ISO 19011:2003. This is a standard that covers quality and environmental management systems auditing.

5.30
Lead auditors¹⁰ must meet the skill and competency requirements of AS/NZS ISO 19011:2003. The Ministry will also require DAA auditors to have a qualification in auditing quality management systems (see paragraph 4.38). When third-party accreditation is reintroduced, audit teams will have to comply with ISO/IEC 17021:2006 (an international standard that will provide assurance that DAAs have suitable management systems in place). There is also a requirement that teams auditing rest homes include at least one registered nurse.

5.31
DAAs have sought clarification on auditor competency since the introduction of certification.

5.32
Most auditing is carried out by freelance auditors contracted to DAAs. This makes it difficult for the Ministry to measure compliance. In 2009, the Ministry introduced a new register of all auditors to help it oversee auditor competency.

How the setting of standards could be improved

5.33
In our view, the DAA Handbook would be more useful if it contained more detailed information about the standard of auditing expected of DAAs, and how DAAs will be measured against these auditing standards.

5.34
Third-party accreditation is expected to make the standard of auditing more consistent and clear. DAAs will have to comply with ISO/IEC 17021:2006. In the draft DAA Handbook, the Ministry has included further qualification requirements (see paragraph 4.38). However, the Ministry will need to provide DAAs with greater clarity about the competency requirements of auditors.

Providing technical advice and assistance

5.35
From 2002 to 2008, communication between the Ministry and DAAs was poor, and DAAs received patchy and inconsistent responses to requests for technical advice. This has improved in 2009.

5.36
Until recently, Ministry staff spent most of their time checking audit reports rather than providing advice. This was a cause of frustration for some staff, who felt that their experience and expertise was not being used. There was a period of high staff turnover but the Ministry's HealthCERT team is now fully staffed. The senior advisors within the team have experience relevant to their roles.

5.37
In 2009, the Ministry introduced a programme of quarterly "train the trainer" workshops for DAAs. These workshops include technical advice and guidance.

How the technical advice and assistance could be improved

5.38
In our view, the Ministry needs to continue to maintain communication with DAAs. We also consider that the HealthCERT team needs to continue to build its skills, given the number of new staff in the team.

5.39
We note that the DAA Handbook has been in draft format since May 2009. The Ministry's support for DAAs would be improved if the guidance that DAAs rely on was properly published and authorised.

Carrying out quality assurance reviews

5.40
The Ministry's monitoring of DAAs mostly happens when DAAs send in their audit reports. The Ministry has recently started to observe DAA audits to check whether the audits meet the expected standards, and seeking feedback from rest homes and DHBs on the quality of DAA auditing.

5.41
From 2002 to 2008, there was little regular communication between the Ministry and DAAs. DAAs received little feedback on the quality of their auditing. In our file reviews, the Ministry had completed evaluation reports for 30% of the audit reports. Of these evaluation sheets, 21% record that the technical expertise of the auditors was not identified, but there is no evidence that this information was communicated to DAAs or to the New Zealand Association of Designated auditing agencies (the NZADAA).¹¹

5.42
In October and November 2009, the Ministry observed two DAA audits and plans to observe more in 2010. The Ministry is evaluating audit reports more thoroughly, and has improved its evaluation template.

How quality assurance could be improved

5.43
The Ministry expects third-party accreditation to introduce more rigour to the quality assurance of DAAs. It will include observing the audits that DAAs carry out.

5.44
DAAs told us that they would like to receive more regular feedback from the Ministry. The Ministry has responded to this by communicating more regularly with the NZADAA, and now meets with the NZADAA every two months.

5.45
In October 2009, the Ministry analysed a sample of the information recorded on the new evaluation sheets. In our view, the Ministry needs to continue this work and use the information it has to compare and benchmark the different DAAs.

5.46
The quality assurance of DAA auditing would also be improved if the Ministry checked DAA auditing more rigorously, and observed DAA audits more regularly.

5.47
In our view, the Ministry cannot rely solely on third-party accreditation to ensure the quality of DAA auditing.

Considering other ideas or approaches to certification

5.48
We are pleased that the Ministry has increased its efforts to improve the effectiveness of certification, and appears committed to making many changes. However, because the weaknesses in the certification arrangements are only now being adequately addressed, we have not been able to form a view about whether the current arrangements are the most effective and reliable.

5.49
For example, we have noted our views on having more than 206 criteria to check during the time available for a certification audit. We have also noted that, in the sample of files we reviewed, the average number of "partially attained" ratings for each audit has hardly changed since 2006.

5.50
There are currently eight DAAs. If the Ministry were to cancel the designation of every DAA that has failed to meet the criteria in section 33(b) of the Act, the consequences for certification auditing, and the wider health and disability sector, might be serious.

5.51
We also note that our Office changed several years ago from a competitive model to an allocated audit model, to encourage greater sharing of information among auditors and with us – which is essential to effective risk identification and management.

5.52
Although the existing legislation appears enabling and permissive, it is possible that the Ministry may not be able to make the certification of rest homes more effective and reliable without legislative change.

5.53
We encourage the Ministry to remain open to other ideas or approaches to the certification of rest homes.

10: The lead auditor in an audit team manages the audit and is responsible for authorising the audit report before it is submitted to the Ministry.

11: The NZADAA is made up of representatives from each DAA, and represents the interests of DAAs.