Part 4: Managing risks in the certification arrangements

4.1
In this Part, we discuss the Ministry's management of:

• risks generally;
• the risk of conflicts of interest;
• the risk that rest homes might select the cheapest or most lenient DAA;
• the risk that commercial pressures might influence an auditor's independence;
• the risk that DAAs might interpret and audit the Standards differently; and
• the risk that auditors might have inadequate skills and expertise.

Our overall findings

4.2
The Ministry could have responded more quickly to risks identified in the certification arrangements – particularly risks to the independence of the auditors, because auditor independence is integral to the certification arrangements. The Ministry now has a risk register, and has plans for managing several of the most significant risks.

General risk management

The Ministry's risk management has not been thorough, particularly in relation to the many contractors who carry out the audits for DAAs. The Ministry's actions have also tended to be more reactive than proactive.

4.3
The design of the system for certification has some inherent risks. In 2008, the Ministry acknowledged that the risks in rest homes choosing their DAA, and the competition between DAAs for business, had the potential to create a "moral hazard". Managing the performance of the different DAAs is also a challenge for the Ministry, particularly because most of the auditors who work for DAAs are self-employed contractors. In September 2009, the Ministry developed a risk register for managing these and other risks. In our view, this should have been introduced sooner, given the level of risk that the Ministry had known about and acknowledged in Cabinet papers written in 2008.⁷

4.4
We are pleased that the Ministry now has a risk register to help it manage the risks associated with certification. In our view, the Ministry needs to continue to improve its risk management by working with DHBs and rest homes to identify the severity and likelihood of risks in certification. Because risks change over time, the Ministry will need to review its risk register each year.

Risk of conflicts of interest

The Ministry's efforts to manage potential conflicts of interest have not been wholly effective. The Ministry intends to publish a code of conduct for DAAs and the contracted auditors they use. In our view, the Ministry should have done this sooner.

4.5
There is a risk that DAAs may offer services over and above certification auditing, which could compromise the objectivity of their audits. The likelihood of this risk is high because many DAA auditors provide consultancy and other services in addition to auditing. Also, most auditors are independent contractors and DAAs have limited control over what they do outside the audits they carry out on the DAAs' behalf.

4.6
We found examples, in our file reviews and interviews, of instances where:

• a DAA was advertising its consulting services in its auditing brochures;
• a DAA wrote to a rest home, offering consultancy advice on the Standards before the DAA carried out an audit; and
• a DAA contractor sold consultancy services to a rest home after carrying out a certification audit of that rest home.

4.7
The Ministry has written to one DAA, after discovering a conflict of interest, to request that the activity cease. The Ministry proposes, in the latest draft of the DAA Handbook (which is out for consultation), that all auditors complete a conflict of interest declaration before each audit.

How the Ministry manages this risk

4.8
To manage the risk of conflicts of interest, the Ministry checks each DAA's internal audit reports and corporate brochures when the DAA applies for re-designation.

Our view of the Ministry's management of this risk

4.9
In our view, the Ministry needs to check more thoroughly that DAAs have adequate systems in place to prevent conflicts of interest. The Ministry told us that it will implement methods of identifying and preventing conflicts of interest. We consider that the process needs to be in place quickly, and certainly before the end of 2010.

4.10
There is also scope for more regular monitoring of conflicts of interest. Better communication between the Ministry and rest homes could improve the Ministry's knowledge of the types of services DAAs provide or offer, in addition to auditing.

4.11
The Ministry has prepared a code of conduct, which is now in the draft DAA Handbook, that all auditors of rest homes will have to comply with. Again, we consider that this needs to be in place no later than the end of 2010.

Risk that rest homes select the cheapest or most lenient designated auditing agencies

The Ministry has found it challenging to manage this risk effectively. It has recently started to gather information from DAAs about their audit fees, and intends to use it to analyse market share and performance factors. In our view, the Ministry should have been doing this since certification was introduced.

4.12
Allowing rest homes to choose their DAA is intended to introduce competition among DAAs. The intention of competition is to provide an incentive to DAAs to improve quality and to help keep the costs down. But competition also brings risks. The relationship between rest homes and DAAs can become compromised. Rest homes might choose DAAs on the basis of cost rather than quality. DAAs, in an effort to win business, might offer consultancy services to rest homes, which would compromise the objectivity of future audits. To keep costs down, contracted DAA auditors might carry out audits in too short a time.

4.13
In 2008, a Cabinet Paper described these risks as a "moral hazard" that the Ministry has found difficult to manage effectively.

4.14
In our view, there is a high risk that rest homes might select their DAA based on price or leniency rather than on quality.

How the Ministry manages this risk

4.15
The Ministry has begun to collect information on audit fees from the DAAs, intending to compare this with market share and performance of DAAs, as well as information from rest homes.

Our view of the Ministry's management of this risk

4.16
In our view, the Ministry should continue to collect information about audit fees and attainment ratings. Analysing this information will help the Ministry to manage this risk by alerting the Ministry to any major inconsistencies in audit fees and ratings.

4.17
We would have expected the Ministry to have been collecting this information since certification was introduced in 2002.

Risk that commercial pressures might influence an auditor

The Ministry checks whether the evidence in audit reports corresponds to the ratings that auditors provide. The Ministry could do more, like comparing the ratings that different DAAs and different contracted auditors have given and looking for unusual trends.

4.18
There is a risk that commercial pressures might unduly influence an auditor. A DAA could provide the Ministry with a more positive report than a rest home's performance warrants, to retain the rest home as a client.

How the Ministry manages this risk

4.19
The Ministry checks the evidence in audit reports against the attainment ratings, to make sure that they match. The manager of HealthCERT writes to individual DAAs when there is evidence of poor auditing practice. Auditors are aware that their reports will be checked, and this approach should help to ensure that the ratings they give are well considered.

4.20
We note that by observing audits by DAAs, the Ministry will be able to compare the rigour and quality of auditing by different DAAs and their contracted auditors.

Our view of the Ministry's management of this risk

4.21
The Ministry needs to supplement its existing methods of managing this risk by analysing the information from past audit reports. It would allow the Ministry to compare the different levels of attainment given by different DAAs and their contracted auditors. This could reveal any unusual trends that might need to be investigated or managed more closely.

Risk that designated auditing agencies might interpret and audit the Health and Disability Services Standards differently

In our view, the Ministry could and should have managed this risk more actively and earlier. The number of criteria to be checked during an audit is likely to be adding to the risk of inconsistency.

4.22
There is a risk that DAAs might interpret and audit the Standards differently. For example:

• rest homes, DHBs, and Age Concern local offices all told us about inconsistency in the interpretation of the Standards by DAAs; and
• in our file reviews, we found that some DAAs reported on an exceptions basis – stating if a rest home did not meet a criterion, rather than providing evidence that all criteria were met.

4.23
An operator with multiple rest homes and the managers of rest homes we interviewed told us that different DAAs, and sometimes different auditors from the same DAA, interpret the Standards and criteria differently and apply different levels of judgement. This is supported by the results of our file reviews – when rest homes change auditors there is often a significant change in the number of "partially attained" ratings (see Figure 6). There is also large variation in the average number of "partially attained" ratings given by different DAAs – from 7 to 24.

4.24
We would expect to see some variation in the average number of "partially attained" ratings given by different DAAs. However, the extent of variation between the average ratings given by different DAAs indicates a degree of inconsistency that risks the reliability of decisions about periods of certification made by the Ministry.

Figure 6
Changes in ratings when rest homes change their designated auditing agency

Number of criteria to be checked

4.25
The large number of criteria that must be checked for certification within the time that is normally provided creates risks to consistency. When a DAA audits a rest home for certification the auditors must check at least 206 criteria. The auditor needs to check each one and provide evidence supporting the level attained for each of the criteria.

4.26
The criteria are intended to be outcome-focused. This means that auditors are expected to check that policies and procedures are leading to the intended outcomes. For example, DAAs should check that rest homes have the correct procedures in place for staff training, but also check, through staff interviews, that staff have received adequate training and that they have the right skills as a result.

4.27
Some criteria take longer than others to check. For example, checking that the records kept about residents are legible (criterion 1.2.9.9) is straightforward. Checking that residents "and where appropriate their family/whānau of choice or other representative" are actively involved when different stages of service are planned for them (criterion 1.3.3.2) requires more in-depth work, including interviews with residents and their families. It is a significant challenge for auditors to check all criteria thoroughly in the time available to them (which includes, at most, two days on site).⁸

How the Ministry manages this risk

4.28
The Ministry manages the risk that auditors might interpret the Standards differently by producing revised versions of the DAA Handbook. The Ministry held workshops throughout 2009 on the latest version of the Standards. Ministry advisors also check the evidence in DAA audit reports against attainment ratings to judge if they match appropriately.

4.29
During 2009, the Ministry introduced a new audit reporting template to bring greater consistency in reporting. It introduced workshops and training sessions to support greater consistency in auditing and interpretation of the Standards. In September and October, the Ministry analysed its evaluations of audit reports.

4.30
The Ministry is planning to introduce audit-specific accreditation of DAAs in 2010, in an effort to bring greater consistency. All DAAs will have to be accredited by one of two organisations, which will be approved by the Ministry (see paragraphs 5.11-5.16).

Our view of the Ministry's management of this risk

4.31
In our view, the Ministry could and should have managed this risk more actively and earlier.

4.32
As well as the measures already under way and planned, the Ministry needs to continue its analysis of audit report evaluations and use it to benchmark the performance of DAAs. We also consider that the Ministry's information about DAA audit reports should be available to rest homes.

4.33
A risk-based approach to selecting which criteria to audit most closely would be helpful. For this to happen, greater knowledge and intelligence sharing between DHBs, DAAs, the Ministry, and non-governmental organisations will be needed.

Risk of inadequate skills and expertise

The Ministry has started to put measures in place to manage the risk that auditors of rest homes lack the necessary skills and expertise. In our view, the Ministry should have acted sooner.

4.34
There is a risk that auditors might lack the skills and expertise required to audit rest homes effectively.

4.35
In the files we reviewed, 21% of the evaluations carried out by the Ministry's senior advisors indicated that the audit report had failed to identify the technical expertise of the auditor. The Ministry is aware that some audits are of a poor quality.

How the Ministry manages this risk

4.36
In 2003, a senior advisor raised concerns about the quality of audits carried out by a particular DAA contractor. The Ministry did not write to the DAA about these concerns until 2006. The Ministry told the DAA that the auditor did not have the appropriate clinical qualifications to carry out health and disability audits on their own. The auditor could be part of an audit team that included a member with appropriate clinical qualifications. The auditor continued to conduct certification audits in the meantime. In our file reviews, this auditor was the sole auditor on three audits before the letter was sent to the DAA. After the letter, the auditor carried out at least one more audit on their own for a different DAA.

4.37
To manage the risk that auditors used by DAAs do not have the necessary skills and expertise, the Ministry has issued guidelines for auditor competency, based on AS/NZS ISO 19011:2003: Guidelines for quality and/or environmental management systems auditing. This standard describes the training and performance management requirements DAAs should have in place to manage the competency of the auditors they employ, which should include an evaluation of auditors while they carry out an audit.

4.38
The latest draft of the DAA Handbook includes a requirement that DAAs employ or contract with auditors who have gained a qualification in auditing quality management systems. The qualification is based on a five-day training course, and does not include a work-based assessment or examination.⁹

4.39
In 2009, the Ministry established a register to monitor the work of DAA auditors more closely. DAAs must now provide the Ministry with evidence that each of their auditors has had their performance reviewed at least once a year. The Ministry has clarified in the draft DAA Handbook the requirements for minimum qualifications and ongoing competency.

Our view of the Ministry's management of this risk

4.40
In our view, the competency and qualification requirements for DAAs could be clearer. The assessment of skills and competencies needs to be more rigorous. It could include a qualification that requires an examination and a work-based assessment.

7: The minutes of a Cabinet Social Development Committee meeting in 2008 stated that there "is an inherent risk of conflict of interest in having agencies auditing their own clients."

8: The Ministry requires a two-stage audit process. In stage one, auditors review documents and policies (usually not at the rest home premises). In stage two, the auditors visit the rest home to check that the policies are complied with. Not all DAAs have followed this process.

9: Unit 8086, Demonstrate knowledge required for quality auditing, New Zealand Qualifications Authority.