Appendix: Findings and recommendations in our 2013 report
Earthquake Commission: Managing the Canterbury Home Repair Programme - follow-up audit.
In this Appendix, we describe what we found and recommended in our 2013 report. The information is organised around each of the recommendations we made in our 2013 report.
Paragraph number references are to our 2013 report.
Repair quality
| What we recommended that EQC should do | Continue to improve its approach to auditing repairs in the home-repair programme so the Commission is well informed about the scale and type of repair quality risks, can mitigate those risks where possible, and can match the resourcing of its quality assurance processes to the significance of those risks. |
| Findings in our 2013 report | Positive findings Site monitoring was carried out by contract supervisor staff. Monthly auditing of about a quarter of all completed repairs against repair work standards set for the programme. EQC started a post-repair completion survey in February 2013 that it intended to regularly administer. A quality assurance team was set up in March 2013 to review quality concerns. This team receives referrals from the complaints management process. (Paragraph 3.27.) Negative findings As at December 2012, quality controls were "yet to be fully embedded". The controls for ensuring compliance with the Building Code were not consistently followed. There were risks with the accreditation process because there was no centralised database containing all of the data relevant to contractors. Some contractors were accredited and inducted before criminal, credit, and conflict of interest checks became a routine part of the accreditation process. (A May 2013 internal audit report described these checks as "only recently" implemented.) The issuing of performance improvement notices (or PINs) to contractors had not been centrally recorded for most of the programme, with only six contractors (out of about 1200) losing their accreditation. There had been instances where the cost of fixing defects evident during the warranty period exceeded the amount of money withheld from the contractor until the end of the warranty period. (Paragraph 3.39.) |
Informing homeowners
| What we recommended that EQC should do | Continue to improve communication with individual homeowners about their claims, giving homeowners as much certainty as possible as early as possible. |
| Findings in our 2013 report | Positive findings EQC had prepared a set of criteria, in consultation with other agencies, to identify vulnerable people. (Paragraph 3.88.) From November 2012, EQC had set a monthly target of completing 100 vulnerable people's dwelling repairs and had generally met that target every month up to May 2013. (Paragraph 3.89.) EQC used its website, the media, its community contact centre, social media, an outbound calling programme, and EQC staff attendance at public meetings to communicate with homeowners. (Paragraph 4.21.) EQC knew that it needed to continue to improve communication with homeowners about their claims. EQC has previously acknowledged publicly that it has provided conflicting and inadequate information to claimants. (Paragraph 4.22.) EQC was planning, beginning in February 2013, to send letters to claimants providing information about the status of their claims. EQC's Chief Executive subsequently indicated that the shutting down of EQC's systems after a privacy breach delayed the time frame for informing claimants. Subsequently, EQC completed a 90-day programme for communicating with all claimants about their claims and was intending to continue its communication programme with these claimants. (Paragraph 4.23.) Negative findings Homeowners experienced inconsistency in information and processes, and long periods without specific information from EQC about their claim, leading to a lack of certainty while waiting for repairs; (Auditor-General's overview.) If the surveyed level of dissatisfaction with repairs in the programme in 2013 applied to the whole programme, then the owners of more than 14,000 repaired homes would be dissatisfied or very dissatisfied with the repairs. (Auditor-General's overview.) EQC did not know enough about homeowners' experiences of the home-repair programme. (Paragraph 3.69.) In our view, deciding to cash-settle a claim rather than manage repairs needed to include an assessment of the homeowner's capacity to manage the repairs themselves. (Paragraph 3.93.) As at May 2013, EQC was not completing repairs to properties with vulnerable occupants any "faster" than repairs to other properties. (Paragraph 3.94.) It was late in the programme before repair slots were actively allocated to the homes of vulnerable people. In our view, this was too late. (Auditor-General's overview and Paragraph 3.97.) In February 2013, EQC started surveying the satisfaction of homeowners who had just had their repairs completed. In our view, this work started too late. Communication, the availability of information, and the quality of repairs had been issues for homeowners. (Paragraphs 4.7-4.20.) EQC previously acknowledged publicly that it had provided conflicting and inadequate information to claimants. (Paragraph 4.22.) |
Key performance indicators
| What we recommended that EQC should do | Continue to refine key performance indicators for the home-repair programme to consistently and meaningfully cover cost, time, quality, and safety, with targets where practicable. |
| Findings in our 2013 report | Positive findings EQC performed well in managing repair costs and setting the programme up quickly. (Auditor-General's overview.) EQC set targets for finishing all repairs in the programme and for overall claims-handling costs as a percentage of total costs (this measure is wider than the programme). (Paragraph 5.31.) When we were finalising our report, EQC was reviewing the key performance indicators for the programme. We encouraged EQC to also consider the relationship between the key performance indicators and the objectives of the programme. (Paragraph 5.36.) Repair costs had been reasonable to date, but there were risks to that continuing. (Paragraphs 6.14-6.35.) Negative findings Some metrics reported to the Project Control Group were not consistently reported (for example, the results of quality audits of completed repairs), some metrics had no targets in the reported material (for example, the number of houses to be repaired each month), and there was no comparative data reported for some metrics (for example, the number and types of complaints received). (Paragraph 5.30.) "Value for money" metrics were identified in monthly reports to the Project Control Group. These were percentage differences between:
Our analysis suggested that project management costs (on average, about 12% of the cost of a repair) had been at the higher end of what we considered reasonable in the circumstances. Achieving reasonable project management costs at the end of the programme depended heavily on EQC completing the programme by December 2014, managing the hub reconfiguration project effectively to deliver the expected benefits, and continuing to control repair-cost inflation. (Paragraphs 6.36-6.54.) |
Programme configuration
| What we recommended that EQC should do | Continue to review and, if necessary, adjust the configuration of repair and project management services in the home-repair programme to deliver the best value and results in the circumstances and treat homeowners fairly and consistently. |
| Findings in our 2013 report | Context Achieving reasonable project management costs at the end of the programme depended heavily on EQC:
Positive findings Repair costs had been reasonable to date, but there were risks to that continuing. Keeping repair costs at a reasonable level depended on EQC managing essential controls and systems, staying ahead of the private insurance and central city repair and rebuild work, and completing the programme by the December 2014 deadline set by EQC. (Auditor-General's overview.) EQC was committed to improving how it manages the programme and had acted on the findings of various reviews and audit work. (Paragraphs 7.6-7.9) EQC prepared and put in place a sensible internal audit work programme, acted on the findings of our financial audit work, and responded to the findings of various reviews it commissioned on aspects of its operation. (Paragraph 7.6.) The recovery environment in Canterbury was continuing to evolve. EQC needed to continue to assess and amend aspects of the programme to ensure that it was appropriate as the environment changed. (Paragraph 7.7.) When we finalised our report, EQC was in the process of reviewing and/or changing important aspects of the home-repair programme, including governance arrangements, key performance indicators, and repair hub structures and locations. The hub reconfiguration was intended to support greater consistency of practice within the programme. It was important that EQC continued these review and improvement activities. (Paragraph 7.9.) Negative findings Our analysis suggested that project management costs (on average, about 12% of the cost of a repair) were at the higher end of what we considered reasonable in the circumstances. Achieving reasonable project management costs at the end of the programme depended heavily on EQC completing the home-repair programme by December 2014, managing the hub reconfiguration project effectively to deliver the expected benefits, and continuing to control repair-cost inflation. (Paragraphs 6.36-6.54.) |
Lessons learned
| What we recommended that EQC should do | Identify and record the lessons, tools, and information from the home-repair programme that could usefully support responses to future large-scale natural disasters. |
| Findings in our 2013 report | Context EQC had no comparable situations to draw direct experience and lessons from given that the scale and complexity of the repair activity was unprecedented in New Zealand. The affected population was a higher proportion of the country's total population, a much higher proportion of damage was covered by insurance, and the effect on the overall economy was proportionally bigger, especially when compared to the effects of large-scale natural disasters in other countries. (Auditor-General's overview.) Progress was complicated by more earthquakes and the need to apportion damage correctly to each earthquake, evolving repair techniques and guidance, and the effect of "zoning" land in Christchurch. Complications did not arise in a linear sequence but with many complexities coinciding (including land remediation and dwelling repairs). (Auditor-General's overview.) Public entities naturally concentrate their planning around likely events. But the uncertainty and complexity of the contemporary world mean that this alone is not enough to serve the future needs of New Zealanders well. (Auditor-General's overview.) Public entities need to sensibly prepare for potentially catastrophic but unlikely events. Those events can require public entities to deliver large and complex initiatives that must be quickly set up. Examples of such events include the failure of significant financial institutions, a large mining disaster, a global pandemic, a shipping disaster, or a food contamination scare. (Auditor-General's overview.) Being prepared for these types of situation is difficult but possible. Although detailed action planning cannot be done before an event, entities can prepare a coherent strategic approach, or framework, ahead of such events. A disciplined approach is required when responding to these events, particularly once the immediate emergency phase of the event has passed. (Auditor-General's overview.) EQC's Board and Ministers decided on a home-repair programme because of its potential to contain inflation in the cost of repairs, ensure that funds were used for repairs, and maintain the quality of housing stock in Canterbury. Maintaining the quality of housing stock was considered important in encouraging people to say in the region. (Paragraph 2.2.) Positive findings EQC subjected itself to scrutiny and review through internal audit and commissioned review work. This scrutiny and ongoing improvement needed to continue. (Paragraph 7.2.) Negative findings The considerable inconsistency in the repair process and in the information recorded was a consequence of EQC concentrating on getting repairs done before supporting systems and controls had been fully prepared and implemented. (Paragraphs 4.49 and 4.50.) |
Complaints
| What we recommended that EQC should do | We did not make a recommendation relating to EQC's management of complaints. |
| Findings in our 2013 report | Positive findings EQC identified two actions to improve complaints management, in response to the findings of the May 2013 internal audit of the programme: a review of EQC's complaints process based in Wellington, before complaints were transferred to Fletcher Construction; and a "follow-up" to the process used from November 2012 where claims not resolved within 10 days by Fletcher Construction were referred back to EQC. (Paragraph 4.28.) The internal audit work confirmed that these actions have been carried out. (Paragraph 4.29.) Negative findings Complaints had been made directly to EQC and to other organisations about the programme. EQC needed to better integrate how complaints were managed between it and Fletcher Construction. (Paragraphs 4.24-4.30.) EQC needed to do more to analyse and learn from complaints and better integrate complaints management processes and systems with Fletcher Construction. The number of complaints received was one means by which EQC intended to judge the success of its performance in response to a large-scale natural disaster. (Paragraph 4.24.) EQC and Fletcher Construction began integrating complaints information from late November 2012, but their complaints systems were not technically connected. There were inconsistent complaints processes between repair hubs. (Paragraph 4.27.) |