Appendix 5: Findings and prompt action for non-compliance

The Civil Aviation Authority's progress with improving certification and surveillance.

A5.1
In 2005, we recommended that:

  • CAA auditors issue a finding for all identified instances of non-compliance and non-conformance; and
  • the CAA establish a system that ensures that operators take quick and effective corrective action when auditors tell them to do so. This system should include re-assignment of responsibility for that function when an auditor leaves the CAA.

A5.2
In this Appendix, we look at:

Our overall findings

A5.3
The CAA has implemented a system to ensure that operators appropriately address findings.

A5.4
There is significant variation in the number of findings issued by auditors from within the same units. We expected the unit managers to have investigated the reason for these differences and to have taken action to moderate the number of findings issued by the auditors they are responsible for. In our view, not doing so raises questions about the quality of the surveillance audits carried out, and also, from an operator's point of view, raises questions about the consistency and predictability of the regulatory focus.

A5.5
Corrective actions that CAA auditors had identified to resolve the findings were not always addressing the "root cause" of the findings. This means that the reason for the non-compliance will not necessarily be addressed and there is an increased risk that the operator will revert to being non-compliant.

A5.6
The number of major and minor findings not being closed by their due dates has been increasing since our 2005 audit, and the CAA is falling well short of its target. In our view, the CAA needs to address the timely closing of findings, because while the findings remain unaddressed the operator is non-compliant with the Rules and could potentially pose a higher safety risk.

A5.7
We consider that our 2005 report recommendations have been only partly addressed.

Issuing findings

We found instances where auditors had not issued findings as required, and instances where it was not clear why there were no findings in response to problems found during inspections.

A5.8
Recording all instances of non-compliance in the electronic surveillance tool and issuing findings is important because:

  • the number and severity of findings recorded affects an operator's risk profile, so not recording findings reduces the risk profile of the operator; and
  • the CAA intends to start analysing the type and frequency of findings, so not recording them may hide an issue that needs to be dealt with (for example, through education or changing a Rule to clarify its intent).

A5.9
The new surveillance process is designed so that, if non-compliance with the Rules is identified and entered in the tool by the auditor, a finding is generated. Administrative staff now monitor the closure of findings and follow up on overdue findings by sending reminders to the operators. The auditor who issued the finding is responsible for checking that appropriate evidence is provided to the CAA that the operator has fixed the non-compliance. We saw evidence that this was happening.

A5.10
However, auditors were still not always noting instances of non-compliance in the system and issuing findings. During our audit, we found instances in the Airlines Group and the General Aviation Group where findings had not been issued for identified non-compliance. For example, for one general aviation operator, an auditor found that the main rotor blade of a helicopter had been in use for more than 20 hours beyond its airworthiness limit. No finding was issued for this non-compliance. In another example, for an airline operator, the auditors noted some problems with the adequacy of procedures for co-ordinating crew during an en-route audit. The manager noted that the operator needed to review these procedures, but it was not clear why no finding was issued.

A5.11
The operators we spoke to during our audit were dissatisfied that they were not receiving the findings at the end of the audit, but rather that findings were emailed or sent several days – or in one case up to two weeks – after the audit was completed. Several operators also told us that some of the findings were not expected and had not been discussed with them at the time of the audit. This is contrary to the CAA's internal policies.

Timeliness of issuing findings

We found examples of findings where the issues identified dated back several years but were not identified in earlier audits or certifications.

A5.12
For six of the operators in our sample, we found examples where the operator had been non-compliant for several years before a finding was issued. These instances of non-compliance should have been identified at earlier audits or as part of certification. For example, findings were issued:

  • during a spot check in December 2008, because it was noted that airworthiness directives had been inappropriately recorded and certified in the aircraft logbooks since December 2006;
  • in March 2007, because there was no evidence of an assessment and record of the manufacturer's service information since April 1998; and
  • because an aircraft was operated from the left-hand seat and the flight manual required operation from the right-hand seat. For 15 years, the aircraft had been certified and had been operated from the left-hand seat.

Consistency in issuing findings

We found bigger variations in the numbers of findings issued by auditors from within the same units than we had expected.

A5.13
Auditors are required to apply judgement when deciding what the Rules mean and whether the procedures followed by the operators meet the Rule requirements. This means that there will not be complete consistency between auditors.

A5.14
Several of the operators we spoke to disagreed with the auditor's interpretation of some Rules and were concerned about the confrontational attitude some CAA auditors adopt when the operators challenge them about Rule interpretation. We were told that there was a general concern within the industry that challenging the auditor would result in the CAA being more critical of the operator.

A5.15
The operators also expressed concern about some auditors finding what the operators considered to be "obscure" Rules that did not necessarily have a safety focus – for example, the necessity for the maintenance controller to check the wording in the documentation provided to the operator from the maintenance provider, and to ensure that it was the original documentation rather than a photocopy.

A5.16
We consider that, to retain credibility as a regulator, the CAA needs to ensure that its auditors' interpretation and application of the Rules is consistent enough to ensure that:

  • the operators know what is expected of them and how the Rules will be interpreted;
  • the minimum requirements for compliance with the Rules are distinguished from what the CAA considers to be best practice; and
  • the number and type of findings (instances of non-compliance identified) do not depend on which particular auditor is doing the surveillance.

A5.17
These expectations are in line with the CAA's surveillance policy, which requires both consistency and predictability in the way the Rules are applied.

A5.18
During our audit, we noted evidence from our file reviews, from interviews with a sample of operators, and from the information provided to us by the CAA that, allowing for differences in judgement, there was more than the expected variation in the number of findings issued by auditors from within the same units and also between the units.

A5.19
For example, our analysis of the data provided by the CAA on the number of findings issued from 1 July 2005 to 30 June 2009 showed that:

  • in the Flight Operations Unit in the Airlines Group, the most findings issued by an auditor was 158 and the next highest number in the same period was 40; and
  • in the Rotary Wing and Agricultural Operations Unit in the General Aviation Group, the most findings issued by an auditor was 338 and the lowest number of findings issued in the same period was 128.

A5.20
We also noted examples in our sample of operators where the number of findings for the same operator increased significantly when certain auditors were carrying out the surveillance. For example, an operator was issued with four findings in 2005, one finding in 2006, and 24 findings in 2007. The audit in 2007 was carried out by different auditors and 19 of the findings related to one aircraft.

A5.21
We noted a difference in approach between the auditors of rotary-wing and fixed-wing aircraft. At times, the auditors of fixed-wing aircraft found minor matters of non-compliance but did not issue a finding, but the auditors of rotary-wing aircraft issued findings for similar minor non-compliance. For example, the auditors of fixed-wing aircraft identified, but did not issue a finding for, a failure to have a placard showing the location of a fire extinguisher in the aircraft, and for incorrect completion of training records. We found several examples where auditors of rotary-wing aircraft had issued findings for similar minor matters, such as for a failure to have placards showing the location of the first aid kit, and documents incorrectly filled out or not signed.

A5.22
In our view, the level of inconsistency we found in the number of findings issued by auditors suggests that auditors are taking different approaches to surveillance. That is:

  • some auditors may not be carrying out audits to a sufficient depth to identify non-compliance;
  • some auditors may not be issuing findings when they identify non-compliance; and
  • some auditors may be issuing a series of findings for individual instances of non-compliance when one finding that addresses the "root cause" may suffice.

A5.23
We expected the CAA's unit managers to have investigated the reason for these differences and to take action to moderate the number of findings issued by the auditors they are responsible for. In our view, not doing so raises questions about the quality of the surveillance audits carried out and also, from an operator's point of view, raises questions about the consistency and predictability of the regulatory focus.

Identifying the cause of the non-compliance

We found many examples where the cause of the non-compliance had not been clearly identified, and therefore the action needed to rectify the non-compliance was not obvious.

A5.24
Under the surveillance policy, auditors are required to direct operators to the cause of the finding, as well as to any system deficiency or error that contributed to the non-compliance.

A5.25
We noted many examples of findings within the General Aviation Group where the cause of the non-compliance was not clear and the corrective action required to address the non-compliance was ongoing. For example, for one operator, the auditor found that the pilots were not using the company procedure for determining payload, and some aircraft did not have a copy of the relevant chart for tracking this. The auditor identified the cause as being inadequate control and monitoring. The corrective action required by the auditor was to survey company aircraft and ensure that payload charts were available. The corrective action may have solved the availability of the chart but not necessarily why the procedure for determining payload was not being used nor ensured that it was used in the future.

A5.26
We also noted examples where there were a large number of findings for one single aircraft. A corrective action had been identified for each instance of non-compliance, but the cause for so many findings on a single aircraft was not addressed. For example, an audit resulted in 23 findings, 12 of which related to maintenance issues on one aircraft. The auditors noted that, while most maintenance-related findings were relatively "minor", they showed that the company needed to pay more attention to compliance with the Rules. However, the auditors did not appear to consider possible system deficiencies that could have caused the individual issues on the aircraft.

A5.27
In our view, it is important that corrective actions for findings address the underlying cause of the findings. Otherwise, the reason for the non-compliance may not be addressed and there is a risk the operator will revert to being non-compliant.

Timeliness of following up findings

The CAA is falling well below its target for closing all findings by the due date, but those not closed on time tend to be minor and most are closed within one month of the due date.

A5.28
We saw evidence on the files that findings were monitored to ensure that the appropriate corrective action was carried out. However, the CAA is falling well short of its target for closing all findings by the due date. The CAA annual report for 2007/08 reported that, for both the Airlines Group and the General Aviation Group, "[i]n 2007/08 and 2006/07, the rate of on-time implementation of corrective action was almost static at 57% and 56% respectively".21 The rate reduced to 46% in 2008/09. The target is 100%.

A5.29
Since 2006/07, fewer of the major or critical findings – and more of the minor findings – are not closed by their due date. Most overdue findings are addressed within one month after their due date. However, since 1 July 2005, there have been 33 findings that were not addressed for more than six months after their due date.

A5.30
We were also concerned to note the deterioration in addressing critical findings for 2008/09. Of the critical findings identified during that year, only one was closed by the agreed date, two were closed within a month of the due date, six within two months, and five within three months.22 These delays in addressing critical findings are particularly concerning given that a critical finding is defined by the CAA (in its surveillance policy) to be "an occurrence or deficiency that caused, or on its own had the potential to cause, loss of life or limb".

21: Civil Aviation Authority of New Zealand (2008), Annual Report 2007/08, Wellington, page 23.

22: Civil Aviation Authority of New Zealand (2009), Annual Report 2008/09, Wellington, page 29.

page top