Summary and recommendations

Civil Aviation Authority: Certification and surveillance functions.

Introduction

The Civil Aviation Authority of New Zealand (CAA) was established on 10 August 1992 by amendment to the Civil Aviation Act 1990 (the Act). The objective of the CAA is to undertake its safety, security, and other functions in a way that contributes to the aim of achieving an integrated, safe, responsive, and sustainable transport system. The CAA’s functions include promoting civil aviation safety and security in New Zealand and beyond, in accordance with New Zealand’s international obligations.

In New Zealand, the Civil Aviation Safety System is based on:

  • setting a minimum standard for entry into and operation within the civil aviation system;
  • allowing entry to only those operators who meet the entry requirements, and are capable of maintaining compliance with the Civil Aviation Rules (CARs) and the conditions of their aviation documents;
  • providing information and advice to operators to assist them to comply with the CARs;
  • monitoring operator adherence to the safety standards and their own documented procedures, including identifying action that operators need to take to ensure that they comply with the safety standards; and
  • where necessary imposing conditions on, or suspending or revoking, the aviation document issued to the operator.

In the year to 30 June 2004, the CAA spent $15.890 million ($14.367 million in 2002-03) on the output described as Safety Assessment and Certification.

Previous audits

In 1997 we audited the risk management capabilities of the 3 transport safety authorities – the CAA, the Land Transport Safety Authority, and the Maritime Safety Authority – and we were concerned about several aspects of the CAA’s surveillance function; in particular:

  • the adequacy of risk management processes within the CAA to identify the most costeffective safety initiatives; and
  • the extent to which audit resources:
    • targeted high-risk operations and operators; and
    • tested whether operators actually applied their quality management systems.

We conducted a follow-up audit in 2000, to establish how the CAA had addressed our concerns. We found that there had been improvements in:

  • inspector understanding and documentation of the safety audit process;
  • establishing confidence for individual operators (Quality Index) and developing broad strategies to address risk areas; and
  • reporting and follow-up of corrective action to fix instances of non-compliance with the CARs.

However, we still had concerns about and made recommendations in relation to:

  • resources not being appropriately targeted at high-risk operators;
  • how consistently the Quality Index was applied to operators;
  • the extent of inspection undertaken with operators who had limited quality management processes; and
  • staff capability.

This audit

Two significant changes have occurred since our 2000 audit:

  • First, the CAA introduced a new organisational structure in May 2000. This resulted in the safety audit unit (at that time a unit within the Safety Certification Group) being amalgamated into the operational groups.
  • Secondly, operators of aircraft with 2 or more engines who were previously operating under a Transitional Air Operator Certificate, were required to gain Part 119/135 certification by the end of February 2001. Single-engine, fixed-wing, and helicopter operators were required to gain their certification by the end of February 2003. Certification has effectively changed the approach taken by CAA inspectors towards surveillance of these operators.

As a result of these changes, this audit covered both the certification and surveillance functions, to assess whether:

  • the certification (or entry) function ensures that prospective operators understand and are capable of complying with the Act, the CARs and the conditions of their aviation document(s); and
  • an effective surveillance function is operating, to ensure that an acceptable level of civil aviation safety is maintained.

Key findings

Overall, we found that:

  • The certification process used by the Airline Group is generally sound, in that the certifications we reviewed were not subsequently found to be deficient through surveillance. However, General Aviation Group inspectors need to be more rigorous in their assessment of operator capability to comply with the Act and the CARs. Out of the 11 certifications that we reviewed relating to the General Aviation sector, the behaviours demonstrated by 6 of the operators within 12 months of certification suggested that they had been certificated without understanding, or being able to comply with, their own expositions or the CARs.
  • As little action had been taken to address the recommendations in our 1997 and 2000 audits, we still have significant concerns with the surveillance function. The areas we were particularly concerned about were:
    • the effectiveness of the risk analysis and risk assessment processes;
    • ensuring that the risk analysis “feeds through” to the surveillance process; and
    • ensuring that operators, or groups of operators, that are assessed as “high-risk” are appropriately targeted, in relation to both depth and frequency of the surveillance undertaken.
  • CAA inspectors were not ensuring, in accordance with the CAA’s Surveillance Policy, that a Finding Notice is issued to operators for all instances identified where the operators are either not complying with the Act or the CARs or not conforming to their own expositions. We were also concerned about the length of time it took inspectors to ensure that corrective action had been taken by operators to address the matters raised in the Finding Notices.
  • CAA inspectors were not recording all the hours that they work on surveillance in the time recording system. Not recording hours worked means that the CAA is not aware of the actual level of resources required to maintain its surveillance programme. It also affects the accuracy of risk assessment tools that use the hours as part of their calculation (for example, the Non-Compliance Index).
  • Due to financial pressures, resource demands, and the high cost of specialised technical training, only essential training of CAA staff (including inspectors) has been carried out over the last 3 years.
  • Although the internal audits help to promote consistent practice across the CAA, the operational groups do not always “buy in” to the internal auditors’ recommendations.

We discussed our concerns with the CAA during the audit, and recommended that the CAA evaluate its surveillance function with a view to increasing the effectiveness and efficiency of the current resources it puts into the process.

We were pleased to note that the CAA has since begun a review of its surveillance function.

Recommendations

The following recommendations from our audit should be incorporated into the CAA’s review:

Recommendation 1: We recommend that the CAA continue to establish measures to better assess the effectiveness of its safety interventions.

Recommendation 2: We recommend that the CAA improve its analysis of industry information by:

  • including more analysis of the information in the Aviation Safety Report and the Aviation Safety Summary Report to support further action, and to improve the timeliness of these reports; and
  • improving analysis of accident and incident data (for example, by identifying further opportunities – such as the CAA’s joint study of pilot-caused and controller-caused airspace incidents), from which the CAA will draft recommendations for safety intervention mechanisms.

Recommendation 3: We recommend that the CAA further develop the tools it uses to assess the risks associated with individual operators. For example:

  • For the Non-Compliance Index to be more effective, CAA inspectors need to correctly record all instances of non-compliance, as well as the actual audit hours spent with each operator. Operators need to be further encouraged to advise the CAA of instances of non-compliance.
  • For the Quality Index score to be more consistent, it should be supported by the information in the routine audit report, and reasons for significant changes should be explained.
  • For Client Risk Assessments to be more useful to the surveillance process, the CAA needs to re-assess their function. These assessments identify changes to a company’s operation, but not necessarily changes to risk. We recommend that this tool be used to highlight any changes in the company’s operations for inspectors, who would then be responsible for assessing the effect of those changes on the risk of an individual operator.

Recommendation 4: We recommend that the CAA use better indicators of the financial status of operators when assessing operator risk, both at certification and during surveillance.

Recommendation 5: We recommend that the CAA ensure that its inspectors follow the policies and procedures set down for certification.

Recommendation 6: We recommend that the CAA continue with its review of its surveillance function. In undertaking this review and designing a new approach, the CAA should:

  • ensure that the audit process directs resources at the highest-risk operators;
  • direct appropriate activities and interventions at high-risk Safety Target Groups;
  • give priority to the sampling project (a sampling methodology will allow inspectors to make informed decisions on the work necessary to cover the assessed risk);
  • assess where reliance can be placed on operators’ own quality and risk management systems, so that audits can be targeted at higher-risk areas;
  • ensure that the depth and frequency of surveillance is adjusted to reflect operator and operation risk; and
  • develop guidelines to indicate when instances of non-compliance should be referred to the CAA’s Law Enforcement Unit for further action.

Recommendation 7: We recommend that CAA inspectors issue a Finding Notice for all identified instances of non-compliance and non-conformance.

Recommendation 8: We recommend that the CAA establish a system that ensures that operators take quick and effective corrective action when inspectors tell them to do so. This system should include re-assignment of responsibility for that function when an inspector leaves the CAA.

Recommendation 9: We recommend that CAA inspectors ensure that they record all time spent on the surveillance function. Continuing to do otherwise will affect the accuracy of the CAA’s risk analysis tools, and its ability to produce accurate business cases.

Recommendation 10: We recommend that the CAA:

  • ensure sufficient investment in training CAA staff so that they develop and maintain the appropriate skills to carry out their functions;
  • review its staffing levels when the current review of the surveillance function has been completed, to ensure that it has sufficient resources to undertake this function (Both the review of the surveillance function and the review of staffing levels need to take account of the potential pressures or “surges” put on inspectors as a result of unanticipated requests for certifications.);
  • ensure that the operational groups comply with the CAA’s generic policies and procedures (particularly relating to Quality Assurance);
  • promote consistent standards of quality and practices throughout the operational groups by ensuring that they address internal audit Finding Notices; and
  • ensure that the internal audit section is appropriately staffed to enable the CAA’s operations and inspectors to be audited on a more regular basis.
page top