Part 1: Background to our audit

Civil Aviation Authority: Certification and surveillance functions.

1.1

In this Part, we discuss:

Our 1997 audit and report1

1.2
In 1997 we audited the risk management capabilities of the 3 transport safety authorities – the CAA, the Land Transport Safety Authority, and the Maritime Safety Authority. We made findings and recommendations for all 3 safety authorities, but the more serious of these concerned the CAA.

1.3
In particular, we concluded that the CAA did not systematically assess the potential consequences of aviation accidents and incidents in order to identify accurately:

  • the areas of highest risk; and
  • the aviation safety initiatives that offered the best safety benefits to New Zealand.2

1.4
We also found that the CAA’s surveillance was focused on checking an operator’s documentation rather than seeking to confirm that the procedures specified in the operator’s manuals were being carried out in practice.

1.5
The standard of the routine audits also did not meet our expectations. Individual inspectors applied their own standards, and as a result could reach different conclusions. This was because there were no routine audit plans, senior staff did not review routine audit findings, and there was little follow-up of previous routine audit recommendations.

1.6
We therefore recommended that the CAA should:

  • develop a cost-benefit analysis system which would accurately identify aviation safety initiatives that offered the best safety benefits, and therefore adequately manage risk;
  • continue a broad-based approach to routine audits with a focus on more rigorous audits of high-risk operators or types of aircraft;
  • conduct more inspections to confirm that operators applied their quality management systems in practice; and
  • develop the proposed confidence rating system to allow audit resources to be targeted at high-risk operations and operators.

Our 2000 follow-up audit and report3

1.7
A follow-up audit in 2000 focused on how the CAA had addressed the recommendations made in our 1997 report.

1.8
Our audit found that:

  • the CAA had established priority areas and developed broad strategies to address risk areas, and had developed a system for establishing a level of confidence for individual operators (the Quality Index);
  • the quality of routine audit reports and documentation had generally improved, and the routine audit process appeared to be well understood by CAA inspectors; and
  • the CAA had a well-established system for requiring operators to take corrective action to fix instances of non-compliance with the CARs, and for following up operators’ responses.

1.9
However, we still had concerns about the surveillance process. In particular:

  • Routine safety audits were being undertaken annually, irrespective of each operator’s risk profile.
  • Some operators were dissatisfied with these routine audits. Large operators felt that the CAA could do more to add value to its quality assurance processes, and smaller operators felt that routine audits of their operations should focus less on checking documentation and more on physical inspections.
  • There were inconsistencies in the approach to routine audits between individual inspectors, some of which the CAA had attempted to address. Management review of routine audit plans was limited.

1.10
We therefore recommended in our 2000 report that the CAA should:

  • ensure that its new organisational structure did not reduce the effectiveness of its surveillance resources;
  • consider what resources were required to gain assurance on the safety of low-risk operators;
  • more appropriately target surveillance resources at high-risk operators;
  • improve the consistency of the use of the Quality Index – for example, by Group Managers selectively reviewing audit plans and how inspectors arrived at their scores;
  • increase the proportion of routine audit time spent on physical inspections for operators with limited quality management processes; and
  • seek to strengthen staff capability by seconding skilled staff from large operators (recognising relevant constraints).

What this audit covered

1.11
Rather than carry out a follow-up review, we re-audited the CAA’s surveillance function because:

  • The CAA had introduced a new organisational structure in May 2000, which resulted in the safety audit unit (at that time a unit within the Safety Certification Group) being amalgamated into the operational groups.
  • Since our 2000 audit, operators of aircraft with 2 or more engines who were previously operating under a Transitional Air Operator Certificate were required to gain Part 119/135 certification by the end of February 2001. Single-engine, fixed-wing, and helicopter operators were required to gain their certification by the end of February 2003. Certification has effectively changed the approach taken by CAA inspectors towards these operators.

1.12
To establish whether our 1997 and 2000 recommendations had been addressed, we focused on the surveillance function. In particular, we looked at the extent to which the CAA’s risk assessments influence the depth and frequency of surveillance.

1.13
In addition, we considered whether the certification process was sufficiently robust to keep out potentially unsafe operators.

1.14
We did not cover the following areas, as we considered them outside the scope of the certification and surveillance functions:

  • pilot licensing and aircraft certification;
  • sport and recreation activities;
  • the CAA’s “fit and proper person” assessment criteria; and
  • the CAA’s role in monitoring the occupational safety and health of aircrews during domestic operations4.

How we conducted this audit

1.15
We carried out this audit by interviewing:

  • senior CAA managers, including General Managers and the Director of Civil Aviation;
  • a sample of CAA staff, including team managers, inspectors, internal auditors, and staff from the Safety Research, Education and Publishing Group;
  • the then Chair and the Deputy Chair of the CAA;
  • staff from the Ministry of Transport and the Transport Accident Investigation Commission; and
  • representatives from the Aviation Industry Association.

1.16
We also:

  • surveyed, by phone, 27 operators of varying sizes and types;
  • observed CAA routine audits in practice and spoke with the operators being audited; and
  • reviewed CAA documentation – including audit methodology, the draft strategic plan, the Aviation Safety Report, and certification and routine audit files.

1: Report of the Controller and Auditor-General: Fourth Report for 1997, parliamentary paper B.29[97d], pages 77-121.

2: Ibid, page 86, paragraph 4.031.

3: Civil Aviation Authority Safety Audits – Follow-up Audit, ISBN 0-477-02874-8.

4: The CAA has been assigned this responsibility through the Health and Safety in Employment Amendment Act 2002.

page top