Part 2: Regulatory structure and measuring performance

2.1
In this Part, we discuss:

• the role and structure of the CAA;
• the Civil Aviation Safety System;
• the CAA’s measurement of performance;
• training courses conducted by the CAA to improve operator performance; and
• the responsibility of the participants in the system.

The role and structure of the Civil Aviation Authority

2.2
The CAA’s regulatory role and responsibilities are set out in the Act, section 14 of which states that the objectives of the Minister of Transport (the Minister) are–

(a) to undertake the Minister’s functions in a way that contributes to the aim of achieving an integrated, safe, responsive, and sustainable transport system; and

(b) to ensure that New Zealand’s obligations under international civil aviation agreements are implemented.

2.3
The functions of the Minister include promoting safety in civil aviation and administering New Zealand’s participation in the Chicago Convention (see paragraph 2.11) and any other international aviation convention, agreement, or understanding to which the Government of New Zealand is a party.

2.4
Section 22 of the Act allows the Minister’s functions and powers to be delegated to a 5-member board, known as the Civil Aviation Authority (the Authority). It enables the Authority to appoint a Director of Civil Aviation (the Director) and delegate responsibilities and powers to him or her.

2.5
In addition to the responsibilities and powers delegated by the Authority, the Director also has a range of functions and powers conferred or imposed by the Act, including 2 functions which are relevant to the surveillance process:

• controlling entry into and operation within the civil aviation system, through the granting, monitoring, suspending and revoking of aviation documents; and
• taking such action as may be in the public interest to enforce the provisions of the Act and the CARs made under the Act, including inspections and monitoring.

2.6
In exercising his or her statutory functions and powers, the Director acts independently, and is not responsible to the Minister or the Authority in relation to any particular case.

2.7
The organisation through which the Authority and the Director discharge their functions or powers is also known as the Civil Aviation Authority. To distinguish between the 2 bodies, we refer to the 5-member board as “the Authority”, and the organisation as “the CAA”.

2.8
The CAA is organised into divisional groupings (as shown in Figure 1), of which the following 3 are aligned with sectors of the civil aviation industry:

• the Airline Group is responsible for overseeing the activities of operators of aircraft weighing more than 5670kg⁵, or containing 10 or more seats⁶, along with the associated maintenance, training, design, manufacturing and supply organisations;
• the General Aviation Group covers the operators of aircraft that weigh less than 5670kg and have 9 seats or less, all helicopter, agricultural and balloon operations, and all sport and recreation aviation (both commercial and private); and
• the Personnel Licensing and Aviation Services Group covers:
  • personnel licensing of pilots, maintenance engineers, air traffic controllers, flight engineers, and flight examiners (including medical certification);
  • aviation service – air traffic service providers, airports and aerodromes, training organisations, meteorological services, communication services, aviation security and dangerous goods; and
  • search and rescue co-ordination.

2.9
This structure has the advantage of building expertise in distinct aviation sectors. To maintain effective oversight, the CAA requires expertise in each of these sectors, and adequate training needs to be provided to ensure that this expertise remains current.

Independent internal inquiry of the CAA

2.10
Although not related to the audits of the CAA that we have undertaken, we note that the Director has recently launched an independent internal inquiry into some aspects of the CAA’s performance. The report of this inquiry is to be completed by the end of August 2005, and will be made public.

Figure 1
Structure of the Civil Aviation Authority (excluding the Aviation Security Service)

The role of the regulator in the international context

2.11
The Convention on International Civil Aviation (Chicago 1944) – “the Chicago Convention” – established the International Civil Aviation Organisation (ICAO). It was signed on behalf of the Government of New Zealand in Chicago on 7 December 1944.

2.12
Article 37 of the Chicago Convention states that the ICAO shall adopt international standards and recommended practices and procedures regarding safety, regularity and efficiency of air navigation. Standards and recommended practices are designated as Annexes to the Chicago Convention. At present, there are 18 Annexes.

2.13
Each contracting state (of which New Zealand is one) is responsible for developing and promulgating the national legislation, regulations and standards necessary to comply with the ICAO commitments, and to implement national decisions in discretionary areas. New Zealand legislation provides for this in section 14(b) of the Act (see paragraphs 2.2 and 2.3 above).

The Civil Aviation Safety System

2.14
The Civil Aviation Safety System is based on the “life cycle” approach advocated in the Swedavia-McGregor Report.⁷ This System is based on:

• setting a minimum standard of safety behaviour through CARs and by placing conditions on aviation documents;
• allowing entry into the civil aviation system to those operators who have the capability to meet the required minimum standard for certification and the conditions placed on their aviation documents (certification);
• providing information and advice to operators to assist them to comply with the CARs;
• monitoring operator adherence to the safety standards and their aviation documents, including identifying action that the participants need to take to ensure that they comply with the safety standards (surveillance); and
• where necessary in the interests of safety, imposing conditions on, or suspending or revoking, the aviation document issued to the operator.

The Civil Aviation Rules

2.15
The Act provides for 2 principal tiers of legislation – the Act and the rules made under Part III of the Act. The rules are “secondary” legislation, like regulations.

2.16
The rules that apply to general aviation operators are: Part 119 for compliance and operating requirements, Part 135 for flight operation requirements, and Parts 91 and 137 for agricultural operators. Parts 119, 121, and 125 apply to airline operators, and cover compliance with certification and operating requirements. Part 129 covers certification for foreign airline operators that fly to and from New Zealand.

2.17
The CAA initiated a Rules Review Implementation project in April 2004 to improve the rules development process. The Director engaged an independent reviewer⁸ in 2002 to make recommendations to increase the efficiency and effectiveness of the rule making process.

2.18
The Rules Review Implementation project addresses the recommendations from the review, which include:

• better identification of necessary rules changes;
• filtering out issues that could be dealt with using other processes;
• using risk management processes; and
• improving aviation community participation in the rules development process.

Certification

2.19
The certification process is intended to ensure that prospective operators understand and are capable of complying with the Act and the CARs.

2.20
Prospective operators are required to complete an “exposition”, and submit to the CAA the career histories of nominated staff members performing key roles in the organisation. The CAA completes a “fit and proper person” assessment for these staff members, and checks the exposition to ensure that it complies with the CARs. Airline operators must also have an internal quality assurance system in place that ensures compliance with the procedures specified by Part 119.

2.21
Once the CAA accepts the exposition, staff nominations, and internal quality assurance system, inspectors carry out an entry-level inspection.⁹ If successful, the operator is certificated for an initial period of 6 months. Within that period, inspectors perform a spot check and a “compliance inspection”.¹⁰ At the end of this process, if CAA requirements have been met, the operator is re-issued a certificate for a total period not exceeding 5 years.

2.22
At the end of the 5 years, operators are required to “re-enter” the system by going through the certification process again.

Surveillance

2.23
Section 15 of the Act empowers the Director to carry out such inspections and monitoring as he or she considers necessary in the interests of civil aviation safety.

2.24
CAA surveillance programmes cover all aspects of the civil aviation system (e.g. operators, design and training organisations, aircraft and components, as well as aerodromes and airspace in respect of which CAA approvals are granted). Our audit focused on CAA surveillance of those operators with Air Operator Certificates (Part 119/121,125,135) and Agricultural Aircraft Operator Certificates.

2.25
The CAA considers that the surveillance function is the prime means of ensuring that an acceptable level of aviation safety is maintained, in that it:

• checks that operators are complying with the CARs and the conditions of their aviation documents; and
• aims to identify and correct non-compliant behaviour and unsafe practices before they cause an accident or incident.

2.26
The CAA’s Surveillance Policy sets out the CAA’s surveillance requirements, and is designed to provide the foundation for the day-to-day surveillance operations of the operational groups, viz. the Airline, General Aviation, and Personnel Licensing and Aviation Services Groups.

2.27
The Surveillance Policy refers to both the depth and frequency of audits and states that auditing frequency can only be varied based on the results of the audit, or series of audits over a period and the level of confidence the CAA has in the organisation’s activities. The inspector’s level of confidence is reflected in the Quality Index score.

2.28
The Surveillance Policy includes the range of surveillance tools shown in Figure 2.

2.29
The Surveillance Policy requires that all failures by an operator to comply with the CARs, conditions of their aviation document, or their organisation’s exposition are to be raised with the operator and included in a Finding Notice, which is to be given to the operator at the end of the audit or inspection.

2.30
The Finding Notice lists:

• instances of non-compliance (failure to comply with the CARs) or nonconformance (failure to comply with any additional standards detailed in the organisation’s exposition) identified by the inspector;
• the severity of the finding¹¹, and the cause of each instance of non-compliance and non-conformance;
• the corrective action the operator must take to address the finding; and
• the deadline by which the action has to be taken.

Figure 2
Audit tools in the CAA’s Surveillance Policy

2.31
To “close” the finding in accordance with the Surveillance Policy, the operator must forward evidence that the corrective action has been taken. Until the finding is “closed”, the operator remains non-compliant or non-conforming. CAA inspectors are required to ensure that the corrective action is taken within the stipulated time.

2.32
At the end of the audit or inspection, the inspector is required to prepare an audit report for the Director that includes details of the operator’s business, audit coverage, the findings identified during the audit, and the required corrective action(s). A copy of this report is also to be given to the operator.

Measuring performance

2.33
The CAA uses “safety targets” to measure the safety performance of the aviation industry, areas where it needs to take action, and the consequences of those actions. Safety targets were first set for the 1995-2000 period, and new targets were set for 2000-2005.

2.34
Safety targets are set for each Safety Target Group (STG). There are 9 STGs, distinguished by the type of aircraft, the weight of the aircraft, and the type of operation being carried out.

2.35
Both primary and secondary measures are used for each STG. The primary measure is the number of aircraft accidents per 100,000 flight hours. Secondary measures assess the number of the following factors per 100,000 flight hours:

• aircraft incidents;
• airspace incidents; and
• reportable aircraft defects.

2.36
Industry non-compliance with the CARs is also assessed. This assessment is based on the median level of non-compliance detected during the routine audit and inspections (for the previous 12 months) weighted for severity and divided by CAA routine audit hours (for the 12 months) as a measure of organisational size.

2.37
=An analysis of industry performance against the safety targets shows:

• A decreasing trend in accidents for 8 of the 9 STGs, but increasing trends in the 2721 to 5670kg group (STG 3), which historically has shown the highest level of risk.
• The targets for 7 of the 9 STGs (including the 2 largest STGs, which make up about 96% of the passenger hours of New Zealand’s civil aviation industry) are being achieved.

2.38
However, the report for the Transport Safety Strategies Project¹² questioned the ability of the safety targets to measure the effectiveness of aviation safety interventions. The report highlighted the fact that relatively small numbers of aviation incidents and casualties in New Zealand made it difficult to evaluate interventions or establish statistically researched causal links between death and injury outcomes and safety programmes.

2.39
The CAA’s Annual Reports for 2001-02, 2002-03 and 2003-04 have noted that the current safety targets are not a reliable measure of trends in the safety performance of the civil aviation industry. The reports referred to both the primary and secondary measures, and noted–

The significance of reporting trends cannot be determined using current systems, as there is currently no means of determining if a changing trend represents a change in actual safety performance or a change in reporting patterns by industry.

2.40
The CAA’s concerns relate primarily to the reliability of the data on which the measures are based, especially in relation to the General Aviation sector. The concerns are that:

• Aircraft flying hours are being under-reported by owners, which means that the safety rates can look worse than they really are.
• Accidents, incidents and defects are also being under-reported by either the pilots-in-command or the operators, which means that the safety rates may look better than they really are. Under-reporting also means that the CAA’s Safety Investigation Unit does not have access to all accidents and incidents to see if there is a systemic problem requiring an Airworthiness Directive or a CAR change.

2.41
The CAA’s Statement of Intent 2004/2005 – 2006/2007 also noted concerns about the reliability of incident data, especially as there is no information on what is actually occurring against which to test the accuracy of what the operators and pilots-in-command are reporting.

2.42
The CAA is currently reviewing the measures to establish their reliability and, if necessary, to develop replacements.

2.43
In 2000, the CAA published a booklet (how to... report your accidents and incidents¹³). However, the pilots-in-command, or the operators, are still not reporting all incidents to the CAA as they are required to do. The Aviation Safety Report for 2002 notes that the majority of pilot related airspace incidents continue to be reported to the Authority by the ATS [Air Traffic Service] provider and not by the pilot or aircraft operator.

Training courses conducted by the CAA

2.44
The CAA conducts regular courses for operators. For example, in relation to general aviation, the following courses were held during 2004:

• The Aviation Safety Co-ordinators Course. This 2-day course was held in September-October 2004 at Rotorua, Palmerston North and Queenstown. It covered safety programmes and their structure (including risk management, hazard identification, accident/incident report and analysis), and accident prevention concepts. 60 people attended this course.
• Av-Kiwi – Recent Aircraft Accidents. This course was held at a variety of locations throughout New Zealand from February to September 2004. The course lasted approximately 2 hours and covered the causes and the lessons to be learned from recent air accidents. 429 people attended this course.
• Av-Kiwi Safety Seminars – A to Z Flight Planning. These seminars were held over November-December 2004 at a wide variety of locations throughout New Zealand, and lasted approximately 3 hours. They focused on pre-flight planning and in-flight considerations, and discussed new visual navigation charts, an internet weather service for general aviation pilots and the booklet New Zealand Airspace – Good Aviation Practice, which was revised in November 2004. 749 people attended this course.

Responsibility of participants

2.45
All participants in the civil aviation system are required to take their share of responsibility for safety by fully understanding and complying with their obligations under the Act. Section 12 of the Act requires participants to ensure that all activities and functions are carried out safely and in accordance with the relevant safety standard and practices. This includes ensuring that their employees are appropriately trained and supervised, that the organisation is appropriately resourced, and that its management system will ensure compliance with the CARs and any conditions attached to the aviation document.

5: This figure is normally rounded to 5700kg.

6: This class of aircraft makes up about 96% of passenger hours in the aviation industry.

7: The Swedavia-McGregor Report (1988) was the result of a study ... to consider the need, in the interests of safety, for regulatory controls of civil aviation and their enforcement, to identify the appropriate level of regulation, and to determine the resources needed for a civil aviation safety authority. At the time of the Swedavia-McGregor Report, the Ministry of Transport undertook that function. The findings and recommendations in the report are the foundation of the present regulatory framework in New Zealand.

8: Mary Scholtens QC, December 2002 Review of Participation of Interested Persons in the Development of Ordinary Civil Aviation Rules (also known as “the Scholtens Report”).

9: The main purpose of this inspection is to establish whether management systems detailed in the exposition are in place. This inspection also involves on-site evaluations of support facilities, aircraft, training facilities, maintenance equipment and facilities, and an evaluation of the likely effectiveness of the policies, methods, procedures, and instructions described in the applicant’s exposition.

10: The purpose of a compliance inspection is to:

• confirm that the certificate-holder is able to demonstrate compliance with their documented systems and procedures; and

• establish whether their documented systems and procedures are adequate for the nature and size of the operation.

11: Findings are classified as critical (an occurrence or deficiency that caused, or on its own had the potential to cause, loss of life or limb), major (an occurrence or deficiency that caused, or had the potential to cause, significant problems to the function or effectiveness of the system) or minor (an isolated occurrence or deficiency not indicative of a significant system problem).

12: This project involved the CAA, Land Transport Safety Authority, Maritime Safety Authority and the Ministry of Transport engaging in a collaborative planning process to determine the first steps towards a co-ordinated and timetabled approach to the development of aviation, maritime and rail safety strategies.

13: Safety Education and Publishing Unit, Civil Aviation Authority, Lower Hutt.