Appendix 3: Ensuring that auditors follow certification policy and procedures

The Civil Aviation Authority's progress with improving certification and surveillance.

A3.1
In 2005, we recommended that the CAA ensure that its auditors follow the policy and procedures set down for certification.

A3.2
In this Appendix, we look at:

the Certification Project;
updates to the certification policy and procedures; and
issuing short-term certificates.

Our overall findings

A3.3
The CAA has updated its certification policy and procedures. These provide a better defined process of what auditors need to do during certification.

A3.4
However, we found differences in approach between the auditors in the Airlines Group and the General Aviation Group. The Airlines Group auditors are using the new procedures set out in the Entry Process Sheets, which provided us with evidence that auditors had followed the required steps set out in the certification policy and procedures. Because the General Aviation Group auditors are not using the Entry Process Sheets, we were not always able to establish that auditors had completed the certification process. The CAA told us that the General Aviation Group started using the Entry Process Sheets after our audit visit.

A3.5
The unit managers in the Airlines Group and the General Aviation Group are not tailoring the Rule checklists to reflect the size and risk of the operator's organisation, as was originally intended. This means that efficiencies intended from the new certification and surveillance processes are not being achieved.

A3.6
Both groups need to better document the results of the work carried out and the weight given to evidence provided for assessing senior persons as "fit and proper" when the evidence requires serious consideration for a judgement to be made.

A3.7
The Act and the certification policy require the Director to be satisfied that all requirements have been met before issuing a certificate. However, we noted several examples where certificates had been issued before resolving issues that had potential safety implications.

A3.8
We found nine examples in our sample where the CAA had issued short-term certificates (for example, for three- and six-month periods) while operators addressed the concerns that auditors had identified during the certification process. This was either to give the operator time to prepare documentation or because the CAA staff did not appear confident in the operator's capacity and resources to continue to comply with the Rules. Although issuing short-term certificates does not necessarily have safety implications, we were concerned that in some instances the CAA's regulatory role was potentially weakened when short-term certificates were issued while operators addressed identified concerns.

A3.9
We consider that our 2005 report recommendation has been partly addressed.

The Certification Project

The scope of the Certification Project was scaled down – from a fully automated process producing a tailored checklist for each operator to "Entry Process Sheets".

A3.10
The CAA began the Certification Project in 2005. It was one of three projects that the CAA intended integrating to produce a fully automated process. This process would produce a tailored checklist identifying all the Rule Parts applicable to each operator. The results of the certification process were then to feed into the initial risk profile and surveillance plan for the operator to create a fully integrated system. It was originally intended that the electronic process would contain adequate checks to ensure that auditors were following the certification policy.

A3.11
However, the scope of the Certification Project was reduced in July 2007, and the tailored electronic checklists were put on hold. The CAA considered that the manual system provided the benefits sought without having to invest in the technology necessary to fully automate the certification process. Instead, in April 2008, the CAA issued detailed procedures and Entry Process Sheets (electronic checklists covering each step in the certification process) to ensure that auditors follow the certification procedures and policy.

Updates to certification policy and procedures

The Certification Project included an update of the certification policy and procedures. The procedures are not always followed, and the adequacy of documentation needs to improve. We found examples where the CAA was issuing certificates even though compliance issues had not been resolved, contravening the Act and the CAA's certification policy.

A3.12
As part of the Certification Project, the CAA has updated its certification policy. The policy was issued in June 2009 and sets out the purpose and scope of certification, the phases of the certification process, and (briefly) what each phase entails. The policy also briefly addresses natural justice considerations, procedural fairness, and what needs to be done when an adverse decision is recommended.

A3.13
The CAA also issued procedures in April 2008 that have to be followed in certification. The procedures are a "how to" guide and are tailored according to the requirements of each type of certificate. The procedures document is electronically linked to the Entry Process Sheets.

A3.14
If completed, the Entry Process Sheets are designed to ensure that auditors follow the certification policy and procedures.

Auditor compliance with the certification policy and procedures

A3.15
To assess whether the auditors were complying with the CAA's certification policy and procedures, we reviewed the most recent recertifications of five operators (two airline operators and three airline maintenance organisations) by the Airlines Group. We also reviewed the most recent certification of 13 fixed-wing and 13 rotary-wing operators by the General Aviation Group.

A3.16
Specifically, we looked at whether:

certification was planned sufficiently;
the operator's exposition (where applicable) was reviewed;
senior persons were assessed as "fit and proper" for the role;
an inspection of the operator's facilities, equipment, documentation, and processes had been done to test the operator's resources and capability to comply with the Rules; and
whether all requirements had been met before the certificate was issued.

A3.17
We also asked a number of operators what they felt about the time that it takes to complete the certification process and what was looked at during the process.

A3.18
Most of the operators we spoke to thought that the timeliness of the certification process was good and that the review was very thorough. Some of the operators we spoke to were concerned about the length of time that it took CAA staff to process amendments to their aviation documents. Most had been told that this was because of a shortage of resources.

Planning for certification

A3.19
The Airlines Group was completing Entry Process Sheets, which provided us with a record that:

planning had been carried out (this included reviewing past audit reports, reported occurrences, and other safety information about the organisation);
the exposition had been reviewed;
senior persons' qualifications and experience had been checked and a "fit and proper" assessment had been done; and
an inspection had been done.

A3.20
In contrast, we were not able to establish what planning had occurred during certification for 20 of the 26 general aviation operators in our sample because the General Aviation Group auditors were not using the Entry Process Sheets or an alternative certification plan.

A3.21
We were told that the General Aviation Group has started using the Entry Process Sheets since our audit visit. We were told that workload pressures discouraged the General Aviation Group from adopting the Entry Process Sheets earlier. We do not consider this to be an appropriate reason for not adopting them. In our view, compliance with the certification policy and procedures should be required and included in staff performance agreements.

Exposition review

A3.22
The Entry Process Sheets provided us with evidence that the expositions had been reviewed in the Airlines Group.

A3.23
In the General Aviation Group, there were checklists on file to show the exposition had been reviewed for 10 operators. However, for nine operators there was no evidence of this on file. For two of these nine operators, the manager had indicated on the certification plan that the exposition did not need to be reviewed. No explanation was provided for this.

A3.24
For the remaining seven operators in our sample, the certification report indicated that the exposition had been reviewed for three rotary-wing operators but there was no documentation supporting this on file. The other four operators had agricultural aircraft operator certificates, which do not require an exposition.

Assessing senior persons for their role

A3.25
Overall, in both the Airlines Group and the General Aviation Group, the information recording the work that had been carried out to assess a senior person as "fit and proper" was inadequate. For example:

inadequate notes from senior person interviews about what was discussed and the interviewer's opinion of how well the applicant had performed during the interview;
inadequate evidence of what was looked at during the inspection; and
inadequate information recording whether the auditor considered that the applicant had the resources and capability to comply with the Rules.

A3.26
We were not able to establish whether the "fit and proper person" checks of senior persons had been completed for three operators.

A3.27
Documenting the reasons for decisions is particularly important when the files contain adverse information (for example, a Ministry of Justice report that recorded a history of serious traffic offences), or recent surveillance reports that note serious concerns with the operator's resources and capability to comply with the Rules. These situations require the exercise of judgement and make the assessment less straightforward.

A3.28
The Act requires the Director to be satisfied that a senior person:

has the relevant qualifications and experience;
is a "fit and proper person" to hold the aviation document and have its privileges; and
meets all other relevant prescribed requirements.

A3.29
When assessing whether a senior person is "fit and proper", the Director is required to consider a number of factors, including the person's knowledge of the applicable regulatory requirements and any convictions for any transport safety offences. In making an assessment, the circumstances of each particular applicant, including the degree and nature of the person's involvement in the civil aviation system (their "privileges", level of responsibility, and degree of control), needs to be considered and the appropriate weight given to the available information (its credibility, relevance, and relative importance).

A3.30
Figure 6 outlines one example in the General Aviation Group where we had concerns about the lack of documentation on file to record how the assessment and decision was made to approve a senior person as "fit and proper" to be the chief executive of the organisation.

Figure 6
Example of a lack of documentation to support a "fit and proper person" decision

An operator was certified to carry fare-paying passengers and freight for the first time in January 2009. The person nominated for the chief executive's position had been convicted of driving with excess blood alcohol in August 2008, and at the time of the "fit and proper" person assessment was still disqualified from driving. He also had a previous three-month driver licence suspension in June 2006 for accumulated traffic offences. The CAA auditor was concerned enough about the conviction history to elevate the final decision assessment to his unit manager. The unit manager and the general manager of the General Aviation Group interviewed the nominated person to determine his suitability for the role and to express their concerns about his behaviour. The only record of this interview that we could find was a short summary in the Management Information System about the meeting with the operator. CAA staff considered that the chief executive had accepted the message that his behaviour would not be tolerated, and "the impression was gained that he would make a serious effort to not repeat any of the things that led to the bad reports". In our view, there was inadequate information on file about the decision to approve the nominated person to the chief executive role. This included a lack of documentation about the weighting given to the conviction history and the effect that this had on the person's ability to hold the certificate and their role as the chief executive, especially given there was a pattern of offences. There was no explanation on file about what the chief executive had agreed to do to give the CAA assurance that his behaviour would improve.

A3.31
The case study at the end of Appendix 4 (see Figure 10) sets out another example from the Airlines Group where we had concerns about the lack of documentation on file supporting the assessment and decision for the chief executive role.

A3.32
In our view, a lack of documentation about how these decisions were made could be problematic if the validity of the decision was later challenged. CAA staff told us that, where an adverse decision is made (that is, an applicant is assessed as not "fit and proper"), the assessment and reasons why the decision is made are documented in detail. We were provided with one example where this had occurred. We consider that this process of fully documenting the assessment and reasons for the decision should be extended to include situations where the evidence requires serious consideration for a judgement to be made.

A3.33
We also noted, where a senior person who had previously been assessed as "fit and proper" was being re-assessed, a section 15A¹⁹ investigation was carried out. This investigation provided detailed analysis and support for the decisions made.

Carrying out inspections

A3.34
We noted that there was no record on file that an inspection had been completed for five operators in our general aviation sample (two fixed-wing and three rotary-wing operators). In one instance, a desk-top review rather than an inspection had been carried out and there was no record explaining why this had been done (see Figure 7). In our view, the surveillance history with the operator suggested that an inspection was warranted.

Figure 7 Example where recertification of a general aviation operator was carried out as a desk-top review

After a routine audit of a general aviation operator carried out in November 2007 (which identified 10 findings, including an outdated exposition), the frequency of surveillance was changed to six monthly. When the operating certificate expired in January 2008, a new certificate was issued after a desk-top review. We would have expected an inspection, given the number of findings in November 2007 and the CAA manager's decision to increase the frequency of surveillance. At the next routine audit (carried out in October 2008, not after six months as the CAA manager had indicated), 17 findings were identified. Some of these findings were for issues that had not been resolved since the previous audit, and were, in our view, issues that should have been corrected before the certificate was re-issued (for example, the internal quality assurance system needed improving).

A3.35
In our sample, we found five examples (three airline operators and two general aviation operators) where we had concerns about the quality of the inspection that had been carried out. Issues were later identified that we consider should have been picked up during the certification inspection. Figure 8 outlines two of the five examples we found (one each in the Airlines Group and the General Aviation Group) where we had concerns about the quality of the work done for certification.

Figure 8
Examples where we had concerns about how thorough the Civil Aviation Authority's certification work had been

Example 1 An airline maintenance organisation had been recertified for five years from January 2007. However, the following serious concerns were identified with the organisation during surveillance in March and July 2007: a lack of resources, with too few staff to support the flight operations side and maintenance side of the organisation; and aircraft defects were incorrectly identified as "observations" and not recorded in the aircraft records or fixed by the maintenance organisation – some were for serious issues that needed to be fixed before the aircraft flew again. We would have expected these concerns, which were ongoing, to have been identified and corrected as part of the recertification process. Example 2 A general aviation operator was certified in June 2005. The first surveillance audit in August 2006 identified that the operations manual did not fully comply with the Rules. In 2007, the auditors found that the exposition was providing little guidance to staff in their day-to-day activities. There have been ongoing issues since then with getting the exposition up to date. In September 2008, a finding was issued to track progress. In our view, if certification had been done properly, issues such as the exposition not complying with the Rules, and the fact that the exposition provided little guidance to the staff in their day-to-day activities, would have been resolved before the operator was certified.

A3.36
The case study at the end of Appendix 4 (see Figure 10) describes another example in the Airlines Group where we had concerns about the quality of the work carried out as part of recertification.

Meeting requirements to issue a certificate

A3.37
The Director has delegated his authority to issue operator certificates to the general managers and unit managers. We expected that a manager using the delegated authority would ensure that all issues identified during the certification process were resolved before issuing a certificate.

A3.38
For three airline operators in our sample, the CAA manager had issued a certificate before resolving the issues:

In one example, the certificate was issued before the exposition was completed (CAA staff had considered the safety implications of this) and there had been continuing issues with the operator's compliance with the Rules since the certificate was issued. The CAA was concerned enough at one stage to halt the operator's international operations until the issues were addressed.
In another example, the operator was certified for a six-month period without the "fit and proper person" checks being completed for the chief executive and the security manager.

A3.39
In the General Aviation Group, the two units had various issues unresolved before issuing certificates. Overall, the Fixed Wing Unit was better at ensuring that there were no unresolved issues before issuing the certificates:

For five rotary-wing operators, it was not clear from the documentation on file whether matters that had been identified during the certification process were addressed before issuing the certificate. Also, for one of these five rotary-wing operators, the "fit and proper person" checks were not completed until after the certificate had been issued.
We noted just one fixed-wing operator who had a minor finding still unresolved when the certificate was issued, and one operator for whom the "fit and proper person" checks were not completed until after the certificate was issued.

Issuing short-term certificates

The CAA was issuing short-term certificates as a risk management measure, and we are concerned that in some instances this may be done when operators have not reached the required standard.

A3.40
The Rules provide for a certificate to be issued or renewed for up to five years. The Director may issue a certificate for a shorter period.

A3.41
The CAA's certification policy requires that successful new entrants to the civil aviation system are issued with a six-month certificate. This allows the CAA to carry out a compliance inspection to establish that the applicant has the resources and the capability to comply with the Rules before a longer-term certificate is issued (for up to four-and-a-half years).

A3.42
In our view, although issuing short-term certificates is good practice for new entrants, we do not expect to see this happen for operators and organisations that have been in the civil aviation system for more than five years unless they are undergoing a significant change to their operation or organisation (for example, a new type of operator certificate that changes the scope of their current operation).

A3.43
However, we found nine examples in our sample where the CAA had issued short-term certificates (for example, for three- and six-month periods) so that operators could address concerns that auditors had identified during the certification process. This was either to enable the operator time to prepare documentation or because the CAA staff did not appear confident in the operator's capacity and resources to continue ongoing compliance with the Rules to issue the certificate for a longer period.

A3.44
The CAA general managers told us that the term of the certificate is used as a risk management tool because it gives the CAA a chance to do a further thorough review of the organisation and it also puts the operator on notice that they have to lift their standards.

A3.45
However, in a couple of instances, the operator had an action plan in place to improve their performance before the next recertification. In these cases, we were concerned that a short-term certificate had been issued when the operator did not appear to have met the required standards.

A3.46
We were also concerned about the amount of ongoing assistance the CAA had given some operators to help them achieve compliance.

A3.47
Figure 9 sets out two examples we found in the General Aviation Group where the CAA had issued short-term certificates so that the operators could improve their compliance.

A3.48
The case study at the end of Appendix 4 (see Figure 10) sets out another example, from the Airlines Group, where we had concerns about the issuing of a short-term certificate and the amount of assistance the CAA had given to help the operator achieve compliance.

A3.49
Although issuing short-term certificates does not necessarily have safety implications, we were concerned that in some instances the CAA's regulatory role was potentially weakened because it issued short-term certificates while operators made necessary improvements.

Figure 9
Examples where short-term certificates had been issued to general aviation operators

Example 1 The operator's certificate was due for renewal on 31 December 2007. A certification inspection was carried out in October 2007. In December 2007, the manager of the CAA's Safety Investigations Unit sent a memo to the Rotary Wing and Agricultural Operations Unit manager setting out his concerns about the safety record of the operator. The operator's risk profile had increased from 16% to 31% in three months, and since 2002 there had been four fatal accidents resulting in five employee deaths. The general manager of the General Aviation Group met with the operator's chief executive in December 2007 to discuss concerns about the company's organisation, geographical location, and operational supervision. A recent incident where an undercarriage wheel had fallen off the aircraft in flight was also discussed because it raised airworthiness and maintenance issues. As a result of the interview, the CAA issued a certificate for a six-month period to June 2008 to give the chief executive time to address the issues. The chief executive was also required to provide an "action plan" detailing how four key issues that had been identified would be addressed. In June 2008, the CAA issued another short-term certificate (to expire in August 2008) because the chief executive had addressed only three of the four major issues. Operational oversight of the operator's line stations still needed to be improved and a chief pilot appointed. In August 2008, the CAA issued a five-year certificate after it had provisionally accepted the appointment of an acting chief pilot. (The acceptance of the chief pilot by the Director was still subject to the CAA interviewing the chief pilot.) Example 2 Concerns were noted about the completeness and accuracy of an operator's records (particularly for the agricultural side of the business) during a routine audit in June 2008. When the agricultural aircraft operator certificate expired in August 2008, the certificate was renewed for only one year. The next routine audit of this operator, in August 2009, identified 13 instances of non-compliance with the Rules, mainly in maintenance control. The CAA manager noted that there was no evidence that the concerns noted in the August 2008 audit had been addressed. The operator was put on a six-monthly audit regime with spot checks of both the air operator certificate and the agricultural aircraft operator certificate. However, an agricultural aircraft operator certificate was issued for four years. The audit report did not explain why a four-year certificate was issued. Twelve months earlier, lesser concerns had resulted in a one-year certificate. The risk profile for the agricultural certificate was relatively high at 26%.

A3.50
In our view, this process is contrary to the strong regulatory message that the Director is trying to send to both the industry and staff because it places the CAA in a position of educator and advisor rather than regulator. It also increases the risk that the CAA will certify operators who do not have the capability and resources to comply with the Rules, and increases the amount of future surveillance that the CAA will have to carry out to address the risk of these operators.

A3.51
We have discussed this with the CAA. Although the CAA agrees with us in principle, it notes that many of the Rules require an operator to have a procedure or programme. If the operator is being recertified, the requirement for compliance with the Rules was met at the time that the certificate was originally issued and the Director then has to assess the adequacy of the procedure or programme. The CAA considers that there is a very high probability that the courts would not uphold a refusal to renew a certificate unless there was a significant safety risk and, because it would be seen as taking an operator's livelihood away, it would require a higher burden of proof. The CAA believes that the courts would support the issuing of short-term certificates as a means of dealing with these matters. The CAA has not tested the matter in court.

A3.52
Although we agree with the CAA's views on this for minor non-compliance, we are concerned that the CAA has not established what constitutes a "significant safety risk". This deficiency means that the treatment of serious non-compliance by operators in our sample was not consistent. For example, the certificate of an agricultural aircraft operator was suspended because of inadequate procedures, while a larger passenger-carrying operator's certificate was issued for shorter periods for similar non-compliance. We consider that, when the "livelihood test" is applied, the "consequences of the risk" should also be given at least equal weight.

A3.53
We also note that the CAA's certification policy states that, for re-entry or recertification, "it is important for the CAA to confirm that there is 100 percent compliance with all applicable rule requirements". It goes on to say that "it is still necessary to ensure that all certification requirements are satisfied".

19: In the interests of civil aviation safety and security, the Director may, under section 15A of the Act, require operators to undergo an investigation.

page top