Part 4: Why the Civil Aviation Authority has been slow to improve certification and surveillance

4.1
The CAA accepted our 2005 recommendations and has been working on improvements to its certification and surveillance policies, processes, and tools. Since our audit in 2005, the CAA has assured us that it was addressing our recommendations and it has also reported publicly that it was making good progress in addressing our recommendations.

4.2
After our 2005 audit, the CAA presented us with a model of how it would integrate its certification, surveillance, and risk assessment and intervention projects using an electronic system and supporting tools. These projects were designed to improve the effectiveness and efficiency of the certification and surveillance processes.

4.3
We deferred the timing of our most recent audit to ensure that the CAA had enough time to fully implement those changes and complete enough audits using the new processes and tools so that we could test how well the integrated system was working in practice.

4.4
In our view, the progress that the CAA has made in addressing our 2005 recommendations is inadequate. There is still considerable scope for improvement in how effectively and efficiently the CAA conducts certification and surveillance. We provide the evidence to support our view in Appendices 1 to 6. Part 3 provided a summary of our evidence.

4.5
We consider that the CAA continues to have weaknesses that we have previously identified, and that these weaknesses affect the rigour, consistency, and transparency of regulation.

4.6
In our view, the CAA has failed to understand and effectively address the underlying causes of the weaknesses in its certification and surveillance work. We consider that the following factors have contributed to its inadequate response:

• Governance of, and accountability for, the CAA's certification and surveillance functions are ineffective.
• The strength of the CAA's regulatory focus is unclear, and there is insufficient guidance to ensure that regulatory responses are appropriately and consistently applied.
• The CAA's management practices are not focused on improving staff performance, and it has not been receptive to change.
• The CAA's management oversight of implementing and using the new certification and surveillance processes is inadequate.
• The CAA has not given enough attention to improving its organisational proficiency in auditing.

4.7
In this Part, we discuss the reasons for our observations and make recommendations to improve the CAA's governance and management, and the training of its auditors. We also include the CAA's views on how a lack of capacity and capability has constrained change.

Governance of, and accountability for, the certification and surveillance functions are ineffective

There are deficiencies in how the Board holds the Director to account for the CAA's outcomes, the measures used to assess the CAA's effectiveness in carrying out certification and surveillance work, and the efficacy of the CAA's internal audit work. Also, the Ministry of Transport's monitoring of the CAA has not been active enough.

The Director's accountability to the Board for the CAA's outcomes

4.8
The Director, as chief executive officer of the CAA, is accountable to the Board for the CAA's performance in achieving the Board's strategic priorities and operating intentions. These are set out in the CAA's statement of intent.

4.9
However, in carrying out the functions and exercising the powers given to him under the Act, the Director acts independently and is not responsible to the Minister or to the Board in relation to any particular case. For certification and surveillance, this includes controlling entry into, and operation within, the civil aviation system, including inspecting and monitoring the behaviour of the participants in the civil aviation system. The combined effect of these individual actions by the Director establishes the level of the CAA's regulatory focus.

4.10
The Board is responsible, through the CAA, for providing the Director with resources to carry out his functions. These resources include adequate certification and surveillance processes, appropriately qualified staff, and a robust information system.

4.11
We reviewed the Board minutes since our 2005 audit and found that the CAA had regularly provided the Board with positive updates on its progress with the three projects that were to make up the integrated certification and surveillance system. However, there was, in fact, inadequate progress with these projects, especially the certification and surveillance projects (see Part 3).

4.12
The Board consequently received inaccurate advice. However, because the measures the CAA provides to assess the effectiveness of its certification and surveillance work are not robust enough to identify the inadequate progress (see paragraphs 4.18-4.20), the Board had no reason to question this advice.

4.13
We consider that, if the Board had been using appropriate measures to assess the CAA's performance in carrying out certification and surveillance, the Board would have been able to see that progress was inadequate.

4.14
We expected that the CAA would be able to demonstrate the effect that its certification and surveillance work was having on its outcomes. The CAA's outcomes are:

• a reduced number and cost of civil aviation accidents and incidents;
• a reduced number of civil aviation fatalities and injuries; and
• a high level of compliance with legislation and the Rules by the civil aviation participants.

4.15
Although the CAA is beginning to identify initiatives aimed at particular types of operators in a Safety Target Group, there is little in the groups' annual plans to identify how the group and unit outputs will address the outcomes. To address these, the CAA will need to understand more fully the effect that its surveillance and certification work is having on its outcomes.

4.16
Implementing the new risk assessment tool should be useful in tracking trends in individual operator risk, and this should be able to be done by Safety Target Groups. In addition, the CAA is currently trying to establish how it can measure the effect that the various types of intervention (for example, routine audits and spot checks) have on improving operator compliance with the Rules.

4.17
If the Board is to hold the CAA accountable, the CAA needs appropriate measures to show the effect of its certification and surveillance work on its outcomes.

4.18
For the Airlines Group and the General Aviation Group, the effectiveness of their certification and surveillance work is measured by the number of certification requests that they have met and by the number of hours spent on surveillance and spot checks. The quality of certification and surveillance is measured by the results found by the CAA's internal audit unit.

4.19
In our view, these are not good measures of the effectiveness of certification and surveillance. The measures do not give assurance that surveillance resources have been appropriately targeted at the higher-risk operators, nor do they provide any information about the effect of the surveillance and certification on operator behaviour.

4.20
The CAA is currently looking at addressing this issue. It has looked to see what other countries' aviation safety regulators are doing. The CAA considers that no other regulator has developed robust measures of the effectiveness of their interventions, and that what the CAA is doing is "world-leading". The CAA is considering measuring the effect that surveillance has in reducing operator risk profiles and other measures. We support this approach.

Recommendation 1
We recommend that the Civil Aviation Authority put in place measures to better assess the effectiveness of its certification and surveillance functions and use these measures to report and account to the Board for its performance in achieving its outcomes.

Internal audit

4.21
Each year, the Board's Audit and Risk Management Committee (the Committee) approved the Board's internal audit programme, and the results of the internal audits were reported to the Committee. Although the annual internal audit programme covered all the major groups within the CAA, it was focused on checking procedural compliance.

4.22
In our view, the internal audit focus needs to be extended to provide the Board with assurance about the CAA's assessment of how well the groups are contributing to the CAA's strategic priorities for certification and surveillance, and the extent to which certification and surveillance are helping to achieve the overall goals and objectives of the CAA.

Recommendation 2
We recommend that the Board extend its internal audit of the Civil Aviation Authority to include assurance over the executive management team's assessment of how well the Airlines Group's and General Aviation Group's certification and surveillance are contributing to its strategic priorities and achieving its overall goals and objectives.

4.23
In addition, we also noted that the CAA's internal audit unit had reviewed the Airlines Group's compliance with the new surveillance process in February 2008. The review found that, although the Airline Maintenance Unit was generally complying with the new policy, the Flight Operations Unit was not complying. Examples of non-compliance included not planning and conducting audits to reflect the operator's risk.

4.24
The internal audit unit also reviewed the General Aviation Group's compliance with the new surveillance process in April/May 2008. Overall, this review found that, although the rotary-wing section of the Rotary Wing and Agricultural Operations Unit was complying with the new surveillance process, the Fixed Wing Unit was "not using the surveillance process as prescribed".

4.25
The findings about both groups do not appear to have been rechecked by the internal audit unit during the next audit.

Recommendation 3
We recommend that the Board's Audit and Risk Management Committee take a more active role to ensure that the Airlines Group and the General Aviation Group actually address the internal audit findings.

Monitoring by the Ministry of Transport

4.26
The CAA provides quarterly reports to the Minister, copied to the Ministry of Transport, on how the CAA is performing against its performance measures. The Ministry meets regularly with the CAA management team and these meetings include discussion on performance issues. Key performance issues are also discussed as part of the Ministry's six-monthly meetings with the Chairman of the Board.

4.27
Ministry of Transport officials told us that it is the Board's responsibility to monitor the CAA. They see that the main accountability relationship is between the Minister and the Board, rather than between the Minister and the CAA's chief executive officer.

4.28
In our view, although the Board is primarily accountable to the Minister for the CAA's performance, the Ministry of Transport also has an important role in advising and supporting the Minister by monitoring the Crown entities for which the Minister is responsible. We would expect this monitoring to include scanning for emerging issues or risks that might require a response, advising the Minister of these as early as possible, and providing support for the relationship between the Minister and the Board. Our expectations of Crown entity monitoring are set out in our 2009 report, How government departments monitor Crown entities.¹⁵

4.29
In our view, monitoring progress on how well the CAA is addressing our recommendations on its certification and surveillance functions fits with the Ministry of Transport's general role of supporting the Minister.

4.30
We consider that the Ministry of Transport should have been more comprehensive and timely in its monitoring of the sufficiency, appropriateness, and timeliness of the action taken by the CAA in addressing the recommendations in our 2005 report. The Ministry set up an "action sheet" to monitor the CAA's progress in implementing our recommendations from 2005. This action sheet was posted on the websites of the CAA and the Ministry. The most recent status report for the action sheet was dated December 2007.

Recommendation 4
We recommend that the Ministry of Transport, on behalf of the Minister of Transport, more actively monitor the Civil Aviation Authority to provide assurance to the Minister of Transport that the Civil Aviation Authority is addressing our recommendations and performing certification and surveillance effectively and efficiently.

Strength of the regulatory focus is unclear and guidance is insufficient to ensure appropriate and consistent responses

The strength of the CAA's regulatory focus is not clear and, because of a lack of guidance and training, auditors do not respond consistently to safety concerns.

The strength of the CAA's regulatory focus

4.31
An external review of the CAA in 2006 noted that, to be an effective regulator, the CAA needed to focus on the public as its main client and adopt the position of a regulator rather than that of an industry advisor.

4.32
Shortly after taking office in July 2007, the new Director told both the aviation industry and CAA staff that the CAA would be adopting a stronger, compliance-driven regulatory role. The Director said that this would include a more rigorous certification process and strengthening of the surveillance process by more thorough auditing. However, this stronger regulatory focus has not always been reflected in the CAA's day-to-day certification and surveillance work.

4.33
In our view, the assistance provided to some operators in our sample to achieve certification placed the CAA more in the role of advisor than regulator. The assistance included issuing aviation documents for a short time to enable operators to either complete the certification process or lift the standard of their operations (see Appendix 3, paragraphs A3.40-A3.53).

4.34
Issuing short-term certificates can be an appropriate regulatory response and does not necessarily have safety implications, but in issuing short-term certificates the CAA needs to show that it has considered the increased risks in doing so. In the sample that we reviewed, we saw no evidence that the CAA had considered the risks involved. In our view, the CAA's regulatory role was potentially weakened because it issued short-term certificates while operators made necessary improvements.

4.35
For example, we noted that one operator was told before recertification that they would have to carry out a major overhaul of their company to meet the requirements for recertification (see Figure 10 at the end of Appendix 4). The CAA had issued a "milestones" document to the operator at the same time that it issued the certificate. The document listed the actions that the airline had to take during the term of the certificate and set out the CAA's planned oversight activities. In this instance, a higher-risk operator was allowed to operate while rectification actions were taken.

4.36
There was not always a clear link between the CAA's surveillance findings, the effect that the findings had on safety, and the level of regulatory response that the auditor should make. We also noted instances in our sample of operators where there were significant increases in findings from one surveillance audit to the next, although there had been no change to the size or scope of operations, to the systems used in performing the operations, or to personnel.

4.37
CAA staff told us that the increase was because of the new, stronger, regulatory focus. However, we found that a significant number of the findings were not directly related to safety. For example, a finding was issued because:

• the registration marks on the tail fin of an aircraft were too small (220mm rather than 250mm); and
• the operator's maintenance review statement did not use the specific wording required by the Rule.

4.38
The Director and the Chairman of the Board were not able to tell us how strong the regulatory focus was at the time of our audit and how either of them would know if it was strong enough. In our view, there are indicators that could be used – for example, monitoring changes in the behaviour of aviation sector members, monitoring the types and frequency of complaints from aviation sector members, and monitoring the results of cases that have been taken to court.

4.39
The two general managers we spoke to at the CAA told us that they would recommend revoking or suspending an operator's certificate to the Director if they felt that safety had been compromised. However, they were unclear about the level of non-compliance where this would happen. The Director told us that he would suspend or revoke an aviation document if he received such a recommendation, supported with adequate evidence, from the general managers. The CAA has suspended three operator certificates during the last five years.

4.40
In our view, the CAA does not have a consistent, organisational-wide, regulatory focus. Instead, the strength of the regulatory focus differed between the groups, the units, and (at times) the individual auditors.

Recommendation 5
We recommend that the Civil Aviation Authority prepare and implement better measures of the strength and effectiveness of its regulation of the civil aviation sector, including measures to assess the relative effectiveness of advisory and enforcement actions.

Guidance provided to CAA staff on applying regulatory responses

4.41
CAA staff have been provided with high-level guidance in the form of regulatory principles (in the surveillance policy), to use when making decisions on applying regulatory tools. Training is also given on applying the principles of natural justice and assessing senior persons as "fit and proper".

4.42
In our view, the guidance and the training are not enough to ensure that auditors consistently apply regulatory responses. The surveillance policy states that the:

… guidelines indicate, in a general way [our emphasis], the basis upon which decisions concerning the exercise of the Director's powers with respect to the choice and application of regulatory tools will be made.¹⁶

4.43
We found examples where the regulatory response to address safety concerns differed in similar circumstances. There was not enough documentation for us to understand why the regulatory responses differed between these examples. This is contrary to two of the CAA's principles:

• Consistency – the Rules and standards must be applied consistently and fairly across the sector and regulation should be predictable to give stability and certainty to those being regulated.
• Accountability – regulators must be able to justify the decision and be subject to public scrutiny, and regulators should clearly explain how and why decisions have been reached.

4.44
Figure 5 gives details about four operators in our file review where we found that the CAA had responded differently to safety concerns. For these examples, we are not questioning the individual decisions but we are concerned about inconsistent responses for what appear to be similar circumstances.

Figure 5
Examples of the Civil Aviation Authority's variability in responding to safety concerns

Operator 1 In October 2008, the CAA found that an agricultural operator was in serious non-compliance with the Rules. For example, the chief executive officer had been operating an aircraft for six months without a valid licence to carry out aerial distribution of agricultural chemicals. The CAA suspended the operator's certificate while the operator addressed the non-compliance. The suspension was lifted on the closure of all findings. The matter was not referred to the CAA's Law Enforcement Unit.

Operator 2 Operator 2's certificate was due for renewal in December 2007. The CAA noted its concern about the operator's safety record, which included four fatal accidents since 2002. The CAA issued the operator with several short-term certificates while the operator addressed the findings and safety concerns, and also required the operator to have an action plan for improving compliance.

Operator 3 In August 2008, the CAA found that Operator 3 had operated an aircraft after its Annual Review of Airworthiness (ARA) had expired. The aircraft had been flown on seven occasions before the ARA was renewed. The CAA also found that the pilot had operated for more than two years without an agricultural competency check. The CAA referred the operator to its Law Enforcement Unit. The pilot was prosecuted and fined for being overdue with the competency checks. The operator's certificate expired in October 2008, and the CAA issued a short-term certificate for two months to allow the operator time to resolve the findings.

Operator 4 The CAA began a section 15A investigation* of Operator 4 in July 2008 because safety concerns and audit results indicated that the operator's performance had deteriorated during the previous two years. Although the CAA did not continue this investigation after February 2009 because the operator was to exit the regulatory system within a few weeks, it included looking at whether the chief executive officer continued to be a "fit and proper person" for the role. If the investigation had found that he was not "fit and proper", he would have been removed from this role.

* The Director may, under section 15A of the Act, require operators to undergo any inspections and monitoring that the Director considers necessary in the interests of civil aviation safety and security.

4.45
We consider that the CAA should provide auditors with enough reference information to enable them to make consistent decisions. This guidance should at least indicate:

• what constitutes a risk to public safety and at what level of risk regulatory action should be considered;
• what information needs to be collected and considered when deciding on whether regulatory action needs to be taken;
• the process that needs to be followed when considering regulatory action; and
• the legislative authority for regulatory action and how this is applied in the courts through case law.

Recommendation 6
We recommend that the Civil Aviation Authority clarify how its regulatory focus is to be applied in practice through certification, surveillance, and other regulatory action by providing more detailed guidance to staff about what circumstances constitute a significant risk to public safety, and what action they should take when these safety risks are identified.

Informing regulatory decision-making

4.46
The CAA needs to improve its analysis of the safety data that it collects in its Management Information System. (This safety data is generated by the CAA as well as by the aviation industry.) The safety data needs to be used to better inform the strength of the regulatory focus for different groups of operators.

4.47
The CAA is currently taking action to improve the reliability of the safety data. In July 2008, it approved a policy on collecting and using safety data. Carrying out this policy is a "key" project of the Safety Information Group, and the CAA expects it to be completed before December 2011. However, this date is unlikely to be met. CAA staff told us that a lack of funding is constraining progress with this project. In our view, this project should be carried out as soon as possible.

Recommendation 7
We recommend that the Civil Aviation Authority give priority to completing the project to improve the integrity and reliability of safety data in its Management Information System, and improve the analysis of this data so that it can be used to better inform regulatory decision-making.

Management practices are not focused on improving staff performance, and the organisation has not been receptive to change

Management practices have improved within the Airlines Group, but not within the General Aviation Group. Performance agreements in the General Aviation Group do not set clear expectations nor require auditors to properly understand the importance of consistently applying the policies and processes for certification and surveillance. Overall, we consider that the CAA's culture is resistant to change and too risk averse when it comes to introducing innovation and continuous quality improvement.

Managing and leading staff to perform

4.48
There has been greater focus recently on effectively managing staff in the Airlines Group, where certification and surveillance improvements had gained more traction, than in the General Aviation Group. The management of the Airlines Group was more proactive and focused on quality improvement. For example:

• The general manager of the Airlines Group led an audit of the management aspects of a large airline, in which he demonstrated to the audit team how he would like them to plan audits. This included widening the scope of an audit in response to identified risks, and preparing a plan outlining specific questions to ask each senior person at the airline and identifying the records to be sampled.
• The unit managers' performance and development agreements focused on building capacity and capability within their units, management, and internal quality assurance.
• The Airlines Group worked with the Safety Information Group to produce regular reports highlighting the "top five" trends in occurrences from the airline sector, and has carried out work to investigate and address the causes of these trends.
• The Airlines Group carried out a complete review of the Entry Process Sheets (for certification) to reduce repetition and make them more user-friendly for staff within the group.

4.49
In the General Aviation Group, the leadership style had not changed since our last audit in 2005. The General Aviation Group managers were more focused on day-to-day operational work, rather than on process improvement or staff development. For example, the General Aviation Group had not adopted, nor required its auditors to use, the new certification process. Also the expectations in performance agreements for managers and staff were not aimed at improving the capacity and capability within the group.

4.50
Performance agreements for managers and staff in the Airlines Group and the General Aviation Group are linked back to the Groups' annual plans. This is designed to provide a stronger link between the strategic priorities in the statement of intent and day-to-day work.

4.51
We reviewed the agreed objectives in the performance agreements for staff in the Airlines Group and General Aviation Group. The performance and development agreements for the Airlines Group set clear expectations, and required staff to have a proper understanding of the importance of consistently applying the certification and surveillance policies and processes. The agreements were aimed at building on current performance and improving the capacity and capability within the group. The two Airlines Group unit managers are required to develop and maintain an accurate skills matrix for their units.

4.52
The performance goals for the General Aviation Group unit managers were less demanding of the managers and staff. They were more superficial and less specific, did not link staff performance to complying with the certification policy and processes, and were not clear about building on the capability and capacity within the group.

Recommendation 8
We recommend that the Civil Aviation Authority assess, and, where necessary, provide training to improve its managers' capability to effectively manage and lead staff. This includes improving the staff performance assessment process in the General Aviation Group.

The CAA's openness to change

4.53
During our audit and through other dealings that we have had with the CAA, we have found that the organisation has not been receptive to change. Auditors appear to be comfortable with past processes and in some instances have resisted the introduction of new processes. For example, the General Aviation Group auditors do not use the Entry Process Sheets even though new policies and procedures require them to. The very slow (13 years) and inadequate response to our recommendations, as well as a reluctance to address issues noted in other external reports, are also evidence that the CAA has not been receptive to change.

4.54
In our view, the pace of change within the CAA needs to be much quicker. The CAA needs to accept and promote continuous quality improvement. Most of the issues that we identified during our audit should have been identified and addressed earlier by the CAA.

4.55
The CAA's culture is too risk averse when it comes to introducing innovation and continuous quality improvement in the way that the CAA carries out its certification and surveillance work.

4.56
We found evidence that the CAA's auditors responded to high risk by increasing the frequency of surveillance but have been slower to respond to low risk through reducing the scope, or frequency, of surveillance – although this is now beginning to happen.

4.57
We found little evidence that auditors were adjusting the depth of the audits to reflect risk. This means that the underlying cause of the safety issue may not be addressed and the operator could become non-compliant with the Rules again.

4.58
In our view, the CAA needs to better match the intensity of its certification and surveillance work to its perception of risk. If an operator is low risk, then the CAA can improve its effectiveness by adopting the most appropriate audit methodology for that level of risk. In our view, the approach taken by the Aviation Security Unit of the CAA does this (see Appendix 4, paragraph A4.23). However, the Airlines Group and General Aviation Group will need to change the way that they perceive risk in the audit methodology that they use for certification and surveillance if risk-based auditing is to be consistently applied by these groups.

Management oversight of implementing and using the new certification and surveillance processes has been inadequate

Management oversight of the implementation and use of the new certification and surveillance processes has been inadequate, and so has the quality assurance of day-to-day certification and surveillance work. Auditors have not been told what information to record on file for the managers to be able to provide assurance about the quality of the auditors' decisions and recommendations.

Management oversight of implementing the new certification and surveillance processes

4.59
After our 2004 audit fieldwork, the CAA began to develop an integrated electronic system to improve the effectiveness and efficiency of the certification and surveillance processes. To achieve this integrated system, the CAA started three projects:

• the Certification Project;
• the Surveillance Review Project; and
• the Risk Assessment and Intervention Project.

4.60
The CAA executive team reduced the scope of the Certification Project in July 2007, putting on hold the development of electronic checklists tailored to each operator. Instead, the CAA introduced an alternative manual procedure for certification (Entry Process Sheets – see Appendix 3). Reducing the scope of this project to exclude the tailored checklists has meant that the efficiency gains intended from integrating the new certification and surveillance processes have not been achieved. The CAA unit managers are not tailoring the electronic checklists before each audit to adjust the scope and depth of the audits.

4.61
The auditors in the General Aviation Group were not required to use the new Entry Process Sheets even though the sheets had been formally adopted as part of the CAA's new certification policy and process.

4.62
As part of the Surveillance Review Project, the CAA introduced an electronic surveillance tool (including computer tablets for use by the auditors) to automate the surveillance process. However, the tool is not being used as intended by the Airlines Group and the General Aviation Group. Auditors have experienced problems with the tool, including issues with the software and a lack of training in using the new technology. The CAA has made major upgrades to the tool, and refinement is ongoing.

4.63
During our audit, the CAA started a project to review and improve the surveillance process and tool. If the problems that have been identified by this most recent review are addressed, it will address most of our recommendations in this report relating to the surveillance function.

Recommendation 9
We recommend that the Civil Aviation Authority give priority to completing the project to review and improve the surveillance process and tools, and ensure that all managers and auditors are using the new certification and surveillance processes.

Quality assurance by management of day-to-day certification and surveillance work

4.64
The new certification and surveillance processes have built-in managerial quality assurance reviews. However, they are effective only when the new processes (and relevant tools) are used. The robustness of the review also depends on the individual manager.

4.65
The unit managers differed in the extent of their review of auditors' certification and surveillance work, but, overall, the scope and depth of their review needed to increase.

4.66
There was not enough information on file for the managers to determine the scope and depth of audit work that had been done, and for the managers to be able to assess that the auditors' decisions and recommendations were based on robust evidence and analysis. This was particularly concerning in the General Aviation Group, where the differences in the numbers of findings made by auditors suggested inadequate moderation by the managers.

4.67
We therefore asked the two unit managers of the General Aviation Group how they obtained assurance that the auditors had carried out enough work and identified the risk areas of an operator's operation. Both managers told us that the auditors briefed them at the conclusion of each audit and that they "got a good feel" from these briefings and from the audit reports about the amount of work done. We consider that a verbal briefing is not enough.

4.68
In our view, the CAA needs to provide better guidance for auditors on how much information they are required to record about what was looked at during the audit (including the scope and the depth of testing), the analysis of the findings, and the effect that this has on the risk profile and future surveillance work.

4.69
We consider that the CAA' senior managers are responsible for providing auditors with this guidance, and for ensuring through their ongoing quality assurance activity that auditors are using the guidance and complying with policies and processes.

Recommendation 10
We recommend that the Civil Aviation Authority introduce more robust quality assurance of certification and surveillance work, including input into planning for certification and surveillance, reviewing the results, and moderating auditors' findings.
Recommendation 11
We recommend that the Civil Aviation Authority provide better guidance to its auditors on the level of documentation that needs to be retained as evidence of the certification and surveillance work that has been carried out, and reinforce the importance of clearly documenting the basis for decisions that involve serious consideration of evidence for a judgement to be made.
Recommendation 12
We recommend that the Civil Aviation Authority provide better guidance to its auditors on how to apply the "fit and proper person" criteria when carrying out assessments of senior persons.

Not enough attention has been given to improving organisational proficiency in auditing

Training has focused on the aviation technical proficiency of auditors, with not enough time given to enhancing audit skills and understanding of a risk-based audit methodology.

4.70
The CAA managers told us that a lot of time is spent training the auditors so that auditors are able to maintain their technical aviation proficiency. However, all training time was reduced in the CAA's 2009/10 budget in response to a projected significant reduction in revenue from fees and levies because of the economic downturn.

4.71
All CAA auditors are required to attend and pass a recognised "lead auditor" course. In response to the poor feedback by participants, the CAA is currently looking for alternative providers. At the time of our audit, the CAA was considering using the lead auditor course provided by the "CASA Academy" (Australia's Civil Aviation Safety Authority).

4.72
Recent work carried out by CAA staff has also identified significant gaps in the audit training available for auditors. This is discussed in Appendix 6.

4.73
In our view, the CAA needs to give priority to providing training and guidance for auditors to ensure that they have the appropriate skills and knowledge to effectively carry out certification and risk-based surveillance.

Recommendation 13
We recommend that the Civil Aviation Authority give priority to providing training in risk-based audit methodologies for its auditors, to ensure that they have the appropriate skills to carry out effective certification and risk-based surveillance.
Recommendation 14
We recommend that the Civil Aviation Authority provide detailed guidance to its auditors on risk-based auditing, including how information about risk can be used to tailor audits at the planning stage, how this information should be documented, how systems-based auditing should be applied, and how risk influences the size of samples checked during audits.

The Civil Aviation Authority's views on how a lack of capacity and capability has constrained change

4.74
ICAO noted during its 2006 audit that staff levels at that time did not allow for the development of regulation and guidance material and attendance at training courses in addition to ongoing work. The number of staff employed in the Airlines Group and the General Aviation Group has not increased significantly since 2006. In one unit of the Airlines Group, the number of staff has decreased. The CAA noted that, since 2006, the demand on CAA resources had increased through an increase in the number of registered aircraft and the number of certified operators, and through the increasing complexity of the operations of airline operators.

4.75
The CAA also noted that the Civil Aviation Safety Authority of Australia (CASA) in "very approximate terms" oversees three times the number of registered aircraft and pilots than CAA. It employs three times as many staff, but has an operating budget almost five times larger than the CAA's budget. The CAA believes that, even allowing for higher salaries and assuming higher expenses in Australia, CASA has significantly greater financial resources to apply to its operations and to facilitating change than the CAA does.

4.76
During our audit, the general managers of the Airlines Group and General Aviation Group told us that they thought that the staff levels were adequate for carrying out certification and surveillance work but not for also supporting the rules development programme or improving essential guidance material.

4.77
We reviewed the number of reported hours that auditors spent on surveillance, and found that the hours were fewer than we expected (see Appendix 6, paragraphs A6.13-A6.15). It was not possible for us to form a clear view on the productivity of auditors because the CAA did not have reliable information about how much time auditors spent on certification and other tasks.

4.78
We did not examine the adequacy of the operating budget for the CAA because this was outside the scope of our audit. We note that the CAA, supported by the Ministry of Transport, is carrying out a funding review. One of the aims of the review is that the CAA achieves greater efficiencies with its current resources. This review is not expected to be completed, or its effect on the amount of resources known, until early 2011.

---

15: This report is available on our website, www.oag.govt.nz.

16: Civil Aviation Authority (2009), Civil Aviation Authority Surveillance Policy, CAA, Wellington, page 12. This document is available on the CAA's website, www.caa.govt.nz.

page top