Part 3: The Civil Aviation Authority's response to our 2005 recommendations

The Civil Aviation Authority's progress with improving certification and surveillance.

3.1
In our 2005 report, we made 10 recommendations for the CAA in six broad categories:

  • improving the analysis of safety information;
  • developing the risk assessment tools;
  • ensuring that the CAA auditors follow certification policy and procedures;
  • increasing the effectiveness and efficiency of surveillance resources;
  • ensuring that findings14 are issued and prompt action is taken for non-compliance; and
  • improving the use of resources, including investing in training and complying with quality assurance processes.

3.2
In this Part, we look at how well the CAA has responded to our recommendations from 2005.

Our overall findings

3.3
Of the 10 recommendations that we made in 2005, we consider that:

  • only one has been fully addressed;
  • eight have been only partly addressed; and
  • one has not been addressed.

3.4
At a strategic level, there has been little improvement since 2005 in the CAA's analysis and use of safety information, such as accident and incident data, to inform how tightly it regulates different groups of operators and focuses certification and surveillance on risk areas.

3.5
At an operational level, the CAA has strengthened its processes for certifying and monitoring operators by introducing new policies, procedures, and tools. This has resulted in some improvements in practice, including better profiling of the risks of individual operators and adjusting the frequency of surveillance in response to risk. However, the CAA has still not addressed weaknesses that we identified in previous reports. For example:

  • The depth of work completed before certifying operators is not adequately documented.
  • Surveillance is not always targeted at higher-risk areas and operators.
  • Instances found by CAA auditors of non-compliance with the Rules are not consistently reported and followed up.

3.6
In addition, the introduction of the new policies, procedures, and tools at the operational level has been delayed or has been very recent, and they are not yet fully implemented or consistently applied, particularly in the CAA's General Aviation Group. The CAA has not made as much progress in addressing our recommendations as we expected. We discuss our observations on this slow pace of change in Part 4.

How well the Civil Aviation Authority has responded to our recommendations

3.7
Figure 4 sets out our assessment of whether the CAA has addressed our 2005 report recommendations (the recommendations are in the six categories set out in paragraph 3.1).

Figure 4
Our assessment of whether the Civil Aviation Authority has addressed the recommendations in our 2005 report

Improving the analysis of safety information
Our 2005 recommendations Our assessment
We recommended that the CAA continue to establish measures to better assess the effectiveness of its safety interventions.

We recommended that the CAA improve its analysis of industry information by:
  • including more analysis of the information in the Aviation Safety Report and the Aviation Safety Summary Report to support further action, and to improve the timeliness of these reports; and
  • improving analysis of accident and incident data ... from which the CAA will draft recommendations for safety intervention mechanisms.*
 The CAA has revised its safety targets and Safety Target Groups.

The CAA is collecting a lot of useful safety data but, apart from a couple of initiatives, staff are yet to analyse the data to effectively identify and formulate a response to strategic risk issues.

Although the timeliness of the Aviation Safety Reports and the Aviation Safety Summary Reports has improved, the reports are still largely descriptive and contain little interpretative analysis of the information so that it can be used as a basis for further action. We were disappointed to find that there is still not enough analysis of the accident and incident data. The proportion of air accidents for which causal factors have been assigned has reduced since our 2005 audit.

In addition, the CAA has identified significant concerns about the reliability of the safety data and is taking steps to address these issues.

We consider that these two recommendations have been only partly addressed. Our detailed findings are set out in Appendix 1.

* In our 2005 report, this recommendation had slightly more text than we needed to repeat in this summary.

Developing the risk assessment tools
Our 2005 recommendations Our assessment
We recommended that the CAA further develop the tools it uses to assess the risks associated with individual operators.* The risk profiling of operators was reviewed and improved as part of the Risk Assessment and Intervention Project.

The CAA auditors, general managers, and unit managers we spoke to are confident that the system is identifying the high-risk operators.

Individual operators' risk profiles are monitored monthly by CAA management.

We consider that this recommendation has been addressed.
We recommended that the CAA use better indicators of the financial status of operators when assessing operator risk, both at certification and during surveillance. The CAA decided not to use better indicators of the financial status of operators when assessing operator risk because it could find no evidence that the financial status of an operator affects safety.

However, it is still our view that cash-flow shortages increase the risk that costs (for example, maintenance and training) will be deferred. We therefore still consider that financial risk should be assessed as part of certification and surveillance.

We consider that this recommendation has not been addressed by the CAA. Our detailed findings are set out in Appendix 2.

* In our 2005 report, this recommendation had slightly more text than we needed to repeat in this summary.

Ensuring that the CAA auditors follow certification policies and procedures
Our 2005 recommendations Our assessment
We recommended that the CAA ensure that its auditors* follow the policies and procedures set down for certification. The CAA has updated its certification policy and procedures. These provide a better defined process of what auditors need to do during certification.

However, we found differences in approach between the auditors in the Airlines and General Aviation Groups. The Airlines Group auditors are using the new procedures set out in the Entry Process Sheets (electronic checklists), which provided us with evidence that the required steps set out in the certification policy and procedures were followed. Because the General Aviation Group auditors are not using the Entry Process Sheets, we were not always able to establish that the auditors in this group had completed the certification process.

Both groups need to better document the results of the work carried out, and the weight given to evidence provided, when assessing senior persons as "fit and proper" when the evidence requires a judgement to be made.

We consider that this recommendation has been only partly addressed. Our detailed findings are set out in Appendix 3.

* Our 2005 report used the term "inspectors" when referring to CAA auditors. In this report, we have substituted "auditor" when referring to "inspector" in our 2005 recommendations.

Increasing the effectiveness and efficiency of surveillance resources
Our 2005 recommendations Our assessment
We recommended that the CAA continue with its review of its surveillance function. In undertaking this review and designing a new approach, the CAA should:*
  • ensure that the depth and frequency of surveillance is adjusted to reflect operator and operation risk;
  • ensure that the audit process directs resources at the highest-risk operators;
  • assess where reliance can be placed on operators' own quality and risk management systems, so that audits can be targeted at higher-risk areas;
  • direct appropriate activities and interventions at high-risk Safety Target Groups;
  • give priority to the sampling project (a sampling methodology will allow auditors to make informed decisions on the work necessary to cover the assessed risk); and
  • develop guidelines to indicate when instances of non-compliance should be referred to the CAA's Law Enforcement Unit for further action.
 The CAA has continued with its review of its surveillance function, and a new electronic surveillance tool was introduced in March 2007.

The new surveillance process has led to a better allocation of staff between audit and administration tasks, and this has made more time available for auditing. However, the full extent of the expected efficiencies has not been achieved because the tool has not been implemented as intended – checklists tailored for each operator are not generated. These checklists were to form the basis of the surveillance audits and were to be used to adjust the depth and the frequency of the surveillance based on the operators' risk profiles. The implementation of the Surveillance Review Project was late, there were software problems, and auditors felt it did not meet their needs. As a result of this, the new process and tool are used variably by different units.

When we checked whether the auditors are complying with the surveillance policy, we found that:
  • The frequency of audit activity is sometimes adjusted to reflect operator risk (increased activity for higher- risk operators and decreased for lower-risk operators). However, the depth of audits is not adjusted.
  • The auditors are still not assessing where they can rely on operators' quality and risk management systems, so that audits can be targeted at higher-risk aspects of operations.
  • The sampling project did not proceed, so there is still no guidance available to auditors to enable them to make informed decisions on the work necessary to cover the assessed risk.
  • The surveillance policy was updated and a new version was issued in June 2009. This new policy includes a section on how the regulatory tools (for example, imposing conditions on an aviation document, or suspending or revoking an aviation document) should be used. It therefore should assist the auditors to know when to refer instances of non-compliance to the CAA's Law Enforcement Unit for further action. However, the policy was not adopted until June 2009, so we were not able to assess its effect.

We consider that this recommendation has been only partly addressed. Our detailed findings are set out in Appendix 4.

* We have changed the order of the following bullet points from the order in which they appeared in our 2005 report.

Ensuring that findings* are issued and prompt action is taken for non-compliance
Our 2005 recommendations Our assessment
We recommended that the CAA auditors issue a Finding Notice for all identified instances of non-compliance and non-conformance. The auditors are still not always issuing findings when necessary.

There is still a problem in the inconsistency of findings between auditors, both with the number issued as well as the type or severity of the finding.
We recommended that the CAA establish a system that ensures that operators take quick and effective corrective action when auditors tell them to do so. This system should include re-assignment of responsibility for that function when an auditor leaves the CAA. A system has been established to ensure that findings are followed up. Administrative staff monitor whether operators send evidence of corrective action to the CAA. If operators do not send such evidence, the administrator sends reminders. The auditor who issued the finding decides whether the action taken is appropriate and properly evidenced.

The timeliness of following up critical findings has improved. However, the CAA is falling well below its target for closing all findings by the due date.

We consider that these two recommendations have been only partly addressed. Our detailed findings are set out in Appendix 5.

* In our 2005 report, we referred to "Finding Notices" because the CAA at that time was issuing actual hard-copy notices. In this report, we refer to "findings" instead.

Improving the use of resources, including investing in training and complying with quality assurance processes
Our 2005 recommendations Our assessment
We recommended that the CAA auditors ensure that they record all time spent on the surveillance function...* The CAA managers told us that they had advised auditors to make sure that they were recording all the time they spent on surveillance. The managers are confident that the auditors are now doing this.

In our view, the unexplained decreases in auditors' time spent on surveillance, and the fact that the average number of auditor hours recorded for surveillance is so low, suggest that this issue has not yet been fully resolved.

We consider that this recommendation has been only partly addressed.
We recommended that the CAA:
  • ensure sufficient investment in training CAA staff so that they develop and maintain the appropriate skills to carry out their functions;
  • review its staffing levels when the current review of the surveillance function has been completed, to ensure that it has sufficient resources to undertake this function (both the review of the surveillance function and the review of staffing levels need to take account of the potential pressures or "surges" put on auditors as a result of unanticipated requests for certifications);
  • ensure that the operational groups comply with the CAA's generic policies and procedures (particularly relating to Quality Assurance);
  • promote consistent standards of quality and practices throughout the operational groups by ensuring that they address internal audit Finding Notices; and
  • ensure that the internal audit section is appropriately staffed to enable the CAA's operations and auditors to be audited on a more regular basis.
 Training has focused on maintaining auditor proficiency in aviation (that is, pilot competency or aircraft engineer skills) rather than auditing skills. Detailed training was provided to staff on how to use the new electronic surveillance tool.

The managers have reviewed staffing levels and consider them to be adequate for carrying out certification and surveillance work, but not for also supporting the rules development programme or improving essential guidance material. To us, the hours spent on surveillance appear low, but we were not able to form a clear view on the productivity of auditors because the CAA did not have reliable information about how much time auditors spent on certification and other tasks. The CAA continues to have trouble recruiting sufficiently qualified auditors.

Quality assurance reviews are built into the new processes and tools. However, they are only effective when the new processes and tools are used. Also the robustness of the review depends on the individual manager. The lack of supporting evidence on file leads us to question the basis on which managers are assessing whether a quality audit has been carried out. We also consider that manager reviews in the certification process need to be more robust.

The internal audit unit was disbanded in June 2009. An Internal Audit Service Provider was appointed for a term of two years. Their term began in October 2009.

We consider that this recommendation has been only partly addressed. Our detailed findings are set out in Appendix 6.

* In our 2005 report, this recommendation had slightly more text than we needed to repeat in this summary.

14: In our 2005 report, we referred to "Finding Notices" because the CAA was issuing a hard-copy notice for each identified finding. In this report, we refer to "findings".

page top