Appendix 4: Increasing the effectiveness and efficiency of surveillance resources

A4.1
In 2005, we recommended that the CAA continue with its review of its surveillance function and that, in designing a new approach, it should:

• ensure that the audit process directs resources at the highest-risk operators;
• direct appropriate activities and interventions at high-risk Safety Target Groups;
• give priority to the sampling project;
• assess where it can rely on operators' own quality and risk management systems, so that audits can be targeted at higher-risk areas;
• ensure that the depth and frequency of surveillance is adjusted to reflect operator and operation risk; and
• develop guidelines to indicate when instances of non-compliance should be referred to the CAA's Law Enforcement Unit for further action.

A4.2
In this Appendix, we look at:

• the Surveillance Review Project;
• the updated surveillance policy;
• auditor compliance with the surveillance policy;
• how risk influenced the method, depth, and frequency of surveillance;
• operator views on surveillance; and
• the Surveillance Process Improvement Project.

Our overall findings

A4.3
The CAA continued with its review of its surveillance function. This resulted in changes to its surveillance process and the introduction of an electronic surveillance tool in March 2007.

A4.4
The new surveillance process has led to a better allocation of staff between audit and administration tasks, and this has made more time available for audit tasks. However, the full extent of the expected efficiencies has not been achieved. The electronic surveillance tool was not implemented as intended, because the checklists tailored for each operator are not generated. These checklists were to form the basis of the surveillance audits and were to be used to adjust the depth and the frequency of the surveillance based on the operators' risk profiles.

A4.5
Although the Airlines Group and the General Aviation Group are sometimes adjusting the frequency and methods of surveillance in response to risk (especially high risk), neither group is using the new tool to adjust the depth or scope of routine audits and inspections.

A4.6
The implementation of the Surveillance Review Project was late, there were software problems, and auditors felt it did not meet their needs. As a result of this, the units use the new electronic surveillance tool variably.

A4.7
The Airlines Group continues to use customised audit programmes. These are made up of audit modules that are designed to cover all applicable parts of the Rules and to annually test the operator's compliance with its exposition. The modules are completed in a series of visits during the year. We saw evidence that the customised audit programmes for routine audits were repeated from year to year. We found little evidence that the Airlines Group reviews and adjusts the depth of the programmes to reflect the results of the audits.

A4.8
The General Aviation Group continues to use paper-based checklists.

A4.9
Not adjusting the depth of the audit means that the cause of the safety issues may not be identified and addressed, and increases the risk that surveillance may not be effective in checking that operators continue to operate safely. Not adjusting the scope means that the operator's whole operation is subjected to the same level of audit irrespective of the operator's overall level of risk or higher risk in parts of their operation. This approach is not only inefficient but it also means that surveillance may not be targeted at safety risks and may not identify safety issues and their "root cause" (see paragraph A4.35). The underlying cause of the safety issue will not be addressed and the operator will become non-compliant with the Rules again.

A4.10
We found that one unit (the Aviation Security Unit) had tailored the system to work well, with the depth and frequency of audits adjusted in response to risk information.

A4.11
When we checked whether the auditors were complying with the surveillance policy, we found that:

• The frequency of audit work is sometimes adjusted to reflect operator risk (increased audit work for higher-risk operators and, in some instances, decreased work for lower-risk operators). However, the depth of audits is not adjusted.
• The auditors are still not assessing where they can rely on operators' own quality and risk management systems, so that audits can be targeted at higher-risk areas.
• The CAA did not proceed with the sampling project so there is still no guidance available to auditors to enable them to make informed decisions on the work necessary to cover the assessed risk.
• The surveillance policy was updated and a new version was issued in June 2009. This new policy includes a section on how the regulatory tools should be used. It should assist the auditors to know when to refer instances of non-compliance to the CAA's Law Enforcement Unit for further action. However, the policy was not adopted until June 2009 so we were not able to assess its effect.

A4.12
We were pleased to note that the CAA has started a project (the Surveillance Process Improvement Project) to improve procedures for surveillance, including auditor competencies and training. The work done has identified that the electronic surveillance tool is effective if the Rule checklists (possibly supported by other checklists) are tailored to the type and risk of the organisation audited. Managers need to do this as part of the audit planning process. The project has identified that the managers needed training to enable them to do this.

A4.13
The Surveillance Process Improvement Project has also identified further gaps in training and guidance for auditors and managers. Resolving the problems identified by this project will address most of our recommendations about the surveillance function. However, in our view, there is nothing that this project has highlighted that should not have been identified and dealt with earlier if there had been appropriate management oversight of the process.

A4.14
We consider that our 2005 recommendation has been only partly addressed.

Surveillance Review Project

Auditors have experienced problems with the new electronic surveillance tool, and after major upgrades it continues to be refined. The new surveillance process has resulted in a better allocation of staff between audit tasks and administration tasks. However, the full extent of the expected efficiencies is not being achieved because the electronic surveillance tool is not being used as intended.

A4.15
The Surveillance Review Project began in late 2004 and has cost $1.2 million (the budget was $1.112 million). The review focused on improving the effectiveness and efficiency of the surveillance function through:

• identifying and assigning administrative, management, and auditing tasks to the appropriate staff (the new "surveillance process"); and
• using technology to automate the surveillance process where possible, including the use of computer tablets and computer systems that could generate checklists and record information (which feeds into the Management Information System). This technology is referred to as the "electronic surveillance tool".

Implementation of the new electronic surveillance tool

A4.16
The new surveillance tool and updated surveillance process were launched in May 2007. Auditors immediately experienced problems with the tool, which included issues with the software, lack of training in the use of the tool, and a general lack of preparation for the change.

A4.17
In August 2007, a second phase of the Surveillance Review Project was started to fix the issues with the software, and nine major upgrades were made to the tool. Further development of the tool was put on hold in March 2008 to give the tool some stability. The CAA considers the project to be complete. However, refinement of the tool is ongoing, and since March 2008 three further upgrades have been done.

Use of the new tool

A4.18
The Airlines Group and the General Aviation Group are not using the electronic surveillance tool as intended:

• Most auditors are not using the computer tablets and electronic checklists during the audit. Auditors told us that the Rule checklists are not ordered to reflect the audit process, the computer tablets created a barrier when interviewing operators, and the computer tablets were not practical to use when inspecting aircraft and hangars. The auditors also noted occasions when information had been lost from the Management Information System.
• The electronic checklists are not tailored to adjust the scope and the depth of the surveillance audit. Not adjusting the depth means that the cause of the safety issues may not be identified and addressed, and increases the risk that surveillance may not be effective in checking that operators continue to operate safely.

A4.19
The new surveillance process has improved the allocation of work between administration staff, managers, and auditors. We also found that this new process ensures that managers review the audit work. In most cases, managers commented on how the results of the audit affected an operator's risk profile.

A4.20
After the audit, the auditors complete the electronic Rule checklists, which are reviewed by the unit manager. Therefore, the electronic checklists are used to generate the audit report and audit findings.

A4.21
Our findings are supported by the findings of a post-implementation review of the Surveillance Review Project, carried out by an external reviewer in April 2008. The external review found that the expected outputs of the project had been substantially delivered, but that:

• the expected outcomes of improved surveillance efficiency and consistency had not been achieved because of incomplete customised checklists, workflow constraints, and use of alternative processes; and
• consistency of surveillance across the Groups remained an issue because some auditors were not reviewing the exposition against Rule checklists, or were not using the surveillance process and tool correctly.

A4.22
CAA staff told us that part of the reason why the new surveillance process and tool have not been as successful as intended is because the CAA underestimated, and was not prepared for, the amount of change the new process and tool would require. It required not only a change in technology but also a change to the culture of the organisation and the mind-set of managers. As noted elsewhere in this report, in our view, the CAA culture is not receptive to change. Auditors appear comfortable with past processes and, in some instances, have resisted introducing new processes.

A4.23
One unit within the CAA, the Aviation Security Unit,²⁰ was effectively using the process and tool as intended. The Aviation Security Unit focuses on using safety information to inform risk analysis and to respond to emerging risks or threats. The Aviation Security Unit manager uses this information to decide priorities for scheduling audits and surveillance. To prepare for audits of individual organisations, the manager builds an audit checklist based on the relevant Rule. Safety information and risk determine the scope and depth of the audits.

Updated surveillance policy

The surveillance policy was updated in June 2009 to more clearly state that surveillance priorities and methods should reflect identified strategic risks.

A4.24
The CAA's surveillance policy was reviewed and an updated policy document was issued in June 2009. The updated CAA surveillance policy more clearly states that surveillance priorities and methods should reflect identified strategic risks.

A4.25
The surveillance policy requires the CAA's operational groups to adjust surveillance priorities and methods – for example, to reflect the different risks associated with the type of operation or to reflect the strategies in the CAA business plan. The risk of an individual operator determines the frequency and depth of surveillance, including the level of sampling or surveillance method. A change in depth is made by changing the level of sampling done during the audit or inspection, or by carrying out a different method of surveillance (for example, a special purpose audit or inspection, a spot check, or unobserved surveillance).

A4.26
In 2005, we recommended that the CAA develop guidelines to indicate when instances of non-compliance should be referred to the CAA's Law Enforcement Unit for further action.

A4.27
The CAA has responded by updating its surveillance policy to include guidance to CAA staff making (or recommending) decisions about the choice and application of regulatory tools. This should help auditors decide when to refer instances of non-compliance to the CAA's Law Enforcement Unit for further action. However, the policy was not adopted until June 2009, so we were not able to assess its effect.

A4.28
The guidance is also intended to provide information to participants in the civil aviation system and other interested parties on what is taken into consideration in deciding which regulatory tool (for example, providing education, issuing a finding, or suspending an aviation document) is applied, and when it is applied.

Auditor compliance with the surveillance policy

In the Airlines Group, the Maintenance Unit was more consistent than the Flight Operations Unit in its quality of planning for audits. Both units could better document what work would be carried out, taking account of the operator's risk. Planning for audits in the General Aviation Group was poorly documented.

Planning for audits

A4.29
When reviewing our sample of files, we assessed the quality of planning that took place. We did this to establish the extent to which the audits were tailored to address the issues and risks associated with each operator. This included how the audit would check the operator's compliance with its exposition (if applicable), the selection of the appropriate audit method, what areas the audit would cover, and the level of sampling that would be done. We consider this amount of planning necessary to ensure that the audit not only addresses the risk issues but is also effective and efficient.

A4.30
In our view, preparing audit plans should not be a long and arduous process. For example, we noted one instance where the CAA carried out a joint audit with an operator. The audit had a plan setting out concisely (in one page) what would be audited and how it would be done.

A4.31
The amount and quality of planning varied between the units. In the Airlines Group, the Maintenance Unit was more consistent than the Flight Operations Unit in recording what planning had been done before the audit. There was evidence that the last audit report, findings, and occurrences had been reviewed before the audit in some cases. However, we consider that there needed to be more information about how the identified Rule aspects would be audited. For example, "maintenance specification control" was being audited, but there were no details of how the operator's exposition ensured that the organisation was compliant and what work would be carried out to assess whether the operator's procedures were effective.

A4.32
Planning in the Flight Operations Unit was variable. We noted only one example in our sample of 14 Flight Operation audit reports where an audit of an airline operator had been well planned. The preparation and planning for that audit focused on identifying the matters that needed to be examined during the audit as well as identifying a number of recent issues that needed to be investigated. There were comprehensive planning documents on file, including questions to ask each senior person and a detailed list of what records would need to be sampled (and how many of each) during the audit. In other examples, we were not able to find any evidence that the audit had been planned.

A4.33
In the General Aviation Group, it was difficult to establish how much planning and preparation had been done because of the lack of documentation on file.

How risk influenced the method, depth, and frequency of surveillance

The Airlines Group was changing the frequency and method of surveillance in response to high-risk operators. However, it was slow to identify the appropriate method and depth to establish the extent of the issues. In the General Aviation Group, the frequency of the audit had been increased for only half of the high-risk operators that we reviewed. The depth of the audits was not adjusted in response to risk.

A4.34
To assess the extent to which individual operator risk influences the method (routine audits, spot checks, and special purpose audits), depth, and frequency of surveillance, we selected two high-risk operators from the Airlines Group and five high-risk operators from each of the Rotary Wing and Agricultural Operations Unit and Fixed Wing Unit of the General Aviation Group. We expected to see evidence that both the depth and frequency of the audits were altered to reflect the operator's risk.

A4.35
In the Airlines Group, for both operators, we looked at whether the CAA had increased the frequency and changed the method of audits – both had been subject to an increased number of spot checks as well as a special purpose audit. However, the surveillance carried out by the CAA was not deep or thorough enough to get to the "root cause", identify the extent of the risk, and take appropriate action. The case study in Figure 10 at the end of this Appendix outlines an example in the Airlines Group where we had concerns about how robust and thorough the surveillance work had been.

A4.36
In the General Aviation Group, the CAA had increased the frequency of the audits in response to the operator's higher risk profile for only half of the highest-risk operators that we reviewed. Some operators remained on an annual audit cycle despite their high risk profiles.

A4.37
We found three examples in our sample of 10 high-risk operators where the surveillance was not as frequent as we expected, given the risk profile or other issues within the organisation. For example, for one operator, the CAA carried out a routine audit in 2008 that identified some serious concerns, including a lack of ownership of the organisation's new exposition (which had been adopted in 2007), not complying with its exposition, and management systems not keeping up with the rapid growth of the organisation. However, there was no evidence that surveillance was increased in response to these concerns. The operator remained on an annual audit cycle.

A4.38
We also found some cases where the frequency of surveillance had been reduced in response to the operator's low risk profile (for example, to every 18 months).

Guidance provided to auditors about the level of sampling that is needed

A4.39
The level of sampling depends on the judgement of each auditor. At the time of our 2005 report, there was no sampling methodology to guide auditors in exercising their judgement about sampling. The review of the surveillance process was to include a sampling methodology project, but this project has been deferred indefinitely. The adequacy of auditor sampling techniques is within the scope of the current Surveillance Process Improvement Project.

A4.40
Most of the general aviation operators we spoke to noted variations between what the auditors looked at during the audit. The operators thought that the Maintenance Unit auditors were more thorough or "picky" than the Flight Operations Unit auditors, and that the Rotary Wing and Agricultural Operations Unit auditors were more thorough or "picky" than the Fixed Wing Unit auditors.

Operator views on surveillance

Operators told us the surveillance process could be further improved.

A4.41
When we asked the operators what they thought about the annual surveillance audit, all supported the concept and appreciated the assurance the audits gave them. However, a number thought that the audits could be more robust. A couple of operators said that their own internal quality assurance systems or audits by other quality assurance organisations identified more safety issues than the CAA audits.

A4.42
A number of operators thought that the CAA auditors spent too much time looking at paper and not enough time checking what was actually happening. Most of the operators we spoke to thought the spot check was the most effective form of surveillance because it better portrayed their normal operations. Most commented that the announced audits gave operators time to prepare and get their records in order for the visit. Several operators went as far as supporting a "secret shopper" concept, where unidentified CAA auditors would take a flight with the operator.

A4.43
Overall, the operators we spoke to had not noticed a change in how the audits were carried out in the last couple of years. There were some complaints about the auditors using the computer tablets. The operators thought that these were less efficient and made the audit take longer.

Surveillance Process Improvement Project

The CAA has started another project to review and improve the surveillance process. Resolving the problems that have been identified through this review will address most of our recommendations in this report about surveillance.

A4.44
During fieldwork for our latest audit (in September 2009), the CAA started a project to review and improve the surveillance process, and also to improve auditor competencies and training. The CAA's executive team (which includes the chief executive officer and the group general managers) has identified this project as a high priority.

A4.45
The work that was completed in scoping the project established that the electronic surveillance tool was effective if the Rule checklists (supported by other checklists, where appropriate) were tailored to the type and risk of the organisation being audited. The managers needed to do this tailoring as part of the audit planning process, which will require training. Further gaps in training and guidance for auditors and managers were also identified. They needed training in the use of risk and safety information so that they could alter the depth, scope, and frequency of audits. They also needed training and guidance in how to set up audit modules and checklists.

A4.46
Resolving the problems that have so far been identified from the initial stages of the Surveillance Process Improvement Project will address most of our recommendations in this report about the surveillance function. In our view, if there had been appropriate oversight by senior managers of the surveillance function, these issues would have been addressed earlier without needing a further project.

A4.47
The following case study (see Figure 10) highlights various certification and surveillance issues that we found in our audit. It also shows that not carrying out certification work thoroughly can lead to increasing risk for the operator and more surveillance work than would otherwise have been necessary.

Figure 10
Case study, from our sample, of the Civil Aviation Authority's certification and surveillance work with an airline operator

* In the interests of civil aviation safety and security, the Director may, under section 15A of the Act, require operators to undergo an investigation.

20: The Aviation Security Unit is responsible for certification and surveillance of airline security programmes, regulating air cargo agents, and security aspects of aerodromes, airports, and Airways New Zealand Limited.