Part 7: Surveillance resources

• Time spent
• Surveillance audit training
• Quality assurance processes

7.1
In this Part, we:

• review the amount of time CAA inspectors spend on surveillance, to establish that this time has not decreased since the implementation of the new organisational structure in 2000;
• review the amount of surveillance audit training undertaken, to ensure that inspectors have and maintain appropriate skills; and
• assess the CAA’s internal quality assurance processes for the surveillance function, to ensure that the quality of this function is being adequately monitored.

Time spent

7.2
During our audit, we noted that inspectors did not record all the hours spent on the certification and surveillance functions (especially routine audit hours). We believe that this under-reporting is significant enough that we are unable to make a definitive statement on the trends in audit hours over the last 6 years. We are therefore unable to assess whether the CAA’s change in organisational structure in 2000 has affected the level of surveillance undertaken.

7.3
CAA staff told us that:

• the inspectors do not record all the hours that they work in the time recording system; and
• the majority of preparation and report writing time is often charged to an administration code.

7.4
This means that the CAA does not know the actual level of resources required both to maintain the surveillance programme and to improve the accuracy of its risk assessment tools (for example, the Non-Compliance Index uses audit hours as part of its calculation).

7.5
It also makes it difficult to assess the number of staff required. For example, the General Aviation Group has requested additional staff, but the hours reported for airworthiness and flight operations inspectors showed an average of only 288 hours of surveillance work for the 2003-04 year, which was down from the 318 hours recorded for the previous year.

7.6
We were told that one of the main reasons for not recording all the hours worked is that the cost of the routine audits is charged to the operator at $133 per person hour (normally 2 inspectors are involved in a routine audit, which means $266 per hour). Understandably, many smaller operators may be resistant to paying any more than the actual hours spent on-site at the organisation.

7.7
The Professional Standards Group reported this issue to the Authority’s Audit Committee in March 2004 noting –

Management and staff in these groups [the Airline and General Aviation Groups] are working extended hours to meet the industry demands. Time management figures establish that every person in the Groups record an additional 2 hours per person each week. Further, there is strong anecdotal evidence to suggest that both field staff and management are under-reporting hours worked. Invariably, travel time, work taken home and preparation and follow-up activities while in the field in the evening are not being recorded in timesheets.

7.8
The report then went on to say that the current workloads meant that the CAA’s operational groups had limited time for support activities – for example, following up audit findings, developing the CARs, group policy and procedures, and quality assurance and risk management activities.

7.9
The CAA’s internal auditors have also noted a high turnover of managers and staff within the operational groups, some of whom have left for positions with aviation operators.

7.10
The internal auditors have reported that workload and management turnover have resulted in piecemeal attention to various management functions (such as strategic and business planning, project and programme management, and staff oversight). In essence, operational activities (such as certification and audit) have taken priority over management functions, with some managers becoming too closely involved in operational activities.

Surveillance audit training

7.11
Two concerns were raised during our audit about the training of CAA inspectors. The first concern related to the deferral of training due to staff shortages and workloads within the operational groups. The second concern was from industry representatives, who were worried about the level of currency training done by inspectors.

7.12
New inspectors are required to attend a 1-week residential course in basic auditing, which is reinforced by a 2-day refresher course every 3 years (although we noted during our audit that the CAA was trying to locate a course better targeted at civil aviation audits). Inspectors are also required to do sufficient training to maintain their professional qualifications (currency). The CAA has also run courses on the Act, the CARs, risk analysis, report writing, and communication skills.

7.13
Inspectors may also be required to attend familiarisation courses for new aircraft run by aircraft manufacturers. For example, an inspector from the Airline Group travelled overseas to be familiarised with the Airbus A320 passenger aircraft.

7.14
The training budget was increased from $0.4 million in 1999-00 to $0.9 million in 2000-01 in response to criticism from the industry on the competency of CAA personnel. However, due to financial pressures and the high costs of specialised technical training, the CAA decided to “temporarily curtail” this investment. The training budget was reduced to $0.7 million in 2001-02, which allowed for the minimum of essential training to be carried out, and the actual amount spent was only $0.3 million.

7.15
In 2002-03, the training budget was maintained at the same level as in the previous year ($0.7 million) to enable training to be resumed. Resource demands resulted in actual expenditure being limited to $0.6 million.

7.16
In our view, the technical nature of the job is such that the training to which the CAA has committed must be done, notwithstanding competing priorities. The pace of technological change in the industry also reinforces the importance of training to maintain an inspector’s currency.

Quality assurance processes

Peer review

7.17
The CAA’s internal quality assurance processes require that every routine audit report and spot check report produced by the Airline and General Aviation Groups is to have an Audit Work Request Control Sheet attached to it. This control sheet requires audit reports (but not spot check reports) to be peer reviewed and proof-read by a person not involved in the audit.

7.18
We found that this review was often completed long after the audit. Generally, there were no peer review comments. To be meaningful to the inspectors being reviewed, and an effective quality improvement tool, these reviews need to be completed in a timely manner.

7.19
We also had concerns with the quality of some peer review comments noted on the routine audit reports. For example, in one case the peer reviewer noted that recommendations had been made. In our view, the reviewer should have questioned why the recommendations were made (because the CAA’s policies and procedures do not allow for them), and also why a Finding Notice was not issued for one particular recommendation, which was –

That further Competency Checks are to be clearly identified as such and be recorded in Company Records and the applicable Pilot Logbook.

7.20
In addition, the audit report noted–

The following comments were discussed with the Maintenance Controller during the audit and are listed to ensure they have been rectified:

• A loose leaf logbook entry (or direct entry) for a repair carried out on the right lower wing @ 4036.1 hours.
• During the inspection the Maintenance controller made logbook entries for the ELT RCPI battery change and the annual proof testing of the seat belts.

7.21
These seem to be matters where Finding Notices should have been raised to ensure that corrective action was taken by the operator.

7.22
We were also concerned to find a case where the peer reviewer noted that they did not think, given the comments in the report, that the QI was low enough. We were not able to ascertain that any action was taken as a result of this comment. Neither the report nor the QI score were adjusted, and nor was there any evidence that the inspector had justified the QI score in the light of the peer reviewer’s comments.

7.23
Group meetings are another quality assurance tool, but their frequency varies between the CAA’s operational groups. General Aviation Group managers meet once a week, and quarterly to examine the application of the general aviation rules and consistency of practice between inspectors. The Airline Group has a weekly management meeting, monthly unit meetings, and monthly group meetings.

Internal audits

7.24
We were pleased to see that the CAA’s Professional Standards Group has audited both the Airline and General Aviation Groups. For the financial year 2002-03, the internal auditors completed 9 audits of routine audits completed by the inspectors. In addition, the Professional Standards Group also completed 6 audits of activities by the Chief Legal Counsel and the Government Relations Group.

7.25
We reviewed a sample of the internal audits, and consider that they help to promote consistent good practice across the CAA. We also noted that they raised many of the issues that we identified during our audit. However, as the operational groups did not always “buy in” to the internal audit recommendations, corrective action has sometimes not been undertaken.

7.26
The internal audit function operates in a similar way to the surveillance function, in that CAA staff are audited for compliance against ISO standards, legislation, policies and procedures. Finding Notices are issued for any non-compliances or non-conformances, and an internal auditor (called a Co-ordinator Quality Systems) is responsible for ensuring that the appropriate corrective action is taken.

7.27
For example, an internal audit in February-March 2004 raised a Finding Notice against the Airline Group relating to minimal QA procedure and practice. The corrective action was that the Group’s management should adopt and implement CAA generic QA policy and procedures.²⁷

7.28
In response, the General Manager of the Airline Group noted that the group was not currently reporting on quality because it was waiting for CAA-wide policies and procedures to be developed. He considered that–

[while] The adoption and implementation of CAA generic QA policy and procedures is a priority for the CAA, this finding on a lack of procedure and practice in the Airline Group reflects a wider problem within the CAA.

7.29
He added that, while the CAA’s management had been given a presentation on the internal quality assurance system, all CAA staff should be given the same presentation because–

There is, as was demonstrated at a management level, an almost total lack of knowledge of the system.…The lack of a correctly functioning QA system is a serious risk to the CAA.

7.30
The internal auditors are planning regular (bi-monthly) workshops to consolidate action on internal audit findings and other CAA quality assurance policies and procedures.

7.31
The internal auditors normally arrange an internal audit programme for the year ahead. Historically, this has proved difficult to complete, however, because the 2 internal auditors are often taken off-task by various external reviews of the CAA – for example, those done by the Bureau Veritas Quality International (BVQI) and the State Services Commission²⁸ – and the work required by the Health and Safety in Employment Amendment Act 2002.

7.32
We consider that the internal audit process needs to be strengthened. The small size of the Group (2 internal auditors) and competing priorities mean that the practices used by each inspector will be covered only every 2 years. In our view, this review needs to be undertaken at least annually.

7.33
The internal auditors are also responsible for completing special projects, one of which is the audit rate sampling confidence project. This project is designed to help inspectors establish the level of sampling needed to establish confidence levels that any outcomes are representative. Sampling practices vary at present, and a standard for establishing confidence in sampling would promote consistency across the CAA.

ISO accreditation

7.34
The CAA is an ISO9001: 2000-accredited organisation, certified with BVQI, and is audited against this standard once a year. The last BVQI report we sighted, dated 29 January 2004, noted that the CAA’s Internal audits continue to be effectively managed, suitably reported and actioned in a timely manner…. The internal audit process is judged to be very effective.

27: During fieldwork for this audit, the Head of Professional Standards indicated that the development of generic CAA policies and procedures was progressing, and would be made available to staff electronically.

28: The SSC undertook a review of, and made recommendations regarding, the CAA’s internal policies and procedures for identifying and managing conflicts of interest. The SSC also reviewed the adequacy of the CAA’s practices, policies and procedures for the conduct of special purpose inspections and investigations relating to the suspension of aviation documents.