Appendix 6: Improving the use of resources

A6.1
In 2005, we recommended that the CAA's auditors ensure that they recorded all the time that they spent on the surveillance function.

A6.2
We also recommended that the CAA:

• ensure enough investment in training CAA staff so that they develop and maintain the appropriate skills to carry out their functions;
• review its staffing levels when the current review of the surveillance function has been completed, to ensure that it has adequate resources to carry out this function;
• ensure that the operational groups comply with the CAA's generic policies and procedures (particularly for quality assurance);
• promote consistent standards of quality and practices throughout the operational groups by ensuring that they address internal audit findings; and
• ensure that the internal audit section is appropriately staffed to enable the CAA's operations and auditors to be audited more regularly.

A6.3
In this Appendix, we look at:

• training for CAA staff;
• staffing and resource levels for certification and surveillance;
• planning for auditors' time and measuring the effectiveness of certification and surveillance;
• quality assurance; and
• internal audit.

Our overall findings

A6.4
CAA managers had advised auditors to make sure that they were recording all the time that they spent on surveillance. The managers were confident that the auditors are now doing this. In our view, although the managers consider that auditors record all the time that they spend on the surveillance function, there is evidence to suggest this issue has not been fully resolved. We therefore consider that our recommendation, detailed in paragraph A6.1, has been only partly addressed.

A6.5
We also found that:

• Training has focused on maintaining auditor proficiency in aviation (that is, pilot competency or aircraft engineer skills) rather than auditing skills. Detailed training was provided to staff on how to use the new electronic surveillance tool.
• The Airlines Group needs to improve its planning for resources spent on certification and surveillance. This group also needs to analyse the variations between targets and actual hours spent on certification and surveillance.
• The CAA has had trouble recruiting sufficiently qualified auditors.
• The general managers of the Airlines Group and General Aviation Group considered that the staff levels were adequate for carrying out certification and surveillance work but not for also supporting the rules development programme or improving essential guidance material.
• The number of reported hours that auditors spent on surveillance were fewer than we expected. It was not possible for us to form a clear view on the productivity of auditors because the CAA did not have reliable information about how much time auditors spent on certification and other tasks.
• Quality assurance reviews are built into the new certification and surveillance processes. However, they are only effective when the new processes (and relevant tools) are used. Also, the robustness of the review depends on the individual manager. The lack of supporting evidence on file makes us question the basis on which managers are making their assessments. We also consider that reviews by managers in the certification process need to be more robust.
• The internal audit unit was disbanded in June 2009. An Internal Audit Service Provider was appointed for a term of two years. Its engagement began in October 2009.

A6.6
We consider that our recommendation, detailed in paragraph A6.2, has been only partly addressed.

Training for Civil Aviation Authority staff

Training has focused on the aviation technical proficiency of auditors, with not enough time given to developing audit skills and to understanding a risk-based audit methodology.

A6.7
We were told by the managers that a lot of time is spent training the auditors so that they are able to maintain their technical proficiency. However, this time has been reduced in the 2009/10 budget in response to a projected significant reduction in revenue from fees and levies because of the economic downturn.

A6.8
All CAA auditors are required to attend and pass a recognised "lead auditor" course. In response to the poor feedback by participants, the CAA is currently looking for alternative providers. At the time of our audit, the CAA was considering using the lead auditor course provided by the "CASA Academy" (Australia's Civil Aviation Safety Authority).

A6.9
The Director is also providing training on the role of the regulator. We observed the first session of this series of training. In our view, the Director needed to more forcefully convey his expectations of auditors for the certification and surveillance functions, rather than what the auditors considered was appropriate.

A6.10
All the auditors have been trained to use the new electronic surveillance tool.

A6.11
Work carried out for the Surveillance Process Improvement Project has identified that the existing audit training available for auditors does not cover:

• training on Rules applicable to the auditor's particular role;
• communication skills;
• audit report writing;
• analysis of findings, causes, and actions;
• conducting an audit;
• sampling;
• interpreting the rating scores and associated criteria for assessing operator risk to ensure consistency in applying the risk scores;
• use of risk information (by managers); and
• setting and changing audit module scope, depth, and frequency.

A6.12
In our view, the CAA needs to give priority to providing training for auditors to ensure that they have the appropriate skills, including risk-based auditing skills, to effectively carry out certification and surveillance.

Staffing and resource levels for certification and surveillance

The managers have reviewed staffing levels and consider them to be adequate for certification and surveillance work, but not for necessary additional tasks. The information provided to us indicates that the time spent on surveillance was much lower than we expected.

A6.13
Additional auditors have been employed since our last audit, especially in the General Aviation Group. The general managers of the Airlines Group and the General Aviation Group thought that the levels were adequate for carrying out certification and surveillance work but not for also supporting the rules development programme or improving essential guidance material.

A6.14
The CAA told us that it is difficult to recruit appropriately qualified auditors.

A6.15
Based on the hours spent on surveillance reported in Figure 12 and the staffing levels in Figure 1, in the year ended 30 June 2009, the Airlines Group auditors spent, on average, 359 hours on surveillance audits. The General Aviation Group auditors spent, on average, 169 hours on surveillance audits. We have discussed this apparently low workload with the CAA. It told us that the average hours for the General Aviation Group auditors do not include travel time. This means that the hours would double for the General Aviation Group auditors. However, even doubling the hours still means that, in total, fewer hours were spent on surveillance than we expected. It was not possible for us to form a clear view on the productivity of auditors because we were not able to establish with any certainty how much time auditors spent on certification and other tasks.

Planning for auditors' time and measuring the effectiveness of certification and surveillance

The Airlines Group needs to improve its planning for resources spent on certification and surveillance, and analyse the difference between its target and actual hours. The current measures of the effectiveness of certification and surveillance work do not give assurance that auditors' time has been appropriately targeted, nor do they provide any information about the effect of surveillance and certification on operators' behaviour.

A6.16
The Airlines Group and the General Aviation Group assess the:

• number of certification requests that each group will receive during the year (this includes new certifications, recertifications of operator aviation documents, and approving changes to the conditions in the aviation documents); and
• number of hours that the auditors will spend on surveillance.

A6.17
The number of recertifications can be predicted by analysing certificate expiry dates, but the number of changes to aviation documents is driven by demand and is less predictable. Figure 11 shows the actual number of certifications, recertifications, and changes to aviation documents completed for the last five years and the "target" or estimated number of certifications and recertifications to be completed in that time.

6.18
We note that the Airlines Group's certification targets have not been increased in the last five years, despite the fact that the numbers of requests for certification have been consistently increasing and are significantly above the target figures. For 2008/09, the actual number of requests was almost double the 2008/09 target. In our view, if the Airlines Group's planning for resources is to be meaningful, then estimates need to be more realistic. The General Aviation Group is estimating its projected certification workload better than the Airlines Group.

Figure 11
Annual numbers of certification requests received by the Airlines Group and the General Aviation Group, and targets for this work

Note: The figures include recertification and changes to aviation documents.

A6.19
Figure 12 shows the number of actual hours that the Airlines and General Aviation Groups have spent on the surveillance function in the last five years and their "target" or estimated hours that they planned to spend on surveillance.

Figure 12
Annual reported hours for surveillance work by the Airlines Group and the General Aviation Group and targets for this work

* These hours are not directly comparable to other targets and actual hours in this figure because the General Aviation Group's 2006/07 target and 2005/06 data included travel time to and from an audit or inspection.

A6.20
The data in Figure 12 shows that, for the last five years, the Airlines Group's targets for spot checks have not changed. This Group's targets for routine audits have also been the same each year except for 2008/09. In our view, the failure to adjust targets again raises questions about how robustly the Airlines Group plans its work. There was also no explanation about why the hours spent on routine surveillance in 2007/08 was 21% less than the average of the other four years.

A6.21
The General Aviation Group's targets for surveillance work were usually adjusted to reflect the projected changes in workload each year. However, we note that, although there are fluctuations because travel time was included for 2005/06 and the 2006/07 targets, total surveillance time (hours estimated for routine audits and spot checks) was projected to increase from 4000 to 5000-6400 hours. These targets have not been met. The actual hours decreased from 4550 hours in 2004/05 to 3611 hours in 2007/08, but increased again to 4195 hours in 2008/09. We also note that the overall decrease in hours is actually larger than shown in Figure 12 because auditors' time recording has improved in the last five years (previously auditors were not recording all time they spent on surveillance). We were not able to find any reason why the hours have reduced.

A6.22
The managers we spoke to were confident that auditors were recording all the time that they spent on certification and surveillance work.

A6.23
In our view, there is evidence that suggests this is not the case. The unexplained reduction in hours, discussed in paragraph A6.21, together with the apparently low number of hours recorded by auditors (see paragraph A6.15), suggests that auditors are still not recording all their time spent on surveillance.

A6.24
We also noted that, for the Airlines Group and General Aviation Group, the effectiveness of their certification and surveillance work is measured by the number of certification requests that have been done and by the number of hours spent on surveillance and spot checks. The quality of both certification and surveillance is measured by the results found by the CAA's internal audit unit.

A6.25
In our view, these are not good measures of the effectiveness of certification and surveillance. The measures do not give assurance that surveillance resources have been appropriately targeted at the higher-risk operators, nor do they provide any information about the effect of the surveillance and certification on operator behaviour.

A6.26
The CAA is currently looking at addressing this issue, and in future it intends to measure the effect that surveillance has in reducing operator risk profiles. In our view, the risk profiles could also be used to better target the amount of resource assigned to surveillance and certification work.

Quality assurance

Quality assurance activities are not rigorous enough to ensure that auditors use the CAA's policies and procedures, that the scope and depth of the audits is robust enough to identify significant safety issues, and that auditors are consistent in their approach.

A6.27
Because of the lack of documentation, we were not sure how the unit managers in the General Aviation Group were ensuring that their auditors were following the CAA's policies and procedures for both certification and surveillance.

A6.28
There was not enough information to determine the scope and depth of audit work done. We asked the General Aviation Group unit managers how they obtained assurance that a quality audit had been done. They told us that the auditors briefed them at the end of each audit and they "got a good feel for" the amount of work done from these briefings and from the audit reports. We consider that a verbal briefing is not enough and that a record should be maintained of the amount of audit work done, an analysis of what the work found, and how it linked to the overall recommendations and findings.

A6.29
The Maintenance Unit of the Airlines Group was better than the Airline Flight Operations Unit at recording this information as part of the audit report. However, the amount of information the auditors in the Airlines Flight Operations Unit recorded varied significantly. Again, there needs to be better guidance on the amount of information that needs to be recorded about what was looked at during the audit (including both the scope and the depth of testing), the analysis of the findings, and the effect that this has on the risk profile and future surveillance work. Overall, the information that is recorded is inadequate for the manager to be able to assess that the recommendation is based on robust evidence and analysis.

A6.30
We also noted a significant difference in the number of findings that auditors were issuing. The difference is significant enough to suggest that the auditors are taking a different approach (see paragraphs A5.13-A5.23).

Internal audit

The internal audit function was disbanded in June 2009 and an Internal Audit Service Provider began work in October 2009. The internal audit programme does not include giving the Board assurance that the overall goals and objectives of the CAA are being achieved.

A6.31
Internal audit provides the Board with assurance that the CAA is complying with policies and processes. In the past, this internal audit function was largely carried out by the Professional Standards Unit of the CAA, although there was a period from August 2006 to July 2007 when the CAA seconded services from Ernst & Young. During this time, the internal audit activities were supervised by CAA staff and Ernst & Young used the CAA's internal audit practices.

A6.32
The internal audit programme was approved by the Board's Audit and Risk Management Committee, and the results of the internal audits were reported to this Committee. The annual audit programme covered all the units within the CAA. However, we note that the internal audit did not detect the underlying problems with the quality of the data recorded in the Management Information System.

A6.33
In April 2009, the Director recommended that the Board outsource the internal audit function for two years. The decision was endorsed by the Board's Audit and Risk Management Committee. The internal audit unit was disbanded in June 2009.

A6.34
An Internal Audit Service Provider was appointed and began its engagement in October 2009. A member of the Professional Standards Unit has been retained to provide continuity and to act as the planner, co-ordinator, and liaison between the Board's Audit and Risk Management Committee and the Internal Audit Service Provider.

A6.35
In our view, the annual internal audit programme is focused on checking procedural compliance and needs to be extended to assess how well the Groups are contributing to the CAA's strategic priorities and the extent to which the overall goals and objectives of the CAA are being achieved.