Appendix 2: Developing the risk assessment tools
A2.1
In 2005, we recommended that the CAA further develop the following three tools it used to assess the risks associated with individual operators:
- the Non-Compliance Index;
- the Quality Index; and
- the Client Risk Assessments.
A2.2
We also recommended that the CAA use better indicators of the financial status of operators when assessing operator risk, both for certification and during surveillance.
A2.3
In this Appendix, we look at:
- the Risk Assessment and Intervention Project; and
- better indicators of financial status.
Our overall findings
A2.4
The CAA has reviewed and improved the risk profiling of operators as part of the Risk Assessment and Intervention Project.
A2.5
The CAA auditors, general managers, and unit managers we spoke to were confident that the system is identifying the high-risk operators.
A2.6
The unit managers within the Airlines Group and General Aviation Group are told immediately of any changes to the risk profiles of individual operators. In addition, senior managers monitor individual operators' risk profiles each month.
A2.7
We consider that our 2005 report recommendation, described in paragraph A2.1, has been addressed.
A2.8
The CAA decided not to use better indicators of the financial status of operators when assessing operator risk because it could find no evidence that the financial status of the operator affects safety. However, it is still our view that cash-flow shortages increase the risk that some costs (for example, maintenance and training) will be deferred. We still consider that financial risk should be assessed as part of certification and surveillance, and that the CAA has not addressed our 2005 recommendation (described in paragraph A2.2).
The Risk Assessment and Intervention Project
The CAA's auditors and managers are confident that the new risk assessment tool is appropriately identifying risk. The tool is used to monitor operator risk and in some cases is leading to more frequent surveillance when high risks are identified.
A2.9
The Risk Assessment and Intervention Project began in 2005 and focused on improving the way that the CAA assessed the safety risk posed by individual operators. The project was originally intended to be completed by February 2006, but the resulting new risk assessment tool was not in use until February 2007. The project cost $109,000.
A2.10
The new risk assessment tool results in one risk profile for each type of aviation document held by an operator. A risk profile includes up to 34 parameters (which include the main criteria used in the previous three risk assessment tools). Auditors assess about two-thirds of the parameters based on their interaction with the operator (for example, after a surveillance audit or spot check). The CAA has developed rating scores and criteria for meeting the requirements of each score for each risk parameter. These help auditors to assess operators and help achieve consistency between auditors in doing this. The other parameters are calculated electronically as new data about an operator (for example, changes in senior staff, changes in the size or scope of an operation, and safety trends) is entered into the CAA database.
A2.11
The risk profile tool can be tailored (in terms of weighting individual parameters, rating scores and criteria for meeting the requirements of each score, and the type of parameter assessed) to meet the different profiles of the operational groups within the CAA. There was an expectation that this would happen. However, neither the Airlines Group nor the General Aviation Group has asked for the tool to be tailored for their group.
A2.12
The concept and the model for the risk assessment tool were validated by Aerosafe Risk Management Limited of Australia.
The effectiveness of the new risk assessment tool
A2.13
The auditors, unit managers, and general managers we spoke to had confidence in the risk profiles generated by the risk assessment tool. They thought the tool was identifying the high-risk operators.
A2.14
We saw evidence that monthly lists of the risk profiles for each operator were given to, and were monitored by, the managers. The managers were also assessing why the operators with high-risk profile scores had such high scores.
A2.15
Each unit manager receives an email alert when an operator's risk profile changes. These managers review the emails and make comments on the changes. Most alerts about a change in risk profile are then filed with no action needed. Some require a response and are passed to an auditor to follow up or are referred to the operator to see what they are doing to manage the risk.
A2.16
The unit managers also review an auditor's assessment of the operator's risk at the end of each surveillance audit or spot check. We saw evidence that the frequency of the surveillance is adjusted in response to the assessed risk in some cases.
Better indicators of financial status
The CAA decided not to implement our recommendation about using better indicators of financial status because it can find no evidence that the financial status of the operator affects safety.
A2.17
We consider that the risk assessment system should better reflect the operator's financial condition. Currently, financial risk is based on whether the operator has paid the CAA's fees (including any surveillance and certification fees). However, in our view, cash-flow shortages increase the risk that some costs (for example, maintenance, training, and replacing or upgrading aircraft) will be deferred. Cash-flow shortages could also increase the pressure for operators and pilots to fly in marginal weather conditions or at the limit of, or beyond, their capability.
A2.18
Therefore, we considered that the CAA should assess financial risk as part of the certification and surveillance functions.
A2.19
The CAA told us that it decided not to implement this recommendation because it can find no evidence that the financial status of the operator affects safety.
A2.20
The CAA also believes that the auditors' certification and surveillance checks would pick up where operators are "cutting corners" because of financial difficulties. Because our audit has identified that the CAA auditors are not necessarily identifying the underlying cause of non-compliance (see paragraph A5.5), we do not agree that "cutting corners" would always be identified.
A2.21
A number of the operators we spoke to during our audit consider that there is a link between operators under-charging for services and a likelihood that those operators would be involved in an accident.
A2.22
We still consider that the CAA should assess financial risk as part of the certification and surveillance functions.