Appendix 1: Improving the analysis of safety information

A1.1
In 2005, we recommended that the CAA:

• continue to establish measures to better assess the effectiveness of its safety interventions; and
• improve its analysis of industry information by:
  • including more analysis of the information in the Aviation Safety Report and the Aviation Safety Summary Report to support further action, and to improve the timeliness of these reports; and
  • improving the analysis of accident and incident data from which the CAA will draft recommendations for safety intervention mechanisms.

A1.2
In this Appendix, we look at the CAA's response to those recommendations.

Our overall findings

A1.3
The CAA is collecting a lot of useful safety data, but, apart from a couple of initiatives, staff are yet to effectively analyse the data so that they can identify and formulate a response to strategic risks.

A1.4
Although the timeliness of the Aviation Safety Reports and the Aviation Safety Summary Reports has improved, the reports are still largely descriptive and contain little interpretative analysis of the information that can be used as a basis for further action. We were disappointed to find that there is still not enough analysis of the accident and incident data. The proportion of air accidents for which causal factors had been assigned has reduced since our 2005 report.

A1.5
In addition, the CAA has identified significant concerns about the reliability of its safety data (which it holds in its Management Information System). The reliability of this data is important because it is used for some important functions of the CAA (for example, the risk profiling system, the strategic planning process, and identifying trends within the civil aviation sector).

A1.6
The CAA is taking steps to improve the quality of accident and incident data. A policy about collecting and using safety data was approved in July 2008. The policy identifies the purposes for which the data will be used, and this should lead to better classification and analysis of the data. However, the policy will not be fully implemented until December 2011 at the earliest.

A1.7
We consider that the recommendations in our 2005 report have been only partly addressed.

Measures to better assess the effectiveness of safety interventions

The CAA has reviewed safety targets and has increased the number of Safety Target Groups from nine to 13.

A1.8
In our 2005 report, we noted that, in the CAA's annual reports for 2001/02, 2002/03, and 2003/04, the CAA had stated that the safety targets it was using were not a reliable measure of trends in the safety performance of the civil aviation industry.

A1.9
In 2005, the CAA also had concerns about the reliability of the data on which the measures were based, especially for the general aviation sector, because:

• aircraft flying hours were under-reported by owners, which meant that the reported safety rates may have looked worse than they really were; and
• accidents, incidents, and defects were also under-reported by either the pilots-in-command or the operators, which meant that the safety rates may have looked better than they really were.

A1.10
New safety targets and Safety Target Groups were established for the July 2005 to June 2010 period. The new targets included a new measure to quantify safety performance – the social cost of accidents for each unit of activity. The measures for these safety targets are now:

• the social cost of aviation accidents;¹⁷
• the rate of aviation accidents for every 100,000 flight hours; and
• the number of civil aviation fatalities and injuries.

A1.11
The Safety Target Groups were changed to better cover the civil aviation sector and make them more meaningful to the aviation community and the public. There are now 13 such Safety Target Groups, each of which falls under three main "interest areas":

• public air transport – in business to provide public transport for passengers and freight;
• other commercial operations – in business to provide non-public transport services (for example, training, agricultural mapping, and industrial operations); and
• non-commercial operations – all non-commercial operations, such as private flying and club flying.

A1.12
The CAA anticipates that, once the risk assessment system becomes established, increases or decreases in the overall risk profiles of aviation industry groups will be more meaningful indicators for reporting the effectiveness of the safety interventions.

A1.13
The CAA is also carrying out a research project to assess the safety benefit provided by the full range of higher-level interventions – including training and education, inspection and monitoring, and enforcement.

Analysis of industry information

The timeliness of the reports about aviation safety has improved, but they contain little interpretative analysis to inform action. The analysis of accident data to identify causal factors has deteriorated since our 2005 report.

A1.14
Although the timeliness of the Aviation Safety Report and the Aviation Safety Summary Report has improved, they are still largely descriptive. They contain little interpretative analysis of the information so that it can be used as a basis for further action. The CAA has only partly addressed our 2005 recommendation.

A1.15
In June 2008, the Board asked the CAA to include, in all future safety analysis reports, a comprehensive analysis of trends in safety performance and the actions it was taking in response to trends in safety performance. However, the Aviation Safety Summary Report for the second and for the third quarters of 2008 still did not specifically state what the CAA was doing to address the issues. In our view, the CAA does not appear to be able to take the high-level information and identify lower-level interventions to address the issues identified.

A1.16
The Aviation Safety Report also includes an analysis of causal factors for accidents. The purpose of doing this is to identify the appropriate remedial action to prevent future accidents. In our June 2005 report, we noted that, in the 18 months from 1 July 2002 to 31 December 2003, causal factors were assigned to only 37% of the recorded accidents. The proportion of air accidents for which causal factors had been assigned has now reduced. In the two-and-a-half years from 1 January 2006 to 30 June 2008, causal factors were assigned to only 56 of the 226 accidents reported (24%).

New initiatives identified

A1.17
During our audit, we noted two initiatives where the safety data in the Management Information System was used to identify strategic risk. The first initiative, the Agricultural Aircraft Safety Review,¹⁸ was a once-only study that began in March 2007 and was completed in December 2008. The CAA carried out the review in response to concerns expressed by aviation industry members and CAA staff about the increased number of fatal accidents involving agricultural aircraft between 2001 and 2004, as well as an increase in the number of defects in agricultural aircraft reported to the CAA.

A1.18
The review included an analysis of the occurrences (incidents, accidents, and defects) reported for agricultural aircraft and found there were significant increases in the accident and incident rates after the relevant Rule was changed to allow agricultural aircraft to carry heavier loads. It concluded that the CAA should rewrite the Rule to recognise both the economic advantages and the safety implications of agricultural aircraft being able to carry heavier weights. The CAA has included rewriting the Rule in its business plan for 2009/10.

A1.19
The second initiative is ongoing. The Airlines Group is obtaining regular reports highlighting the "top five" trends in occurrences reported to the CAA in the airline sector. Where applicable, the issue is investigated further to identify and address the causes of the trends. For example, a significant increase in Traffic Collision and Avoidance System alerts (an alert given to the pilot that they are too close to another aircraft) was noted, with the highest incidence at Christchurch. The issue was discussed with the Christchurch-based airlines; they changed the way they operated and this reduced the number of Traffic Collision and Avoidance System alerts reported there.

Problems identified with the quality of the safety information

The CAA has identified significant concerns about the reliability of the safety data and is taking steps to address these concerns.

A1.20
Bearing in mind the importance of the safety data in assessing risk, we were concerned to note that the CAA has identified problems with the reliability of the data that it holds in its Management Information System. A December 2008 paper from the CAA to the Board noted that there is "scepticism within the CAA about the reliability of the data and information that ought to predominantly inform regulatory decision making".

A1.21
The Agricultural Aircraft Safety Review also found that the CAA's management of recorded occurrence information could be significantly improved to provide more useful safety management information. This was because of the difficulties experienced in searching for the relevant information in the Management Information System. This CAA database stores the occurrences on the basis of the type of occurrence (there are 14 types) and the aircraft model, but not by the type of operation. This meant that the reviewers had to decide which aircraft models could be classified as "agricultural aircraft".

A1.22
The CAA is currently taking action to improve the reliability of the safety data. This includes the approval of a policy, in July 2008, on collecting and using safety information. Implementing this policy is a "key" project of the Safety Information Group and is expected to be completed before December 2011. However, this date is unlikely to be met. CAA staff told us that a lack of funding is constraining progress with this project. In our view, this project should be implemented as soon as possible.

17: The social cost of each unit of passenger exposure is defined as an economic measure of the cost of accidents to the nation. It assigns a value of statistical life (VOSL) to any deaths, rehabilitation costs from injuries, cost of property damaged or lost in the accident, and other specific external costs. The gross social cost calculated from accidents is pro-rated over the volume of aviation activity in any specified sector of the aviation community. The volume of aviation activity (the unit of passenger exposure) is measured by flying hours for each seat. For targets that are predominantly about carrying passengers, a surrogate of 500kg of aircraft weight is assessed as being the equivalent of an occupied seat.

18: Civil Aviation Authority (2008), Agricultural Aircraft Safety Review. Unpublished report, Wellington.