Efficiency of the Investigation of Transport Accidents and Safety Occurrences
Entities audited: Australian Transport Safety Bureau
Introduction
3.1
The Australian Transport Safety Bureau (ATSB) was established by the Transport Safety Investigation Act 2003 and is classified as a non-corporate Commonwealth entity under the Public Governance Performance and Accountability Act 2013
3.2
In 2017-18, the ATSB had a full time Chief Commissioner, three part-time commissioners and 109 staff across its offices within Australia.
3.3
The ATSB seeks to improve the safety and public confidence in the aviation, marine and rail modes of transport through:
Independent investigation of transport accidents and other safety occurrences;
Safety data recording, analysis and research; and
Fostering safety awareness, knowledge and action.
Audit Scope
3.4
The ATSB was selected for audit as a part of a series of performance audits focusing on the efficiency of entities. The objective was to examine the efficiency of the ATSB’s investigations into accidents and safety occurrences involving aviation, marine and rail modes of transport. To form a conclusion against the audit objective, the Australian National Audit Office (ANAO) adopted the following high level criteria:
Has the ATSB put in place efficient processes for the investigation of transport accidents and safety occurrences?
How well does the ATSB’s investigation efficiency compare to its own previous performance, as well as relevant international comparator organisations?
3.5
In conducting this audit, the ANAO applied the ANAO Special Considerations for Efficiency Auditing Methodology and Guidance, which is based on a general model for assessing public sector performance. In addition, the ANAO applied specific audit procedures that included:
Analysis of entity records;
Interviews of management and key stakeholders;
Reviewing and assessing the case prioritisation model used to determine which notifications are investigated;
Analysis of available data; and
Benchmarking with relevant international comparators and the ATSB’s past performance drawing on data obtained by the ATSB during the audit.
3.6
The audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $301 000 and the audit team consisted of four in house members.
Overall Audit Conclusion
3.7
The audit found that the efficiency with which the ATSB investigates transport accidents and safety occurrences has been declining. The ANAO acknowledged that the ATSB is aware of this trend and have implemented a range of strategies to improve efficiency. Changes include: shifting resources to address the backlog of investigations, dedicating resources to shorter investigations, applying sound processes to decide which notifications merit a safety investigation, and adjusting key performance indicators (KPIs) to identify more realistic completion timeframes for complex investigations.
3.8
The audit concluded that the ATSB has established key elements of a framework to promote efficient investigation processes. Change is underway with the implementation of the Safety Improvement Reference Framework. The ATSB is also taking steps to benchmark its performance against other international transport investigation entities.
3.9
The audit made four recommendations that focused on improving investigation timeframes, resourcing and benchmarking. The ATSB agreed with all four recommendations and has commenced implementation.
Governance
3.10
The ANAO acknowledged that the ATSB has been active in improving investigation efficiencies through improved governance structures and processes.
3.11
In 2017, the ATSB initiated two programs aimed at increasing the efficiency of investigations: The Evolution Program and the Back on Track Program. The Evolution Program enables better resource allocation and utilisation to maximise the efficiency and effectiveness of operations by addressing the organisational structure, culture, efficiency, learning and development, talent management, succession planning and leadership of the ATSB.
3.12
The Evolution Program also included the transformation of the agency to a multimodal, teams-based agency which aims to improve the effectiveness of the ATSB’s investigators. This was achieved by enabling investigators to investigate any type of accident, where previously they investigated one mode of transport only.
3.13
The Back on Track program, implemented in mid-2017, targeted the ongoing backlog of open investigations for completion. Significant resources have been allocated to this initiative to clear this backlog and meet newly framed KPIs. After the 40 investigations identified as a priority are finalised, diverted resources will be brought back in line to perform business as usual tasks.
Performance Measurement and Management
3.14
The ATSB has amended KPIs for investigation timelines. A KPI focusing on demand capacity has been introduced which considers the ATSB’s workload at a point in time and how much work can be taken on, realistically, against the resources on hand. The demand capacity KPI projects a decline from the current caseload of around 100 over the next three to four years. The ATSB explained:
That KPI recognised where we were at, which was a caseload of around 100 investigations, and what it's doing is projecting out to understand the resources we have available to get us down to a realistic case load. That is projecting out over the next three to four years, to dropping down to 50 or 60 complex cases on our books at any one time, and that’s being reviewed on a rolling monthly basis.
3.15
The ATSB advised that work is underway to improve data collection and analysis that will help inform assessments of investigation efficiency. This will assist the ATSB with the development of more realistic and effective investigation timeframes.
3.16
Since the audit the ATSB has taken on 17 new transport safety investigators who will be trained, skilled and mentored to maturity within approximately three years. Further, investigator training will now be handled by RMIT University. This change releases resources within the ATSB previously utilised for training new staff. This sits well against the Chief Executive Officer’s description of the ATSB as a ‘doing organisation not a training organisation’. Operations will slow down during this training and ‘getting up to speed phase’ but the efficiency metrics will improve once training has been completed.
Benchmarking
3.17
In order to assist the ANAO to address the audit criteria and in order to learn lessons that may help to inform the future direction of the ATSB’s key performance indicators the ATSB requested assistance from four international multi-modal transport safety investigation entities.
3.18
The entities identified as suitable comparators are from the Asia Pacific region and North American Region.
3.19
At the time of the audit three of the four entities had provided assistance.
3.20
The ANAO undertook additional analysis of publicly available information of three countries to compare the ATSB’s performance against the numbers of commenced, completed and ongoing active investigations, and the number of investigator staff. Compared to Country A, Country B and Country C, the ATSB has commenced more investigations than it has completed. This is being addressed by the Back on Track program.
3.21
The ANAO recommended that the ATSB conduct a benchmarking activity against relevant international comparators to assess relative efficiency. The ATSB agreed with the recommendation, and stated:
The ATSB recognises that benchmarking is a valuable activity and there is significant scope to progress this body of work at an international level with counterpart agencies. Benchmarking is a shared responsibility and no other international transport safety investigation agency has yet conducted such an activity. The ATSB is pleased to lead the way.
Reporting
3.22
The audit found that the ATSB had limited visibility over the workload and resource pool of investigators. The ANAO noted that at the time of the audit the ATSB were developing a tool to provide a holistic view of resource demand. Microsoft Project will be utilised to track and gain a better understanding of the current workload for all operational resources, both investigative and non-investigative. This is likely to address the need identified by the ANAO.
3.23
The Committee heard that as part of the Evolution Program, the ATSB is adopting a project management approach to investigations that will deliver a more interconnected reporting view of caseloads and capacity at individual, team and organisational levels. This, in turn, will support decision making around the allocation and relocation of resources to meet demand.
3.24
The ANAO and ATSB have worked together to develop more suitable key performance indicators for demand/capacity and benchmarking which has subsequently been detailed in the 2019-20 Corporate Plan.
Stakeholder Engagement
3.25
The ATSB has multiple stakeholders in the aviation, marine and rail industries, both domestically and internationally. Engagement with these stakeholders can impact investigation timelines. The ATSB explained:
That report from go to whoah took 19 months to publish and that's because was so complex. It involved an aeroplane manufactured in the States. In this particular case it involved four widows that resided in the States. We had to deal with the overseas agencies—the NTSB, the FAA, the propeller manufacturers of the United States, the engine manufacturers in Canada. And so the collection of evidence, then the analysis of evidence and then the crosschecking of the analysis with those various parties overseas takes some time.
Concluding Comment
3.26
The Committee recognises that several programs and governance structures have been put in place both during and following the audit process. As such, results in relation to timeframes have yet to be fully realised. The Committee acknowledges the efforts of the ATSB regarding benchmarking and the prioritisation of investigative resourcing.
3.27
The Committee commends the ATSB on the benchmarking activities underway and understands the proposed international benchmarking framework paper that will be presented at the next International Transportation Safety Association (ITSA) meeting.