2.1
The Joint Committee of Public Accounts and Audit (JCPAA) inquiry into the Defence 2015-16 Major Projects Report was based on ANAO Audit Report No. 40 (2016-17) 2015-16 Major Projects Report. This inquiry also includes analysis of ANAO Audit Report No. 11 (2016-17) Tiger - Army’s Armed Reconnaissance Helicopter.
2.2
The purpose of the MPR is to report on the performance of selected major Defence equipment acquisition projects (Major Projects), since Second Pass Approval, and associated sustainment activities (where applicable), managed by Defence.
2.3
The Committee is concurrently conducting an inquiry into Defence Sustainment Expenditure which includes consideration of ANAO Audit Report No. 2 (2017-18) Defence’s Management of Materiel Sustainment.
Principles underpinning the Major Projects Report
Qualified audit findings
2.4
A significant issue to the Committee in its review of the Major Projects Report (MPR) was the Auditor-General’s qualified audit finding. Part of the reason for the qualified conclusion was that the ARH Tiger Helicopters (hereafter ‘Tiger’) Project Data Summary Sheets have: ‘not been prepared on the basis of the Guidelines’, as:
Funding for caveats, capability deficiencies and obsolescence issues need to be provided separately to the acquisition project, as this cost is unable to be identified
the Project Maturity Score given to the Tiger project ‘does not accurately or completely represent the project’s maturity as at 30 June 2016’
2.5
Further, the ANAO found the Tiger’s Project Financial Assurance Statement did not address significant caveats, capability deficiencies and obsolescence issues identified when Final Operational Capability (FOC) was declared in April 2016.
2.6
Funding to fix these issues would need to be provided separately to the acquisition project, an amount that is unable to be quantified. Additionally, material inconsistencies were identified in the forecast information for the Project Data Summary Sheets (PDSSs) of the Tiger project, and the LHD Landing Craft.
2.7
The Committee notes the clear difference between the ANAO’s past qualified findings for the MPR and this year’s qualified audit finding. Historic qualifications could be regarded as ‘growing pains’, as the MPR was developed and refined, whereas the findings in this MPR constitute material disagreement between Defence and the ANAO. The Committee considers qualified audit findings are a significant matter for an auditor, and are not entered into lightly – auditors will generally seek further information to clarify the position stated by the auditee and try to find common ground before issuing a qualified finding. Accordingly, the Committee will continue to monitor both the Tiger project, the content of PDSSs and the funding of caveats, capability deficiencies and obsolescence issues in future MPRs.
Transparency across the project life cycle
2.8
Transparency is one of the key principles of the MPR. The MPR delivers transparency to Parliament and the Australian public, and the Committee will seek to maintain and enhance transparency where possible.
2.9
The Committee is concerned about the declaration of FOC resulting in a project being removed from the MPR and concluded in the sustainment phase. The Committee takes project transparency seriously, and emphasises the importance of project transparency across the entire project life cycle. Accordingly, the Committee was pleased to hear Defence and the ANAO confirm that the Tiger project will remain part of the next MPR, with the ANAO also confirming examination of the remediation of issues caveated at FOC. The Committee will continue to monitor the completion of the Tiger project through the MPR process.
Guidelines around project removal
2.10
The Committee notes possible differences between Defence’s and the ANAO’s interpretation of the MPR guidelines for project removal. However, the Committee believes the guidelines for project removal are clear, and it reserves the right, as per the guidelines, to decide when a project should exit the MPR. For clarity, if either the ANAO or Defence wish to amend the guidelines, they should follow the appropriate protocol and jointly agree on a form of words to propose to the Committee.
Implementation of Committee recommendations
First Principles Review
2.11
The Committee acknowledges the complexity around the implementation of First Principles Review recommendations. The Committee notes that Defence is completing implementation and will provide adequate reporting as soon as possible.
Capability performance reporting
2.12
In 2014 the Committee recommended a more objective method be applied to capability performance reporting, noting Defence had no systems in place to track inputs to capability. The Committee followed up this matter on notice, with Defence replying that it had disagreed with the Committee recommendation as at the time Defence had limited ability to ‘have full visibility of all Fundamental Inputs to Capability’ and ‘due to the intrinsic difficulty in objectively measuring the capability provided compared to that originally sought.’
2.13
The ANAO noted that ‘Defence has not developed this measure, reporting that the difficulties relate to the varied nature of projects being managed, the inherent subjectivity of the content, and the lack of a system that tracks at a sufficient level of detail the progress of inputs to capability.’
2.14
The Committee emphasises that more work needs to be done to improve the reporting of expected capability performance.
Capability performance analysis
2.15
The methodology used for capability performance analysis was also of interest to the Committee, which considered current Defence reporting, and two worked examples of another methodology used by the ANAO to provide another perspective on capability performance analysis.
2.16
Considering the two worked examples, and examining the current reporting, while Defence may ‘expect’ that the AWD Ships elements will be delivered, it is difficult to state with certainty that 100 per cent capability delivery performance will be achieved.
2.17
The ANAO analysis provided more clarity on reporting, advising that no ships have been delivered, initial and final sustainment arrangements had not been made, and crew training had not been completed.
2.18
Considering Bushmaster Vehicles, the worked example provided by the ANAO indicates a match in Defence and ANAO’s analysis. All Production Period 1 vehicles were delivered, and as at 30 June 2016, 1,008 of 1,016 vehicles had been delivered.
2.19
While the method deployed by the ANAO may not suit all projects, improving the way capability development progress is defined and reported will reduce the likelihood of projects at vastly different developmental stages reporting the same capability delivery.
2.20
To ensure consistency with project level risk information and to improve reliability, the Committee recommends that the Department of Defence review the procedure for the development of expected capability estimates for future Major Projects Reports. The outcome of this review should be provided to the Committee within six months of the tabling of this report. Further, the Committee requests that Defence provide a progress report within three months of the tabling of this report.
Project Maturity Scores
2.21
Project Maturity Scores are present in the MPR to provide ‘quantification, in a simple and communicable manner, of the relative maturity of capital investment projects as they progress through the capability development and acquisition life cycle.’ They have been a topic of ongoing discussion for the last four years, with the Committee maintaining a watching brief on project maturity scores and related issues.
2.22
At the public hearing, the Committee discussed Project Maturity Scores in further detail, and noted that there was agreement that it was worthwhile for Defence and the ANAO to attempt to come to a consensus on project maturity scores, starting from the basis of what the objective of the score is, and then moving onward from there.
2.23
The Committee was disappointed to find that there had been little progress in the implementation of the recommendation it made in 2016 regarding Project Maturity Scores and encourages Defence, its contractor, and the ANAO to come together to progress improvement in this area.
2.24
While acknowledging the complexity of the issue, the Committee recommends that Defence and the ANAO should work to come to a consensus on Project Maturity Scores, reiterating the recommendation the Committee made in JCPAA report 458 in 2016.
2.25
The Committee recommends that the Department of Defence:
commence discussions with the Australian National Audit Office on updating Project Maturity Scores, with a view to advising the Committee on a way forward prior to the first sitting week of 2018
provide the Committee with an update on the progress of this dialogue within six months of the tabling of this report.
2.26
The Committee queried Defence about similar reporting regimes in other jurisdictions, and sought further feedback on these models. Defence noted that the United Kingdom used a similar methodology in its reporting regime, and stated that it would source further information on the guidelines behind the methodology.
2.27
The Committee recommends that the Department of Defence examine the guidelines and methodology used by the United Kingdom’s Ministry of Defence to report on major projects, and report back to the Committee within six months of the tabling of this report. Defence’s report back to the Committee should consider whether any features of the United Kingdom’s guidelines and methodology would enhance major projects reporting in Australia. If Defence finds the United Kingdom’s guidelines and methodology unsuitable, the Committee requests that Defence provide reasons why they reached this conclusion.
Schedule slippage
2.28
The ANAO has monitored schedule slippage (the amount of time lost over the course of a project) across multiple MPRs. It has found that total schedule slippage has trended downward over the last three years.
2.29
The Committee notes that military off the shelf (MOTS) equipment was less likely to experience slippage than projects categorised as ‘Developmental’.
2.30
Further, the Committee notes that Tiger was sold as an off-the-shelf project, but was more like a developmental project, and that wrongly classifying it had led to unrealistic expectations around the timeliness of capability delivery. Defence replied that this was one of the ‘fundamental lessons’ Defence learnt from Tiger.
2.31
As to whether Defence used a clear or official definition of Australianised MOTS compared to MOTS, or off-the shelf, Defence stated that its Capability Development Manual contained more detail and information on this has been included in an answer to a question on notice from the Committee.
Risk management and use of spreadsheets
2.32
The Committee agrees with both Defence and the ANAO on the issues identified in the MPR and at the public hearing in ensuring successful risk management. Defence noted that the ‘intellectual rigor’ applied to risk management is important, and the ANAO noted that tools featuring change and version control are important to preserve the accuracy of risk assessment and support that intellectual rigor.
2.33
The Committee supports Defence’s commitment to ‘consistent and transparent’ risk management, and notes the ANAO’s interest in bringing Defence’s risk management framework into the scope of its audit work. If the ANAO decides that bringing risk management into scope will improve Defence’s processes, the Committee would welcome such an approach.
Cost Per Flying Hour
2.34
The Committee notes the different calculations in cost per flying hour between the MPR and ANAO Audit No. 11 (2016-17) on the Tiger helicopters. The Committee agrees with the ANAO that Defence should use a ‘defined, consistent methodology’ to calculate the cost per flying hour for fixed wing aircraft and helicopters.
2.35
The Committee recommends that the Department of Defence devise a single methodology to calculate the cost per flying hour for fixed wing aircraft and helicopters, and use this single methodology in all internal and public reporting and in responses to queries from the Australian National Audit Office. Defence should provide this information to the Committee prior to the first sitting week of 2018.
Individual projects
ARH Tiger Helicopters
2.36
The Committee notes the continuing issues with the ARH Tiger Helicopters and will continue to monitor this major project through the MPR process.
Collins Class Submarine Reliability and Sustainability
2.37
The Committee is satisfied with Defence’s response regarding the rectified fault on the Special Forces Exit and Re-Entry on HMAS Dechaineaux.
LHD Landing Craft
2.38
The Committee notes the disagreement between Defence and the ANAO surrounding the materiel capability performance of the LHD Landing Craft. The Committee will continue to monitor this issue through future MPRs and any relevant performance audits.