3.1
As part of its review, the Joint Committee of Public Accounts and Audit (JCPAA) held one public hearing in Canberra on 23 March 2018 with representatives from the Australian National Audit Office (ANAO) and the Department of Defence (Defence). A list of witnesses is at Appendix C. A list of submissions to the inquiry is provided at Appendix B.
3.2
The JCPAA inquiry focused on the following key issues:
Principles underpinning the Major Projects Report (MPR)
Implementation of Committee recommendations
Materiel Capability Delivery Performance
Risk management and the use of spreadsheets
Projects of Concern and Projects of Interest
Project team maintenance costs and liquidated damages
Principles underpinning the Major Projects Report
Qualified Audit Findings
3.3
The Auditor-General made a qualified audit finding in the MPR. The Auditor-General outlines the basis for the qualified conclusion in his letter to the Presiding Officers. In part, it states that the ARH Tiger Helicopters (Tiger) Project Data Summary Sheets (PDSS) does not:
Accurately or completely represent the project’s maturity at 30 June 2017. This represents a departure from the Guidelines. The lifting of two of the nine caveats in July 2017 was a result of events occurring prior to 30 June 2017, and, accordingly, my conclusion has had regard to the caveats being lifted.
…a material inconsistency has been identified in the forecast information. Section 4.1 in the ARH Tiger Helicopters PDSS reports that materiel capability delivery performance is at 100 per cent, indicating that materiel capability delivery performance has been met. Rate of effort continues to be lower than planned, and expert analysis commissioned by Defence in April 2016 indicates that the program will remain incapable of fully meeting expectations relating to reliability, availability, maintainability, and rate of effort.
3.5
The inquiry asked about the significance of a qualified audit finding, with the ANAO replying that the Tiger PDSS ‘doesn’t fairly represent the position with respect to that project’, and ‘It’s a substantial statement, because we’re saying that it’s not materially correct.’
3.6
The inquiry noted the official statement by the Secretary of Defence did not explicitly address the Auditor-General’s qualified finding. Defence stated that a reference to a ‘difference of opinion in relation to the guidelines around the reporting’ constituted a response, with the ANAO replying:
When we undertake any form of audit assurance work, we can’t allow our opinion to be based upon technicalities of definition. We’re required under the standards to audit the substantive nature of an issue, not technicalities of definitions. There may be grounds for a difference of opinion about what the guidelines say when a project ends, which is the difference between Defence and us, but the substantive issue is not about the technicality of that; it’s about making sure Parliament isn’t misled by the information provided. I can’t sign off on a report that’s technically correct but misleading as being accurate.
3.7
Defence stated that it was examining improving project maturity scores to ensure transparency, and stated ‘for the level that goes into [the MPR], to end up with a qualified [audit finding] is not in anybody’s interest’, and that ‘no organisation likes a qualified report from their auditor.’
Implementation of Committee recommendations
3.8
Defence has committed to implementing the recommendations made by the Committee in Report 468: Defence Major Projects Report (2015-16): Inquiry based on Auditor-General’s report 40 (2016-17), by updating the Committee on the progress of implementation.
3.9
Recommendation 1 called for Defence to review the procedure for the development of expected capability estimates for future MPRs, with Defence stating it would conduct a baseline validation activity to drive greater consistency in schedule reporting.
3.10
Recommendation 2 called for Defence and the ANAO to work together to update Project Maturity scores. This issue is examined in more detail below.
3.11
Recommendation 3 called for Defence to examine major projects reporting in the United Kingdom, with Defence reporting on changes in this area.
3.12
Recommendation 4 sought a single methodology for determining the cost per flying hour for fixed wing aircraft and helicopters, with Defence advising that revised definitions would be included in an updated version of its internal financial manual.
Project Maturity Scores
3.13
The inquiry noted Defence’s undertaking in its submission to critically review the Project Maturity Score concept in conjunction with the ANAO, asking whether a timeframe for this work had been set. Defence stated that work was currently being undertaken. At the hearing, the Committee sought further assurance on this work, with Defence stating that it was ‘currently working with the ANAO in relation to options around recalibrating the maturity scores and our approach,’ and that progress reporting would be included in the 2018-19 Major Projects Report. The ANAO stated that ‘there’ll be greater changes necessary to systems, procedures, training et cetera. That will have a longer time frame.’
Materiel Capability Delivery Performance
3.14
The Committee considered materiel capability delivery performances (displayed as pie charts in the MPR), noting the ARH Tiger Helicopters chart displayed 100 per cent achieved even though the project had two caveats still in effect and was the reason for a qualified audit finding. Defence replied that the chart was developed against the materiel acquisition agreement, and that adopting a more holistic assessment of the capability may require the discussion of matters under security classification.
Schedule slippage
3.15
The Committee examined schedule slippage and the ANAO advised that it was ‘a bit more flat lining than anything else’. Defence advised that the vast majority of current slippage was as a result of the LHD Landing Craft and Civilian Military Air Traffic Control System program (CMATS). The LHD Landing Craft project is discussed in more detail below. Defence stated that the CMATS project had been delayed by two years but that the project was ‘now in contract’. Further, a real cost increase on the CMATS project approved by the Government was $243 million.
Risk management and the use of spreadsheets
3.16
The inquiry discussed the ANAO’s finding that some areas of Defence continued to use spreadsheets as a risk management tool, a practice identified by the ANAO as a: ‘high risk approach’. Spreadsheets lack formalised change/version control and reporting, thereby increasing the risk of error’.
3.17
Defence stated that the Capability Acquisition and Sustainment Group (CASG) had developed a standardised risk approach at the group level consistent with that used at Defence, and advanced the same argument used in the last MPR review that ‘the tool used to assess risk is not the determinant of the outcome.’
3.18
The ANAO agreed that the most important input into a risk framework was the content, not the system, but reemphasised the lack of controls that surround spreadsheets with respect to version control.
3.19
The inquiry discussed Defence’s risk identification assessment communication and management and its relationship to Defence’s organisational culture. Defence replied that a top-down approach had been taken to examine how risk is identified, communicated, and culturally discussed as a group. The Committee asked whether Defence had a culture of sharing good news and not sharing bad news in risk assessments, with Defence replying that there was not a culture, but that there had been ‘small, isolated examples’ where internal reviews had found that risks hadn’t been identified ‘in totality’. Defence also added that ‘…the true skill here is to pick the right risks that management needs to pay attention to and mitigate as you go…’
3.20
The inquiry discussed the finding of an external consultant that only four of eight CASG domains actually documented a risk management approach, with Defence replying that all of the domains now had a comprehensive risk approach, the last of which was finished recently. The ANAO stated that it would examine these developments in the next MPR, ‘because risk is something we keep an eye on’.
3.21
The inquiry also discussed project classifications such as Military Off The Shelf (MOTS), Civilian Off The Shelf (COTS), and Developmental, asking whether Defence would instead rely more on descriptions or risk, rather than trying to label a project. Defence replied that ‘those terms are not particularly useful in and of themselves’, and ‘[these terms] have the potential to lull people into a false sense of security about the real complexity of the project or the acquisition.’
3.22
The inquiry asked whether provision of project risk registers would give the Parliament a better appreciation of an individual project’s risks. Defence replied that each PDSS in the MPR highlighted major risks for each project, and this was assessed by the ANAO.
Projects of Concern and Projects of Interest
3.23
The Inquiry noted references to ‘Projects of Interest’, and asked for more information on this classification compared to the ‘Projects of Concern’ list. Mr Gillis, Deputy Secretary, CASG advised that ‘Projects of Interest’ 'get much closer attention from me and from the capability manager, and they are the potential ones that could end up on the projects of concern list'.
3.24
A ‘Project of Concern’ requires involvement and decisions of the two Defence Ministers. Mr Gillis advised that he provides advice to the Ministers who would 'make a decision about whether a project goes on or off' the ‘Projects of Concern’ list. Further:
As an ex-industry CEO, being on a project of concern is a really, really bad thing. You do not want to be, and you want to get off it. But what it means is there is direct ministerial engagement to ensure that those companies are engaging fully, to make sure that they’re remediating their projects and getting off.
Project team maintenance costs and liquidated damages
3.26
Noting the length of time the Tiger project had been included in the MPR process, the inquiry asked about the costs of keeping project teams together for long periods of time and the costs of keeping current capability active in the absence of new capability. Defence replied that the ‘Commonwealth actually applies liquidated damages to the companies. That goes back into consolidated revenue, because it doesn’t go directly into the project team. Therefore, those liquidated damages are at the pre-agreed quantums that we negotiate.’
Individual Projects
ARH Tiger Helicopters
3.27
The Committee asked about progress on the ARH Tiger Helicopters project, with Defence replying that the ANAO report was ‘an accurate reflection of where Tiger is in its life cycle’, and that the project continued to progress. Two caveats remained open, relating to communications and mission planning (identified as ‘a fairly extensive technical issue’), and availability and rate of effort (‘which has shown improvement’). Further, Defence stated of the caveats: ‘we hope to have them resolved in the next 18 months or so’, and that industry was under contractual obligation on a performance-based contract to resolve the caveats.
LHD Landing Craft
3.28
The Committee asked about progress of the LHD Landing Craft, noting delayed 2017 trials had been rescheduled for the second quarter of 2018. Defence replied that the trial program was continuing and a major exercise was planned for the second quarter of 2018. Defence added that it had also done work with the Mexeflotes powered carrying system to ‘carry a tank and landing crafts very safely in a high seas state.’
…last year we had a qualification in the report which related to LHDs and which went to this issue. We don’t this year, because the evidence that the department was able to provide us with this year was consistent with the statement the department has just made about the amount of capability that that particular function was meant to deliver.
12 September 2018