Auditor-General Report No. 19 2020–21
Introduction
1.1
The Major Projects Report (MPR) is prepared each year by the Department of Defence (Defence) and the Australian National Audit Office (ANAO). The MPR contains detailed information on a selection of large‑scale Defence materiel acquisition projects.
1.2
Expenditure of taxpayer funds on Defence capability acquisition projects is substantial. As at 30 June 2020 Defence’s Capability Acquisition and Sustainment Group (CASG) managed projects worth a total of $130 billion, with expenditure in 2019-20 of $8.7 billion.
1.3
The objective of the MPR is to improve the accountability and transparency of this expenditure through a process of review and analysis of a selection of the most significant projects, for the benefit of the Parliament, stakeholders, and the public. The 2019-20 MPR is the thirteenth report of its kind.
1.4
The MPR is prepared by Defence and the ANAO in line with the Joint Committee of Public Accounts and Audit 2019‑20 Major Projects Report Guidelines (the Guidelines) as approved by the Joint Committee of Public Accounts and Audit (JCPAA, the Committee). The MPR is automatically referred to the Committee in line with its statutory obligation to examine all Auditor-General reports tabled in the Parliament.
1.5
The MPR consists of detailed project information in the form of Project Data Summary Sheets (PDSSs) prepared by Defence according to the Guidelines, commentary on that information by the Secretary of Defence, and assurance and analysis of that information by the ANAO.
1.6
The Committee adopted the Inquiry into the Defence Major Projects Report (2019-20) on 23 June 2021. A public hearing for the inquiry was held on 15 September 2021 and the inquiry received 3 submissions and 4 supplementary submissions.
1.8
This report considers the Auditor-General’s findings in the 2019-20 MPR, including the ANAO’s analysis of:
project schedule performance, and
project capability performance.
1.9
The report then considers project management and governance issues arising from the inquiry, including:
the early impacts of the COVID-19 pandemic
Projects of Concern and Projects of Interest
the declaration of milestones with caveats and deficiencies,
the discontinuation of Project Directives, and
The 2019-20 Major Projects Report
1.10
There were 25 projects included in the 2019-20 MPR. Combined, the projects had a budget of nearly $79 billion. In-year expenditure on Major Projects was $5.7 billion against a budget of $6.1 billion.
1.11
The value of projects included in the MPR has increased steadily in recent years. In 2017-18 MPR projects had a combined budget of $59.4 billion and the 26 Major Projects in the 2018-19 report had a budget of $64.1 billion. As Defence noted in its submission, this is consistent with a 10‑year trend of increases in the number, complexity and scale of major capability acquisition projects being managed by the department.
1.12
Six projects included in the 2018-19 MPR were removed from the report because they achieved Final Operational Capability (FOC) or were considered to have a low risk in achieving final capabilities. They were:
SEA 1448 Phase 2B – ANZAC Anti-Ship Missile Defence
JP 2072 Phase 2A – Battlespace Communications System (Land)
JP 2048 Phase 4A/4B – Amphibious Ships (LHD) (LHD Ships)
JP 2048 Phase 3 – Amphibious Watercraft Replacement (LHD Landing Craft)
AIR 7403 Phase 3 – Additional KC-30A Multi-role Tanker Transport
JP 9000 Phase 7 – Helicopter Aircrew Training System
1.13
The five new projects which replaced them in the 2019-20 MPR were:
SEA 1000 Phase 1B – Future Submarine Design Acquisition
SEA 5000 Phase 1 – Future Frigates
AIR 7000 Phase 1B – MQ-4C Triton Remotely Piloted Aircraft System
LAND 400 Phase 2 – Mounted Combat Reconnaissance Capability
Land 200 Tranche 2 – Battlefield Command System.
1.14
During the course of this inquiry, the Government announced the termination of the Future Submarines project and the early retirement of the MRH90 ‘Taipan’ helicopter fleet. It is expected that the replacement submarine project will be included in the MPR in due course.
The Auditor-General’s findings
1.15
The Auditor-General is required to provide a ‘limited assurance review’ of the data provided by Defence in its PDSSs. A limited assurance review provides a lower level of certainty than that provided by an ANAO performance audit: whereas performance audits provide a ‘reasonable but not absolute’ level of assurance, a limited assurance review expresses its conclusion in negative terms.
1.16
In line with this obligation, the Auditor-General found that:
Based on the procedures I have performed and the evidence I have obtained, nothing has come to my attention that causes me to believe that the information in the 25 Project Data Summary Sheets in Part 3 (PDSSs) and the Statement by the Secretary of Defence, excluding the forecast information, has not been prepared in all material respects in accordance with the 2019–20 Major Projects Report Guidelines (the Guidelines), as endorsed by the Joint Committee of Public Accounts and Audit.
1.17
This is the third consecutive MPR to have an unqualified audit finding from the Auditor-General. The last qualified finding was in 2016-17, when the Auditor-General identified issues with the accuracy and consistency of the information in the PDSS for the ARH Tiger project. That project has subsequently exited the MPR.
1.18
The ANAO noted that some information in the PDSSs is excluded from the scope of the Auditor-General’s Independent Assurance Report:
Forecast information in each PDSS relating to capability forecasts, forecast dates, or project risks and issues, is out of scope. The forecast information has not been included in the scope of the ANAO’s assurance engagement due to the lack of Defence systems from which to provide complete and accurate evidence, in a sufficiently timely manner to complete the review.
1.19
However, the Auditor-General does take any material inconsistencies in this forecast information into account when forming the review’s conclusion. As such, the MPR may still include analysis of Defence’s risk management and capability forecasting.
Cost performance
1.20
In-year expenditure in 2019-20 for the 25 Major Projects was lower than forecast. The ANAO noted that:
In total, actual expenditure for the 25 projects at 30 June 2020 was $5651.9 million. This is compared against an initial Portfolio Budget Statements (PBS) forecast expenditure of $6621.6 million, a mid-year Portfolio Additional Estimates Statements (PAES) forecast of $6171.7 million, and a final forecast of $6110.2 million (Final Plan, approved during May 2020).
1.21
The in-year variance was caused by a number of different factors. These included delays against contracted activities, expenditure refinements based on new information from the US-based Joint Project Office, reprogramming of activities, and the initial impacts of the COVID-19 pandemic, among others.
1.22
Overall budget performance showed a 30.7 per cent increase in approved budget for the 25 Major Projects since Second Pass Approval, representing an increase of $24.2 billion. Of this increase, $14.1 billion arose from project scope changes (for example, the purchase of an extra 58 Joint Strike Fighters at Stage 2 Second Pass Approval), all of which was incurred prior to the period covered by this MPR. Price indexation and exchange rate variations accounted for an additional $7.3 billion.
1.23
All projects reported that they could continue to operate within their total approved budget, which totals $78.7 billion for all 25 projects. One project (MRH‑90 Helicopters) drew on contingency funds in 2019‑20 to address supportability and performance risks. Access to these funds was approved in previous financial years.
Schedule performance
1.24
Schedule performance continued to be a significant issue for major projects:
Delivering Major Projects on schedule continues to present challenges for Defence, affecting when the capability is made available for operational release and deployment by the Australian Defence Force, as well as the cost of delivery.
1.25
Major Projects reported a combined schedule slippage of 68 months during 2019-20. The ANAO noted that slippage tends to be highest for projects with the most complex ‘developmental’ content, while comparatively simpler ‘Military Off The Shelf’ (MOTS) or ‘Australianised Military Off the Shelf’ purchases experience less slippage:
A focus on MOTS acquisitions has assisted in reducing schedule slippage. Prima facie, the more developmental in nature a project is, the more likely it will result in a greater degree of project slippage.
1.26
The MQ-4C Triton Remotely Piloted Aircraft System, being developed in a Cooperative Program partnership with the United States Navy, accounted for 43 out of the 68 months of in-year schedule slippage. The delay arose from a US Government decision for a two-year pause in production funding for the Triton project.
1.27
Total slippage for the 25 Major Projects was 507 months as at 30 June 2020, which represents a 21 per cent increase since the Second Pass Approval stage. The total of 507 months slippage is 144 months lower when compared to the 2018-19 report. This is as a result of projects with a combined 218 months slippage exiting this MPR.
1.28
The Defence submission argued that:
The 2019-20 Major Projects Report reflects an improvement in the average schedule slippage of the projects and Defence will continue to work with Industry to improve schedule certainty and achievement. By their nature the projects pursue an aggressive schedule in order to provide the ADF with technologically advanced equipment and support.
1.29
Although schedule slippage against FOC has the potential to impact capability, Defence pointed out that programs often provide real capability to the Defence Forces before reaching that milestone:
For example, the MRH-90 is reporting 89 months delay to FOC, however it is operational and providing extensive support locally for natural disaster relief as well as support to Pacific Island nations. The P-8A Poseidon maritime patrol aircraft, which is reporting a 29 month delay to FOC has been deployed on multiple operational deployments and conducted reconnaissance flights over fire-affected areas in New South Wales, Victoria and South Australia as part of Operation Bushfire Assist.
1.30
At the same time, the significant delay reported in the MQ-4C Triton project arising from the project partner deciding to delay the program provides some warning about the risk associated with diminished control over joint developments with overseas partners. The impact of the delay in availability of the capability, if any, has not been made clear.
Capability performance
1.31
The third component of project performance contained in the MPR relates to the analysis of capability performance. Reporting on a project’s likelihood of meeting its capability requirements is a difficult endeavour, since it involves the making of assumptions and predictions and is therefore inherently subjective.
1.32
In previous MPR reviews, the JCPAA has urged Defence to try and find a more robust and objective measure for capability performance. The ANAO again noted in this MPR that Defence’s capability forecasts are subjective and that they ‘may be optimistic’.
1.33
The ANAO submission noted that:
Following a JCPAA recommendation made in May 2016, Defence is yet to implement a system of materiel capability delivery performance/scope reporting, with a robust methodology applicable to materiel acquisition.
1.34
An example of the subjectivity and occasional optimism involved in capability forecasting is provided by the Battlefield Airlifter project. The project involved the acquisition of 10 US Joint Cargo Aircraft variant Leonardo C-27J Spartan fixed-wing aircraft to deliver ‘enhanced intra-theatre and regional airlift capability’.
1.35
The project reported a ‘100 per cent Green’ capability prediction on its inclusion in the MPR in 2013-14 despite reporting major risks caused by the US Government’s decision to divest its C-27J fleet. The ANAO noted in the MPR that:
These risks have continued to affect the project, with a mature training system and a number of baseline capability requirements now not expected to be delivered until after FOC. These capability issues were reported in the PDSS Pie Chart for the first time in 2018–19, indicating that the earlier level of confidence in the ability to achieve the required capability may have been overly optimistic.
1.36
The risks have subsequently been realised. At the Committee’s public hearing Defence advised that the role of the C-27J aircraft has been ‘redefined’ as a result of an inability to achieve the necessary capability and it will now operate in a Humanitarian and Disaster Relief role. Defence advised that its initially planned use for intra-theatre operations is still possible, but it would now carry higher risk to Australian personnel as a result of the unmet capabilities.
1.37
Noting difficulties of this kind in its capability predictions, Defence reported in this MPR that 24 of the 25 Major Projects are expected to deliver all of their key capability requirements. Five projects report capability that is ‘under threat’ but the risk of which is assessed as ‘manageable’. Only one project, the Air Warfare Destroyer, reported a capability requirement that it will not meet. The Air Warfare Destroyer initially reported this in the 2018‑19 MPR.
Project management and governance issues
1.38
This section considers a number of project management and governance issues arising from the inquiry. These are:
the impact of the COVID-19 pandemic
projects of Concern and Projects of Interest
Defence’s risk management policies and procedures
Defence’s use of caveats and deficiencies
Project Directives and the Capability Life Cycle Management Tool, and
COVID-19 impacts
1.39
This MPR covers the period in which the first impacts of COVID-19 affected Defence and its local and international partners. In the first half of 2020 Australia and many other parts of the world experienced their first wave of COVID-19 infections and the subsequent public health measures, including lockdowns and travel restrictions. Although many projects in this MPR were affected by COVID‑19 to some degree, it is expected that the pandemic’s impact on Major Projects will be more pronounced in future MPRs.
1.40
In its analysis of the MPR, the ANAO noted that Defence had paused Independent Assurance Reviews (IARs) in June 2020 as a result of the pandemic. IARs are an important project management mechanism. According to the ANAO:
The Defence Independent Assurance Review (IAR) process provides the Defence Senior Executive with assurance that projects and products will deliver approved objectives and are prepared to progress to the next stage of activity. These management initiated reviews consider a project’s status while sufficient time remains for corrective action to be implemented. IARs are intended to commence at project initiation and are usually conducted on an annual basis through to FOC. They are an important input to key acquisition and sustainment decision points or milestones.
1.41
Defence advised the Committee that the pause in IARs resulted from a temporary reassignment of staff to assist with the department’s COVID-19 response, and that IARs have subsequently resumed.
1.42
The pandemic’s impact on project cost performance was mixed. In the early stages of the pandemic, Defence instituted accelerated payment terms to defence primes to assist Australian defence industry partners:
Defence worked with industry to rapidly process invoice payments for work undertaken, and for the period of the 2019-20 Major Projects Report, this was 100,000 invoices to a value of $8.0 billion. Despite the difficulties of COVID-19, Defence and industry have continued to equip and sustain the Australian Defence Force through the pandemic.
1.43
One project, the Joint Strike Fighter, reported a year-end overspend of 2.9 per cent as a result of accelerated payment terms.
1.44
Other projects experienced underspends as a result of delays caused by the pandemic. The ‘Hawkei’ Light Protected Mobility Vehicle, for example, underspent by approximately $19 million during the year because of delays in contract negotiations, and its Initial Materiel Release (IMR) milestone was declared with caveats related to the pandemic.
1.45
Schedule impacts were more commonly reported as a result of work restrictions, lockdowns, and limits on the movement of people. Defence reported the following for the period covered by this MPR:
The full COVID-19 impacts on Defence’s contracts are still being assessed under the evolving COVID- 19 circumstances overseas. For those projects impacted, current delay is in the order of three to six months. For each contract affected by COVID-19, the Contractor will generate a Recovery Plan which will set out how they propose to address the COVID-19 impacts on the contract. These Recovery Plans will, in conjunction with any necessary evidence, be used to inform Defence about any contract changes that will need to be effected.
1.46
Projects also reported increased risks arising from the pandemic which may be realised in future MPRs. For example, the Pacific Patrol Boat Replacement project reported risks arising from Pacific nation crews unable to travel to Australia for training. Similarly, the Overlander Medium and Heavy Vehicle project reported risks to project milestones due to potential impacts on supply chains, cancellation of events, and suspension of training, among others. The effects of Defence’s pandemic mitigation efforts on risks such as these will undoubtedly figure in future MPRs.
Projects of Concern and Projects of Interest
1.47
The Projects of Concern process is ‘intended to focus the attention of the highest levels of government, Defence, and industry on remediating problem projects’. In its annual report, Defence describes the Projects of Concern regime as a:
… process for managing the remediation of underperforming projects. This is done by implementing an agreed plan to resolve significant commercial, technical, cost and/or schedule difficulties and increasing senior management and ministerial oversight.
1.48
The MRH90 ‘Taipan’ helicopter is the only project in this MPR on the Projects of Concern list. It has been on the list since 2011. In December 2021 the Defence Minister announced that the MRH90 helicopter fleet would be retired a decade earlier than planned and replaced by a fleet of UH-60 Black Hawk helicopters purchased from the United States Government.
1.49
Projects of Interest are defined by Defence as:
Projects (and products) showing heightened risks in the areas of cost, scope, schedule, capability, commercial strategy and/or other issues are monitored through a variety of sources. Consultation with senior stakeholders occurs before determining a Project of Interest. Once listed, reporting requirements are increased with a more detailed summary of issues, along with proposed remediation strategies to get the project/product back on track.
1.50
Nine projects in this MPR were classified as Projects of Interest at 30 June 2020. They were:
‘Hawkei’ Light Protected Mobility Vehicle
Replacement Replenishment Ships (Maritime Operational Support Capability)
MQ-4C Triton Remotely Piloted Aircraft System
Civil Military Air Traffic Management System (CMATS)
Battlefield Command System, and
1.51
These projects were placed on the Projects of Interest list for reasons including COVID-19 impacts, their size and complexity, project risks, schedule risks or issues, and rectification issues, among others.
1.52
The ANAO conducted a performance audit of the Projects of Concern process in 2019, finding that Defence was ‘not able to demonstrate the effectiveness of its regime in managing the recovery of underperforming projects’, and that Defence had been ‘inconsistent’ in applying the process.
1.53
The report made two recommendations: first, that Defence introduce a consistent process for managing entry to and exit from the Projects of Concern list, and second, that Defence evaluate its Projects of Concern regime. As discussed in the 2018-19 MPR report, Defence agreed to both recommendations but expressed disagreement with the reasoning behind the ANAO’s conclusions and also with statements that, in Defence’s view, implied that the department was reluctant to place projects on the Projects of Concern list to preserve positive relationships with industry.
1.54
Defence advised that it had closed both of these recommendations in July 2020, noting that CASG had evaluated the Projects of Concern regime and had put in place effective assurance mechanisms.
1.55
However, the ANAO published an audit report in April 2021 on Defence’s implementation of parliamentary committee and ANAO recommendations, which found that Defence had failed to implement the two recommendations made with respect to Projects of Concern. The audit found that ‘there is no evidence that Defence established a clear basis or criteria to ensure a consistent approach to entry to and exit from the Projects of Concern or Projects of Interest lists’. At the Committee ‘s public hearing the ANAO advised that:
We recently published a performance audit report on Defence's implementation of both the parliamentary committee recommendations and the ANAO recommendations. One series of audit report recommendations that was reviewed in the context of that more recent audit report went to our audit of projects of concern. We basically found that in respect of the two recommendations that we made in the audit that we judged they had not been implemented by the department … our assessment to date has been that neither recommendation has been implemented.
1.56
Defence told the ANAO in March 2021 that:
For reason [sic] of structural and technical complexity, the nature of commercial arrangements, integration and interdependencies designing hard criteria or 'triggers' for entry as a Project of Concern would be absurd. Accommodating such a multiplicity of factors and variables would be experimental. There would be some binominal variables (yes/no) but the categorical variables could extend into the hundreds. This would not deliver a value for money outcome for Defence projects or sustainment. Thus, the IAR process and focussing questions based on project management standards is the best practice.
1.57
Defence advised the Committee that:
Entry and exit processes have been formalised through policy, in line with Recommendation 1 of the ANAO’s performance audit of Projects of Concern … Independent Assurance Reviews continue to routinely consider whether performance issues warrant heightened attention as either a Project/Product of Interest/Concern and will make a recommendation.
Risk management
1.58
Major Project risks and issues identified in PDSSs are excluded from the scope of the Auditor-General’s limited assurance review. However, the management of risk within Defence has been a focus of the ANAO and the Committee since the MPR’s inception. In line with JCPAA recommendations, a risk management reform program has been underway in Defence since May 2017. Defence advised that the final phase of the program, involving updated risk management policies and toolsets for use by projects, was delivered in November 2020.
1.59
A key object of Committee concern has been Defence’s transition away from the use of spreadsheets to manage risk to a more robust set of tools. Defence advised in May 2020 that the ‘Predict!’ risk management system will be mandated for use in future, though its adoption in the period covered by this MPR was partial:
The ANAO’s review of the documentation from CASG’s 25 project offices at 30 June indicates that:
nine utilise spreadsheets as their primary risk management tool; nine utilise Predict!;
three (Maritime Comms, Combat Recon. Vehicle and Future Subs) utilise both Microsoft Excel and Predict!;
two (Joint Strike Fighter and CMATS) utilise a bespoke SharePoint based tool;
one (Night Fighting Equip Repl) utilises the Project Performance Report; and
one (Future Frigates) uses Predict! and Defence’s Capability Lifecycle Management Tool.
1.60
The ANAO also identified several risk management issues relating to projects’ compliance with risk management policy and procedure, and the maintenance and quality control of risk management logs.
1.61
Defence advised that the rollout of a new risk management system has been underway since June 2020:
During 2019-20, the design of the new Capability Acquisition and Sustainment Risk Management System was finalised. The rollout commenced in June 2020 with the release of the Group Policy Package, including the Directive, Risk Management Strategy, Framework and a series of handbooks. Predict! was mandated as the common risk management tool. Transition of projects and products to the latest version of Predict! commenced in October 2020
As at 28 October 2021, 90 projects and 47 products have transitioned. Some projects and products that are soon to close will not be transitioned. However, the remaining projects and products will be transitioned by end of February 2022.
1.62
In relation to the deficiencies in its risk management processes, Defence advised that, upon completion of the rollout, a ‘continuous improvement process’ will commence which aims to ensure that ‘the system evolves with business needs and risks’.
Project maturity scores
1.63
Project Maturity Scores have been used in the MPR for many years as an indicator of a project’s progress towards FOC. The ANAO described maturity scores as follows:
Maturity scores are a composite indicator, cumulatively constructed through the assessment and summation of seven different attributes. The attributes are: Schedule, Cost, Requirement, Technical Understanding, Technical Difficulty, Commercial, and Operations and Support, which are assessed on a scale of one to 10. Comparing the maturity score against its expected life cycle gate benchmark provided internal and external stakeholders with a useful indication of a project’s progress.
1.64
The Committee and the ANAO have had an ongoing interest in the improvement of Project Maturity Scores as a measure of project progress. The Committee’s report on the 2016-17 MPR observed that Defence and the ANAO were making a joint effort to review the scores. The report noted Defence’s undertaking to ‘critically review the Project Maturity Score concept in conjunction with the ANAO’, with the intention of ‘recalibrating the maturity scores and our approach’.
1.65
During the 2018-19 MPR inquiry Defence advised the committee that it had finalised a preliminary review of the framework, having developed a two-stage process in consultation with the ANAO that better reflected the project capability lifecycle. However, internal stakeholder feedback identified several issues with the new framework, which had been retitled as the ‘Project Progress Score’. Defence advised that it would retain the existing Project Maturity Score system until it had resolved the issues with the Project Progress Score framework.
1.66
In this MPR, Defence advised that it no longer intends to use the Project Maturity Score as an input to project performance management. Further, Defence does not intend to include a replacement to the Project Maturity Score framework in future PDSSs. Defence argued that:
Project Maturity Scores and Project Performance Scores are not a reliable indicator of project maturity or performance as scores often remain static for several years as the project progresses through key milestone evens such as Critical Design Review, production (low rate and full rate), delivery, and testing programs.
1.67
Defence said that it instead intends to use a combination of its Monthly Reporting Module (MRM) and its Project Performance Review Information Platform (PRRIP) to manage project progress:
The implementation of PPRIP, supported by the MRM enables rich conversations on a monthly basis between Project Managers and Branch Heads to take place based on contemporary project performance data. This empowers Branch Heads to make informed decisions and to implement corrective actions if projects begin to trend away from or exceed agreed tolerances across a range of metrics.
1.68
Defence advised the Committee that:
[The] Project Performance Review Information Platform (PPRIP) and the Monthly Reporting Module, in contrast to the lagging and less dynamic Project Maturity Scores, provides project performance and associated risks on a monthly basis for cost, schedule and scope and facilitate senior management risk based decision-making and, where necessary, enables appropriate and proportionate interventions to maintain approved project delivery requirements.
1.69
The ANAO has used Project Maturity Scores in its analysis of project schedule performance, by comparing a project’s time elapsed with its Project Maturity Score. In future MPRs, the ANAO will analyse project schedule and cost performance through different comparisons:
Analysis of cost performance, by comparing budget expended with project maturity, and schedule performance, by comparing time elapsed and project maturity, has not been replaced and is not replicated in future MPRs … The analysis in the MPR going forward will focus on comparisons of cost performance and time elapsed, as well as being supplemented by analysis of schedule performance against acquisition type and categorisation.
1.70
The ANAO also noted that since the MRM was introduced in the 2020-21 financial year, it will be reviewed in the next MPR.
Caveats and deficiencies
1.71
Defence is yet to provide a definition of the terms ‘caveat’ or ‘deficiency’ in its internal documentation. However, the ANAO noted that it ‘has observed the use of these terms by Defence to represent exceptions to the achievement of significant milestones declared by Defence such as IMR [Initial Material Release], IOC [Initial Operational Capability], FMR [Final Material Release] and FOC [Final Operational Capability]’.
1.72
The ANAO noted that the 2017-18 MPR pointed to a trend of fewer projects declaring major milestones with caveats. However, the two subsequent MPRs have seen an increase to four projects declaring milestones with caveats in each year. In 2019-20 these were:
P-8A Poseidon Maritime Patrol and Response Aircraft – which expected to declare Material Release 3 with three caveats
‘Overlander’ Medium/Heavy Vehicle – which declared IOC with one caveat
‘Hawkei’ Light Protected Mobility Vehicle – which declared IMR with four caveats, and
Additional Multi-Role Tanker Transport – which declared FOC with caveats and exited the MPR.
1.73
Defence also recorded two ‘concessions’ to the declaration of FOC for the Battle Comm. Sys (Land) 2A project due to some configurations exceeding weight limitations and consequently experiencing capability limitations. The project exited this year’s MPR.
1.74
If used indiscriminately, caveats, deficiencies, concessions, or other qualifying terms have the potential to undermine the usefulness of milestones as a guide to project progress and the overall transparency of the MPR. The absence of specific definitions of caveats, deficiencies, or other qualifiers lay behind the Committee’s recommendation in the 2018-19 MPR report that these terms should be defined.
1.75
The ANAO argued that the use of caveats is not concerning in and of itself:
Declaring milestones with caveats or deficiencies can provide clarity over the completeness of the delivery of materiel capability or milestones. It can also identify impacts on future stages of the project and issues to be managed, and the need to apply specific risk management strategies.
1.76
However, it noted that:
Robust definitions of terms such as ‘caveat’ and ‘deficiency’ would contribute to the transparency of Defence reporting in the MPR and would better inform the ANAO’s assessment of progress against key project milestones. This is particularly where caveats are placed on the Capability Manager's declaration of significant milestones.
1.77
Defence argued that the declaration of milestones with caveats ‘is a useful method to assess the project’s performance in terms of ability to meet capability requirements while transparently acknowledging there may be an element of scope or performance that is outstanding.’
1.78
However, the Department did not address questions relating to the definition of the terms, noting only that they ‘are used where a milestone (Initial Operational Capability, Final Operational Capability, Initial Materiel Release, Final Materiel Release) has been achieved in principle, with outstanding actions to be rectified or mitigated.’
Project directives and the Capability Lifecycle Management Tool
1.79
Project Directives have been foundational documents for Defence materiel acquisition projects. According to the ANAO, Project Directives ‘state the terms of government approval, reflecting the approved scope and timeframes for activities, responsibilities and resources allocated, and key risks and issues’. The ANAO further noted that:
Project Directives have historically been used to inform internal Defence documentation such as Materiel Acquisition Agreements (MAAs) between Capability Acquisition and Sustainment Group (CASG) and the Service Chiefs. Project Directives had previously been described as a key governance document under the Capability Life Cycle, intended to ensure that all parties in Defence are informed of Government decisions.
1.80
In June 2020 Defence updated its Capability Life Cycle Manual, removing references to Project Directives as governance documents. Defence said that Government decisions in relation to Major Projects will instead be recorded in Defence’s Capability Life Cycle Management Tool.
1.81
However, the ANAO noted Defence advice that of the five new projects entering this MPR, three projects – Future Subs, the Combat Recon. Vehicle and the Battlefield Command System – did not have access to Government approvals through the Capability Life Cycle Management Tool. Further, key dates in the Combat Recon. Vehicle did not align with Government approvals.
1.82
The ANAO noted that where projects have not been provided access to Government decisions, they can gain access via an application to the Cabinet Liaison Services area:
[W]here a Project has not been identified as having a need to know, the Project can request access to relevant Cabinet documents via a business case. If MPR projects access Cabinet documents in this way, there is no residual impact.
1.83
The ANAO also noted that its access to internal Defence documentation – required to validate the requirements of projects – is not always possible as a result of these changes. However, the ANAO can source original approval documents from Defence’s Cabinet Liaison Services area or from the Department of the Prime Minister and Cabinet.
1.84
Defence advised the Committee that:
The removal of the requirement for Project Directive [sic] occurred to strengthen the focus on the primary artefacts related to project approvals, being the Ministerial/Cabinet submission and associated approval. Therefore, it is not anticipated there will be an impact on MPR projects as a result of removing the requirement for Project Directives. All Defence staff have access to their Government approval of the project, as appropriate. Therefore, no impact is expected.
Other issues
1.85
This section briefly considers:
ongoing efforts to make the MPR more comprehensible
project staffing costs, and
Defence’s review of the MPR process
1.86
The Committee made recommendations relating to the clarity and accessibility of the MPR in its 2018-19 report. Although these are expected to be implemented in the 2020-21 MPR there are issues of clarity in the MPR worth further mention. There are, for example, two separate glossaries included in this MPR – one each compiled by the ANAO and Defence.
1.87
Even read in combination, however, the two glossaries are not comprehensive. Neither defines PDSSs, for example, and the Defence glossary includes a definition of a Memorandum of Understanding but does not define Defence project management jargon or frequently used acronyms/initialisms in the PDSSs such as LRIP (Low Rate Initial Production) and FRP (Full Rate Production). Lacunae such as these make the MPR less understandable and in doing so decrease its usefulness as a transparency mechanism.
1.88
As a further example, the term ‘C4I’ is used in three PDSSs without definition: the 'Boxer' Combat Recon. Vehicle, the ‘Overlander’ Medium and Heavy Vehicle, and the ‘Hawkei’ Protected Mobility Vehicle. A search through the entire MPR reveals that the term is never defined. Its meaning can only be found in the MPR by visiting the Battlespace Communications Systems PDSS, where the term 'JC4ISPO' is defined as the Joint Command, Control, Communications, Computers and Intelligence Systems Program Office.
1.89
The MPR is also made harder to understand by the ‘coded’ language which is sometimes used in PDSSs. For example, prior to redefining the role of the Battlefield Airlifter project because of an inability to deliver required capability (as noted above), its PDSS described the project’s capability progress in these terms:
The MAA identifies a requirement for Defence to deliver a response on retention, replacement or upgrade of the Missile Approach Warning System (MAWS). Options have been considered by the project and Defence in 2019 and a remediation decision forms part of the overall project capability consideration by Defence in 2020 for the execution strategy for all residual acquisition activity.
1.90
Accurately reporting on project staffing costs also remains an issue in the MPR. The ANAO noted that:
Following a JCPAA request made in 2018 ‘on how Defence major project cost variations and the costs of retaining project staff over time might be reported annually in future Major Projects Reports,’ Defence advised that it is not yet in a position to provide the staff cost component of projects and its systems are not capable of calculating the cost of retaining project staff over time.
1.91
Defence noted that ‘Defence’s staffing costs are included in the Defence budget, separate to acquisition and sustainment provisions. Defence will continue to identify how best to measure the impact of delayed projects, in a standardised manner.’
1.92
Finally, Defence proposed a review of the MPR process in its submission to the 2018‑19 MPR review. In its submission the ANAO noted that:
… the Department of Defence had proposed a review of the MPR process. Defence subsequently undertook an internal Smart Buyer review of the MPR and provided a copy of the review report to the ANAO.
1.93
As the ANAO noted, however:
It remains appropriate for the Committee to drive any review activity going forward. The ANAO would continue to provide the Committee with advice on both the auditability of any proposals for change and their impact on transparency.
1.94
The Committee has not been provided with a copy of the review.
Committee comment
1.95
The MPR is a multifaceted document which involves input from the many project teams at Defence, review and analysis by the ANAO, and direction from the Committee via the PDSS Guidelines. The final report contains complex technical and financial information and layers of Defence and ANAO jargon.
1.96
Because of how the MPR cycle functions, the implementation of the Committee’s 2020 recommendations on the clarity of the MPR will be more fully realised in the next MPR. In the meantime, the Committee takes this opportunity to restate its view that Defence and the ANAO should continue to work together to make the MPR as understandable and useful to non-specialists as possible, in order to maximise transparency and accountability to the Parliament and the public.
1.97
In relation to the COVID-19 pandemic, Defence’s accelerated payment terms to assist Australian defence industry partners and its agility in reassigning staff in the early stages of the pandemic to aid those efforts are commendable. Future MPRs will give a more complete picture of the pandemic’s effects on the cost and schedule of Major Projects. The Committee will observe this area with interest. In the meantime, however, it would be useful for the Committee to be provided with up to date information on the pandemic’s impact on MPR projects.
1.98
The Committee recommends that the Department of Defence provide a short update to the Committee within the first two months of the 47th Parliament on the latest impacts of the COVID-19 pandemic on major projects.
1.99
The Committee notes the differing views between Defence and the ANAO in relation to Defence’s implementation of the ANAO’s two recommendations on Projects of Concern. The Committee understands that one of Defence’s objections to creating consistent processes for entering and exiting the Projects of Concern list is that designing rigorous ‘hard’ triggers for entry and exit would be too complex. However, the processes need not be quantitively rigorous to be useful and their consistent application need not depend on such rigour. The department should revisit entry and exit processes for the Projects of Concern and Projects of Interest lists, and the ANAO should assess Defence’s efforts in future MPRs. The Committee will actively monitor this issue in future MPR inquiries.
1.100
The Committee recommends that the Department of Defence revisit its effort to provide criteria for projects to enter and exit the Projects of Concern and Projects of Interest categories and create processes for their consistent application, enabling these to be reviewed as part of the next MPR, and that the ANAO gives further consideration to these issues in the next MPR.
1.101
The Committee will continue to monitor progress in the reform of Defence’s risk management processes as changes are rolled out over time. Of particular interest will be the move away from spreadsheet-based risk management and the adoption and consistent use of the newly mandated risk management system across all Major Projects.
1.102
The Committee recommends that the Department of Defence provide an update within the first two months of the 47th Parliament on the implementation of the new risk management system and advise which, if any, projects have not fully transitioned.
1.103
In relation to the use of caveats and deficiencies, the Committee shares the ANAO’s view that their use does not in itself undermine milestones as a guide to a project’s progress. However, terms used to define any ‘delta’ or deviation from a declared milestone should be properly defined to ensure that their use does not undermine the clarity or validity of that milestone being achieved.
1.104
The Committee recommends that the Department of Defence provide a clear definition of any term used in Project Data Summary Sheets or elsewhere in the Major Projects Report that is associated with a delta or deviation from a project milestone being achieved, to ensure that the use of such a term does not undermine the validity of the milestone having been achieved.
1.105
Finally, the Committee notes that the Smart Buyer review of the MPR process discussed in the previous MPR inquiry report was conducted but that the results of the review have not yet been shared with the JCPAA. The Committee is interested in learning more.
1.106
The Committee recommends that the Department of Defence and the ANAO provide the Committee with a joint briefing on the outcome of the Smart Buyer review in the next Parliament so that the Committee can consider changes to MPR processes and scope if warranted.
Lucy Wicks MP
Chair
25 March 2022