- Capability assurance mechanism
Overview
4.1It is imperative that the Defence (including the Department of Defence and the Australia Defence Force (ADF)) can effectively identify and manage the complexities associated with material procurement and sustainment including acceptance of new capability into service. The Subcommittee reviewed Defence’s approach to capability assurance including ‘test and evaluation’ and how it impacts and informs accountability and risk identification.
4.2This section of the report discusses the ADF’s and domain capability assurance, test and evaluation, acquisitions, minimum viable capability and workforce issues.
4.3The 2020Defence Strategic Update identified that the Indo-Pacific was in the midst of consequential strategic realignment, regional military modernisation, technological disruption with an increasing potential for state-on-state conflict. The update highlighted Defence should be better prepared for the prospect of high-intensity conflict and should assume that ten-year strategic warning time was no longer an appropriate basis for Defence planning.
4.4The National Defence: Defence Strategic Review 2023 (2023 DSR)further amplified these concerns stating that Australia’s strategic circumstances have radically changed. The Indo-Pacific has seen a major power strategic competition, with Australia’s alliance partner, the United States of America no longer the leader of the Indo-Pacific.
4.5The 2023 DSR concluded that the ADF as currently constituted and equipped, was not fully fit for purpose to ‘maximise the deterrent effect and response options from ADF capabilities’. The 2023 DSR found that the ADF force structure needs to evolve, explaining:
… the current joint force, namely the combined effect of Navy, Army and Air Force working together does not appropriately reflect the growth of domains. The evolution to five domains – maritime, land, air, space and cyber – demands a new approach. The ADF must rapidly evolve into a genuine Integrated Force, which harnesses capability across the five domains.
4.6The Subcommittee’s inquiry into Defence’s capability assurance and ‘test and evaluation’ sought to understand how these measures inform risk identification and accountability and how potential systemic issues are addressed.
4.7The Subcommittee received evidence from a diverse range of stakeholders regarding the importance of capability assurance but also potential areas for its improvement. However, as Defence capability assurance is closely linked to Defence’s capabilities and preparedness directives, there was a limit to what information Defence can provide the Subcommittee and thus what the Subcommittee can report on.
Defence capability assurance overview
4.8To maintain operational effectiveness and relevance Defence must continually introduce and maintain capabilities and then decommission redundant capabilities from service within scope, available resources and scheduling constraints. Defence must review and consider the relevance of a capability, integration, safety and sustainability overtime in identifying required capabilities.
4.9Capability acquisition processes ensure acquisitions meet these requirements and test their relevance to Australia’s changing strategic and operational environments, until they are superseded and withdrawn from service to be timely replaced with an alternate system or capability.
4.10In seeking out and acquiring capabilities, the capability assurance process requires the Verification and Validation (V&V) of potential systems against the context of the mission, operating intent, usage and system certification standards. An Objective Quality Evidence assessment is undertaken to ensuring a system is fit-for-purpose and meets V&V requirements. An Objective Quality Evidence may include inspection, demonstration, analysis and test and evaluation (T&E) of the desired capability.
Domain capability assurance
4.11Defence has ‘three mature “worthiness” domains - Air, Land and Sea’ - in support of the capability assurance accountabilities. These worthiness authorities are required to identify capability risks, review recent incidents and T&E reporting. Worthiness domains are required to provide domain leads annual reports detailing the domain capabilities and key issues affecting ‘worthiness’ of the respective domains.
4.12Cyber and Space worthiness agencies and processes are under development to provide equivalent capability assurance to the mature domains in the future.
Capability assurance accountabilities
4.13Overarching capability assurance accountabilities are assigned, to ensure capability assurance is aligned with domain specialisation, as follows:
The three Service Chiefs are accountable for the assurance of effective Defence capabilities within the Air, Land and Maritime domains in relation to each of these aspects of capability. Accountability for the assurance of purely joint capabilities, for example Health, Space and Cyber, rest with the Chief of Joint Capabilities. The Vice Chief of Defence Force is accountable for the assurance of integrated joint capabilities that are formed through collective, coordinated application of capability elements from multiple domains.
Test and evaluation assurance
4.14In August 2021 the new Defence Test and Evaluation Strategy aimed to ensure that T&E across Defence supported risk-based capability decisions. In 2024 a new T&E Governance Model was planned for introduction to enhance the:
- Identification of potential capability risks and prioritise T&E activity;
- Monitoring of T&E reporting throughout the capability life cycle ensuring T&E outcomes and capability risks are actively reported to Head Force Integration (HFI); and
- Upgrade to Defence’s capability decision support tool - ‘Capability-One’ - improving fidelity and transparency of T&E activities.
- Defence’s submission outlined the extensive range of T&E activities across various levels in the support of capability assurance including:
- Preview T&E (PT&E) – to compare different platforms for selection, fill knowledge gaps and determine suitability for the intended role or environment and to refine requirements development.
- Developmental T&E (DT&E) – to develop the design of a platform prior to Defence acceptance...
- Acceptance T&E (AT&E) – to confirm that the platform meets contracted and certification requirements…
- Operational T&E (OT&E) – to confirm the platform meets operational requirements when employed in the intended environment, by the intended operators using the intended procedures…
- Force-Level OT&E (FLOT&E) – to confirm a combination of capabilities or platforms operating as an integrated system meet operational requirements when employed in the intended operating environment, by the intended operators using the intended doctrine and procedures.
- In-service T&E (IT&E) – can be any combination preview, development, acceptance, operational and force level T&E applied to verify and validate the capability, or modernisation to meet the needs of its changing environment.
Acquisition and investment prioritisation
4.16The Defence submission stepped out the early life-cycle capability assurance process:
High level capability requirements, often expressed as Critical Operating Issues (COIs) are articulated in a Joint Capability Needs Statement (JNCS) developed by the sponsoring Capability Manager and ratified by the Vice Chief of the Defence Force Group through Head of Force Design (HFD) and Head of Force Integration (HFI). Once agreed, the JCNS, Project Execution Strategy, associated business case and draft cabinet submission are assessed at a Capability Gate Review (CGR) prior to being considered by the Investment Committee… and presented to government for approval…
CGRs involve consideration of the submission by HFD and HFI supported by Contestability Division at monthly sittings.
4.17The Investment Committee meets monthly to consider acquisitions in the context of the wider Integrated Investment Program, deciding if a capability proposal is fit for government consideration. The Investment Committee is also informed of significant capability risks as assessed by the capability sponsor and reviewed by Force Design, Force Integration and Contestability Divisions.
Minimum viable capability
4.18The 2023 DSR assessed Defence’s approach to capability acquisition as ‘not fit-for-purpose’. The 2023 DSR found that the capability acquisition system:
… needs to abandon its pursuit of the perfect solution or process and focus on delivering timely and relevant capability…
Defence must move away from processes based around project management risk rather than strategic risk management. It must be based on minimum viable capability in the shortest possible time.
4.19Defence highlighted in its submission that:
A competitive tension exists between scope, cost and schedule, which is heightened by the need identified in the [2023] DSR to deliver minimum viable capability.
4.20The Defence submission further stated that ‘agile design and delivery methods for iterative and evolutionary design’ can define the minimum viable capability requirements, with the benefits that it will, ‘allow rapid design and integration of new technology or innovative solutions and allows early identification of risks and issues’.
4.21RAND Australia described minimum viable capability as ‘representing a threshold level of capability that offers operational benefit in the shortest time’. RAND also recommended Defence capability should be delivered ‘within the defined requirements of scope, budget and schedule, which are established prior to the acquisition phase’. RAND added that the schedule needs to be prioritised, acquisition projects be more ‘disciplined’, to deliver within planned schedule and budget. This contrasts with the ‘traditional approach of prioritising higher levels of capability regardless of cost and schedule implications’ (which the 2023 DSR characterises as ‘pursuit of the perfect solution’). Putting it simply, RAND Australia concluded: ‘greater speed to achieve capability is a key element of the DSR’.
4.22RAND further proposed that this minimum viable capability approach represents ‘a new epoch in which capability assurance is relevant throughout the capability lifecycle’. This aligns with an increased expectation that defence systems will be incrementally upgraded to stay competitive, because:
… defence processes need to shift from a primary focus of accepting new capabilities into service, to an ongoing monitoring of capabilities as their effectiveness against emerging threats diminish.
4.23It was emphasised that such a program of evolutionary upgrades during the lifecycle is ‘not well represented in defence capability processes’ but may be reflected in future updates of the Defence Capability Manual.
4.24RAND highlighted the criticality of the T&E function is ‘correlated with the level of risk in the related capability activity, and is especially relevant to developmental and integration risk.’ Therefore, they advised that ‘care must be exercised so that T&E isn’t associated with the quest for perfectionism but is undertaken consistent with the level of risk’.
Regulatory and international legal compliance
4.25The Law and Future of War Research Group recommended that an ‘assessment of domestic regulatory risk and international legal compliance should be integrated more broadly’ into the procurement and sustainment of new capabilities into service.
4.26It was also recommended that when considering capabilities:
… different edge case testing scenarios must also contemplate the likely legal framework in which novel capabilities might be applied. For example, the use of a capability in a “grey zone” operation as compared to use during armed conflict have profoundly different risk outcomes from a legal perspective.
Workforce
4.27Defence submitted that ‘demand for T&E services within acquisition projects exceeds the capacity’ of Defence’s T&E. As a result, many projects rely on contractor support to deliver T&E outcomes, such as the utilisation of the Original Equipment Manufacturer for DT&E monitored by the project’s Engineering or T&E manager.
4.28RAND contended that a ‘significant risk’ to Defence capability assurance would arise from ‘competition for suitably qualified, experienced and cleared workforce’. The context being an ‘expected shortage of engineering skills in Australia’ coupled with security clearance requirements that will preclude or limit Defence sector utilisation of skilled migrants. Defence capability assurance would be further adversely affected by departments reducing use of ‘“inappropriate outsourcing”, thus putting more pressure onto the retention of the professional workforce overseeing capability acquisition and sustainment activities’.
4.29RAND also highlighted the challenges of the ‘higher workforce turnover within the defence sector’ impacting capability assurance, noting ‘experience is relevant to productivity and quality of advice and decision making’. RAND analysis benchmarked shipbuilding studies where at least five years’ experience is required before employees can be presumed to be fully productive.
Committee comment
4.30The Subcommittee understands from the evidence received that the Defence capability assurance system is highly complex, traversing five domains with individual lines of capability monitoring, capability assurance assessment agencies and reporting responsibilities. The complexity extends from the day-to-day capability operator to the accountable capability assurance officer for reporting and prioritising, to the Investment Committee, the Chief of the Defence Force and ultimately, the Australian Government. The Subcommittee concluded that while complex, the system was clearly documented, clarified responsibilities, assigned accountabilities and had clear reporting and feedback mechanisms, which gave the Defence executive and government the opportunity to appraise capability priorities, risk and inform future capability investment.
4.31The Subcommittee acknowledges the maturity of the Air, Land and Sea ‘worthiness’ agencies and the robust interfaces and feedback mechanisms these systems provide to ADF capability and ongoing capability assurance. The Subcommittee noted the developing nature of the Cyber and Space worthiness systems and recommends prioritising the maturing of these processes as soon as possible to support Whole-of-Government and Whole-of-Nation national defence efforts.
4.32The Subcommittee also acknowledges the evidence given to it by Lieutenant General Natasha Fox that the strategic health review is currently assessing Defence health capability and its ongoing capability assurance requirements in that context.
4.33The Subcommittee is mindful of the highly integrated and essential input of testing and evaluation processes to ongoing capability assurance across a range of roles and levels within Defence and Defence Industry. A significant vulnerability, as highlighted by Dr Andrew Dowse, Director at RAND Australia, is the paucity of suitably qualified, experienced and appropriately cleared workforce.
4.34The Subcommittee believes that Defence and Defence Industry need to prioritise investing in growing and retaining test and evaluation qualified and experienced workforces. In the absence of testing and evaluation capabilities, Defence and government will need to accept higher levels of risk to achieve necessary capability acquisitions and assurance.
4.35The 2023 DSR stated that Defence needs to continue prioritising the introduction into service of minimum viable capabilities. The Subcommittee acknowledges that with the release of the 2024 National Defence Strategy and 2024 Integrated Investment Program, Defence has a prioritised list of capabilities to pursue, with allocated funding and a greater focus on delivering within planned schedules. The Subcommittee believes this will give Defence the best opportunity to achieve its directed tasks and realise the 2031 Integrated Force.
4.36Finally, the Subcommittee wishes to acknowledge the Australian National Audit Office submission and the Auditor-General reports pertaining to the Hunter class frigates and delivery of Health Services. The Auditor-General identified in both reports’ significant deficiencies in the execution of the acquisition or contractual framework and contract management of these services.
4.37The Subcommittee notes that Defence has accepted all Auditor-General report recommendations. The Subcommittee acknowledges that over multiple meetings and case studies throughout the year, Defence senior leadership has actively engaged with it to clarify ongoing issues and planned remediation steps. The Defence executive has the confidence of the Subcommittee to remediate these issues while prosecuting the capability priorities within the 2024 National Defence Strategyand Integrated Investment Program.