Chapter 12

Technical analysis and test and evaluation

12.1      Proponents of OTS acquisition have highlighted that the selection of developmental products is a source of added complexity and greater uncertainty to an acquisition project, thereby increasing the risk of problems emerging during the procurement process. The committee observes that not only has this view led to the current preference for OTS acquisition expressed by many in government, Defence and the media, it may well have given rise to the increasing practice of manufacturers claiming that products are OTS when in fact they turn out to be developmental. Witnesses have presented numerous cases whereby the expectation that a procurement activity is OTS has led Defence to believe that a product is more mature or an outcome more predictable than experience (or an experienced review) would indicate. The conspiracy of optimism, referred to by a number of witnesses, appears to have led Defence to undervalue the role that developmental test and evaluation can play in the early stages of the acquisition cycle to identify and analyse risk in a quantifiable and defensible manner.

12.2      The committee also notes observations made by the Helmsman Institute suggesting that the complexity of some Defence projects was so high that they were 'placed at risk of never delivering the required capability, and failing to work'.[1] This has proven to be true even for some projects that were presented to be OTS, such as the MU90 where integration across a number of platforms was compounded even further by a decision to constrain phasing to line up with other projects (for example JASSM on AP3). It has also proven to be the case where other purchasers withdrew, leaving Australia holding more of the risk. There is a moral hazard faced by industry and CDG in that both parties have an incentive to support the view that a particular technological reach or level of integration with other weapons systems is achievable.

12.3      The committee notes that this conspiracy of optimism may have tended to crowd out or ignore dissenting voices that could alert Defence to the true extent of capability, technology, integration and certification (hence cost and schedule) risk represented by a proposed project.

12.4      In the previous chapter, the committee referred to the importance of Defence being a knowledge-based organisation: of having a deep understanding of the products it intends to purchase; and of the critical importance of having the right people able to ask the right questions. It is particularly important to note that the problems experienced by some projects were due to an underestimation of the amount of developmental work required. This lack of knowledge about the technical maturity of a capability raises the question about the analysis undertaken of the proposed project, and highlights to the committee the absence of early test and evaluation conducted by qualified ADF T&E practitioners as part of a structured risk identification process. In this chapter, the committee uses test and evaluation as a means of assessing the quality of analysis underpinning Defence's capability development process.

Off-the-shelf purchase

12.5      According to Pappas, technical risk accounts for more than 50 per cent of post approval slippage in projects approved after the Kinnaird review.[2] Many analysts, advisers and Defence and industry personnel familiar with defence procurement recognise that purchasing off-the-shelf can reduce the risk of things going wrong. Usually, the costs are known and the performance is proven.[3] The government endorses these views. The 2009 White Paper and Defence's procurement manual make clear that off-the-shelf solutions to Defence's capability requirements 'will be the benchmark against which a rigorous cost-benefit analysis of the military effects and schedule aspects of all proposals will be undertaken'.[4] As noted, in the previous chapter, this discipline while necessary to limit the developmental risks of service wish lists, restricts the need for industry locally to grow engineers through developmental activities. It has also unfortunately been used as a rationale to limit the same development of skills with Defence, both at the high tech end of capability assessment, and for operational and sustainment activities. The longer term effect is that Defence has fewer qualified people able to test and evaluate thoroughly information provided by industry early in the procurement process, especially where it  is not all that it is marketed to be (i.e. the system is still really developmental or that the level of integration sought with other platforms may in fact be difficult to achieve).

12.6      It does not automatically follow that MOTS requires sustainment to be outsourced either domestically or overseas. Such decisions should depend on the normal costs and benefits, local industry capacity, and any strategic needs for self-reliance. Regardless, there must be in place within Defence a cadre of technical skill to manage properly both procurement and sustainment with assured continuity, integrated organisationally under single line accountability, drawing on a superior skill base supported with career paths, and without the risk of complete dependency on suppliers.

Possible secondary risks

12.7      Although recognised as an effective way to reduce risk, purchasing OTS may introduce secondary risks that need to be assessed, treated, monitored and reviewed. Miller Costello and Co noted that a MOTS procurement can be 'either a model for risk management or it can disguise risk and lead government into painfully bad decisions'.[5] So while OTS may initially be the preferred option, it may also pose significant risks that involve:

• modifications that may be necessary to meet Australian standards and operational conditions (yet perversely the emphasis on OTS, and therefore funding and involvement of agencies, may preclude adequate planning for this to occur);
• the integration of an OTS purchase into the ADF's existing and anticipated future capability;
• issues related to the sustainment and upgrading of the asset throughout its anticipated life and decommissioning and the associated costs; and
• medium to long term degradation of ADF's capacity to be a smart customer.

12.8      With regard to the last dot point, the committee notes that OTS may be purchased under terms that preclude any ADF unique modifications which may further reduce the opportunities where Defence can grow and sustain skill sets such as engineering, certification, T&E and R&D, and increase dependency on overseas suppliers. 

12.9      Other witnesses similarly underscored the caution Defence needs to exercise when purchasing an OTS product.[6]

12.10         Defence also noted the limitations of an OTS purchase. It acknowledged that while OTS equipment minimises procurement risk, such equipment would 'not always meet the needed long-term capability requirement'. It stated further an OTS may not readily integrate with other capabilities in service; may not always be available; may not suit Australia's geographic and strategic circumstances; and/or may not be available in a timeframe that allows Australia to avoid gaps in its defence capability.[7]

12.11         Despite this awareness of possible technical complications associated with OTS, decisions have been taken on such purchases that clearly indicate no robust consideration was applied to such risks. Indeed, one of the identified causes of problems in defence procurement has been the underlining assumptions about products purchased off-the-shelf. The Helmsman Institute noted:

A number of projects started with the assumption that as a product was being offered as an existing design by a supplier, that the product was 'Off the Shelf'. The approach that was then applied assumed that the product could move into mass production immediately. Helmsman believes that true 'Off the Shelf' approaches can only be used if the products have achieved a high volume production rate and are in service in significant numbers in military service, and will only have limited customisation to fit local regulatory requirements.

All other projects need to assume that high levels of testing and evaluation will be required for testing and acceptance given the ADF regulatory environment. Helmsman believes that some of the highest complexity added to projects was that created by 'First of Type' or 'Early in Type' products being treated as 'Off the Shelf'. The unplanned need for substantial certification, systems integration, design and modification created addition complexity in stakeholder management, cultural clashes and journey complexity.[8]

12.12        Accordingly, even an apparently straightforward purchase requires a deep knowledge of the product.[9] Thus, with OTS products, Defence needs the capability to exercise a rigorous test and evaluation regime in order to understand fully the maturity of the capability it intends to acquire. Clearly then, for customised purchases and developmental projects the need for sound and comprehensive analysis is even greater. For example, the committee has referred to the conspiracy of optimism where both industry and the customer are drawn toward the leading edge technology. The danger is that 'an ambitious set of specifications' could be locked in before the associated risks are properly identified and understood.[10] Mr Bruce Green noted that going beyond the 'leading edge to the bleeding edge of technology is a recipe for disaster as these types of projects just bleed money'.[11]

Analysis—test and evaluation

12.13         Kinnaird fully appreciated the central role of analysis in defence procurement practices. In his view, there must be:

• detailed analysis of the options to achieve a required military effect before adopting a platform-based solution;
• a higher proportion of project funds spent on early analysis to improve project outcomes, which would represent an investment that can return dividends in terms of greater certainty in regard to costs and a better understanding of project risks; and
• rigorous analysis of technology, cost and schedule risks, backed by external verification, which is essential before any project is put to tender.[12]

12.14         Mortimer also recommended that 'any decisions to move beyond the requirements of an off-the-shelf solution must be based on a rigorous cost-benefit analysis of the additional capability sought against the cost and risk of doing so'. He stated that this analysis must be clearly communicated to government so that it is informed for decision-making purposes.[13] For projects that are not genuine MOTS, Pappas similarly recommended that 'improving technical risk management practices would help reduce schedule and cost escalation'. Specifically, this would involve:

• greater investment of project expenditure in de-risking projects;
• not progressing projects until they reach the required level of technical maturity;
• separating technology development from product development; and
• greater involvement of technical scrutineers (who need developmental activities if the ADF is to grow this skill) and better use of a risk register.[14]

12.15         Many witnesses underscored the importance of good quality and 'systematic independent analysis'.[15] In this regard, the ANAO noted that 'International experience shows that adopting a systems engineering approach in concert with program management of a high order offers the greatest likelihood of success for the delivery of complex and large scale projects, including Defence major capital acquisitions.[16] It explained:

Systems engineering involves the orderly process of bringing complicated systems into being through an integrated set of phased processes covering user requirements definition, system design, development and production, and operational system support.[17]

12.16         People familiar with the complexity of defence acquisitions appreciate the critical role of test and evaluation (T&E) activities in providing information about risk and empirical data to validate models and simulations.[18] Defence similarly recognises T&E as an integral part of the systems engineering process for identifying and reducing technical risk in the acquisition of defence equipment, though this recognition is on paper and not necessarily in practice.[19] The focus on OTS however has led many to believe that Defence only requires a T&E capability at the end of the process: i.e. operational T&E as part of introduction into service. What numerous Defence projects have shown however is that Defence must sustain, develop and employ personnel with experience in developmental T&E in order to conduct pre-contract analysis with rigour.

Early testing

12.17         T&E is a process that can be applied at the initial feasibility stage of a project and continues through to its delivery into service. Clearly, T&E at the feasibility stage helps ensure that a capability will operate as intended and can be produced in line with cost, schedule, and quality targets.[20] This observation about the importance of early analysis is based on wide project experience.[21] For example, with regard to technology risk, the GAO noted:

When technology risks are not managed early, an acquisition program can run into difficulties in later phases. Having a feasible, stable preliminary design for a weapons program early in the acquisition process is also important in lessening risk...by demonstrating that a product's design can meet customer requirements, as well as cost, schedule, and reliability targets.[22]

12.18         Supporting this contention that the customer cannot leave all design activities to the manufacturer, the GAO found that in recent years programs that have held critical design reviews reported higher levels of design knowledge. The committee notes the Haddon-Cave Review (UK) which found that a critical design review is only of value if the stakeholders involved (including the customer) have the necessary qualifications and design/certification experience to understand and challenge the information presented to them.

12.19         Witnesses similarly referred to the value of early research and development.[23] The Australian Business Defence Industry Unit noted that in order to avoid problems, a project must be set on the right course from the start. It suggested that 80 per cent of problems occur in the first 20 per cent of a project's life.[24]

12.20         Air Marshal Harvey agreed with the proposition that there is a case for conducting detailed technical risk analysis of a proposed capability at an earlier stage. He indicated that Defence do so, though not initially, in a formal technical risk analysis sense. According to Air Marshal Harvey technical risk analysis supports both first pass and second pass and forms part of the capability gate review board.[25] He made clear that DSTO follows 'a very rigorous process' for its technical assessments for first and second pass.[26]

12.21         The committee notes that the Air Marshal was referring to the process and not the capacity to analyse relevant risks (technology, integration, capability and certification). The committee has already noted the difference between process and application: i.e. what the manuals prescribe and what actually happens, as noted on many occasions by ANAO. Also, Air Marshal Harvey referred only to DSTO advice. The visit to the Aerospace Operational Support Group at RAAF Edinburgh highlighted to the committee that Defence has other centres of expertise that should be more effectively utilized early in the procurement process to identity the full range of risks presented by a particular solution.

12.22         The committee notes that a capacity to conduct developmental T&E is the same skill set as that needed to conduct effective risk identification and analysis. Proponents of OTS acquisition rightly point out that the original equipment manufacturer does not require ADF to have a developmental T&E capability—that it is industry's job to provide people to run that part of the process. But without a developmental T&E capability, Defence cannot assess the veracity of what they are being told or shown, either in absolute terms or within the certification and training frameworks required by Australia. The number of products accepted as OTS when they were in fact developmental has a strong correlation to situations where T&E expertise was not available, not engaged or not listened to.

Implementation

12.23         Ultimately, under the current process, the Chief Defence Scientist is responsible for the provision of technical risk assessments, technical risk certifications, the development of project S&T plans and for providing other S&T support as required.[27] As one of the fundamental documents that support the first pass approval, the TRA forms part of the Capability Proposal First Pass and needs to be in place.[28]

12.24         The Project Science and Technology Advisor, a DSTO officer,  prepares the TRA for second pass approval. It is intended to allow Defence to advise government on the areas and levels of technical risk of the options being proposed for acquisition. The Chief Defence Scientist signs off on the Technical Risk Certification which is included in the ministerial or cabinet submission.

12.25         There is no doubt that the procurement system should be sufficiently robust to ensure that information on the readiness of a platform for operational service is known. But as noted in chapter 5, one of the problems with risk management is the lack of awareness or the unresponsiveness of some personnel to emerging risk. Evidence suggested that despite Defence's recognition of the importance of test and evaluation, Defence does not pay sufficient attention to this most important aspect of risk management.

12.26         For example, notwithstanding Kinnaird's recommendation for small amounts of early up-front investment to quantify and minimise risk in complex projects, some witnesses were concerned that that was not happening.[29] In his review of the latest Major Projects Review, Air Commodore (retired) Bushell stated:

...the primary cause of project risk lies in the operational and technical areas of the project, and that these (largely potential and manageable) risks demand a very different approach, an approach requiring skills and competencies different from commercial (contract terms and conditions) management. Effective capability management requires that all capability functions—operational, systems and equipment engineering, test and acceptance functions and support requirements, including their associated risks, must come under tight Project and Systems Engineering management, and that commercial management must be constrained to contract management that supports project management objectives.[30]

12.27         In his view, the difficulties that are endemic throughout Defence's major projects indicate that 'the DSTO's capability development, test and acceptance and technical risk assessment and management input have not been adequate'. According to Air Commodore Bushell, such tasks 'were historically, and still are, a natural extension of the fundamental responsibility of the Capability Managers for raising, training and sustaining force'. He argued that DSTO has a role to play, but 'it is one that supports the Capability Managers, not replacing or double-guessing them'.[31]

12.28         In the previous chapter, the committee highlighted the overall shortage of skilled engineers in the area of defence procurement, especially in the Services, and most notably the hollowing out of such skills in the Navy. This shortage has serious implications for test and evaluation.

12.29         Air Marshal Brown gave the AEW&C as 'a classic example' of where there was inadequate T&E. He named two core things that were not done correctly on that project. The first was the contractor's decision to use emulators instead of real equipment on the systems integration lab. He explained that this decision meant that 'a lot of the integrations problems, instead of occurring inside the lab, occurred when we built the aeroplane'. According to Air Marshal Brown:

That decision was objected to by the Commonwealth quite strenuously at the time, but it was taken on a cost basis by the contractor. He decided that that was one way to save money, and they were confident in their design.[32]

12.30         He then referred to the AEW&C program's six-month development, test and evaluation program. He informed the committee that:

If you benchmark that against any other similar sort of highly developmental program, you will find that most people allow about three years. Guess what? That is about the time that it has taken us to do...My view of that program is that we have lost time but we are going to end up with the capability we contracted for.[33]

12.31         During its visit to South Australia, the committee learnt of another example of inadequate T&E. The committee was told that Defence believed the MRTT to be effectively an OTS purchase, with all indications in the tender process pointing to a purchase with a proven performance record for each of the major systems involved. Late in the program, however, the Australian test team needed to be boosted in numbers in order to get the data it needed to have the aircraft accepted into service. The committee also heard that Defence did not articulate clearly enough the Air Force's certification requirements in the contract. Finally, Defence did not manage its observation of the overseas tests at all well, resulting in a gap in its understanding of the tests. Defence did not make early investment in developmental T&E qualified staff on the resident project team a sufficient priority.

12.32         The committee's findings on T&E in defence acquisition projects are consistent with those of the broader issue of risk management in Defence's procurement of major capital equipment. Defence believes that its procedures are appropriate and should ensure that up front analysis followed by systematic test and evaluation activities would prevent unexpected major technical difficulties surfacing later in a product's build. The type of problems that emerged with the Super Seasprite, Landing Watercraft, Wedgetail, Tiger, the MRTT and the MRH-90 Helicopter suggest otherwise.[34]

12.33         If in fact DSTO is solely responsible for technical risk analysis as has been asserted, then the committee suggests that Defence fails to understand the full gamut of technical risk analysis and management from project inception to completion. If in fact CDG is no longer required to fund a preview evaluation by a qualified developmental test team, the committee's concern is amplified. The difference in quality of risk analysis from a CDG officer without relevant experience who is following a 'more thorough checklist of questions' as compared to that provided by a subject matter expert drawing on experience seems to be lost on Defence.

12.34         The committee has considered the underlying causes for the discrepancy between written guidelines and procedures and the implementation of sound risk management practices. The same causes are evident with Defence's T&E regime—non-compliance with policy and guidelines and unawareness or unresponsiveness to risk. As an example, the ANAO found that in a number of cases, the description of technical risk for project proposals did not provide sufficient guidance for decision-makers, or provide confidence that an adequate risk assessment had been conducted.[35] The committee has already referred to the observation made by Pappas that DSTO's technical risk assessments were not always paid the respect they warranted.[36] As the examples in chapter 2 clearly attest, the same observation can be applied to risk assessed by other Defence T&E personnel.

12.35         The lessons to be learned from recent projects underscore the need for improvement in test and evaluation. Such observations have particular relevance for defence projects still in the early stages of their capability development especially the need for up-front investment in research and analysis.

Resourcing test and evaluation

12.36         The committee notes that Kinnaird found that greater resources needed to be allocated to conduct comprehensive and rigorous T&E programs as part of project funding.[37] In this regard, the committee highlights a stark message that came out of the committee's site visits to South Australia:

An organisation cannot support high technical capability without the ability to test it. If it does complex things, it should set requirements but importantly it must understand the skills set it needs to validate requirements.

12.37         Dr Davies stressed a recurring theme throughout this inquiry that improving the quality of analysis is needed rather than improving the quantity of process and of information.[38] He also acknowledged that it takes a long time to grow that analytical capability. In his view, Defence, in the first instance, might have to rely on external contractors with expertise such as the RAND Corporation and Access Economics and use this expertise at least until in-house analytic capability can be built up.[39]

12.38         In this regard, the committee notes the challenges facing the capability managers in developing this level of expertise which to date, only exists in a formal sense for the aerospace domain. For example, Service chiefs are responsible for the initial officer training and specialist training (engineer, pilot etc) and for the 2–3 years of operational experience. Each individual T&E practitioner requires a further year of masters level full time training at a cost of around $1 million. After training, there is normally a period of 1–2 years of supervised T&E conduct and involvement in the ADF airworthiness and certification systems before an individual would be deemed competent to support DMO in a project role away from the test centre. Thus T&E personnel would need to enter the training pipeline several years well in advance of a project's need. This capacity therefore has to exist ahead of the project but given the high cost of training, should be an integral part of a consolidation capability procurement and sustainment team under the direct control of the capability manager, in line with the committee's preferred organisational model.

Long-standing concern

12.39         In its report on materiel acquisition and management in Defence, tabled in March 2003, the committee expressed a lack of confidence in Defence's 'capacity or will to address T&E concerns seriously'. At that time, Defence was preparing a revised T&E policy. The committee was particularly keen to ensure that the policy would be fully integrated (planned and funded) with the capability development process; provide for T&E to be carried out in an independent fashion; and embed a 'cradle to grave' philosophy.[40]  

12.40         Five years later, in its 2008 T&E Roadmap, Defence highlighted a raft of shortcomings in Defence's T&E pointing to a need for greater funding, improved training and attracting and retaining skilled and experienced personnel. They included:

• many existing Defence T&E facilities were approaching (or had reached) the end of their useful live and without significant upgrade and investment, they would be unable to adequately meet future weapon system or joint capability T&E requirements;
• there was no coordinated approach at a Defence level to identify T&E facilities necessary to support the DCP;
• spending needed to be prioritised to ensure that the appropriate T&E facilities exist or are accessible...;
• Defence would need to develop a T&E approach that enables the application and coordination of T&E across platforms and environments;
• inadequate funding—there are constant pressures on projects to under-resource T&E activities and funding for reworking and regression testing, which appears to be inadequate in many current projects;
• significant gaps in competency recognition, which potentially made T&E less attractive than other career paths;
• no formalised career management of personnel with T&E expertise and experience;
• most training was 'on the job' and there was high staff turnover as well as  difficulty recruiting and retaining technical officers and engineers;
• limited, and in some instances, a complete lack of T&E expertise in some specialist capabilities within Defence or Australian industry;
• while Maritime and Aerospace T&E was generally well conducted, the Land  environment did not have an organisation responsible for undertaking OT&E and acceptance into service;
• at the combat environment and Joint Force Operating levels, Defence lacked both organisations and resources to manage, coordinate and undertake T&E to ensure the integration of capabilities through a common methodology that was based on, and aligned with, relevant standards and procedures;
• information to assist in the development and conduct of T&E activities was often difficult to obtain; this was often the case when the equipment was procured from foreign organisations;
• current approach to the T&E during the acquisition phase often appeared to have industry performing T&E activities in isolation from the Commonwealth. (The committee understands that sometimes because of cost pressures and limits on overseas led projects choosing not to fund the posting of T&E practitioners to the resident team location (often overseas) contract negotiations do not provide for Defence T&E staff participation);
• project schedules were tight, placing pressure on all aspects of the project including the conduct of effective T&E, particularly the critical elements such as testing of key FIC related elements pertaining to the measures of suitability; and
• it would be more efficient if Defence adopted a model where T&E subject matter experts were engaged during the concept and requirements development stage for each project to assist with the specification of project T&E requirements (for example, Air 87).[41]

12.41         The Roadmap indicated that steps would be taken to address these findings.

12.42         Vice Admiral Jones, the sponsor for T&E, recognised that the Roadmap was 'quite a significant document' though he noted that there were 'a lot of utopian views in it and a bit of nirvana'.[42] This observation appears to be at odds with the clear articulation of the need for a robust T&E capability in Defence from previous reviews, reports and witness statements which lend weight to the recommendations of the 2008 T&E Roadmap. Vice Admiral Jones referred to work done since the publication of the Roadmap which has resulted in:

• a much more coordinated approach to T&E within Defence and the creation of an Australian Defence Test and Evaluation Office (ADTEO) within the Capability Development Stakeholder Group (CDSG);
• issuing unifying policies on T&E, acceptance into operational service amongst the Services and the conduct of trials;
• the formation of a T&E community within Defence with annual meetings of T&E principals; and
• a 'big increase' in pre first and pre second pass trials.[43]

12.43         In particular, he referred to the early test planning directorate, a group of six individuals, who specialise in writing test concept documents which they write in conjunction with the relevant T&E organisations. Group Captain Keith Joiner, Director General Test and Evaluation, explained further:

We are tightening that journey of discovery process there, so we have introduced a large number of additional questions into the test concept document writing guide as a result of some of the experiences we have had bringing into service military off-the-shelf and commercial off-the-shelf. That is delivered annually to the T&E principals, so it gets input from all domains, not just land and joint.[44]

12.44         The committee notes with concern that there appears to be a significant investment in form and process but not necessarily in the professional qualifications and work opportunities to gain relevant experience that will—over time—lead to real capacity to identify and analyse risk prior to contract signature.

12.45         Mr King accepted that at one time Defence 'did too much trusting and not enough verification' but was also of the view that Defence had 'moved on quite a distance from there'. Even so, he thought there was a role 'for improved analysis and testing of clams of maturity'.[45] Based on the committee's 2003 report, the 2008 Roadmap and more recent evidence, the committee is not convinced that Defence is moving quickly or decisively enough to address the matters raised in 2003 and 2008.

12.46         For example, the committee understands that ADTEO largely coordinates or conducts operational T&E for the land domain and coordinates some joint OT&E activities. However, the committee is also advised that its staff have no capacity for developmental T&E and the organisation plays no role in the management of the ADF's only developmental T&E agencies— Aircraft Research and Development Unit and Aircraft Maintenance and Flight Trials Unit. Their regulations are contained in ADF airworthiness regulations maintained by Air Force. If this is the case then the committee believes that significant rationalisation of both T&E policy and practice is required.

12.47         Moreover, looking ahead to where Defence needed to go, Vice Admiral Jones indicated that there would be a defence manual 'which all the services and DMO and DSTO and CDG have to sign up to'. He explained that in the manual 'we actually have to start to chart where we are going to go with the workforce and how we are actually going to grow and sustain the workforce'.[46] According to the Vice Admiral, standardisation and professionalisation remains an area where 'there is a lot of work that we have to do'. He stated:

At the moment we are in this situation where we have started to really get a much greater appreciation across the board of the importance of T&E, so that has been a big change probably in the last five years and we are seeing the value of that objective data for our decision making. But what we have to...have is a sustainable path for our workforce, and we see this next iteration and development of a manual as an opportunity whereby we can tease some of those issues out and then actually have some goals to set for ourselves to get to where we need to be.[47]

12.48         The committee notes that Defence was reviewing and developing a T&E concept paper and policy in 2003 and that its T&E Roadmap was produced in 2008. Now, Defence is still talking about producing a manual—that is about process. In this regard, the Haddon-Cave Review into the loss of the RAF Nimrod aircraft has some salutary advice for Defence:

The instinctive reaction of many governmental organisations to problems is the creation of more complexity, not less, and the 'bolting on' of more process, procedures, boards, committees, working parties, etc rather than stripping away the excess and getting down to the essential elements. The net result for the MOD was, unfortunately, an increasingly complicated safety and airworthiness system which was accompanied by a significant weakening of airworthiness oversight and culture during the period leading up to the loss of XV230 in September 2006. Over the past decade, responsibility for risk and risk management has been divided, dissipated and dispersed. Risk has effectively been 'orphaned' by being made part of an extended family, with everyone involved but no-one responsible.[48]

12.49         In the committee's view, while the people who own the process are talking about manuals, those with the responsibility and competence are not being heard.

Conclusion

12.50         Defence would have the committee believe that the organisation has an integrated and effective T&E regime operating throughout the capability life cycle to minimise the chance of unexpected technical difficulties arising. The T&E activities are meant to ensure the delivery of a fully functioning platform with safety-critical systems meeting all requirements. In practice, however, the failings indentified in some major projects stem from poor quality or inadequate analysis. The committee reinforces the message that early investment in analysis is an indispensable component of an acquisition. The Service Chiefs, in particular, as the ultimate users of an acquisition, must have the personnel with the skills and experience to stipulate from the early stages of a capability development cycle the test and evaluation activities required before they will accept an asset into service. Hence the committee's concern in principle about the real responsibility of the capability manager.

Recommendation

12.51         The committee recommends that the government make a long-term commitment to building technical competence in the ADF by requiring Defence to create the opportunities for the development of relevant experience.

Recommendation

12.52         The committee recommends that capability managers should require their developmental T&E practitioners to be an equal stakeholder with DSTO in the pre-first pass risk analysis and specifically to conduct the pre-contract evaluation so they are aware of risks before committing to the project.

12.53         Given that the capability to conduct this T&E and analysis needs to be extant prior to the commencement of any given project, the committee is concerned that cost pressures will lead individual services and projects to degrade this capability over time.

Recommendation

12.54         The committee recommends:

• the immediate finalisation of central defence policy on T&E to be implemented by capability managers in line with the committee’s recommended shift of full accountability for capability managers for all technical assessment of capability procurement and sustainment (independently assessed in conjunction with DSTO);
• full responsibility for the implementation of prescribed T&E processes be assigned to capability managers for all procurement activity from inception through to acquisition and sustainment; and
• each capability manager should ensure adequate skilled resources to oversee all T&E activity in line with central policy, as part of all acquisitions, including MOTS, as part of the capability managers’ total responsibility for procurement, but prior to as well as after second pass.

Recommendation

12.55         The committee recommends that Defence build on the capability already extant in aerospace to identify training and experience requirements for operators and engineers in the land and maritime domains and apply these to ADTEO. Capability managers will need to invest in a comparable level of training to enable their personnel to conduct (or at least participate in) developmental testing. The intention is to provide a base of expertise from which Defence can draw on as a smart customer during the first pass stage and to assist in the acceptance testing of capability.  

Recommendation

12.56         The committee recommends that Defence mandate a default position of engaging specialist T&E personnel pre-first pass during the project and on acceptance in order to stay abreast of potential or realised risk and subsequent management. This requirement to apply also to MOTS/COTS acquisition.  

Navigation: Previous Page | Contents | Next Page