Chapter 6


Chapter 6

Compliance and awareness

6.1        Based on its examination of Defence's acquisition process the committee found that on paper at least Defence has a robust risk management regime, which is comprehensive, systematic and engages all stakeholders. Further, that if followed correctly, risk would be considered from the outset or formative phase of a project when critical decisions are made and then managed throughout the project including a continuous process of identifying, analysing and mitigating risk. Defence's key policy documents explicitly recognise risk management as an essential part of corporate governance and senior Defence leaders have stated their commitment to sound risk management practices.

6.2        In this chapter, the committee examines the implementation of Defence's risk management strategies. It compares Defence's stated policy on risk management and the advice contained in its relevant guides on procurement with practice and actions. Having determined that Defence's policy and advice on risk management is not the problem, the committee's purpose in drawing these connections is to better locate the source of poor decision-making and performance.

Problems in defence procurement

6.3        Evidence before the committee identified significant failings in a number of major projects. They included inadequate description of risk during capability definition and planning phase; underestimation of the maturity of the technology and/or complexity of integration; and miscalculation of industry's capacity to deliver. In essence, a failure to understand, appreciate and mitigate risk. Indeed, Defence in its submission recognised that the common causes of poor project performance noted from past and current projects of concern are:

  • unachievable expectations in terms of technology, performance or schedule;
  • scope changes;
  • ineffective defence stakeholder engagement and interaction; and
  • challenging commercial or business relations.

6.4        In this context, the committee believes that it is important to refer again to the finding of the Helmsman Institute that some of the complexity in Defence's acquisition projects was 'self-inflicted'. It cited factors such as embarking on highly developmental projects; level of customisation; limited clarity on the key drivers of the project; lack of clear plans to achieve target dates and results; and tension between the needs of the military chain of command and the requirement to deliver against defined contracts and commitments.[1] The causes of poor project performance identified by Defence and the Institute's observation about 'self-inflicted' complexity indicate that although Defence has a solid risk management policy, in practice it is not working to full effect.

Culture of risk management

6.5        Having examined risk management in the UK MoD, Mr Chris Maughan, defence analyst, was compelled to ask that if the MoD had the right process, guidance documentation and tools why then was risk management not delivering the anticipated benefits. In his opinion 'the answer can only lie in its actual implementation'.[2] He found:

For improvements to be experienced there needs to be a major shift, by the MoD, away from process and towards a concentration on comprehensive quantitative schedule and cost risk analysis. There needs to be an appreciation, within both MoD and the wider defence industry, of the root causes of the failure of risk management and a willingness to take the necessary actions to resolve them.[3]

6.6        This observation has direct relevance for Australia's Defence organisation. Indeed, a number of the independent members of the gate reviews cited risk identification, mitigation and management as one of the major challenges for Defence and an area in need of 'significant attention'.[4] Dr Ralph Neumann stated:

It is not a matter of process: the process exists. It is a matter of better understanding the business, focusing on things that matter and better utilising the opportunities to reduce risk rather than managing the fallout of the risks.[5]

6.7        In the previous chapter, the committee noted that to be effective a risk management regime should be:

...fully integrated and embedded in an organisation's culture so that risk management policy and practice is part of management thinking and actions and permeates all levels of the organisation—enterprise level, function level or business unit level—senior managers in particular must show leadership and commitment and managers at all levels must take responsibility.[6]

6.8        Despite the clear statement of commitment to risk management, evidence presented to the committee suggests that risk management may not be front and centre of people's thinking in defence procurement. The first indication is the extent to which personnel adhere to the guidance or directions issued in Defence's handbooks and instructions.

Adherence to procurement policy and guidelines

6.9        Compliance is essential if Defence's risk management policies and their supporting guidelines and manuals are to translate into organisation-wide practice. In its preliminary report, the committee noted problems caused by non-compliance with such directions and advice. For example, the Defence Teaming Centre described the Defence Procurement Policy Manual as 'robust', but noted that 'it is the differential tailoring and interpretation of these policies by the DMO that causes significant frustration and confusion for industry'.[7] It suggested that training in the interpretation of the manual across DMO would create 'a consistent interpretation and implementation' of the Manual.[8] This practice would encourage a 'more fluid and efficient procurement process with both the customer and contractor understanding and having the same interpretation of the policy'.[9]

6.10      Likewise, the Australian Industry Defence Network agreed that DMO's procurement procedures and processes as detailed in the procurement manual appear sound. It noted, however, that the poor implementation and apparent non-compliance with the DCP, Defence Procurement Policy Manual and the Defence Capability Manual schedules and processes adversely affected the acquisition and sustainment of ADF capability on a regular basis.[10] In this regard, the committee notes ANAO's audit report on Planning and Approval of Defence Major Capital Equipment Projects, which examined the key capability development documents from a sample of 20 Defence projects. The ANAO found that Defence was not consistently adhering to its 'administrative framework for implementing the process'.[11]

6.11      Along the same lines, the Pappas Report observed that the manner in which projects approach the management of risk was somewhat variable. According to Mr Pappas, the quality of detail on the type/level of risk, residual risk post-treatment, and ownership of risk was also inconsistent. He noted that a risk register had been in place for some post-Kinnaird projects, but there was no standardised template. According to the Project Management Manual, a project risk log should be established in the Needs Phase and is mandatory for second pass.[12] The log should be used 'to record all project risks, the likelihood, consequence and level assigned to each, the treatment strategies (if the risk is unacceptable), the amount of Project Contingency Budget assigned to each treatment and the individual responsible for managing risk'.[13] The integrated project team is to review the risk register and treatment strategies, at least monthly.[14]

6.12      Despite the existence of a risk register, Pappas found that 'some mitigation strategies had not been implemented and lacked a rationale or timeline indicating when the action was to be implemented and the success of the mitigation reviewed'. He recommended that technical risks should be measured and managed through a risk register with a standard format and clear action plans.[15]

6.13      In its performance audit into acceptance into Service of Navy capability, the ANAO observed that mis-matched expectations between DMO and Navy had adversely affected the acceptance into service process. It identified a range of factors that could result in misunderstandings or disagreements including instances of projects proceeding with high-level risk because of a lack of agreed Capability Definition Documents and Certification Plans and Systems Safety Plans.[16] The audit report found:

...without the application of greater discipline by defence in the implementation of its own policies and procedures, improved communication and collaboration across the relevant parts of the defence organisation during a project's life cycle and the maintenance of adequate records to support appropriate monitoring of capability development performance, the necessary improvements in acquisition outcomes will not be achieved.[17]

6.14      Finally, the committee draws attention to the ANAO's observations in the annual Major Projects Reviews where it continues to report on a lack of consistency in the application of policies, practices and systems relevant to risk management. In the most recent reviews, it noted that the different practices at a project level 'impact on a consistent and strategic risk management approach at the whole of the DMO level'.[18]

6.15      There could be a number of reasons for this non-compliance, inconsistency or laxity in applying guidelines including a lack of awareness, complacency, or no one person or group having responsibility or being accountable for their part in the process. Assumptions that someone else will check the veracity of the information before them or an absence of, or ineffective, oversight of the process may also contribute to the lack of regard shown toward the manuals and guidelines. A combination of both these cultural and structural factors may be at work that results in non-compliance. It may well be that the culture took root and flourished in Defence's environment of ill-defined organisational accountability.

Awareness and ownership of risk

6.16      A healthy risk management environment is one where all members of an organisation are fully aware of the risks, controls and tasks for which they are accountable.[19] For example, in 2002 the Deputy Director, ANAO, referred to the importance of having a clear view on what is an acceptable level of risk.[20] In this regard, Dr Thomson cited the project for 12 new submarines, suggesting that:

You cannot pretend that risk away, you have to look at that risk and stare it in the face. It has to be part of your decision making but I do not think we should throw up our hands and give up on doing things. We should simply take an objective and sober recognition of the risks that some of these options carry because of the present state of our engineering and other expertise.[21]

6.17      DMO's Project Management Manual makes absolutely clear that there is 'ownership of risks and controls'.[22] Two of the key principles enunciated in the manual are:

  • risks are not avoided, but rather managed at the level at which people have the authority, responsibility and resources to take action; and
  • a risk management culture is promoted and is part of everyone's job.[23]

6.18      In their recent audit of acceptance into service of Navy capability, the ANAO found some significant issues with Navy projects including 'that Navy, CDG and DMO did not have a shared understanding of the risks to the generation of the expected capability from Navy projects and had not taken shared responsibility for mitigating those risks'.[24] The Pappas Review also suggested that a 'clearer indication of the most critical risks would help those tasked with risk management to know where to focus'. Worryingly, it observed that DSTO's involvement and assessments of project options were 'not always paid the respect they should be'.[25] It should be noted that DSTO has a central role in providing technical risk assessments especially for first and second pass approval.

6.19      This devaluing of advice from technical experts by non-experts points to an organisational weakness. Furthermore, as noted in chapter 2, DSTO is not the only body of technical experts whose advice may be neglected. Within Defence the advice of domain experts and operators does not always inform key decisions, sometimes with unfortunate results. There appears to be no effective mechanism to ensure that critical technical advice is accurately reflected in submissions on major acquisitions to senior decision-makers and ultimately to government—no real contestability; no visibility of risk.

6.20      In respect of risk awareness, Mr King expressed concern that some people in Defence do not fully appreciate the critical importance of risk analysis, monitoring and management. He stated:

There is a problem we need to deal with in defence more rigorously than we sometimes do: we become a bit unreactive to red alarms. In other words, we see a risk and we watch it go through to fruition and say, 'Oh, yes, indeed it did happen'. That is happening less and less where we are focusing on what is a risk and what we are doing about it. Unfortunately, sometimes that materialises in a project of concern, when we have to go and do a new remediation project to get it right.[26]

6.21      Mr King stated that he tells his personnel that there are really only two sins they could commit—not knowing their risks or problems, and not telling anybody about it or not doing something about it. He explained that DMO is trying to encourage its people, when they have this risk, just not to talk about how they are 'monitoring it' or 'actively checking it', but to have a real plan to mitigate or treat it. According to him, more often than he would like, Defence have had a risk that it has 'allowed to come to fruition without a real remediation plan'. He told the committee that 'we need to work harder at that'.[27]

6.22      The Rizzo Report observed that Defence was beginning to develop mechanisms to quantify its appetite for risk 'in a formal way and to promote this vertically through the organisation'. It noted, however, that this practice 'needs to become part of everyday life in Defence, with effective risk management being adopted and linked throughout'.[28]

Committee view

6.23      Despite Defence's clear commitment to sound risk management and to the principle of promoting a risk management culture which is seen as 'part of everyone's job', some personnel fail to own risk and avoid rather than manage it. Indeed, evidence before the committee presents a compelling case that Defence must take risk management more seriously. Mr Pappas' description of the 'variable' approach to recording risk management activities is consistent with Mr King's comments about some personnel being unresponsive to emerging risks.

6.24      The fact that some defence personnel appear inattentive to, or unmindful of, risk or uncertain about their role in risk management must be symptomatic of a deeper systemic problem in defence procurement. This failure to own risk is not a process problem—it is clearly an organisational weakness that effectively permits people to avoid taking responsibility.

Learning lessons and recordkeeping

6.25      As noted in the previous chapter, to be effective, risk management should be part of a continuous improvement system where experiences in risk inform revised risk assessment and management strategies. This means that lessons must be learnt from previous experience and applied to future decisions and actions regarding risk management.[29] As Air Marshal Binskin, Vice Chief of the Defence Force, told the committee:

It is only a lesson learnt if you do not repeat it: otherwise it is just a lesson identified and it is useless.[30]

6.26      Industry representatives were of the view, however, that:

At the moment Defence is not capable of being able to capture lessons learnt and project those lessons learnt forward a decade. What tends to happen is that they end up repeating a number of mistakes which lead to relearning of those lessons.[31]

6.27      For example, the Defence Teaming Centre stated that the DMO 'appears to lack any capacity to learn from failings in previous projects'. It suggested that there does 'not appear to be any drive or motivation within the DMO to capture lessons learned and pass them on internally and to industry'.[32] The pattern of repeated shortcomings in projects as detailed in chapter 2 attests to Defence's difficulties in learning from past mistakes.

6.28      In its guide to risk management, Standards Australia suggests that the 'results of monitoring and review should be recorded and externally and internally reported as appropriate, and should also be used as an input to the review of the risk management framework'.[33] It stated further that risk management activities should be traceable.

6.29      In some cases, however, it was not the absence of records that was the problem but the quality of the documentation, which reflected a poor understanding of what was important and what was not. Many witnesses referred to Defence's procurement of major capital equipment as process bound. One referred to people in Defence getting 'bogged down' with too much paper work.[34] A number of independent members of the gate review boards observed that although improving, the standard of documentation could be lifted.[35] One noted 'a certain amount of nugatory work...and at times a lack of guidance of project direction that can occur pre project approval'. In Dr William's view there was 'an issue of quality and consistency'. He noted:

On some occasions I think there is an enormous amount of work put in to produce extremely large documents which are probably far more so than is needed—and it is done with the best will in the world but it must tie up a lot of resources. I think perhaps in some cases if we could not actually remove documents we could at least streamline them, and that would be quite a resource saver.[36]

6.30      Mr Gallacher was similarly aware of instances where the project team were 'spending enormous amounts of effort on doing detailed work but then missing important things that were going on'. He supported 'simplifying rather than adding complexity'.[37]

6.31      In the risk management process, records provide the basis for improving methods and tools, as well as the overall process.[38] The committee has commented on the haphazard use of the risk register—an important accountability and learning tool—which not only highlights Defence's poor record keeping but points to a deeper problem with risk management in the organisation. The observations about the inability of personnel to discern the important issues from the less important when producing documentation similarly suggests that other factors are at work when it comes to effective risk management. For example, evidence presented later in this report suggests that even though people are diligent and hard working they may feel disempowered or unable to effect change, may be the wrong person to make decisions about risk, or may not have the requisite qualifications and experience to recognise the significance of risks.

Conclusion

6.32      In order to identify deficiencies in the acquisition process, the committee considered the practical application of Defence's risk management practices and procedures as set down in its written guidelines and manuals. The committee found that, if followed correctly, the acquisition process should ensure that risks are identified early and managed appropriately. Clearly, however, in some cases problems emerge or are exacerbated in an acquisition project because of poor implementation of Defence's policy and guidelines. The committee finds statements indicating that defence personnel are not alert to risk most disturbing. There can be no excuse for such personnel disregarding their own procedures, which can result in the organisation being unaware of, downplaying or ignoring, risks that threaten the success of a major acquisition. In effect, as stated by Mr King, Defence must not allow situations to develop where personnel watch risk emerge and come to fruition without a remediation plan. Poor recordkeeping and inappropriate or incomplete documentation is yet another indicator of a poor risk management regime. In essence, despite Defence's risk management policies and guidelines, the evidence is clear and unequivocal that in practice Defence's risk management in a number of major defence acquisition projects has:

  • failed to identify risk during the early stages of an acquisition project or, as highlighted in chapter 2, if identified, especially by domain experts, risk was downplayed, misinterpreted, or ignored;
  • failed to monitor risk and its treatment on a systematic basis throughout the procurement process; and
  • failed to ensure that senior leaders and government were fully apprised of the nature and extent of risk resident in a project.

6.33      The question must then be asked—who is responsible and accountable for risk management: for ensuring that 'things do not go wrong', or if they do, for prompt remedial action. In the following chapters, the committee continues to seek to understand the reasons for poor performance when it comes to identifying and/or acting on potential problems. It considers accountability and responsibility; communication and reporting within the organisation.

Navigation: Previous Page | Contents | Next Page