House of Representatives Committees

| Parliamentary Joint Committee on Public Accounts and Audit

Navigation: Previous Page | Contents | Next Page

Chapter 3 Auditor-General’s Review

Conduct and scope

3.1                   The ANAO’s assurance review of the 2009-10 MPR was undertaken as an audit by arrangement with the DMO pursuant to subsection 20(1)(c) of the Auditor-General Act 1997 (Cwlth). The assurance review is additional to the ANAO’s regular performance and financial audits of the Defence portfolio.

3.2                   Under a section 20 agreement, the ANAO ‘conducts an assurance review of all projects included in the MPR, in accordance with the Auditor-General’s Independent Review Report’s scope, criteria and methodology.’ The ANAO determines its review methodology in formulating an opinion.[1]

3.3                   The audit was performed in accordance with Australian Standard on Assurance Engagements 3000[2] and provides for a lesser level assurance than a typical performance audit on an individual defence project in regard to:

3.4                   When assessing information contained in the PDSS, the ANAO examines the project performance taking into consideration: ‘budgeted cost, schedule and progress towards delivering the planned capability.’[4]

3.5                   In addition to assessing the PDSS, the ANAO’s assurance review also examined the longitudinal analysis of projects over time and ‘further insights by the DMO on issues highlighted during the year’.[5]

3.6                   The assurance review did not include ‘review of PDSS data on the achievement of future dates or events’ including Measures of Effectiveness, and major risks and issues.[6] In regard to the exclusion of these items the ANAO stated:

By its nature, this information relates to events and depends on circumstances that have not yet occurred or may not occur, or have occurred but have not yet been identified. Accordingly, the conclusion of this review does not provide any assurance in relation to this information.[7]

Formal audit opinion

Audit qualification

3.7                   The assurance review found (apart from the qualifications made in regard to expenditure in base date dollars and contract price in base date dollars), that the PDSS had been prepared ‘in all material aspects, in accordance with’ PDSS Guidelines.[8]

3.8                   Specifically, qualifications were provided for the ‘non inclusion of project expenditure history expressed in base date dollars for 19 major projects and the prime contract price in base date dollars for four major projects.’[9] Similar conditions for audit qualification were also provided for the two previous MPRs of 2007-08 and 2008-09.

3.9                   The 2009-10 MPR defines base date dollars as ‘the amount adjusted for the impact of inflation (prices) and foreign exchange movement over the period from a specified date.’[10]

3.10               The 2009-10 MPR further states that recording prices in base date dollars allows for the cost of a project to be compared (in like terms) to the actual expenditure over time. In addition, the financial tables in the PDSS ‘adjust for real variations to budgeted costs, which involve:

3.11               In regard to expressing expenditure data in base date dollars in the PDSS the ANAO stated this method of reporting demonstrates expenditure performance against the initial approved budget at the second pass approval stage (or when the Government commits to the particular acquisitions). The ANAO explained:

The provision of information in terms of base date dollars is one way of demonstrating DMO’s expenditure performance against the originally approved budget at second pass approval, which, broadly speaking is the stage when Government commits to the particular acquisitions.[12]

3.12               The ANAO also stated that reporting on projects’ using base date dollars simplifies the assessments that can be undertaken on the information and clarifies information so that it can be readily understood by the reader. Importantly, expressing expenditure and price data in base date dollars (or also possibly using the different method of out-turned dollars) allows for comparison of the initial approvals to the final outcome in a constant manner.[13] The ANAO stated:

The analysis we are able to do when we have base date dollars or out-turned dollars so that we can actually compare the information from second pass approval right through to final expenditure, is something that I think is of assistance to us in our analysis. It would tell us whether a project has come inside the budget and whether it would realise a surplus or whether there would be a deficit at the end of the project. I think it adds information for the reader because it comes back to the notion of whether you got ten planes for $10 billion and it simplifies the assessments that you can do on the information. ... DMO are currently looking at the usefulness of putting out-turn dollars there and I think that both are aimed at doing the same thing, which would be allowing you to compare the initial approvals to the final outcome in a constant manner.

3.13               In an effort to address the issues associated with a qualified audit conclusion for the 2008-09 MPR in relation to not recording expenditure in base date dollars, the DMO proposed to use Assets Under Construction (AUC) data instead. The previous committee was aware of the DMO’s proposed approach. However, the DMO was not able to implement this approach (through inclusion of AUC data) in the 2009-10 MPR.[14]

3.14               In an alternative approach to solving the issues surrounding qualification of price and expenditure data in regard to base date dollars for the 2009-10 MPR, the ANAO advised that the DMO is now examining reporting using out-turned dollars.[15]

3.15               The DMO commented that the ‘out-turned dollar’ reporting approach represents the future value of the amount expended on a project. The DMO is testing this approach by applying it to project approval value and determining to what extent, if any, project approval value may be exceeded. The DMO stated:

... the issue of base date dollar is still the same. Now we are really talking about it in an out-turning sense. What we mean is the future value of that dollar—which is when the project is approved by Government as second-pass. In reality, in future the project approval value—which we are trying to look at: whether we have exceeded or DMO will exceed that project approval value—will always be the same figure from today to 10 years in the future. In other words, it will be in those year dollars from when the project was approved rather than reverse back to what was done some time ago. I think we are consistent in our thinking; it is just a different terminology.[16]

3.16               Further, the DMO stated that while the current preference is to use out-turned dollars for reporting on new projects, this has not always been the case, and those older projects reported on in the MPR need to be updated using the new method.[17]

3.17               The approach to report in out-turned dollars is in contrast to the approach suggested by the DMO during the previous JCPAA Inquiry into financial reporting and equipment acquisition at the Department of Defence and Defence Materiel Organisation.[18]

3.18               In evidence provided in 2007 to the previous JCPAA inquiry, the DMO was in favour of reporting in constant dollars whereby the effects of inflation and foreign exchange are eliminated. The DMO also noted that out-turned prices were not very useful in determining whether a project has had a real cost increase or not over time. The DMO stated:

... the best way to talk about the quality of the project is to have everything in real dollars-constant dollars. That eliminates the effects of inflation and foreign exchange, and it stops this non-farm deflator instrument being used for out-turned prices, which you need to do to get appropriations but it is a piece of mathematics that is used for appropriation purposes but is not terribly helpful when we are trying to work out whether a project has had a real cost increase or not.[19]

3.19               Further, in evidence to a previous JCPAA, the Chief Executive Officer of the DMO advised that there are legitimate reasons for using different pricing methods and stated:

I have attempted over the last two years to use real pricing wherever possible and that is what I have asked my people to use wherever possible. But there are other bodies with legitimate needs for different types of pricing. If you are chasing appropriation, you have to use out-turning, so there are legitimate reasons why you have to use other ones. In all of the work I have done with media and when I do my business plan reviews, I talk to industry and so on, I try and use real prices throughout.[20]

3.20               In response to reporting on older projects in base date dollars or out-turned dollars the ANAO acknowledged that it would require additional time to investigate this approach and suggested that it could be implemented for the 2011-12 MPR.[21]

3.21               The ANAO stated that reporting using out-turned dollars for older projects such as for the Guided Missile Frigate Upgrade Implementation or FFG may not be feasible, but that it would be possible for all new projects included in the MPR.[22] The DMO and the ANAO advised that they would work together to provide the committee with a list of projects included in the MPR that are able to be reported on using out-turned dollars.[23]

3.22               In an effort to allow for differentiation between the two reporting approaches and clarification on the effect of the accounting standard change, the DMO stated that it would be in the public interest to report the old and the new methods in comparison tables in the MPR. The DMO stated:

... it would be of interest to the public also that in the year that we do make a change—which might be next year or the year after—we will compute the data by both methods, the old method and the new method, and have the tables next to each other, so that everyone always understands the effect of the accounting standard change, if you like, compared with the real thing going on in life. I think we need to measure it both ways for one year, so you can differentiate the two effects.[24]

3.23               In regard to timing of the assessment on reporting on out-turned dollars, the ANAO advised that the DMO and the ANAO would work together to provide the committee with a ‘comprehensive proposal for transition to a new arrangement and [ANAO’s] subsequent review’ by August 2011. This would allow for alignment with the process for project selection for the 2011-12 MPR.[25]

3.24               The DMO and ANAO consequently provided the committee with Guidelines for the development of the PDSS for the 2010-11 MPR. This included a list of 11 projects which will be reported on in base date dollars and current dollars for expenditure, with the remaining 17 projects disclosing only current dollars expenditure.[26] These 11 projects are:

Other findings

Projects’ capability performance

Measures of Effectiveness

3.25               At the project acquisition stage, the scope of a project is determined through a Materiel Acquisition Agreement (MAA) made between the Department of Defence (Capability Development Group) and the DMO.

3.26               MAAs incorporate a number of Measures of Effectiveness (MOE). MOE ‘are designed to set out the key capability performance attributes of the system to be delivered by the DMO.’ Further, MOE represent the expected technical status of a project once completed and ‘provide insight into a system’s likely suitability for planned operational release.’[28]

3.27               In regard to specification and focus of MOE, the ANAO found that ‘there is not a clear underlying consistency in the identification and articulation of MOE in the MAAs.’ [29] The ANAO made a similar finding in regard to MOE in the 2008-09 MPR.

3.28               As MOE data relates to forecasting future achievements it is excluded from the scope of the ANAO’s formal review. However, this data is provided by DMO to the ANAO so that the accuracy of MOE data included in the PDSS can be assessed.[30]

3.29               In line with the confidentiality requirements, associated with national security data for projects, only the status of the ‘assessment of the likelihood of delivering the required MOE is contained in the 2009-10 MPR and earlier reports.’ This assessment is presented through a traffic light analysis whereby:

3.30               The ANAO stated that the MOE provided in regard to the information contained in the 2009-10 MPR PDSS allows for broad trends to be identified and examined in regard to project delivery and capability performance. The ANAO stated:

While a multi-year comparison of capability performance needs to be treated with caution due to year-to-year changes in the bases of the data, this third MPR provides the opportunity to start to examine broad trends in the DMO’s assessment of the likelihood of projects delivering the required capabilities over time. This comparison can be done through examining this year’s PDSS and data reported by the DMO in its part of the MPRs for 2007-08 and 2008-09.[32]

3.31               Using the current MOE, the ANAO identified five projects which may be classified as under threat and one which at this stage is unlikely to be met.[33]

3.32               The DMO advised that it is unlikely that MOE will be reported on in their current form in the 2010-11 MPR.[34]

3.33               The 2010-11 MPR Guidelines provide that the previous section 3.5 Measures of Effectiveness is moved to section 4.1 and has been revised so that the:

Capability pie chart and associated narratives will provide a percentage breakdown of the FMR Milestones and Completion Criteria, as identified in the respective MAA, prior to ANAO site visit.[35]

Projects’ governance over acquisition processes

3.34               In the 2009-10 MPR, the ANAO reviewed a number of major governance aspects of projects. These were: Gate Review Assurance Boards, management of projects of concern, business systems, measures to improve clarity of financial information in the MPR, identification of projects’ emergent risks, contingency budgets, the use of Earned Value Management Systems, and skills development.[36]

3.35               Discussion on the issues associated with the financial control framework for projects and the Gate Review Assurance Boards follows.

Financial control framework for projects

3.36               Examination of the DMO’s acquisition process provides the ANAO with an understanding of DMO’s operations and design and implementation of the ANAO’s test program for review. The test program includes ‘the examination of the DMO’s financial control framework, enterprise risk management arrangements and a formal assurance mechanism.’[37]

3.37               With the aim of addressing the qualification provided in 2008-09 for the non disclosure of information in relation to prime contract price and expenditure in base date dollars, the ANAO reviewed the financial control framework underpinning DMO’s management of its projects.[38]

3.38               Broadly, the ANAO found that the financial control framework differed for each of the projects examined. This was because of the ‘wide range of corporate and project management systems being employed and the varying financial management policies being adopted by different project offices.’ This meant that there was inconsistency of information recorded across projects and that ‘efficiencies could not be gained by adopting a consistent approach to developing and subsequently reviewing each PDSS.’[39]

3.39               The DMO stated that there is a process of amalgamation under way where legacy projects are being standardised in terms of current management systems and that it would be another two to four years before completion of the process. The DMO stated:

DMO was an amalgamation of the acquisition commands of the three services and then an amalgamation of the sustainment commands of the three services, and then the two acquisition sustainment commands got together to make a further amalgamation. That happened in about 2000. Bashing together all the systems and making them all identical has been an enormous challenge. You have to also ask: what do you do with legacy projects? If a project was started in 1998, do you keep it on the system it started with or do you move it over to a new system? We could work a lot harder in getting standardisation across. We have a standardisation office of about 20 people whose sole job it is to try and bring all the projects into a common standard. But it is long and arduous work, and it will probably be two, three or four years before we finish it.[40]

3.40               The ANAO also found that for some projects there were issues of accuracy and completeness of information ‘in the current DMO systems for reporting on project status to senior management.’[41] The ANAO provided an explanation of this finding and stated:

...as we go through an audit or assurance process, we look at the controls and the structure around the information that we are looking at. We are talking here about information and reporting that is predominantly collected manually. Obviously, as we look at an assurance process, at times we find errors. I could be wrong, but as that information would have gone through the chain in DMO it would have probably been picked up also, but we are reflecting that, as the information is provided, there will be errors in it—and there are a string of [quality assurance] processes and those kinds of things that need to occur to weed out that kind of error that is introduced through manual processes.[42]

Gate Review Assurance Boards

3.41               The process of using Gate Review Assurance Boards (GRAB) was examined by the ANAO as it is a key part of the governance framework of projects.

3.42               The GRAB process was introduced in 2008 in response to recommendations arising from the Defence Procurement and Sustainment Review or Mortimer Review.[43]

3.43               The GRAB consists of senior line management, other relevant DMO staff with key skills sets, and an external independent member. There is a requirement that a GRAB reviews projects at three specified points or gates at: First Pass Approval, Second Pass Approval, and contract signature. At a minimum a project must be cleared by a GRAB at these stages if it is to progress to the next stage.[44]

3.44               The GRAB cycle provides assurance to the Chief Executive Officer of the DMO ‘that all the identified risks for a project (cost, schedule, technical/capability) are manageable.’[45]

3.45               The ANAO reported that in total 20 projects have been subject to a GRAB review with five of these projects included in the 2009-10 MPR. In regard to the five 2009-10 MPR projects, the ANAO found that the GRAB Review outcomes were consistent with the relevant information reported in the MPR.[46]

3.46               The DMO stated that the GRAB process of review provides a check on project management processes. The DMO stated:

One of the advantages of a Gate review is that it is not always obvious to a project manager what domain they are working in. When have you changed it enough that it is an important change or when have you changed it so that you are really into the area of new design development? One of the great advantages of the experienced people on the gate review boards is to be able to highlight some of the risks that the project manager might be getting into unintentionally—it has grown from it being off-the-shelf, changes have been introduced as the project has matured and now they are starting to get into another area. As you can see from that illustration that can have some quite profound effects on schedule in particular in our case if you are not seeing the changes. So the boards are doing that and the boards are feeding back into the project managers’ lessons learned—long-term lessons learned, the risks of integration, schedule risks and so on.[47]

3.47               The ANAO stated that it would continue to assess the GRAB review process for potential efficiencies and has sought information from the DMO on performance measures in relation to the delivery of projects ‘within the agreed milestones.’[48]

Earned Value Management Systems

3.48               The previous committee requested that through the 2009-10 MPR, the ANAO cover specific governance areas of projects. This included Earned Value Management Systems (EVMS) at the project level, ‘the controls over the use of contingency budgets, and the management of prepayments to contractors.’[49]

3.49               The ANAO provided an explanation of EVMS as:

A method of using actual cost and schedule information to measure and report project performance, as well as forecast future performance, and can be used to ensure that project payments do not exceed the value of work performed.[50]

3.50               In particular, the previous committee was interested in including EVMS data at the project level as a requirement in the PDSS. As a result, the ANAO reviewed the extent to which EVMS are being used in projects in the 2009-10 MPR.[51]

3.51               In its response to the previous committee, in relation to the 2007-08 MPR, the DMO stated that it was unable to provide EVMS data ‘for those projects with contract arrangements that do not have Earned Value Management requirements’, which includes Foreign Military Sales procurements. In addition, the DMO was concerned that providing EVMS data for selected projects only would not meet the objective behind the MPR, which is ‘to have a standardised set of data across all MPR projects.’[52]

3.52               In the 2009-10 MPR, the DMO reported on the use of EVMS in regard to its projects and stated that 14 of the 22 MPR projects are currently or have previously used EVMS ‘as a payment or contract management method.’[53]

3.53               The DMO also reported that while EVMS is commonly used for contracts valued at $20 million or more, the majority of major capital acquisition projects use ‘milestone payment options, as this is a more appropriate way of ensuring the delivery of goods and services as specified in the contract.’[54]

3.54               The ANAO reported that:

Where projects’ contracts and schedules had been re-base lined, EVMS was no longer linked to contractor payments, and instead those projects use a milestone-only approach (that is, all payments are made on the achievement of the agreed milestones).[55]

Concluding comments

Audit qualification

3.55               The Auditor-General through the Australian National Audit Office (ANAO) provided a qualified audit opinion in relation to the 2007-08, 2008-09 and 2009-10 Major Projects Reports (MPRs). Similarly, all three qualified opinions were given in regard to price and expenditure information not being provided in base date dollars.

3.56               The committee understands that presenting price and expenditure information in base date dollars excludes the impact of inflation and foreign exchange movement on prices and presents dollars in a constant manner. The committee agrees with the ANAO that using the base date dollar approach facilitates the understanding of expenditure on projects and so provides clarity in determining whether a project has met, come under or exceeded its approved budget.

3.57               The Defence Materiel Organisation (DMO) has previously suggested the use of Assets Under Construction data as a reporting methodology, to address the qualification received on the 2007-08 MPR. This approach was not able to be implemented.

3.58               The DMO is currently exploring using out-turned dollars to address the qualification. Out-turned dollars will present expenditure on project information for the future value of the project from the year in which the project was approved.

3.59               The committee notes DMO’s previous stance on the use of constant dollars to report on projects. This is in line with the approach of using base date dollars, but in contrast to its preferred current approach and its examination of the out-turned dollars approach to reporting expenditure on projects. The committee would like to understand the rationale for the change in preference for reporting methods for expenditure.

3.60               The committee acknowledges that presenting MPR project information in out-turned dollars would be possible for new projects entered into the MPR, but may present a challenge for the DMO in regard to older projects. The DMO has proposed to present in a comparison table, the expenditure on 11 projects in both out-turned dollars, base date dollars and current dollars. The committee supports this approach.

3.61               The committee also notes that a number of the projects included in the list of eleven are older projects such as the Airborne Early Warning and Control Aircraft, the Bushmaster Protected Mobility Vehicle and the Armidale Patrol Boat and if they readily lend themselves to being expressed in out-turned dollars, then it should also be possible for other older projects. The committee would support this approach for all projects with the exception of the Guided Missile Frigate Upgrade Implementation which has been ongoing since 1999 and may present a significant investment in time and resources for reporting.

3.62               The committee’s preference would be for project expenditure information to be reported on in a constant manner. This approach would allow for a constant set of expenditure information to be presented, allowing for ease of comparison between years, identification of any project concerns and overall more effective and efficient scrutiny. In turn, improvements in scrutiny would allow for increased transparency of information and greater ease of monitoring, leading to policy and practice improvements. In the longer term, the DMO’s accountability in its project management would also be strengthened.

Measures of Effectiveness

3.63               Measures of Effectiveness (MOE) represent the key capability performance attributes of the system to be delivered by the DMO. MOE also represent the expected technical status of a project once completed and provide insight into a system’s likely suitability for planned operational release.

3.64               The ANAO’s review found that in regard to specification and focus of MOE, there is not a clear underlying consistency in the identification and articulation of MOE in the Materiel Acquisition Agreements, which are the scope documents for major capital acquisition projects.

3.65               The ANAO noted that while MOE data relates to forecasting future achievements and so is excluded from the scope of the ANAO’s formal review, MOE data is made available to the ANAO so that the accuracy of MOE data that is included in the Project Data Summary Sheets (PDSS) can be assessed.

3.66               The DMO advised that it is unlikely that MOE data will be reported on in its current form in the 2010-11 MPR. The committee did not receive evidence about the implications of changing the form of MOE data in the 2010-11 MPR and so is uncertain about how this information will be presented.

3.67               The committee believes that in terms of consistency and in line with the need to be able to compare data across consecutive MPRs that the form in which MOE data is presented in the 2010-11 not be changed from how it is presented in the 2009-10 MPR without firstly undertaking a thorough analysis of any proposed changes and with subsequent endorsement of the preferred approach by the committee.

Financial control framework for projects

3.68               The ANAO’s test program for review includes the examination of the DMO’s financial control framework, enterprise risk management arrangements and a formal assurance mechanism.

3.69               The DMO stated that it is still in the process of amalgamation in terms of the management process applied to legacy projects and that it will be another two to four years before this is completed.

3.70               The committee is interested in the ongoing effects of this amalgamation on how MPR projects are managed and what impact this will have for meeting project cost, schedule and capability.

Gate Review Assurance Boards

3.71               The committee supports the principle underlying the Gate Review Assurance Boards (GRAB) process which is to provide assurance to the Chief Executive Office of the DMO that all identified risks in regard to cost, schedule and capability are manageable.

3.72               The ANAO stated that it will continue to assess the GRAB process for potential efficiencies and has sought relevant information from the DMO to enable this to occur.

3.73               The committee supports the ANAO’s review of the GRAB process and looks forward to receiving the results of the ANAO’s analysis.

Earned Value Management Systems

3.74               Earned Value Management Systems (EVMS) data is used to monitor the cost and schedule of projects. In the interest of improving the accountability and transparency of the management of major projects, the previous committee requested that EVMS data be included in the PDSS.

3.75               The committee agrees with the previous committee’s request that the DMO in conjunction with the ANAO develop a standardised graphical representation of each project’s cost and schedule variance for inclusion in the PDSS.

3.76               The committee understands that it may take time to develop a standard method for graphical representation of EVMS or milestone data in the PDSS, but since it has already received the 2010-11 Guidelines, would like to see this format included in the 2011-12 MPR.

Recommendation 6

3.77

The committee recommends that the Defence Materiel Organisation include in the format of a comparison table, for the listed eleven projects included in the Major Projects Report, columns appearing side by side showing base date dollars, out-turned dollars and current dollars for expenditure information.


Recommendation 7

3.78

The committee recommends that the Defence Materiel Organisation present the findings of its examination of the presentation of financial data on all possible methods for project expenditure information (Eg. Base date dollars, out-turned dollars and current dollars) to the Joint Committee of Public Accounts and Audit (JCPAA) as soon as it is completed and no later than 31 August 2011.

This examination should include a: (1) preferred method, and (2) comprehensive proposal for transition towards the proposed new arrangement. In addition, the proposed examination should be reviewed by the Australian National Audit Office before it is submitted to the JCPAA for consideration and recommendation prior to inclusion in the MPR.


Recommendation 8

3.79

The committee recommends that the way that Measures of Effectiveness data is presented in the Major Projects Report not be changed until a thorough analysis outlining the reasons for and implications of the change has been undertaken and presented to the Joint Committee of Public Accounts and Audit for consideration and endorsement.


Recommendation 9

3.80

The committee recommends, in line with the previous committee’s recommendation, that the Defence Materiel Organisation in conjunction with the Australian National Audit Office develop a standardised graphical representation of each project’s cost and schedule variance for inclusion in the Project Data Summary Sheets for the 2011-12 Major Projects Report Guidelines.

 

Mr Robert Oakeshott MP

Chair

29 April 2011

We acknowledge the traditional owners and custodians of country throughout Australia and acknowledge their continuing connection to land, waters and community. We pay our respects to the people, the cultures and the elders past, present and emerging.

Aboriginal and Torres Strait Islander people are advised that this website may contain images and voices of deceased people.