ANAO Report No.28 (2012–13)

Chapter 3 The Australian Government Performance Measurement and Reporting Framework – Pilot Project to Audit Key Performance Indicators

Introduction

3.1                   The Federal Government’s Outcomes and Programs Framework, introduced in 2009–10, requires entities to identify and report against the programs that contribute to government outcomes over the Budget and forward years. This requires clearly specified outcomes, program objectives and appropriate Key Performance Indicators (KPIs).[1]

Previous audit reports and JCPAA reviews

3.2                   The topic of performance measurement and reporting has been the subject of a range of Australian National Audit Office (ANAO) audits over a number of years.[2] Most recently, an audit of agency development and implementation of KPIs, released in September 2011, found that:

… many entities continue to find it challenging to develop and implement KPIs; in particular, effectiveness KPIs that provide quantitative and measureable information, allowing for an informed and comprehensive assessment and reporting of achievements against stated objectives.[3]

3.3                   The audit recommended improvements to entity business planning processes around program objectives and KPIs; assessments by entities of their current use of costing information and the allocation of costs to programs; and for the Department of Finance and Deregulation (Finance) to review the development and implementation of effectiveness KPIs and to improve elements of its guidance to other entities.[4]

3.4                   A JCPAA review of the 2011 audit, tabled in May 2012, supported the ANAO’s findings and recommended:

n  that Finance include at least one recognised KPI methodology in its guidance to other entities;

n  that the methodology used by entities in the preparation of KPIs be available for review and potentially included in annual reports; and

n  that Finance report back to the Committee within six months on progress made on improving guidance to agencies, and how the ANAO’s audit methodology was envisaged to fit within and support the overall KPI framework and support ongoing policy enhancements.[5]

3.5                   In its February 2013 response to the Committee’s recommendations, Finance advised that it would ‘revise its policy and guidance materials for the development, monitoring and reporting of program-level KPIs’. The response indicated that the methodology for the construction of program-level KPIs would be clearer in the revised guidance, which would incorporate a practical user guide. Finance further indicated that it ‘would seek to brief the [JCPAA] on this work after the 2013–14 Budget’.

3.6                   The full text of the JCPAA’s recommendations and the responses from Finance are included at Appendix C to this report.

The pilot project

3.7                   The ANAO commenced a KPI audit pilot project in 2012–13 in the context of new powers given to the Auditor-General in 2011 with amendments to the Auditor-General Act 1997.

3.8                   The amendments provided the Auditor-General with explicit authority to undertake audits of entity KPIs and reporting against KPIs, and were consistent with a JCPAA recommendation arising from its 2010 review of the Act.[6] Specifically, the amendments to the Act enable the Auditor-General to ‘at any time’ conduct an audit of:

n  the appropriateness of the performance indicators of a Commonwealth entity; and

n  the reporting by an entity against those indicators.[7]

Objective and scope

3.9                   The ANAO’s pilot project was designed to:

… assess the status of the Australian Government performance measurement and reporting framework as a basis for implementation of a future program of audits of entities’ KPIs, and to develop a suitable audit methodology.[8]

3.10               The objective of the pilot was to:

n  build an understanding of experiences from other jurisdictions currently performing audits of KPIs as part of their financial statement audit processes, including the development of an approach and methodology;

n  initiate and maintain ongoing discussions with Finance and the Department of Prime Minister and Cabinet (PM&C) in regard to strengthening the administrative framework relating to performance measurement; and

n  develop and test an audit methodology and criteria to address the practical challenges of auditing the appropriateness of KPIs and the completeness and accuracy of reporting against them.[9]

3.11               The implementation of the pilot included:

n  assessing the Australian Government performance measurement and reporting framework as a basis for implementation of a future program of audits of entities’ KPIs;

n  reviewing the approaches taken by other relevant jurisdictions in the implementation of KPI audit methodologies;

n  working with the responsible Australian Government central agencies; and

n  testing a KPI audit approach and methodology within three entities.[10]

3.12               Four entities participated collaboratively in the pilot project. Finance contributed in view of its responsibility for administering the Outcomes and Programs framework. The Australian Taxation Office, the Department of Education, Employment and Workplace Relations, and the Department of Health and Ageing, contributed as agencies with experience in applying the principles of the Outcomes and Programs framework. One outcome and program from each of these three entities was selected for assessment.[11]

3.13               The ANAO also worked with audit offices in other jurisdictions that have work programs with a performance information focus, in particular the Office of the Auditor-General for Western Australia and the Office of the Auditor-General of New Zealand.[12]

Funding considerations

3.14               The ANAO’s report noted that the pilot project had been funded to date from its existing resource base, and that ‘the future development and implementation of a broader ANAO KPI audit work program will require the ANAO to be appropriately resourced’.[13]

3.15               The ANAO was not successful in obtaining the budget supplementation it had requested in order to continue its KPI activities in 2013–14, but was given the opportunity to bring back the proposal for consideration in the 2014–15 Budget. Despite not receiving supplementation in 2013–14, the ANAO has informed the Committee that it aims to continue work on developing a potential KPI audit work program within its existing budget.

Findings and conclusions

3.16               The ANAO’s report concluded that ‘entities continue to experience challenges in developing and implementing meaningful KPIs, and that the administrative framework supporting the development and auditing of KPIs remains problematic’.[14]

The current framework and supporting guidance

3.17               While not making any formal recommendations, the ANAO’s report made a range of observations about the current performance framework and areas for potential improvement. These included:

n  The need a framework that better accounts for different entity types:

The development of a framework that accommodates the diversity of public administration, and provides entities with the ability to report appropriate performance information regardless of role, is critical. … it would be desirable for the framework to recognise that the primary function of some entities is the delivery of services whereas other entities’ responsibilities include assessing the impact on the Government’s outcomes by those deliverables.[15]

n  The use of intermediate ’milestone’ objectives for longer term outcomes:

The Outcomes and Programs framework would benefit from further consideration of intermediate objectives where an overall outcome can only be achieved over the longer‐term.[16]

n  The introduction of ‘efficiency’ performance indicators:

The focus of a more comprehensive model for performance measurement and reporting in the Commonwealth would include consideration of the development and implementation of ‘efficiency’ indicators to complement the ‘effectiveness’ indicator focus within the current model.[17]

3.18               The ANAO reported that Finance’s guidance to agencies in relation to the development of KPIs had not been significantly updated since the introduction of the Outcomes and Programs framework, and there was no single comprehensive source of guidance and policy.[18] The ANAO also found the current framework and guidance to be unsuitable as a basis for auditing:

It is also clear from the Pilot that the current framework and accompanying guidance does not provide an effective framework against which entities’ KPIs can be reliably evaluated through an assurance audit process, as it does not specify clear standards or criteria that KPIs should satisfy. That said, it does need to be recognised that the current framework was not designed with this specific purpose in mind.[19]

Findings of the pilot audit

3.19               In conducting its pilot audit to assess the appropriateness of entity KPIs and the completeness and accuracy of their reporting, the ANAO developed the criteria shown in Table 3.1 and Table 3.2 below.

Table 3.1        ANAO criteria for the evaluation of the appropriateness of KPIs

Criteria		Characteristics	Explanation
Individual Assessment	Relevant	Focused	The KPI should assist significantly in informing whether the program objective is being achieved.
		Understandable	The KPI should be stated in plain English and signal the impacts of program activities to inform users.
	Reliable	Measureable	The KPI should be capable of being measured to demonstrate the performance of the program.
		Free from bias	The KPI should allow for clear interpretation of results.
Overall Assessment	Complete	Balanced	The set of KPIs should provide an overall picture of the impact of a program on the target group/s.
		Collective	The set of KPIs should demonstrate the extent of achievement against the program objective.

Table 3.2        ANAO criteria for the evaluation of the completeness and accuracy of KPI reporting

Criteria	Explanation
Data Completeness and Accuracy	KPIs should be reported on the basis of data and information that reflects accurately and completely all events that should have been recorded.
Disclosures Completeness and Accuracy	All disclosures relating to KPIs that should have been included in the annual report have been included (in accordance with PM&C’s Annual Reporting Requirements), and all KPIs and information relating to them in the annual reports is disclosed fairly and, where applicable, at the appropriate amounts.

Source        ANAO Audit Report No.28 2012–13, pp.63– 64.

3.20               The ANAO report included the following findings based on its pilot audit of a selection of KPIs from the three participating agencies:

n  Five of the 31 KPIs examined ‘clearly did not meet the Finance definition and were not assessed further against the appropriateness criteria’. Those KPIs not meeting the definition were either descriptions of activities or output indicators.[20]

n  Only one KPI met all of the characteristics outlined in the appropriateness criteria. Of the remaining 25 KPIs assessed, 22 met at least one of the characteristics and three partially met all but one.[21]

n  In analysing the entities’ processes for collating KPI data, areas for improvement included: the use of manual data entry; reliance on unverified external data sources; a lack of formally documented processes; limited quality assurance practices; and infrequency of KPI data measurement.[22]

n  The entities’ annual reports generally met PM&C’s Annual Reporting Requirements, although one of the entities did not identify whether its KPIs had been met and the other two entities did not provide explanations where KPIs were reported as not met.[23]

3.21               The report indicated that the entities involved in the pilot had been ‘receptive to the ANAO’s feedback and planned to revisit their current approach where required’.[24]

Future reforms

3.22               The report’s final conclusion was that:

… it is time for greater attention, investment and resourcing to be given to the quality and integrity of KPIs used by public sector entities to inform decisions about the performance of government programs. This requires a stronger and sustained focus by entities to enhance KPIs, and support provided by Finance through improved guidance. Entity leadership will be critical to success here.[25]

3.23               The report identified a key role for both the Government and the Parliament to encourage improvement in this area:

Encouragement to achieving better performance measurement can also be given by the Government and the Parliament through reviews, inquiries and the questions asked about the changes being brought about by specific programs having regard to the program’s objectives.[26]

3.24               The ANAO noted that the Commonwealth Financial Accountability Review (CFAR) currently being undertaken by Finance would provide an opportunity ‘to consider how to position the current performance measurement and reporting framework to respond to contemporary issues in public administration’.[27]

3.25               Since the ANAO’s report was released, the Government introduced into Parliament the first piece of legislation arising from CFAR. The Public Governance, Performance and Accountability Bill 2013 (PGPA Bill) seeks to replace the existing Financial Management and Accountability Act 1997 and Commonwealth Authorities and Companies Act 1997 with a single Act to govern the management of public resources and the performance of Commonwealth bodies. The Bill would be a ‘fundamental part’ of broader reforms to be introduced through CFAR.[28]

3.26               In relation to performance, the PGPA Bill aims to introduce a ‘framework for measuring and assessing performance, including requiring effective monitoring and evaluation’.[29] The Bill, if passed, would embed performance monitoring and reporting requirements in legislation for the first time. The Bill includes clauses that require entities to measure and assess their performance in achieving their purposes, keep records of their performance, and produce annual performance statements which may be examined by the Auditor-General.[30] The Bill’s Explanatory Memorandum states that the requirements for measuring performance would be outlined in rules associated with the Bill that would focus on ‘… enhancing the quality and integration of performance information required by government and the Parliament to assess actual against planned results.’[31]

The Committee’s review

3.27               Representatives of the following organisations gave evidence at the Committee’s public hearing on 19 June 2013:

n  Australian National Audit Office

n  Department of Finance and Deregulation.

3.28               The Committee’s evidence covered the following issues:

n  The need for greater focus on performance assessment

n  A less homogenous framework

n  Future directions for reform.

The need for greater focus on performance assessment

3.29               The need for a sustained and greater level of focus on performance monitoring, reporting and evaluation was a key message given to the Committee by the Auditor-General and by representatives of Finance at the public hearing.

3.30               Calling for more attention to be given to KPIs, the Auditor‑General said that there was a need to set a ‘much higher level of expectation’ around agencies using KPIs to measure the effectiveness of their programs. This was a message that would need to be put out ‘collectively’ by the ANAO, Finance and the broader Government.[32]

3.31               The importance of government being able to assess the effectiveness of their programs was concisely summarised for the Committee by the Auditor-General:

At the end of the day, government does need to understand the impact or the effectiveness of programs because that can help government take decisions about the targeting of programs and whether they can be more effectively targeted in order to allow resources to be allocated to other areas if the lens can be tightened on some programs. So it is terribly important information, but, at the moment, there is not enough emphasis given to it within government, and we need to change that in some way.[33]

3.32               The Auditor-General stated his concern that, while agencies were ‘very clear’ about the expected benefits of their programs in submissions to Cabinet, when it came to measuring whether those objectives had been achieved ‘there is a whole range of externalities that departments then mention as to reasons why it is hard and why they cannot do it et cetera’.[34]

3.33               The Auditor‑General particularly emphasised that KPIs needed to be part of the original design of a program, with expectations that they ‘will be carried through and delivered on’.[35] He expressed optimism that the Public Service was capable of developing more effective, outcomes-focused KPIs, but that ‘we just need this particular issue to be given a higher priority in program management than it has been to date’.[36] This would have implications for the allocation of resources for government programs:

Across the years, we have had different approaches to try to make sure agencies are doing this work, but the most important thing is that agencies, as part of their design and submissions to government, need to make sure they are appropriately resourced to do the full program implementation, which includes assessing the impact of programs.[37]

3.34               Finance agreed with the Auditor–General that the ‘capacity exists within government’ to develop rigorous, outcome-focused KPIs, adding that it was ‘very important to pay sustained attention over a period of time’.[38] It suggested that a broader, evaluation-based approach might be needed, rather than an approach that focused only on KPIs:

Ultimately, we have to keep focus on assessing performance and achievement rather than focusing on the tools themselves. The tools are just a means to the end. There really is nothing better than sustained attention, both within the Public Service and from outside the Public Service to the inside, to really drive this home and ensure that adequate effort is paid in this particular area.[39]

3.35               Finance also cautioned that reform would require more than just changes to the architecture of the framework. It argued that ‘consistent and concerted’ interest was needed if good quality information and time series of performance data were to be achieved, and that this was what is needed to obtain a ‘solid basis on which to assess how a program is really going’:

The system needs to value performance information. You can change the architecture of the framework and you can put in place requirements and so on, but, until there is a concerted and consistent interest from government, parliament, the public and people who are responsible for administering programs, collecting performance information, using it and making it part of the public debate about the utility of public programs and so on, it is going to be hard to get progress on this front … To be honest, the Public Service has struggled on this front and I think the political system has struggled on this front.[40]

A more flexible framework

3.36               As noted above, the ANAO report discussed a need for the performance assessment framework to accommodate the diversity of public sector organisations, recognising the wide variety of functions carried out across different agencies.[41]

3.37               At the public hearing, the Committee asked Finance whether the performance assessment framework was capable of measuring agency performance in activities that went beyond the delivery of programs, such as the quality of the advice provided to government.

3.38               Finance advised that ‘the intention is and always has been to cover the totality of government activity, the bulk of which is programs’. It added that, in addition to programs, other expectations of the Public Service—such as stewardship and advice—‘need to be assessable or assessed in some kind of way’.[42] Noting the diverse range of functions across the government, Finance described the difficulties associated with having a single framework:

What we have been learning and what we have experienced is that, if we take all of the different things the government does and all of the different expectations that people have of government entities such as Finance compared to, say, DHS, and put them all into the same conceptual framework or the same bucket, we will probably miss something there.[43]

3.39               Expanding on this topic, Finance reiterated its view that it was important to have ‘a range of tools at your disposal: not just KPIs’. It identified evaluation and review activity as other important tools, noting that the Treasury has recently undertaken an independent review of its forecasting activities, which had been publicly released.[44]

3.40               Finance also drew the Committee’s attention to the recent development of public service ‘capability reviews’, which apply a consistent framework to evaluating whole-of-department strengths and weaknesses. It noted that capability reviews, which were being conducted for every department and made public after 12 months, had become ‘quite a valued and a valuable part of the evaluative tool kit’, adding that:

They would have value as a benchmark or a reference point for looking at what the capability of that organisation was at a point in time, but more importantly the value would be enhanced if government was to come back in a couple of years’ time and say, ‘Let’s redo this and have a look at what has improved and what has not improved’.[45]

Future directions for reform

3.41               The Committee sought more information at the public hearing on how the Public Governance, Performance and Accountability Bill 2013, if passed, would impact the existing performance monitoring, reporting and evaluation framework.

3.42               Finance described the PGPA Bill as being ‘an important contribution’, but ‘certainly not a silver bullet’. It noted that the Bill, by enshrining requirements in legislation, would make performance monitoring and reporting an obligation ‘for all who work within the public sector’. This would remove the ‘level of interest’ variable that had led to inconsistent efforts in the past:

One way you can think about this area is that for the last 20 or 30 years we have relied on best endeavours and the energy and interest that individuals and organisations have had over that period. When the interest is high, then we get progress. When the interest drops off, then the quality drops off. We really need to try to break that cycle … [The PGPA Bill] is certainly not a silver bullet, because it does not deal with the skills or the capability or the systems, but it does put a minimum requirement, an obligation, in place and people then have to do something. That is the real contribution that the PGPA Bill seeks to make.[46]

3.43               Finance further explained that one of the ‘key underpinnings’ of the CFAR process was to ‘focus more on performance’, and that the PGPA Bill was one way to provide a ‘high-level emphasis’ on this. If passed, the Bill would also have the effect of giving Finance’s work on improving requirements and skills a ‘kick along’, and would provide an opportunity for Finance to look at the work the ANAO had undertaken to try and strengthen elements of the framework.[47]

3.44               Asked about how the JCPAA could best support Finance in its efforts, the department indicated that ‘sustained attention’ was a fundamental issue:

It is a willingness to come back and look at terrain that you might have traversed before and to ask: ‘What has changed there’—and perhaps not just to do that in a context where it is prompted by another follow-up ANAO report but prompted by the committee itself saying: ‘We raised some issues in the past here and there around this. Let's go back and ask the people what they are doing and what they have done’. I think sustained interest would be particularly useful.[48]

3.45               Finance also referred to an undertaking by the Minister for Finance and Deregulation to submit the rules developed under the PGPA Act (if passed) to scrutiny by the JCPAA prior to their tabling.[49] Finance suggested that engagement by the Committee on the rules for performance assessment and KPIs would be ‘particularly valued’.[50] However, it noted that, while improvements to the framework were important, attention would need to shift to the ‘people who work within the frameworks’ in order to make ‘real, genuine and sustained improvements’.[51]

3.46               In regard to its role in the framework, the ANAO advised that it was working with Finance to ensure its KPI audit processes would align with any new developments. However, it noted that ‘without visibility around the content of the rules that will underpin the [PGPA] Bill, it is not clear how some provisions of the Bill … align and interact with the Auditor-General Act 1997’.[52]

3.47               The ANAO informed the Committee that it would ‘continue to invest resources in developing and refining the approach of the systematic audit of the appropriateness of entities’ KPIs’ in 2013–14, but that after this resources would ‘become an issue’.[53] The ANAO also foreshadowed that it would be requesting ‘additional funds for 2014–15 and the forward estimates for an ongoing program of auditing performance indicators’.[54]

Committee Comment

3.48               The Committee welcomes the ANAO’s report for its timely and constructive analysis that will assist in the development of a more effective KPI framework, supported by a program of ongoing KPI audits.

3.49               The ANAO and Finance have both expressed their desire for a stronger focus on assessing performance in the public sector. The Committee welcomes Finance’s commitment to improving the performance monitoring and assessment framework, as was flagged in the department’s response to JCPAA Report 430. The Committee encourages Finance to continue to work closely with the ANAO in its development of a framework that better takes into account the diversity of Commonwealth agencies and provides meaningful and reliable indications of performance to the Government and the Parliament.

3.50               The Committee recognises that evaluating program and agency effectiveness is about more than just KPIs, and that there is a role for other mechanisms such as independent evaluations and capability reviews. However, KPIs remain an essential tool for informing these broader evaluations, for continuous reporting and for project management discipline.

3.51               The Committee also firmly agrees with Finance and the ANAO that it is not just the framework that is important. There is a need for strong leadership and sustained effort—both within the Government and in the Parliament, including the JCPAA—to ‘raise the level of expectation’ and promote a culture in the public service that values performance assessment. Indeed, high quality performance monitoring, reporting and evaluation should be more than aspirational: it should be demanded.

3.52               Towards this objective, the Committee supports the Auditor-General’s call for KPIs and evaluation to be built into the design of programs from their initial stages, and that appropriate funding is allocated for this purpose. It is also critical that agencies work together to ensure these activities actually take place—for which monitoring and reporting will be essential.

3.53               The PGPA Bill, by elevating performance assessment requirements into legislation, offers a positive step towards achieving cultural change. The Bill, if passed, also provides an opportunity to reform the existing KPI and broader performance framework, taking into account the findings and proposals in the ANAO’s report.

3.54               To fulfil part of its responsibility to provide strong leadership and sustained effort in this area, the Committee commits to ongoing engagement with Finance on the prioritised improvement of the framework, irrespective of the passage of the PGPA rules.

3.55               The Committee notes that the ANAO’s pilot project included the examination of approaches being taken in New Zealand and Western Australia, which both have existing frameworks for the audit of KPIs. The Committee suggests that there would be value in Finance and the ANAO engaging with the Australasian Council of Auditors-General (ACAG) during the redevelopment of the Commonwealth’s performance assessment framework. For example, there could be substantial benefits in terms of driving cultural change across governments and services from any promulgation by ACAG of consolidated ‘best practice’ guidance on KPI‑focused audits.

3.56               The Committee continues to support the adoption of ongoing audits of agency KPIs, and notes that the relatively small investment of resources required will be far outweighed by long term benefits. As was noted in the recent JCPAA statement to the Parliament on the 2013–14 budget for the ANAO, the Committee will closely monitor funding to the ANAO for its implementation of a full KPI audit program.[55] The Committee’s view is that additional funding should be provided for this important purpose in the 2014–15 Budget, contingent on the final outcome of the pilot, the PGPA Bill and other related developments.

 

Recommendation 1

The Joint Committee of Public Accounts and Audit recommends that the Department of Finance and Deregulation, in consultation with the Australian National Audit Office, prioritise the review and update of the performance measurement and reporting framework. A goal should be to have clear policy and guidance in place for the 2014–15 financial year that can be used by agencies to produce auditable Key Performance Indicators, irrespective of the passage of the Public Governance, Performance and Accountability Bill 2013.

 

Recommendation 2

The Joint Committee of Public Accounts and Audit recommends that: n  the Government reinforce the requirements for agencies to incorporate specific performance monitoring, reporting and evaluation activities into the design and costing of their programs; n  agencies be appropriately funded to carry out these activities; and n  monitoring be used to provide assurance that these activities are implemented.

 

 

 

Rob Oakeshott MP

Chair
June 2013