- Performance and impact measurement
Overview
5.1The Commonwealth Performance Framework, developed by the Department of Finance (Finance), states that:
- commonwealth entities and companies must report on how their performance, in achieving its purposes, will be measured, and assessed.
- performance planning and reporting should draw clear links between the entity’s key activities and the results achieved and provides meaningful performance information with a clear line of sight between planned and actual performance.
- Performance indicators allow qualified rather than absolute judgements. This allows programs to be reviewed and assessed on a variety of indicators, as few programs are unambiguous successes or failures, and Commonwealth practice requires every new policy proposal to include a plan of action for evaluation.
- The ANAO highlights that all government agencies are required to evaluate programs, taking account of materiality, policy significance and risk. In addition to specific evaluation activity, the Public Governance, Performance and Accountability Act 2013 (PGPA Act) sets out a broad system for accountability and performance management in Commonwealth entities. Establishing an appropriate performance framework up front will position an entity to assess the extent to which it is achieving the intended purpose.
- The Commonwealth Performance Framework requires entities to publish performance information that outlines how the entity’s performance will be measured, and the results of that measurement. This allows an assessment of an entity’s progress towards achieving the purposes and the objectives set by government for the funds received.
Findings
5.5The Australian National Audit Office (ANAO) listed key messages regarding performance and impact measurement that were identified as important for the successful implementation:
- To support a meaningful assessment of progress and improvements, regulators should establish clear measurables so that performance can be monitored, evaluated, and reported on an ongoing basis.
- The need for fit-for-purpose monitoring and evaluation is heightened during periods of emergency response, when rapidly introduced measures may need to be refined based on early experiences.
- Entities delivering services through multiple teams should establish and maintain quality assurance frameworks that identify systemic issues, and ensure that these issues are communicated, addressed, and monitored to provide consistent outcomes across the network.
- Successful implementation is underpinned by effective monitoring, review and evaluation processes. Effective monitoring collects timely and relevant information allows progress to be tracked towards outcomes and adjustments made, as necessary. Progress should be tracked in a deliberate and systematic manner during implementation.
Department of Health and Aged Care
5.6In Auditor-General Report No.10 2022–23, the ANAO found that the Department of Health (Health) monitored MBS billing data had no performance monitoring plan or performance measures nominated for the telehealth expansion. Health did not have an evaluation plan for temporary or permanent telehealth. It did not establish performance measures for the telehealth expansion and used MBS billing data to monitor telehealth usage patterns, on the assumption that telehealth usage and billing behaviours were sufficient indicators of successful telehealth implementation. There were no performance targets. Related performance audits undertaken by the ANAO identified weaknesses in Health’s integrity and quality controls, as third-party data sources were used to monitor and report on performance.
5.7The ANAO determined that Health did not maintain a formal process to assess the implications of this research, and the extent to which the reporting was considered by responsible officials is undocumented. The ANAO identified evidence of implementation issues through complaints and inquiries received by Health that included a lack of clarity in policy guidance on telehealth use.
Department of Education
5.8In Auditor-General Report No. 42 2022–23 the ANAO identified that the Department of Education (Education) was unable to demonstrate how the higher education data it collects is used to measure progress against achieving package level outcomes. Education undertook evaluation activities for individual programs and initiatives on an ad hoc basis but did not take a broader strategic approach to package evaluation.
5.9The ANAO audit findings concluded that Education was unable to demonstrate how higher education data is considered or used to inform progress of the access and participation program and initiative activities, or the measurement of progress against package level outcomes for regional and remote students.
5.10Education is yet to implement a structured approach for monitoring and reporting access and participation programs and initiatives against the achievement of intended package outcomes. The consequence of an unstructured approach is that assessment cannot be measured, and trending data cannot be compared to similar programs.
Department of Agriculture, Fisheries and Forestry
5.11The Auditor-General Report No.6 2022–23 identified that the Department of Agriculture, Fisheries and Forestry (DAFF) had not established arrangements to monitor and measure the intended benefits of the legislative framework for the implementation of the export control program by the commencement date. Post-commencement, there were documents outlining DAFF’s approach to benefits management. The ANAO also noted that implementation milestones had not been established to support the monitoring and reporting of benefits realisation.
5.12DAFF has export-related performance measures in its corporate plan and annual reporting, most of which relate to the objects of the Act. It has reported an ‘emerging’ maturity level under the regulatory performance framework. The Act sets out the overarching legal framework for the regulation of exported goods, including food and agricultural products from Australia. The objects of the Act include:
- to ensure that goods that are exported meet the relevant importing country requirements, comply with government or industry standards, or other requirements relating to the goods exported from Australia
- to ensure goods that are exported are traceable and, if necessary, can be recalled
- to ensure the integrity of the goods
- to ensure that trade descriptions of the exported goods are accurate
- to give effect to Australia’s rights and obligations under international agreements (in relation to the goods that are exported), to which Australia is a party.
- During a public hearing in Canberra, 2 February 2024, the ANAO outlined the requirements that need to be considered when undertaking large-scale changes, stating that:
[W]hen you’re implementing a change and you develop a rationale for the change and expected benefits, unless you start the process for how you can evaluate and determine those benefits at the beginning and start collecting data, then you’re unlikely to be able to demonstrate success or otherwise of the program further down the track…[DAFF] talked about the establishment of a benefits realisation framework, which was linked to the RIS-type activity, and then it wasn’t robustly implemented and data wasn’t collected et cetera in order to achieve that.
5.14In Auditor-General Report No. 17 2022–23, the ANAO found that the Department of Agriculture, Water and Environment (DAWE, later DAFF) did not establish performance measures that enabled it to evaluate the achievement of desired outcomes for its cultural reform program.
5.15The framework for monitoring and measuring reform progress did not provide sufficient information over the entire reform program. Progress reporting did not provide appropriate assurance over the status of the projects. Performance measures primarily focused on staff sentiment which provided some indication of the impact of the reform program. However, these indicators did not provide a complete measure of the impact of the reform program, clearly align to intended outcomes, or enable identification of significant trends in impact.
5.16DAWE did not establish appropriate assurance over the achievement of the capability transfer and uplift under the strategic and delivery partner model. Recommendation three from the ANAO report outlined the need for DAFF to establish fit-for-purpose performance criteria to support evaluation and provide a basis for assessing the performance of its current or any future reform program. DAFF agreed.
5.17In support of the findings made by the ANAO regarding auditability of non-financial performance, specifically relating to performance and impact measurement requirements, ANAO articulated the shortcomings with the PGPA Rule in a public hearing in Canberra, on 2 February 2024, highlighting that:
The PGPA Rule, which creates the framework to enable the parliament to look at the non-financial performance information, isn’t set up for auditability. It’s set up for the measures to assist the parliament understand the performance of the entity. One of our challenges, as we continue to roll out the performance statements auditing, is people doing it because of the audit, not because it is good practice. What we are hoping to see is some of the deficiencies that we identify through the auditing actually leading to better performance statements, not just auditable performance statements, so that they’re meaningful to the parliament.
Agency Responses
Department of Health and Aged Care
5.18Health addressed recommendation one from the ANAO in its submission to the inquiry, which included Health needing to improve planning and performance monitoring and evaluation. Health outlined in its agreement to this recommendation a progress update, noting that it has a project underway and the benefits, specifically that:
The outcome from this project will assist MBS policy staff with documentation for the management of implementation and integrity risks, and mechanisms for oversight, performance monitoring, and evaluation.
5.19The Committee questioned Health at the public hearing in Canberra, 2 February 2024, about what Health is doing to improve performance monitoring and evaluation for the permanent telehealth expansion. Health provided an update to the Committee, stating that:
[W]e have a Medicare review advisory committee, an independent group, that is currently evaluating telehealth to give advice to government. These are things that we’ve done in the last two years, where we’ve actually looked at how we do formalise evaluation and review of all MBS changes, and we bring forward to the independent group things that are also significant to review and seek advice and submissions on.
5.20Health went on to clarify the specific address of impact measure, highlighting the shortfalls of the MBS, stating that:
In terms of measures of the effectiveness of telehealth, I think it’s important to note that an MBS item is a rather blunt item. It tells us if a service is being delivered. We don’t necessarily know what happens in a consultation with a GP. The data that we get out of the MBS essentially tells us that this service was delivered by this provider to that person.
Department of Education
5.21The ANAO’s first recommendation to Education was to establish a methodology to measure the performance of programs and initiatives that contribute to access and participation for regional and remote students, and their progress in achieving Strategy outcomes. Education agreed.
Education [is] reviewing its Performance and Data Framework covering Napthine Review measures to incorporate further relevant higher education programs and initiatives that improve access and participation outcomes for regional and remote higher education students.
5.22The Committee revisited Education’s address of the ANAO recommendations in the public hearing held in Canberra, on 2 February 2024. The main concern highlighted was the repetition of issues previous ANAO reports had shown a pattern of being unable to demonstrate success of implemented programs. Education responded by stating that:
…[W]e’ve agreed with the recommendation that we did need to step back from the individual initiatives and have a more joined up view of various programs and initiatives to be able to then draw on the data that we have in the department to start to measure where those initiatives were making a difference.
5.23Education continued its address to the Committee and highlighted the steps it’s taking in response to the ANAO findings. Education also provided updates on the progress of current initiatives and advised that it is too early to adequately measure impact, but that the data being regularly collected and reviewed is showing where impacts have been made.
5.24In response to the ANAO’s findings in paragraph 5.10, Education confirmed that it has developed and finalised a Performance and Data Framework Plan and a Stakeholder Engagement Plan, to address recommendations one and two from the ANAO audit. Education provided a detailed response, stating that:
- the Performance and Data Framework and Stakeholder Engagement Plan are the only tools specifically developed in response to the audit
- the Performance and Data Framework sets out the measurement methodology at both package and individual measure levels, and the related data requirements, assurance processes, risk management and evaluation activities that will be monitored and reported on by the department. Under the framework, a report will be prepared annually and provided to the senior executive, covering all measures, and considering the way they operate as a package. The Stakeholder Engagement Plan supports the implementation of the Performance and Data Framework by identifying key stakeholders and outlining the timing of engagement.
Department of Agriculture, Fisheries and Forestry
5.25The Committee questioned DAFF in a public hearing, in Canberra on 2 February 2024, about its claims of benefits that would result from the major legislative change and changes in the regulatory framework, but that DAFF was unable to demonstrate any benefits. DAFF addressed the Committee’s concerns, stating that:
We have benefits we were trying to measure through our reform package and through the legislation at the same time. We were trying to work out how we combine that from how we actually, as a regulator, measure our performance, of which the legislation is one part of…[W]e actually have a better system as an agency around performance reporting more generally.
5.26DAFF continued in its response to the Committee, highlighting that:
As part of the benefits management approach, the department, and the Agricultural Trade Group in particular, have developed a portfolio benefits management strategy thorough the former trade reform portfolio office. This is consistent with Department of Finance guidelines. These benefit management arrangements have been fully implemented within the agricultural trade group.
…
In relation to the legislation itself, we have implemented ongoing monitoring and reporting of regulatory performance under the Export Control Act 2020. It includes external reviews, such as by the Inspector-General of Animal Welfare and Live Animal Exports.
5.27In its submission to the inquiry, DAFF provided a brief review of its response to recommendations received by the audit report. The response provided little detail regarding how DAFF was going to improve its frameworks and process, stating that:
The department is developing an action plan in response to both this audit and the recent Australian Public Service Commission (APSC) Capability Review.
5.28During a follow up hearing with DAFF, the Committee discussed the changes that DAFF has implemented in response to the ANAO reports. DAFF highlighted the improvements it has made regarding improvements to performance and impact measurement, stating that:
There’s a requirement now for a project plan chaired by a steering Committee that defines the problem, the objective, the activities, the benefits’ risks for each priority action—so very much picking up those ANAO recommendations—we’ve had monthly monitoring and reporting to the steering Committee and quarterly reporting to the department’s executive boars…on progress in implementing those actions and activities.
Committee Comment
5.29Regarding Health’s telehealth expansion, both the temporary and permanent expansion should have had performance and impact measurement as a key framework to ensure appropriate decision-making occurred based on evidence and data collected in response to the rapid implementation and changes to the MBS. The Committee would emphasise how invaluable performance measurement and the data created is in driving good decision-making practices and in adjustments to programs. Even in urgent circumstances such as temporary expansion, if it is not possible to finalise performance and impact measures immediately upon implementation, then they should be completed as quickly as possible.
5.30The Committee requests that the Department of Health and Aged Care reports back on improvements to performance and impact measuring related to the telehealth expansion MBS items within 12 months of the date of this report.
5.31The Committee requests that the Department of Education reports back on the implementation of its Performance and Data Framework Plan and Stakeholder Engagement Plan within 12 months of the date of this report.
Hon Linda Burney MP
Chair
12 September 2024