Chapter 3


Navigation: Previous Page | Contents | Next Page

Chapter 3

Australian Crime Commission performance measurement

3.1        This chapter considers the ACC's performance against its Key Performance Indicators (KPIs). It examines the measurement tools utilised to inform the KPIs and to track performance over time.

Key Performance Indicators

3.2        The Outcomes and Outputs Framework (the framework) provides the basis for the government's approach to budgeting and reporting for public sector agencies and the means by which the Parliament appropriates funds in the annual budget context.[1] In 2010, the Joint Committee on Public Accounts and Audit observed that measuring key aspects of an agency's performance is a critical part of the Government's Outcomes Framework.[2] Within the context of the framework, KPIs are 'established to provide information (either qualitative or quantitative) on the effectiveness of programs in achieving objectives in support of respective outcomes'.[3]

3.3        The Department of Finance (Finance) has provided the following guidance for agencies in developing KPIs:

Agencies should focus on reporting a strategic and meaningful level of performance indicators, demonstrating the link between the program performance indicators and the outcome.[4]

3.4        Agencies are required to provide a relevant, informative and useful range of performance indicators that can be tracked over time. In advice to entities on developing KPIs, Finance recommended that agencies use both qualitative and quantitative information to measure program performance and provided the following definitions:

Quantitative: This type of reporting is represented by numbers or percentages in a table.

Qualitative: This type of reporting is represented by narrative text. Agencies should identify aspirational goals or milestones that are intended to be achieved by the program.[5]

3.5        Finance noted that KPIs must be designed to be 'capable of signalling to government, Parliament and the community whether programs are delivering intended results'.[6] Further, consistent, clear reporting on performance provides an important record of an agency's 'progress towards meeting government policy objectives, how well public money is being spent and whether planned achievements are on track'.[7]

3.6        A performance audit report from the Australian National Audit Office (ANAO) titled Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework emphasised the importance of an 'appropriate mix of qualitative and quantitative KPIs including targets against which progress towards program objectives could be assessed'.[8] The ANAO noted that a tendency to rely on qualitative KPIs reduces the ability of an agency to measure the results of program activities over time. Whereas:

A mix of effectiveness KPIs, that place greater emphasis on quantitative KPIs and targets, would provide a more measurable basis for performance assessment.[9]

3.7        The ANAO argued that because KPIs are statements of the pre‐defined and expected impacts of a program, it is important that they are:

  • specific—so as to focus on those results that can be attributed to the particular intervention/program;
  • measurable—include quantifiable units or targets that can be readily compared over time;
  • achievable—realistic when compared with baseline performance and the resources to be made available;
  • relevant—embody a direct link between the program’s objective and the respective effectiveness KPI; and
  • timed—include specific timeframes for completion.[10]

ACC's outcome and program structure

3.8        The ACC's outcome and program structure are set out in its annual report:

Outcome 1

Reduction in the threat and impact of serious and organised crime, through analysis of and operations against national criminal activity, for governments, law enforcement agencies and private sector organisations.

Outcome strategy

Collaborate with law enforcement and related government agencies and private industry as part of the national effort against serious and organised crime.[11]

3.9        The ACC's single outcome is underpinned by two programs:

  • Program 1.1.1—Strategic criminal intelligence services, the performance of which is measured by two KPIs; and
  • Program 1.1.2—Investigations and intelligence operations into federally relevant criminal activity, which are measured by six KPIs.[12]

Program 1.1.1

3.10      Program 1.1.1 has a set of deliverables to meet the overall aim that:

The ACC's criminal intelligence services are designed to provide Commonwealth, state and territory law enforcement and relevant government agencies with the criminal intelligence necessary to effectively and efficiently disrupt serious and organised criminal activity and reduce the vulnerabilities posed to the Australian community.[13]

3.11      The following table lists Program 1.1.1's KPIs, their targets and results for 2012-13.


Program 1.1.1—ACC KPIs and performance 2012-13[14]

KPI

2012-13 target

2012-13 result

1.    Strategic intelligence products align with ACC Board endorsed National Criminal Intelligence Priorities.

90%

100%

2.    Partner agencies agree or strongly that the ACC's intelligence enhances their understanding of serious and organised crime.

90%

90% of senior executives of partner agencies agree/strongly agree

3.12      In relation to the first KPI, the annual report noted that a combined 1866 analytical and tactical intelligence products were produced during 2012-13. Of these products, 100 per cent align with the National Criminal Intelligence Priorities (NCIPs), including all 16 of the ACC strategic products in the Picture of Criminality in Australia suite.[15]

3.13      The second KPI is measured by stakeholder feedback and the explanatory note in relation to it states that:

Each partner agency has its own role and priorities, and each has different needs for and uses of criminal intelligence. We continue to work with our stakeholders to deepen our understanding of their needs and improve the value of our intelligence from their perspective.[16]

Program 1.1.2

3.14      The overall aim of program 1.1.2 is as follows:

The ACC's investigations and intelligence operations underpin its criminal intelligence services by providing unique intelligence collection capabilities. ACC investigations are conducted in partnership with law enforcement agencies with the objective of disrupting and deterring federally relevant serious and organised criminal activity. In 2012-13, the ACC, under the guidance of its Board, will further focus its coercive powers determinations to more comprehensively address emerging issues in the organised crime environment.[17]

Program 1.1.2—ACC KPIs and performance 2012-13[18]

KPIs

2012-13 result

1.    Partner agencies agree, or strongly agree, that the ACC's contribution enhances efforts to combat serious and organised crime.

100% of managers and senior executives agree

(target was 80%)

2.    Partner agencies agree, or strongly agree, that the ACC's coercive powers are effective.

80%

(target was 90%)

3.    Targeted ACC investigations and operations are aligned with ACC Board priorities and approved by the ACC Board.

100% of senior executives of partner agencies agree/strongly agree

4.    The ACC's operational intelligence and contribution to joint intelligence investigations and operations enhance the efficiency and/or effectiveness of law enforcement efforts to disrupt and deter serious and organised crime, as measured by stakeholder feedback.

100% of senior executives of partner agencies agree/strongly agree

5.    The activities of targeted criminal entities are disrupted as a result of ACC intelligence, investigations and operations, and activity is undertaken to confiscate proceeds of crime.

26 disruptions

185 people charged

332 charges laid

38 people convicted

$83.87 million proceeds of crime restrained

$26.02 million proceeds of crime forfeited

$159.98 million tax assessments issued

34 firearms seized

$16.34 million in cash seized

$589.52 million estimated street value of drugs seized

$84.74 million value of the illicit drug production potential of precursors seized.[19]

6.    Availability of the Australian Criminal Intelligence Database and the Australian Law Enforcement Intelligence Network.

>99%

(target was 98%)

 

3.15      The availability of ACID and ALEIN achieved greater than 99 per cent in 2012-13 and 2011-12. In  2012-13, there were 1 302 439 searches on ACID compared to 331 664 in 2011-12, a fourfold increase.[20] The ACC noted that this was 'due primarily to increased multiple-entity searches by the ACC-led National Criminal Intelligence Fusion Capability'.[21]

3.16      The committee notes that the government has responded to the recommendation made in its last report[22] that the ACC review and re-examine its KPI concerning the ACID and the ALIEN. In response, the government stated that it accepts the recommendation and that the ACC is currently undertaking a scoping study to determine a replacement system for ACID and to improve ALEIN. In addition, the ACC has provided a new KPI for ACID and ALEIN. The performance of ACID and ALEIN will now be measured against 'provision of a national criminal intelligence database and analytical tools, which facilitate the sharing and analysis of criminal intelligence across jurisdictions.'[23]

3.17      The committee will carefully examine the ACC's 2013-14 Annual Report with a view to determining the extent to which the change in KPI has resulted in improved performance information.

Challenges in measuring ACC performance

3.18      The annual report highlights some of the complexities in measuring ACC results. It notes that most of the work undertaken by the ACC is classified, or has long-term results often being realised years after the ACC's initial involvement, and once court decisions, law reforms and changes in community behaviour have concluded or been implemented.[24]

3.19      During its examination, the committee asked the ACC to comment on the difficulties surrounding the measurement of the performance of the ACC, as much of the work undertaken by the ACC is utilised by other agencies. Mr Jevtovic stated that the ACC is eager to demonstrate the worth of investment in the organisation for crime fighting, but that it is difficult to monitor how much of the information is used:

...we might produce 2500 information reports, which we disseminate to Australian law enforcement over the country. Once we disseminate that information, there is no technical solution to monitoring which component of our 2500 products has actually been accessed or used if the people using it do not tell us...that is the reality and we accept that.[25]

3.20      However, the level of reporting by the ACC was improved through changes made to the ACC's Performance Measurement framework, as recommended by the committee in its last report.[26] This involved adapting the framework to:

  • more accurately reflect both quantitative and qualitative results—capturing short-term and long-term, direct and indirect impacts of activities; and
  • measure the performance against the new Strategic Plan 2013–18, which highlights the evolution of serious and organised crime and the need to continually adapt and develop the right capabilities to respond.[27]

3.21      The annual report notes that these changes have had a positive outcome on performance:

Our revised indicators enable us to better self-assess our performance, to complement feedback from our stakeholders which we intend to gather more regularly throughout each reporting year. This will enable us to report in new and different ways on the value we add to the national fight against serious and organised crime.[28]

3.22      The revised performance management framework will require the ACC to design new systems for collecting performance data, which will be implemented over the course of the five year Strategic Plan. The committee will examine the ACC's 2013-14 Annual Report with a view to determining the efficiency of the new systems for collecting and reporting on performance data over the course of the Strategic Plan.

Stakeholder surveys

3.23      The annual report notes that stakeholder relationships are critically important to the ACC's success in combating serious and organised crime. The ACC uses stakeholder surveys to help inform ACC reporting obligations. In 2012-13, the ACC revised its stakeholder research methodology. Rather than engaging market researchers to gather data through stakeholder survey forms, the ACC conducted its research in-house with a wider range of qualitative interviews. According to the ACC, the result provided 'more in-depth information for analysis, [and] a more focused indication of how [the ACC] performed', while reducing costs.[29]

Committee view

3.24      The committee appreciates the complexities involved for the ACC in developing meaningful qualitative KPIs that can be measured over time. The committee also acknowledges that the nature of some of the ACC's work may not be directly quantifiable.

3.25      In its previous report the committee recommended that the ACC Annual Report 2012-13 should include information on progress made towards establishing a balance of quantitative and qualitative KPIs, which can be measured over time.[30] The committee notes that the government has accepted this recommendation,[31] and that the ACC has amended its key performance indicators for 2013-14 for both its Portfolio Budget Statement and its annual report.[32]

3.26      The new KPIs for outcome 1 include:

  • the ACC produces useful intelligence that identifies and provides insights on new and emerging serious and organised crime threats.
  • the ACC fills intelligence gaps through the identification of vulnerabilities and indicators of serious and organised crime.
  • the ACC collects and maintains national intelligence holdings of serious and organised crime threats and targets.
  • the ACC interprets and analyses national holdings to create a national serious and organised crime intelligence picture.
  • the ACC informs and influences hardening of the environment against serious and organised crime;
  • the ACC influences or enables the disruption, disabling or dismantling of serious and organised crime; and
  • the ACC participates in or coordinates collaboration in joint investigations and operations to prevent and disrupt serious and organised crime.[33]

3.27      The committee notes these new KPIs and will closely examine the relevant information included in the ACC's 2013‑14 annual report, to ensure that it provides a more measurable basis for performance assessment.

Navigation: Previous Page | Contents | Next Page