Chapter 24
Evaluation and accountability
24.1
This chapter is based on the premise that evaluation improves
performance. The committee starts by looking at the importance of evaluating a
peacekeeping operation. It then describes the approaches taken by the ADF and the
AFP to assessing their performance in peacekeeping activities. The committee
considers the performance indicators needed by government to assess adequately the
success or otherwise of an operation and the challenges of ensuring that
lessons learnt from an operation are captured for future operations. Finally, the
committee looks at the evaluation of a peacekeeping operation as an important
accountability tool.
Evaluating peacekeeping operations
24.2
Evaluation is used to improve performance in a number of important ways.
It provides feedback to decision makers so that they are better able to assess the
objectives of a mission and, if required, change the mission's mandate and make
informed and necessary adjustments to the conduct of the operation. Such assessments
may also identify deficiencies in training and resources and can be used to make
immediate improvements to the peacekeepers' preparation or equipment. In the
longer term, evaluations add to the body of knowledge and understanding of
peacekeeping operations and are central to developing best practice doctrine
and procedures. Finally, evaluations are an important accountability tool
whereby those responsible for an operation are answerable for the conduct of
the mission.
Agencies' current evaluation practices
24.3
The committee believes that although peacekeeping is a
whole-of-government undertaking, government agencies have a responsibility to
assess their own performance.[1]
In the committee's view, self assessment enables an agency to identify the
strengths and weaknesses in its performance and provides the necessary impetus
for change and improvement. It follows that agencies should have 'mechanisms in
place to measure [their] effectiveness'.[2]
Both the ADF and the AFP are aware of the importance of evaluating their
performance in peacekeeping operations.
ADF
24.4
According to the ADF, it develops its training objectives through incorporating
lessons learnt, both from its own experience and that of other countries.[3]
It uses a 'Defence-wide evaluation system' to capture and disseminate those
lessons. The ADF Activity Analysis Database System covers 'all ADF operations
and major exercises and has been in operation for the last decade'.[4]
24.5
The ADF informed the committee that it has operational analysis teams of
two to three personnel, from the Defence Science and Technology Organisation
(DSTO) and the ADF. They 'deploy regularly to identify and record lessons'.[5]
The teams consider lessons from ADF operations, from a Defence perspective, but
also in cooperation with other government agencies. The reporting is passed up
the chain of command as appropriate.[6]
The ADF explained further:
...the ADF Warfare Centre and the Centre for Army Lessons act as repositories
for ADF lessons learned. Previous peacekeeping operations are analysed, and
lessons applied to doctrine. These are also incorporated into standard
operating procedures and result in changes to tactics, techniques and
procedures. In addition, these lessons inform consequent mission rehearsal
exercises for elements preparing to deploy or replacing elements already
deployed.[7]
24.6
The post-deployment debriefing process also provides an opportunity for
the ADF to obtain feedback from their peacekeepers in order to improve
procedures and operations.
24.7
It should be noted, however, that the committee received evidence
suggesting there were shortcomings in the ADF's debriefing process. Captain Wayne
McInnes, who served with ATST-EM, claimed that the team received no debriefing
in East Timor. In his words, they basically prepared their 'kit for inspection
by AQIS and 48 hours later were out of the country'. On return to Australia,
there was no debrief either. He explained, 'There were no observations, lessons
learnt, how can we do things better or what went wrong'.[8]
24.8
The committee discusses more fully the ADF's approach to lessons learnt
later in this chapter.
AFP
24.9
As noted in Chapter 21, the AFP mission debrief is mandatory. It is
conducted through an anonymous questionnaire covering three stages of
deployment.[9]
The committee has also referred to the joint project funded by the AFP and the Australian
Research Council, 'Policing the Neighbourhood'. One of the key objectives is to
develop 'an analytical framework by which Australian police assistance missions
can be better understood, assessed' and modified to 'contribute effectively,
equitably and sensitively to the improvement of law and order in host
countries'. The AFP is also collaborating with the University of Queensland in 'the
development of performance measures to assist in evaluating AFP contributions
to peace and stability operations, and capacity development missions'.[10]
24.10
These projects indicate the willingness of the AFP to address seriously
the matter of evaluation. This approach is particularly important in light of
the literature that points to deficiencies in the way peacekeeping operations
are evaluated. The committee now looks at the difficulties organisations have
in determining their effectiveness in a peacekeeping operation.
Performance indicators
24.11
In its report on public diplomacy, the committee discussed the
difficulties measuring the effectiveness of government programs. For example, Mr
Meert, ANAO, noted that because it is easy to measure, 'a lot of the agencies
are stuck at the activity measure' and hence 'struggling with how to determine
effectiveness'.[11]
He suggested that a range of indicators are needed to ascertain whether the
activities being undertaken are 'having the desired effect'. He noted that
there are methods available to measure changes in attitudes or perceptions. He
suggested, however, that 'One indicator on its own may not give you the result
but a range of indicators may give you that indication'.[12]
Based on the advice from ANAO, the committee noted that if ANAO were to
undertake a performance audit, it would likely concentrate on the performance
indicators a department uses to evaluate the effectiveness of its programs and
how it sets targets.[13]
ANAO would be looking to see whether a department has the mechanisms in place
to evaluate its own programs.
24.12
This observation on having performance indicators is particularly
pertinent for peacekeeping operations. The United States General Accounting
Office noted in 2003 that the UN DPKO acknowledges that it 'needs better
indicators by which to measure the progress peacekeeping operations are making
in attaining sustainable peace. It stated:
Although some measures for the peacekeeping operations are
quantifiable [include minimising and containing cease-fire violations and
opening roads and removing roadblocks], the United Nations faces challenges in
developing results-oriented measures about conditions in the country that the
peace operations are supposed to improve.[14]
24.13
It stated further:
Although U.N. missions are using measures of progress for their
operations, most measures are tasks and outputs rather than measures of
underlying conditions in the country that the peace operation is to improve.[15]
24.14
For example, in East Timor a key objective was to increase the capacity
of the national police force to provide internal security. The US General
Accounting Office noted, however, that the number of police 'does not measure
the quality of their training and whether they are improving security in the
country'.[16]
It warned against relying on measures that are process oriented and do 'not
measure changes in country conditions that the peacekeeping operations were
working to improve'.[17]
24.15
When considering exit strategies in Chapter 8, the committee made
similar findings about having performance indicators that are credible and
useful. It noted that an exit strategy should relate back to the objectives set
out in the mandate which must be clearly defined, realistic and attainable and
based on a sound understanding of all facets of the problem. The committee
found further that the strategy should contain milestones against which the outcomes
of peacekeeping and peace enforcing measures can be assessed. These benchmarks,
however, should be more than indicators of 'technical' achievements and while
identifying key attainments such as an election, should also take cognizance
of, and mark progress toward, the ultimate goal of sustainable peace.[18]
24.16
Sergio Vieira De Mello, Special Representative of the Secretary-General
and Transitional Administrator for East Timor, also used East Timor as an example
to highlight the need for substance in determining milestones. He noted that
success would be judged not just on the number of schools rebuilt or roofs
replaced. The yardstick would be the extent to which the mission assists the
people of East Timor to fully realise 'their independence as masters of their
own future' and their own 'independent State'. Using what are called 'technical
achievements' may give a false reading of the situation.[19]
24.17
In its consideration of exit strategies, the committee also noted the
danger of relying on indicators that do not provide a complete assessment of
the operation. The committee acknowledged that most observers agree that the
ADF pacification in Solomon Islands was a success with over 6,000 militiamen
arrested, over 9,000 charges laid and more than 3,000 guns confiscated. It went
on to cite the findings of a recent report that concluded that, while civil
stability had brought security, these gains would 'prove temporary if the
underlying economic stagnation that led to civil unrest is not addressed'. Thus,
while measures such as the number of local police trained, schools rebuilt or
weapons confiscated are quantifiable, they may mask the fact that deep-seated
conflicts remain with the potential to flare up and return violence and
instability to the country.[20]
24.18
The committee notes that Assistant Commissioner Paul Jevtovic in an
interview stated that one of the indicators the AFP uses to measure its
performance is 'how we develop our counterparts in that country [Solomon Islands],
so our success is ultimately measured on the development of good future leaders'.[21]
This benchmark is an improvement on just quantifying the number of police
trained and is looking to measure change that indicates real achievements have
been made. The question remains, however, what indicators the AFP would use to
determine 'the development of good leaders'. Having credible and useful
performance indicators is critical to effective evaluation but they are also
difficult to measure.
Committee view
24.19
The committee believes that to be effective, performance indicators must
anticipate the difficulties of achieving sustainable peace. They should not be
tasks or outputs but rather objective result-oriented measures that effectively
plot progress toward achieving the objectives of the mission. In this way, they
provide the necessary feedback to determine whether there is a need to change
the objectives and conduct of the mission and what these changes may be. The committee
accepts, however, that formulating performance indicators that measure shifts
in attitude or fundamental changes in a society is not easy. Nevertheless, the committee
believes that it could and should be done.
24.20
Another important reason for evaluating performance is to build up a
body of knowledge and understanding that can be applied to future operations.
Capturing lessons learnt
24.21
The UN has for sometime been concerned that it did not have a process
'for converting lessons into policies and procedures that could guide
subsequent operations'. In a report to the General Assembly in 2007, the Secretary-General
stated:
To be 'learned', lessons identified need to be validated and
endorsed by the Organization in the form of standardized guidance materials,
and guidance needs to be disseminated and its implementation monitored. The
identification of lessons and good practices does not itself lead to
performance improvement. It must be accompanied by a process that seeks to
implement improvements in the way operations are managed and conducted, which
must also be monitored and re-evaluated on an ongoing basis.
...Learning lessons may be a natural process, but sharing and
implementing lessons across different missions is not. Although most staff
members say that they regularly engage in learning lessons, unless systematic
efforts are made to document and share those lessons, their impact remains
limited to local teams.[22]
24.22
As noted earlier, the ADF asserted that it incorporates lessons learnt
from missions to develop its training objectives. It has operational analysis
teams that deploy regularly to identify and record lessons, and the Warfare
Centre and the Centre for Army Lessons act as repositories for ADF lessons
learnt. In this way, previous peacekeeping operations are analysed, and lessons
applied to doctrine.
24.23
Professor Raymond Apthorpe and Mr Jacob Townsend, however, doubted the efficacy
of the ADF's learning lessons process. In general, they noted that many
organisations 'tend to misunderstand the phrase "lessons learnt"':
At the end of a peacekeeping operation, identifying problems and
summarising these under the title 'lessons learnt' obscures the otherwise
obvious point that the lesson has yet to be learnt—or the problem would not
have occurred. Easy access to in-house evaluations of Australian Defence Force
(ADF) peacekeeping operations eluded us, so we will restrict ourselves to the
general point that identifying problems is easy; the hard part is making
real and lasting change.[23]
24.24
Professor Apthorpe and Mr Townsend therefore suggested that the committee
investigate 'the degree to which [organisations] have learned any lessons from
their "lessons learnt"'. They continued:
A warning sign would be the same issues re-appearing in many
lessons learnt sections of evaluations. In our experience, it is not uncommon
that an organisation seems to be cutting and pasting 'lessons learnt' from one
evaluation to the next, which indicates dysfunction in its learning processes.[24]
24.25
Furthermore, in their view, 'lessons are often not presented in a
learnable form' and 'expressed in a way that cannot inform future policy or
strategy'. They argued:
Like proverbs, two equally sensible suggestions can be in
direct contradiction...Like proverbs, these 'lessons' are post facto
comments that cannot prepare us for the future unless we place them in context.
If organisations are to learn from their experiences, then the 'lessons
learnt' process needs to identify contributing contextual factors that can be
recognised in new scenarios, so that something learnt in one context might
be transferable to another.[25]
24.26
The committee believes that it would be of value to have an audit of the
ADF's operational analysis teams, the Warfare Centre and the Centre for Army
Lessons to determine the effectiveness of their work in capturing the lessons
learnt from current and recent peacekeeping operations.
Request to Auditor-General
The committee requests that the Auditor-General
consider conducting a performance audit on the mechanisms that the ADF has in
place for capturing lessons from current and recent peacekeeping operations
including:
- the adequacy of its performance indicators;
- whether lessons to be learnt from its evaluation processes are
documented and inform the development or refinement of ADF's doctrine and
practices; and
- how these lessons are shared with other relevant agencies
engaged in peacekeeping operations and incorporated into the
whole-of-government decision-making process.
24.27
The committee has confined this request to the ADF because, as noted
earlier, the AFP has commissioned the University of Queensland to develop
performance indicators. An ANAO audit should provide guidance for the AFP, and
indeed for all relevant government agencies, in further developing their performance
indicators. The audit should also provide an insight into how effectively
government agencies are sharing information from lessons learnt and
contributing to improving Australia's whole-of-government performance in
peacekeeping activities.
Recommendation 34
24.28
The committee recommends that the relevant government agencies jointly
develop standard measurable performance indicators that, where applicable,
would be used across all agencies when evaluating the effectiveness of their
peacekeeping activities (also see Recommendation 36).
Whole-of-government evaluation
24.29
The government also confronts the challenge of devising effective
performance indicators when assessing its whole-of-government performance in a
peacekeeping operation. The task of measuring the effectiveness of the
whole-of-government contribution means examining issues such as
interoperability and CIMIC. Such matters raise questions about who should
assess this overall performance and how an agency's assessment of its
performance feeds into a whole-of-government appraisal. Having credible and useful
performance indicators is also important. Ensuring that information is shared
and lessons learnt across government agencies poses problems for the
government. Also, the government must decide who is to monitor and ensure that
the lessons learnt are implemented.
24.30
In this regard, the committee believes that the proposed peacekeeping
institute would have a vital role in the evaluation and continuous improvement
of Australia's peacekeeping performance. It believes that the institute is the
ideal mechanism for ensuring that Australia has:
- appropriate performance indicators to measure the success or
otherwise of its whole-of-government performance in peacekeeping activities;
- a repository for lessons learned; and
- a central body responsible for ensuring that doctrine and
practices are developed and refined in light of past experiences.
24.31
The establishment of the institute would not in any way counter or make
redundant the work on peacekeeping of the ANAO, should it undertake the audit,
or of projects such as that underway by the University of Queensland. The
institute would complement and indeed add value to the findings of such
organisations. The committee believes that it could play a vital role in
building a culture of learning and continuous improvement.
Independent assessment
24.32
The committee also notes the importance of obtaining independent outside
evaluation as a means of gauging the performance of Australian peacekeepers and
the success of a peacekeeping operation.[26]
Such information comes from the government and people of the host country as
well as Australia's partners in an operation. The proposed institute could also
take responsibility for gathering this type of information.
Accountability
24.33
Accountability lies at the heart of efforts to
improve the government's management of its peacekeeping activities. Not only
does it provide a means for parliamentary and public scrutiny of government
expenditure in this area but it also allows people outside the executive
government to make an informed assessment of the government's performance.
In this way, Australians may not only feed into the overall evaluation of Australia's
performance in peacekeeping operations but they are better placed to hold the
government answerable for its performance.
Annual reports
24.34
The starting point for examining Australia's involvement in peacekeeping
is to understand the nature of the contribution that the government makes
through each of its departments and agencies. It is difficult to find
comprehensive information about Australia's whole-of-government commitment to
peacekeeping operations. Defence provides clear information in its annual
report and on its website about current international deployments, including
each operation's aims, the number of personnel and other capabilities deployed.[27]
However, Defence does not distinguish peacekeeping operations from other
deployments. Its annual report lists operations under the headings 'Operations
contributing to the security of the immediate neighbourhood', 'Operations
supporting wider interests' and 'Peacetime national tasks'.
24.35
Referring to the Defence Cooperation Program (DCP), outlined in Chapter 20,
the committee is concerned that Defence's Annual Report 2006-2007
offered no examples of the expenditure in the Defence Cooperation Program's $80
million (approx.) capacity-building activities.
24.36
Similarly, the AFP provides information about its international missions
on its website and in its annual report.[28]
The latest AFP Annual Report has been enhanced by the inclusion of a snapshot
providing a summary of the number of personnel deployed to each operation as at
the end of the financial year.[29]
The AFP also does not identify deployments as 'peacekeeping operations': its
snapshot distinguishes missions as 'United Nations missions', 'AFP Capacity Building
missions' and 'Regional Assistance missions'.
24.37
Several other government departments refer to their involvement in peacekeeping
operations in their annual reports. There is, however, no centralised source
providing comprehensive information about the whole-of-government contribution.
DFAT's annual report provides an overview of its activity with relevant
countries or regions in support of its Outcome 1: 'Australia's national
interests protected and advanced through contributions to international security,
national economic and trade performance and global cooperation'.[30]
In this context DFAT reports, for example, that it 'continued to coordinate Australia's
whole of government contribution to the Regional Assistance Mission to Solomon
Islands (RAMSI)'.[31]
24.38
In response to the conclusions of an ANAO report, DFAT's 2006–07 annual
report also lists the government agencies contributing to RAMSI. This is an
improvement, but falls short of the ANAO's suggestion, which was to include at
a minimum each agency name, its contribution in financial and human resource
terms and a brief description of its role.[32]
24.39
In this inquiry, the committee considered Australia's involvement in a
number of current peacekeeping operations, not only RAMSI. In response to a
question as to whether the ANAO's recommendation would be applied to other
operations with multi-agency input, DFAT responded that this 'will depend on
the nature of the individual operation'.[33]
24.40
Despite the inadequacy of easily accessible information, it is clear
from the nature of the missions and their mandates that Australia's
contribution to peacekeeping operations has extended well beyond military and
police involvement. In an answer to a question on notice, DFAT informed the committee
that the following departments and agencies have contributed staff to peace
operations in Bougainville, East Timor or Solomon Islands: Defence, AFP, DFAT,
Attorney-General's Department, AusAID, Australian Electoral Commission,
Customs, Finance, Office of Financial Management, National Archives, Treasury
and Department of Veterans' Affairs.[34]
Committee view
24.41
Australia's contribution to peacekeeping operations now extends well
beyond the military. It is important that this whole-of-government contribution
is accompanied by whole-of-government reporting, so that the Parliament and the
Australian public can identify the size and nature of the resources allocated
by government to peace operations.
Recommendation 35
24.42
The committee recommends that the Australian Government designate an
appropriate agency to take responsibility for the whole-of-government reporting
on Australia's contribution to peacekeeping. This means that the agency's
annual report would include a description of all peacekeeping operations, a
list of the contributing government agencies, and, for each relevant agency:
- a description of its role in the operation;
- the agency's financial contribution to the operation during that
reporting year;
- the peak number of personnel deployed by the agency during the
reporting year and the date at which the peak occurred; and
- the number of personnel deployed as at the end of the reporting
year.
Recommendation 36
24.43
In light of the committee's discussion on the adequacy of performance
indicators, the committee also recommends that the agencies reporting on
peacekeeping activities provide in their annual reports measurable performance
indicators on the effectiveness of these activities.
24.44
Following on from this consideration of how the government reports its
whole-of-government contribution to peacekeeping, the committee notes that
there appears to be a similar lack of coherence in explaining the government's
policy on peacekeeping.
White paper on peacekeeping
24.45
The committee is of the view that the changing nature and expanding
scope of Australia's involvement in peacekeeping operations have profound
implications for Australia as a member of the UN and a long-time contributor to
peacekeeping missions. Further, peacekeeping operations are no longer the
domain of the military, with a range of government agencies and NGOs now
needing to work together as an integrated team to achieve the mission's
objectives. Peacekeeping operations can also be costly and dangerous
undertakings, with failure a real prospect. Moreover, international doctrine and
practice on issues such as the responsibility to protect, the significance of
exit strategies, interoperability, CIMIC and women and peacekeeping, to name
just a few, continues to develop and poses challenges for policy makers.
24.46
These developments have a direct bearing on the formulation of Australia's
policy on peacekeeping and on its decisions about the composition and structure
of its deployment, the training and preparation of its personnel and how it
coordinates its effort. Yet to date, there is not a single coherent policy
document that covers the joint efforts of all government agencies contributing
to peacekeeping. The committee believes that it is time for the government to
produce a white paper on peacekeeping that would explain the whole-of-government
policy on peacekeeping operations, including the factors that shape the
government's decision on Australia's involvement in such operations. The committee
envisages that the paper would discuss the whole range of complex matters
involved in peacekeeping, including matters raised in this report. It would
also provide detail on the recently-announced Asia–Pacific Centre for Civil–Military
Cooperation and how this initiative fits into the broader policy on
peacekeeping.
24.47
The committee does not favour the proposed white paper forming a subset
of the new Defence White Paper. Although peacekeeping may figure prominently in
this document, the committee believes that because of the involvement of many
key government agencies in today's peacekeeping operations, the government
should produce a separate white paper on peacekeeping. The production of a white
paper would provide the government and the relevant agencies with the
opportunity to review their policies and practices and to better understand how
their activities contribute to the whole-of-government effort. It would also require
the government to articulate its policy across the full spectrum of Australian
peacekeeping activities, thereby allowing more informed public scrutiny of this
important area of government engagement.
Recommendation 37
24.48
The committee recommends that the Australian Government produce a white
paper on Australia's engagement in peacekeeping activities.
Conclusion
24.49
The committee has underlined the importance of evaluation as a means of
improving performance both in the short and long term. It has also referred to
the role that the committee's proposed peacekeeping institute could have in evaluating
operations and capturing the lessons learnt. In the following chapter, the committee
discusses in detail the proposed institute.
Navigation: Previous Page | Contents | Next Page