Part IV

Part IV

Transparency and accountability

This final part of the report is concerned with the way in which Australia assesses and reports on the effectiveness of its aid to Afghanistan. To this stage of the report, the committee has identified a number of Australian-funded programs, their achievements and in some cases their shortcomings. The committee has also considered the mechanisms through which Australia channels its funds—ARTF, multilaterals, NGOs and the PRT in Uruzgan. While acknowledging the impressive gains that have been made in Afghanistan with Australian support, the committee has not yet determined whether Australian funds are being used to best effect—whether they provide value for money.

In this part of the report the committee looks at the ways in which Australia holds those delivering aid to account for their performance and how Australia evaluates its own performance and importantly reports its findings.


Chapter 17

Evaluating the effectiveness of Australian aid

17.1      Australia has provided over $700 million in aid to Afghanistan since 2001 with tangible results.[1] The question before the committee is whether this aid was used most effectively. There are suggestions that some projects have not measured up to expectations—AliceGhan and the Australian Leadership Awards Scholarships for Afghanistan students. There are other projects where the indications are that, while impressive on paper, the achievements on the ground may not be as substantial as initial indicators suggested. In this regard, some witnesses referred to schools being built but without substantial evidence to show increased school attendance. In this chapter, the committee looks at the transparency of Australia's aid funding to Afghanistan: at how Australia monitors, analyses and evaluates the effectiveness of its own performance.

17.2      The committee looks first at the monitoring and assessment of programs delivered through Afghanistan's national budget via the ARTF, multilateral organisations and NGOs.

ARTF

17.3      The committee has looked in detail at the transparency and accountability mechanisms employed to ensure that the ARTF uses its funds effectively and efficiently. Based on solid evidence, it found that the fund is open and transparent and subject to a high level of scrutiny. Even so, there were aspects of its accountability that could be strengthened. Importantly, the recent independent review of the ARTF, part-funded by Australia, highlighted the importance of 'intensive and detailed reporting'. It found that there is a need to achieve a 'consistent, comprehensive and critical tracking and reporting system'.[2] With regard to the fund's National Solidarity Program and the Community Development Committee model, it was also looking for a critical assessment of achievements against political-social, mobilisation and livelihood objectives.[3]

17.4      The committee believes that the findings of this independent review in relation to robust tracking and reporting systems and undertaking critical evaluations of achievements as they affect, for example, livelihoods, should hold true for Australian aid programs in Afghanistan.

Mutual accountability 

17.5      A number of witnesses also recognised the importance of the Afghan Government improving its performance as part of a mutual accountability framework. They underlined the importance of attaching measurable conditions to assistance in order to engender positive incentives for the transition decade and to foster accountability. In managing the transition process, CARE was of the view that shifts towards on-budget aid through the government should be sequenced on the basis of demonstrated progress against sector-specific benchmarks in state capacity and accountability at central and sub-national levels.[4] Professor Maley noted that, if one is working at the central level, one needs some pretty tough conditions attached to funding.[5]

Budget transparency

17.6      Budget accountability is one area where Afghanistan has made some headway. Since 2008, the country has made 'steady and impressive progress' toward greater budget transparency, particularly in the last two years. On the Open Budget Index in 2010, Afghanistan scored 21 out of 100 and subsequently jumped 38 points to register 59 in the 2012 survey. Although with considerable room to improve on budget transparency, Afghanistan has demonstrated a willingness to provide greater information. For the first time, it now publishes a Pre-budget Statement, the Executive's Budget Proposal and a Citizens Budget.[6] The 2012 open budget survey attributed Afghanistan's improved transparency to:

...the political will of the leadership of the Ministry of Finance, as well as the government's desire to improve its international image...Donor organisations and international financial institutions also increasingly focused their attention on fiscal transparency as a means to reduce corruption in the country. Their pressure, coupled with technical assistance provided to the Ministry of Finance, facilitated quick improvements. As part and parcel of these developments, civil society organizations and researchers have started engaging with the government, primarily through the Ministry of Finance, on budget-related issues, publishing budget analyses and organizing public awareness campaigns through the media, and conducting meetings and workshops to highlight the importance of budget transparency for citizen monitoring and government accountability.[7]

17.7      The Survey noted that in 2010 donors committed to channelling up to 50 per cent of their aid through the budget 'on the condition that the government make its budget more transparent and increase its spending capacity'.[8] Clearly, with encouragement and technical support from the international donor community, the Afghan Government can undertake reforms necessary to improve transparency and accountability. Having conditions that are tangible, specific and measurable are important, as is the concerted effort and good will of the Afghan Government, the international community and civil society to help the government meet those conditions.

17.8      This one example of improved governance is, however, not matched in other areas. The committee has noted that the perception of corruption in Afghanistan has changed little since 2001. Indeed, Afghanistan still has a long way to go to improve governance across a number of facets. The worldwide governance indicators shows that Afghanistan remains in the bottom 0–10th percentile on voice and accountability; political stability and absence of violence; government effectiveness; regulatory quality; rule of law and control of corruption.[9] Moreover, in some cases, Afghanistan has reversed its performance—a few years ago, in voice and accountability and government effectiveness Afghanistan managed to score in the 10–25th percentile but has since fallen back.

Importance of monitoring and evaluation

17.9      Although multilateral organisations (including the World Bank through its ARTF) and international NGOs deliver the bulk of Australia's ODA to Afghanistan, the Australian Government is ultimately accountable for how its money is spent and its effectiveness. AusAID provided the committee with a detailed account of the measures it takes to ensure that its partners in delivering development assistance do so efficiently and effectively.

17.10         Australia has signed a memorandum of understanding with the Government of Afghanistan on development cooperation, which includes some conditions. According to AusAID, the agreement contains 'a number of quite robust commitments including fiduciary and administrative capacity within the country's institutions in Afghanistan at national and local levels.' It also includes 'a number of commitments on tackling corruption and improving public expenditure systems'. She explained that it is through this framework that Australia's increased aid to Afghanistan over the next few years would take place.[10]

17.11         Dr Bizhan noted, however, that some of these measures in the memorandum of understanding were broad and could be interpreted differently by various stakeholders in Australian and Afghan. He referred to commitments by the Afghan Government to fight corruption and build effective administration. For example, the Afghan Government has undertaken to 'make tangible progress toward a democratic society, where the equality of men and women, and the active participation of both in Afghan society are respected'. Other equally broad commitments include reducing corruption; addressing injustices and increasing people's access to justice; managing revenue; and building capacity for accountable and fair service delivery.[11]

17.12         Clearly, it would be helpful if the benchmarks against which improvements could be gauged were concrete, practical and, indeed, measurable such as the extent to which Afghanistan had developed its own institutions, their strength and resilience and capacity to deliver services.

Multilaterals and NGOs own systems

17.13         AusAID stated that it promotes effectiveness and accountability for Australian aid to Afghanistan in a number of ways. It works through credible development partners, such as the World Bank, that have demonstrated in-country experience and effectiveness and have robust monitoring of fiduciary risk management systems in place.[12] AusAID informed the committee that it employs a range of other management and evaluation approaches, including direct monitoring, monitoring through trusted partners, communities and third parties.[13]

Monitoring

17.14         According to AusAID, it ensures that all its contractual agreements with implementing partners contain explicit provisions against fraud and corruption and it engages closely with implementing partners to ensure that they have robust scrutiny systems in place. It holds its partners to account through strict reporting requirements against their agreed deliverables and monitors and reviews programs directly wherever it is safe to do so.[14] According to AusAID, it responds quickly if it detects any financial irregularities and encourages its partners to do the same in those circumstances.[15]

17.15         Looking more specifically at Uruzgan, Mr Dawson indicated that since AusAID's early engagement in the province, it has been looking to build its capacity for assessment. He explained:

Initially the work with the liaison office was one way of providing access to information about circumstances in the province; but over a period of time our own capacity to provide that same information and level of analysis of developments in the province has increased.[16]

17.16         Mr Dawson mentioned the PRT and the embassy and DFAT people on the ground who talk to the Afghan Government and other stakeholders—including community groups—to provide direct contact with Australian agencies operating in Afghanistan.[17] According to Mr Dawson, AusAID's capacity for monitoring and evaluation had increased 'very substantially' since the agency engaged a liaison office in November 2010. These include support over three years for the annual Afghan people's survey. The survey canvasses the views of Afghan citizens from all provinces on a wide variety of issues, including economic development, political participation, corruption and the status of women.[18] It does not, however, specifically canvass views on Australian-funded projects.

17.17         In partnership with another service provider, AusAID is also to deliver a specific Uruzgan monitoring and evaluation program—a data collection system, which will monitor and analyse AusAID's programs in Uruzgan. The program commenced in October 2012 and will run for one year, initially. It will include an online database to house collated information on baselines and the results of Australia's activities. According to Mr Dawson, it will go a substantial way to addressing information gaps and limitations that AusAID and other development partners face in Uruzgan. Significant access to information that AusAID obtains through its own sources and direct contacts in Uruzgan will supplement the program.[19]

17.18         Mr Dawson also referred to AusAID forming relationships with other organisations operating in Afghanistan that can provide information 'on the development circumstances and the progress of transition in different parts of the country'.[20] He explained that given this combination of other organisations and AusAID's own increased resources, the agency was confident of its capacity to understand and assess the situation in the province.[21]

17.19         The committee has considered AusAID's detailed description of the measures it takes to ensure the effective delivery of aid in Afghanistan. In this context, the committee notes that attention is heavily focused on intentions and process—steps taken to improve and strengthen administrative procedures—but not on the actual outcomes, particularly what reaches intended beneficiaries and what it means for them. As Dr Bizhan pointed out:

The basic issue should be to measure the effectiveness of aid as it influences targeted recipients or objectives rather than 'so called efficiency in aid administration'.

17.20         Dr Bizhan observed that sometimes 'an efficiently administered project could have a negative impact'.[22] Professor Maley also noted that compliance with process was relatively easy to assess. He noted, however, that the exact outcomes of policy initiatives may be 'far from obvious', which may be 'tempting to replace appraisal based outcomes with appraisal based on process'. He concluded that ultimately, development policies need to be judged by outcomes, not by processes.[23] For example, Professor Maley noted that:

With trained teachers and basic teaching materials one can run a basic school without a dedicated building, but a school building without teachers is simply an aggregation of bricks, mortar and concrete.[24]

17.21         The committee appreciates that having correct processes and procedures in place is important, but that is only part of the picture. The committee now turns to consider the effectiveness of Australia's aid program: not on process and inputs but on what has been delivered on the ground.

Effectiveness of aid as delivered and used

17.22         For many years, the committee has commented on Australian-funded aid projects that have failed in a number of aspects but importantly in their sustainability. The committee is referring to projects completed successfully but then underutilised, neglected, or abandoned completely because of shortfalls in resources to cover operational and maintenance costs or the community's reluctance to use them.

17.23         The problem for the committee is that such deficiencies tend to surface after the completion of a project—when there are insufficient means to operate and maintain a facility or it is not suitable for the environment or intended community. For example, when children are not attending a new school or patients not using a health clinic; when farmers do not have access to, or are unable to take advantage of, improvements in crop production or animal husbandry. There is also the risk of aid having unintended consequences such as laying the foundations for future conflict by favouring particular individuals. As mentioned already, there is the possibility that Australian funds in Uruzgan may have helped to empower individuals looking to promote their own interests rather than those of the local communities. In particular, the committee mentioned Matilluah Khan. There appears, however, to have been no real analysis or serious consideration given to understanding or appreciating the longer term consequences of such funding.

17.24         The committee has only a few specific examples of Australian-funded projects in Afghanistan that had fallen short or not delivered to their potential. Some of the evidence before the committee, however, hints at other instances of the ineffective use of aid funds. For example, Professor Maley argued that the record for achieving local capacity building in Afghanistan was 'notably patchy'. He cited the successful work that went into preparation for the 2004 election. At that time, the Joint Electoral Management Body, in which Australian experts were actively involved, adopted a very positive approach to local capacity building. The intention was to have a strong cohort of trained Afghan staff able to do the bulk of the technical work required for a proper process. According to Professor Maley, the United Nations failed to make effective use of these skilled personnel with little effort made to retain their services for the 2005 parliamentary elections. In his assessment:

The net result was that the 2005 election went over budget, and the 2009 presidential election was marred by very serious fraud. The lesson here is that the failure to engage in effective local capacity building can have potentially grave long-term consequences.[25]

17.25         The current concern is that, with the anticipated decline in government revenue and the closure of the PRTs, some projects in Afghanistan will be unsustainable. The committee has highlighted the importance of Australian aid focusing on sectors with a proven track record. To do so, decision makers must rely on a sound understanding of what is working in Afghanistan and, importantly, likely to prove durable in a vulnerable economic and security environment. Thus, evaluation of projects, with their accompanying lessons to be learnt, is central to this process.

17.26         From the committee's perspective, however, there is a dearth of hard-nosed and searching examination of projects funded by Australia, even for those that have drawn criticism. There is no publicly available assessment of the AliceGhan project or of the circumstances that led to the delayed visas for Afghans invited to a workshop at the ANU. It would appear that an independent review of the Australian Leadership Awards Scholarships for Afghanistan was precipitated by allegations of fraud.

Difficulty evaluating projects in Afghanistan

17.27         The committee accepts that there are major obstacles to evaluating the effectiveness of development assistance to Afghanistan. In particular, the inability to move about the country means that aid officials do not have the opportunities to talk to local people in order to receive feedback, to gather data, inspect, monitor and evaluate a project on the ground. Professor Maley referred to the limitations personnel in Afghanistan encounter in monitoring and evaluating aid projects:

Most foreign embassies are swathed in security constraints which make it exceedingly difficult for them to perform...some of the basic tasks of diplomatic reporting. [26]

17.28         He made clear that this statement was not a reflection on Australian staff deployed to Afghanistan, some of whom, in his opinion; were outstandingly competent people.[27] Professor Stephen Howes and Mr Jonathan Pryke also accepted the view that evaluating aid effectiveness in Afghanistan was 'not an easy exercise':

Information is scarce, and feedback difficult to obtain. While measures can be taken to mitigate them, these problems are very much in the nature of aid given the fundamental geographical disconnect which underlies all aid: the fact that aid funds are raised in one country and disbursed in another.

Evaluating aid to Afghanistan raises special challenges. It is very difficult to visit Afghanistan. There is a huge shortage of impartial information.[28]

17.29         While they acknowledged that monitoring and evaluation was 'harder in Afghanistan than just about anywhere else in receipt of Australian aid', they argued that this situation was all the more reason for the Australian Government itself to monitor and evaluate its aid programs.[29]

17.30         As noted earlier, Australian civilians in Uruzgan are likely to pull back to Kabul. In this regard, Professor Howes argued that Australian personnel would not be able to do as much fieldwork because of the security situation, but again that drawback was no reason for failing to produce annual performance reports or for Afghanistan not being part of the transparency reform underway in the aid program.[30]

Independent evaluation

17.31         The high level of corruption in Afghanistan underscores the importance of transparency in Australia's aid program and robust evaluation and reporting. Professor Maley emphasised that Australia has a strong legislative framework to address the problem of bribery of foreign officials, but it was difficult to put into effect when dealing with the Afghan environment. In his view, careful monitoring of on-the-ground activity was one way to begin to deal with this problem. He then stated, however, that it was 'precisely this kind of monitoring which seems unlikely to be sustainable in the long run with the mooted withdrawal of Australian personnel from Uruzgan to Kabul'.[31]

17.32         Professor Maley noted the importance of putting in place mechanisms that would ensure the effective, independent appraisal of how Australia's policy initiatives have affected the situation on the ground in Afghanistan. He underlined the importance of independence in the appraisal process—that this aspect matters most.[32] According to Professor Maley:

...understandably, aid agencies may have a subliminal tendency to value what they have done, just as parents can often see in their children a beauty which is invisible to all other observers.[33]

17.33         In his view, for this reason the widely-reported termination of the relationship between AusAID and the Liaison Office (TLO) was unfortunate.[34]

17.34         CARE Australia also highlighted the need to establish a set of benchmarks and targets for the phased transition.[35] For example, Mr Poulter underlined the critical importance of 'monitoring, against benchmarks—what actually gets down to the most affected in society and gets out of Kabul'.[36]

Committee view

17.35         While restrictions hamper evaluation, they are not an excuse for failing to do so. Indeed, they underscore the need for sound and thorough analysis and assessment of projects so that Australia can improve on its delivery of aid. If a donor country, such as Australia, is committed to the effective delivery of aid then it would also welcome open and independent scrutiny of the projects it funds.

17.36         Mr John Eyers, however, conducted a survey of published evaluation and reviews of Australian aid to fragile and conflict-affected states (FCA) and found:

...readers must gather together for themselves the observations they contain about where the performance of aid programs has been impaired by countries’ fragile situations; and while there are references to innovations intended to address difficulties particular to FCA countries, most of these are not followed by reporting in later years on how successful or otherwise they had proved to be.

Similarly, readers must make their own inferences about how the effectiveness of Australia’s aid in FCA countries compares with that of programs in other countries.

As far as he could discern, none of the published evaluations or reviews had addressed this question directly.[37]

17.37         Professor Howes and Mr Pryke argued that Australian aid to Afghanistan had hardly been evaluated at all. In particular, they noted that it was remarkable that AusAID had 'not thought it necessary to provide a report by management on its aid to Afghanistan even though there are so many questions around whether it represents value-for-money'.[38] According to Professor Howes and Mr Pryke, the Australian aid program in general has become more transparent and monitoring and evaluation had improved over time. With regard to Afghanistan, however, they argued that practice had lagged even when it came to the internal management reports.[39] They cited a number of indicators that applied as at mid-September 2012 in support of this finding:

17.38         With regard to annual performance reports, Professor Howes noted that one annual performance report, for 2010, had been published for Afghanistan. He argued, however, that given Afghanistan:

...is the fourth largest program, you would expect a report every year, and that is something that the country program is responsible for. So, even if they are sitting in the embassy, they can still write it.[43]

17.39         Professor Howes acknowledged that to its credit, AusAID had made some effort to evaluate projects in Afghanistan—it financed a Feinstein study and, until recently, the Liaison Office.[44] He understood that because of the security situation, AusAID officers would not be able to do as much fieldwork but, as noted earlier, that was not a reason for failing to produce annual performance reports or for Afghanistan not being part of the transparency reform underway in the aid program. Overall, he concluded that the same arrangements that apply to other aid programs should apply to Afghanistan: that Afghanistan should be quickly pulled into line with the rest of the aid program with respect to transparency, program monitoring and evaluation.[45]

17.40         It should be noted that since Professor Howes and Mr Pryke lodged their submission, AusAid has produced a 2011 Annual Program Performance Report for Afghanistan and, on 21 December 2012, launched the Afghanistan Transparency Page.[46] The question then arises whether these measures, together with other reporting mechanisms, provide the level and quality of information indicative of robust analysis and assessment of Australia's aid programs in Afghanistan.

17.41         The Director General, AusAID, Mr Baxter, informed the committee that one can get a sense of the effectiveness of Australia's bilateral aid in a country context in the agency's annual report to Parliament. He explained that AusAID reports extensively on the progress of each of its country programs against individual MDGs. He argued that the agency provides a higher level of detail in its annual report to parliament than ever before and further that it is the only organisation in the Commonwealth required to report to Cabinet on the totality of its program on an annual basis. In his view, AusAID receives a level of scrutiny that does not apply to any other agency in the Commonwealth.[47]

17.42         While AusAID's Annual Report provides information on the amount spent in Afghanistan, and describes some of the programs funded by Australia, it does not give any indication of the effectiveness of programs. Some of the achievements listed cannot be attributed directly to Australian funds but more generally to the international donor community, for example achieving a longer life expectancy for Afghan women. The committee agrees that although the report may give a sense of the effectiveness of Australia's bilateral aid to Afghanistan, it in no way provides analysis or evaluation.

17.43         Similarly, the committee argues that while the Afghanistan Annual Program Performance Report provides a wealth of information on AusAID's development assistance to Afghanistan, it is mainly descriptive, provides little hard data and makes it difficult to determine how Australian aid to Afghanistan is performing. The document is strong on describing programs and activities; on detailing inputs such as the amount of money disbursed to various organisations and on intentions but extremely weak on analysis and evaluation. Often the report lists a range of achievements without any direct connection to a specific Australian program. There are numerous examples of broad statements bordering on meaningless. For example AusAID's staff in Kabul will work closely with the Afghan Government and continue to play an active and influential role in donor coordination (pp. 21–22). Such statements simply invite more questions—what is meant by influential, how has it changed behaviour? Moreover, the statement appears to be one of intention, which is worrying since one would assume that such activity is the very bread and butter of Australian diplomacy.

17.44         The Annual Program Performance does contain a table meant to show progress against objectives. Firstly, the four stated objectives are broad:

17.45         Secondly, the ratings are crude indicators of performance with all four above objectives obtaining the rating of 'will be partly achieved within the timeframe of the strategy'.[48] The committee has no idea what to make of the ratings. 

17.46         Where the performance program does give an indication of results on the ground such as 1,578 farmers trained in improved cropping techniques and 5,016 in improved livestock management, there is no indication whether Australian funds contributed fully or only partially through the Aga Khan Foundation to this result. More importantly, there is no assessment as to the extent that the projects have in fact changed practices for the better and whether Australia's contribution was a cost effective way to help the farmers.[49]

17.47         Throughout this report, the committee has referred to observations about the construction of schools but with attendance not matching enrolments or without reference to retention rates. Local residents have referred to 'white elephants'. Such observations may be unfair, but the committee believes that they should be tested, otherwise the achievements trumpeted may well mask little or no real gains for the Afghan people.[50]

17.48         There can be no denying that improvements due to Australian aid have been made. But considering the hundreds of millions of dollars spent in Afghanistan to help the people rebuild their country and their lives, it would be unusual not to have some obvious improvement. The committee does not want to appear to be too critical, but without a robust evaluation of Australia's aid projects in Afghanistan, there can be no genuine understanding of whether the various programs represent value for money and are likely to make a lasting difference for the better for the Afghan people.

ADF projects

17.49         The reconstruction work undertaken by the ADF in Uruzgan came under harsh criticism for its failure to evaluate the effectiveness of its development assistance.

17.50         Save the Children, Oxfam and World Vision noted the importance of evaluating projects. It noted:

Development and reconstruction projects implemented by the ADF in Afghanistan have not been independently evaluated for cost-effectiveness, impact or sustainability. Nor has the ADF, in its financial reporting, disaggregated its aid operations in Afghanistan from its military operations.[51]

17.51         An Australian Council for International Development publication also found that the ADF does not appear to disaggregate its aid operations from military operations in Afghanistan and further that ADF-supported development projects have not been evaluated for cost-effectiveness, impact or outcome'.[52] AID/WATCH argued that the extent to which problems (cost-effectiveness, focus on strategic goals, quick fix projects and poor accountability structures) apply to Australian assistance was unclear due to 'a lack of transparency in aid delivered by the military'.[53]

17.52         Defence in its submission informed the committee that circumstances in Afghanistan militated against the conduct of formal cost/benefit evaluations. These included: the overall security situation; the relatively small scale of the individual projects undertaken by the military Reconstruction and Task Force and ADF Managed Works Team; and the time imperatives to consistently deliver immediate and visible benefits to local communities.[54] Defence made clear that the extent of its monitoring and evaluation finished when the construction was complete, the defect liability period had expired and the project handed over to the relevant government authority. Defence does not go back to completed projects to do an evaluation as to effectiveness.[55]

17.53         The committee understands the ADF's position that it is not an aid agency. Nonetheless, millions of dollars have been expended on substantial reconstruction work in Uruzgan as part of a whole-of-government effort. The committee cannot accept the lack of any subsequent assessments of the effectiveness of this type of development assistance. The committee has evidence that the quality of work produced under ADF supervision is high but understands that while a project can be 'beautifully constructed' it may not be operational.[56]

17.54         Throughout this report, the committee has quoted from departmental officials or official documents referring to Australia's 'integrated whole-of-government effort involving interlinked, diplomatic and development and military objectives' in Afghanistan.[57] It would seem that AusAID, in the absence of any Defence evaluation, could have been involved in monitoring and reporting on these facilities. AusAID made clear that it does not 'assess, evaluate or monitor ADF projects for effectiveness or how they fit with the MDGs'.[58] This lack of coordination, of long term vision calls into question the working of this so-called integrated whole-of-government effort.

17.55         There does not appear to have been any serious analysis of Australia's whole-of-government approach in Afghanistan. DFAT made clear that it 'does not generate separate reports/recommendations on the whole-of-government performance in Afghanistan.[59]

Whole-of-government

17.56         John Eyers, who undertook a survey of evaluations of Australian aid to fragile and conflict-afflicted states, noted that the effectiveness of Australia's whole-of-government overseas aid was a surprising gap in recent such evaluations.[60] He suggested that the independent reviews of Australia's aid to fragile and conflict-affected countries, including Afghanistan, would provide more evaluation of the parts played by agencies other than AusAID and with more prompt publication.[61] The committee agrees with this observation.

Committee view

17.57         Based mainly on Annual Reports, the committee has provided a detailed description of the development activities undertaken by the various agencies in Afghanistan. Generally, the accounts are simply descriptions providing no indication about the extent to which they reflect the effectiveness of the aid. Where this activity has been part of a multilateral contribution it is difficult to discern the effect of Australia's contribution. The committee has seen no evidence suggesting that Australian government agencies delivering aid to Afghanistan have attempted any genuine critical evaluation of the effectiveness of Australian aid, including an assessment of the cost-effectiveness of aid programs. Information is available on the inputs and when recording outcomes the information is often restricted to quantitative information such as schools, clinics and roads built but with no indication about how these facilities are making a difference: that is the quality of the change that is being achieved. Such reporting presents an incomplete picture and may mask serious underachievement.

17.58         One of the main difficulties, however, is to comprehend the extent to which the projects have had a lasting positive effect, particularly in light of running costs—the need for trained people, to pay salaries, and to fund the operation and maintenance of the project. This consideration is particularly relevant in a fragile and conflict-affected country already dependant on aid to provide essential services and with an uncertain future. Clearly, there is a need for periodic systematic follow-up to determine whether the project was and remains viable after completion and when aid funding for it has ceased.

17.59         In this regard, a number of witnesses also looked at the stated objectives of Australia's aid program in Afghanistan. To make the reporting more robust, the committee notes that the development goals need to be clear and specific. Thus, while measuring development against the MDGs may provide a general sense of the effectiveness of aid, it is too broad to give any certain indication that particular Australian programs were value for money.

Reporting

17.60         The Australian Government agreed with the recommendation of the Independent Review of Aid Effectiveness that all Australian government departments and agencies adopt a three-tiered reporting system in relation to their use of ODA funds. The three tiers for reporting would be:

17.61         The committee has drawn attention to the lack of rigour in AusAID's reporting on ODA to Afghanistan. In light of the committee's findings on the inadequacy of analysis and reporting on the effectiveness of Australian aid, the committee believes that the three tiers of reporting required from other government departments and agencies also needs to meet higher standards.

17.62         For a start, the committee believes that either operational effectiveness should be better defined or a fourth tier should be added. This forth tier would require an assessment and evaluation of the effectiveness of ODA funds delivered. It would not be about process, about vague connections to improvements in development goals but about assessing the way in which the intended recipients of the aid have experienced direct real, sustainable and beneficial changes to the way they live. The reporting would focus on quality over quantity, it would go beyond recording the construction of facilities to providing an account of their use—attendance at school or health clinics with measurable indicators to demonstrate improved education or health standards, farmers using roads to transport their goods to markets etc. Departments and agencies would be required to show how their projects have taken into account the maintenance and operational costs and the skilled people need to operate or manage facilities. It would also require the department or agency to explain how their projects form part of a whole-of-government coherent strategic development plan for the recipient country including projects it is intended to complement.

Recommendation 33

17.63         The committee recommends that AusAID review its Afghanistan Annual Program Performance Report in order to ensure that the document reflects its title—program performance report. This means that the report's main aim would be to convey information on:

It should contain a section providing a comprehensive account of the effectiveness of Australia's whole-of-government effort in Afghanistan.

Recommendation 34

17.64         The committee recommends that the Australian Government implement new reporting and evaluation requirements for departments and agencies delivering Australian ODA that are timely, consistent, transparent and more stringent. They should also require information on:

Unless there is a compelling reason otherwise, reporting and evaluation reports should be publicly available from AusAID’s website.

Recommendation 35

17.65         The committee recommends that the Office of Development Effectiveness conduct a critical analysis of the effectiveness of Australia's ODA to Afghanistan with a particular emphasis on the sustainability of projects and Australia's whole-of-government effort.

17.66         Although the committee believes that the best and strongest critics of the effectiveness of aid should be the agencies themselves, it recognises the critical role of independent scrutiny, especially of parliamentary oversight.

Parliamentary oversight

17.67         The committee does not share AusAID's confidence in the robustness of its evaluation and reporting on Australia's ODA to Afghanistan. Professor Howes suggested there should be more parliamentary reviews of aid.[63] The committee agrees. It believes that a dedicated parliamentary committee is needed to provide regular, systematic and rigorous scrutiny of Australia's ODA and stands ready to inquire into relevant matters as they may arise.

Recommendation 36

17.68         The committee recommends that the Parliament consider establishing a parliamentary standing committee or dedicated subcommittee of an existing standing committee charged with examining and reporting on Australia's ODA. Among other benefits, this committee could be the catalyst needed to improve the standard of reporting on Australia's ODA, especially Australia's whole-of-government effort in delivering overseas aid. It may also be a means of raising public awareness of the work being done with Australia's ODA.

Request to Auditor-General

17.69         In the introduction, the committee highlighted the miscalculation of Australian ODA since 2006.

17.70         With this in mind, the committee requests that the Auditor-General consider conducting an audit of Australia's ODA to Afghanistan with a view to determining whether the guidelines for classifying funding as ODA are appropriate, well understood and applied properly.

Conclusion

17.71         The committee understands the difficulties confronting donors in delivering aid to Afghanistan and accepts that some projects will inevitably suffer setbacks. It also acknowledges the work of Australian personnel in Afghanistan and commends their commitment. Even so, the committee believes that the contribution that Australia is making in Afghanistan should come under close and critical scrutiny.

 

Senator Alan Eggleston

Chair
Senate Foreign Affairs, Defence and Trade References Committee

Navigation: Previous Page | Contents | Next Page

Top