Chapter 7


Chapter 7

Accurately measuring results

7.1        The efficacy and success of a social economy organisation depends on social measures to facilitate analysis and performance comparison, establish track records and offer potential investors an assessment of the risks and returns related to different investments.[1] In addition, measurement can improve an organisation's performance, provide insight into the operation of the sector as a whole and improve public policy outcomes.[2]

7.2        The terms of reference for this inquiry do not explicitly mention measuring social value as an imperative to attracting investment to the social economy sector. The committee has not received much evidence on the issue of measurement. However, the committee recognises that the ability to measure social value is critical to social impact investment (see chapter 6).[3] Social impact bonds (SIBs) in particular, can only be applied to policy areas that have measureable results[4] as they require a baseline outcome and set targets to monitor the achievement of outcomes.[5] The benefit of a greater emphasis on the measurement of social outcomes (rather than simply inputs and outputs) provides incentive for government and investors to allocate resources to achieve the greatest social impact.[6]

7.3        The Productivity Commission (PC) report, Contribution of the Not-for-Profit Sector, provided substantive recommendations for Australian governments to adopt a common measurement framework for the sector which takes into account the diversity of the sector's activities and structures. The PC outlined that a framework applied consistently across governments will minimise compliance costs and maximise the value of data collected.[7] The report stated that building knowledge about the sector and its impacts on society 'is an important element in building confidence in the sector, as well as guiding policy and program design. Evaluation of the sector’s effectiveness is essential if the sector and government are to embrace an evidence-based approach to social investment.'[8] The PC's recommendations are discussed further below.

7.4        Mr Les Hems and Associate Professor Cheryl Kernot of the Centre for Social Impact (CSI) provided a broad description of the global trends in the evaluation of social outcomes and impact, and the growing recognition of the need to move beyond economic indicators to more holistic measures that include social and environmental progress. These trends are interrelated with a shift towards socially responsible investment (SRI):[9]

Mr Hems: ...there is an international initiative at the moment around integrated reporting. It is very much building on the blended value and triple bottom line... In the documents that have been issued to date there is recognition that the straightforward financial accounting is inadequate. Indeed investors are requesting information on social impact and environmental impact. So the argument behind the need for integrated reporting is that investors are requesting it. Similarly, other stakeholders are wanting—

CHAIR: Are they requesting it across the board for all investments or specifically for investments in this area?

Mr Hems: All investments. These are mainstream institutional investors requesting effectively triple-bottom-line reporting and our argument is that the social impact component is going to be the most difficult to standardise and report in a meaningful way. Therefore, the not-for-profit sector and social enterprises should actually provide the set of case studies to allow integrated reporting to develop.

...

Prof. Kernot: In the corporate sector there has been a bit of a hijacking of the whole notion of reporting sustainability simply along environmental lines. Most people now recognise the challenge of incorporating the social in that triple-bottom-line, blended value approach. There is a huge shift going on in the corporate sector in its understanding in that particular sphere.[10]

7.5        The committee has heard that there is a growing body of international social measurement techniques to support these developments[11] and an increasing willingness and recognition among Australian social economy organisations of the importance to report on social impact.[12] Despite these developments, measuring social value presents a number of challenges for social economy organisations, which in turn places limitations on the amount of investment in the sector. Some barriers to effective measurement of the activities of social economy organisations include:

  • social and environmental measurements and metrics have not yet been fully developed;[13]
  • metrics for social outcomes and impacts cannot always be assigned a numerical value;[14]
  • focusing on milestones or achievements, rather than on the quality of the service provided when measuring success may provide a skewed result;[15]
  • a highly diverse social sector creates tensions between the need for a standardised reporting system across organisations, and the need for organisations to report on the most material aspects of their individual work;[16]
  • the financial costs of, and the requirement of considerable expertise for reporting and evaluation can place undue burdens on social economy organisations;[17] and
  • tools used to determine social outcomes and assess participants may be considered invasive, and can be dependent on the cooperation of the service users.[18]

International developments

7.6        The Young Foundation recently analysed over 150 measurement tools that have developed internationally and concluded that while these are encouraging developments, 'there are not yet any widely-agreed metrics on gauging social returns or the impact of social ventures'. The same observation was made by the UK Social Investment Task Force which noted that measuring social impact was a significant challenge for the market.[19]

7.7        The departments' submission[20] and the Centre for Social Impact (CSI)[21] noted the leading work of a number of international organisations:

  • The Impact Investment Reporting and Investment Standards (IRIS)[22] are an initiative of the Global Impact Investing Network (GIIN) which is developing an independent and credible set of metrics for measuring social and environmental performance. As part of IRIS, a beta database was recently made available on the web working towards mechanisms for a rateable system in the US.[23]
  • The Global Impact Investing Rating System (GIIRS),[24] which is overseen by B-Lab, is exploring a ratings approach to assessing social and environmental impact and was launched at the Clinton Global Initiative in September 2011, New York.[25]
  • The International Integrated Reporting Committee's (IIRC) Integrated Reporting Framework is aiming to integrate multiple sources of value (social, environmental and financial) in a single reporting approach.[26]
Integrated reporting framework

7.8        Integrated reporting is intended to reduce the reporting burden of organisations while improving investors' insight and understanding. The provision of an integrated information set, for example, will help inform those who invest on behalf of others to discharge their fiduciary duty (see chapter 4) by taking into account the range of issues that affect organisational, and therefore financial, success and investment returns.[27] The IIRC has released a discussion paper with guiding principles that offers initial proposals for an International Integrated Reporting Framework prior to the publication of an exposure draft for integrated reporting in 2012.[28]

7.9        South Africa has become the first jurisdiction to legislate and mandate that organisations[29] prepare reports on an integrated basis. Organisations are required to provide an explanation if they have not reported on an integrated basis.[30] The Integrated Reporting Committee of South Africa has released a discussion paper outlining a non-prescriptive, principles-based approach which offers practical direction on the integrated report.[31]

Social return on investment framework

7.10      A Social Return on Investment (SROI) framework has been developed by the Measuring Social Value consortium in the UK and is promoted through the SROI Network International.[32] The framework was devised to understand, measure and manage the outcomes of an organisation's activities. A SROI is similar to a cost‑benefit analysis. It places a monetary value on outcomes so that they can be added up and compared with the investment made. This results in a ratio of total benefits (a sum of all the outcomes) to total investments. For example, an organisation might have a ratio of £4 of social value created for every £1 spent on its activities.

7.11      The Measuring Social Value consortium noted that there are two types of SROI:

  • evaluative SROIs are conducted retrospectively and are based on outcomes that have already taken place; and
  • forecast SROIs predict how much social value will be created if the activities meet their intended outcomes. Forecast SROIs are useful at the planning stage of a project, or if the right kinds of outcomes data have not been collected to enable an evaluative SROI.[33]

In addition, the Gates Foundation has published an overview of some emerging measurement tools.[34] Social Traders also lists a number of resources on measuring social impact on its website,[35] and CSI has published a paper, Recent approaches to measuring social impact in the Third sector: An overview.[36]

7.12      Other measurement methods are discussed in detail in the PC report and include cost-benefit analysis, social accounting and audit, stakeholder value management analysis, results-based accountability and logical framework. Appendix B of the report, 'Techniques of social evaluation' provides further detail.[37]

Australian developments

7.13      Within Australia, there is some tension between the need for a standardised reporting system across organisations,[38] and the need for organisations to report on the most material aspects of their individual work.[39] Progress has been made towards standardised reporting of inputs through the establishment of the Standard Chart of Accounts (SCOA) (see chapter 3).[40] However, there is currently no uniform approach to measuring the social outcomes and impacts of social economy organisations operating in Australia. The Finding Australia's Social Enterprise Sector survey, for example, reported that only 65 per cent of organisations even sought to evaluate or measure impacts in relation to their missions.[41]

7.14      CSI is undertaking work on a new reporting system being developed as part of its Integrated Reporting Project.[42] It aims to assist stakeholders to better understand how an organisation creates value, including social, financial, environmental and human capital in the short, medium and longer term.[43] The CSI website states that its Integrated Reporting Project 'presents a unique opportunity for innovation in impact measurement and reporting of shared value in the Australian not-for-profit sector'.[44]

7.15      The project is a collaboration between CSI, the Australian School of Business and private, not-for-profit (NFP) and government sector partners. The project is intended to develop a framework to implement Integrated Reporting in the NFP sector and will be tested across a variety of sites in collaboration with a network of partners. The project leverages international momentum[45] and is hoped to 'place Australia as a global thought leader in the future of NFP reporting':

Collaboration with an array of stakeholders will be undertaken to ensure that Integrated Reporting meets a range of expectations including government as funders and regulators (DEEWR, FACHSIA, PM&C), donors, and may help lay the foundations for a capital market for social investment in Australia. The test sites will comprise a range of not-for-profit organisations operating in a small number of policy fields.

The data collected and experiences of this process will form an evidence base for evaluating the key success factors and recommendations for the future of Integrated Reporting in the not-for-profit sector in Australia.[46]

7.16      CSI is working in partnership with the Salvation Army to evaluate its Employment Plus program. This will involve capturing the impact of the program over a five year period. It is intended that the project will contribute to the evidence base to illustrate that 'organisations across the board have got to take evidence and systematically capture the impact of their programs into their DNA'.[47] Mr Les Hems of CSI told the committee that while impact based reporting will require capacity building (see chapters 3 and 5), the majority of organisations are willing to adopt the new approach:

This will require a lot of capacity building, but I would say almost every not-for-profit organisation and social enterprise that we have spoken to has recognised the need for doing this and are willing to engage in programs to help build up these case studies to build up a methodology to systematically report social impact, building on the pathfinding work on social return on investment and social accounting.[48]

7.17      Mission Australia, for example, has a sophisticated case management system which enables it to capture data and integrate it across the entire organisation. This system involves maintaining contact with clients post program treatment and assessing whether they are still in a job six months, 12 months, 18 months and two years later.[49]

7.18      In addition, as part of the capital injected into the sector through the government's Social Enterprise Development and Investment Fund (SEDIF), the University of Sydney Business School's Innovation and Entrepreneurship Research Group (IERG) is working with Social Enterprise Finance Australia (SEFA) to develop key performance indicators and metrics to measure social impacts.[50] Foresters Community Finance, also a recipient of the SEDIF funding, uses IRIS for evaluation.[51] Social Ventures Australia (SVA) offers consultation using SROI and tailors the approach to the specific needs of the client. SVA can conduct the analysis, or assist an organisation to develop the skills to carry out its own analysis.[52]

7.19      Dr Richard Seymour of the IERG noted that '[m]ultiple approaches [for measurement] are required at this stage of research, as no one group or approach has solved the problem of impact measurement'.[53]

...with regards impacts and indirect value creation, we are seeking to prioritise the subjective: We have questioned whether it can be appropriate or meaningful to objectify the impacts/outcomes such as ‘enhanced sense of self’ or ‘changed attitudes or values.’ Our framework considers that these would be better celebrated ‘subjectively’ and from the appropriate perspective (such as a beneficiary or participant). Furthermore, we think this is more aligned with the way employees, participants, beneficiaries, donors and other stakeholders engage and understand their purposeful activity and impact.

...

Measurement frameworks often ignore the perspectives of clients, beneficiaries or participants, and end up presenting condescending or objectifying data. Measurement frameworks often promise supporters or funders transparency and rich information but end up being lists of comparative lists and rankings. We will endeavour to keep the measurements aligned with the interests (quantitative and objective as appropriate, qualitative and subjective as appropriate). The tensions between such data should be celebrated rather than ignored.[54]

Financial costs of evaluation

7.20      The PC and Christian Super noted that the financial costs of reporting and evaluation may place undue burdens on social economy organisations and require considerable expertise.[55] Associate Professor Kernot of the CSI emphasised that the government, and other partners, may need to assist in developing reporting resources for the sector:

The problem is that many non-profits do not have the resources to invest in that even though it is a fundamental requirement of being capacity ready for greater social investment. There is an enabling role for government and other partners there.[56]

7.21      The departments' submission noted that international developments and collaborations will reduce the transaction costs of building market infrastructure:

Leaders in the social impact investment market are collaborating to reduce high transaction costs by building market infrastructure such as common social return metrics and framework...These new measurement tools are guiding financiers, including governments and philanthropists seeking to determine the most effective way of acting and bring a focus and discipline to the task of choosing interventions that are shown to be the most effective relative to the resources invested.[57]

7.22      The PC report proposed a commonly agreed measurement framework with the intention that any type of organisation in the sector should be able to adopt it, including those that lack the resources to undertake comprehensive evaluation.[58]

Adopting a measurement framework

7.23      An evaluation framework is a compulsory requirement for a capital market for social economy organisations.[59] The PC explored the importance of a measurement framework for the social economy, drawing on international research and current techniques such as SROI and results based accounting. The PC report recommended that all nine Australian governments adopt a framework  based on four distinct levels:

  • inputs—measures of the resources used;
  • outputs—indicators of the level of activity undertaken;
  • outcomes—direct costs and benefits to the activity of participants; and
  • impacts—longer term net benefits to the participants, and other costs and benefits to the broader community.[60]

7.24      Inputs and outputs provide an insight into the processes and activities of organisations and, when aggregated, the scale of the sector. The Australian Bureau of Statistics' (ABS) Non-profit Institutions Satellite Account provides input and output data on economically significant NFPs in the sector. Outcomes measures provide information on the direct benefit of an organisation's activities. At the far end of the spectrum, impacts aim to reflect the net benefit for the broader community, and its longer term effects. In an analogy of medical services, for example, the impacts would include the general productivity and social benefits of a healthy population.[61] The report emphasised that the measurement of outcomes and impacts will inform the governments' resource allocation for the sector, and without these measurements, the understanding of the contribution of the sector will be understated.[62] The framework recommended by the PC is shown in Diagram 7.1.

7.25      Mr Robert Fitzgerald, the commissioner that led the PC inquiry into the contribution of the sector, elaborated on the four levels of the proposed measurement framework in evidence to the committee:

Mr Fitzgerald: ...As you move down, going from inputs, outputs, outcomes and down to impacts, the measurement techniques become less and less robust. As a consequence of that, when you are looking at impacts you need to use a range of measures. You need to use both quantitative and qualitative approaches, which draws off good evaluation, good case studies and other evidence, but absolutely measuring impacts, both at an organisational and sectoral level is where we want to get to. There is work being done on that internationally and I think we have made a significant contribution to that.

CHAIR: In some areas the ability to measure impacts would be more mature than in other areas, depending on the social benefit you are trying to deliver.

Mr Fitzgerald: Absolutely. Since this report has come out—and I am still involved in the sector in a number of ways—one of the good things is that there has been a real desire by the sector, including the philanthropic part of the sector, to move to looking at impacts and trying to design ways by which that can be measured in the broad sense. I think that is an area where there has been great progress. The point I make is that we should not fall into the trap of trying to turn everything into a number. The point that we have made in the report is that where it is measurable you can measure it, but where it is not measurable you can still in fact try to ascertain its value through other processes and other techniques. I think that in the social policy area that is an appropriate way to go.[63]

Diagram 7.1: Measurement framework and types of indicators

Diagram 7.1: Measurement framework and types of indicators

Productivity Commission, Contribution of the Not-for-Profit Sector, January 2010, p. 40.

7.26      Mr Fitzgerald also highlighted the need to establish an evaluation framework during the design of a program, and not simply at a point of failure as the evaluation component 'will inform the design features' of the program. The Logical Framework approach, for example, is used to establish outputs, outcomes and impacts of the activity early in the design of a program.[64] Mr Fitzgerald also emphasised the need to use evaluation as part of a quality improvement framework, rather than simply a mechanism to determine success or failure. He elaborated on this point, using indigenous social policy as an example:

One of the issues with social policy is that often you have to adapt as you learn. Not only do you need early intervention by the government in terms of bringing the right parties together, you need good-quality evaluation frameworks designed at the beginning. You need them to be used to inform and then you need an ability to adapt as you go forward.

If we look at the Indigenous social policy area—and you are aware that the commission convenes the working group that produced the Overcoming Indigenous disadvantage report—we see over and over again a lack of good-quality evaluation, and evaluations used once the program has already started to fail, and no capacity to use them for quality improvement. That is a micro example of a much greater issue. Again, our commentary in the report about evaluation is very important. It is not just about program evaluation; it can be applied to a much broader concept than just very small programs. It can be looking across a range of programs or initiatives.[65]

7.27      A March 2010 paper by Social Finance underlined the distinction between a program's outputs and outcomes, noting that a focus on milestones (outputs) rather than the quality of the service provided (outcomes) may provide a skewed result. The report explained:

A youth training scheme, for example, might be evaluated on the basis of how many young people attend the training course (an input) or receive a Health and Safety certificate (an output). The hope (the desired outcome) is that better qualified young people will be more likely to find jobs. Too often such outcomes are not used to measure the success of such schemes or to judge whether or not to renew service provider contracts.

This trend is driven by the fact that contracting for inputs and outputs is simpler than contracting for outcomes. It is easier for a commissioner to measure and attribute outputs to a service provider – they can count the number of Health and Safety certificates awarded. It is more difficult to attribute an outcome, such as employment, to the work of one agency as inputs from multiple organisations may be required to deliver a positive outcome.[66]

7.28      The Cape York Institute also noted that SIBs could act as a catalyst for the sector to move from outputs based measurement to outcomes, and that this would have particular benefits for indigenous programs.[67]

7.29      The PC measurement framework proposes a full spectrum of techniques to accommodate different measurement needs and purposes for organisations. The PC report concluded that there is no standard measurement 'which will be best suited to all NFPs under every circumstance' and acknowledges that certain activities pose specific measurement challenges. It cites the spiritual value of retaining connection to land for Indigenous Australians as an example.[68]

Government response

7.30      The Office for the Not-for-Profit Sector in the Department of the Prime Minister and Cabinet is acutely aware of the need for an evaluation framework for social economy organisations. The Office found that SROI was one of three critical areas that need to be examined to create a robust capital market for the social economy and to establish 'transparency and accountability of social initiatives for building credibility with investors, funders, donors and the like':[69]

Certainly at the Office of the Not-for-Profit Sector we are acutely aware of how critical it is that we have the right sort of data and that we have evaluation frameworks. Whether the best outcome is that we have one evaluation framework that we are all able to use I am not at all sure. It may be that we have a range of evaluation frameworks that are appropriate for different policy settings, but the underlying need to improve our data to have a much greater emphasis—and this is from a government perspective as well as from a not-for-profit sector perspective and a for-profit sector perspective—on evaluation and a strong measurement of outcomes is undoubtedly a critical issue in all of this.[70]

7.31      The Office told the committee it is in the initial stages of responding to the PC report's recommendations on evaluation, and has begun work with the Australian Bureau of Statistics around framing the PC's recommendations.[71]

7.32      In addition, CSI discussed the need for government to move towards outcomes-based assessment systems and to develop this approach through a series of demonstration projects. CSI proposed that these projects could focus on policy areas where results based funding is already utilised (such as pathways to employment) or 'other policy areas where robust outcome measures can be formulated and where there is a robust evidence base'.[72]

Conclusion

7.33      An effective measurement framework is fundamental to the development of a robust capital market for social economy organisations in Australia. The PC has provided the groundwork for a measurement framework to be applied to the sector by government, social economy organisations and investors.[73] CSI has also made significant contributions towards a measurement framework through its Integrated Reporting Project.

7.34      The Council of Australian Governments' adoption of the SCOA has provided initial steps towards standardisation of input measurements. The committee acknowledges, however, that standardisation of outcomes and impacts has added challenges, such as measuring unquantifiable contributions and producing comparable results across a sector undertaking a diverse set of activities. As a result, many social organisations will not have the expertise or funding to adopt emerging measurement techniques.

7.35      In the committee's view, the government has an enabling role to develop reporting resources to guide the sector and promote measurement techniques that have proven to be the most effective relative to the resources available to social economy organisations. This will improve efficiencies and consistency and facilitate better analysis and performance comparison across the sector. It will also reduce the transaction costs for social economy organisations while the market infrastructure is being established.

7.36      The committee believes the PC's measurement framework captures a broad range of social economy activity, and is flexible enough to allow users to choose measurement techniques most suited to their circumstances. The committee notes that the Office for the Not-for-Profit sector has begun work with the Australian Bureau of Statistics to respond to the PC's recommendations on evaluation. The committee recommends that in addition to this work, the Office for the Not-for-Profit Sector develop a guide on the PC's proposed evaluation framework for social economy organisations and government departments and agencies. The guide should take into consideration emerging international measurement tools.

7.37      The committee notes the CSI's proposition that government should move towards outcomes-based assessment systems and that this should be developed through a series of demonstration projects. The committee recommends that the government identify existing policy areas where results based funding is already utilised and document an evidence base of these programs. The evidence base should contribute towards the development of a robust measurement framework and the proposed guidance material on measurement for the sector and departments and agencies.

Recommendation 7.1

7.38      The committee recommends that the Department of the Prime Minister and Cabinet identify policy areas where results based funding is already utilised and use any relevant programs as an evidence base towards the development of a robust measurement framework for social economy organisations in Australia.

Recommendation 7.2

7.39      The committee recommends that the Office for the Not-for-Profit Sector prepare a guide for social economy organisations to assist in the evaluation of their performance. The guide should be based on the evaluation framework recommended by the Productivity Commission using inputs, outputs, outcomes and impacts and include Australian case studies and emerging international measurement tools.

7.40      The guide should provide social economy organisations with a number of measurement techniques as options to measure their outcomes and impacts. The committee recommends that the guide be adopted by the Council of Australian Governments and distributed to all government departments and agencies.

Navigation: Previous Page | Contents | Next Page