The politics of social impact: ‘value for money’ versus ‘active citizenship’?

Jenny Onyx

University of Technology, Sydney

Abstract

There is growing interest in identifying the social impact of everything: academic research, funded projects, organisations themselves, whether in public, private, or community sectors. The central questions are first what benefits do organisations create and deliver for society and second how do we measure these benefits? These questions are notoriously difficult to answer and yet go to the heart of efforts by governments and civil society organisations to create a better world, to generate social value. The importance of finding a way to measure social impact becomes all the more crucial when it comes to arguing that the benefits obtained far outweigh the cost of producing those benefits, and indeed the benefits may directly or indirectly increase economic wealth. This line of thinking has started to generate various attempts in Australia and elsewhere in the neo-liberal world, to find objective indicators of social impact, and preferably to frame these in terms of monetary cost and benefit. Indeed there is increasing insistence on the part of funding bodies that we measure the social impact. However, exactly what it is that we should be measuring remains contested and elusive.

Cosmopolitan Civil Societies Journal, Vol.6, No.2, 2014
ISSN: 1837-5391; https://epress.lib.uts.edu.au/journals/index.php/mcs
CCS Journal is published under the auspices of UTSePress, Sydney, Australia

Cosmopolitan Civil Societies: An Interdisciplinary Journal 2014. © 2014 Jenny Onyx. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 Unported (CC BY 4.0) License (https://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: Cosmopolitan Civil Societies Journal 2014, 6(2): 3923,- http://dx.doi.org/10.5130/ccs.v6i2.3923

Introduction

There is growing interest in identifying the social impact of everything: academic research, funded projects, organisations themselves, whether in public, private, or community sectors. The central questions are first what benefits do organisations create and deliver for society and second how do we measure these benefits? These questions are notoriously difficult to answer and yet go to the heart of efforts by governments and civil society organisations to create a better world, to generate social value. The importance of finding a way to measure social impact becomes all the more crucial when it comes to arguing that the benefits obtained far outweigh the cost of producing those benefits, and indeed the benefits may directly or indirectly increase economic wealth. This line of thinking has started to generate various attempts in Australia to find objective indicators of social impact, and, preferably to frame these in terms of monetary cost and benefit. Calls for impact measurement are particularly strong in systems aligned with neo-liberalism with its emphasis on evidence based management. Indeed there appears to be increasing insistence on the part of funding bodies that we measure the social impact. However, exactly what it is that we should be measuring, and how to ascribe its “value”, remains contested and elusive.

Over the past five years, the University of Technology, Sydney’s Cosmopolitan Civil Societies Research Centre (CCS) has been actively involved in attempts to grapple with the concept of social impact, as applied in different contexts. More recently this involvement has taken a distinctly political edge. This paper traces some of that attempt to interrogate the concept, and the disputed outcomes that have emerged.

In 2011, CCS convened a major symposium to explore the use of social impact. As a lead-up to the Symposium on Social Impact, researchers at CCS decided to selectively take the pulse on this issue in the not-for-profit (NFP) sector in Sydney and regional NSW, to gain a sense of their views, concerns, frustrations, and issues about measuring the social impact of their organisations and beyond. This preliminary commissioned research was conducted by Dr. Barbara Bloch and subsequently reported to the symposium and published in this journal.

The symposium itself focused on the question of how to measure the social impact of organisations and programs within the local community (as opposed to an evaluation of programs). The event consisted of presentations and discussions of current approaches and issues. In the afternoon, small groups attempted to develop some potential solutions and a future research agenda. A major outcome of that symposium was a special issue of Cosmopolitan Civil Societies Journal on social impact (vol. 4, no. 3, 2012).

The papers delivered at the symposium and subsequently published in the special issue give testament to the range of situations in which social impact is, or should be measured. Thus for instance, Kate Barclay examined the nature of sustainability within fishing communities in NSW and concluded that while there is general acknowledgement of the three pillars of sustainability, that is the economic, environmental and social, nonetheless ‘there is a lack of recognition of the role of social factors in sustainability, and a related lack of understanding of how to analyse them in conjunction with economic and environmental factors’ (Barclay 2012, p 38). This especially is true of fishing communities. Another paper similarly examined the concept of urban regeneration and the association between social capital (SC), local communities and the culture-led regeneration process at Sydney Olympic Park (Prior and Blessi 2012), while a third examined the use of a toolkit to increase wellbeing within Aboriginal communities (Batten and Stanford 2012). In these papers, social impact is associated with community wellbeing or social capital; in each case there is an identified need to understand the broader impact an event or industry or program has on the wider community in which it is embedded.

However the commissioned research and analysis by Bloch illustrates that social impact is usually defined much more narrowly when applied to programs within the community sector. Nonetheless the organisations interviewed clearly saw social impact as much broader than program evaluation, ‘that it was a planning and a predictive tool, which inevitably went beyond the stated objectives of a particular program or service. They regarded it as being about long term measurement. Importantly, they saw it as being about trying to capture unintended consequences, or ‘spill-over’ effects. Social impact was about client and community outcomes, and it asked the fundamental question: What is making a difference?’ (Bloch 2012, p 7). Interestingly, while several organisations tried to assess this longer term impact, usually by qualitative means, they seldom mentioned their findings to the funding body, which invariably was interested in ‘objective’ and quantifiable outcomes from specific funded programs.

From the symposium itself a number of questions emerged:

  • What do organisations need to measure in social impact?
  • How do we capture spill-over effects, that is, the wider impacts beyond those anticipated by the organisation or program?
  • What are the tensions between short-term measures imposed by funding bodies and measures which resonate with community members?
  • Can we convert impact into a monetary base? What happens when we try to do so?

Potential answers to these questions centred for example on health, levels of safety, human capital, social capital, social justice, diversity, empowerment – including measures of change in attitudes and values, and the impact on government policies and practices.

A brief overview of the literature

This overview has been covered in a number of publications through CCS and elsewhere. Evaluation models are commonly used to test for social impacts. The model used by the Australian Productivity Commission and many funding bodies seeks to distinguish short term outputs or outcomes from more far reaching impact. Within the logic model of evaluation, a distinction is made between the different stages of evaluation as indicated in Figure 1 below (Arvidson, 2009; Ebrahim and Rangan, 2010; Maas and Liket, 2011; Productivity Commission, 2010).

392301.jpg

Figure 1: The logic model of evaluation

According to the Logic Model of evaluation, there are several distinct stages in any organisational process and these can be evaluated sequentially. Inputs refer to the resources provided for the program, both material and human. Activities refer to the implementation of the program. Outputs are specific, immediate and countable products of the program while Outcomes refer to the benefits of the program for the intended beneficiaries, as identified in the program objectives and are sometimes expressed in terms of a hierarchy of outcomes moving from more specific to more general. Impacts within the Logic Model relate to all changes in the wider environment and the community at large, that occur as a result of the program whether intended or unintended, positive or negative, short term or long term. Impacts usually refer to effects beyond the immediate boundaries of the organisation and its programs. (Zappalà and Lyons, 2009, Arvidson, 2009).

Most evaluations focus on specific programs and not the organisation as a whole. These evaluations are evidence-based and aim to assess outcomes against intended objectives of the program, without attempting to examine the wider implications of these, even where the wider impact is the ultimate goal of the organisation in question. Impact, then, is normally regarded as an extension of the formal objectives of the organisation’s program, and is thus limited to the parameters set by the program. The broader intangible social effects on members and the community are rarely considered. They are what may be termed ‘spill-over’ effects (Productivity Commission 2010). These are effects that may be unintended, and are not specified in the statement of organisational objectives. They are effects that are produced outside the planned intervention, either directly or indirectly as a result of the intervention. In accounting terms they are externalities, and as such not measured. They may however be very important in terms of wider impact of organisational activities on the wider community, with potential positive and negative implications.

The monetisation of social impact

More recently there have been attempts to develop a framework of evaluation that can be used across the sector, that provides a clear and consistent approach and that can potentially measure effects beyond the immediate outputs (Maas and Liket 2011). Of these, the Social Return on Investment (SROI) is gaining considerable attention in the UK and Australia (Nicholls, Lawlor, Neitzert, and Goodspeed 2009). SROI is essentially a monetising exercise, identifying a dollar value for each nominated activity or event to put in the ratio equation of investment against return. This leaves open the question of identifying the key variables to include in the equation, and the appropriate dollar value to impute to each variable. Any variable that cannot be readily given an attributed value is simply omitted from the equation. A recent meta-analysis of social return on investment studies was conducted by the Centre for Social Investment at Heidelburg University (Kriev, Munscher and Mulbert 2013). That analysis was based on 114 published studies of SROI, the majority of which were conducted in the Anglo world of UK, Canada and Australia. While supporting the potential for the use of SROI, the study nonetheless found that the majority of studies gave relatively little detail on the methodology making it difficult to assess the validity of the results. Where information was provided, there was relatively little use of indicators of genuine social impact as opposed to existing available indirect statistical data such as health status, household income or local crime statistics.

Perhaps more seriously, the use of SROI and similar monetised evaluation frameworks reduces organisational performance to financial values at the expense of the human and mission-based values of the service provided (Zappalà and Lyons 2009; Arvidson 2009; Mook, Richmond and Quarter 2003; Ebrahim and Rangan 2010). So, for example there is little or no capacity to monetise self-esteem, community cohesion, social capital networks, or social justice despite various ineffectual efforts to do so (Kiev et al. 2013).

The political will

Despite these limitations, or perhaps because of them, government funding bodies and commercial organisations are increasingly applying, or demanding, SROI measures as a basis for assessing social impact. Indeed this is rapidly developing into a lucrative consulting industry, lead in the first instance in Australia by Social Ventures Australia (SVA) and later picked up by the Centre for Social Impact (CSI) at UNSW and the broader SIMNA (Social Impact Measurement Network Australia).

While SIMNA now professes a broader brief, its origins were clearly fostered from the SROI tradition as indicated by statements on the SIMNA website:

‘SIMNA came about following a three year “Investing in Impact Partnership” established by Social Ventures Australia, The Centre for Social Impact and PwC from 2009 to 2012. One of the recommendations from this report was the establishment of an SROI Network in Australia.

In October 2011, a Social Impact Measurement in Australia conference was held and supported by the “Investing in Impact Partnership” project partners. Following this conference a number of attendees came together to establish an Australian Network about social impact and Social Return on Investment (SROI). Open meetings about establishing a Network were held in Sydney and Melbourne throughout 2012 and promoted as an Australian SROI Network.

Attendees included people from NFP, business, philanthropic, social investment, social enterprise, consulting, corporates and academic organisations. Their feedback indicated that they wanted a network about the practice of social impact measurement generally rather than a network for a specific framework.’ (SIMNA.com.au 2014)

While the network is now seeking a broader analysis of social impact, much of its work continues to lie in the promotion of SROI or other monetised impact measures. The consulting arm of SVA is one of the very few organisations ‘accredited’ to carry out SROI. They appear to be successfully lobbying the State Government to impose SROI as a condition of grant acquittal. This, combined with specialist industry conferences on SROI, has created a lucrative industry, one however that few small organisations can afford to pay. The ethics of this are highly questionable given that SVA had a direct interest in the model, through its consulting arm, and given the high cost of undertaking SROI when it is imposed on funded agencies.

An alternative: Ripple impacts and the SLSA

Debates at CCS have taken a different turn. In 2010, CCS was successful in obtaining a UTS funded grant between UTS and SLSA (Surf Life Saving Australia) with a broad brief to identify a means of measuring the social impact (social capital) generated by Surf Life Saving, apart from its core life saving mission. The project was carried out over two years (2010/ 2011) and involved initially a set of focus groups in four states followed by a national survey of items developed from the initial qualitative study. The initial study was published in conjunction with the social impact symposium and special issue (Edwards et al. 2012) while the survey results were first presented at the International Society for Third Sector Research (ISTR) biennial conference in Sienna in 2012 (Onyx et al. 2012).

The research results demonstrated that social impact is related to the development of active citizenship and not the specific targeted outcomes of programs, monetised or otherwise. It became clear that social impact is a process which flows from the initial establishment of a welcoming culture within the organisation, to increasingly wider effects within the organisation itself and ultimately to wider contributions to the host community. That is, the welcoming organisation creates a climate in which members develop a sense of identity with the organisation, and broader social/ citizenship values, such as a desire to work as a team, and to contribute to the community. From that develops a set of skills and knowledge (human capital) and networks (bridging capital). The skills are the kind of soft skills valued in any corporate setting, such as team work, time management and leadership skills. The networks begin within the immediate organisation but quickly spread beyond; these networks ensure the flow of information and mutual support, perhaps for example leading to new job opportunities or access to external resources That is, there appears to be a flow, or ripple effect, from a central organisational core to increasingly wider impacts both within the organisation and beyond it. The effects can be measured both for the individual participants and for the organisation itself, though the two are linked. The factor analysis identified as separate factors: a welcoming organisational culture, individual sense of belonging, then organisational and individual social/ citizenship values, then skills and networks at the individual and organisational level, and finally organisational and individual contributions. The Structural Equation modelling verified the projected causal path. The theoretical underpinnings of this ‘ripple’ model have been published in the CCS Journal (Onyx 2014) together with a set of propositions which now need to be tested in other contexts in order to establish the wider validity of the model and its measure.

One follow up study now underway by the CCS team involves an analysis of a series of disability arts programs funded by NSW Arts in order to attempt to develop a measure of social impact, using the ripple model, in a very different context. While the analysis is incomplete at this stage, it does appear that the model is appropriate; the conceptual structure is similar albeit with a slightly different detailed content of each factor.

Wherever the model with the SLSA data has been presented to those interested and involved in civil society, there has been an extremely enthusiastic response. Practitioners ‘get it’ and want immediate access to the developing tool. The response of academic reviewers has been less positive. Academics seem most concerned that the model is not derived from the Logic Model or anything like it, and indeed that there is no obvious literature to justify some of the factors obtained, they having emerged in a grounded manner from the data itself. It appears that the ripple model represents a paradigm shift, at least in the field of evaluation and impact measurement, and consequently there is some resistance to its acceptance, particularly as it avoids any direct attempt at monetising costs and benefits. However, demonstrating and recognising the worth of non-monetary impacts, in non-monetary terms, holds an intrinsic benefit, particularly in the many situations where social benefits have no clear or relevant financial reference. Once fully developed as a measuring tool, it should also be relatively simple and cheap to apply, while still providing an objective, quantified measure of social impact.

It is interesting that SIMNA itself has now professed an interest in the model, and indeed the most recent seminar on social impact hosted by CCS and held early in 2014 was badged as a joint CCS/ SIMNA event, following which a number of participants have requested access to the detailed instrument.

Clearly these are early days. The propositions derived from the model need to be tested in other contexts to determine just how robust the model is. Both the theory and the empirical measure will no doubt undergo some modification as testing proceeds. It is the intention of the research team that this measure of social impact, if it is formalised, should be available to all community groups at little or no cost, should be readily applied and analysed and provide meaningful data to participating organisations for their own purposes. The instrument, once fully developed, will allow organisations to assess the extent to which they engage with their members in a process that delivers the characteristics of active citizenship: which enhance the members’ individual capacity, but which also strengthens the wider community.

References

Arvidson, M. 2009, Impact and evaluation in the UK third sector: Reviewing literature and exploring ideas. Third Sector Research Centre, working paper 27. Universities of Birmingham and Southampton. Accessed on 4 August 2014, available at: http://eprints.soton.ac.uk/183217/1/arvidson_working_paper_27.pdf

Barclay, K. 2012, ‘The social in assessing for sustainability: Fisheries in Australia’, Cosmopolitan Civil Societies: an Interdisciplinary Journal, vol.4, no.3, pp.38-53. DOI: http://dx.doi.org/10.5130/ccs.v4i3.2655

Batten, B. and Stanford, K. 2012, ‘Strengthening Aboriginal community wellbeing’, Cosmopolitan Civil Societies: an Interdisciplinary Journal, vol.4, no.3, pp.54-77. DOI: http://dx.doi.org/10.5130/ccs.v4i3.2574

Bloch, B. 2012, ‘Everyone’s story counts: Measuring social impact in the not-for-profit sector’, Cosmopolitan Civil Societies: an Interdisciplinary Journal, vol4, no 3, pp1-17. DOI: http://dx.doi.org/10.5130/ccs.v4i3.2623

Ebrahim, A. and Rangan, K. 2010, The limits of non-profit impact: a Contingency framework for measuring social performance, Harvard Business School Working Paper 10-099. Harvard University, Accessed on 4 August 2014, available at: http://www.hbs.edu/faculty/Publication%20Files/10-099.pdf

Edwards, M., Onyx, J., Maxwell, H., Darcy, S. 2012, Meso level Social Impact: Meaningful indicators of community contribution’, Cosmopolitan Civil Societies: an Interdisciplinary Journal, vol.4, no.3, pp.18-37. DOI: http://dx.doi.org/10.5130/ccs.v4i3.2576

Kriev, G., Munscher, R. and Mulbert, K. 2013, Social Return on Investment (SROI): State – of-the-art and Perspectives. Centre for Social Investment, Heidelberg University

Maas, K. and Liket, K. 2011, ‘Social impact measurement: Classification of methods’, in R. Burritt, et al. (eds) Environmental Management Accounting and Supply Chain Management, Springer, New York, pp. 171-202. doi: http://dx.doi.org/10.1007/978-94-007-1390-1_8

Mook, L, Richmond, B. and Quarter, J. 2003, ‘Integrating social accounting for nonprofits; A case from Canada’, Voluntas, vol. 14, pp. 283-297. doi: http://dx.doi.org/10.1023/A:1025614619742

Nicholls, J., Lawlor, E., Neitzert, E. and Goodspeed, T. A. 2009, Guide to Social Return on Investment, The Cabinet Office, Office for the Third Sector, London. Accessed on 4 August 2014, available at: http://www.bond.org.uk/data/files/Cabinet_office_A_guide_to_Social_Return_on_Investment.pdf

Onyx, J. Edwards, M., Maxwell, H., Darcy, S., Bullen, P. and Sherker, S. 2012, ‘Measuring social impact’ Paper presented at International Society for Third Sector Research ISTR International conference, Sienna.

Prior, J. and Blessi, G. 2012, ‘Social capital, local communities and culture-led urban regeneration: The Sydney Olympic Park experience’, Cosmopolitan Civil Societies: an Interdisciplinary Journal, vol.4, no.3, pp.78-96. DOI: http://dx.doi.org/10.5130/ccs.v4i3.2684

Productivity Commission 2010, Contribution of the Not-for-Profit Sector, Research Report, Canberra. Accessed on 4 August 2014, available at: http://www.pc.gov.au/__data/assets/pdf_file/0003/94548/not-for-profit-report.pdf

PriceWaterhouseCooper 2011, What is the Economic Contribution of Surf Life Saving in Australia? Price Waterhouse Coopers International. Accessed on 4 August 2014, available at: http://www.surflifesaving.com.au/about/pwc-report.pdf

SIMNA (Social Impact Measurement Network Australia) 2014, www.SIMNA.com.au Accessed on 4 August 2014.

Surf Life Saving Australia 2011, Surf Life Saving Australia Annual Report 2010-2011, Surf Life Saving Australia, Sydney, Australia. Accessed on 4 August 2014, available at: http://sls.com.au/sites/sls.com.au/files/Annual Report 2011-12_Online Single Pages.pdf

Zappalà, G. and Lyons, M. 2009, Recent approaches to measuring social impact in the third sector: an Overview. CSI Background Paper No. 5, Centre for Social Impact, University of New South Wales, Australia.