Professors Anne-Claire Pache, ESSEC Business School Philanthropy Chair, Greg Molecke, University of Exeter Business School, and Researcher Eléonore Delanoë, contend that we need to focus on what organisations need and what they can actually do if we want impact assessments to truly drive development and increase impact.

For a Pragmatic Approach to Social Impact Assessment. With kind acknowledgements to the NVSQ Blog

The Nobel prize in economics awarded to Esther Duflo, Abhijit Banerjee and Michael Kremer in December 2019 has consecrated their game-changing work against poverty. At the heart of their work are experimental approaches using Randomized Controlled Trials (RCTs), which have shed new light on the way the impact of social innovations can be assessed. RCTs compare the impact of a measure between a treatment group and a control group whose participants are selected at random. They are a powerful way to remove biases and isolate a specific action from the great swirl of other factors that may affect the result.

However, they are far from being a “one-size-fits-all” approach because they are complex to set up and impose significant technical and financial demands on the organization. They also frequently require long timeframes to set up and run – running into years and decades – making them poor tools to help businesses and investors execute short- to medium-term strategies. RCTs work well to establish causal links between a given intervention and social impact. However, in many instances, the impact evaluation needs for innovators and their supports are quite different – with much more need for tools that can guide performance improvements rather than prove outcomes.

The latest research by Anne-Claire Pache and Greg Molecke for the Handbook of Inclusive Innovation suggests that these needs vary based on where social innovators stand in the innovation cycle. We need to focus on what organizations need and what they can actually do if we want impact assessments to truly drive development and increase impact.

The growing concern for social impact assessment

The best impact assessment method depends on their specific evaluation needs, and these needs vary depending on where they are in the innovation life cycle. They often fall into two categories: needs to “improve” operations to enhance impact and needs to “prove” their impact to attract external stakeholders.

The relative importance of proving vs improving shifts as enterprises progress through their innovation’s lifecycle. Competitive funding environments and performance-based management schemes have led to a large array of new assessment methods, with varying costs and scope of proof: outcome-based metrics, before/after comparisons, experimental methods, ethnographic thick descriptions… Assessing social impact however remains highly challenging. Assessment must deal with scarce data and difficult-to-measure effects, such as self-respect, freedom, or quality of life.

Causal relationships between the impact and the intervention are often difficult to accurately trace and translate through the entire chain of cause and effect. Further, funders seek comparability across assessments; but comparing the effect of different innovations remains difficult, especially when innovations improve lives in very different ways and in different contexts. Social innovators also find that different assessments unevenly support internal (staff, beneficiaries…) and external (funders, regulators…) stakeholders with different priorities.

The experimentation stage: an iterative process

The social innovation life cycle starts with the experimentation stage. During this stage, resources dedicated to the project are scarce. The need to prove is low, while the need to improve is high. Innovators iteratively design their solution and attempt to get feedback. Their main focus is to understand the needs of beneficiaries and improve their design to best address them.

This is often done with the help of in-depth qualitative methods (interviews, ethnographic observations), which provide “thick” descriptions of the lived experiences of the beneficiaries These methods are relatively easier and less costly to do. They enable iteration and provide actionable levers.

They also allow the creation of detailed “theories of change” and “logic models” that can map how their enterprise’s activities will lead to social impact. An organization attempting to tackle homelessness may, for example, map out the key factors that cause homelessness and then create a chain of steps that would lead to alleviating each factor (called logic models). An innovation that improves only one of the causes of homelessness – a telehealth app, for example – can then track their progress on each of the steps in the specific logic model on how their innovation alleviates the specific underlying cause they are targeting. This way they can demonstrate social performance through progress on the steps in their chains, even if their innovation won’t singlehandedly end homelessness.

On the flip side, they cannot allow comparability across heterogeneous innovations, and don’t give certitude as to whether the expected social changes observed will actually result.

The business model stage: improving and proving the pertinence of the innovation

As an innovation enters the next lifecycle stage – the business model stage – innovators need to mobilize financial and political resources. The need to prove the potential of the innovation to external stakeholders becomes prominent. As more resources come in, a new array of tools start to come in handy: crafting “indicators” and “scorecards” to monitor and optimize their activities.

Outputs and Outcomes measured by performance monitoring can be used as proxies for impact measurement. Take solar lanterns, which can be used to replace polluting, hazardous kerosene lamps in off-grid rural areas in Sub Saharan Africa. The number of solar lamps distributed can be used as a proxy for impact, given the direct link between the use of the lamp and its impact (improved quality of life, decreased indoor air pollution).

While “key performance indicators” reduce impact to simplified metrics that cannot fully capture the subtleties of the change, they can be useful tools to monitor and improve operations, as well as to report a sense of impact to the first external funders.

Scaling up impact: Taking the innovation to the next level

As the innovation matures and reaches the scaling stage, it is time to provide rigorous proof of impact to funders, regulators and policymakers. As institutional funders, impact investors, and philanthropists get involved, they can bring in expertise and resources which make sophisticated impact assessments more accessible.

Among them, monetary measures translate social value into monetary terms. This sets the ground for comparability and considerations of cost-efficiency. These measures include Trucost Environmental Impact Metrics, Avoided Cost Methods, and Social Return on Investment (SROI). SROI measures compare the social impact generated to the investment costs required to launch the innovation.

One Acre Fund, a nonprofit which works with smallholders in rural Africa, uses SROI to compare the additional monetary value of the crops their programs help farmers grow to sell and eat to the net costs of the program. The nonprofit has quickly found that this metric works best when included in scorecards which take other factors into account such as nutrition or soil health as well as scale. Monetary measures often raise a tricky question: how can a monetary value be attributed to social impact?

Economists attempt to overcome this challenge by using proxies for the quantitative value of the innovation, such as the willingness to pay for an innovation, even though it is free. How much would people be willing to pay for free malaria bed nets? Once an economic value has been attributed, it can be included in a cost-benefit analysis. Avoided Cost Methods, might calculate repair, replacement, and substitution costs that are avoided when shifting course away from damaging status quo trajectories. For example, the malaria treatment costs avoided by providing bed nets.

Health-based methods, such as disability adjusted life years (DALYs), can also be useful. Through DALYs, health issues are converted into the number of years of life impaired or lost or due to sickness, disability or early death. So, if an enterprise successfully transitions rural areas from cooking over fire to cooking with natural gas, the extra years of healthy life and freedom from disability and burden that are gained by the woman and girls whose lungs no longer fill with smoke every day can be calculated. And the costs per extra DALY generated can be compared to the cost per DALY gained. This process allows funders and investors to direct their resources to places with the best cost/DALY.

These methods, unfortunately, can be difficult to calculate accurately and rely on best estimates of “what would have happened without the innovation” to calculate the actual impact. Only stringent experimental methods, such as RCTs can truly provide conclusive answers to the impact to a specific intervention. Yet, their outreach is limited by their significant cost, technical demands, and timescale. The scaling and mainstream lifecycle stages provide opportunities to implement them, but often only with external help to start.

Few organizations reach the mainstream stage. For those which do, their focus shifts to maintaining the innovation disruptive and making it adopted by others. It is then that organizations are most likely to choose methods that really suit their needs: they are no longer constrained by limited resources and legitimacy concerns.

A new way to look at impact assessment?

What does this lifecycle analysis teach us about assessing the impact of social innovations? Mainly that social innovators need to move beyond the mere question of “how to assess my social impact” to instead ask: “What do we want to do with social impact assessment?” Funders and investors have a key role in this process.

First, they can help by aligning their impact assessment demands with other funders’ so that reporting requirements do not balloon with each new stakeholder brought aboard.

Second, they can understand which measures are appropriate at different phases of the innovation lifecycle.

Third, they can encourage impact assessments that focus on innovators’ needs to learn and improve over funders and investors’ need to prove and compare impact – especially during earlier stages in an innovation’s lifecycle.

And finally, they can realize that social impact assessment requires specialized knowledge and resource that most innovators are unlikely to have until the very height of their innovation’s life cycles, if ever. It may be wiser for funders and investors to provide the social impact assessments as a form of additional support they can deploy across their portfolios rather than having each innovator “reinvent the wheel” of social impact proofs. This keeps each innovation’s resources and attention where both the innovators and their stakeholders want it – on improving and scaling their impact.

Useful links:

• Link up with Anne-Claire Pache, Greg Molecke, and Eléonore Delanoë on LinkedIn.
• Discover the ESSEC Philanthropy Chair and its research and initiatives
• Apply for the new ESSEC MSc in Sustainability Transformation
• Read a related article: Women’s Philanthropy – the end of an invisible phenomenon?

Learn more about the Council on Business & Society

• Website: www.council-business-society.org
• Twitter: @The_CoBS
• LinkedIn: the-council-on-business-&-society

The Council on Business & Society (The CoBS), visionary in its conception and purpose, was created in 2011, and is dedicated to promoting responsible leadership and tackling issues at the crossroads of business and society including sustainability, diversity, ethical leadership and the place responsible business has to play in contributing to the common good.  

In 2020, member schools now number 7, all “Triple Crown” accredited AACSB, EQUIS and AMBA and leaders in their respective countries.

• ESSEC Business School, France-Singapore-Morocco
• FGV-EAESP, Brazil
• School of Management Fudan University, China
• IE Business School, Spain
• Keio Business School, Japan
• Trinity Business School, Trinity College Dublin, Ireland
• Warwick Business School, United Kingdom.