The Impact Measurement Conundrum

The Merriam-Webster dictionary defines a conundrum as an intricate and difficult problem, giving the example, ‘He is faced with the conundrum of trying to find a job without having experience’.

In many ways, this describes the situation that some companies, social enterprises and NGOs in Trinidad and Tobago find themselves in. They are faced with the conundrum of trying to measure their social impact, without having any data.

What is social impact anyway?

Though there are many definitions and overlap with terms such as social value creation and social accounting, social impact inherently relates to the social value produced by for-profit or non-profit organisations. In addition, social impact is usually expressed in line with four key principles.

  • Value is created (or destroyed) as a result of activities of persons or organisations;
  • Value is experienced by beneficiaries and all others affected (including staff);
  • Positive and negative impacts should be considered; and
  • Impacts should be judged against what would have happened in the absence of the activity in question (this is the counterfactual).

Social impact measurement then, aims to assess the social value produced by the activities or operations of any for-profit or non-profit organisation. The diagram included outlines the impact measurement journey, from inputs to value.

Although any business can have a social impact, social enterprises and NGOs are explicitly designed to create social value by addressing social or environmental challenges. They are therefore expected to produce social impact (European Commission and OECD, 2015).

It’s all the rage, but . . .

A clear theme has come out of the author’s interactions with dozens of companies, social enterprises and NGOs in Trinidad and Tobago. A significant proportion, including some with modest investments in Corporate Social Responsibility (CSR), want to measure the social impacts of their activities. Motives vary but include the need to get the biggest bang for each CSR buck, a desire to improve existing CSR activities, and to support ongoing PR and marketing campaigns, to name a few.

The problem is, very few of the companies that want to measure social impact i) have these skills in house, or ii) have monitoring and evaluation processes in place to collect the required longitudinal data. This is before even considering the significant challenges of establishing counterfactuals, causality and attribution (fundamental components of any impact measurement exercise). Thus, there are parallel knowledge and expectation gaps between what organisations want and what they can realistically achieve.

The reality is, unless your organisation has been measuring its CSR inputs and outputs and making use of the data in decision making and reporting, it is very unlikely to be in a position to measure its short or long-term impacts.

Walk before you can run

The following quote from Alnoor Ebrahim’s 2013 Harvard Business Review Article, ‘Let’s be realistic about measuring impact’, is fitting.

“Surely measuring impact matters, but we need to be realistic about the constraints. It requires a level of research expertise, commitment to longitudinal study, and allocation of resources that are typically beyond the capabilities of implementing organizations. It is crucial to identify when it makes sense to measure impacts and when it might be best to stick with outputs — especially when an organization’s control over results is limited, and causality remains poorly understood.”

But all hope is not lost. There are a number of steps that organisations can take to move themselves closer to impact measurement. For example:

  1. Clearly articulate what social impacts you are trying to create
  2. Develop Logic Models or Theories of Change for each activity you want to measure for impact (see definitions below)
  3. Use these Logic Models or Theories of Change to i) identify the input, output and outcome metrics you need to measure, and ii) identify improvements that can be made to boost the activities’ efficacy
  4. Determine whether you already have data for any of the metrics identified, or can source it within your organisation or from community partner organisations
  5. Develop and implement mechanisms to collect the data you don’t already have access to (start with input and output metrics)

Logic Models are tools used to describe the effectiveness of programs. They describe linkages among program inputs (resources), activities and outputs, and short and long-term outcomes. Once a program has been described by a logic model, critical performance metrics can be identified. They are relatively simple graphical depictions of processes that communicate the assumptions upon which an activity is expected to lead to a specific result (adapted from McCawley, undated).

Theories of Change are more complex and robust than Logic Models, but there is no requirement that they be used in their place. They provide a detailed description and illustration of how and why a desired change is expected to happen in a given context. In so doing, they map out how activities lead to desired goals being achieved (adapted from Center for Theory of Change, 2017).

Principles to remember

Regardless of the approach you choose to adopt, it is worth keeping a few principles in mind:

  • Don’t bite off more than you can chew. Start small by identifying a relatively simple activity and establishing strict boundaries for what is in and out of scope.
  • Be balanced. An impact assessment that ignores relevant negative impacts, either from the activity or from the organisation’s operations, is not valid.
  • Look inward. CSR activities may positively impact your own participating staff in addition to external beneficiaries (e.g. improved staff morale or enhanced skills). In many cases these internal impacts are faster and easier to measure, while having direct benefits to the business.

This article is not intended to discourage organisations that want to measure their social impact. Rather, it aims to highlight some common challenges and misconceptions, while providing actional steps to move forward. Now you’re ready to start your impact measurement journey.


Leave a Reply

Your email address will not be published. Required fields are marked *