In the red corner… we have the hippies who want to let it all hang out, the Marxists who maintain that ownership is everything and the radical heretics who dare to trust in people and believe in compassion.
The fight rages over the debate around a NICE for social policy or at social enterprise conferences where the command of “You need to measure your social impact” is met with a mumbled “You mean you need to measure our social impact”.
Can we take the sting out of this fight? Would a mature debate not ask rather when it is appropriate and proportionate to measure our social impact and when is it not? Employing the methods of the metrics moguls, what might be the costs and benefits of the methodology of measurement?
The benefits - while unquantified - are much heralded: convincing funders and investors of what you do; proving to improve your work and more. How significant these benefits are is uncertain in a world politicians have preferences, some grant-makers will always like dogs and some will always like cats, and commissioning practice is as truly scientific as Wallace and Gromit.
But what might be the costs and problems?
1. It might be too complicated
As long as reality remains more complex than the most complex spreadsheet, social impact metrics will inevitably be incomplete. Even Geoff Mulgan of NESTA (the mysterious door-handle-less hi-tech bunker which serves as the headquarters of the Impact Taliban) points out that “unlike molecules, which follow the rules of physics rather obediently, human beings have minds of their own, and are subject to many social, psychological, and environmental forces…. Very few domains allow precise predictions about what causes will lead to what effects.” People are more complex than bacteria, social programmes are not vaccines, homeless people are not a disease. Evidence can be as unreliable and contingent as humans are irrational and unpredictable.
2. How can we really know what would have happened otherwise?
Attempts to capture the ‘deadweight’ or ‘counterfactual’ will always be limited in their reliability. Logically, any impact measure that takes into account "what would have happened" is to some extent guesswork. To overcome this, runs the retort, we can employ Randomised Control Trials (RCTs). But a RCT tells is what happened elsewhere, not what would have happened here. As the World Health Organsiation says “As the complexity of interventions or contexts increases, randomization alone will rarely suffice to identify true causal mechanisms.”
3. It might depend on the context.
The impact of any social action is always to some degree context specific, to the particular family, community or individual in question. So just as some measurement Maharishis argue that we need to move towards sectoral approaches to narrow the variations in context, then logically, we would also need to narrow to the regional, the local, to the street-level and all the way down to each individual. The metrics of dogs aren’t comparable to cats and neither will the numbers for Bristol necessarily pan out the same in Brixton.
4. What is impact anyway?
Or rather, when is impact? Even if x leads to y, might y not lead to z? As Zhou Enlai replied, when asked about the impact of the French Revolution, it is too early to say.
5. "Past performance is no guarantee of future results."
Even if, somehow, you do overcome the issues of complexity, deadweight, context and time, your metrics won’t necessarily be much use to you in the future. The world changes. 'What works' changes.
6. Your methodology will be immediately discredited by another social impact expert
The story of social impact measurement looks something like this:
- Social impact guru 1: “My social impact measurement framework is better than your social impact measurement framework.”
- Social impact guru 2: “No, my framework is not a framework. It’s a set of principles”
- Social impact guru 1: “OK. But they’re the wrong principles.”
- Social impact guru 2: “Well then. Let’s develop an overarching set of social impact measurement principles to create a new framework.”
- Social impact guru (3 years later): “OK.”
- Social impact guru 3: ““My social impact measurement framework is better than your social impact measurement framework.”
7. It might be dangerous for society.
What are we going to do with all these metrics? Presumably, they will inform future social action. So there is a risk here that a rational approach to social impact, money and society will increasingly bring a focus on the young and those who can be helped while drifting away from the lost, the dying, the old, and those with the most intractable problems. Only a risk. But a risk nonetheless.
8. It might be dangerous for your staff and your organisation
What else are we going to do with these metrics? Presumably they will inform how your organisation works in future, for setting priorities, staff objectives, business planning and targets. The perils of targets have been well documented and too often ignored, including:
- perverse incentives – as evidenced by Bevan and Hood that "target based performance management always creates gaming” and the infamous ‘creaming and parking’ of the Work Programme.
- demotivation of staff with a wider public service ethos - as seen in the teaching profession and the crowding out of creativity, freedom, intuition, trust and the human touch.
- unintended consequences – of the kind which baffled Tony Blair when a Question Time audience member pointed out that she couldn’t book a GP appointment more than 2 weeks in advance.
9. It costs time and money to measure.
This is really the point. Understanding impact may well bring considerable benefits to an organisation, to funders and to society. But if sometimes the metrics are complex, uncertain, contingent, temporary, of limited practical application, contested and dangerous for you and others AND they cost money, then maybe, just maybe metrics will not always be the order of the day.
Sometimes they will be. But sometimes they won’t. Sometimes, we might better focus on means, values and behaviours, ownership, co-operation, openness and respect. Just sometimes. Peace comrades!