There's no shortage of guidance out there for those of us interested in the evaluation of frontline services. Data Cymru's 'Introductory Guide to Evaluation' is a one stop shop that can bring anyone quickly up to speed with the key terms, concepts and principles. UK Government's 'Green' and 'Magenta' Books provide detailed guidelines for policy makers and analysts on how policies and projects should be assessed and reviewed. Organisations and projects like Inspiring Impact and Better Evaluation offer a wealth of great free evaluation resources online, as do we on the Co-production Network for Wales Knowledge Base. And this is only to scratch the surface!
Before I turn to any of these though, my starting point is to keep three simple things at the forefront of my mind – mantras that ensure I never lose sight of what the best evaluation requires.
1. We are condemned to have an impact
Jean-Paul Sartre famously said 'man is condemned to be free'(1), for once we are thrown into the world, we have no choice but to take responsibility for our actions. Taking inspiration from him, we might say that evaluators are 'condemned to have an impact', for once we throw ourselves into evaluation, we have no choice but to impact upon people's lives.
When we evaluate front line services, it is rare that our evaluation does not involve the users of those services in some way: and whether it does or not, the act of evaluating (or not evaluating!) has an impact. When we ask someone to share their experience, it affects them. When we don't ask them, it affects them. When we ask to observe someone's behaviour, it affects them. When we observe someone but don't tell them about it, if affects them. Every action we take as evaluators has consequences, and it is our responsibility to ensure that those consequences lead to positive impacts for as many people as possible. When done well, evaluation can enhance the impacts achieved for the people we serve, and it should be considered every bit as important as every other part of the service.
2. People are not data points
We all make use of data when we evaluate, but we must always remember that people are not data points! Every 'x' on a graph represents a person, with thoughts, feelings and experiences: an understanding of which is invaluable if we are to gain the deep and nuanced insights needed to make sense of the surface-level picture the statistics paint for us, and to make meaningful improvements to the services we deliver. Lest we forget these people are the true experts in the services which we strive to improve, and to fail to fully capture their expertise in our analysis would be a tremendous missed opportunity.
Not only that, but showing respect for the inherent value of the people we evaluate is simply the right thing to do! Co-production is underpinned by five principles, the first of which is to 'Value all participants, and build on their strengths'. Whenever we have good reason to believe that those people who are the subject of our evaluation want to be directly involved with it, they should be, and to the full extent that they desire.
3. You have permission to be creative!
Evaluating in a way which is rigorous, robust and reliable is not at odds with being creative in our approaches. On the contrary, without being creative, we will be far less likely to find ways to evaluate which fully meet the wants and needs of the people we involve, or deliver the greatest impacts for the people we serve. We also won't have anywhere near as much fun!
Being creative doesn't mean breaking the rules, or neglecting the obligations we hold to our funders and stakeholders. Rather, it means understanding how, within these constraints, we can tailor and bespoke our approaches to our own specific contexts. If Claude Monet and Jackson Pollock could create works of such diversity within the constraints of few square feet of canvas, we can find ways to always deliver impact within the constraints of our funder's reporting requirements. A lot of the hard work has been done for us. Across the world, a panoply of creative approaches to engaging with services users, collecting data and evaluating services are in practice: tried, tested and underpinned by clear, respected and
freely available guidance.
In 2019, the Co-production Network for Wales launched
'Measuring What Matters'. Co-commissioned, co-designed, and co-developed by our membership, Measuring What Matters is an open-access thinking tool which, through a short and sweet set of self-assessment questions, leads you directly to published guidance on the variety of evaluation approaches commensurate with the desires of the people you evaluate with, the people your evaluate for, and you – a great way to get the creative juices flowing!
From L-R: Mike Corcoran, Co-Production Network for Wales; Nick Andrews, Wales Centre for Social Care Research, Swansea University; Mair Rigby, WCVA; Leanne Teichner, Data Cymru
Mike helps organisations of all shapes and sizes around the world to clarify their strategies, communicate their ideas and engage with their audiences in innovative and creative ways. He is a long-term associate of the Co-production Network for Wales, and led the development of the Network's 'Measuring What Matters' evaluation tool.
Mike Corcoran was speaking at the Data Cymru and Wales Council for Voluntary Action (WCVA) Inspiring Impact workshop, ‘Evaluating frontline services - WHAT WORKS?’ on Wednesday 12th February 2020 at Local Government House, Cardiff.
This blog post first appeared on the WCVA website on 19th February, 2020
(1) Jean-Paul Sartre, from the lecture, “Existentialism is a Humanism” (1946)