Not all cultural sector organisations have the luxury of being able to hire a freelance evaluator or create a data monitoring role within their team to navigate through what can be quite a daunting and complex task. Staff are busy delivering their programmes, often under tight deadlines and with limited capacity. Evaluation can end up at the bottom of the to-do list rather than embedded in delivery. The pressure is real – we need, and I believe we should, evidence the impact we are having. Not just for our funders, and the public, but to assess whether we’ve achieved our own organisational aims. It’s a big morale boost if nothing else. But at what cost are we doing this?
Care in planning
Start with your why (rather than your funders) and create a monitoring and evaluation plan.
Make a plan that you care about, driven by your organisation rather than your funder/s monitoring requirements. I’ve seen so many evaluation plans that aren’t really evaluation plans – they’re a list of monitoring requirements for funders. But what do you want to achieve? And how do those monitoring requirements align or fit into the mix? Sidenote: if they don’t, I’d be reflecting on whether you’re working with the right funders.
Think about how you can be inclusive with your planning process.
Is the framework you’re developing socially engaged or people-centred? Is it co-created with the target market? How do you know which outcomes your participants or audiences want to experience?
In a recent evaluation framework development session with Ideas Test, one of my National Portfolio Organisation (NPO) Creative People and Places (CPP) clients (funded by Arts Council England, ACE), we invited young people taking part in the programme to help shape our evaluation framework for their particular project. We spent an afternoon at the start of their engagement collaboratively reflecting on the benefit, difference or change they wanted to experience. We all (hopefully) put audiences at the heart of other aspects of our work, so why should evaluation be any different? This isn’t unusual, it’s a core element of the SROI process (certainly the one recommended by the UK Cabinet Office’s 2012 guide) but it’s not something you often see talked about in the cultural sector.
Streamline multiple funder requirements.
Without grant funding many of my clients wouldn’t exist (and I’d be out of a job). But it’s completely impractical (especially for smaller organisations) to collect output data for numerous different funders when their requirements ever so slightly vary for the same thing - for example, if a range of funders all want to know visitor age, but their banded age response categories are marginally different.
This also happens with outcomes, with similar expectations (such as improved health and wellbeing) worded slightly differently. Visually mapping these varying funder requirements alongside your own outputs and outcomes onto a giant wall can help to reduce anxiety and streamline what looks complex on paper. I’d encourage you to have conversations with your grant managers if you’re finding it difficult to collect everything in the same way – can you push back on those exact age banding categories, or streamline and supply the same information across the board?
Care in your methodology and design
Do what’s right for your audience.
I’ve seen so many organisations rely on surveys when they’re not the most appropriate methodology for either the target audience (e.g. those with limited literacy) or the type of evidence required (e.g. if something is better evidenced via a qualitative approach).
Some funders acknowledge this and encourage the use of alternative methodologies, whereas others aren’t all that explicit. Don’t force a square peg in a round hole – if it’s not right, speak to your funder to explain why you don’t think a survey is appropriate, rather than just assume you have no choice and go ahead regardless.
I’m working on a project with Scottish Book Trust at the moment called Reading is Caring. It’s a programme which uses reading to support those living with dementia and their family or professional care partners. Our approach is time-consuming, but care-driven. We’ve designed a methodology which is people-centred and tailored per project participant by giving them flexible options for feeding back. This is essential for both those living with dementia, and the people looking after them who are often time poor.
Give context to why you’re asking sensitive questions.
One of my clients recently piloted their new NPO survey on a group of young people using the mandatory NPO monitoring questions. The majority of respondents opted not to answer the demographic questions as they found them too intrusive. More worryingly, questions on gender and assignation at birth were reported as potentially traumatic. This is clearly problematic from the perspective of care and may indicate the need for trauma-informed research (TIR) practice.
It’s not unique to NPOs – cultural organisations ask demographic questions across a lot of surveys in order to align with the census or other local authority statistics (so we can understand if we are representative of our catchment area for example). But I often see surveys given out with absolutely no context about why demographic data is being collected.
If you want to collect this data (or have no choice due to funder monitoring requirements) how can you approach it in a care-focused way? Can you give a verbal or written trigger warning at the start of a demographic question set? If there’s little room for providing written context (as is the case at the start of the current NPO Illuminate survey) my advice would be to add it into the email body copy when you send your e-survey.
Or if you’re conducting face-to-face surveys, ensure your fieldworkers are briefed with a consistent boilerplate script to use. This should typically include a thorough explanation about why you’re asking for the information, how it’s used and how it will help your organisation. But I’d go further than that and flip it – what’s the benefit to them in handing over this type of information? It’s quite simply a matter of ethics.
Although you could argue that the quantity and personal nature of demographic questions has changed, the challenges or concerns with getting audiences to complete them isn’t a new issue. Back in 2002, I did an A/B test on an exit survey with audiences at Kelham Island Museum (at the time part of Sheffield Industrial Museums Trust, now Sheffield Museums Trust) during my MA Arts and Heritage Management degree. I conducted half of the surveys with no contextual explanation for the demographic questions, and the other with an (albeit basic) explanation. I think you can probably guess which one was the most successful in getting fewer ‘prefer not to say’ responses (the latter of course).
Care in your data collection
Sell the benefits.
We’re asking a lot from our target audiences. Surveys are doing a lot of heavy lifting for us. One of my NPO clients tested the completion time for their survey which included the new NPO mandatory question set plus five other questions to help evidence outcomes in their organisational evaluation framework. It took nearly 15 minutes with over half of the respondents giving up before they’d even got to the end. Is this what caring about our audiences looks like?
This issue isn’t unique to ACE-funded organisations. I’ve seen several twenty-page surveys in my time working in the cultural sector. Ask yourself: why should the respondent bother? Would you? The ‘ask’ is more important than ever. Although I encourage the use of non-client incentives for surveys (and there are best practice rules on these), we’re effectively asking for long survey completions with no benefit at all to the respondent.
What can you legitimately offer people for taking the time to give their feedback? What can you tell them, so they understand the importance? Like the demographic question contextual narrative, what’s going to genuinely convince them it’s worth their while? Do you share the responses from surveys with your audiences? Could you do this in a visually appealing way in your space, in your promotional communication printed materials, on your website?
Be secure.
I sometimes see a lack of care in how personal data is stored. Make sure you’re adhering to your data processing and storage obligations. One organisation I worked with a few years ago couldn’t understand why I was concerned that their SurveyMonkey data had been downloaded and shared with a partner, without removing the e-database ‘opt in’ personal emails. Care runs from the start of the planning process, right through to the end.
Care in our wellbeing as evaluators
‘Care’ is also important to us as human beings who are designing and delivering this type of evaluation activity. I’ve worked on a variety of projects over the last few years which have involved talking to participants who’ve experienced severe trauma. For example, I’ve just finished working on Legacy of 67, an NHLF-funded project which involved capturing the (often sensitive or traumatic) stories of those within the LGBTQ+ community who were impacted before and after the partial decriminalisation of homosexuality in 1967. If you’re working with potentially traumatic or triggering material, make sure you’re prepared and take care with your own wellbeing. I’ve attended a two-day Adult Mental Health First Aid course and ensured I’ve got processes in place for dealing with my own triggers.
In summary
- Create a plan that starts with you (your organisation).
- Think about how you can build in a ‘care check’ at each stage of the monitoring and evaluation process, from planning, design, data collection, storage, analysis and write up.
- Consider how you can make your evaluation approach more socially engaged and/or person-centred, and wherever possible do this before any funding bids so that your outcomes match what you actually submit.
- Don’t be afraid of speaking to funders to share your concerns and discuss your ideas for alternative methodologies. Push back on anything that you feel is absolutely not doable and make a case about why.
- Wherever possible, before applying for funding ask your grant manager what their expectations are with monitoring and evaluation – if their response doesn’t match your values or capabilities think twice before applying.
- If you’re an NPO or CPP NPO, share feedback and ask questions about the Illuminate survey platform with the Price Waterhouse Cooper helpdesk. Help make things better by testing, reporting, revising, and trying again.
- Think about ways you can provide support to staff who are collecting potentially challenging impact story data.
Continue training staff in monitoring and evaluation – how to do it and why it’s important. The more methodological approaches up your sleeve, the less likely you’ll be to default to surveys.
Further resources to explore on care
Author Bio:
Marge Ainsley is a cultural consultant with specialist expertise in marketing, audience consultation and evaluation. Since launching her consultancy in 2008, she’s worked with a range of arts and heritage organisations across the UK and internationally. These have ranged from evaluation of major grants programmes (often funded by Arts Council England or NHLF) through to delivering training and facilitation in monitoring, evaluation, and research for organisations with limited in-house expertise. Her previous clients include the National Trust, London Transport Museum, National Museums Liverpool, Manchester Art Gallery, LIANZA, Scottish Book Trust and many other archives, museums, galleries, and heritage sites.