How to use Transformative Evaluation to capture the complexity and diversity of your work

19/10/23

Transformative Evaluation is a game-changing method that helps you evaluate your project in a way that is equitable, inclusive, and empowering. It helps you learn from the people who are most affected by the challenge or issue you are trying to solve. We used this approach in the NSPCC’s evaluation of Together for Childhood and we have learned a lot along the way. 

What is Transformative Evaluation? 

Transformative Evaluation is a way of understanding how well a project is working by listening to different people’s stories of change. It combines the best of three worlds: appreciative inquiry, participatory evaluation, and transformative learning. The approach is based on the following principles:

• It starts with ethics and values to commit to justice and equity.
• It questions whose reality is being voiced and privileged, and the consequences of that.
• It listens and amplifies the voice of those who are most negatively impacted by the problem or issue.
• It brings forward solutions identified by the community to address inequities. 

Transformative Evaluation is not one-size-fits-all. It can be applied to different contexts and purposes, depending on the needs and goals of the stakeholders. 

How does Transformative Evaluation Work?

Transformative Evaluation uses the Most Significant Change method within its overall framework. This consists of two phases: gathering stories and sharing stories. 

Gathering stories

To get started, you generate stories of change with the people that you are working with through informal and reflective conversations about the outcome of their involvement with the project you are evaluating. Each story should include a description of what has changed, for who, how, and why the change is significant.  

Recording and collecting stories of change is a powerful way to capture the impact of your work, but it also requires careful attention to ethics and safeguarding. You will need to train practitioners on how to do this well.  

Sharing stories  

After story collection, you analyse and organise the stories into themes. This can be in a series of workshops where the learning from each story is discussed. You use different criteria to group the stories, e.g., the type or nature of change or the level or scale of change. 

By the end of this phase, you end up with a significant change story that best represents each theme. You should document this workshop discussion and commentary, and the reason for selecting the story as the most significant in that group or theme.  

Then you bring together a Stakeholder Group (e.g., partners, funders, the community) to discuss and select the most significant story of change overall. This is agreed by consensus and with a reason for their selection. 

How is the process of story selection important? It invites the community affected by the issue to share and discuss the stories that matter to them. When a story is chosen, we do not ignore the other challenges or successes, nor minimise them. Rather, the process of story selection opens a space for dialogue with partners, beneficiaries, and stakeholders. 

It is essential that everyone feels part of the reflection and learning process. That is why we share the story and the reason behind the selection with the whole community. This way, we learn from each other and improve our practice. 

Why is Transformative Evaluation valuable? 

• It promotes interaction and dialogue between stakeholders with distinct experiences and perspectives to facilitate learning about how change is achieved. 
• It helps to create a shared understanding of the project or initiative and its outcomes, and to foster a culture of learning and improvement. 
• It uses ethical and participatory principles to guide the evaluation process and ensure that it is respectful, inclusive of local knowledge and diversity. 

Imagine you could listen to the stories of the people who are working with you to solve a complex social issue. Well, that is exactly what we have done at the NSPCC. 

We have used Transformative Evaluation to evaluate Together for Childhood. It is an initiative where the NSPCC is working with four local communities in the UK to make them safer for children. The initiative aims to prevent abuse before it occurs. 

But how do we know if our initiative is working? How do we measure the changes that we are creating? That is where Transformative Evaluation comes in. It helps us learn from the stories of people who are part of our initiative and see the effect we are having. 

Practitioners acted as story collectors, generating stories of change from local partners and community members who lived in the Together for Childhood sites. This has helped us to learn that: 

1. The connections between activities in Together for Childhood help us to address multiple outcomes at different levels.
2. Participation and collaboration with children, young people, and families can support Early Help and safeguarding outcomes.
3. Community relationships can catalyse learning that leads to policy and practice change.

While we may have achieved this learning through traditional methods, such as surveys or interviews, it may have taken longer. Transformative Evaluation allows for ongoing data collection, which helps us capture important yet unintended outcomes. It provides valuable insights into what is changing and why. 

But that is not all. It also puts those involved in the change-making process at the heart of sense-making. By analysing and interpreting the stories, they contribute the added value of diverse perspectives and decentralised research. This can help shift from research being done to, to being done with - which can be powerful. 

Tips for doing Transformative Evaluation well 

Transformative Evaluation is not always easy to do well. We learned a lot from the challenges we encountered along the way, such as: 

• How to prepare practitioners who collected stories on ethical and safe evaluation practices, especially in complex and sensitive settings. We also learned we had to keep reminding ourselves and the practitioners of these principles throughout the project, as new situations and dilemmas arose. 
• How to show the value and outcomes of our initiative using stories of change, when different stakeholders had different needs and expectations from the evaluation. We had to communicate clearly and convincingly why stories matter, and how they can reveal the complexity of a place-based initiative. 
• How to respect the storytellers' voice and confidentiality, while giving them agency and choice over how they want to share their stories. Some storytellers wanted to use their real names and felt proud of their stories. We worked with them to find other ways to share their experiences and annoymise their stories in a way that did not take away their power or change their meaning.
• How to learn from what works less well and what needs to be improved and recognise the limitations of this methodology. You might not always get the feedback or insights you need from the stories, and  we had to be humble and honest about what we can and cannot learn from them 

Here are some tips for using Transformative Evaluation: 

•  It needs to be locally owned and adapted to the context and needs of each community. Otherwise, it can be seen as a burden rather than an opportunity for participation and learning. 
• It needs enough time to engage a diverse range of people, so that we can capture changes in different parts of the system. 
• It needs to let us hear the voices of people who are usually unheard. They can tell us how the initiative is changing their communities for the better. 
• It can give us rich and meaningful data that goes beyond numbers and statistics. It can shows us the real changes in how people think, feel, and act to prevent child abuse. 

We hope that by sharing these lessons, we can inspire others who are interested in using Transformative Evaluation. What do you think? Would you try it? Let us know what you think in the comments!