It is often thought that the key to delivering a successful evaluation of any intervention lies in producing an appropriate and detailed evaluation plan. Then enacting that plan with adequate time and resources, and with the full collaboration of the intervention team and other stakeholders. Consequently, most guidance focuses on the elements of an evaluation that should be considered in advance in order to develop, design and schedule an evaluation plan that, in appearance at least, will meet the needs of the stakeholders.

Of course, in practice, and even with the best possible design in place, evaluation projects do not always follow a smooth course

Of course, in practice, and even with the best possible design in place, evaluation projects do not always follow a smooth course. It is impossible to anticipate all the problems that may arise, such as the practical difficulties of accessing data or a lack of resources at key moments. And any changes to the intervention itself, to say nothing of external factors such as policy changes and global pandemics, can all serve to throw plans off course and to undermine evaluation activities. In the case where evaluations do go wrong, or at least not according to plan, the existing guides offer little in the way of practical advice about what to do to rectify matters and rescue the evaluation process – if, indeed, it can be rescued.

Evaluation of quality improvement (QI) projects seems particularly susceptible to these problems since the latter are often complex and changeable interventions in open systems that can have unanticipated emergent characteristics. There are a number of areas of tension that can potentially derail the evaluation of QI projects. For instance, differing priorities mean that QI projects will often place the implementation of change above the requirements for evaluation of that change: for good or for ill, change becomes an end in itself.

…differing priorities mean that QI projects will often place the implementation of change above the requirements for evaluation of that change

These conclusions are among the findings of the Connecting to integrate quality improvement and evaluation practice project being carried out by a team led by researchers from the University of Sheffield and supported through Q’s Connecting Q Locally programme. This project has undertaken a series of evidence reviews and consultations with QI practitioners and evaluators with the aim of bridging the divide between evaluation theory and QI practice. One thing that has emerged is the lack of practical guidance both for recognising the practical problems that can arise and for indicating how to respond when serious problems do occur while the evaluation is underway. But also apparent is the breadth of the practical experience that exists in the QI and evaluation communities, and the potential of this, if properly harnessed, to provide an invaluable shared resource for improving evaluation processes.

To create such a resource, we first need to understand the nature of the problems that do arise. Our review of the published literature about the evaluation of QI projects (in both the UK and abroad) suggests that problems can be clustered around a relatively small set of themes or dimensions of the evaluation:

• Data access, collection and analysis
• The purpose and parameters of evaluation
• Stakeholder management and engagement
• Evaluation approaches and methods
• Resources, knowledge and skills
• Timing and timeliness
• Culture and context

How can you support this work?

Note that certain problems can be related to more than one of these themes, which is unsurprising given the complexity of most evaluation efforts (as well as the somewhat artificial nature of any attempt to classify them into neat categories). Nonetheless, these themes give us a scaffold for talking about the problems that do occur. While not all the problems will have easy solutions, we believe that the collected experience of evaluation and QI practitioners offers a pool of practical knowledge for both recognising problems and suggesting solutions.

As a first step we would like to try to “crowdsource” a knowledge base of experience and advice about how to improve the integration of evaluation and QI; we invite anyone with any experience of the evaluation of QI or other interventions to complete an initial questionnaire which takes one of these themes – that of data access, collection and analysis – as its starting point. You’ll be asked to provide your particular experiences of tackling problems that have arisen relating to this theme. You’ll also be able to share your opinions on the approach that we’re taking. Fill out the questionnaire to share your insight.

And keep an eye out for the next questionnaire in the series, which we’ll be sharing soon. We look forward to receiving your responses!