Hi All,

I wanted to flag an excellent opportunity to share your knowledge, experience, expertise and information on overcoming the challenges of evaluating QI through a series of surveys that will crowdsource insights from the community. Please see the following blog for more information:

The trouble with evaluation:
It is often thought that the key to delivering a successful evaluation of any intervention lies in producing an appropriate and detailed evaluation plan, and then enacting that plan with adequate time and resources, and with the full collaboration of the intervention team and other stakeholders. Consequently, most guidance focuses on the elements of an evaluation that should be considered in advance in order to develop, design and schedule an evaluation plan that, in appearance at least, will meet the needs of the stakeholders.

Of course, in practice, and even with the best possible design in place, evaluation projects do not always follow a smooth course. It is impossible to anticipate all the problems that may arise, such as the practical difficulties of accessing data or a lack of resources at key moments. And any changes to the intervention itself, to say nothing of external factors such as policy changes and global pandemics, can all serve to throw plans off course and to undermine evaluation activities. In the case where evaluations do go wrong, or at least not according to plan, the existing guides offer little in the way of practical advice about what to do to rectify matters and rescue the evaluation process – if, indeed, it can be rescued.

Evaluation of quality improvement (QI) projects seems particularly susceptible to these problems since the latter are often complex and changeable interventions in open systems that can have unanticipated emergent characteristics. There are a number of areas of tension that can potentially derail the evaluation of QI projects. For instance, differing priorities mean that QI projects will often place the implementation of change above the requirements for evaluation of that change: for good or for ill, change becomes an end in itself.

These conclusions are among the findings of the Connecting to integrate Quality Improvement and evaluation practice project being carried out by a team led by researchers from the University of Sheffield and supported through the Connecting Q programme of The Health Foundation. This project has undertaken a series of evidence reviews and consultations with QI practitioners and evaluators with the aim of bridging the divide between evaluation theory and QI practice. One thing that has emerged is the lack of practical guidance both for recognising the practical problems that can arise and for indicating how to respond when serious problems do occur while the evaluation is underway. But also apparent is the breadth of the practical experience that exists in the QI and evaluation communities, and the potential of this, if properly harnessed, to provide an invaluable shared resource for improving evaluation processes.

To create such a resource, we first need to understand the nature of the problems that do arise. Our review of the published literature about the evaluation of QI projects (in both the UK and abroad) suggests that problems can be clustered around a relatively small set of themes or dimensions of the evaluation:
Data access, collection and analysis
The purpose and parameters of evaluation
Stakeholder management and engagement
Evaluation approaches and methods
Resources, knowledge and skills
Timing and timeliness
Culture and context

Note that certain problems can be related to more than one of these themes, which is unsurprising given the complexity of most evaluation efforts (as well as the somewhat artificial nature of any attempt to classify them into neat categories). Nonetheless, these themes give us a scaffold for talking about the problems that do occur. While not all the problems will have easy solutions, we believe that the collected experience of evaluation and QI practitioners offers a pool of practical knowledge for both recognising problems and suggesting solutions. As a first step we would like to try to “crowdsource” a knowledge base of experience and advice about how to improve the integration of evaluation and QI; we invite anyone with any experience of the evaluation of QI or other interventions to complete an initial questionnaire which takes one of these themes – that of data access, collection and analysis – as its starting point. You’ll be asked to provide your particular experiences of tackling problems that have arisen relating to this theme. You’ll also be able to share your opinions on the approach that we’re taking. To access the first questionnaire using this link: https://scharr.eu.qualtrics.com/jfe/form/SV_abiDRf2XrOvk7vU

We look forward to receiving your responses and watch out for the next survey!

On behalf of the University of Sheffield and Evaluation SIG Project Team.