Evaluating QI capacity building

In a recent member-led event, Dr Iain Smith shared the findings of his latest paper with 70 attendees, funded by Q, on using the Kirkpatrick model to evaluate the success of QI capacity building. 

Below he summarises the findings of his work:

The Kirkpatrick framework, as designed, has four levels: reaction, learning, behaviour, and results.

Reaction is what did people think about the training or the learning or the development program. Did they like it? Did they enjoy it? Did they react to it in a positive way? 

Learning is what it says on the tin. Did participants acquire new knowledge? Did they learn things? 

Behaviour is largely about what people did with what they learned. Did they have the confidence to translate that and use it to do something? 

Results are what happened as a consequence of putting it into practice. Did it achieve some impact? Did it help to achieve organisational goals? 

One thing that we find with this framework, and particularly where it’s being applied in a limited way to massive online courses that I could find, is usually the evaluations go at the first couple of levels. 

How did people react to it and what did they learn?

Various papers suggest that we’re not getting into the higher levels. 

Why is this relevant more generally? I’m guessing in this community, pretty much everyone’s come across NHS Impact; NHS England’s new approach to improvement. 

On their resource site on NHS Futures, they have tools and resources around the Kirkpatrick framework.

There’s a proposal on there that it’s a useful thing to evaluate your improvement work and your improvement training. 

There are wider resources about how you can use it in evaluation of improvement and there are links to some papers that have used it. 

It brings it up to date and it’s quite contemporary looking at Kirkpatrick through an NHS Impact lens, given that they are recommending that.

He also touched on other highly sought topics like waiting list reduction – one of the post-course reports he received claims an 80% waiting list reduction and a 75% waiting time reduction. 

Watch the full presentation: