Q Exchange

Transfer of learning from QI training for better impact on care

Our project will identify what is needed to support the translation of QI training for healthcare staff into real impact on service user and care outcomes.

Read comments 52

Winning idea
2018

Meet the team: QITrainingforImpact

Also:

Mark McCrory, Lecturer in Management, Ulster University

The Challenge

The need for system transformation and innovation is widely recognised in health and social care in Northern Ireland (NI). The Department of Health published the Quality 20:20 Attributes Framework to develop the knowledge, skills and capacity of organisations in Quality Improvement (QI). Since then organisations in NI have been investing significantly in training staff to support and drive QI in practice.

Thus far, resource has been dedicated to the delivery of training rather than evaluating whether the training has had the intended impacts and how. In a study with more than 140 companies, Bersin (2006) found that organisations struggled to practically determine the value of training and were only spending about 2.6% of their total training budget on evaluation.

Evaluation is recognised as important but for many of the QI training programmes delivered, evaluation has been restricted to levels 1 and 2 of Kirkpatrick’s model (reaction and learning). Whilst important, this does not tell us whether the programmes are leading to participants changing their on-the-job behaviour (level 3 – training transfer) and driving QI within their teams and services (level 4 – results).

Again, this is not unusual. An evidence scan published by the Health Foundation (2006) reported that most evaluations of QI training focus on perceived changes in knowledge rather than delving deeper into the longer-term outcomes for professionals and patients. We have received similar feedback from the Q community as can be seen below.

Proposed idea

The aim is to develop an evaluative framework for QI development programmes delivered across Northern Ireland with a specific focus on post programme impact. From this, we hope to identify the enablers and barriers experienced by participants when driving QI from which resources can be developed to guide future programmes to support participants apply their QI learning post programme.

Our project will consist of an evaluation of programmes already delivered across NI. Access across the region can be achieved as we are representing a regional Community of Practice for QI evaluation across the HSC Trusts, service users, both NI universities and the Health Leadership Centre.

We will draw on various evaluation methods but primarily, Brinkerhoff’s (2003) Success Case Method (SCM). The SCM is particularly useful in looking back at programmes already delivered to identify what has worked and how. The approach is based on comparing successful and unsuccessful cases through story-telling.

First, we will co-design a survey which can be used to identify successful cases i.e. QI training participants who have adopted the behaviours and habits of an ‘improver’ (Health Foundation 2015) and have embedded these within their practice (level 3 of Kirkpatrick), and who can demonstrate impact and outcomes within their teams and services through QI (level 4). The survey will be distributed to all the previous participants of QI training programmes in NI. Next, based on the survey results, we will conduct in-depth, semi-structured interviews with multiple success cases as well as non-success cases (hopefully between 25 to 30 interviews in total). Here we will document, with evidence, the success case stories, as well the enablers and barriers experienced by these participants. With the non-success cases, we will concentrate on barriers to achieving impact.

Impact and learning

The first major impact for the Q community will be an evaluative framework which allows healthcare organisations to efficiently measure impact of QI training at levels 3 and 4.

Secondly, our project will identify enablers and barriers experienced by participants post training. Research has shown that the work environment is one of the key elements of training transfer including: culture; available support/resources; and follow-up through additional learning opportunities (Grossman and Salas, 2011). We expect these issues to emerge but this project will help identify specifically how they emerge in a QI context as well as other unexpected issues.

From this we will produce a ‘transfer resource’ which we will share with the Q community, especially relevant to those involved in delivering, supporting or commissioning QI training, and with the wider healthcare sector who want to support transfer. We will share:

Our progress through social media, our project website and Q newsletter,
Our findings via our report to the Health Foundation, publication including paper based short reports distributed at suitable events; digital interactive reports including infographics via our project website and social media as well as contribution to the ‘Q talks’ programme.

The HSCQI have a website and newsletter which we would avail of. We will also seek to collaborate with Q members (e.g. through team members links with multiple SIGs) and Q Exchange projects involved in QI training.

Review with colleagues involved in QI training will allow us to demonstrate the benefit of the evaluative framework. With future QI training participants we will replicate the Success Case Method survey we design post programme to demonstrate the benefits of the ‘transfer resources’ provided.

How you can contribute

Tell us how you have measured the impact of QI training
Share your experiences of successful and unsuccessful impacts of training on care practices and outcomes
If you have completed QI training, what has supported you to maintain the habits of an improver within your organisation?
What are the barriers to transfering your QI learning and skills to the workplace?

Reviewer feedback

This is a great project because…

The project is well structured, thoughtful and builds on existing evidence. Not enough is known about what makes QI training effective beyond initial feedback and understanding the impact it can have will provide useful learning for the Q community.

By the time of the event we encourage the project team to think more about…

Progressing work on identifying the programmes they will be evaluating will add significantly to the project - helping to make the learning potential for other Q members more tangible.

Comments

Simon Dodds 15 Jun 2019

Hi Declan, it is possible to post an update on the progress of this exciting project? I can see from the posted comments that there was a lot of support for the idea ... so I'm sure the whole Q Community will want to benefit from the lessons learned. Warts and all. Regards Simon
Bob Brown 24 Sep 2018

This initiative is great to see progressing and it's encouraging to read the wholehearted support colleagues are providing, which I am keen to further endorse. Across Northern Ireland we have focused on building capability for QI during recent years, and this will continue while we consider how best to create capacity to lead and undertake QI in what are increasingly challenging workplace contexts. Therefore the evaluation framework to help us better understand what investment in QI training is delivering in terms of improving care and outcomes, and how best to sustain improvements and further develop priorities for QI work, will be enhanced through this work, which should also be of national and international significance
Thomas John Rose 18 Sep 2018

Hi, Could you please provide the full titles for the references sited in your brief. Regards Tom
Gill Smith 18 Sep 2018

Great work pulling this proposal together team. A wonderful opportunity for us to learn from the investment in QI capability we have made in Northern Ireland and really help us shape our thinking as grow our HSCQI community. Looking forward to being involved!
Cathy Mccusker 17 Sep 2018

I fully support this Project and look forward to helping the team in anyway I can. this will be excellent for NI.
Amanda Gormley 14 Sep 2018

Brilliant initiative which will support the growth of knowledge and skills in quality improvement in Northern Ireland. Evaluation of training and review of opportunities to use these new skills will allow us to demonstrate improved outcomes for patients and clients.
Athinyaa Thiraviaraj 14 Sep 2018

Great work on pulling this proposal together. The concept is good and a key piece of work that will deepen our understanding of QI training programs. 'Spray and pray' is not an effective method and current evaluation tools have their blind spots. I'm curious and interested to know the outcome of your project as I can see direct application in my work as a Clinical lead on QI training delivered to trainee doctors, pharmacists and consultants
Catherine McDonnell 14 Sep 2018

This project brings the spirit of QI to QI itself. How rigorous have we been to date about "measuring" whether QI training is actually leading to improvement? Northern Ireland is well positioned to do this piece of work, building on Quality 2020 Training Strategy and Attributes Framework. This is a well structured and solid evaluation process which should deliver rich learning for all of us. We, in the WHSCT will be keen to participate in and support this project. We have already been inspired to look at how we apply this type of longer term evaluation to our own training programs. I look forward to the learning that will come out of this.Good luck.
Eileen McKay 14 Sep 2018

Hi - I am delighted to hear about this project and strongly support the need for us to be able to show outcomes from QI training - I have a lead responsibility for training of social work and social care staff in SEHSCT and would be really interested in using learning from the project to measure the impact on care beyond the traditional 'happy sheet' evaluation right across our training provision.
1. Ruth Gray 14 Sep 2018
  
  Eileen - I have to say I am partial to a happy sheet! But totally agree we need to understand what makes training impact positively and sustainably on service outcomes. We are hoping this learning will be applicable across disciplines and organisations.
Julie Hamilton 14 Sep 2018

I fully support this proposal. I am approaching the end of my participation on the Regional QI programme within the SE Trust and can't emphasise enough the learning I have gained throughout including development of my skills which have in turn enabled me to implement and lead a small project to improve outcomes for young people in care, whilst being continually supported by tutors, QI mentor and line managers. It is important therefore that we are evaluating outcomes and measuring impact of training. This proposal should definitely be high on the agenda.
1. Ruth Gray 14 Sep 2018
  
  Thanks Julie. We are going to explore what helps people apply their training effectively - your mention of tutors, QI mentors and a supportive line manager, seem key to this transfer.
Aideen Keaney 13 Sep 2018

Really great idea- will support and look forward to getting involved!
Mary McElroy 13 Sep 2018

I would welcome this project, it would be a positive to collate the impact of all the QI training currently available across HSC to inform the future plans for a joined up approach to QI training across the province.
1. Noleen Mccorry 14 Sep 2018
  
  Thanks Mary. I think the context in Northern Ireland at the minute is really ripe for this work. The recently established HSCQI Alliance brings a system wide focus on QI - which can be exploited to increase awareness and impact around this work. Our findings however will also be of great value to other UK jurisdictions, particularly our draft framework for evaluation of impact of training.
Barbara Campbell 12 Sep 2018

Really important approach to understanding the outcome of QI training.....much learning to be gained for all in the UK from this proposal.
1. Mark McCrory 13 Sep 2018
  
  Thanks Barbara, while the project will be completed in NI, we hope to engage with colleagues across the Q community and share what we learn as widely as possible.
Colleen Harkin 12 Sep 2018

I fully support this project to evaluate the outcomes of QI to enable sustainability and further rollout across the region.
Siobhan Jasztal 12 Sep 2018

A brilliant project and one which i fully support. I am a Clinical Sister and I am almost at the completion stage of a Quality Improvement certificate provided by Southern Health and Social Care Trust. It has been a fabulous course and my ideas and hopes of improving a patient's sleep experience in Intensive care have come to fruition. My ideas have been implemented in practice and i think it will make a real difference for our very valued patients. Through regular audits i will be able to evaluate and sustain these changes. I wish you all the best of luck with your bid.
1. Mark McCrory 13 Sep 2018
  
  Thanks for taking the time to comment Siobhan. It is great to hear that your ideas for improvement have taken effect. We hope to capture and better understand what supported this in your case and how any barriers were overcome in order to help others in their QI journey.
Clifford Mitchell 12 Sep 2018

I fully support this project. As a quality improvement lead, I believe that evaluation of "sustain and spread" is vital. It will enable us to measure the effectiveness and impact of QI training for staff and service users - are they confident with their QI skills, and are they supporting and mentoring others.

It will also be important to evaluate return on investment - are QI skills being evidenced by completion of a work based QI project aligned to team / corporate objectives, and is spread and sustain effective.
1. Ruth Gray 14 Sep 2018
  
  Thanks Clifford. If we are successful in the Q Exchange Process, we plan to gather HSCQI members to workshop: What are the hoped outcomes of QI training? Projects being aligned to team and corporate objectives will definitely be explored.
Jimmy Hamill 12 Sep 2018

It is of no doubt that the raising of awareness of quality improvement through seminars, training, etc is of great importance. It is now becoming of importance in NI to ensure that the next step is achieved - that effective implementation of QI programmes improves delivery of services. This proposal will enable proper comparative analysis of these programmes ensuring that the key barriers are identifiedf and hopefully acted on in subsequent programmes. The bid should be welcomed by all.
1. Mark McCrory 13 Sep 2018
  
  Thanks Jimmy - we hope others see the benefit and importance of the project too and will support us on the day.
Damien Carson 12 Sep 2018

Brilliant idea for a project as its findings are potentially quite generic and thus very beneficial to the entire Q community rather than any one subsection.

It is time we really focused the enormous educational resources we put into QI towards ensuring we are delivering real level 4 results and outcomes and finding out what made that happen - rather than just ticking the box on numbers trained - because it doesn't matter how many we train if they deliver nothing. 100% support from me.
1. Ruth Gray 13 Sep 2018
  
  Thanks Damien. By working on a regional evaluation tool we hope to bring better understanding to shape future QI training transfer across the UK.
Liz Henderson 12 Sep 2018

Having an evaluation framework around QI across NI which will surface barriers and enablers to QI as well as focusing on impact sounds like an excellent proposal and one which I fully support.
denise boulter 12 Sep 2018

this sounds like an excellent idea and a great way to improve networking for QI in NI I support this bid
Mark Roberts 12 Sep 2018

Really important proposal - to strengthen our QI position and understanding of evaluation in Northern Ireland & to continue to make connections and contributions with the wider UK family. I wholeheartedly support this.
kate latimer 12 Sep 2018

Sounds great. The explosion of different Qi training and courses in NI is wonderful and very welcome and now is definitely the time to start evaluating them. I was at an interesting talk on joy at work today and have been reflecting on how we have so many sub cultures in the healthcare teams in NI. This could be an opportunity to reflect and help develop a wider approach to QI across NI with a shared vision/culture of the best way forward
Linda Kelly 17 Aug 2018

I want to express my support for this bid As the development of QI training programmes accelerates in Northern Ireland, it would be beneficial to have an evaluation programme that demonstrates impact back in the workplace. The identification of enablers and barriers for participants applying QI post programme is essential as we continue on our journey to establish our infrastructure to support implementation, scale and spread of improvement initiatives.
Ceara Gallagher 16 Aug 2018

I think this is an excellent bid as many Trust staff are engaging in various forms of QI we need to see who is the better off. Measuring impact of quality training using storytelling is an ingenious way to do this.
Lesley Edgar 2 Aug 2018

This shortlisted project would help address a common problem experienced by a significant number of organisations. Not only will it provide an evaluation framework (Kirkpatrick level 3) for QI programmes, it will also provide a useful planning tool to support QI training participants to make a success of their training. Good luck!
1. Brenda Carson 2 Aug 2018
  
  Thanks Lesley we feel it will be a very valuable piece of work given the amount of investment in skilling staff up in QI - does this change behaviours and how do they use the skills they learn back into the workplace.
Marie-louise Sloan 31 Jul 2018

I fully support this project, it would be really helpful to hear how QI is being developed across the region, the challenges and the good news stories . I am particularly interested in how scale and spread is applied and to hear how organisations are maximising the skills and expertise of practice leaders following completion of programmes
Thomas John Rose 24 Jul 2018

Please have a look at my B2B idea and let me know if you would be interested in its content. Regards Tom
1. Brenda Carson 2 Aug 2018
  
  Thanks Tom for your kind offer as we are continuing to work up our bid all suggestions are welcomed. The aim of our project is to develop useful tools to support organisations evaluate the impact both on the individual and the organisation of staff undertaking some of the QI training programmes available. This is a challenge for all organisations who invest the time and money to support these staff
Janet Haines-Wood 24 Jul 2018

Brenda, Ruth, Declan and Noleen - Are any of you involved with the Evaluation SIG in Q - planning a half day session at the National Q Event on 19.9.18
Jacqueline Morton 20 Jun 2018

This is a key piece of work that requires attention in our NI HSCQI journey. To get the best impact from this approach will require the full engagement and support of all Trusts with representation from all professional groups and grades involved in QI. As SHSCT also have offer accredited QI training to service users and carers we would be keen that they would be involved in the evaluation.
1. Mark McCrory 20 Jun 2018
  
  Thanks Jacqueline for offering to be involved, it is very much appreciated. We hope to gain the support of all the Trusts in NI who have been offering QI training, as you say, this is key to achieving the best impact.
Pauline Thompson 20 Jun 2018

Delighted to support this initiative across Health and Social Care in N.I. I fully support this application to measure impact following training and to identify how best to support staff to continue to implement quality improvement approaches in their practice following training.

Pauline Thompson - Social Work QI team, South Eastern Trust
Sarah Williamson 19 Jun 2018

A brilliant initiative and the beginnings of really important collaboration to shape the future of QI training across Health and Social Care in NI by in depth evaluation of the benefits of the training carried out to date. Looking forward to being involved!
1. Mark McCrory 20 Jun 2018
  
  Thanks for the support and offer to get involved. Much appreciated.
Anne Dixon 19 Jun 2018

I have recently completed the QI Fellowship within the South Eastern Health & Social Care Trust and during this period of time I have been fully supported both by the programme Tutors and by my own department. Completing the QI Fellowship has enabled me to actively look at everything from a different perspective and in everything I now do I would seek ways to bring quality improvements into practice. I would fully support this project as the impact QI has within the HSC will be something that can really make a huge difference to the quality of care we provide to our service users.
1. Ruth Gray 20 Jun 2018
  
  Anne- we hope to collate the impact of this training collectively. To explore how this postive energy you mention, can be harnessed and focused to bring transformation to our HSC structures and services. Thanks for the support.
Gill Smith 19 Jun 2018

Would be great to see this progressed regionally - and an excellent example of some collaborative work for HSCQI. Fully supportive of this important project.
Anne Kilgallen 19 Jun 2018

I strongly support this proposal. As a senior leader in NI, I want to know the added value of our current approaches to developing capability and capacity for improvement here. I also want to better understand barriers and enablers that staff experience so that I can influence these.

This is the first evaluation of our training approaches here in NI. And it has been developed and led by a community of practice, which gives it particular strength of ownership. If this bid is successful, this acknowledgement sends an important signal to improvers here that the regional system for improvement is capable of making a tangible difference to our collective work.
1. Noleen Mccorry 19 Jun 2018
  
  Thanks Anne and Gill. We hope to build a programme theory to explain how training is successfully transferred into practice. Others will then be able to use this programme theory for a range of goals - design or modification of programme elements, support tools, evaluation etc, and can interrogate the context of their own training and practice environments.
  
  Normal
  0
  
  false
  false
  false
  
  EN-GB
  X-NONE
  X-NONE
  
  /* Style Definitions */
  table.MsoNormalTable
  {mso-style-name:"Table Normal";
  mso-tstyle-rowband-size:0;
  mso-tstyle-colband-size:0;
  mso-style-noshow:yes;
  mso-style-priority:99;
  mso-style-parent:"";
  mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
  mso-para-margin-top:0cm;
  mso-para-margin-right:0cm;
  mso-para-margin-bottom:8.0pt;
  mso-para-margin-left:0cm;
  line-height:107%;
  mso-pagination:widow-orphan;
  font-size:11.0pt;
  font-family:"Calibri",sans-serif;
  mso-ascii-font-family:Calibri;
  mso-ascii-theme-font:minor-latin;
  mso-hansi-font-family:Calibri;
  mso-hansi-theme-font:minor-latin;
  mso-bidi-font-family:"Times New Roman";
  mso-bidi-theme-font:minor-bidi;
  mso-fareast-language:EN-US;}
Christine Mcgowan 14 Jun 2018

This is an excellent project. In addition to the anticipated benefits of being able to evaluate behaviour transfer and impact of QI training and identifying enablers/barriers to help future participants, there will be significant scope to use the framework to help QI trainers build evaluation in at the design and planning stage of their programmes.

By taking this upstream approach, trainers will be able to use the framework to identify the tools/approaches most appropriate for their programme and to proactively collect information before and during the programme, rather than focusing mainly on post programme follow up, which is largely what happens at the minute.

I would be very happy to get involved and share approaches to leadership programme evaluation that might be relevant, if that would be helpful.
1. Ruth Gray 17 Jun 2018
  
  Thanks Christine
  
  We would really appreciate yours and the Leadership Centre's input into the team. We hope to develop a framework which should be adaptable to QI and leadership training across the region and be transferable elsewhere.
Kayleigh Price 5 Jun 2018

Hi, this sounds like a really great piece of work, and I think something that will benefit all of us delivering QI programmes, as its notoriously difficult to evaluate the long term impact of delivering QI training. We ask for a project submission some time after they have completed the programme to evidence learning, but I dont feel this really hits the nail on the head and would be really interested in connecting to share ideas and thoughts should you wish to do so
1. Ruth Gray 10 Jun 2018
  
  Thanks Kayleigh
  
  In Northern Ireland there is a regional development of quality improvement across health and social care, @HSCQI. It is an opportune time to build a framework for the evaluation of long term impact of the effort made. We have a community of practice of educators, academics, frontline staff and service users devising a regional evaluation model.
  
  This is very much a dynamic process, initially using a realist approach to evaluation to build a framework which should be transferable to other regions hoping to robustly assess QI outcome.
  
  We would love to hear your views and share any methods and tools you are already using.