Overview

Q has commissioned Picker to undertake a two-year independent evaluation of its overall impact on members, its improvement work and the wider health and care system.

The independent evaluation team, led by Picker, will help Q to understand how impact is achieved, and ways that it might increase impact in the future so that it can improve health and care.

Picker’s research protocol, which sets out how they and their partners will conduct their research, is below.

---

Aim of the research

Picker is excited to lead the independent evaluation of Q.

The aim is to evaluate the impact of Q, understand how impact is achieved, and suggest ways to increase impact in the future.

The specific evaluation questions are as follows:

• What is the individual and collective impact of Q’s offers at individual member, organisational and system level?
• What factors create this impact and how do they interact?​
• How can the impact of Q be increased beyond 2025?

Why is this important and what does it mean for Q members?

The evaluation will provide high quality evidence on Q’s impact and play a key role in informing the decisions regarding Q’s future strategic development beyond 2025. For example, it has the potential to inform evidence-based decisions around adapting Q’s existing offers and the development of new offers, to achieve maximum impact on Q members as well as their organisations and local health systems.

We are keen to hear from as many Q members and stakeholders, eg senior leaders, as possible and will be inviting them to participate in workshops, interviews and surveys over the course of the evaluation. Participation will be voluntary and will not affect Q membership in any way. All contributions will be treated with anonymity, unless we gain individuals’ explicit consent to be named, and participation will be kept confidential from the public.

What is Picker doing?

The evaluation team will use a combination of qualitative (eg interviews, workshops and desk research) and quantitative methods (eg surveys) to collect data and iteratively perform a realist evaluation and a performance evaluation. This allows us to get under the skin of the evaluation questions by bringing together a range of perspectives.

There are two strands of the evaluation:

• The realist evaluation: this will highlight factors that drive Q’s impact, explore synergies between Q’s offers, and identify ways to increase impact.
• The performance evaluation: this will include different types of analyses including causal inference, multi-criteria decision analysis, and portfolio optimisation.

Ultimately, this evaluation will provide evidence on:

• Perceived impact on individual members: eg skills / competencies, morale, relationships at work.
• Perceived impact on organisations eg project efficacy, teams and colleagues, informational connectors.
• Perceived impact at local, regional and system-levels: eg evidence of impact within an Integrated Care System (ICS).
• Sustainability considerations at different levels of impact.

The key activities to collect data are outlined below:

Scoping interviews

We will carry out up to six interviews with internal stakeholders (Health Foundation colleagues and Q leadership and delivery team members) at the start of the project to understand the Q landscape, strategic focus and its offers. This will give the evaluation team the relevant context about Q and inform the design of survey questions and the interview topic guide.

Data review

Subject to approval of data sharing agreements, we will request access to information about Q members including their role, details about their work setting and information about their engagement with Q offers to understand how members interact with Q. We will also request names and email addresses of members to be able to send survey invitations.

Surveys with Q members

We will conduct two surveys with Q members and stakeholders at different timepoints. The surveys will ask about the range of offers that people have interacted with and their perceived impact of such interactions. The surveys will also explore characteristics of members and other collaborations (outside Q) they are involved in. Finally, the surveys will help us understand potential barriers to engagement with Q and its offers. For the survey data, we will look at members’ views about the impact of Q by (i) different Q initiatives; and by (ii) the extent to which they interact with Q’s offers.

The surveys will be conducted online, with invitations electronically sent to all Q members. Participation will be voluntary, and people will be able to opt out of receiving reminders if they do not wish to take part.

Interviews with Q members and system leaders

Approximately 40 qualitative interviews will be conducted (30 with Q members and 10 with system leaders – some of whom may also be Q members). These will explore how people have interacted with the Q offers, how this has informed their understanding of quality improvement, the actions that they have taken in their own organisations, and any evidence that they have gathered to understand impact.

The interviews will also investigate how Q impacts individuals, for example via relationships and collaborations, sharing of skills and knowledge, aspirations and confidence. We will use feedback from the interviews to summarise views on the various impacts of Q.

All interviews will be conducted remotely (eg via Zoom or Microsoft Teams) to minimise cost and maximise convenience, and will follow a topic guide to ensure all intended topics are explored and discussed. Interviews will be recorded and transcribed.

Workshops

An initial stakeholder workshop will be conducted in the early stages of the evaluation to explore impact in relation to Q and what this means for members (for them as an individual, for their teams, for their organisation and at a system level). It will also allow us to gather feedback about the methods we plan to use throughout the evaluation.

Two further stakeholder workshops will be facilitated – one at the midpoint to drive engagement as the evaluation develops, and one towards the end of the evaluation. The workshops will present and sense-check emerging findings and provide an opportunity to identify potential case studies (see below). Both Q members and system leaders will be invited to these two workshop events.

All workshops will be held remotely for convenience, greater accessibility, and to reduce costs for the overall evaluation and the participants.

Case study development

We will identify a set of Q members to engage in the creation of five case studies to summarise examples of Q membership’s impact across a range of settings and relating to different Q offers. The case studies will provide a detailed and in-depth presentation of Q projects and/or organisational change at different levels, from small to large-scale systems impact. We wish for the case studies to be informative to capture achievements and opportunities for learning and knowledge transfer, as well as to be visually impactful.

How will Picker use this data?

We will use a number of different analysis techniques to gain rich insight from the data we collect. Anonymised qualitative data will be analysed thematically – this means that we will be interrogating the data to discover which subjects in relation to the evaluation questions are of particular salience to the participants. In line with our realist evaluation approach, we wish to understand what about Q works for whom, when and under what circumstances. We are interested in context and in finding out what can cause impact or change as a result of Q.

The data collected through the survey will be used to investigate the causal relationship of Q offers with outcomes. Propensity score matching will be used in the analysis to ensure attribution of impact of specific Q interventions on outcomes. Multi-criteria decision analysis will be performed to assess the value for money of Q’s offers, and a portfolio optimisation exercise will be conducted to explore the combination of Q pathways that maximise Q impact within given resources, based on a simulation model.

The analysis techniques being used will provide a new depth to the understanding of Q’s impact at individual, organisational and system levels, and provide clear recommendations for how Q can amplify impact in the future.

How will the findings be shared?

We aim to provide regular updates to Q members about activities and progress throughout the course of the evaluation via Q’s newsletter ‘Q-municate’ and social media channels. Brief quarterly progress reports and an interim report will be provided to the Q team at the Health Foundation.

A final report will detail the methods, key findings and recommendations and will be made publicly available in summer 2025. Three stakeholder workshops will be facilitated during the life of the evaluation to help sustain evaluation engagement and to promote a collaborative approach to sense checking the findings and emerging recommendations with members.

We will also develop a range of accessible outputs to support dissemination of evaluation findings with Q members and key stakeholders. This may include infographics, slide decks, Q blog posts, and webinars to share learning.

Ethics and Equality, Diversity and Inclusion

Ethical approval is not required for this project in line with the Health Research Authority guidelines for evaluation studies. The project does not involve primary research with NHS patients or social care users and raises no material ethical issues. However, we conduct our evaluation work in line with the Market Research Society Code of Conduct.

We have designed our evaluation approach with Equality, Diversity and Inclusion (EDI) considerations in mind. We have considered how to make participation as accessible as possible, for example through the use of online technologies, longer recruitment time frames, and flexible researcher availability. We shall also consider accessibility in how we present our interim and final reports. We wish to ensure that our methods are not exclusionary, particularly to typically underrepresented and underserved members of the community. We also wish to explore the extent to which Q is perceived as equal, diverse and inclusive by its members as part of the evaluation. We want to ensure that those participating in the evaluation feel supported to share their views in confidence with the evaluation team.

Timeline

The evaluation commenced in April 2023. Throughout the course of the evaluation, we will be meeting regularly with the Q delivery team to feed back on progress. Key project timings are outlined below:

Stage	Timing
Project inception and set up	April 2023
Initial interviews with Q delivery team stakeholders (up to 6)	May-June 2023
Initial stakeholder workshop	June-July 2023
Desk review of Q member and management data	July-August 2023
Implement first survey with Q members	October-November 2023
Second stakeholder workshop	November-December 2023
Recruitment of Q members to qualitative interviews	January-February 2024
Conduct qualitative interviews with Q members	March-July 2024
Implement second survey with Q members	September-October 2024
Case study development	October 2024-February 2025
Data analysis and interpretation	Ongoing and iterative
Final stakeholder workshop	December 2024-January 2025
Final report	June-July 2025

Who is conducting the independent evaluation?

The evaluation is being conducted by Picker Institute Europe, (‘Picker’) along with support from ResPeo and Health Care Priority Solutions LTD. Picker is an international charity with a vision of ‘the highest quality person centred care for all, always’. We use people’s experiences of care to measure, understand, and improve the quality of person centredness and to identify priorities in the design and delivery of health and social care. Our ‘Picker Principles of Person Centred Care’ are an internationally renowned quality improvement framework that underpin measurement of high quality person centred care. The Principles articulate our belief that everybody deserves timely access to safe, high quality care that addresses their own individual preferences and needs. We are committed to sharing our knowledge and expertise, developing new services and tools that enable people’s experiences to be valued and used. We are here to:

• Influence policy and practice so that health and social care systems are always centred around people’s needs and preferences.
• Inspire the delivery of the highest quality person centred care, developing tools and services that enable people’s experiences to be better understood.
• Empower staff working in health and social care to improve experiences by effectively measuring and acting upon people’s feedback.

ResPeo provides research, evaluation and facilitation services to public sector, charity and community interest clients. Set up in 2018, ResPeo predominantly works with health and social care organisations on projects which hold the potential for societal betterment. Like Picker, ResPeo are driven by person-centred values, drawing on Rogerian principles of supporting human thriving and meaningful relationships through their work. ResPeo are a team of academics, evaluators and methodologists, several of whom are former clinicians and public sector leaders. They recognise the value of listening to the most significant members of organisations – staff, clients and stakeholders. At the core of their work is the use of rigorous research to understand people and to help organisations and their members flourish.

Healthcare Priority Solutions (HPS) was founded by scientific researchers with deep expertise and broad experience in priority setting and health technology assessment. HPS has access to a network of scientific researchers, mainly health economists, that allows an optimal matching of customer needs with the most suitable expertise and skills. Our vision is to support governments, health authorities, and other public (or charitable) organisations to make informed decisions that maximise the impact of health care resources and improve population health. HPS specialises in economic evaluation, decision support modelling, cost and budget impact analysis, prioritisation and value-for-money analysis, and Multi-Criteria Decision Analysis.

The work will be guided by an Evaluation Advisory Group, which will be independently chaired and include a combination of academics and researchers, Q members (including patient representatives), Q country partners, and Q staff.

The project team

How do I find out more?

If you have any questions about the evaluation, o would like more information, please contact:

Amy Tallett, Head of Research at Picker

+44 (0) 1865 208115

Amy.Tallett@PickerEurope.ac.uk

Molly Blackwell, Senior Research Associate at Picker

+44 (0) 1865 208111

Molly.Blackwell@PickerEurope.ac.uk

Download the Research Protocol

Download the Research Protocol