Developing effective feedback on quality of anaesthetic care: what are its most valuable characteristics from a clinical perspective?

Abstract

Objectives

Research suggests that better feedback from quality and safety indicators leads to enhanced capability of clinicians and departments to improve care and change behaviour. The aim of the current study was to investigate the characteristics of feedback perceived by clinicians to be of most value.

Methods

Data were collected using a survey designed as part of a wider evaluation of a data feedback initiative in anaesthesia. Eighty-nine consultant anaesthetists from two English NHS acute Trusts completed the survey. Multiple linear regression with hierarchical variable entry was used to investigate which characteristics of feedback predict its perceived usefulness for monitoring variation and improving care.

Results

The final model demonstrated that the relevance of the quality indicators to the specific service area (β = 0.64, p = 0.01) and the credibility of the data as coming from a trustworthy, unbiased source (β = 0.55, p = 0.01) were the significant predictors, having controlled for all other covariates.

Conclusion

For clinicians to engage with effective quality monitoring and feedback, the perceived local relevance of indicators and trust in the credibility of the resulting data are paramount.

Keywords

feedback patient safety quality indicator

Introduction

The Francis Report called for information that is accessible and useable by all, allowing for effective comparison of performance by individuals, services and organizations.¹ Effective monitoring and feedback from quality indicators makes variations visible to health care professionals and can potentially support timely remedial action.

In the UK, clinician revalidation has been introduced as a mechanism to uphold and improve practice through continuous professional development.² Supporting professional development requires effective design of quality monitoring systems capable of delivering accurate, timely and useful feedback to clinicians based upon valid and reliable quality indicators.³

A recent systematic review demonstrated that performance feedback to clinicians has a positive impact on behaviour and outcomes, resulting in small to moderate positive effects.⁴ Such effects have been observed in terms of reduction of mortality rates and improved compliance with guidelines, amongst other outcomes.^5,6 Tailoring feedback to the specific clinical setting has been shown to have a positive influence on its effectiveness along with ensuring those issuing the feedback are perceived as experts.^4,7–10 High intensity and frequency of feedback improves outcomes along with sustained monitoring.^7,10–13 A number of strategies have been identified to support the effectiveness of feedback, including providing recipients with information on specific areas for improvement, action planning and educational components.^{4,7,8,10–12} It has been suggested that the departmental context in which feedback is administered may be important.^14,15

Little systematic research exists, however, to guide development of quality monitoring and feedback mechanisms that clinicians themselves regard as effective.^3,16 One study of this type used a qualitative methodology and identified a number of characteristics for effective feedback, according to clinical and administrative staff. They included: the perceived validity and credibility of the data; their source and timeliness; the way units are benchmarked and the avoidance of individual profiling that could be misconstrued as punitive.¹⁷ Other relevant studies focussed on evaluating existing approaches to feedback from the perspective of clinicians. These include multisource feedback^18–21 and benchmarked feedback of patient reported outcome measures.²²

Existing approaches are more concerned with evaluation of audit and feedback interventions with limited attention to what makes specific interventions successful. Greater research effort needs to be devoted to understanding the underlying mechanisms through which feedback is effective.²³ Identifying the characteristics of feedback perceived by clinicians to be of most value may go some way in commenting on the ingredients required to produce the most desirable effects from audit and feedback as a quality improvement tool.

In anaesthetic services, intraoperative process data and recovery room quality indicators such as core temperature, patient reported pain, nausea, and quality of recovery may serve as the basis for feedback.^17,24,25 The challenges of developing sensitive and reliable quality indicators and patient satisfaction measures in this clinical area are well documented.^2,24,26,27 There is currently considerable specialty interest in developing national standardized guidance for perioperative quality monitoring and reporting in anaesthesia.²⁸

Currently, anaesthetists rarely receive continuous, systematic feedback on anaesthetic quality to support professional behaviour change and service improvement.^17,24 Given the likelihood that feedback mechanisms are linked to organizational and departmental quality assurance, perceptions of feedback are likely to vary as a function of tenure, organizational context, and local unit climate for quality improvement. The role of the level of feedback, its intensity and more specific design characteristics may additionally influence perceived utility. Given the limited evidence as to the important characteristics of feedback from a clinician's perspective, the aim of the current study is to investigate which characteristics are of most value. We use baseline data collected in the evaluation of an initiative for anaesthetists to explore the role of a range of demographic, contextual, and design characteristics in predicting anaesthetists’ perceptions of utility of data feedback.

Methods

Study design

A cross-sectional survey design was used with data collected at a single time point.

Participants and data collection

Two UK NHS organizations with large perioperative service departments were selected as the basis for the study sample. These sites were chosen because research leads based at both sites were collaborators on a programme to develop perioperative quality indicators as part of a national initiative in the UK.²⁸

Data were collected from consultant anaesthetists prior to any local development of quality monitoring programmes.

Research measures

The survey measure was developed by a multidisciplinary research team working at the primary study site (Organization A). Initial survey items were developed based upon literature review of emerging theory in the area of data feedback for quality improvement. Questions were designed to quantify specific characteristics that have been identified as important, e.g. relevance, validity, reliability and applicability.³ The approach taken was that anaesthetists should evaluate outcome measures against their potential to lead to improvements in standards of care and benchmarking.²

Iterations of the survey items were discussed and refined by three consultant anaesthetists and a social sciences researcher with experience in survey design. The survey was piloted using a cognitive walkthrough technique with two additional consultant anaesthetists in which presentation, item interpretation and wording were clarified through a structured interview in which participant interpretations and responses to the survey items were verbalized and discussed with the researchers. Survey items were refined based upon the results from this exercise.

The survey measure comprised four items assessing the comprehensiveness of local quality monitoring, i.e. whether clinicians received regular quantitative feedback on a number of quality dimensions. These items were summed into a scale representing comprehensiveness of monitoring for the purposes of the regression analysis. The level of care that the feedback focused upon (i.e. care at the departmental/individual level) was then assessed.

Twenty five items evaluated three key areas: (1) perceptions of current quality indicators (comprehensiveness/relevance/reliability/improvability), (2) perceptions of current feedback from quality indicators (level of analysis/timeliness/means of communication/data presentation/data credibility), and (3) local departmental climate (comprising 16 items designed to assess features of the local departmental context and climate for quality improvement). Responses to items evaluating the effectiveness and usefulness of quality indicators and feedback were measured on an eight-point Likert scale ranging from 1 (completely inadequate) to 8 (excellent). Responses to items evaluating the departmental climate for quality improvement were measured on an eight-point Likert agreement scale ranging from 1 (strongly disagree) to 8 (strongly agree) and aggregated into a single scale score. Box 1 provides examples of the variables included, along with an internal consistency metric (Cronbach’s alpha) for the aggregated score. The dependent measure for the study was based upon a single survey item: ‘the degree to which current data feedback is useful in monitoring variations and improving care’ and rated on an eight-point scale ranging from 1 (completely inadequate) to 8 (excellent).

Box 1.

Departmental climate for quality improvement.

Examples of variables included in a 16-item scale to measure departmental climate for quality improvement
•Processes of monitoring and quality improvement at departmental level
•Constructive response to observed variations in care
•Openness of professional climate for discussing failures
•Organisational support for departmental quality improvement initiatives
•Professional competency with quality improvement and statistics amongst clinicians
•Responsibility for acting on observed variations in care
•Willingness to disclose personal performance data to the department
Cronbach’s alpha of the combined items = 0.91

Statistical analysis

Multiple linear regression analysis with hierarchical variable entry was performed. The following hypotheses were tested using hierarchical entry of specific predictors in steps. For details of the predictors that were entered to test each hypothesis please refer to Table 4 in the ‘Results’ section.

Hypothesis 1: Length of time since qualification (tenure) will influence perception of the degree to which current local data feedback is useful for monitoring variation and improving care in anaesthesia.

Hypothesis 2: Organizational membership will influence perception of the degree to which current local data feedback is useful for monitoring variation and improving care in anaesthesia.

Hypothesis 3: The reported local departmental climate for quality improvement will influence perception of the degree to which current local data feedback is useful for monitoring variation and improving care in anaesthesia.

Hypothesis 4: The reported scope of local quality monitoring will influence perception of the degree to which current local data feedback is useful for monitoring variation and improving care in anaesthesia.

Hypothesis 5: The design characteristics of feedback will influence perception of the degree to which current local data feedback is useful for monitoring variation and improving care in anaesthesia.

The statistical significance of the additional proportion of variance in the dependent measure accounted for by each successive entry of variables was assessed in order to establish the role of each specific hypothesized predictor, having controlled for previously entered factors.²⁹ Forced entry regression was selected as the most appropriate method, as preexisting research findings and theory (outlined earlier) are available to support the order of causal and temporal priority amongst the independent variables.

Results

Descriptives

Eighty-nine respondents from two acute healthcare organizations participated in the study. This represents a response rate of 59% (70% for Organization A and 48% for Organization B). Eighty-two (92.1%) participants were consultants, six (6.7%) were trainees and one (1.1%) was nonconsultant faculty. Anaesthetists included were from a mixture of specialties typical of a large, urban, academic teaching hospital. Following exclusion due to missing data, 78 survey responses were included in the regression analysis.

Seventy six per cent of participants had been qualified (Medical Undergraduate Degree) for between 11 and 30 years and the mean length of time since qualification was 20 years (SD = 8.1). Respondent characteristics are presented within Table 1.

Table 1.

Respondents’ characteristics: descriptive information about the clinicians that were included in the analysis.

	N (Number of Cases)	% (Percentage)
Length of time since Qualification (Tenure)
0–10 years	10	13
11–20 years	31	40
21–30 years	28	36
31–40 years	8	10
41–50 years	1	1
Acute Healthcare Organisation
Acute healthcare organisation A	52	67
Acute healthcare organisation B	26	33

The overall scope of local quality monitoring, with a mean value of 0.85 (SD = 1.20), was notably low (from a range of 0 to 4). This was reflected in the amount of feedback being received by participants on both levels of care (departmental and individual). The dependent variable, with a mean value of 2.83 (SD = 2.01), indicates that perceived usefulness of feedback for monitoring variations and improving care at these organizations was generally low. Table 2 presents categorical items and their frequency of responses whilst Table 3 presents mean scores and standard deviations of all scale items included in the regression model.

Table 2.

Descriptive statistics of survey items: percentage responses to categorical variables, with number of responses shown in brackets, split by acute healthcare organisation.

		Percentage (N)
		Acute Healthcare Organisation A	Acute Healthcare Organisation B	Total
Level of care focussed upon by quality monitoring	I receive monthly or more regular feedback concerning the care delivered by my department
	True	23 (12)	19 (5)	22 (17)
	False	77 (40)	81 (21)	78 (61)
	I receive monthly or more regular feedback concerning the care that I delivered personally
	True	15 (8)	8 (2)	10 (13)
	False	85 (44)	92 (24)	68 (87)

Table 3.

Descriptive statistics of survey items: mean responses to scale variables, with standard deviation shown in brackets, split by acute healthcare organisation.

Survey Item	Mean (SD) Acute Healthcare Organisation A	Mean (SD) Acute Healthcare Organisation B	Mean (SD) Total
	Scope of local quality monitoring (0–4)
Aggregated scale of categorical responses to the four quality dimensions (clinical effectiveness of care; compliance with best practice guidelines; patient safety; patient experience)	0.83 (1.15)	0.88 (1.31)	0.85 (1.20)
	Perceptions of the current quality indicators in your area (1–8)
Comprehensiveness: The degree to which the data you receive are comprehensive and cover all important dimensions of care quality	2.48 (1.58)	2.46 (1.66)	2.47 (1.59)
Relevance: The degree to which care quality indicators are unambiguous and specific to our service area and the care we routinely deliver to patients	2.63 (1.74)	2.42 (1.72)	2.56 (1.73)
Reliability: The degree to which indicators are objective and reliable indicators of current standards of care, promoting confidence in the accuracy of the data over time	2.77 (1.83)	2.54 (1.79)	2.69 (1.81)
Improvability: The degree to which indicators measure aspects of care that you and your unit can have a direct impact upon through changing behaviour, the care process or local systems	3.21 (2.12)	2.58 (1.84)	3.00 (2.04)
	Perceptions of the current feedback that you receive (1–8)
Level of analysis: The degree to which the data that you receive is broken down to a level that is directly relevant to you (e.g. for your team, your ward, your operating theatre, your patients)	2.46 (1.71)	2.35 (1.83)	2.42 (1.74)
Timeliness: The degree to which the frequency of feedback you receive helps you to monitor how care quality varies over time	2.50 (1.76)	2.50 (1.88)	2.50 (1.79)
Means of communication: The degree to which the channel and method for dissemination (e.g. meetings, email, reports, posters) are useful and engaging	2.87 (1.86)	3.35 (2.10)	3.03 (1.94)
Data presentation: The degree to which the format in which data are presented (e.g. tables, graphs, scorecards) is clear and easy to use with the right amount of data presented	2.56 (1.81)	3.04 (2.01)	2.72 (1.88)
Data credibility: The degree to which the data are viewed as credible and from a trustworthy, unbiased source	2.56 (2.07)	2.50 (1.77)	2.54 (1.97)
	Local departmental climate for quality improvement (1–8)
Aggregated scale of all 16 scale items	4.67 (1.25)	4.77 (1.36)	4.71 (1.28)
	Dependent variable: Usefulness of current local data feedback (1–8)
Overall usefulness for improvement: The degree to which current data feedback is useful in monitoring variations and improving care	2.83 (2.09)	2.85 (1.87)	2.83 (2.01)

Regression analysis

The statistical model parameters of the different stages of the regression analysis examining the significance of the hypothesized predictors of usefulness of data feedback are given in Table 4. Regarding study hypotheses one and two, neither tenure nor organizational membership significantly predicted perceived usefulness of current data feedback. The departmental climate for quality improvement (hypothesis 3) explained an additional 27.5% of the variance in the usefulness measure (p < .0001). The stronger the perception of a departmental climate for quality improvement, the greater the perception of the degree to which current local data feedback was viewed as useful for monitoring variations and improving care. In the third model in Table 4, partialling out the effects of all prior predictors resulted in departmental climate for quality improvement making a significant positive contribution to the dependent variable (β = 0.83, p < .0001). When hypothesis 4 was investigated, the scope of local quality monitoring explained a further significant 11.2% of the variance in local usefulness of data feedback (p = 0.006). In this model, both comprehensiveness of feedback received (β = 0.45, p = 0.02) and provision of feedback at the level of the individual clinician (β = 1.19, p = 0.049), as opposed to department level feedback, were significant predictors of local usefulness, once prior factors had been controlled for.

Table 4.

Model summary: overview of statistics illustrating model fit for each of the five study hypotheses.

Model sequence and description	R	R Square	R Square Change	F Change	Significance of F Change
Tenure (hypothesis 1)	0.14	0.02	0.02	1.60	0.21
Tenure + Trust (hypothesis 2)	0.14	0.02	0.00	0.002	0.97
Tenure + Trust + Departmental climate for quality improvement (hypothesis 3)	0.54	0.30	0.28	28.94	<.0001
Tenure + Trust + Departmental climate for quality improvement + Scope of local quality monitoring/Level of care focussed upon (hypothesis 4)	0.64	0.41	0.11	4.48	0.006
Tenure + Trust + Departmental climate for quality improvement + Scope of local quality monitoring/Level of care focussed upon + Generic characteristics of feedback (hypothesis 5)	0.82	0.67	0.26	5.55	<.0001

In the final fitted model (hypothesis 5), a number of variables representing feedback design characteristics were entered in the model, after controlling for all prior entered factors, including tenure, organizational membership, local contextual factors, and the scope of any local quality monitoring initiatives. Feedback characteristics explained a further 26.4% of the variance in perceived local usefulness (p < .0001). The final model demonstrated that with the effects of all other factors held constant, two characteristics were significant predictors of usefulness (Table 5). These were the perceived relevance of the quality indicators to the specific service area (β = 0.64, p = 0.01) and the perceived credibility of the data as coming from a trustworthy, unbiased source (β = 0.55, p = 0.01).

Table 5.

Coefficients: model parameters for final fitted model with all variables entered.

Final model with all variables entered	Unstandardized coefficients		Standardized coefficients	t	Sig	95% confidence interval for β
Final model with all variables entered	β	Std error	β	t	Sig	Lower bound	Upper bound
Constant	−0.40	0.72		−0.56	0.58	−1.85	1.04
Tenure	−0.01	0.02	−0.03	−0.33	0.74	−0.05	0.03
Trust	0.01	0.34	0.001	0.01	0.99	−0.68	0.68
Departmental climate for quality improvement	0.30	0.16	0.19	1.82	0.07	−0.03	0.62
Comprehensiveness of dimensions of feedback received	0.12	0.17	0.07	0.70	0.49	−0.22	0.46
Departmental feedback	−0.52	0.45	−0.11	−1.16	0.25	−1.41	0.37
Personal feedback	−0.43	0.60	−0.07	−0.72	0.47	−1.64	0.77
The degree to which data are comprehensive	−0.19	0.22	−0.15	−0.86	0.39	−0.62	0.24
The degree to which indicators are relevant to the specific service area	0.64	0.25	0.55	2.57	0.01	0.14	1.13
The degree to which indicators are reliable and accurate	−0.05	0.24	−0.04	−0.20	0.84	−0.52	0.42
The degree to which indicators measure aspects of care that can be improved	0.02	0.20	0.02	0.08	0.94	−0.38	0.41
The degree to which data analysis is at a level which is relevant to you	−0.36	0.19	−0.31	−1.94	0.06	−0.73	0.01
The degree to which frequency of feedback helps monitor trends	0.07	0.16	0.06	0.41	0.69	−0.26	0.39
The degree to which feedback is communicated effectively	0.001	0.13	0.001	0.01	0.996	−0.26	0.27
The degree to which data presentation is adequate for effective use	0.09	0.22	0.089	0.42	0.67	−0.34	0.53
The degree to which the source of the data is credible	0.55	0.22	0.54	2.57	0.01	0.12	0.98

Discussion

This study aimed to investigate the characteristics of feedback that are perceived by clinicians to be of most value.

Neither tenure nor organizational membership significantly influenced perceptions of usefulness, demonstrating that there were no significant differences in perceptions of current local feedback attributable to professional experience or due to location at either study site. Variations in perceptions of local departmental climate for quality improvement, however, was a significant predictor accounting for a large proportion of the variance in the dependent measure (27.5%). This finding is interesting as it suggests that without a supportive local context, providing information on variations in care may not result in improvement. It is additionally compatible with prior research, which suggests that having a local operating culture conducive to quality and safety improvement is an important contextual factor influencing the success of local initiatives.^14,15,30

Both the scope of local quality monitoring and the level of feedback were significant predictors of perceived usefulness. This suggests that the more dimensions of care on which an individual receives feedback, the more useful information they have to interpret and from which to learn. Higher overall intensity of feedback has been shown to increase its effectiveness.⁴ In our model, receiving feedback on care delivered by the individual practicing clinician was a strong positive predictor of perceived usefulness. This finding reinforces the notion that personal professional feedback is important in learning and improving practice.^9,16

The final regression model investigated the role of feedback design characteristics, which were found collectively to explain a large proportion (26.4%) of the remaining variance in perceived usefulness, once all prior factors had been controlled for. As expected, the design of feedback is clearly the most important predictor of its utility. When all design characteristics were entered into the model simultaneously, only two factors were found to have a significant unique effect upon perceived usefulness of feedback. These were the relevance of the quality indicators to the specific service area and the credibility of the data as coming from a trustworthy, unbiased source.

Prior research has highlighted the importance of the perceived credibility of data from quality indicators and the extent to which it originates from a trusted source.^3,4,9,31 Qualitative findings highlight the importance of investing time to establish the credibility of performance data and involving respected members of senior staff to achieve this.¹⁷ Two systematic reviews support this concept further by identifying feedback provided by experts as being more effective.^7,10

The local relevance of quality indicators has additionally been highlighted as an important characteristic of effective feedback. Two systematic reviews concluded that tailoring a feedback intervention to the local setting augmented its effectiveness.^8,9 In anaesthesia, a recent national survey study in the UK has demonstrated that current practice in monitoring and feedback is focussed upon high-level perioperative outcomes, productivity, and efficiency indicators, rather than quality of care or patient experience measures.²⁸ The findings from the current study suggest these types of indicators may be perceived as less useful for quality improvement due to their limited local or clinical relevance.

Although the findings from this study add to our understanding of how to design better feedback from a clinician’s perspective, limitations mean the results should be treated with caution. The sample of respondents was relatively small and based on two organizations only. However, these organizations were both large hospitals and overall response rate was good. It is possible that findings based upon teaching hospitals may reduce generalizability. However, it might conversely be argued that in the preliminary stages of research in this area, clinicians in an academic medical setting are likely to be practicing within a data-rich environment and therefore better able to rate utility. Survey measures are open to a number of respondent biases, though subjectivity may be considered a strength where an ‘end-user’ or ‘stakeholder’ perspective is required. Whilst perceived usefulness is not synonymous with effectiveness, it is likely to govern engagement and uptake of the results from quality monitoring programmes.

Our findings should serve as a basis for further research, which might usefully employ longitudinal investigation of the effects of feedback interventions with specific design and implementation characteristics. Such work might, for example, clarify the potentially reciprocal relationship between the development of quality monitoring processes and local departmental climate for open and effective use of performance data. Does a more supportive climate lead to enhanced feedback or vice versa? Our study suggests that where feedback is limited, the local climate is important in determining its usefulness.

Given current enthusiasm for using quality and safety monitoring to drive improvements in practice, there is surprisingly little evidence to inform development of effective feedback from quality indicators. The findings from this study suggest clinicians perceive a range of factors as important in determining the usefulness of feedback. Specifically, local departmental context and its support of quality improvement is an important determinant of how instrumental feedback from monitoring quality indicators is likely to be. Furthermore, feedback tailored to the personal professional practice of the individual clinician is an important predictor of usefulness. In terms of the feedback design characteristics clinicians value most, the perceived credibility of the data and the local relevance of the quality indicators are paramount.

Footnotes

Acknowledgements

The authors wish to thank Dr Mark Sacks, Dr Christopher Stonell and Dr Claire Moloney for their contributions to the development of the survey measure and data collection for this study. We additionally thank Ms Floor Aleva for her contributions to the broader research programme upon which this study is based.

Declaration of competing interests

The authors declare that there is no conflict of interest.

Funding

This work was supported by funding from the National Institute for Health Research (NIHR) Health Services and Delivery Research (HSDR) Programme (project number 11/1015/21), the Collaboration for Leadership in Applied Health Research and Care (CLAHRC), and the NIHR Imperial Patient Safety Translational Research Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

Ethics approval

This study was approved as a service development project.

References

Francis

. Independent inquiry into care provided by Mid Staffordshire NHS Foundation Trust January 2005—March 2009, London: The Stationary Office, 2010.

Moonesinghe

Tomlinson

. Quality improvement and revalidation: two goals, same strategy? Brit J Anaesth 2011; 106: 447–450.

Wollersheim

Hermens

Hulscher

. Clinical indicators: development and applications. Neth J Med 2007; 65: 15–22.

Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; 6, CD000259.

Wright

Dugdale

Hammond

. Learning from death: a hospital mortality reduction programme. JRSM 2006; 99: 303–308.

O’Reilly

Talsma

VanRiper

. An anesthesia information system designed to provide physician-specific feedback improves timely administration of prophylactic antibiotics. Anesth Analg 2006; 103: 908–912.

Alvero

Bucklin

Austin

. An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings (1985–1998). J Organ Behav Manage 2001; 21: 3–29.

Chaillet

Dube

Dugas

. Evidence-based strategies for implementing guidelines in obstetrics: a systematic review. Obstet Gynecol 2006; 108: 1234–1245.

van der Veer

de Keizer

Ravelli

. Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform 2010; 79: 305–323.

10.

Veloski

Boex

Grasberger

. Systematic review of the literature on assessment, feedback and physicians’ clinical performance*: BEME Guide No 7. Med Teach 2006; 28: 117–128.

11.

De Vos

Graafmans

Kooistra

. Using quality indicators to improve hospital care: a review of the literature. Int J Qual Health Care 2009; 21: 119–129.

12.

Hysong

. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care 2009; 47: 356–356.

13.

Aita

Zanier

Rijavec

. Toward incident reporting (IR) systems: a safety culture survey within the oncology network of a northern Italian region. J Clin Oncol 2009; 1: e17527– e17527.

14.

Ovretveit

Shekelle

. How does context affect interventions to improve patient safety? BMJ Qual Saf 2011; 20: 604–610.

15.

Kaplan

Provost

Froehle

. The Model for Understanding Success in Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf 2012; 21: 13–20.

16.

Benn

Arnold

Wei

. Using quality indicators in anaesthesia: feeding back data to improve care. Brit J Anaesth 2012; 109: 80–91.

17.

Bradley

Holmboe

Mattera

. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Safety Health Care 2004; 13: 26–26.

18.

Sargeant

Mann

Sinclair

. Challenges in multisource feedback: intended and unintended outcomes. Med Educ 2007; 41: 583–591.

19.

Burford

Illing

Kergon

. User perceptions of multi-source feedback tools for junior doctors. Med Educ 2010; 44: 165–176.

20.

Violato

Lockyer

Fidler

. Changes in performance: a 5-year longitudinal study of participants in a multi-source feedback programme. Med Educ 2008; 42: 1007–1013.

21.

Overeem

Wollersheim

Driessen

. Doctors’ perceptions of why 360-degree feedback does (not) work: a qualitative study. Med Educ 2009; 43: 874–882.

22.

Boyce

Browne

Greenhalgh

. Surgeon’s experiences of receiving peer benchmarked feedback using patient-reported outcome measures: a qualitative study. Implement Sci 2014; 9: 84–84.

23.

Ivers

Sales

Colquhoun

. No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014; 9: 14–14.

24.

Barnett

Alagar

Grocott

MPW

. Patient-satisfaction measures in anesthesia: qualitative systematic review. Anesthesiology 2013; 119: 452–478.

25.

Myles

Weitkamp

Jones

. Validity and reliability of a postoperative quality of recovery score: the QoR-40. Brit J Anaesth 2000; 84: 11–15.

26.

Haller

Stoelwinder

Myles

. Quality and safety indicators in anesthesia: a systematic review. Anesthesiology 2009; 110: 1158–1175.

27.

Castanelli

Kitto

. Perceptions, attitudes, and beliefs of staff anaesthetists related to multi-source feedback used for their performance appraisal. Brit J Anaesth 2011; 107: 372–377.

28.

The Royal College of Anaesthetists. The use of perioperative quality indicators in NHS services: A national survey. In Press.

29.

Cohen

West

. Applied multiple regression/correlation analysis for the behavioral sciences, 3rd ed. London: Lawrence Erlbaum Associates, 2003.

30.

Pronovost

Needham

Berenholtz

. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med 2006; 355: 2725–2732.

31.

Ilgen

Fisher

Taylor

. Consequences of individual feedback on behavior in organizations. J Appl Psychol 1979; 64: 349–349.