Double reading rates and quality assurance practices in Norwegian hospital radiology departments: two parallel national surveys

Abstract

Background

Double reading as a quality assurance (QA) tool is employed extensively in Norwegian hospital radiology departments. The practice is resource consuming and regularly debated.

Purpose

To investigate the rates of double reading in Norwegian hospital radiology departments, to identify department characteristics associated with double reading rates, and to investigate associations between double reading and other quality improvement.

Material and Methods

We issued two parallel national surveys to management and to consultant radiologists, respectively. Management was defined as the chief medical officer and/or the head of the radiology department. The management survey covered staffing, perceived resource situation, double reading, guidelines, and quality improvement. The radiologist survey served to validate management responses concerning double reading. Management survey items concerning practices of quality improvement were organized into three indices reflecting different quality approaches, namely: appropriateness of investigations; personal performance feedback; and system performance feedback.

Results

The response rates of the surveys were 100% (45/45) for management and 55% (266/483) for radiologists. Of all exams read by consultants, 33% were double read. The double reading rate was highest in university hospital departments (59%), intermediate in other teaching departments (30%), and lowest in non-teaching departments (11%) (P = 0.01). Among the quality indices, mean scores were highest on appropriateness index (68%), intermediate on the person index (56%), and lowest on system index (37%). There were no correlations between double reading rates and scores on any of the quality indices.

Conclusion

The rate of double reading in Norwegian hospital radiology is significantly correlated to department teaching status, but not to other practices of quality work.

Keywords

QA/QC safety health policy and practice and observer performance

Introduction

Double reading is a practice in which two readers interpret an imaging exam. It can be carried out in a number of ways and for a number of purposes. Double reading as a quality assurance (QA) tool in radiology has been demonstrated to reduce errors and increase sensitivity though it may be at the expense of specificity (1,2). Accordingly the practice reduces the number of false-negative interpretations by uncovering pathology missed in the initial interpretation but also increase the number of false-positive findings by misinterpretation of normal findings as pathology.

Coupled with systematic analysis of discrepancies double reading may reveal patterns of errors in the work of individual readers, allowing for appropriate feedback about the need for further education (3). It is routinely employed in education and QA when resident radiologists read exams independently, but may benefit experienced and inexperienced radiologists alike (3 –5). However, the cost efficiency of double reading has yet to be established (3).

Independent double reading, in which readers are blinded to the interpretation of their colleague, is used in several mammography-screening programs (6). In clinical radiology double reading is usually sequential and non-independent as the second reader is not blinded to the preceding interpretation of the first reader (3).

In the USA 75% of radiology departments conduct retrospective double readings, as a random review of 5% of cases is required for department accreditation by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) (7 –9).

In Scandinavian countries there is a tradition of extensively double reading, even exams read by consultant radiologists (4,10). This practice predates the introduction of Picture Archiving and Communications Systems (PACS). Residents or consultants would first read the imaging exams continuously as they were prepared on image alternators. A second reading would usually be performed the following morning by a consultant radiologist in preparation for daily radiology rounds, when all exams from the preceding day would be presented to the referring department for discussion (11).

The introduction of PACS made images available to the clinicians and radiology rounds were gradually substituted by less frequent clinical radiology conferences (11). In many departments consultant radiologists retained the option to submit exams for a second reading for the purpose of QA or to sign their own reports immediately (11). The consultant may decide to assign a particularly problematic exam, or an exam not in their field of expertise, for double reading. Few Norwegian radiology departments conduct random double reading or have guidelines for double reading (10).

As expected, in the absence of national guidelines and recommendations, Norwegian radiology departments vary substantially in their rates of double reading (10). Similar variation might be expected with regards to other practices of continuous quality improvement (CQI) and QA. Our objectives were: first, to investigate the rates of double reading in Norwegian hospital radiology departments; second, to identify department characteristics associated with the rates of double reading; and third, to investigate possible associations between double reading rates and other practices of CQI and QA.

Material and Methods

Survey design and participants

Between 27 March and 27 May 2012 two parallel nationwide electronic surveys were conducted in the Norwegian radiology community. These were directed at department management and consultant radiologists, respectively. Management was defined as the chief medical officer and/or the head of the radiology department. Although private imaging centers account for 23% of all performed imaging exams in Norway (2008) private imaging centers were not included in the surveys (12). This was because a previous study has shown that private imaging centers perform double reading to a limited degree compared with hospital radiology departments (10).

Questionnaires

A group consisting of a professor of medicine and three radiologists, two of whom were also department heads, developed the surveys.

Items in the management survey concerning department staffing, practice of double reading, and perceived resource situation are listed in Table 1. Double reading was defined as exams read by two consultants. Responders were instructed not to include over reading of residents by consultants. Management survey items concerning department guidelines, CQI and QA work are listed in Table 2. Estimates of working hours assigned to double reading were requested, with identical response options, in both (management and radiologist) surveys.

Table 1.

Management survey items included in report.

Survey item	Possible responses, if limited
Name of department Consultant FTEs (n) Resident FTEs (n)
Estimated double reading rate of all exams read by consultants Estimated percent of consultant working hours spent double reading	… %
Estimated double reading rate of CT exams read by consultants Estimated double reading rate of MRI exams read by consultants Estimated double reading rate of X-ray exams read by consultants Estimated double reading rate of ultrasonography exams read by consultants Estimated double reading rate of interventional procedures performed by consultants	0% / 1–33% / 34–66% / 67–99% / 100%
My department has sufficient resources to meet demands The radiologist manpower of my department is insufficient Insufficient radiologist manpower limits double reading rates I would prioritize increasing double reading rates if department resources were increased	Completely agree / Moderately agree / Indifferent / Moderately disagree / Completely disagree

CT, computed tomography; FTE, full-time equivalent; MRI, magnetic resonance imaging.

Table 2.

Items included in indices of three approaches to quality.

Item*	Affirmative n (%)
Person index (Person monitoring and feedback approach)^†
• Our guidelines dictate explicit notification to the first reader of errors uncovered by double reading	37 (82)
• Our guidelines dictate explicit notification to the referring physician of errors uncovered by double reading	36 (80)
• Incorrect diagnoses are recorded according to specific radiologists	4 (9)
• Complications to invasive procedures are recorded according to specific radiologists	24 (53)
System index (System monitoring and feedback approach)^†
• Our guidelines dictate logging of errors to prevent systematic errors and recurrences	7 (16)
• Regular meetings are held to discuss selected cases	36 (80)
• The department has a system in place to identify missed diagnoses	8 (18)
• Diagnostic disagreements uncovered by interpretation of subsequent exams are systematically collected and processed.	9 (20)
• Surgical and histological diagnoses are systematically correlated to preceding radiology reports	8 (18)
• Complications to invasive procedures are systematically recorded	25 (56)
• Complications to invasive procedures are recorded according to type of procedure	30 (67)
• Quality assessment data is reviewed at regular intervals	25 (56)
• Routine retrospective double readings of randomly chosen completed examinations are performed	3 (7)
Appropriateness index (Appropriateness of investigation approach)^†
• There are guidelines to assess the appropriateness of imaging	14 (31)
• There are appropriateness criteria for exams with a high radiation dose	21 (47)
• There are appropriateness criteria for invasive procedures	28 (62)
• Our guidelines state that referrals are assessed by a radiologist prior to the exam when involving intravenous contrast media	39 (87)
• Our guidelines state that CT referrals are assessed prior to the exam by a radiologist	41 (91)
• Our guidelines state that MRI referrals are assessed prior to the exam by a radiologist.^‡	36 (86)
• Our guidelines state that ultrasonography referrals are assessed prior to the exam by a radiologist.	32 (71)

A total of 874 of 897 possible responses (97%) were given by the 45 responders to these 20 items.

†

Indices are percentages of possible affirmative responses on index items.

‡

Three departments did not conduct MRI.

CT, computed tomography; MRI, magnetic resonance imaging.

Recruitment

The management of all (45) hospital radiology departments in Norway was invited to participate in the survey. All management invitees were contacted by telephone prior to receiving the electronic survey. Non-responders were reminded by email and again by telephone prior to closing the survey.

The target population of the radiologist survey was consultant radiologists performing clinical work in Norwegian hospital radiology departments (excluding management). In order to reach this target population, all 726 members of the Norwegian association of radiologists were invited by email to participate in the electronic survey. Non-responders were reminded by email. Responders not belonging to the target population were excluded.

Adjuvant information

Information about staffing was also acquired from the SERUS (system for electronic reporting of educational activities in hospital departments) reporting system. All approved university hospital departments and other teaching departments in Norway annually report details of staffing, production, and educational activities to the SERUS reporting system handled by the Norwegian Medical Association. The staffing information was used to calculate the size of the target population of the radiologist survey and the proportion of residents in the radiologist staff.

Analysis

The department management was assumed to provide the most reliable data regarding department operations as a whole. These data were used in the main analysis. Consultant radiologists accounted their individual practice, and grouped according to workplace they served to validate department management responses about double reading.

Duplicate management responses from the same department (one from the department head and one from the chief medical officer) were merged into one response (Fig. 1). When merging discrepant responses to individual survey items priority was given according to the responsibilities of the management representatives for each survey item.

Fig. 1.

Overview of management responders.

For purposes of analysis management survey items concerning guidelines, CQI, and QA practices were organized in three quality indices. These indices were devised for the present study to reflect different approaches to quality work. The items included in each index are shown in Table 2. The four items constituting the person index reflect an approach focusing on feedback to individuals on performance. The nine items constituting the system index reflect an approach focusing on feedback to staff on system performance. The seven items constituting the appropriateness index reflect an approach focusing on the appropriateness of investigations. Index scores are percentages of possible affirmative responses. Negative, uncertain, or no response scored 0.

Departments were categorized according to teaching status into university hospital departments, other teaching departments, and non-teaching departments. The university hospital departments have the highest level of residency training program, and a minimum of 18 months residency at a university hospital department is compulsory. The other teaching departments are licensed to train residents for up to 42 months of their residency period. Non-teaching departments may not train residents.

Statistical analysis

Management reported double reading rates according to modalities in the following categories: 0%, 1–33%, 34–66%, 67–99%, or 100%. For purposes of statistical analysis the intervals 1–33%, 34–66%, and 67–99% were converted to their middle values of 17%, 50%, and 83%, respectively.

Comparisons of departments according to teaching status were made using Kruskall Wallis’ test. Linear regression was used to assess whether differences in rates of double reading between departments of different teaching status remained significant when adjusting for size. All correlations were tested with Spearman correlation.

The data were analyzed using the Statistical Package for Social Sciences, Version 20 (SPSS Inc., Chicago, IL, USA). All P values are two-sided. P < 0.05 indicates statistical significance.

Results

The management survey received 53 responses from 55 invited management representatives at all 45 departments resulting in a department response rate of 100%. Details of management responders are shown in Fig. 1.

The radiologist survey yielded 266 responses from the target population of 483 (non-management) consultant radiologists in Norwegian hospitals. This constitutes a response rate of 55%. Fig. 2 details the representativeness of responders to the radiologist survey.

Fig. 2.

Target population and responders to radiologist survey.

The mean rate of double reading reported by management was 33% of all exams read by consultants. Management estimated that double reading consumed 20% of the working hours of their consultant radiologists, while the radiologists themselves estimated spending 25% of their working hours on double reading. There was a strong and statistically significant correlation (Spearman’s rho = 0.57, P < 0.001) between the estimates of the radiologists and those of their respective managers. Table 3 summarizes rates of double reading in total and by modality.

Table 3.

Characteristics of departments; n = 45, numbers are n unless otherwise specified.

Item	University hospital departments (n = 10)	Other teaching departments (n = 27)	Non-teaching departments (n = 8)	All departments (n = 45)
Staffing, mean (SD)
• Consultant FTEs	28.1 (10.9)	8.1 (4.4)	2.7 (1.0)	11.6 (10.9)
• Resident FTEs	12.4 (3.4)	3.6 (2.5)	0	4.9 (4.9)
Quality indices*, mean (SD)
• Person index^†	63% (21)	54% (23)	56% (26)	56% (23)
• System index^‡	39% (16)	37% (20)	35% (23)	37% (19)
• Appropriateness index^§	70% (20)	70% (16)	56% (30)	68% (20)
Double reading rate**, mean (SD)
• All modalities	59% (27)	30% (30)	11% (14)	33% (31)
• CT	58% (31)	36% (35)	24% (32)	39% (35)
• MRI	60% (22)	48% (38)	4% (8)	47% (36)
• X-ray	48% (32)	20% (29)	10% (18)	24% (30)
• Ultrasonography	25% (34)	15% (30)	2% (6)	16% (29)
• Fluoroscopy	30% (31)	22% (33)	17% (41)	23% (33)
• Intervention	25% (23)	16% (26)	0	16% (24)
Proportion of working hours spent double reading, mean (SD)
• Reported by management	27% (12)	20% (13)	8% (12)	20% (13)
• Reported by radiologists (n = 263)	27% (17)	25% (17)	8% (7)	25% (17)
Perceived resource situation^††
• Resources are sufficient	0	7	4	11
• Radiologist manpower is insufficient	8	23	6	37
• Insufficient manpower limits double reading	7	17	7	31
• Would increase double reading if resources increased	4	10	5	19

Mean score on a scale from 0 to 100%.

†

Person index: Person monitoring and feedback approach.

‡

System index: System monitoring and feedback approach.

Appropriateness index: Appropriateness of investigations approach.

Proportion of exams being double read estimated by department head.

††

Number of department heads agreeing to statements.

CT, computed tomography; FTE, full-time equivalent; MRI, magnetic resonance imaging; SD, standard deviation.

As shown in Table 3, the rate of double reading reported by management varied significantly (P = 0.01) between departments of different teaching status. University hospital departments reported the highest rates of double reading, while other teaching departments reported intermediate rates and non-teaching departments reported the lowest rates. The proportion of radiologist working hours consumed by double reading reported by management showed a similar trend (P = 0.02). The differences in rates of double reading between departments of different teaching status remained significant when correcting for size in a linear regression model (P = 0.04).

Double reading rates were significantly lower in departments reporting insufficient radiologist manpower as a limiting factor (Spearman’s rho 0.47, P < 0.01). Lower rates of double reading were significantly associated with willingness to increase double reading if resources improved (Spearman’s rho 0.46, P < 0.01). None of the university hospital departments reported sufficiency, while half of the non-teaching departments did. Other teaching departments took an intermediate position (P = 0.03). Management perception of sufficient resources was significantly associated with higher scores on system index (Spearman’s rho 0.33, P = 0.03). Management perception of sufficient radiologist manpower was significantly associated with higher scores on person index (Spearman’s rho 0.30, P < 0.05). There were no significant associations between perceived resource situation and appropriateness index scores. Responses concerning perceived resource situation are listed in Table 3.

Per modality rates of double reading were highest for magnetic resonance imaging (MRI), followed by computed tomography (CT). The rates of double reading of X-ray and fluoroscopy were intermediate, while ultrasonography and intervention were double read the least, as shown in Table 3.

Consultant and resident radiologist staffing information from the management survey and SERUS matched completely. The numbers are summarized in Table 3. There was no difference in the proportion of residents in the radiologist staff in university hospital departments (0.31) and other teaching departments (0.31).

Among the three quality indices, departments scored highest on the appropriateness index, an intermediate score on the person index, and the lowest score on the system index. There was no correlation between scores on any of the three quality indices and the rate of double reading. Nor were there any significant differences in scores on the three quality indices between departments of different teaching status. Table 2 summarizes the items in each index, while the scores are shown in Table 3.

Discussion

According to the management survey 33% of all radiology exams read by consultants in Norwegian hospitals is double read. The rate of double reading was significantly associated with the teaching status but not to the CQI and QA activities of departments assessed in our three quality indices.

Double reading rates

A rate of double reading in Norwegian hospitals of 33% is considerable and consistent with a previous study (10). Literature for international comparison is inadequate, but indicates substantial differences. One report from a university hospital in Sweden indicates that all exams are double read (4). In USA a random review of 5% of cases is required for credentialing by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) (8). An American non-hospital radiology group reported a double reading rate of 2% between 1997 and 2001 (13).

There are several explanations for the association between teaching status and higher rates of double reading. Mere size does not explain the difference fully, as the effect of teaching status remains significant even when adjusting for size. The education of resident radiologists may play a part as the non-teaching departments double read the least. However, there is no difference in the proportion of residents in the radiologist staff between university hospital departments and other teaching departments. The university hospitals are highly specialized hospitals with regional and national responsibilities. Increased complexity of investigations may necessitate double reading, and the opportunities may be more plentiful with the more frequent use of clinical radiologic conferences.

There are considerable differences in double reading rates between modalities. Dynamic exams such as ultrasonography, fluoroscopy, and interventional procedures are double read the least. This is not surprising as the ability to make a contribution of value to the assessment is dependent on being present at the time of the exam. MRI and CT exams are double read the most, probably as a combination of feasibility and necessity. The non-dynamic nature of MRI and CT is suitable for double reading.

Double reading is applied extensively in Norwegian hospitals, however, the practice is not systematic and detached from other QA initiatives. In the USA double reading is applied more selectively to a random sample of exams and the results of the review are documented in electronic systems (such as RADPEER) that allow for retrieval and comparison of performance data (7,8).

A significant effect on patient safety must be expected in order to justify assigning such substantial resources to this practice. Most reports of discordance rates concern cancer screening or reporting by residents on call, although there are some reports on the discordance rates of consultant radiologists in clinical radiology. Several previous studies have shown discordance rates of 1.3–3.0% between consultant radiologists; however, one study reports considerably higher (23%) discordance rates (4,13 –16). In the Norwegian mammography-screening program discordant interpretations have been reported in 5.3% of exams (17). Several studies report on staff radiologists reviewing resident on call interpretation of emergency CT exams with significant discordance rates ranging from 0.9% to 2.0% (18 –25). However, significantly higher rates ranging from 10.0% to 36.7% have been reported for more complex exams and exams with a high frequency of pathology (26 –28).

Lower rates of double reading were significantly associated with reporting insufficient radiologist manpower as a limitation, but not to the perceived resource situation in general. Furthermore, the university hospital departments, which most frequently reported insufficient resources to meet demands, had the highest double reading rates. This is consistent with the considerable workload of radiologists and the challenge some departments face in recruiting them, and it suggests that it is the workload and not the monetary cost of double reading that necessitates priority.

Given that double reading is conducted for quality purposes, one might assume that departments committed to this practice engage in more comprehensive QA activities in general, for example by focusing more on appropriateness of investigations. The lack of association between double reading rates and the quality indices, does not support this assumption, and suggests that double reading is not part of a systematic approach towards quality work. However, the survey did not take into account why departments refrain from particular CQI and QA activities, be it lack of confidence in their efficacy, lack of resources, or limitations posed by technical solutions.

We found that the radiologist survey data supported the management responses since there was a significant correlation between the estimates of working hours assigned to this practice in the two populations. There was also complete agreement between the staffing information acquired from management and the SERUS reporting system. We used this matching of data as validation of our findings. However, as the data in the surveys are self-reported, possible biases should be taken into account in the interpretation.

We achieved an excellent response rate of 100% in the management survey, which is high compared to a previous study (10). The radiologist survey was the first of its kind in Norway. In this survey 55% of our target population responded, which is average for a web-based survey (29). Some consultant radiologists may not be members of the Norwegian association of radiologists, and therefore may not have received an invite. In the radiologist survey non-response bias cannot be ruled out, as there was no alternate source of data allowing for comparison of the responders to the target population.

Private imaging centers account for 23% of all performed imaging exams in Norway (12). Nonetheless they were not invited to participate in the study since it has previously been shown that they practice double reading to a limited degree compared with hospital units (10). In this context it makes good sense to look at hospital radiology units and private imaging centers separately as they serve different patient populations and are organized differently.

The surveys have not been validated. However, they were not intended as instruments for standardized evaluation of CQI activities. The surveys were conducted in an exploratory manner, and there were no previously stated research hypotheses. As a result even statistically significant findings must be cautiously interpreted. This is to some degree remedied by the consistency of data acquired from different sources and of our findings with previous findings.

In conclusion, double reading is extensively applied in Norwegian hospital radiology. The rate of double reading is significantly correlated to department teaching status, but not to other methods of quality work.

Footnotes

Acknowledgements

We thank the Norwegian Association of Radiologists and the Norwegian Medical Association for their assistance in distributing the radiologist survey to their members and for volunteering data from the SERUS reporting system. Thanks to professor Jarl Jakobsen for suggesting the survey and for his enthusiasm. Thanks to Fredrik A. Dahl for help and supervision during statistical analysis.

Funding

The study received funding from the Norwegian Medical Association and the Norwegian Association of Radiologists.

References

Markus

Somers

O'Malley

. Double-contrast barium enema studies: effect of multiple reading on perception error. Radiology 1990; 175: 155–156.

Quekel

Goei

Kessels

. Detection of lung cancer on the chest radiograph: impact of previous films, clinical information, double reading, and dual reading. J Clin Epidemiol 2001; 54: 1146–1150.

Federle

Gur

. Double reading of certain examinations such as barium enemas and mammograms can increase sensitivity at the expense of specificity. Am J Roentgenol 2005; 164: 1291–1292.

Diaz

Ekberg

. The frequency of diagnostic errors in radiologic reports depends on the patient's age. Acta Radiol 2010; 51: 934–938.

Dinnes

Moss

Melia

. Effectiveness and cost-effectiveness of double reading of mammograms in breast cancer screening: findings of a systematic review. Breast 2001; 10: 455–463.

Ballard-Barbash

Klabunde

Paci

. Breast cancer screening in 21 countries: delivery of services, notification of results and outcomes ascertainment. Eur J Cancer Prev 1999; 8: 417–426.

Mahgerefteh

Kruskal

Yam

. Peer review in diagnostic radiology: current state and a vision for the future. Radiographics 2009; 29: 1221–1231.

McEnery

Suitor

Hildebrand

. Integration of radiologist peer review into clinical review workstation. J Digit Imaging 2000; 13: 101–104.

Deitch

Chan

Sunshine

. Quality assessment and improvement: what radiologists do and think. Am J Roentgenol 1994; 163: 1245–1254.

10.

Husby

Espeland

Kalyanpur

. Double reading of radiological examinations in Norway. Acta Radiol 2011; 52: 516–521.

11.

Hurlen

Østbye

Borthne

. Introducing PACS to the late majority. A longitudinal study. J Digit Imaging 2008; 23: 87–94.

12.

Almén A, Friberg EG, Widmark A, et al. Radiology in Norway anno 2008. Trends in examination frequency and collective effective dose to the population. Report No.:2010:12. Oslo: Norwegian Radiation Protection Authority; 2010. [In Norwegian].

13.

Soffa

Lewis

Sunshine

. Disagreement in interpretation: a method for the development of benchmarks for quality assurance in imaging. J Am Coll Radiol 2011; 1: 212–217.

14.

Branstetter

Barton

Morgan

. Preliminary reports in the emergency department: is a subspecialist radiologist more accurate than a radiology resident? Acad Radiol 2007; 14: 201–206.

15.

Jordan

Lightfoote

Jordan

. Quality outcomes of reinterpretation of brain CT imaging studies by subspecialty experts in neuroradiology. J Natl Med Assoc 2006; 98: 1326–1328.

16.

Wakeley

Jones

Kabala

. Audit of the value of double reading magnetic resonance imaging films. Br J Radiol 1995; 68: 358–360.

17.

Hofvind

Geller

Rosenberg

. Screening-detected breast cancers: discordant independent double reading in a population-based screening program. Radiology 2009; 253: 652–660.

18.

Carney

Kempf

DeCarvalho

. Preliminary interpretations of after-hours CT and sonography by radiology residents versus final interpretations by body imaging radiologists at a level 1 trauma center. Am J Roentgenol 2003; 181: 367–373.

19.

Cooper

Goodhartz

Nemcek

. Radiology resident interpretations of on-call imaging studies. Acad Radiol 2008; 15: 1198–1204.

20.

Erly

Berger

Krupinski

. Radiology resident evaluation of head CT scan orders in the emergency department. Am J Neuroradiol 2002; 23: 103–107.

21.

Chung

Strigel

Chew

. Overnight resident interpretation of torso CT at a level 1 trauma center. Acad Radiol 2009; 16: 1155–1160.

22.

Lal

Murray

Eldevik

. Clinical consequences of misinterpretations of neuroradiologic CT scans by on-call radiology residents. Am J Neuroradiol 2000; 21: 124–129.

23.

Roszler

McCarroll

Rashid

. Resident interpretation of emergency computed tomographic scans. Invest Radiol 1991; 26: 374–376.

24.

Wechsler

Spettell

Kurtz

. Effects of training and experience in interpretation of emergency body CT scans. Radiology 1996; 199: 717–720.

25.

Wysoki

Nassar

Koenigsberg

. Head trauma: CT scan interpretation by radiology residents versus staff radiologists. Radiology 2005; 208: 125–128.

26.

Agostini

Durieux

Milot

. Value of double reading of whole body CT in polytrauma patients. J Radiol 2008; 89: 325–330.

27.

Meyer

Nickerson

Burbank

. Discrepancy rates of on-call radiology residents' interpretations of CT angiography studies of the neck and circle of Willis. Am J Roentgenol 2009; 193: 527–532.

28.

Tieng

Grinberg

. Discrepancies in interpretation of ED body computed tomographic scans by radiology residents. Am J Emerg Med 2007; 25: 45–48.

29.

Braithwaite

Emery

De Lusignan

. Using the Internet to conduct surveys of health professionals: a valid alternative? Fam Pract 2003; 20: 545–551.