Risk of Bias in Liver Imaging Reporting and Data System Studies Using QUADAS-2

Abstract

French

Purpose: Use a tailored version of the Quality Assessment of Diagnostic Accuracy Studies tool to evaluate risk of bias and applicability across LIRADS related publications. Method: A tailored QUADAS-2 tool was created through consensus approach to assess risk of bias and applicability across 37 LI-RADS related publications. Studies were selected from 2017 to 2022 using the assistance of experienced hospital librarians to search for studies evaluating the diagnostic accuracy of CT, MRI, or contrast-enhanced ultrasound for HCC using LI-RADS through multiple different databases. QUADAS-2 assessments were performed in duplicate and independently by 2 authors with experience using the QUADAS-2 tool. Disagreements were resolved with a third expert reviewer. Consensus QUADAS-2 assessments were tabulated for each domain. Results: Using the tailored QUADAS-2 tool, 31 of the 37 included LI-RADS studies were assessed as high risk of bias, and 9 out of 37 studies demonstrated concerns for applicability. Patient selection (21 out of 37 studies) and flow/timing (24 out of 37 studies) domains demonstrated the highest risk of bias. 6 out of 37 studies in the index domain demonstrated high risk of bias. 2 out of 37 studies showed high risk of bias in the reference standard domain. Conclusion: A significant proportion of LI-RADS research is at risk of bias with concerns for applicability. Identifying risk of bias in such research is essential to recognize limitations of a study that may affect the validity of the results. Areas for improvement in LI-RADS research include reducing selection bias, avoiding inappropriate exclusions, and decreasing verification bias.

Visual Abstract

This is a visual representation of the abstract.

Keywords

LIRADS QUADAS-2 applicability risk of bias hepatocellular carcinoma

Introduction

The Liver Reporting and Data System (LI-RADS^®) standardizes the nomenclature, technique, interpretation, data collection, and reporting of imaging examinations in patients at risk for developing hepatocellular carcinoma (HCC).¹ Since its inception in 2008, diagnostic LI-RADS algorithms have undergone many updates, with the most recent version released in 2018. Extensive peer reviewed studies have been published which have evaluated various components of LI-RADS. Unfortunately, there has not been much research into the specific risks of bias for LI-RADS related research studies. Risk of bias (ROB) is the chance that the characteristics of a study design or how a study is conducted will lead to misleading results, and can occur when there are significant flaws or limitations in the design or conduct of a study which can alter the veracity of results.² Discerning bias in LI-RADS research can be challenging given the complexity of LI-RADS, variability of study methodology and variable completeness of reporting.³ Several tools are available to evaluate studies for risk of bias, including the SIGN methodology (Scottish Intercollegiate Guidelines Network), AMSTAR (A Measurement Tool to Assess systematic Reviews), and CASP (critical appraisal skills program).⁴ The Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool is appropriate for most LI-RADS research because these studies often evaluate diagnostic accuracy.⁵

The QUADAS-2 tool is used to evaluate for risk of bias and concerns regarding applicability. Four domains are used to assess risk of bias: patient selection, index test, reference standard, and flow/timing.² Signalling questions are developed to help classify the level of risk of bias in each of these 4 domains as low, unclear, or high. Concerns regarding applicability of the study are also assessed in the patient selection, index test, and reference standard domains as low, high, or unclear. The LI-RADS Individual Participant Data (IPD) group conducts systematic reviews to investigate the diagnostic performance of LI-RADS for diagnosis of HCC.⁶ As part of the group’s work, the QUADAS-2 tool has been optimized for assessing risk of bias and applicability in LI-RADS research.⁷ The QUADAS-2 tool was initially tailored for evaluation of LIRADS research using a consensus approach by a team of radiologists, data specialists, and a hepatologist.³ Signalling questions were created and utilized to assess LI-RADS research studies for potential risks of bias and concerns regarding applicability.

The purpose of this study is to present results from our application of the tailored QUADAS-2 tool to LI-RADS research, and to identify common areas of sub-optimal methodology and reporting. This work may serve as a guide for readers to critically appraise LI-RADS diagnostic accuracy studies and for researchers to optimize the design and reporting of LI-RADS diagnostic accuracy studies.

Materials and Methods

Study Design

The IRB for IPD analysis and transfer of data was approved by the Research Ethics Board at The Ottawa Hospital Research Institute (Protocol ID#: 20190664-01H). Thirty-seven studies from our LI-RADS IPD database have been assessed with the QUADAS-2 tool in published IPD-meta-analyses,^3,8 ranging from 2017 to 2022. Only relevant LI-RADS studies where the author consented to participating in the IPD data base were included. Using the assistance of experienced hospital librarians, studies evaluating the diagnostic accuracy of CT, MRI, or contrast-enhanced ultrasound (CEUS) for HCC using LI-RADS (CT/MRI v2014, v2017, or v2018; CEUS v2016 or v2017) were searched for through MEDLINE, Embase, Cochrane Central Register of Controlled Trials (CENTRAL), and Scopus databases. Studies were excluded because of incomplete or redundant data with other studies, unavailable data, patient level rather than observation level data, and issues with the data formatting^7,9-12 (Figure 1). No language or publication type restrictions were applied. Dates included 2014 to Jan 2022 of when our studies were performed based on publication date of LI-RADS 2014. Grey literature was assessed via search of abstracts presented from 2014 to Jan 2022 of when our studies were presented at the annual meetings of the Radiological Society of North America (RSNA), American Roentgen Ray Society (ARRS), European Society of Radiology (ESR), International Society for Magnetic Resonance in Medicine (ISMRM), Society of Abdominal Radiology (SAR), and European Society of Gastrointestinal and Abdominal Radiology (ESGAR).

Figure 1.

Flow diagram demonstrating study inclusion and exclusion strategy.

Development and Application of the QUADAS-2 Tool

The tailored QUADAS-2 tool used by the LI-RADS IPD team is provided as Table 1. Using a consensus-based process, 4 radiologists with 5 to 10 years of experience using the QUADAS-2 tool developed the signalling questions; co-authors were also given the opportunity to contribute to the creation of these questions. The questions were piloted and refined after several rounds before their application in each of the QUADAS-2 domains (patient selection, index test, flow and timing, and reference standard).² Each article was then analyzed systematically using the signalling questions for potential sources of bias. If an article answered one or more signalling questions per each domain as “no,” the article was designated as “high risk” for bias.^2,13 If all signalling questions for each domain were answered “yes,” risk of bias was assessed as low. When there were insufficient data to answer the signalling questions, the domain was assessed as unclear risk of bias. QUADAS-2 assessments were performed in duplicate and independently by 2 authors (JPS, CvdP) with experience using the QUADAS-2 tool. Disagreements were resolved with a third expert reviewer (MM). Consensus QUADAS-2 assessments were tabulated for each domain. Proportions of studies at low, unclear, and high risk of bias were calculated.

Table 1.

Signalling Questions by Domain for the Tailored QUADAS-2 Tool.

Patient and observation selection
Risk of bias
Was a consecutive sample of patients, random sample of patients, or all patients over a given time period enrolled?
Was a case-control design avoided?
Did the study avoid inappropriate exclusions? (eg, If a non-pathology-based reference standard was used, might it have inappropriately excluded observations?)
Was it clear when more than one observation arose from the same patient?
Could the selection of patients have introduced bias? (Low = “yes” to all, otherwise refer to phase 2)
Could the selection of observations have introduced bias? (Low = “yes” to all, otherwise refer to phase 2)
Concerns regarding applicability: Is there concern that the included patients do not match the review question? Is there concern that the included observations do not match the review question?
Index test
CT risk of bias
Describe how multiphase contrast-enhanced CT was conducted and interpreted
Were the multiphase CT results interpreted without knowledge of the results of the reference standard?
Were the multiphase CT results unlikely to be biased by findings on other imaging exams?
Was a delayed phase consistently used? (delayed between 3 and 5 min has increased sensitivity for washout)
Was the index test interpreted by more than one radiologist and were discrepancies resolved in an objective way?
Could the conduct or interpretation of the multiphase CT have introduced bias? (Low = “yes” to all, otherwise refer to phase 2)
CT concerns regarding applicability: is there concern that the multiphase CT, its conduct, or interpretation differ from the review question?
MRI risk of bias
Describe how multiphase contrast-enhanced MRI was conducted and interpreted
Were the MRI results interpreted without knowledge of the results of the reference standard?
Were the MRI results unlikely to be biased by findings on other imaging exams?
Was the index test interpreted by more than one radiologist and were discrepancies resolved in an objective way?
Could the conduct or interpretation of the MRI have introduced bias? (Low = “yes” to all, otherwise refer to phase 2)
MRI concerns regarding applicability: is there concern that the MRI, its conduct, or interpretation differ from the review question?
CEUS risk of bias
Describe how contrast-enhanced US (CEUS) was conducted and interpreted
Were the CEUS results interpreted without knowledge of the results of the reference standard?
Were the CEUS results unlikely to be biased by findings on other imaging exams?
Did CEUS technical parameters meet the minimum standard in LI-RADS?
Was the index test interpreted by more than one radiologist and were discrepancies resolved in an objective way?
Could the conduct or interpretation of the CEUS have introduced bias? (Low = “yes” to all, otherwise refer to phase 2)
CEUS concerns regarding applicability: is there concern that the CEUS, its conduct, or interpretation differ from the review question?
Reference standard
Risk of bias
Describe the reference standard and how it was conducted and interpreted
Is the reference standard likely to correctly classify the target condition at the observation level (particularly non-pathology-based reference standards and also explant reference standards—was the method of lesion matching described and likely to be robust)?
Were the reference standard results interpreted without knowledge of the results of the CT, MRI, or CEUS?
Could the reference standard, its conduct or interpretation have introduced bias? (Low = “yes” to all, otherwise refer to phase 2)
Concerns regarding applicability: is there concern that the target condition as defined by the reference standard does not match the review question?
Flow and timing
Risk of bias
Describe any patients who did not receive multiphase CT, MRI, CEUS, and/or reference standard, or who were excluded from the 2 × 2 table (refer to flow diagram)
Describe the time interval and any interventions between multiphase CT, MRI, CEUS, and reference standard
Describe any observations within included patients who were excluded from the 2 × 2 table
Was there an appropriate interval between multiphase CT, MRI, or CEUS and reference standard?
Did all patients and observations receive a reference standard?
Did patients and observations receive the same reference standard?
Were all patients and observations included in the analysis?
Is there unlikely to be a selection bias from selection of patients only with liver explantation?
Is there unlikely to be verification bias from tissue sampling of only a subset of observations?
Is there unlikely to be incorporation bias for studies including LI-RADS 5 as a reference standard?
Could the patient and observation flow have introduced bias? (Low = “yes” to all, otherwise refer to phase 2)

“Concerns for applicability” were assessed for each study and structured similarly to each bias section (patient selection, index test, reference standard) using questions which differed from the signalling questions (Table 1). If a study had one or more questions for each domain answered as “high,” the article was deemed as “concerning” for applicability. If all questions for concerns of applicably were answered “low,” the study was categorized as “low” concerns for applicability. If data were insufficient for classification, the study was categorized as “unclear” concerns for applicability. Analysis of the results was performed through descriptive measures and summarized in a standardized Excel sheet format.⁷ Consensus QUADAS-2 assessments were tabulated for each domain.

Results

The cohort of original research studies and their characteristics are summarized in Table 2. Using the QUADAS-2 tool, 31 of the 37 included studies were assessed as high risk of bias, and 6 studies were at low risk of bias (meaning all domains were deemed to be at low risk of bias).⁸ 9 out of 37 studies demonstrated concerns for applicability (Figures 2 and 3).

Table 2.

Characteristics of Included Studies.

Study details					Imaging technique				Observation data
Author	Journal	Country	Design	Prevailing risk factor	Modality	IV contrast agent type	LI-RADS version	No. of readers	No. of liver observations/no. of patients	HCC	LR-1	LR-2	LR-3	LR-4	LR-5	LR-TIV	LR-M	Ref standard
Alhasan 2019¹⁴	Abdom Radiol (NY)	CAN	RC	Cirrhosis >> HBV	CT	ECA	v2017	2	104/59	72	1	5	15	29	41	10	3	Pathology
Allen 2018¹⁵	AJR Am J Roentgenol	USA	RCCon	Cirrhosis >> HBV	MRI	HPB	v2014	3	247/127	42	0	0	79	50	118	0	0	Pathology and CCRS
An 2017¹⁶	AJR Am J Roentgenol	KR	RC	HBV>> cirrhosis	MRI	HPB	v2014	2	225/225	218	0	0	1	43	170	0	11	Pathology
Cerny 2018¹⁷	Radiology	CAN	RC	Cirrhosis >> HBV	MRI	ECA	v2014	2	275/102	113	38	52	57	53	58	15	2	Pathology and CCRS
Chen 2019¹⁸	AJR Am J Roentgenol	CN	RC	HBV > cirrhosis	MRI	ECA, HPB	v2018	2	149/149	149	0	0	0	0	149	0	0	Pathology
Chen 2018¹⁹	World J Gastroenterol	CN	RCCon	HBV with cirrhosis	CEUS	Blood pool	v2017	2	176/176	88	0	0	1	6	49	0	0	Pathology
Choi 2022²⁰	Abdom Radiol (NY)	KR	RC	HBV >> cirrhosis	MRI	Primovist	v2018	2	279/253	247	0	0	18	46	190	6	19	Pathology and Clinical
Choi 2019²¹	European Association for the Study of the Liver	KR	RC	HBV > cirrhosis	MRI	HPB	v2018	NR	372/258	273	0	0	18	154	180	4	16	Pathology and CCRS
Clarke 2021²²	Clin Radiol	UK	RC	Cirrhosis	MRI	HPB	v2018	2	105/47	70	0	0	29	38	36	2	0	Explant
Fraum 2018²³	Radiology	USA	RC	Cirrhosis >> HBV	MRI	HPB	v2014	2	212/212	132	3	14	6	28	96	20	45	Pathology
Fraum 2020^24,25,*	Eur Radiol	USA	RCCon	Cirrhosis >> HBV	CT+MRI	ECA	v2018	2	331/331	81	0	0	2	12	63	24	230	Pathology
Hu 2020²⁶	Abdom Radiol (NY)	CAN	PC	Cirrhosis >> HBV	CEUS, MRI	CEUS: Blood pool MRI: ECA, HPB	CEUS: v2017 MRI: v2018	2	CEUS: 39/35 MRI: 38/34	CEUS:27 MRI: 26	CEUS:22 MRI: NR	CEUS:1 MRI: NR	CEUS:4 MRI: NR	CEUS:1 MRI: NR	CEUS:10 MRI: NR	CEUS:0 MRI: NR	CEUS:1 MRI: NR	Pathology and CCRS
Jeon 2019²⁷	Eur Radiol	KR	RCCon	HBV >> cirrhosis	MRI	HPB	v2017	2	140/140	70	0	0	0	21	67	2	50	Pathology
Jiang 2019²⁸	Cancer Imaging	CN	PC	HBV >> cirrhosis	MRI	HPB	v2018	2	272/272	215	1	3	4	28	151	57	28	Pathology and CCRS
Joo 2017²⁹	J Magn Reson Imaging	KR	RC	HBV >> cirrhosis	MRI	CT: ECA MRI:HPB	v2014	2	140/140	106	0	0	0	21	67	2	50	Pathology
Kang 2020³⁰	Abdom Radiol (NY)	KR	PC	HBV > cirrhosis	CEUS, CT, MRI	CEUS: Blood pool CT:ECA MRI: HPB	v2017	2	CEUS: 43/43 CT:35/35 MRI: 8/8	CEUS:20 CT:16 MRI:4	CEUS:0 CT:18 MRI:4	0	CEUS:16 CT:NR MRI:NR	CEUS:16 CT:NR MRI:NR	CEUS:10 CT:0 MRI:0	0	CEUS:1 CT:0 MRI:0	Pathology and CCRS
Kang 2019³¹	Eur J Radiol Open	CN	RC	Cirrhosis >> HBV	MRI	ECA	v2014	2	19/19	15	0	0	4	2	11	1	1	Pathology and CCRS
Kierans 2019³²	J Magn Reson Imaging	USA	RC	Cirrhosis > HBV	MRI	ECA, HPB	v2017	3	144/114	82	5	8	45	25	41	10	10	Pathology and CCRS
Kim 2019³³	Radiology	KR	RC	HBV >> cirrhosis	MRI	HPB	v2018	2	203/160	186	NR	NR	NR	NR	NR	NR	NR	Pathology
Kim 2021³⁴	J Magn Reson Imaging	KR	RC	Cirrhosis > HBV	MRI	Primovist	v2018	3	113/113	0	0	0	0	24	32	13	44	Pathology
Kim 2018³⁵	Eur Radiol	KR	RC	HBV >> cirrhosis	MRI	HPB	v2014	1	202/109	186	6	NR	NR	NR	NR	NR	NR	Pathology
Kim 2019³⁶	Radiology	KR	RC	Cirrhosis > HBV	MRI	HPB	v2018	2	220/220	165	0	0	5	10	70	135	0	Pathology
Lee 2021³⁷	Hepatol Int	KR	RC	HBV >> cirrhosis	CT+MRI	Primovist	v2018	2	291/222	208	0	0	15	104	154	2	16	Pathology
Lee 2019³⁸	Eur Radiol	KR	RCCon	HBV > cirrhosis	MRI	HPB	v2017	2	99/99	66	0	NR	NR	NR	NR	NR	65	Pathology
Lewis 2019³⁹	Abdom Radiol (NY)	USA	RC	HBV ~ cirrhosis	MRI	ECA, HPB	v2017	2	65/63	36	0	0	0	1	31	27	6	Pathology
Lim 2020⁴⁰	Abdom Radiol (NY)	KR	RC	Cirrhosis >> HBV	MRI	HPB	v2018	2	65/65	23	0	0	0	0	0	0	65	Pathology
Lim 2022⁴¹	Br J Radiol	KR	RC	Cirrhosis >> HBV	CT+MRI	CT: Ultravist MRI: Primovist	v2018	2	161/112	107	0	0	15	146	0	0	0	Pathology and CCRS
Makoyeva 2020⁴²	Radiol Imaging Cancer	CAN	RC	Cirrhosis >> HBV	CEUS	Blood pool	v2016	3	196/184	139	10	1	24	8	116	8	29	Pathology and CCRS
Mulazzani 2019⁴³	European Association for the Study of the Liver	IT	PC	Cirrhosis >> HBV	CEUS	Blood pool	v2017	NR	54/34	33	6	3	4	7	25	3	1	Pathology and CCRS
Ronot 2018⁴⁴	J Hepatol	FR	PC	Cirrhosis >> HBV	CT+MRI	HPB	v2014	1	595/422	336	CT: 0 MRI: 0	CT: 61 MRI: 61	CT: 116 MRI: 133	CT: 195 MRI: 152	CT: 146 MRI: 207	CT: 0 MRI: 0	CT: 0 MRI: 0	Pathology and CCRS
Rosiak 2018⁴⁵	Biomed Res Int	PL	RC	Cirrhosis >> HBV	MRI	HPB	v2017	2	69/32	50	0	0	18	13	38	0	0	Pathology
Seo 2019⁴⁶	Eur Radiol	KR	RC	HBV ~ cirrhosis	CT	ECA	v2014	2	R1: 67/50 R2: 102/65	R1: 42 R2: 54	R1:11 R2:16	R1:1 R2:18	R1:11 R2:14	R1:16 R2:21	R1:28 R2:31	NR	R1:0 R2:2	Pathology
Song 2019⁴⁷	Eur Radiol	KR	RC	HBV ~ cirrhosis	MRI	ECA, HPB	v2014	2	154/154	154	0	0	2	64	88	0	0	Pathology and CCRS
Stocker 2020⁴⁸	Eur Radiol	CH	RC	Cirrhosis > HBV	MRI	ECA	v2018	4	71/60	28	18	11	15	6	21	0	0	Pathology and CCRS
Terzi 2017⁴⁹	European Association for the Study of the Liver	IT	RC	Cirrhosis >> HBV	CEUS	Blood pool	v2017	NR	333/NR	289	0	0	74	97	144	0	18	Pathology and CCRS
van der Pol 2021⁵⁰	AJR Am J Roentgenol	CAN	RC	HBV > cirrhosis	MRI	ECA, HPB	v2018	2	222/81	72	23	33	68	42	56	0	0	Pathology and CCRS
Zhang 2019⁵¹	Front Oncol	CN	RC	HBV ~ cirrhosis	MRI	ECA	v2018	2	82/82	82	0	0	7	7	68	0	0	Pathology

Note. The > symbol indicates the first risk factor was more represented in the cohort than the second risk factor. The ~ symbol indicates both risk factors were represented approximately equally. The >> symbols indicate that the first risk factor was substantially more represented in the cohort than the second risk factor. LI-RADS = Liver Imaging Reporting and Data System; LR = Liver Imaging Reporting and Data System; HBV = Hepatitis B Virus; HCC = Hepatocellular Carcinoma; CT/MRI = Computed Tomography/Magnetic Resonance Imaging; CAN = Canada; CH = Switzerland; CN = China; FR = France; IT = Italy; KR = Korea; USA = United States of America; UK = United Kingdom; RC = retrospective cohort; RCCon = retrospective case-control; PC = prospective cohort; ECA = Extracellular Contrast Agent; HPB = Hepatobiliary Contrast Agent; Ref = Reference; CCRS = Composite Reference Standard; NR = not recorded.^14-51

Two authors and two studies contributed data for Fraum 2020, but there was data overlap and the Fraum 2020 study included all the data from both studies. Both are cited here for completeness.

Figure 2.

Summary graph highlighting percentages and number of articles at high risk of bias by domain and concerns for applicability.

Figure 3.

Risk of bias and applicability concerns for each article used in our QUADAS-2 tool assessment.

Patient Selection Domain

The patient selection domain assesses for bias through analysis of the methods for selection of patients and liver observations included in a study. For the patient selection domain, 21 studies demonstrated high risk of bias, 12 studies demonstrated low risk of bias, and 4 studies were at unclear risk of bias. For the signalling question “was a consecutive sample of patients, random sample of patients, or all patients over a given time period enrolled?” 15 studies showed high risk for either choosing patients treated for suspected HCC, including only patients with malignancy, or excluding definitely benign/probably benign (LR-1/LR-2) observations. These choices led to study populations biased toward higher LI-RADS categories. 12 out of 37 studies were at high risk of bias for the signalling question “Did the study avoid inappropriate exclusions?” because the studies included only patients with one liver observation, or limited inclusion to only one liver observation per patient. One study was at high risk of bias for the signalling question, “Was it clear when more than one observation arose from the same patient?” Finally, for the signalling question “Was a case-control design avoided?” 5 out of 37 studies were at high risk for bias (Figure 4). Two studies were found to have concerns for applicability with the question “Is there concerns that the included patients/observations did not match the review question?”

Figure 4.

Percentage of articles by signalling question demonstrating high risk of bias in the Patient Selection Domain.

Index Test Domain

The index test domain assesses for bias through the analysis of how imaging was performed and interpreted. For the index test domain, 6 studies demonstrated high risk of bias, 27 studies demonstrated low risk of bias, and 4 studies were at unclear risk of bias. For the signalling question “Were the CT/MRI/CEUS results interpreted without knowledge of the results of the reference standard?” 3 studies utilizing MRI and one article using CT as the modality demonstrated high risk for bias. For the signalling question “Described how the (CT/CEUS/MRI) was conducted and interpreted” 2 studies did not follow the technical parameters required by LI-RADS. One study using MRI did not include in and out of phase T1-weighted imaging. The second article using CT did not have adequate delayed post contrast imaging and instead used a calculated washout.⁵² Finally, for the question “Was the index test interpreted by more than one radiologist and were discrepancies resolved in an objective way?” one study using CEUS showed high risk of bias (Figure 5). Two studies were found to have concerns for applicability with the question “Is there concern that the multiphase CT/MRI/CEUS, its conduct or interpretation differ from the review question?”

Figure 5.

Percentage of articles by signalling question demonstrating high risk of bias in the Index Test Domain.

Reference Standard Domain

Bias in the reference standard domain is determined through description of the reference standard and analysis of how it is interpreted and conducted. For the reference standard domain, 2 studies were at high risk of bias, 29 studies were at low risk of bias, and 6 studies were at unclear risk of bias. While not a distinct signalling question, many studies included herein relied on explant/pathology, which was considered a high risk of bias in the patient selection domain as it results in evaluating only high LI-RADS category observations. Finally, 2 studies demonstrated high risk of bias for the signalling question, “Were the reference standard results interpreted without knowledge of the results of the CT, MRI, or CEUS?” by interpreting reference standard results with knowledge of the imaging results or using an inappropriate reference standard (Figure 6). One study was found to have concerns for applicability with the question “Is there concern that the target condition as defined by the reference standard does not match the review question?”

Figure 6.

Percentage of articles by signalling question demonstrating high risk of bias in the Reference Standard Domain.

Flow and Timing Domain

Bias in the flow and timing domain relates to patients who did not receive the index test or reference standard, were excluded from the 2 × 2 contingency table, and evaluates the time lapse and any potential intervention between the index test and the reference standard. For the flow and timing domain, 24 out of 37 studies were at high risk of bias, 2 were at unclear risk of bias, and 11 demonstrated low risk of bias. For the signalling question “Was there an appropriate interval between multiphase CT, MRI, or CEUS and reference standard?” 2 studies were at high risk of bias. For the signalling question, “Is there unlikely to be verification bias from tissue sampling of only a subset of observations?” 14 studies demonstrated high risk of bias. Three studies were at high risk of bias based on the signalling question, “Did patients and observations receive the same reference standards?” due to differences in reference standards within studies. Four studies were at high risk of bias for the signalling question “Were all patients and observations included in the analysis?” due to not including all patients and observations. Two studies were at high risk of bias for the signalling question, “Is there unlikely to be a selection bias from selection of patients only with liver explanation?” Finally, 3 studies showed high risk of bias regarding the signalling question, “Is there unlikely to be incorporation bias for studies including LI-RADS 5 as a reference standard?” by having LI-RADS 5 observations as the reference standard (Figure 7).

Figure 7.

Percentage of articles by signalling question demonstrating high risk of bias in the Flow and Timing Domain.

Discussion

This in-depth QUADAS-2 review of 37 studies retrieved from a systematic review identified several areas of potential bias that readers and future researchers should be aware of. These results add to a previous review article and editorial discussing other aspects of LIRADS diagnostic accuracy research and IPD meta-analysis, respectively.^53,54 The QUADAS-2 tool helps evaluate for biases by specific domains, using signalling questions in a systematic manner. We identified several common areas that result in high risk of bias within each of the 4 domains, which are discussed as follows. The highest risk of biases and potential solutions are summarized in Table 3.⁵⁵

Table 3.

Top 4 Risk of Biases Based on Signalling Question and Potential Solutions.

Signalling question	Domain	Number of articles	Solutions
“Was a consecutive sample of patients, random sample of patients, or all patients over a given time period enrolled?”	Patient Selection	15/37 (40.5%)	Evaluate consecutive or random selection of patients
	Patient Selection	15/37 (40.5%)	Do not exclude patients with lower LI-RADS categories (LR1 and LR2)
“Did the study avoid inappropriate exclusions?”	Patient Selection	12/37 (32.4%)	Avoid unnecessary exclusions, for example, patients with more than one liver observation
Significant reliance on explant/pathology as the reference standard	Reference Standard	14/37 (37.8%)	Apply a composite reference standard that includes clinical and imaging standards (eg, stability on imaging for several years), in addition to pathology
“Is there unlikely to be verification bias from tissue sampling of only a subset of observations?”	Flow and Timing	14/37 (37.8%)	Using a robust reference standard not limited to pathology, such as those incorporating clinical factors and imaging findings
	Flow and Timing	14/37 (37.8%)	Use advanced statistical methods such as imputation to estimate the impact of missing reference standard data in unconfirmed patients.

In the patient selection domain, a major source of bias in this cohort of studies was the selection of only patients with a higher LI-RADS category, choosing patients treated for suspected HCC, and choosing patients with known malignancy. These study designs are at high risk of selection bias, where subjects selected for inclusion in the study are likely not representative of the target population. A recent systematic review more fully explored this source of bias, which led to higher observed rates of HCC in LR-2 and LR-3 categories.³ Additionally selecting patients with only higher LI-RADS categories while excluding LR1 and LR2 patients can also lead to spectrum bias (also known as sampling bias). This affects the diagnostic accuracy of these study in terms of sensitivity and specificity.⁵⁶

Several studies inappropriately limited inclusion to only one liver observation per patient or selecting patients with only one observation. This is a significant issue as treatment decisions are based not only the presence of HCC, but also on the size and number of HCCs. As such, analysis at the liver observation level is preferred. Finally, a case-control design was used in several studies. Case-control studies artificially set the prevalence of HCC in the study population, which directly impacts the percentage of HCC in each category, which relates to the positive and negative predictive values. Additionally, due to the retrospective nature of case-control studies, only correlation can be made between variables, not causation. Furthermore, case-control studies tend to include fewer indeterminate or intermediate-category observations, affecting the distribution of outcomes in the study population. The control aspect of these studies needs to be reflective of the general population, which may be an issue in single-institution research.⁵⁷

In the index test domain, readers were unblinded from results from the reference standard in 3 studies. This can lead to review bias, whereby knowledge of the gold standard can potentially influence the interpretation of an index test.⁵⁸ Two studies did not adhere to the LI-RADS technical guidelines. Following the LI-RADS imaging parameters for each modality is paramount in evaluating the diagnostic performance of LIRADS, so that the imaging tests assessed in research are reflective of the tests performed in practice; this also allows for consistent data collection for future studies.⁵²

The reference standard corresponds to the method used to determine the presence or absence of disease to which the index test is compared to, with the assumption that the reference standard is 100% accurate. In practice, reference standards are rarely able to achieve this, leading to bias. For the reference standard domain, biases stemmed mostly from a heavy reliance on pathology and explant as the reference standard. A recent meta-analysis assessing the impact of the reference standard in the diagnosis of HCC found that pathology-based reference standards were used 4 times more often than clinical reference standards.⁸ Additionally, observations that were confirmed to be HCC were twice as likely to have used a pathology reference standard than a clinical reference standard. This can lead to selection bias secondary to focusing on a larger number of observations diagnosed as HCC. For example, LR1-LR3 and even LR5 are typically not pathologically proven. Misclassification bias can also occur when using pathology as the reference standard, as sampling error can lead to false negative results.⁵⁹ The emphasis on pathology in designing the reference standard also adversely affects other domains of bias, namely patient selection and flow and timing. Including reference standards such as follow-up imaging and imaging with other modalities in addition to pathology-based reference standards can help these biases.⁸

It is also important to evaluate the reference standard without knowledge of the results of the index test. Several studies demonstrated high risk of bias in the reference standard domain secondary to evaluating the reference standard while knowing the LI-RADS category. This can lead to diagnostic review bias, where the results of the index test if positive may drive researchers to scrutinize the reference standard more intensely.

When determining the choice for a reference standard, it is important to note that the appropriateness of the reference standard depends on the research question being asked. In the meta-analysis setting for example, a different study research question can lead to rejection of quality research papers secondary to the reference standard. This can occur despite the reference standard being appropriate to the original paper.

The domain in which the largest number of studies demonstrated high risk of bias was with flow and timing. Many of the reviewed studies were either unclear or did not have an appropriate interval between the imaging study and the reference standard. Too long of a period between the index test (imaging study) and reference standard can lead to misclassification bias, where the disease progresses or improves during the extended interval. Ideally, the timing between performing the index test and reference test should be as short as possible to avoid this type of bias.⁵⁹

Additionally, a significant number of studies showed high risk of verification bias through either sampling only a subset of liver observations based on the findings on the index test or using different reference standards for patients. There are several subtypes of verification bias such as partial verification bias and differential verification bias. Partial verification bias occurs if only a certain number of patients or subset of liver observations receive the reference standard (such as tissue sampling), resulting in a low number of false negatives and overestimation of the sensitivity of the test. Sometimes this is unavoidable if the reference standard is invasive, as it may be unethical to apply a particular reference standard to every patient in clinical practice.⁶⁰ Differential verification bias occurs when at least 2 different reference standards are used for the study, again leading to an overestimation of sensitivity. Verification bias can be corrected for via statistical methods, and these methods should be considered when either sampling a subset of observations or when having to use more than one reference standard is unavoidable.

Several studies had risk for incorporation bias by including LR-5 classification with the same modality as the index test as a reference standard (eg, using MR LI-RADS 5 status as all or part of the reference standard when evaluating MR as the index test). Incorporation bias occurs when the index test results are part of the adjudication process, leading to a falsely elevated sensitivity and specificity. Sensitivity analyses (excluding studies at high risk of bias) can help mitigate some of the risks of using the index test as the reference standard.⁵⁸

Applicability of a research study involves looking at the extent to which conclusions from the primary study apply to the review question. As stated previously, these differ from the signalling questions used in the ROB assessment and fall under the patient and observation, index test, and reference standard domains only when developing the tailored QUADAS-2 tool. This helps address the potential mismatches between the review questions at an individual study versus a systematic review level. Several studies demonstrated concerns for applicability because the included patients or observations did not match the review question. This occurred when only patients with malignancy (both HCC or non-HCC malignancies) were included. Only including patients with malignancy limits a studies generalizability to the population it is targeting. Several studies also raised concerns because the conduct or interpretation of the CT/MRI/CEUS differed from the review question. This happened when the study did not follow the technical parameters for LI-RADS, which is a significant limiting factor when the focus of the review question is analyzing the diagnostic accuracy of LI-RADS. Finally, one study demonstrated concern that the target condition as defined by the reference standard did not match the review question. This study used a reference standard which may not detect the target condition defined in the review question (HCC), which again limits the studies applicability to the target population.⁶¹

Many of the described biases in the LI-RADS literature can be addressed by adhering to the STARD guidelines, a check list of 30 recommendations for research study design to improve evaluation of diagnostic accuracy.⁶² A problem in systematic reviews of LI-RADS has been suboptimal reporting of primary diagnostic accuracy studies. Non-reporting of this essential information leads to difficulty in identifying, critically appraising, and replicating studies for future research. While this challenge is prevalent in the imaging literature, the number of “unclear” domains indicates that there is opportunity for improvement. In our cohort of studies, 14 demonstrated unclear domains.^63,64 Currently, many of the major radiology journals endorse and require adherence to the STARD guidelines.^65,66

While the QUADAS-2 algorithm is an important starting point in determining the quality of research, there remain several limitations. First, while QUADAS-2 assesses for risk of bias, it is also based on assumptions to provide a balance between quality and practicality. Second, QUADAS-2 does not assess for data integrity or directly assess quality of statistical methods, both of which are important aspects in assessing research quality. Finally, the LIRADS IPD only included data from studies that met inclusion criteria and also needed the primary investigators of these studies to be willing to share this data. As such, the QUADAS-2 assessments are not necessarily representative of all LI-RADS research being conducted worldwide.

Conclusion

The advent of LI-RADS in the diagnosis of HCC has led to a plethora of research over the past decade. In our analysis of selected LI-RADS literature, we have demonstrated several areas in study design that are at high risk for bias. Recognizing these sources of bias is important to help readers evaluate the validity and generalizability of results from these studies to the population. Using the QUADAS-2 tool, this systematic review has provided recommendations for designing future studies to avoid these biases. Adherence to the STARD checklist is also essential when designing research studies in LI-RADS, as this can also help mitigate bias and comply with major radiology journal requirements.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: CIHR Operating Grant, Joan Sealy trust.

ORCID iDs

Haresh Naringrekar

Andreu F. Costa

Christian B. van der Pol

Matthew D. F. McInnes

References

Chernyak

Fowler

Kamaya

, et al. Liver Imaging Reporting and Data System (LI-RADS) version 2018: imaging of hepatocellular carcinoma in at-risk patients. Radiology. 2018;289(3):816-830. doi:10.1148/radiol.2018181494

Whiting

Rutjes

Westwood

, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529-536. doi:10.7326/0003-4819-155-8-201110180-00009

van der Pol

Lim

Sirlin

, et al. Accuracy of the liver imaging reporting and data system in computed tomography and magnetic resonance image analysis of hepatocellular carcinoma or overall malignancy—a systematic review. Gastroenterology. 2019;156(4):976-986. doi:10.1053/j.gastro.2018.11.020

Shea

Grimshaw

Wells

, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10. doi:10.1186/1471-2288-7-10

Whiting

Weswood

Rutjes

AWS

Reitsma

Bossuyt

PNM

Kleijnen

. Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol. 2006;6(1):9. doi:10.1186/1471-2288-6-9

LI-RADS IPD Group. OSF LI-RADS IPD group protocols. 2023. Accessed May 3, 2024. https://osf.io/tdv7j/wiki/Who%20are%20we/

Lam

Adamo

Goins

McInnes

van der Pol

Al-Ghita

. LIRADS IPD group protocols. June 1, 2023. Accessed May 3, 2024. https://osf.io/tdv7j/

van der Pol

McInnes

MDF

Salameh

J-P

, et al. Impact of reference standard on CT, MRI, and contrast-enhanced US LI-RADS diagnosis of hepatocellular carcinoma: a meta-analysis. Radiology. 2022;303(3):544-545. doi:10.1148/radiol.212340

Khatri

Pedrosa

Ananthakrishnan

, et al. Abbreviated-protocol screening MRI vs. complete-protocol diagnostic MRI for detection of hepatocellular carcinoma in patients with cirrhosis: an equivalence study using LI-RADS v2018. J Magn Reson Imaging. 2020;51(2):415-425. doi:10.1002/jmri.26835

10.

Zhang

Huang

Wei

, et al. Hepatocellular carcinoma: can LI-RADS v2017 with gadoxetic-acid enhancement magnetic resonance and diffusion-weighted imaging improve diagnostic accuracy? World J Gastroenterol. 2019;25(5):622-631. doi:10.3748/wjg.v25.i5.622

11.

Chen

Ruan

Lin

, et al. Comparison between M-score and LR-M in the reporting system of contrast-enhanced ultrasound LI-RADS. Eur Radiol. 2019;29(8):4249-4257. doi:10.1007/s00330-018-5927-8

12.

Terzi

Iavarone

Pompili

, et al. Contrast ultrasound LI-RADS LR-5 identifies hepatocellular carcinoma in cirrhosis in a multicenter restropective study of 1,006 nodules. J Hepatol. 2018;68(3):485-492. doi:10.1016/j.jhep.2017.11.007

13.

Whiting

Rutjes

AWS

Reitsma

Bossuyt

PMM

Kleijnen

. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3(1):25. doi:10.1186/1471-2288-3-25

14.

Alhasan

Cerny

Olivié

, et al. LI-RADS for CT diagnosis of hepatocellular carcinoma: performance of major and ancillary features. Abdom Radiol (NY). 2019;44(2):517-528.

15.

Allen

Jaffe

Miller

Mazurowski

Bashir

. Comparison of visualization rates of LI-RADS version 2014 major features with IV gadobenate dimeglumine or gadoxetate disodium in patients at risk for hepatocellular carcinoma. AJR Am J Roentgenol. 2018;210(6):1266-1272.

16.

Park

Chung

, et al. Curative resection of single primary hepatic malignancy: liver imaging reporting and data system category LR-M portends a worse prognosis. AJR Am J Roentgenol. 2017;209(3):576-583.

17.

Cerny

Bergeron

Billiard

, et al. LI-RADS for MR imaging diagnosis of hepatocellular carcinoma: performance of major and ancillary features. Radiology. 2018;288(1):118-128.

18.

Chen

Zhou

Kuang

, et al. Liver imaging reporting and data system category 5: MRI predictors of microvascular invasion and recurrence after hepatectomy for hepatocellular carcinoma. AJR Am J Roentgenol. 2019;213(4):821-830.

19.

Chen

Ruan

Liang

, et al. Differentiation of intrahepatic cholangiocarcinoma from hepatocellular carcinoma in high-risk patients: a predictive model using contrast-enhanced ultrasound. World J Gastroenterol. 2018;24(33):3786-3798.

20.

Choi

Kang

Kim

. Comparison of gadoxetate disodium-enhanced MRI sequences for measuring hepatic observation size and its implication of LI-RADS classification. Abdom Radiol (NY). 2022;47(3):1024-1031.

21.

Choi

Kim

Byun

, et al. Subtraction arterial images of hepatocyte-specific contrast-enhanced MRI: added value for the diagnosis of hepatocellular carcinoma in the LI-RADS v2018. Poster presented at: The International Liver Congress; April 10-14, 2019; Vienna, Austria. EASL: European Association for the Study of the Liver.

22.

Clarke

CGD

Albazaz

Smith

, et al. Comparison of LI-RADS with other non-invasive liver MRI criteria and radiological opinion for diagnosing hepatocellular carcinoma in cirrhotic livers using gadoxetic acid with histopathological explant correlation. Clin Radiol. 2021;76(5):333-341.

23.

Fraum

Tsai

Rohe

, et al. Differentiation of hepatocellular carcinoma from other hepatic malignancies in patients at risk: diagnostic performance of the liver imaging reporting and data system version 2014. Radiology. 2018;286(1):158-172.

24.

Fraum

Cannella

Ludwig

, et al. Assessment of primary liver carcinomas other than hepatocellular carcinoma (HCC) with LI-RADS v2018: comparison of the LI-RADS target population to patients without LI-RADS-defined HCC risk factors. Eur Radiol. 2020;30(2):996-1007.

25.

Ludwig

Fraum

Cannella

, et al. Hepatocellular carcinoma (HCC) versus non-HCC: accuracy and reliability of Liver Imaging Reporting and Data System v2018. Abdom Radiol (NY). 2019;44(6):2116-2132.

26.

Bhayana

Burak

Wilson

. Resolution of indeterminate MRI with CEUS in patients at high risk for hepatocellular carcinoma. Abdom Radiol (NY). 2020;45(1):123-133.

27.

Jeon

Joo

Lee

, et al. Combined hepatocellular cholangiocarcinoma: LI-RADS v2017 categorisation for differential diagnosis and prognostication on gadoxetic acid-enhanced MR imaging. Eur Radiol. 2019;29(1):373-382.

28.

Jiang

Liu

Chen

, et al. Man or machine? Prospective comparison of the version 2018 EASL, LI-RADS criteria and a radiomics model to diagnose hepatocellular carcinoma. Cancer Imaging. 2019;19(1):84.

29.

Joo

Lee

Ahn

Lee

Han

. Liver imaging reporting and data system v2014 categorization of hepatocellular carcinoma on gadoxetic acid-enhanced MRI: comparison with multiphasic multidetector computed tomography. J Magn Reson Imaging. 2017;45(3):731-740.

30.

Kang

Kim

Joo

Han

. Additional value of contrast-enhanced ultrasound (CEUS) on arterial phase non-hyperenhancement observations (≥2 cm) of CT/MRI for high-risk patients: focusing on the CT/MRI LI-RADS categories LR-3 and LR-4. Abdom Radiol (NY). 2020;45(1):55-63.

31.

Kang

Wang

. Digital subtract angiography and lipiodol deposits following embolization in cirrhotic nodules of LIRADS category ≥3. Eur J Radiol Open. 2019;6:106-112.

32.

Kierans

Makkar

Guniganti

, et al. Validation of liver imaging reporting and data system 2017 (LI-RADS) criteria for imaging diagnosis of hepatocellular carcinoma. J Magn Reson Imaging. 2019;49(7):e205-e215.

33.

Kim

Choi

Kim

Lee

Byun

. Gadoxetic acid-enhanced MRI of hepatocellular carcinoma: value of washout in transitional and hepatobiliary phases. Radiology. 2019;291(3):651-657.

34.

Kim

Choi

Kim

, et al. Combined hepatocellular-cholangiocarcinoma: magnetic resonance imaging features and prognosis according to risk factors for hepatocellular carcinoma. J Magn Reson Imaging. 2021;53(6):1803-1812.

35.

Kim

. Diagnostic accuracy of prospective application of the Liver Imaging Reporting and Data System (LI-RADS) in gadoxetate-enhanced MRI. Eur Radiol. 2018;28(5):2038-2046.

36.

Kim

Roh

. Hepatocellular carcinoma versus other hepatic malignancy in cirrhosis: performance of LI-RADS version 2018. Radiology. 2019;291(1):72-80.

37.

Lee

Choi

Byun

, et al. Combined computed tomography and magnetic resonance imaging improves diagnosis of hepatocellular carcinoma ≤3.0 cm. Hepatol Int. 2021;15(3):676-684.

38.

Lee

Kim

. How to utilize LR-M features of the LI-RADS to improve the diagnosis of combined hepatocellular-cholangiocarcinoma on gadoxetate-enhanced MRI? Eur Radiol. 2019;29(5):2408-2416.

39.

Lewis

Peti

Hectors

, et al. Volumetric quantitative histogram analysis using diffusion-weighted magnetic resonance imaging to differentiate HCC from other primary liver cancers. Abdom Radiol (NY). 2019;44(3):912-922.

40.

Lim

Kwon

Cho

. Inter-reader agreement and imaging-pathology correlation of the LI-RADS M on gadoxetic acid-enhanced magnetic resonance imaging: efforts to improve diagnostic performance. Abdom Radiol. (NY) 2020;45(8):2430-2439.

41.

Lim

Kwon

Cho

, et al. Added value of enhanced CT on LR-3 and LR-4 observation of Gd-EOB-DTPA MRI for the diagnosis of HCC: are CT and MR washout features interchangeable? Br J Radiol. 2022;95(1132):20210738.

42.

Makoyeva

Kim

Jang

Medellin

Wilson

. Use of CEUS LI-RADS for the accurate diagnosis of nodules in patients at risk for hepatocellular carcinoma: a validation study. Radiol Imaging Cancer. 2020;2(2):e190014.

43.

Mulazzani

Sansone

Giordano

, et al. Clinical validation of the role of contrast-enhanced ultrasound in the EASL guidelines for the diagnosis of hepatocellular carcinoma. Poster presented at: The International Liver Congress; April 10-19, 2019; Vienna, Austria. EASL: European Association for the Study of the Liver.

44.

Ronot

Fouque

Esvan

Lebigot

Aubé

Vilgrain

. Comparison of the accuracy of AASLD and LI-RADS criteria for the non-invasive diagnosis of HCC smaller than 3 cm. J Hepatol. 2018;68(4):715-723.

45.

Rosiak

Podgorska

Rosiak

Cieszanowski

. Comparison of LI- RADS v.2017 and ESGAR guidelines imaging criteria in HCC diagnosis using MRI with hepatobiliary contrast agents. Biomed Res Int. 2018;2018:7465126.

46.

Seo

Kim

Park

, et al. Optimal criteria for hepatocellular carcinoma diagnosis using CT in patients undergoing liver transplantation. Eur Radiol. 2019;29(2):1022-1031.

47.

Song

Choi

Hwang

Choi

. LI-RADS v2014 categorization of hepatocellular carcinoma: intraindividual comparison between gadopentetate dimeglumine-enhanced MRI and gadoxetic acid-enhanced MRI. Eur Radiol. 2019;29(1):401-410.

48.

Stocker

Becker

Barth

, et al. Does quantitative assessment of arterial phase hyperenhancement and washout improve LI-RADS v2018-based classification of liver lesions? Eur Radiol. 2020;30(5):2922-2933.

49.

Terzi

De Bonis

Leoni

, et al. CEUS LI-RADS are effective in predicting the risk hepatocellular carcinoma of liver nodules. Poster presented at: The International Liver Congress; 2017; Amsterdam, The Netherlands. EASL: European Association for the Study of the Liver.

50.

van der Pol

Dhindsa

Shergill

, et al. MRI LI-RADS version 2018: impact of and reduction in ancillary features. AJR Am J Roentgenol. 2021;216(4):935-942.

51.

Zhang

Kuang

Chen

, et al. The role of preoperative dynamic contrast-enhanced 3.0-T MR imaging in predicting early recurrence in patients with early-stage hepatocellular carcinomas after curative resection. Front Oncol. 2019;9:1336.

52.

Kambadakone

Fung

Gupta

, et al. LI-RADS technical requirements for CT, MRI, and contrast-enhanced ultrasound. Abdom Radiol (NY). 2018;43(1):56-74. doi:10.1007/s00261-017-1325-y

53.

van der Pol

Costa

Lam

Dawit

Bashir

McInnes

MDF

. Best practice for MRI diagnostic accuracy research with lessons and examples from the LI-RADS Individual Participant Data Group. J Magn Reson Imaging. 2024;60(1):21-28. doi:10.1002/jmri.29049

54.

Costa

McInnes

MDF

van der Pol

, et al. Individual participant data meta-analyses for diagnostic accuracy research: challenges and lessons learned from the LI-RADS IPD Group. Radiol Imaging Cancer. 2024;6(3):e240015. doi:10.1148/rycan.240015

55.

de Groot

JAH

Bossuyt

PMM

Reitsma

, et al. Verification problems in diagnostic accuracy studies: consequences and solutions. BMJ. 2011;343:d4770. doi:10.1136/bmj.d4770

56.

Willis

. Spectrum bias—why clinicians need to be cautious when applying diagnostic test studies. Fam Pract. 2008;25(5):390-396. doi:10.1093/fampra/cmn051

57.

Tenny

Kerndt

Hoffman

. Case Control Studies. StatPearls Publishing LLC.; 2023.

58.

Kea

Hall

Wang

. Recognising bias in studies of diagnostic tests part 2: interpreting and verifying the index test. Emerg Med J. 2019;36(8):501-505. doi:10.1136/emermed-2019-208447

59.

Pavlou

Kurtz

Song

. Diagnostic accuracy studies in radiology: how to recognize and address potential sources of bias. Radiol Res Pract. 2021;2021:5801662. doi:10.1155/2021/5801662

60.

O’Sullivan

Banerjee

Heneghan

Pluddemann

. Verification bias. BMJ Evid Based Med. 2018;23(2):54-55. doi:10.1136/bmjebm-2018-110919

61.

Reitsma

. In: Deeks

Bossuyt

Leeflang

Takwoingi

., eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023) Chapter 8. Cochrane; 2023. https://training.cochrane.org/handbook-diagnostic-test-accuracy/current

62.

Simel

Rennie

Bossuyt

. The STARD statement for reporting diagnostic accuracy studies: application to the history and physical examination. J Gen Intern Med. 2008;23(6):768-774. doi:10.1007/s11606-008-0583-3

63.

Hong

Korevaar

McGrath

, et al. Reporting of imaging diagnostic accuracy studies with focus on MRI subgroup: adherence to STARD 2015. J Magn Reson Imaging. 2018;47(2):523-544. doi:10.1002/jmri.25797

64.

Whiting

Rutjes

Dinnes

Reitsma

Bossuyt

Kleijnen

. A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol. 2005;58(1):1-12. doi:10.1016/j.jclinepi.2004.04.008

65.

Cohen

Korevaar

Altman

, et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open. 2016;6(11):e012799. doi:10.1136/bmjopen-2016-012799

66.

Bossuyt

Reitsma

Bruns

, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. Clin Chem. 2003;49(1):1-6. doi:10.1373/49.1.1