Inter- and intra-rater agreement in the assessment of the vascularity of spinal metastases using digital subtraction angiography tumor blush

Abstract

Background

Preoperative embolization is based on the preoperative digital subtraction angiography (DSA) tumor blush, and as such is considered the “gold standard” for determining tumor vascularity. However, to our knowledge reliability studies evaluating vascularity ratings of DSA tumor blush in spinal metastases have not been published previously.

Purpose

To evaluate inter- and intra-rater agreement in the assessment of the vascularity of spinal metastases using DSA tumor blush.

Material and Methods

This reliability study included 46 patients with symptomatic metastatic spinal cord compression requiring surgery. DSA data stored in the hospital picture archiving and communication system (PACS) from the participants of a randomized controlled trial were used. Inter- and intra-rater agreement on vascularity assessment using DSA tumor blush according to a three-step ordinal scale was evaluated: no hypervascularity; moderate hypervascularity; and pronounced hypervascularity. The statistical analysis was based on the linear weighted kappa’s for multiple raters that extend Cohen’s κ. Three raters and κ = 0.2 in the null hypothesis implied that the power of the study was 0.96.

Results

Inter- and intra-rater agreements were moderate in rating the vascularity of spinal metastases and the agreements were significantly higher than the κ = 0.20 in the null hypothesis (P = 0.0002 and P = 0.0001). The κ value for inter-rater agreement was 0.57 (95% confidence interval [CI], 0.41–0.72) and for intra-rater agreement 0.55 (95% CI, 0.38–0.71).

Conclusion

There is moderate inter-rater and intra-rater agreement in classifying the vascularity of spinal metastases on a three-step ordinal scale for DSA tumor blush.

Keywords

Vascularity digital subtraction angiography (DSA)reliability study inter-rater agreement spinal metastases tumor blush

Introduction

Preoperative embolization of hypervascular spinal metastases is an adjunct to surgery to reduce perioperative bleeding and the surgery time. A number of clinical studies, including one randomized trial, have demonstrated a beneficial effect of surgical treatment of symptomatic spinal metastases (1 –4): reducing neurological symptoms and pain and improving mechanical stability (1,5,6). However, perioperative bleeding can represent a major cause of operative morbidity in surgery for metastatic spinal disease (7,8). The extent of the surgical procedure and the vascularity of the metastasis are considered the major predictive factors of perioperative blood loss (9 –12). Patients with spinal metastases considered hypervascular on the basis of tumor histology or on magnetic resonance imaging (MRI) are often referred to preoperative arteriography and embolization. The positive predictive value of MRI identifying hypervascular tumors is 77–94%. However, the accuracy of excluding hypervascularity is low: 33–79% of metastases predicted to be hypovascular according to MRI findings are diagnosed hypervascular on digital subtraction angiography (DSA) (10,13,14). Consequently, the final decision on preoperative embolization is based on the preoperative DSA tumor blush, and as such is considered the “gold standard” for determining tumor vascularity (10,13,15,16). To our knowledge reliability studies evaluating vascularity ratings of DSA tumor blush in spinal metastases have not been published previously. We evaluated inter-rater and intra-rater agreement in the assessment of the vascularity of spinal metastases using DSA tumor blush.

Material and Methods

For this reliability study we used DSA data stored in the hospital picture archiving and communication system (PACS) from the participants of a randomized controlled trial (RCT) evaluating the effect of preoperative embolization of spinal metastases (17). Patients (n = 48) were included in the RCT from May 2011 to March 2013 and the study was approved by the Danish National Committee on Biomedical Research Ethics (ID number H-2-2011-024) and the Danish Data Protection Agency (ID number 2008-41-2128). Informed consent was obtained from all individual participants included in the study. Two patients did not have factual information stored and therefore 46 patients were included in to the present study.

Two experts in interventional radiology (IR) (>20 years of experience) and one resident trained in IR (<3 years’ experience) evaluated the vascularity of the symptomatic metastases at the levels planned to undergo wide surgical decompression and instrumented stabilization of the spine. The vascularity rating of the most hypervascular metastasis was recorded in patients with more than one metastasis requiring surgery. The vascularity was graded by visual evaluation of the intensity of DSA tumor blush based on a three-step ordinal scale. No hypervascularity was defined as tumor blush equal to or less than an adjacent vertebra without tumor involvement (Fig. 1). Metastases presenting with slightly more distinct tumor blush than an adjacent vertebra without tumor involvement was categorized as moderate hypervascular (Fig. 2). Pronounced hypervascularity was defined as markedly strong tumor blush (Fig. 3). Prior to evaluation of the images, all three raters were briefed about the definition of the rating scale and the reference images (Figs. 1 –3). One expert evaluated the DSA images twice with 6 weeks and interval blinded to the histopathological diagnosis. The ordering of the patients was random at each rating.

Digital subtraction angiography (DSA)

The endovascular procedures were performed according to the study protocol, using standard techniques via arterial access to one of the two common femoral arteries under local regional anesthesia. All participants underwent selective catheterization with a 5-F visceral catheter and DSA of spinal segmental arteries at the level/levels of the affected vertebra/vertebrae as well as two levels above and below. A total of 8 mL of contrast fluid (iodixanol, 270 mg iodine/mL), diluted 4:5, was injected by hand. The injection rate was modified by experience according to the caliber of the vessels supplying the metastasis. A micro catheter in a coaxial system was used when required.

Baseline characteristics

The mean age of the patients was 64 years (standard deviation [SD], 8) and 63% were men. The predominant origins of the metastases were lung and breast carcinoma (34.8% and 19.6%) and the most frequently affected spinal level was in the thoraco-lumbar region of the spine (56.5%). Additional patient characteristics are summarized in Table 1.

Table 1.

Patient characteristics.

Parameter	n = 46
Age	64 (SD 8)
Sex (male)	29 (63%)
Primary site of cancer
Lung	16 (34.8%)
Breast	9 (19.6%)
Prostate	4 (8.7%)
Renal	3 (6.5%)
Head and neck carcinoma	3 (6.5%)
Unidentified	3 (6.5%)
Colon	2 (4.3%)
Melanoma	2 (4.3%)
Rectum	1 (2.2%)
Esophagus	1 (2.2%)
Stomach	1 (2.2%)
Myelomatosis	1 (2.2%)
Spinal level
Lumbo-sacral	5 (10.9%)
Lumbar	1 (2.2%)
Thoraco-lumbar	26 (56.5%)
Thoracic	14 (30.4%)

SD, standard deviation.

Statistical analysis

The vascularity was evaluated as an ordinal variable (grade 0–2). For intra-observer agreement two readings by rater A were compared. For inter-observer agreement the first reading by rater A and the readings by rater B and C were compared. Agreement was expressed as a single index of agreement corresponding to Cohen’s linear weighted κ for the ordinal scale. Indices were interpreted according to the recommendations of Landis and Koch (18): κ < 0, less than expected by chance; 0.0 < κ < 0.2, slight; 0.2 < κ < 0.4, fair; 0.4 < κ < 0.6, moderate; 0.6 < κ < 0.8, substantial; and 0.8 < κ < 1.0, almost perfect. The statistical analysis of inter-rater agreement was based on a linear weighted kappa’s for multiple raters described in Warrens (19). 95% confidence intervals (CI) were obtained from 10,000 bootstrap samples. To rule out agreement less than fair, P values were computed for a one-sided test of the null hypothesis “κ = 0.2.” Since the distribution of the weighted κ’s under this null-hypothesis is unknown, the P values were based on a 10,000 parametric bootstrap samples from a multivariate probit-regression model with threshold parameters matching the data and with latent correlation chosen to yield an expected κ-value equal to the hypothesized value of 0.2. All statistical analyses were performed with R version 3.1.0 (Vienna, Austria version) (20).

Sample size calculation

The choice to include the three raters and the n = 46 observations in the evaluation of inter-rater agreement was based on a power analysis using computer simulations from the multivariate probit-model with threshold parameters matching the distribution obtained preliminarily for rater A and with latent correlation expected to yield a κ-value of 0.6. For the bootstrap test of the null hypothesis “κ = 0.2” the expected power of the study was found to be 0.96.

Results

Both the inter-rater and intra-rater agreements were moderate in rating the vascularity of spinal metastases by DSA tumor blush and the agreements were significantly higher than the κ = 0.20 in the null hypothesis (P = 0.0002 and P = 0.0001). The κ value for inter-rater agreement was 0.57 (95% CI, 0.41–0.72). For intra-rater agreement the κ value was 0.55 (95% CI, 0.38–0.71). The results of the inter-rater and intra-rater agreement analyses are displayed in Table 2.

Table 2.

Inter- and intra-rater agreement.

	κ^ᵚ	95% confidence interval	κ in the null hypothesis	Р
Inter-rater agreement	0.57	0.41–0.72	0.20	0.0002
Intra-rater agreement	0.55	0.38–0.71	0.20	0.0001

The inter-rater prevalence of the different tumor blush ratings is summarized in Table 3 and the prevalence in the intra-rater study in Table 4. The metastases were most frequently rated as moderate hypervascular (48–65%). Pronounced hypervascularity was least frequent (7–33%) and no hypervascularity intermediary (20–28%).

Table 3.

Inter-rater prevalence of tumor blush ratings.

	Assessment of vascularity using DSA tumor blush
	No hypervascularity	Moderate hypervascularity	Pronounced hypervascularity
Rater A*	13	30	3
Rater B†	11	25	10
Rater C‡	13	29	4
Total	37	84	17

First reading by rater A (interventional radiologist with >20 years of experience).

†

Reading by rater Rater B (interventional radiologist with >20 years of experience).

‡

Reading by rater C (resident trained in IR with <3 years of experience).

Table 4.

Intra-rater prevalence of tumor blush ratings.

	Assessment of vascularity using digital subtraction angiography (DSA) tumor blush
	No hypervascularity	Moderate hypervascularity	Pronounced hypervascularity
1st reading by rater A	13	30	3
2nd reading by rater A	9	22	15
Total	22	52	18

Rater A (interventional radiologist with >20 years’ experience).

Discussion

Evaluation of tumor blush score at DSA is considered the “gold standard” for determining tumor vascularity of spinal metastases prior to potential preoperative embolization. We demonstrated moderate inter-rater and intra-rater agreement in classifying the vascularity of spinal metastases on a three-step ordinal scale for DSA tumor blush. Ordinal scales of up to five steps are used in studies on preoperative embolization of spinal metastases/tumors rating the vascularity of the lesions by DSA tumor blush (13,14,21,22). It could be speculated that the reliability of vascularity scales decreases if the number of categories are increased. Differentiating moderate hypervascularity from pronounced appeared to have a lower degree of agreement among the three raters in the present study than determining whether hypervascularity was present or not. A two-step scale would therefore probably result in a higher degree of inter-rater agreement. However, differentiating between moderate and pronounced hypervascularity could have clinical relevance because embolization may only reduce intraoperative blood loss in metastases of pronounced hypervascularity (17). Lewandowski et al. investigated the reliability of evaluating chemoembolization endpoints by the degree of reduction of DSA tumor blush on a four-step scale and found a moderate inter-rater agreement (κ = 0.46) (16). Cardiologists have established the Thrombolysis in Myocardial Infarction (TIMI) scale to describe coronary blood flow after thrombolytic therapy (23), but standard DSA criteria to describe tumor blush in IR have not been established (16). Zhang et al. found it feasible to use quantitative DSA endpoints for chemoembolization (24). In quantitative DSA the encoded temporal information is post processed by software during procedures. The quantitative parameters are maximal enhancement and time to peak. However, quantitative measurements require exact concentrations and volume of the contrast fluid as well as a fixed injection rate and injection delay that are only possible when using injection by pump. The relatively small caliber vessels supplying spinal metastases are not suitable for injection by pump and consequently the quantitative DSA evaluation is not directly applicable in this setting.

In the present study, the DSA tumor blush was evaluated on images stored in PACS. Ideally, the visualization of the vascular tumor blush should have been evaluated in real time by all three raters during acquisition, which would probably improve the reliability of the assessment. However, this was not possible in daily routine. In real-time evaluations the radiologist has the non-stored fluoroscopy available and the projection series, and furthermore, real-time evaluation facilitates distinction of other anatomic structures overlaying the metastases, e.g. bowel. The stored data were selected at the discretion of the investigators on call and individually post-processed. The structure of the post-processing might therefore differ from the rater’s preferences. In addition to impeding the evaluation of the vascularity and thereby potentially resulting in a lower degree of inter-rater agreement, this could also have introduced bias. Another potential source of bias is that injection rates, volumes, and iodine concentrations were not strictly consistent; given that injection by pump is not applicable in spinal segmental arteries. The above-mentioned considerations most likely caused a decrease of the κ values; hence a lower degree of inter-rater agreement. Bias potentially increasing κ values was minimized by having a 6-week interval between evaluations by the same rater and by blinding from previous ratings. Furthermore, the order of the patients was different at each rating. The relatively large difference in prevalence of the different vascularity categories could predispose the raters to diagnose moderate hypervascularity more frequently than pronounced when in doubt. Furthermore, chance agreement is increased when categories are not equally frequent and thereby κ values are reduced. Cohen’s weighted kappa with quadratic weights is often used, but quadratically weighted kappa tends to increase as the number of categories increases. This was avoided in the present study by basing the statistical analysis on the linear weighted kappa’s for multiple raters that extend Cohen’s κ (19).

The accuracy of DSA tumor blush vascularity assessment has not been explored; however, it would probably be infeasible to explore due to lack of a proper endpoint. Histological samples from these surgeries are rarely suitable for a definite diagnosis of vascularity and intraoperative blood loss is often biased by for example varying invasiveness of procedures and embolization status. An accurate non-invasive preoperative solution to evaluate the vascularity of spinal metastases in order to select patients for preoperative embolization would be ideal. A pilot study by Mazura et al. addressed this topic and reported the efficacy of measuring vascularity of spinal metastases prior to preoperative embolization in ten patients using dynamic contrast-enhanced MRI perfusion. This MRI technique was significantly correlated with DSA evaluations (15). Further investigation in larger scale is necessary to determine the role of this MRI technique in patient selection for preoperative embolization.

In conclusion, vascularity of spinal metastases is important in selecting patients for preoperative embolization. We have demonstrated a satisfactory moderate inter-rater and intra-rater agreement in classifying the vascularity of spinal metastases on a three-step ordinal scale for tumor blush during contrast medium injection in DSA.

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Benny Dahl receives research support from Medtronic, Globus Medical, and Lundbeck Foundation; Lars Lönn is employed by Mentice and is a consultant for Orzone.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Patchell

Tibbs

Regine

. Direct decompressive surgical resection in the treatment of spinal cord compression caused by metastatic cancer: a randomised trial. Lancet 2005; 366: 643–648.

Falicov

Fisher

Sparkes

. Impact of surgical intervention on quality of life in patients with spinal metastases. Spine 2006; 31: 2849–2856.

Kim

Losina

Bono

. Clinical outcome of metastatic spinal cord compression treated with surgical excision +/− radiation versus radiation therapy alone: a systematic review of literature. Spine 2012; 37: 78–84.

Tancioni

Navarria

Mancosu

. Surgery followed by radiotherapy for the treatment of metastatic epidural spinal cord compression from breast cancer. Spine 2011; 36: 1352–1359.

Ibrahim

Crockard

Antonietti

. Does spinal surgery improve the quality of life for those with extradural (spinal) osseous metastases? An international multicenter prospective observational study of 223 patients. Invited submission from the Joint Section Meeting on Disorders of the Spine and Peripheral Nerves, March 2007. J Neurosurg Spine 2008; 8: 271–278.

Quan

Vital

Aurouer

. Surgery improves pain, function and quality of life in patients with spinal metastases: a prospective study on 118 patients. Eur Spine J 2011; 20: 1970–1978.

Chen

Tai

Nayak

. Blood loss in spinal tumour surgery and surgery for metastatic spinal disease: a meta-analysis. Bone Joint J 2013; 95-B: 683–688.

Clausen

Lonn

Morgen

. Perioperative blood transfusion does not decrease survival after surgical treatment of spinal metastases. Eur Spine J 2014; 23: 1791–1796.

Kobayashi

Ozkan

Tam

. Preoperative embolization of spinal tumors: variables affecting intraoperative blood loss after embolization. Acta Radiol 2012; 53: 935–942.

10.

Prabhu

Bilsky

Jambhekar

. Results of preoperative embolization for metastatic spinal neoplasms. J Neurosurg 2003; 98: 156–164.

11.

Robial

Charles

Bogorin

. Is preoperative embolization a prerequisite for spinal metastases surgical management?. Orthop Traumatol Surg Res 2012; 98: 536–542.

12.

Schmidt

Rupp-Heim

Dammann

. Surgical therapy of vertebral metastases. Are there predictive parameters for intraoperative excessive blood loss despite preoperative embolization?. Tumori 2011; 97: 66–73.

13.

Thiex

Harris

Sides

. The role of preoperative transarterial embolization in spinal tumors. A large single-center experience. Spine J 2013; 13: 141–149.

14.

Bode

Radcliff

Vaccaro

. MRI characterization of vascular spinal tumors. J Spinal Disord Tech 2015; 28: 590–594.

15.

Mazura

Karimi

Pauliah

. Dynamic contrast-enhanced magnetic resonance perfusion compared with digital subtraction angiography for the evaluation of extradural spinal metastases: a pilot study. Spine 2014; 39: 950–954.

16.

Lewandowski

Wang

Gehl

. A comparison of chemoembolization endpoints using angiographic versus transcatheter intraarterial perfusion/MR imaging monitoring. J Vasc Interv Radiol 2007; 18: 1249–1257.

17.

Clausen

Dahl

Frevert

. Preoperative embolization in surgical treatment of spinal metastases: single-blind, randomized controlled clinical trial of efficacy in decreasing intraoperative blood loss. J Vasc Interv Radiol 2015; 26: 402–412.

18.

Landis

Koch

. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–174.

19.

Warrens

. Equivalences of weighted kappas for multiple raters. Statistical Methodology 2012; 9: 407–422.

20.

R Core Team. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing, 2014.

21.

Nair

Gobin

Leng

. Preoperative embolization of hypervascular thoracic, lumbar, and sacral spinal column tumors: technique and outcomes from a single center. Interv Neuroradiol 2013; 19: 377–385.

22.

Kato

Hozumi

Takaki

. Optimal schedule of preoperative embolization for spinal metastasis surgery. Spine 2013; 38: 1964–1969.

23.

The Thrombolysis in Myocardial Infarction (TIMI) trial. Phase I findings. TIMI Study Group. N Engl J Med 1985; 312: 932–936.

24.

Zhang

Zhuang

Beilner

. Objective assessment of transcatheter arterial chemoembolization angiographic endpoints: preliminary study of quantitative digital subtraction angiography. J Vasc Interv Radiol 2013; 24: 667–671.