Abstract
OBJECTIVE: In 1997, the Task Force on Rhinosinusitis (TFR) set forth symptom-based diagnostic guidelines for chronic rhinosinusitis (CRS). In the present study, we examined radiologic correlates of the TFR diagnostic criteria for rhinosinusitis.
STUDY DESIGN AND SETTING: One hundred twenty-five consecutive patients undergoing computed tomography (CT) scans of the sinuses were studied at Oregon Health and Science University. Patients were evaluated prospectively with a questionnaire based on the TFR criteria, and their CT scans were graded according to the Lund-Mackay scoring system.
RESULTS: Of the 125 patients, 115 met the symptom criteria for CRS. However, 40 of 115 had negative scans (Lund-McKay score, 0) despite meeting the diagnostic criteria for rhinosinusitis. Of 115, 75 had positive scans (Lund-McKay score, >1). Of the 10 patients who had negative diagnoses for rhinosinusitis, 9 had a positive CT scan. The Kappa coefficient was −0.103 (±95% confidence interval, −0.201 to −0.004), indicating poor agreement between CRS positivity and CT positivity. The sensitivity of TFR criteria for detecting a positive scan was 89%, but the specificity was poor at only 2%.
CONCLUSION AND SIGNIFICANCE: Based on these pilot data, it appears that the specificity and predictive value of the current TFR criteria may not be adequate to serve as a diagnostic standard for rhinosinusitis. Additional validating data may provide guidance for improving the sensitivity and specificity of symptom-based diagnostic instruments for rhinosinusitis.
The development of consensus guidelines for the diagnosis and treatment of rhinosinusitis has been a significant challenge for otolaryngology and primary care providers. 1 A major breakthrough emerged in 1997 when the Task Force on Rhino-sinusitis (TFR) set forth consensus diagnostic guidelines for acute, subacute, and chronic rhino-sinusitis (CRS). 2 These guidelines, sanctioned by the American Academy of Otolaryngology, were formulated predominantly on a symptom-based algorithm of major and minor diagnostic clinical criteria. The adoption and utilization of these guidelines have had significant implications both for establishing standards of care for rhinosinusitis and for standardization of outcomes research. However, despite the usefulness and applicability of the diagnostic guidelines, rigorous validation of these diagnostic criteria has not yet been performed.
Although the TFR has advocated the use of subjective symptom-based criteria to diagnose rhinosinusitis, many physicians also recognize the diagnostic usefulness of the sinus computed tomography (CT) scan as an objective criterion. 3–7 Radiologic studies were specifically excluded from the TFR criteria, but the predictive value of nasal and sinus symptoms for radiologic evidence of disease remains an important clinical question. 8–11 We therefore sought to evaluate correlations between the TFR diagnostic criteria and radiologic findings on sinus CT scans. We undertook this study using a prospective blinded approach.
METHODS
The protocol for this study was reviewed and approved by the Institutional Review Board at Oregon Health and Science University (OHSU). Informed consent was obtained from all participating subjects.
A patient questionnaire was developed to identify the presence of sinusitis symptoms as specified by the TFR diagnostic criteria (Fig 1). The questionnaire also examined duration of symptoms, history of treatment, and presence or absence of symptoms at the time of the CT. Patients were not examined at the time of the CT scan, and therefore one objective TFR criterion, purulence on nasal examination, was not included in the analysis. All other TFR criteria, which are symptom based, were included.
Sinus symptom questionnaire (excerpt).
The questionnaire was administered to all adult patients presenting to the OHSU Department of Radiology for a screening sinus CT between February 1999 and October 1999. Known preoperative patients were excluded, as were patients with sinonasal tumors. To confirm and/or clarify patient responses, all participants were subsequently contacted by telephone, and their responses were reviewed through a telephone interview with one of the authors (S.I.).
Lund-MacKay staging worksheet (modified from Lund and Mackay 12 ).
The CT scans were obtained using a standard clinical protocol on a GE HiSpeed scanner (GE Medical Systems, Milwaukee, WI). Three-millimeter contiguous scans were obtained in the coronal plane, from anterior to the frontal sinus through the posterior sphenoid sinus, and were reconstructed in bone algorithm. Patients were scanned in a direct coronal plane in the prone, if tolerated, or supine position.
The sinus CT scans were then reviewed in a blinded fashion by a staff neuroradiologist. Each scan was scored using the Lund-MacKay (L-M) staging system, a validated scoring system recommended by the TFR for outcomes research. 12–14 The minimum possible score was 0, with a maximum of 24 (Fig 2). The specialty of the ordering physician was also noted.
For purposes of this study, a normal or “negative” scan was defined as any scan with a Lund-McKay score of 0. Any scan with a score of >0 was considered an abnormal, or “positive,” scan. Because the L-M grading scale does not specify a cutoff score for a “positive” or “negative” scan, we chose to adhere to the strictest possible definition of a negative scan. Our aim was to avoid underestimating the number of positive scans. Of note, a “positive” scan does not imply the presence of rhinosinusitis but only connotes the presence of radiologically evident mucosal abnormalities (eg, nonpathologic retention cysts may count toward a positive score). However, because of a relative paucity of objective measures for evaluating rhinosinusitis, we chose to use the CT scan as our reference standard for evaluating the TFR criteria.
Patient questionnaires were then reviewed to distinguish a subset of patients who met a diagnosis of CRS by TFR criteria (CRS+) from those who did not (CRS-). The algorithm for diagnosing rhinosinusitis by TFR criteria is presented in Table 1. Patients were also subdivided into those with positive CT scans (CT+) and those with negative scans (CT-). Kappa statistics were calculated to assess agreement between CRS positivity (by TFR criteria) and CT scan positivity. The positive predictive value (PPV) for a positive scan was also calculated for each criterion.
Task Force on Rhinosinusitis criteria for the diagnosis of shinosinusitis
From Lanza and Kennedy. 2
RESULTS
One hundred fifty-four consecutive patients met the study entry criteria and completed a questionnaire. Twenty-nine patients were excluded because they could not be reached for follow-up, were younger than 18 years, or had a diagnosis of sinonasal tumor or because the scan could not be located, leaving 125 eligible patients for analysis. The mean age of the final study group was 47.4 years. There were 68% women and 32% men, similar to previously reported gender distributions in prevalence studies. 15
Of the 125 patients, 115 met the TFR criteria for diagnosis of CRS (CRS+ = 92%); 10 patients (8%) were CRS-. No patients were diagnosed with acute or subacute rhinosinusitis. Of the 115 CRS+ patients, 40 of 115 had negative scans (L-M score, 0) despite meeting the diagnostic criteria for rhinosinusitis. Of 115, 75 had positive scans (L-M score, >1). Of the 10 CRS-patients, 9 had a positive scan. The kappa coefficient was −0.103 (±95% confidence interval, −0.201 to −0.004), indicating poor agreement between CRS positivity and CT positivity; possible kappa values range from − 1 (complete disagreement) to +1 (complete agreement), with 0 representing agreement only per chance (Table 2). The distribution of scores for CRS + patients showed a strong skew toward lower scores (Fig 3).
Categorization of cohort by TFR diagnosis and CT result
TFR/CRS refers to a positive or negative diagnosis of chronic rhinosinusitis by TFR criteria. Observed agreement between TFR criteris and CT scan = 60.8%. Chance agreement between TFR criteria and CT scan = 64.4%. Kappa coefficient (chance-corrected agreement) = −0.103.
Of the 115 CRS+ patients, 87 patients (76%) were symptomatic on the day of the scan; 28 patients (24%) were asymptomatic. One hundred eleven patients (97%) had received some form of medical therapy for sinusitis (prescription or over-the-counter) in the preceding 3 months.
Among CRS + patients, there was no significant difference in the number of symptoms observed in those patients with positive CT scans versus those with negative scans. Patients who were CRS+ and CT+ had 3.2 major symptoms and 4.1 minor symptoms; patients who were CRS+ and CT-had 3.5 major symptoms and 3.5 minor symptoms (major symptoms, P = 0.33; minor symptoms, P = 0.42)
The sensitivity of a CRS + diagnosis for detecting a positive scan was 89%, but the specificity of the CRS criteria was only 2%.
The overall PPV of a diagnosis of CRS for a positive scan was 65%. The frequency of positive scans was higher for those studies ordered by otolaryngologists compared with nonotolaryngologists. The PPV when the ordering physician was an otolaryngologist was 71%; for all other physicians, the PPV was 58%.
When the symptom criteria were assessed individually for their predictive value for a positive CT scan, the PPVs for major criteria showed a range of 0.64 to 0.75. For minor criteria, the PPVs ranged from 0.55 to 0.68. The symptom associated with the lowest PPV was halitosis (0.55), whereas the symptom associated with the highest PPV was purulent discharge (0.75). Values for the remaining symptoms are presented in Fig 4.
Distribution of CT scans by Lund-Mackay score (all CRS+ patients).
PPVs of rhinosinusitis symptoms for an abnormal CT.
Within the CRS+ group, 90 of 115 had no previous sinus surgery, and 25 of 115 had a history of previous surgery. Both subgroups were included in the analysis because the Lund-Mackay scoring system can be applied to patients with previous surgery. However, it should be noted that the nonsurgical subgroup showed significantly lower L-M scores than the surgical group (3.03 versus 6.56, P = 0.001). There was no difference in number of major or minor symptoms between the 2 subgroups (major, P = 0.98; minor, P = 0.13).
PPV of TFR criteria for radiographic evidence of disease.
For purposes of this analysis, we intentionally adhered to a stringent threshold of an L-M score >0 to define positive scans. However, under the Lund-Mackay staging system, L-M = 1 could represent a scan with a nonpathologic mucosal variant, yet the scan would be considered “positive” in our study. For example, a scan that was clear except for a single mucosal retention cyst would qualify as an L-M score of 1 and therefore be “positive”; also, a scan showing clear sinuses except for minimal mucosal thickening in a single sinus would also be considered “positive.” We hypothesize that in a practical clinical context, many clinicians would potentially interpret scans with an L-M score of 1 or 2 or 3 as clinically negative or insignificant; sampled mean L-M scores of the general population may be as high as 4.26. 16 We recalculated the PPV of a CRS+ diagnosis using L-M scores of 1, 2, 3, and 4 as cutoffs for “positive” scans and found a linear decline in the PPV as the threshold for determining positive CTs was liberalized (Fig 5).
DISCUSSION
Before 1997, the quest for an optimal diagnostic instrument for rhinosinusitis had yielded much data but little consensus. One approach advocated symptom-based diagnosis and treatment. Because the resolution of patient symptoms is one of the primary goals of treatment, proponents of this approach argued that rhinosinusitis is “best described by its associated symptoms rather than by CT.” 17 Yet, studies have demonstrated little to no correlation between preoperative or postoperative symptom severity and CT findings. 8,18 Another approach argued for the incorporation of nasal endoscopy in the diagnosis of rhinosinusitis. Indeed, nasal endoscopy has shown relatively better accuracy in predicting positive CT findings (74% sensitivity, 84% specificity). 19 However, nasal endoscopy is not a practical diagnostic requirement because it is not readily available to primary care physicians, who see and treat an estimated 87% of all cases of rhinosinusitis. Finally, a third school of thought advocated reliance on radiologic data for diagnosis and treatment. Kennedy 20 determined that the extent of disease on CT was the only factor significantly predictive of surgical prognosis.
In 1996, the TFR convened to develop consensus recommendations on the definition and diagnosis of rhinosinusitis. The TFR recommendations, published in 1997, were intended to establish a working definition for rhinosinusitis and to function as a starting point for further refinements in disease definition. In 1997, these recommendations were approved by the American Academy of Otolaryngology–Head and Neck Surgery, the American Academy of Otolaryngic Allergy, and the American Rhinologic Society. A key feature of the TFR definitions of rhinosinusitis is its emphasis on symptoms rather than on objective findings for establishing a diagnosis. The intent of using symptom-based criteria was to provide clinicians of all specialties a means of diagnosing rhinosinusitis without requiring expensive modalities such as CT or nasal endoscopy. 2
Our study attempted to validate the TFR symptom criteria against an objective indicator—the sinus CT scan. We sought only to correlate symptoms with the presence of radiologically detectable mucosal abnormalities, which may or may not reflect true rhinosinusitis. The results of our study showed that a substantial number of patients who met TFR criteria for CRS had little to no radiologic evidence of mucosal abnormalities. Thirty-five percent of CRS+ patients had completely negative scans, and an additional 10% of patients had minimally positive scans (L-M score, 1). Overall, the agreement between CRS positivity and CT positivity was poor, with a kappa coefficient of −0.103. Furthermore, the TFR criteria had poor specificity of 2%, with better sensitivity of 89%. PPVs dropped linearly as the cutoff score for a positive scan was raised from an L-M score of 1 to an L-M score of 2 or 3.
Although both symptoms and CT are meaningful elements in the work-up of rhinosinusitis, the data suggest that as indicators of morbidity, they are less interdependent than previously thought. It appears that many more patients have symptoms consistent with CRS than have radiologic findings consistent with CRS. These findings raise important questions regarding the relative value of subjective versus objective findings in the diagnosis and treatment of CRS.
By current practice standards, there appear to be inconsistencies in the valuation of subjective versus objective data in clinical decision-making for rhinosinusitis. For example, if a patient were to present with a strong symptom-based history for rhinosinusitis but with no CT scan data, most clinicians would be comfortable initiating medical therapy based on symptoms alone, without resorting to a sinus scan. Yet, if another patient had an identical symptom history but had also had a known negative CT scan, most clinicians would be reluctant to commit the patient to medical therapies, given the negative CT scan. In the former case, subjective symptoms would be sufficient to diagnose and treat rhinosinusitis, yet in the latter case, objective findings would supersede subjective symptoms in ruling out a diagnosis of rhino-sinusitis.
Similar conundrums exist in surgical decision-making. A patient with a positive CT scan but no symptoms would likely not be considered a surgical candidate. Conversely, a patient with markedly positive symptoms but a completely negative scan would not likely be a surgical candidate. In the former case, the decision to not pursue surgery would be based on subjective symptoms alone. Yet in the latter case, the decision to not pursue surgery would be based on an objective measure that would supersede even strongly positive subjective criteria.
Thus, it appears that both subjective symptom findings and objective CT findings are important factors in the diagnosis and treatment of sinusitis. One might hope that the presence of sinus symptoms would be strongly correlated with a positive CT, but our data suggest that this is not the case. One possible explanation for the low specificity of the TFR criteria is that the TFR symptom criteria are not specific for infectious rhinosinusitis and may also be positive in patients with various forms of noninfectious rhinitis. Another possibility is that symptom-positive, CT-negative patients in fact do have a form of rhinosinusitis that cannot be “scored” by radiologic criteria. In fact, the Lund-McKay system was not designed to diagnose rhinosinusitis. Although a negative CT typically ends the work-up for chronic sinusitis, the adoption of a broader definition of rhinosinusitis to include these CT-negative patients might spur one to reconsider the clinical approach to the symptomatic patient with a negative CT scan.
One of the limitations of our study was the lack of access to more extensive medical history in our series of patients. Knowledge of comorbidities in these patients (eg, allergic rhinitis, migraine) would be helpful in understanding potential etiologies of the symptoms experienced by these CT–patients. The ultimate diagnoses in this series of CT–patients are not known, but this deficiency could be remedied in future studies through longitudinal studies of larger groups of patients experiencing nasal and sinus symptoms.
The establishment of the 1997 TFR diagnostic guidelines represented an important attempt to standardize the diagnosis of rhinosinusitis. In its report, the TFR acknowledged that its guidelines were intended to evolve and to be revised as our understanding of rhinosinusitis improves. Our pilot data suggest that the specificity and predictive value of the current TFR criteria for determining radiologic disease may not be sufficient for a universal diagnostic standard. It is our hope that additional studies be undertaken to provide further validating data and to provide guidance for improving the sensitivity and specificity of the TFR criteria as a diagnostic instrument for rhinosinusitis.
The authors thank Donald C. Lanza, MD, for his valued critique and editorial review of the manuscript.