Inter-observer and intra-observer reliability of the myopic traction maculopathy staging system

Abstract

Aim:

The aim of this study is to evaluate the inter-observer and intra-observer agreement of the myopic traction maculopathy (MTM) staging system (MSS).

Methods:

The agreement test for MSS was based on the evaluation of 104 optical coherence tomography (OCT) scans from 104 myopic eyes. According to the MSS, six observers were asked to identify, in each image, one among four retinal patterns and one among three foveal patterns of MTM, the presence of an outer lamellar macular hole (OLMH) and an epiretinal membrane (ERM). Each observer repeated the agreement test after a 60 days interval.

Results:

Inter-observer reliability: the agreement of the test for the retina pattern was substantial (0.724), for the fovea pattern was 0.821, for the OLMH was 0.656, and for the ERM was 0.463. When all the criteria are included in the validation test the agreement was 0.657. Regarding the weighted statistics (Gwet’s AC2, 95% CI), the validation test was statistically significant both when the variables were considered one by one, with an excellent agreement, respectively for the retina pattern (0.955) and the fovea pattern (0.963) and when all the variables were included in the tests (0.930). Intra-observer repeatability: all observers rerun the test after 2 months with a statistically significant percentage of confirmation of the previous test.

Conclusion:

The MSS offers in one Table information on diagnosis, natural history, function, prognosis, and management of MTM. The MSS is user-friendly and highly reproducible.

Keywords

Myopic traction maculopathy interobserver reliability intraobserver reliability optical coherence tomography staging system statistical analysis

Introduction

Myopic traction maculopathy (MTM) is one of the vision-threatening consequences that may develop from the progressive deformation of the posterior scleral eyewall in highly myopic eyes.¹

The term MTM has included, in literature, the following clinical pictures: foveoschisis/maculoschisis/retinoschisis (FS/MS/RS), retinal/foveal detachment (RD/FD), lamellar macular hole (LMH), and full-thickness macular hole (FTMH), with (MHRD) or without RD.¹ In order to have a more comprehensive knowledge of all the types of MTM, the pathogenesis and the natural history of the disease, we studied a large cohort of highly myopic patients. We reviewed a sequence of OCT scans, per each eye, taken at different times in life, collecting data on more than 10 years of natural history of the disease. We noted, on one hand, different possible evolutions of MTM in the same eye and, on the other hand, a statistically significant correlation between the age of the patients and specific types of MTM. These two observations led us to regroup the various clinical pictures of MTM, not in a classification of types, but in a system of stages. The MTM Staging System (MSS) was recently published.² For convenience, we report the MSS Table in Figure 1. Stage 1 is the inner maculoschisis (I-MS) or inner-outer maculoschisis (IO-MS); Stage 2 is a predominantly outer maculoschisis (O-MS); Stage 3 is a combination of maculoschisis and foveal detachment (MS-MD); Stage 4 is macular detachment without schisis (MD). Each of these four stages, which describe the evolution of MTM in a direction perpendicular to the macula, can be associated with an intact fovea (stage a), or with an evolution of MTM in a direction tangential to the macula that leads to an inner LMH (stage b), or to a FTMH (stage c).

Figure 1.

Myopic traction maculopathy (MTM) staging system (MSS).

The present study was aimed to validate the MSS and to explain the main differences of MSS compared to other published classifications of MTM, as well as to highlight the main benefits of MSS Table in real life clinical practice.

Material and methods

Participants and study design

A retrospective review of the charts of patients, affected by high myopia, was made and the available tomographic images were collected from the database of Heidelberg spectral domain optical coherence tomography (SD-OCT) at the Ophthalmology Department of Sant’Anna institute, Brescia, Italy, and from the swept-source OCT (SS-OCT) of Eyecare Clinic, Brescia, Italy.

One hundred and four macular OCT scans from 104 myopic eyes (78 right and 26 left) of 104 patients (86 females and 18 males) were collected. Cases of eyes with high myopia without MTM were also included as controls: one myopic eye with normal fovea, one affected by LMH, three by LMH with ERM, one by ERM, one by FTMH.

All OCT images were randomly numbered. Personal data, as name and birth date, were removed. Six observers, with high-level expertise on pathological myopia, were recruited (A.F., B.P., G.B., P.M., S.J.P., R.F.). Observers were asked to evaluate the OCT images and to assign an MTM stage, according to the MSS Table, and to define “0” the absence of MTM. Each observer made the evaluation individually and repeated the test for a second time after 60 days. For the second test, the OCT images had been given a different number and order of presentation. The six observers were masked to the results of the other observers. Examination time was not restricted. This research study adhered to the tenets of the Declaration of Helsinki. The retrospective review of patient records was approved by the Ethics Committee at our institution.

Statistical analysis

Interobserver and intraobserver reliability was calculated. Absolute agreement and Gwet’s AC1 (unweighted) and AC2 (weighted) statistics, with the confidence intervals, were reported.³ Agreement was evaluated for the polytomous variables retina and fovea, individually and together. Further evaluation included two more dichotomous variables, the outer lamellar macular holes (OLMH) and the epiretinal membrane (ERM). Quadratic weights (Fleiss-Cohen form)³ were applied to evaluate the variables “retina,” “fovea,” “OLMH,” and “ERM” and defined respectively “weight retina” (w_r), “weight fovea (w_f), weight OLMH (w_OLMH), and weight ERM (w_ERM). The w_r were assigned the following possible values: w_r = 1 when same grading was assigned by both observers; w_r = 0.9375 when the difference between observer’s gradings was of one grade; w_r = 0.7500 when the difference was of two grades; w_r = 0.4375 when the difference was of three grades; and w_r = 0 when the difference was of four grades. The w_f were assigned the following possible values: w_f = 1 when same grading was assigned by both observers; w_f = 0.8889 when the difference between observer’s gradings was of one grade; w_f = 0.5556 when the difference was of two grades; w_r = 0 when the difference was of three grades. The w_OLMH and w_ERM = 1 when both observers agreed in recognizing the presence, w_OLMH and w_ERM = 0 when they disagreed. A combination of all criteria (w = w_r × w_f × w_OLMH × w_ERM), where factors assume the values previously defined. Interpretation of outcomes was based on guidelines outlined by Landis and Koch representing poor (<0.00), slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00) agreement. SAS 9.4 statistical software (SAS Institute, Cary, NC, USA) was used for the analysis.⁴

Results

Inter-observer reliability

Table 1 shows the results of the test for the interobserver validation, carried out by the six observers. Applying the unweighted statistics, absolute agreement and Gwet’s AC1 (95% CI) statistically significant were achieved both for the polytomous and dichotomous variables tested individually. The agreement of the test for the retina variable was substantial (0.724), for the fovea variable was almost perfect (0.821), for the OLMH was substantial (0.656), and for the ERM was good (0.463). When all the criteria were included in the validation test, the percentage of absolute agreement and Gwet’s AC1 decreased, but the agreement was substantial (0.657).

Table 1.

Interobserver reliability.

Variable	Unweighted statistics		Weighted statistics
Variable	Absolute agreement (%)	Gwet’s AC₁ (95% CI)	Absolute agreement	Gwet’s AC₂ (95% CI)
Retina	47.1	0.724 (0.628–0.821)		0.955 (0.937–0.974)
0		0.967 (0.932–1.000)
1		0.894 (0.827–0.960)
2		0.825 (0.742–0.908)
3		0.863 (0.787–0.940)
4		0.952 (0.893–1.000)
Fovea	62.5	0.821 (0.700–0.941)		0.963 (0.937–0.990)
a		0.848 (0.733–0.962)
b		0.833 (0.714–0.952)
c		0.982 (0.962–1.000)
OLMH	56.7	0.656 (0.476–0.835)		–
ERM	40.4	0.463 (0.212–0.684)		–
All variables	32.7	0.657 (0.540–0.774)		0.930 (0.898–0.962)

OLMH: outer lamellar macular hole; ERM: epiretinal membrane.

Applying the weighted statistics (Gwet’s AC2, 95% CI), the validation test was statistically significant both when the individual variables were considered with a excellent agreement, respectively for the retina variable (0.955) and the fovea variable (0.963) and when all the variables were included in the tests (0.930).

Intra-observer repeatability

Table 2 reports the results of the intraobserver repeatability. All observers rerun the test after 2 months with a statistically significant confirmation of the previous test.

Table 2.

Intraobserver reliability.

Variables	Observer	Unweighted statistics		Weighted statistics
Variables	Observer	Absolute agreement (%)	Gwet’s AC₁ (95% CI)	Absolute agreement (%)	Gwet’s AC₂ (95% CI)
Retina	A.F.	100.0	1.000 (1.000–1.000)	100.0	1.000 (1.000–1.000)
	B.P.	100.0	1.000 (1.000–1.000)	100.0	1.000 (1.000–1.000)
	G.B.	99.0	0.988 (0.965–1.000)	99.9	0.997 (0.991–1.000)
	M.P.	96.1	0.953 (0.908–0.998)	99.8	0.993 (0.986–0.999)
	R.F.	91.4	0.896 (0.831–0.962)	99.1	0.976 (0.956–0.996)
	S.J.P.	96.1	0.953 (0.909–0.998)	99.8	0.989 (0.979–1.000)
	Total	97.1	0.965 (0.949–0.981)	99.8	0.993 (0.989–0.997)
Fovea	A.F.	100.0	1.000 (1.000–1.000)	100.0	1.000 (1.000–1.000)
	G.B.	99.0	0.988 (0.964–1.000)	99.9	0.995 (0.986–1.000)
	B.P.	100.0	1.000 (1.000–1.000)	100.0	1.000 (1.000–1.000)
	M.P.	94.2	0.930 (0.875–0.985)	99.4	0.982 (0.966–0.998)
	R.F.	99.0	0.988 (0.966–1.000)	99.9	0.998 (0.993–1.000)
	S.J.P.	93.3	0.917 (0.857–0.976)	99.2	0.978 (0.960–0.995)
	Total	97.6	0.970 (0.956–0.985)	99.7	0.994 (0.991–0.997)
OLMH	A.F.	100.0	1.000 (1.000–1.000)	–	–
	G.B.	99.0	0.982 (0.948–1.000)	–	–
	B.P.	100.0	1.000 (1.000–1.000)	–	–
	M.P.	96.1	0.935 (0.871–0.999)	–	–
	R.F.	95.2	0.925 (0.858–0.991)	–	–
	S.J.P.	99.0	0.981 (0.945–1.000)	–	–
	Total	98.2	0.968 (0.950–0.987)	–	–
ERM	A.F.	100.0	1.000 (1.000–1.000)	–	–
	G.B.	97.1	0.946 (0.885–1.000)	–	–
	B.P.	100.0	1.000 (1.000–1.000)	–	–
	M.P.	93.3	0.888 (0.805–0.971)	–	–
	R.F.	95.2	0.908 (0.829–0.987)	–	–
	S.J.P.	96.1	0.936 (0.872–0.999)	–	–
	Total	97.0	0.942 (0.916–0.968)	–	–
All variables	A.F.	100.0	1.000 (1.000–1.000)	100.0	1.000 (1.000–1.000)
	B.P.	100.0	1.000 (1.000–1.000)	100.0	1.000 (1.000–1.000)
	G.B.	98.1	0.980 (0.951–1.000)	99.8	0.996 (0.991–1.000)
	M.P.	90.4	0.898 (0.838–0.958)	99.1	0.981 (0.969–0.993)
	R.F.	90.4	0.898 (0.838–0.958)	99.0	0.979 (0.964–0.995)
	S.J.P.	89.4	0.888 (0.825–0.950)	99.0	0.979 (0.966–0.991)
	Total	94.7	0.944 (0.925–0.963)	99.8	0.989 (0.984–0.993)

OLMH: outer lamellar macular hole; ERM: epiretinal membrane.

Discussion

The MSS has been designed to offer in one Table five main types of data, on eyes with MTM: nomenclature and diagnosis, evolution and pathogenesis, function, prognosis.

First, nomenclature and diagnosis of the various patterns of MTM. The key features that identify MTM are the patterns in the retina, from stage 1 to 4, and the patterns in the fovea, from stage a to c. The epiretinal proliferations, identified as a + sign, and the outer lamellar macular hole (OLMH), identified as an O are only associated findings that can be observed also in other diseases (Figure 1).

Second, pathogenesis and natural history. The MSS Table is designed to describe the evolution of the disease, from one stage to the next. The four patterns in the retinal layers are placed in the rows, along the vertical axis of table, to visually emphasize the tractional forces elongating the eye, perpendicularly to the retinal plane, from stage 1 to stage 2–4. The three patterns of the fovea are placed in the columns, along the horizontal axis of the MSS table, to visually emphasize the tractional forces tangential to the retinal plane, from stage a–c. Each stage may evolve along the vertical, horizontal, or oblique axis of the MSS Table, according to the prevailing direction of the tractional force, in each single eye. It has been reported that spontaneous resolution or improvement of MTM might occur in a minority of cases. It means that the stage of MTM might reverse without treatment for changes in the tractional forces initially causing MTM itself. The Authors also reported that, cases that spontaneously improved, they did so just temporarily and after years the MTM restarted to progress.^5–9 In any case, the MTM table will help to identify with more precision the possible directions of spontaneous improvements, and if some stages are more prone to change than others.

Third, visual function. The authors reported the average best corrected visual acuity (BCVA), per each stage. These data could be used to know what is the potential functional result that can be obtain by reverting the stage with treatment. As well, when comparing the BCVA of eyes in real life to the average BCVA in the MSS Table, the clinician knows what to expect. Any negative deviation from that data should be interpreted as an alert to exclude possible other associated pathologies.

Forth, prognosis. The Table is enriched by data on the time taken to evolve from one stage to the next. Knowing the severity of the MTM stage and predicting the type and timing of their natural progression might help the ophthalmologist to make more rational choices on follow-up.

In literature there have been attempts of classification of MTM.

Several classifications have been focused on foveoschisis (FS). Shimada et al. proposed five categories of outer FS based on location and extension of it, ranging from no apparent FS (S0), extrafoveal FS (S1), only foveal FS (S2), foveal but not involving the entire macula FS (S3), FS with complete macular involvement (S4).¹⁰

Fujiimoto et al. and Ceklic et al. classified the FS on the basis of the location of the splitting of the retina into inner, outer, inner, and outer FS.^11,12

These classifications only partially describe MTM caused by anteroposterior traction, perpendicular to the retinal plane, but not the patterns caused by tangential traction.

Ruiz Medrano et al. published a classification not just of MTM but of myopic maculopathy (MM), addressing, as a whole, the Atrophic, Tractional and Neovascular aspects of MM. MTM is the Tractional side of the ATN Classification with T0 indicating no traction; T1 inner or outer FS; T2 inner + outer FS; T3 indicating FD; T4 a FTMH; T5 −MH + RD.¹³ This classification is more complete then the previous ones and has the merit to offer a wider view of the highly myopic eye and all the associated complications. However, the description of MTM is not complete. None of the above classifications offers information on natural history.

Shimada et al. described different stages of the FS evolving to a FD, through the formation of an outer LMH, and hypotesized a progressive pathological process.¹⁰ While this type of progression is true, it is only one of the possible evolutions of MTM.

The MSS arises from the need to increase and, at the same time, simplify the knowledge and understanding of the disease, by collecting in one single table all the types of MTM but also all the possible evolutions with the addition of information on function, prognosis, and, in the future, guidelines for treatment. The accurate definition of MSS stages is based on direct observation of OCT images. The MSS allows to code, through OCT, even cases that clearly show the transition from one stage to the next, that could not be described with the rigid patterns of a classification in types, once more confirming the dynamic nature of the Staging System. A staging system must be clear and easy to check. The daily real-life usage of the MSS is strictly linked to the ease of application and understanding. Although the number of main stages is 12, the inter-observer reliability test and the intraobserver repeatability test resulted significantly positive, meaning that with a minimum training, the stages are easily identifiable. The weighted statistical analysis confirmed the reliability with an even greater agreement. Although the evaluation of interobserver reliability was significantly positive, it can be noted that among the retina and foveal patterns there are some differences in results. The more extreme patterns of the MSS have a higher agreement than the intermediate ones. The cases affected by inner outer macular schisis and normal foveal profile (1a) and the more advanced cases with macular detachment and macular hole (4c) are the cases that are most easily evaluated in the same way by observers, obtaining in fact the highest agreement value, as they are characterized by lesions that leave no doubts on the interpretation. In contrast, cases with lesions that fall into the intermediate stages, predominantly outer macular schisis, macular schisis detachment, lamellar macular hole, obtain a lower, albeit significantly positive, agreement value. The authors believe that this difference is due to the high variability of the retinal and foveal changes during the evolution of MTM and to the fact that a staging system is by definition dynamic and might show cases that are in clear transition from one stage to the next and could be defined in different stages. Cases that fall into the intermediate stages may present characteristics just at the limit with the previous or next stage, leading the observers to different judgments. The weighted evaluation shows that this disagreement is minimal, highlighting significantly positive agreement values. Figure 2 shows examples of cases which are in transition between stages.

Figure 2.

(a) The optical coherence tomography (OCT) shows the presence of inner and outer schisis. Since the outer schisis is significantly pronounced, it could be classified as 2a but it is Stage 1a for the coexistence of inner and outer schisis, or, more likely, a case of transition from stage 1a to 2a, (b) the OCT shows a case of outer schisis and very thin foveal thickness. In the magnified picture the outer plexiform layer is not completely intact. The lamellar macular hole (LMH) is not easily identifiable and could be interpreted as Stage 2a. This is a case of transition from a stage 2a to a stage 2b, (c) the OCT shows a case of predominantly outer schisis and very thin foveal thickness (white arrow) covered by an epiretinal roof which could be an epiretinal membrane (ERM) or posterior vitreous cortex (black arrow). The observer could interpret this case as stage 2a+, if mistaking the epiretinal roof as intact fovea. The correct stage is 2b+, (d) the OCT shows a case of inner and outer schisis associated to an Inner-LMH (white arrow) and an Outer-LMH (black arrow). This is a case of transition that could evolve from Stage 1bO+ to Stage 2bO+ or 1c+ or 2c+.

Although Cohen’s k is the most used statistical analysis test in the literature, it allows to compare the evaluation on dichotomous scales, that is for criteria that have two choices: yes or no. While this statistical analysis was appropriate to evaluate the presence of absence of OLMH and ERM variables, it is not appropriate for the retina and the fovea variables that require the choice of more than two alternatives (5-level retina: 0, 1, 2, 3, 4, fovea 3: a, b, c). Fleiss created an extension of the Cohen K measure (Fleiss K) to evaluate the polycotomic scales. Both analysis software, Cohen and Fleiss, allow the comparison between two observers, but not between multiple observers. Many authors have tried to create extensions, one of which is the Gwet’s statistic (unweighted AC1 and weighted AC2) that was used in this study. This analysis can be used for both dichotomous and polycotomic scales and compares more than two observers. Furthermore, Cohen’s K test is not reliable when the data are not homogeneously distributed among the alternative choices made by the two observers. In our test we introduced a retinal pattern “0,” as control, in order to evaluate the skills of the observer to discriminate the absence of MTM, even if the MSS Table does not consider the stage 0. The frequency of images with retina pattern 0 in the test was much lower than the frequency of images of other stages, so even just one discordance between two operators on a “0” case could have a significant weight on the analysis with a poor final result but not corresponding to the real agreement between the observers. Gwet’s weighs the frequency of the various choices, overcoming this limit.^14,15

Conclusion

In summary, the proposed MSS Table offers an accurate and complete description of MTM, is highly reproducible and can help ophthalmologists to share information on the disease.

Study limitation

The major limitation of the study is related to the fact that MSS was only tested on a limited number of observers (n. 6) and on a limited number of analyzed cases (n. 104). It is necessary to highlight the complexity of the statistical processing of the data despite the small number of observers: the interobserver reliability of the six tested observers resulted in the evaluation of 3600 possible combinations of answers, of which 3540 are incorrect and 60 are correct.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rino Frisina

Michele Palmieri

Barbara Parolini

References

Panozzo

Mercanti

Optical coherence tomography findings in myopic traction maculopathy. Arch Ophthalmol 2004; 122(10): 1455–1460.

Parolini

Palmieri

Finzi

, et al. The new myopic traction maculopathy staging system. EurJ Ophthalmol 2020. DOI: 10.1177/1120672120930590.

Gwet

Handbook of inter-rater reliability: how to measure the level of agreement between two or multiple raters. Gaithersburg, MD: Stataxis Publishing Company, 2001.

Landis

Koch

GG.

The measurement of observer agreement for categorical data. Biometrics 1977; 33(1): 159–174.

Shimada

Tanaka

Tokoro

, et al. Natural course of myopic traction maculopathy and factors associated with progression or resolution. Am J Ophthalmol 2013; 156(5): 948–957.e1

Polito

Lanzetta

Del Borrello

, et al. Spontaneous resolution of a shallow detachment of the macula in a highly myopic eye. Am J Ophthalmol 2003; 135(4): 546–547.

Lai

T-T

T-C

Yang

C-M.

Spontaneous resolution of foveal detachment in traction maculopathy in high myopia unrelated to posterior vitreous detachment. BMC Ophthalmol 2016; 16: 18.

Rino

Elena

Ivan

, et al. Lamellar macular hole in high myopic eyes with posterior staphyloma: morphological and functional characteristics. Graefes Arch Clin Exp Ophthalmol 2016; 254(11): 2141–2150.

Frisina

Zampedri

Marchesoni

, et al. Erratum to: Lamellar macular hole in high myopic eyes with posterior staphyloma: morphological and functional characteristics. Graefes Arch Clin Exp Ophthalmol 2016; 254(11): 2289.

10.

Shimada

Ohno-Matsui

Yoshida

, et al. Progression from macular retinoschisis to retinal detachment in highly myopic eyes is associated with outer lamellar hole formation. Br J Ophthalmol 2008; 92(6): 762–764.

11.

Fujimoto

Hangai

Suda

, et al. Features associatedwith foveal retinal detachment in myopic macular retinoschisis. Am J Ophthalmol 2010; 150(6): 863–870.e1.

12.

Ceklic

Munk

Wolf-Schnurrbusch

, et al. Visual acuity outcomes of ranibizumab treatment in pathologic myopic eyes with macular retinoschisis and choroidal neovascularization. Retina 2017; 37(4): 687–693.

13.

Ruiz-Medrano

Montero

Flores-Moreno

, et al. Myopic maculopathy: current status and proposal for a new classification and grading system (ATN). Prog Retin Eye Res 2019; 69: 80–115.

14.

Fleiss

Jacob Cohen

The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas 1973; 33(3): 613–619.

15.

Gwet

Kappa statistics is not satisfactory for assessing the extent of agreement between raters. Stat Methods Inter Rater Reliab Assess 2002; 1(6): 1–6.