Accuracy and reliability of teledermatoscopy with images taken by general practitioners during everyday practice

Abstract

We assessed the accuracy and reliability of teledermatoscopy with images taken by a general practitioner (GP) compared to face-to-face dermatological examination. GPs selected patients for teledermatoscopy and took both macro and dermatoscopic photographs. Patients were then referred to the local dermatologist for face-to-face examination. Accuracy and inter-observer reliability were calculated for the diagnosis and management plan. Image quality was rated by two observers on a three-point scale. A total of 108 teledermatoscopy consultations sent by 13 GPs were assessed by four dermatologists. Agreement was 0.61 (kappa) on diagnostic group and 0.23 on management plan. The inter-observer reliability was 0.65 on diagnostic group and 0.36 on management plan. The image quality was reported as bad in 36% of cases, reasonable in 28% and good in 36%. Agreement for cases with good quality images was 0.66 on diagnostic group and 0.42 on management plan. Teledermatoscopy in general practice had overall a lower accuracy and reliability than face-to-face consultation. In cases where a good quality image was reported, the accuracy increased, which emphasises that teledermatoscopy is highly dependent on a good quality images.

Introduction

The prevalence of melanoma continues to increase in Europe and The Netherlands.^1–3 Since the therapeutic options for metastatic melanoma are poor, methods for early diagnosis, which will have a positive effect on the prognosis, are needed. Dermatoscopy is a technique often used by dermatologists, and increasingly in primary care, for early diagnosis of melanoma and other types of clinically suspicious skin lesions.^4,5

Dermatoscopy is a non-invasive examination method. A dermatoscope is a lens with 10x magnification combined with either immersion fluid or light-emitting diode (LED) lighting with polarizing filters for glare reduction of the corneal layer of the skin allowing a clear examination of the entire epidermis and the superficial papillary dermis. The use of dermatoscopy by trained dermatoscopists increases diagnostic accuracy for pigmented skin lesions, especially for melanoma.^6,7 Several review articles have shown teledermatology as diagnostically accurate and reliable for non-pigmented skin conditions.^8–10 A recent study showed that teledermatology provided by general practitioners (GPs) contributes to more efficient and cheaper healthcare.¹¹

Teledermatoscopy has been used since 1999.¹² Since then several papers have been published on the diagnostic accuracy and reliability of teledermatoscopy for pigmented and non-pigmented skin lesions. Some studies have reported good accuracy of teledermatoscopy, similar to the accuracy found in face-to-face dermoscopic examination (Table 1). The accuracy of teledermatoscopy is mostly dependent on the observer’s (usually the dermatologist) level of experience.¹³ Studies reported moderate to good concordance between observers on diagnostic reliability (Table 2).

Table 1.

Diagnostic accuracy of dermoscopy and teledermatoscopy.

Study	Patients	Lesions	Gold standard	Face-to-face (no of observers)	Teledermatoscopy (no of observers)
Piccolo 1999¹²	66	66	Histopathology	92% (2)	86% (1)
Braun 2000²⁵	51	55	Histopathology	κ 0.74 (1)	κ 0.74 (1)
Piccolo 2000¹³	40	43	Histopathology	κ 0.74 (1)	κ 0.35-0.87 (11)
Coras 2003²⁹	–	100	Histopathology	91% (1)	89% (1)
Ferrara 2004³⁰	12	12	Histopathology	–	83% (1)
Moreno-Ramirez 2006³¹	61	–	Histopathology	–	κ 0.94 (1)
Massone 2007* ³²	18	18	Face-to face	–	89% and 94% (2)
Fabbrocini 2008³³	–	44	Histopathology	κ 0.70 (1)	κ 0.45 (1)
Ishioka 2009²⁷	–	64	Histopathology	κ 0.73 (1)	κ 0.66 (1)
Warshaw 2009²⁰	542	542	Histopathology	81% (1)	67% (1)
Tan 2010¹⁹	200	491	Face-to face	–	κ 0.95 (2)
Kroemer 2011*²⁸	88	113	Histopathology	–	κ 0.84 (1)

mobile teledermatoscopy.

Table 2.

Diagnostic inter-observer reliability of teledermatoscopy.

Study	Patients	Lesions	Observers	Inter-observer reliability, kappa
Argenziano 2003¹⁷	128	128	40	0.47–0.63
Piccolo 2004²⁶	73	77	11	0.49–0.88
Di Stefani 2007²¹	18	465	2	0.58
Tan 2010¹⁹	200	491	2	0.92
Tan 2010¹⁸	206	979	5	0.44–0.93

Teledermatoscopy enables a diagnostic tool for clinically suspicious skin lesions to be used in the general practitioner’s practice. A recent study by Lim et al. reported on the first reimbursed teledermatoscopy triage service for skin lesions.¹⁴ Instead of a live visit, patients could be referred to a commercially run virtual lesion clinic. The results showed that 88% of referrals to the dermatologist could be prevented and that access to a screening clinic was two-thirds faster for the teledermatoscopy patient group.

However, most studies on accuracy and reliability have taken place in a laboratory setting with a clinical photographer or a highly skilled dermoscopist with experience in taking dermatoscopic images. The aim of the present study was to assess the accuracy and reliability of teledermatoscopy with images taken by GPs applied during regular practice.

Methods

A digital camera (Cybershot DSC-W560, Sony, Japan) was used for all macro images of the lesions (camera setting: no flash, auto-focus, resolution 2048x1536 pixels, JPEG compression). The dermatoscope (3Gen DermLite Pro II HR, 3Gen, USA) was used with an adapter for the digital camera. A store-and-forward teledermatology consultation system (KSYOS TeleMedical Centre, The Netherlands) was used for the teleconsultations.¹¹ This system is integrated with all GP information systems in the Netherlands.

All participating GPs received a training course lasting one hour in using the teledermatology system and the camera and how to apply the dermatoscope in order to obtain a high quality image. During each teleconsultation the following information was recorded: patient identification information, medication, patient history (patient and family history of skin cancer), location of the lesion, recent changes in the lesion, sun exposure in hours during the last 3 months, estimated number of moles and skin type (Fitzpatrick scale), images and the questions posted to the dermatologist. Patient history and questions were free text. It was recommended that four images of the lesion (1 overview, 1 macro, 2 dermatoscopic) were included.

Figure 1.

A teledermotoscopy consultation.

The dermatologists were required to answer a teledermatoscopy consultation within 2 working days. All patients who had a teledermatoscopy consultation were also referred to the local dermatologist (who was not necessarily the teledermatologist) for face-to-face examination. A histopathological examination was not mandatory, but depended on the decision of the treating dermatologist during the face-to-face consultation. As this decision was a standard care process independent of teledermatoscopy and extra costs for the patient were associated with histopathology, it could not be made mandatory. For those patients who did receive histopathological examination the final diagnosis was collected. The time between the teledermatoscopy consultation and face-to-face consultation was not more than 2 weeks.

Ethics permission for the study was not required. All patients were seen face-to-face by a dermatologist for the delivery of usual care.

Inclusion criteria

All dermatologists who participated in the study had a minimum of 5 years experience of dermatoscopy and 3 years experience of teledermatology. All GPs who participated had a minimum of three years experience using teledermatology, but no experience with dermatoscopy prior to the study. The participating GPs selected all consecutive patients who presented with a (pigmented) skin lesion and who were, in their opinion, suitable for a teledermatoscopy consultation. Urgent cases were excluded. Patients gave informed consent for the teledermatology consultation and its use for research purposes.

Statistics

Both accuracy and reliability were calculated using Cohen’s kappa statistic.¹⁵ The kappa values were interpreted as: 0-0.20 as slight agreement, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, 0.81-1.0 as almost perfect agreement.¹⁶ Power analyses for an intraclass kappa test used the following values: 0.05 significance level, 2-sided test, 0.70 proportion successes, alternative agreement 0.80, null hypothesis agreement 0.40, power at 80% resulted in a sample size of 107 cases.

Accuracy

All teledermatoscopy consultations were assessed by four independent dermatologists, one of whom reported on the initial teleconsultation and three at a later time. Each provided a diagnosis blinded to the diagnosis of the other dermatologists and the three latter dermatologists also provided a management plan. The management plan consisted of four options: no treatment, excision, cryotherapy or topical medication. All diagnoses (face-to-face, histopathology and teledermatology) were assigned to an aggregated diagnosis group, see Table 3 (see XXXX, online archive) based on consensus decision. Diagnostic accuracy was calculated based on the outcome of the diagnosis group for all four teledermatoscopy assessments compared to the face-to-face assessment. Where histology was available it was compared with the four teledermatoscopy assessments and the diagnostic accuracy of the face-to-face assessment. Management accuracy was calculated between the teledermatoscopy assessments provided by the latter three dermatologists and the face-to-face assessment.

Inter-observer reliability

Inter-observer reliability (agreement between observers) was calculated between 4 dermatologists on the outcome of diagnosis group and between 3 dermatologists on management plan.

Image quality

The two most experienced dermatologists (>20 years of practice) of the four who assessed the teleconsultations independently rated the quality of the images provided in each teledermatoscopy consultation on a 3-point scale (bad, reasonable, good). If ratings differed for a case, the lowest quality rating was used.

Results

Between February 2010 and May 2011 a total of 108 teledermatoscopy consultations were performed by 13 GPs and reported by four dermatologists. The teleconsultations concerned 105 patients (3 patients had two lesions) with a median age of 47 years (range 6–84) and consisted of 55% females (n = 59). The number of excluded patients due to urgency is not known. Four images were supplied in 83 teledermatoscopy consultations, three, two and one image(s) in 17, 7 and 1 teledermatoscopy consultation, respectively.

Of the 108 lesions diagnosed through teleconsultation, 76 were seen face-to-face (70%). Table 3 shows the number of cases per face-to-face diagnosis. The face-to-face diagnosis was not available for the other 32 lesions for various reasons: the patient did not go to the dermatologist (n = 25), patient moved away (n = 6), GP did the excision (n = 1). A histopathological diagnosis was made for 36 lesions (33%), of which 35 also had a known face-to-face diagnosis.

Accuracy

The agreement between teledermatoscopy diagnosis and histopathology diagnosis was κ = 0.41-0.63. The agreement between the teledermatoscopy diagnosis and the face-to-face diagnosis was κ = 0.55-0.73. The agreement between face-to-face diagnosis and histopathology diagnosis was κ = 0.90. The agreement between teledermatoscopy management plan and face-to-face management was κ = 0.19-0.29. Histopathology diagnosed 7 cases as skin cancer (2 melanoma, 5 non-melanoma skin cancers); concordance of management plan for these 7 patients was 100%. All kappa values are shown in Table 4 (see XXXX, online archive).

Inter-observer reliability

Agreement for the 4 observers on the diagnostic group was κ = 0.56-0.78. Agreement for 3 observers on the management plan was κ = 0.31-0.38. The kappa values are shown in Table 4.

Image quality

The image quality was reported as bad in 36% of cases, reasonable in 28% and good in 36%. Accuracy for the teleconsultations with a good image quality rating between teledermatoscopy diagnosis and histopathology diagnosis was κ = 0.53-1.0. Agreement between the teledermatoscopy diagnosis and the face-to-face diagnosis was κ = 0.53-0.77. Agreement between face-to-face diagnosis and histopathology diagnosis was 1.00. Agreement between the teledermatoscopy management plan and the face-to-face management plan was κ = 0.34-0.47, see Table 4.

Reliability for teleconsultations with rated good image quality on the diagnostic group was κ = 0.48-0.79 and on the management plan κ = 0.33-0.57, see Table 4. Kappa scores for bad and reasonable image quality are also shown in Table 4.

Discussion

The aim of the study was to assess whether teledermatoscopy is equally accurate and reliable as face-to-face consultation when images are taken by a GP during everyday practice. The results showed a moderate diagnostic accuracy, a moderate diagnostic reliability, fair management accuracy and management reliability. However diagnostic and especially management accuracy was higher in cases with good quality photographs.

The diagnostic accuracy was lower than studies in which dermatologists or highly trained nurses provided the dermatoscopic images (Table 1). Management plan accuracy in other studies was higher than in the present study. A limitation on the accuracy estimation was the incomplete face-to-face diagnoses due to patients not visiting the dermatologist after the physical referral. For 70% (n = 76) of the cases a face-to-face diagnosis was available, and histopathology diagnosis was available in 33% (n = 36). There is a risk of bias (type 1 error), as the most probable reason that patients did not attend for face-to-face assessment was because their complaints had ceased or because they were adequately reassured (the telemedicine diagnosis for the 32 cases with no face-to-face assessment were melanocytic naevus (n = 19), seborrhoeic keratosis (n = 11), 1 lentigo and 1 mongolian spot). Face-to-face assessment of these more simple cases might have improved the overall accuracy. Few other studies that have been based on more than 100 cases can be compared with our results, as the images in those studies were all gathered in either a laboratory setting (by the researchers) or by a dermatoscopic nurse.^17–20 Although no definitive conclusions on the outcome of accuracy could be made, our findings suggest that obtaining high accuracy from dermatoscopy applied in general practice is more difficult than in other settings.

The diagnostic reliability was similar to the upper end of reported reliability outcomes in teledermatoscopy studies (Table 2). Management plan reliability has been reported in one other study as moderate (κ 0.58)²¹ and is thus better than in our own study. However, results on reliability (and maybe even accuracy when compared to face-to-face) may not entirely be ascribed to teledermatoscopy, as face-to-face diagnosis and management of skin lesions can differ among dermatologists as a result of different definitions of terms, lack of consensus guidelines, diagnostic drift and different perspectives on treatment. In contrast with teledermatology and teledermatoscopy, the literature on clinical inter-rater reliability in dermatology is scarce. The results on the reliability of clinical examination are similar to our own.^22–24

Seven cases in the present study were high-risk patients (diagnosed with skin cancer). In these cases diagnostic and management accuracy was 100%. Other studies also show accurate diagnosis and management of high risk patients,^{12,13,21,25,26} but there are also examples of false negatives.^20,27

Image quality was rated as good in only about one-third of the cases. This shows clearly that taking a good quality dermatoscopic image was difficult for the GPs participating in the study. Several reasons were reported:

Lack of time. GP practices were generally busy and overloaded, thus taking the time to take proper dermatoscopic images could have been difficult

Lack of skills. GPs received training in the use of the equipment, but since suitable patients for inclusion did not present themselves every day, skills were lost over time

Equipment problems. GPs reported failing/empty batteries in the camera and the dermatoscope, probably because of long intervals between uses. In addition, difficulties with attaching and detaching the adapter and focussing the camera with the adapter attached were reported.

Our study did not make a distinction between the image quality of macro clinical images and dermatoscopic images. In future research this distinction should be made as the clinical image can provide at least equivalent information for the diagnosis.²⁸

In cases where a good quality image was reported, the accuracy increased. This is similar to other studies (Table 1). This emphasises that teledermatoscopy is highly dependent on a good quality images.

Conclusions

The present study is one of the first to report on both diagnostic accuracy and reliability of teledermatoscopy applied in an everyday general practice setting. When teledermatoscopy is used in general practice under the conditions described in this study, it has mediocre image quality leading to an overall lower accuracy and reliability compared to face-to-face examination. However, high accuracy and reliability can be achieved under the right circumstances. Staff must be properly trained in acquiring high quality dermatoscopic images.

References

de Vries

Bray

Coebergh

Parkin

. Changing epidemiology of malignant cutaneous melanoma in Europe 1953-1997: rising trends in incidence and mortality but recent stabilizations in western Europe and decreases in Scandinavia. Int J Cancer 2003; 107: 119–26.

de Vries

Coebergh

. Cutaneous malignant melanoma in Europe. Eur J Cancer 2004; 40: 2355–66.

Hollestein

van den Akker

Nijsten

Karim-Kos

Coebergh

de Vries

. Trends of cutaneous melanoma in The Netherlands: increasing incidence rates among all Breslow thickness categories and rising mortality rates since 1989. Ann Oncol 2012; 23: 524–530.

Argenziano

Albertini

Castagnetti

. Early diagnosis of melanoma: what is the impact of dermoscopy? Dermatol Ther 2012; 25: 403–9.

Menzies

Emery

Staples

. Impact of dermoscopy and short-term sequential digital dermoscopy imaging for the management of pigmented lesions in primary care: a sequential intervention trial. Br J Dermatol 2009; 161: 1270–7.

Argenziano

Soyer

. Dermoscopy of pigmented skin lesions – a valuable tool for early diagnosis of melanoma. Lancet Oncol 2001; 2: 443–9.

Kittler

Pehamberger

Wolff

Binder

. Diagnostic accuracy of dermoscopy. Lancet Oncol 2002; 3: 159–65.

Levin

Warshaw

. Teledermatology: a review of reliability and accuracy of diagnosis and management. Dermatol Clin 2009; 27: 163–76.

Massone

Wurm

Hofmann-Wellenhof

Soyer

. Teledermatology: an update. Semin Cutan Med Surg 2008; 27: 101–5.

10.

Warshaw

Hillman

Greer

. Teledermatology for diagnosis and management of skin conditions: a systematic review. J Am Acad Dermatol 2011; 64: 759–72.

11.

van der Heijden

de Keizer

Bos

Spuls

Witkamp

. Teledermatology applied following patient selection by general practitioners in daily practice improves efficiency and quality of care at lower cost. Br J Dermatol 2011; 165: 1058–65.

12.

Piccolo

Smolle

Wolf

. Face-to-face diagnosis vs telediagnosis of pigmented skin tumors: a teledermoscopic study. Arch Dermatol 1999; 135: 1467–71.

13.

Piccolo

Smolle

Argenziano

. Teledermoscopy – results of a multicentre study on 43 pigmented skin lesions. J Telemed Telecare 2000; 6: 132–7.

14.

Lim

Oakley

Rademaker

. Better, sooner, more convenient: a successful teledermoscopy service. Australas J Dermatol 2012; 53: 22–5.

15.

Viera

Garrett

. Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37: 360–3.

16.

Koch

Landis

Freeman

Jr Lehnen

. A general methodology for the analysis of experiments with repeated measurement of categorical data. Biometrics 1977; 33: 133–58.

17.

Argenziano

Soyer

Chimenti

. Dermoscopy of pigmented skin lesions: results of a consensus meeting via the Internet. J Am Acad Dermatol 2003; 48: 679–93.

18.

Tan

Oakley

Soyer

. Interobserver variability of teledermoscopy: an international study. Br J Dermatol 2010; 163: 1276–81.

19.

Tan

Yung

Jameson

Oakley

Rademaker

. Successful triage of patients referred to a skin lesion clinic using teledermoscopy (IMAGE IT trial). Br J Dermatol 2010; 162: 803–11.

20.

Warshaw

Lederle

Grill

. Accuracy of teledermatology for pigmented neoplasms. J Am Acad Dermatol 2009; 61: 753–65.

21.

Di Stefani

Zalaudek

Argenziano

Chimenti

Soyer

. Feasibility of a two-step teledermatologic approach for the management of patients with multiple pigmented skin lesions. Dermatol Surg 2007; 33: 686–92.

22.

Barnhill

Roush

Ernstoff

Kirkwood

. Interclinician agreement on the recognition of selected gross morphologic features of pigmented lesions. Studies of melanocytic nevi V. J Am Acad Dermatol 1992; 26: 185–90.

23.

Carli

De Giorgi

Naldi

Dosi

. Reliability and inter-observer agreement of dermoscopic diagnosis of melanoma and melanocytic naevi. Dermoscopy Panel. Eur J Cancer Prev 1998; 7: 397–402.

24.

Fabbrocini

Balato

Rescigno

Mariano

Scalvenzi

Brunetti

. Telediagnosis and face-to-face diagnosis reliability for melanocytic and non-melanocytic ‘pink' lesions. J Eur Acad Dermatol Venereol 2008; 22: 229–34.

25.

Braun

Meier

Pelloni

. Teledermatoscopy in Switzerland: a preliminary evaluation. J Am Acad Dermatol 2000; 42: 770–5.

26.

Piccolo

Soyer

Chimenti

. Diagnosis and categorization of acral melanocytic lesions using teledermoscopy. J Telemed Telecare 2004; 10: 346–50.

27.

Ishioka

Tenório

Lopes

. A comparative study of teledermatoscopy and face-to-face examination of pigmented skin lesions. J Telemed Telecare 2009; 15: 221–5.

28.

Kroemer

Frühauf

Campbell

. Mobile teledermatology for skin tumour screening: diagnostic accuracy of clinical and dermoscopic image tele-evaluation using cellular phones. Br J Dermatol 2011; 164: 973–9.

29.

Coras

Glaessl

Kinateder

. Teledermatoscopy in daily routine–results of the first 100 cases. Curr Probl Dermatol 2003; 32: 207–12.

30.

Ferrara

Argenziano

Cerroni

. A pilot study of a combined dermoscopic-pathological approach to the telediagnosis of melanocytic skin neoplasms. J Telemed Telecare 2004; 10: 34–8.

31.

Moreno-Ramirez

Ferrandiz

Galdeano

Camacho

. Teledermatoscopy as a triage system for pigmented lesions: a pilot study. Clin Exp Dermatol 2006; 31: 13–8.

32.

Massone

Hofmann-Wellenhof

Ahlgrimm-Siess

Gabler

Ebner

Soyer

. Melanoma screening with cellular phones. PLoS One 2007; 2: e483–e483.

33.

Fabbrocini

Balato

Rescigno

Mariano

Scalvenzi

Brunetti

. Telediagnosis and face-to-face diagnosis reliability for melanocytic and non-melanocytic ‘pink' lesions. J Eur Acad Dermatol Venereol 2008; 22: 229–34.