Validation of a Self-Administered Acne Severity Index in a Convenience Sample of Adults

Introduction

Acne vulgaris is the most common skin condition in adolescents and young adults, affecting around 85% of people at some point between the ages of 11 and 30.^1-3 Acne can be debilitating, impacting individuals’ physical and psychological well-being. ⁴ Patients’ subjective ratings of their acne may vary highly from individual to individual, and patients’ subjective perceptions of their acne severity may not correlate well with objective clinical acne grading systems. This may be due to experiential differences and psychological, cultural, and social factors. Misalignment with clinical grading can impact acne management, as self-perception/self-esteem, stigma, and cultural beauty standards may lead to a discordance between physician and patient treatment desires and satisfaction.

A Self-Administered Acne Severity Index (SAASI) may facilitate a more accurate assessment of acne severity for survey research. A formal self-assessment measure could help patients track their condition between office visits and provide researchers a means for assessing acne severity in large scale survey studies. ⁵ Survey research can enhance accessibility, reduce interviewer bias, and enable broader data collection. Although acne severity scales exist, they are usually for provider, not patient, use. ⁶ The Global Acne Grading System and Investigator Global Assessment of Acne are examples of such scales for provider use. ⁷ A patient-oriented scale was created utilizing a visual facial pictorial scale similar to the Wong-Baker pain scale; however, it was intended only for children and adolescents. ⁶ This study assessed the validity of a novel SAASI in adults in survey research and situations where clinician assessment is not feasible.

Methods

Following approval from the Wake Forest University Institutional Review Board (IRB00068817), 590 participants were recruited through Amazon Mechanical Turk (MTurk), an online crowdsourcing platform providing access to a wide demographic and socioeconomic range. An informed consent sheet summarizing the survey, risks of harm, protections, and voluntariness was presented, and subject consent was assumed by proceeding to participate in the study. The survey was designed to assess participants’ ability to rate 8 partial face “gold standard” acne images subjectively and then with the SAASI. These images were obtained from DermNetNZ.org to provide a broader range of skin types, and each image’s severity—clear, mild, moderate, severe—was determined by the consensus of 3 board-certified dermatologists (Supplemental Appendix 1).

For each image, without any training or orientation, participants were asked to rate the severity of the acne as either clear, mild, moderate, or severe. Next, participants were instructed to rate the severity of the same 8 gold standard acne images using the SAASI, which consisted of 8 different reference acne images obtained from the Atrium Health Wake Forest Baptist Dermatology clinic of varying severity for use in rating acne severity, the photo-assisted ratings, and 2 multiple-choice options to assess the quantity and quality of acne lesions for each photograph (Supplemental Appendix 2). The image orders for the subjective and photo-assisted/SAASI surveys were determined using a random number generator. Participants then completed basic sociodemographic questions (Supplemental Appendix 3). Counterintuitive attention check questions were placed immediately after the subjective assessment and after the SAASI to ensure participants were paying attention and thoughtfully completing the survey. Responses from 371 English-speaking participants aged 18 years and older who completed the rating aspects of the survey and correctly answered both attention check questions were included for analysis (Figure 1). Exclusion criteria involved incomplete survey SAASI responses and those who did not answer the attention check questions correctly, as incomplete surveys and improper attention check answers would confound the data.

OPEN IN VIEWER

Figure 1.

Study design.

Subjective, photo-assisted, and SAASI ratings were compared with dermatologist ratings using agreement analysis and quantified rating metrics—with clear to severe ratings quantified as 0 to 3, respectively. SAASI ratings were analyzed using the reference photo-assisted ratings alone, categorized as “photo-assisted,” and as the reference photo-assisted ratings combined with descriptive ratings, categorized as “SAASI.” The SAASI rating was calculated by adding up the individual scoring components and converting the total to a qualitative rating (clear, mild, moderate, severe)—with the photo-assisted rating accounting for half of the score and the 2 descriptive questions accounting for a quarter of the score each (Supplemental Appendix 4). This was carried out to measure extent and local severity, building upon existing approaches (eg, Psoriasis Area and Severity Index or PASI).

Mean participant responses were calculated for each photograph individually and overall, across pooled images, for the subjective, photo-assisted, and SAASI ratings. Logistic regression analysis was performed to evaluate the effects of the classifier used by the rater, the skin tone of the image rated, and the severity of acne on rater accuracy. The accuracy of the raters was determined by comparing their ratings to the gold standard, which was the rating given by a consensus of board-certified dermatologists.

A secondary analysis was conducted to obtain 95% confidence intervals for the mean and standard deviation of rater disagreement for each classifier applied overall, across pooled images, and to each image individually. Precision was measured by the standard deviation of the ordinal ratings with more precise classifiers having lower standard deviations. Disagreement was defined as previously stated with lower disagreement indicating higher agreement with the dermatologist rating. These confidence intervals were obtained via nonparametric percentile bootstrap, as the ordinal variable disagreement was unlikely to have a latent normal distribution.

Statistical analyses were conducted using R Foundation for Statistical Computing version 4.1.1 (2021-08-10). A P-value of less than .05 was considered statistically significant. Values were rounded to 2 decimal places.

Results

Of the 590 participants, 371 individuals met study inclusion criteria. The majority of the subjects were ages 18 to 30 (42.4%); 50.3% identified as female (Table S1). Of the overall 2,968 comparisons (371 participants × 8 images each), 2,173 [73.2%, 95% confidence interval (CI) 71%-75%] participant subjective ratings agreed with the gold standard dermatologists’ rating; 1,991 photo-assisted ratings (67.1%, 95% CI 65%-69%) and 2,081 SAASI ratings (70.1%, 95% CI 68%-72%) agreed with the dermatologists’ rating.

Logistic regression analysis indicates the subjective classification (P < .01), followed by the SAASI (P = .01) and photo-assisted (P < .01), is most accurate overall, across pooled images. The statistically significant Wald tests indicate that the differences in accuracy between the classifiers are unlikely to be due to chance. The odds of a rater accurately classifying an image using the SAASI are 14% lower than when subjectively rating the images, and the odds of a rater accurately classifying an image photo-assisted are 25% lower than when subjectively rating the images (Table S2a,b).

When the classifiers are applied to individual images, the subjective classification again is the most accurate, followed by the SAASI, and photo-assisted classifiers. Images of light skin are rated more accurately than images of dark skin, and images of severe acne are rated most accurately, followed by clear, mild, and moderate acne (Figure 2). The odds of the raters correctly identifying the acne severity of light-skinned images are 53% greater than the odds for a dark-skinned image, controlling for the other factors (OR: 1.53, P < .01).

OPEN IN VIEWER

Figure 2.

Estimated fixed effects with 95% confidence intervals for logistic model of rater accuracy for each classifier applied to each image.

The subjective ratings had the highest agreement, followed by the SAASI and photo-assisted classifiers, according to the bootstrap analysis, which is consistent with the accuracy analysis. The SAASI was the most precise classifier, suggesting the least inter-rater variability (Figure 3). While the subjective classifier had the highest overall mean agreement, it did not perform best on all images. For example, the SAASI was good or better than the subjective classifier on images of moderate acne. The most precise ratings were associated with images of light skin with no acne and dark skin with severe acne, while the least precise ratings were associated with images of moderate acne and dark skin with no acne (Figures 2 and 4).

OPEN IN VIEWER

Figure 3.

95% confidence intervals for the mean (left) and standard deviation (right) of rater disagreement (overall; pooled images). Lower values correspond to greater agreement with the dermatologist rating and classifier precision.

OPEN IN VIEWER

Figure 4.

95% confidence intervals for mean rater disagreement (image by image).

Discussion

While there are acne severity instruments to aid physicians in grading acne, an acne severity self-assessment tool for adult patient use is lacking. This study aimed to establish an acne severity index for at-home patient use to facilitate a greater understanding of disease severity, enhance patient-physician communication, and encourage accurate assessment of acne severity in survey research. Participants’ subjective acne severity ratings without assistance had moderate agreement (73.2%) with board-certified dermatologists’ rating, and the SAASI provided a formal measure with a similar level of agreement (70.1%). Furthermore, the SAASI was a more precise rating system than raters by themselves and improved participants’ ability to rate moderate acne. This suggests the SAASI may be useful for large population survey research, where expert examination would be impractical or cost prohibitive, and potentially for clinical practice situations (audio-only telemedicine encounters) where visual assessment is not possible. In addition, while the SAASI is not designed for in-clinic use, where clinicians can visually assess patients’ disease severity, a SAASI-type instrument could potentially be incorporated into electronic medical records so that patients can document changes in disease severity over time. The reduced inter-rater variability as compared to subjective ratings, suggests that having a formal guide may be useful when trying to minimize variation between raters. Furthermore, the study suggests participants may be more capable at classifying acne severity accurately than previously thought.

Asking participants to rate acne subjectively, without any assisting system, was the most accurate method to classify acne in our study, followed by the SAASI, and using reference photographs. While we anticipated a greater improvement using dermatologist-rated reference images and a formal descriptive ranking system, this was not observed. This is most likely due to multiple factors. Due to the high prevalence of acne,^1,8 its presentation in advertisements, and the impact of social media—where selfies and selfie filters have increased the focus on faces and the idea of perfect skin, respectively^9,10—people potentially have an increased awareness and understanding of their acne severity. However, a limitation of this study is that participants were asked to rate the acne of others, as opposed to themselves. While people may be more objective judges of other individuals’ acne, it is not clear whether this same objectivity can be applied to themselves. With the increasing use of social media and face filters, there has been a rise in body dissatisfaction or body dysmorphia, ¹¹ with one study finding “skin disfigurements” as the leading concern in up to 64% of participants. ¹² The increasing prevalence of body dysmorphia highlights why further research needs to be conducted to understand whether participants are equally capable and still more accurate than the SAASI at rating their own acne.

The SAASI demonstrated the highest precision as compared to subjective or photo-assisted ratings. The increased precision is likely secondary to the standardized nature of the SAASI with reference images and qualitative and quantitative questions. These can reduce inter-rater variability and subjective conceptions. However, it is not clear why this reduced variability did not consistently outperform subjective ratings.

Raters were more accurate in rating images of light skin than images of dark skin. This is not completely unexpected, as images of skin of colour in advertising and popular culture have long been underrepresented. Society is progressing to increase diversity representation throughout popular culture and in skin disease images. In addition, participants had the greatest difficulty identifying moderate acne, followed by mild acne. This is likely because most people agree that severe acne is severe, while for mild and moderate disease, there may be more subjective variability with some people’s responses being coloured by various preconceptions and thinking that the condition is subjectively severe even though there are not many lesions.

This study builds on our experience developing self-administered objective severity measures for psoriasis, atopic dermatitis, hidradenitis suppurativa, and rosacea.^13-16 Severity assessment tools designed for patient use have aided numerous large population survey research, where physician assessment is impractical.^17-19

Limitations of this study include using a convenience sample through an English-based online crowdsourcing survey, therefore excluding non-English speaking individuals and those without regular access to the Internet. In addition, a majority of participants were younger; however, acne is more prevalent in younger people. These factors may introduce selection bias and limit the generalizability of the findings in differing population subsets; for this reason, future research could assess the validity in other populations. Furthermore, while designed to assess facial acne, the reference images were limited as they did not include the full face or other body areas that may impact how one perceives acne. In addition, some images included other blemishes, such as hyperpigmented macules/scars, which may have impacted participants’ rating of images, and using the descriptor “red and inflamed” in the multiple-choice section of the SAASI may introduce bias as red can be poorly visualized in darker skin tones. The images used came from a single institution and did not have standardized lighting; however, since this is a self-administered tool, the lighting users of the SAASI would have at home would not be idealized standardized lighting, either. Future studies could assess whether an instrument using different photographs obtained under standardized conditions would provide greater accuracy. Lastly, a strength of the SAASI is that it is designed to capture objective acne severity; this is also a limitation in that the measure does not assess subjective disease impact. When measurement of subjective impact is also desired, the use of the SAASI could be complemented with the use of a measure of quality-of-life impact such as the Dermatology Life Quality Index.

Conclusion

The SAASI provides moderate agreement and accuracy with board-certified dermatologists’ rating of facial acne vulgaris severity. The reduced inter-rater variability as compared to subjective ratings suggests that having a formal guide may be useful when trying to minimize variation between raters. Further research using the SAASI may help delineate the reproducibility of these results in diverse patient populations, its utility in longitudinal research studies, and its potential impact on clinical management decisions, if any. At present, the SAASI may facilitate acne survey research in large populations, where expert examination would be impractical or cost prohibitive, by providing a straightforward, formal instrument for acne vulgaris severity assessment.

Supplemental Material

Supplemental Material

Supplemental Material