Standardization of Videorecorded Speech Samples for Children with Cleft Palate: Methods to Facilitate High-Quality Speech Outcomes Data Collection

Abstract

Background

Collection of high-quality videorecorded speech samples is essential for speech outcomes research.

Solution

Cleft palate team SLPs were trained to collect standard videorecorded speech samples in the clinic setting across 20 sites in North America. Standard training and equipment was provided.

What We Did That is New

Quality management procedures were developed and utilized to verify video quality and protocol adherence. Over 97% of speech samples collected by trained SLPs met defined quality standards.

Keywords

cleft palate speech assessment outcomes

Introduction

Collection of high-quality, standard speech samples is essential for valid and reliable assessment of speech outcomes for clinical audit or research purposes.^1–3 Videorecording speech samples allows for archiving of clinical outcomes and for use in clinician training on perceptual ratings tools.³ While several international studies of speech outcomes in children with cleft palate have utilized recorded speech samples,^4–6 their inclusion in North American outcomes research has been less common. Barriers to implementation of speech sample collection may include clinician unfamiliarity with and/or lack of access to recording equipment, time and space constraints in clinic settings, patient behavior challenges, and lack of consistency of the speech sample used across speech-language pathologists (SLPs) and institutions. Regardless, relying on live clinic SLP ratings of speech, documented in patients’ medical records, is not ideal for research, due to susceptibility to bias.² Rather, the gold standard for outcomes reporting should include blinded external listener ratings of recorded speech samples to allow for reliable and valid speech measurement and generalization of findings. The purpose of this paper is to describe methods to facilitate efficient collection of high-quality standard videorecorded speech samples by SLPs in the cleft palate clinic setting.

Method

The Speech Sample

As part of two ongoing, prospective, multi-institutional studies of speech outcomes in children with cleft palate, across the US and Canada, we built upon the work of the Americleft Speech Project³ to develop and expand SLP training tools to elicit and videorecord standard speech samples of patients with repaired cleft palate (ages 3–23 years). A standard English (and/or Spanish^a) speech sample is videorecorded (see Appendix A), which parallels the characteristics of samples used in international outcomes studies.^2,4 In one of the studies that is focused on the assessment of speech outcomes in children at 3–4 years of age, the Goldman-Fristoe Test of Articulation-3 (GFTA-3)⁷ is also included in the sample (a picture-naming task that allows for assessment of speech and articulation at the single-word level). The recordings are later rated by trained SLPs using the Cleft Audit Protocol for Speech-Augmented-Americleft Modification (CAPS-A-AM).³ The American English Phrase Sample (based on the American English Sentence Sample,³ but shorter and easier for younger children) and GFTA-3 are also phonetically transcribed.

Equipment and Training

Videorecording equipment was selected to maximize quality and efficiency for collection of samples as part of routine cleft team clinic visits. The equipment was pilot tested at the primary study sites and found to be easy to use, flexible in terms of placement (based on exam room variations across clinical sites), feasible for use with patients ages 3 years and older and yielded high quality recordings. Each site was then provided with this set of equipment and a tip sheet for use (see Appendix B).

All SLPs and site study coordinators (SSCs) were provided with a detailed procedural manual (as well as quick reference guides) describing the speech sample elicitation protocol; visuals of the recording space and equipment set-up, camera and microphone settings, and patient positioning; and protocol for conducting a sound check. The SLPs and SSCs then attended up to 3 virtual training sessions (see training topics in Figure 1). Due to the geographic diversity of institutions, training was conducted live via Zoom, and recorded for later review. Follow-up sessions were also offered to assist with questions or troubleshooting. Following completion of training, but prior to speech sample collection for the study, each SLP was required to submit a practice video for review of video quality and protocol adherence.

Figure 1.

Topics covered during training sessions for speech-language pathologists, prior to obtaining videorecorded speech samples.

Quality Management

Thirty-eight SLPs from 20 cleft palate teams in North America have completed this training and submitted a practice video for approval. Practice videos were uploaded to each study's lead speech analysis site via secure data transfer and storage platforms for later review. Each practice video was systematically reviewed using a detailed quality assurance checklist (see example in Appendix C), created through a consensus process between the studies’ lead SLPs. All practice videos were reviewed by a lead SLP at the analysis site, with 50% of videos reviewed by two SLPs (as practice videos overlapped across sites participating in both studies). Agreement on whether a practice video passed or failed the quality check was 100%. A site SLP was approved to proceed with study data collection if their practice video met specified quality standards regarding (1) completeness of the sample, (2) audio quality, and (3) video quality. Thirty-eight SLPs (87%) were approved after review of their first practice video. Five (13%) SLPs were provided feedback to improve video quality and asked to submit a second practice video, after which all five (100%) were reviewed and approved. For practice videos that did not meet minimum quality standards, common reasons included: insufficient length of conversational sample, excessive background noise; and concerns with speech elicitation technique. Even for videos that were approved, all SLPs received individualized feedback on each practice video submitted. To achieve continued improvement in videorecording quality, SLPs and SSCs continue to receive feedback for all study videos submitted based on this quality management protocol. Common feedback provided is listed in Table 1.

Table 1.

Common Feedback Provided to SLPs to Improve Videorecorded Speech Sample Quality.

Decrease microphone to mouth distance	Position videocamera directly in front of child
Angle the microphone straight up at the mouth	Decrease SLP speech rate; examiner should avoid talking over child
Minimize background noise and visual distractions	When using visual stimuli, place materials slightly to the side, or in front (at eye level), of child
Remind child to not touch/play with the microphone	Have small prizes/stickers ready to go to encourage child to perform tasks
Remind parents that they do not need to join the conversation (nor repeat what the child says)

To date, 562 total videorecorded speech samples have been collected, with ongoing data collection underway. All were reviewed according to these quality management procedures and over 97% (546/562) met the defined standards and were deemed usable for later perceptual speech ratings.

Conclusion

Through SLP training and individualized feedback, high quality standard videorecorded speech samples can be collected in the cleft palate clinic setting for use in multisite audit and outcomes research.

Figure 2.

Screenshot of still image extracted from a practice videorecording, to illustrate microphone and patient positioning.

Footnotes

Acknowledgments

We would like to thank all the patients and families who have participated in these outcomes studies and those who volunteered to be part of the training videos. We would also like to specifically acknowledge Sarah Hatch Pollard and Erica Baimbridge for their study operations support and coordination of quality management activities, and the Americleft Speech Group, who helped create the speech sample stimuli and sample collection procedures that are used across both studies. We would also like to acknowledge the time and effort of all the speech-language pathologists across both studies who completed the training and submitted practice videos for review, the study coordinators, site Principal Investigators and surgeons, and other research staff who assisted with patient recruitment, enrollment, study operations, and video sample acquisition, editing and quality management processes, listed below by site:

Boston Children's Hospital: Carolyn Rogers-Vicenza, MD, Roseanne Clark, MS, CCC-SLP, Liza Catallozzi, MS, CCC-SLP, Ingrid Ganske. MD, Tamia Hargrove, BS, John Meara, MD, Karen Thompson, BM, MA, CCC-SLP.

British Columbia Children's Hospital: Jugpal Arneja, MD, Lindsay Beveridge, MSc, Marija Bucevska, MD, Erika Henkelman, MD, Sally Hynes, MD, Katie Kewin, MSc, S-LP, Sheryl Palm, MCISc, S-LP, Young Ji Tuen, BA

Children's Health Care of Atlanta: Colin Brady, MD, Katherine Dillon, MS, CCC-SLP, Angela LaGambina, MA, CCC-SLP, Kazlin Mason, PhD, CCC-SLP, Magdalena Soldanska, MD, Julia Tisheh, BS, Bethany Watson, MS, Joseph Williams, MD

Children's Hospital of Colorado: Greg Allen, MD, Katie Brellenthin, MS, CCC-SLP, Obinna Diala, MPH, Brooke French, MD, Bridget Harrington, MA, CCC-SLP, Jennifer Maybee, OTR, MA, CCC-SLP, Joyanne McShea, MA, CCC-SLP

Children's Hospital of Los Angeles: Mark Urata, MD, Aaron Tanner, MS, CCC-SLP,

Children's Hospital of Philadelphia: Scott Bartlett, MD, Oksana Jackson, MD, David Low, MD, Sue McCormack, MA, CCC-SLP, Jordan Swanson, MD, Cindy Solot, MA, CCC-SLP, Jesse Taylor, MD, Zachary Valenzuela, BS, Nisha Vora, BS, Jacob Ariel, Rea Chroneos

Cincinnati Children's Hospital: Stacey L. Ishman, MD

East Carolina University: Jamie L. Perry, PhD

Johns Hopkins University Children's Hospital: Richard Redett, MD, Carissa Cooney, MPH, Bethany Kirby, MS, CCC-SLP, Avery Raincook, MA, CCC-SLP, Kim Seifert, BSN, Jonathan Vuillier, BS

Lurie Children's Hospital: Sarah Vetter, MS, CCC-SLP, Mitchell Barna, MS, CCC-SLP, Noopur Gangopadhyay, MD, Arun Gosain, MD, Jennifer McGrath, MD, Sylvie Render, MS, CCC-SLP, Oveyaa Vignesh, MBS, Akira Yamada, MD

Nationwide Children's Hospital: Richard Kirschner, MD, Caitlin Cummings, MA, CCC-SLP, Alyssa Fogolin, BS, Heidi Wyse, MA, CCC-SLP, Jonathan Grischkan, MD, Ibrahim Khansa, MD, Greg Pearson, MD

Penn State/Hershey Medical Center and Lancaster Cleft Palate Clinic: Cathy Henry, MD, Don McKay, MD, Thomas Samson, MD, Ross Long, DDS, Brittany Bilitzo, MS, CCC-SLP, Sarah Stigelman, MSN, RN, Margaret Yoder, BSN, RN, Clare Robel, RN

Phoenix Children's Hospital: Kelly Cordero, PhD, CCC-SLP (Americleft Speech Group), Nicole Kurnik, MD, Davinder Singh, MD, Erica M. Weidler, MEd, MA, Jessica Williams, MS, CCC-SLP, Jenny Philp, MS, CCC-SLP, Yousef Ghazzawi, MBBCh

Rady Children's Hospital: Amanda Gosman, MD, Sam Lance, MD, Edna Montes, MA, Amanda Wacenske, MA, CCC-SLP

Riley Children's Health: Angela Dixon, MA, CCC-SLP (Americleft Speech Group)

Seattle Children's Hospital: Sara Kinter, PhD, MA, CCC-SLP, Raymond W. Tse, MD, Craig Birgfield, MD, Randall Bly, MD, John Dahl, MD, Russell Ettinger, MD, Richard Hopper, MD, Anna Meehan, AuD, CCC-A, EMT, Jonathan Perkins, MD, Kathleen Sie, MD, Lauren Beckett, MS, CCC-SLP

Shriner's Hospital for Children: Lee Aklureishi, MD, Claudia Belluci, MS, Julia Corcoran, MD, Mitch Grasseschi, MD, David Morris, MD, Mary O’Gara, MS, CCC-SLP, Chad Purnell, MD, Sarah Richards, MS, CCC-SLP

Texas Children's Hospital: Kristina Wilson, PhD, CCC-SLP (Americleft Speech Group), Edward Buchanan, MD, Jeff Carter, BS, Robert Dempsey, MD, Laura Monson, MD, Ellen Ortega-Moore, MS, CCC-SLP, Renata Maricevich, MD, Shola Olorunnipa, MD, Amy S. Xue, MD

The Hospital for Sick Children: David Fisher, MD, Karen Wong, MD, Simone Fischbach, MHSc, Kariym Joachim, MSc, Paula Klaiman, MCISc

University of Alabama – Birmingham: John Grant, MD, Meghan Amerson, MCD, CCC-SLP, Emily Berteau, MS, CCC-SLP, Jordan Brown, MS, CCC-SLP, Ashley Hood, MA, CCC-SLP, Rene Meyers, MD, Timothy Smith, Emily Turner

University of California – Davis: Travis Tollefson, MD, Christina Roth, MS, CCC-SLP, Randev Sandhu, Craig Senders, MD, Angela Beliveau, MPH,

University of Iowa: Scott Dailey, PhD, CCC-SLP, Deborah Kacmarynski, MD, Jessica Danielson, DDS, Briana Horwath, MA, Brandon Viet, MA, CCC-SLP, Kesten Anderson,

University of Wisconsin-Madison: Susan Thibeault, PhD, CCC-SLP, Catharine Garland, MD, Rebecca Johnson, MS, CCC-SLP, Marie de Stadler, MS, CCC-SLP, Mary Lynch, BA, Maddie Baranek, BS, Robert Fujiki, PhD, CCC-SLP, John Munday, MS, CCC-SLP

Cindy Dobbelsteyn, MS (Americleft Speech Group).

We would also like to acknowledge funding from the National Institutes of Health (R01DE027493, A comparative effectiveness study of speech and surgical treatments using a Cleft Palate Registry/Research Outcomes Network (CORNet) and U01DE029750, The Velopharyngeal Insufficiency Outcomes Prediction Study (VPI-OPS): A comparative effectiveness study of existing surgical procedures for velopharyngeal insufficiency following cleft palate repair and identification of criteria to guide a personalized treatment approach) for this project. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

This research was presented at the 2022 American Cleft Palate Craniofacial Association Annual Meeting.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical Approval

These studies were approved by the University of Utah Institutional Review Board and Advarra. Written informed consent was obtained from parents of children who were videorecorded for each study. Written permission was also obtained from the parent of the child depicted in for image use.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the NIH NIDCR, (grant number R01DE027493, U01DE029750).

ORCID iDs

Adriane L. Baylis

Kari M. Lien

Thomas J. Sitzman

Kathy L. Chapman

Notes

Appendix A

American English speech sample components rated with the CAPS-A-AM (Chapman et al., 2016). *Additional suggestions for eliciting conversational or connected speech from younger children or those with developmental delay are listed below. (**Counting from only 1–10 is elicited for younger children.)

Appendix B

Videorecording equipment and compatible accessories for use in multiple recording/clinic settings (eg, table or floor set-up) and Microphone guidelines.

Appendix C

Example of quality assurance checklist for speech video recordings used in the VPI-OPS study.

Videorecorded Speech Sample - Quality Assurance Checklist

Video file ID# Date of QM review:

Completeness of the Speech Sample:

Speech Sample Rating:

□ Acceptable (sufficient data for speech ratings: at least Conversation and Phrases are present)

□ Unacceptable (insufficient data for speech ratings)

Audio Quality:

Child's speech sample is clearly audible to the listener/rater.

□ Acceptable (minimal to no background noise; child occasionally varies mic to mouth distance; but overall audio quality is sufficient for speech ratings)

□ Unacceptable (excessive background nose, excessive microphone noise/artifact, child too far from microphone to capture speech signal clearly throughout majority of video)

Video (visual) Quality:

Child's full face visible in the image frame during the speech sample.

□ Acceptable (face is in frame for most of the video; some visual/movement artifact may be present but does not impact ability to rate child's speech)

□ Unacceptable* (face/mouth is not visible in entire video)

References

John

Sell

Sweeney

, et al. The cleft audit protocol for speech-augmented: A validated and reliable measure for auditing cleft speech. Cleft Palate Craniofac J. 2006;43(3):272-288.

Sell

John

Harding-Bell

, et al. Cleft audit protocol for speech (CAPS-A): A comprehensive training package for speech analysis. Int J Lang Comm Dis. 2009;44(4):529-548.

Chapman

Baylis

Trost-Cardamone

, et al. The americleft speech project: A training and reliability study. Cleft Palate Craniofac J. 2016;53(1):93-108.

Shaw

Semb

Lohmander

, et al. Timing of primary surgery for cleft palate (TOPS): Protocol for a randomised trial of palate surgery at 6 months versus 12 months of age. BMJ Open. 2019;9(7):e029780.

Klintö

Brunnegård

Havstam

, et al. Speech in 5-year-olds born with unilateral cleft lip and palate: A prospective Swedish intercenter study. J Plas Surg Hand Surg. 2019;53(5):309-315.

Willadsen

Lohmander

Persson

, et al. Scandcleft randomised trials of primary surgery for unilateral cleft lip and palate: 5. Speech outcomes in 5-year-olds-consonant proficiency and errors. J Plas Surg Hand Surg. 2017;51(1):38-51.

Goldman

Fristoe

. GFTA-3: Goldman Fristoe-3 Test of Articulation. 3^rd Edition. Psych Corp; 2015.