Abstract
The low incidence of visual impairments (i.e., blindness and low vision) naturally results in a small field of research. However, drawing from the knowledge base of other fields can enrich the work of individuals in the field of visual impairment. It is through this lens that we sought to apply principles from the fields of Applied Behavior Analysis and Early Childhood Special Education to our work with infants and toddlers with visual impairments. Specifically, we analyzed the social validity of online training and coaching as we implemented a researcher-developed approach to early intervention service delivery.
Methodological Framework
Kazdin (2011) credits Skinner’s early work with animals (i.e., operant conditioning) as foundational to the study of human behavior and single-case research methods. By the late 1960s, applied behavior analysis was formally recognized as a means to intervene and measure socially important behaviors (Baer, Wolf, Risley, 1968). However, as this approach developed, Wolf (1978) questioned the idea of changing human behavior without attending to stakeholders’ perspectives. He suggested that, within single-case study designs, researchers should also gather and report what he called the “social validity” of a study. In his seminal work, Wolf proposed that social validity should encompass feedback on the study’s goals, procedures used to achieve those goals, and outcomes (intended and unintended) attained as a result of an intervention (Wolf, 1978).
Social validity data are not meant to be linked to the primary dependent variable of single-case research (Schwartz & Baer, 1991). Instead, they can help us understand the value an intervention offers to stakeholders. Strain, Barton, Dunlap (2012) suggest that single-case research lends itself to opportunities for learning that can inform sustainability, usability, and feasibility. Social validity data can help hone program improvements. Such information can be gathered using qualitative methods such as open-ended interviews with participants prior to and after intervention (Leko, 2014).
Although Horner, Carr, Halle, McGee, Odom (2005) included social validity as one of the four cornerstones when evaluating the quality of a single-case research study, literature reviews consistently show that social validity data are reported in a limited number of articles (Carr, Austin, Britton, Kellum, Bailey, 1999; Ferguson, Cihon, Leaf, VanMeter, McEachin, 2018; Kennedy, 1992). In fact, in a comprehensive review of studies reported in the Journal of Applied Behavior Analysis, Ferguson et al. (2018) found that only 12% (n = 141) of the studies published between 1999 and 2016 (n = 1,209) included social validity data. Ferguson and colleagues analyzed these 12% using the framework outlined by Wolf (1978) and found that 85% of the research teams assessed procedures, 60% assessed outcomes, and only 12% assessed goals. Snodgrass, Chung, Meadan, Halle (2018) reviewed single-case studies (n = 429) published in six top-tier journals beginning with articles published after Horner et al. (2005) identified social validity as a quality indicator. In their analysis, Snodgrass et al. found that although 27% (n = 115) of research teams reported some social validity data, only 6% (n = 28) included all three areas (goal, procedures, and outcomes) recommended by Wolf (1978).
Thus, given the importance of stakeholders’ perspectives to the practical application of behavioral interventions, we sought to incorporate a comprehensive investigation of social validity with parents and practitioners as they participated in a single-case study of early intervention practices. A brief overview of this larger study is described next along with the positioning of the social validity component within the investigation.
Early Intervention
Foundational Practices
It is recommended that services in early intervention (with children aged from birth to 3 years) focus on family-centered and relationship-based practices (Division of Early Childhood, 2014). In this delivery system, goals emphasize family priorities while promoting parental involvement during early intervention sessions. Such services build competence in parents and increase the likelihood that they will implement learned strategies during daily routines. Yet, the early childhood literature shows that such practices are under-utilized across disciplines. In fact, teachers of students with visual impairments that serve families of infants and toddlers show limited use of family-centered relationship–based practices, even though these practices are believed to be beneficial for the population (Ely & Ostrosky, 2017).
A Researcher-Designed Approach
Given this need for a change to practices, the first two authors designed a single-case study to utilize online training and coaching paired with a researcher-made framework called the Matrix Approach with the goal of helping teachers of students with visual impairments implement family-centered relationship–based practices in their work with families and their children (Ely & Ostrosky, 2021). The Matrix Approach is a framework for organizing early intervention sessions to emphasize parental involvement and decision-making. The approach utilizes a matrix to organize family-identified goals and routines with potential strategies to meet their priorities. The framework includes three phases. In the planning phase, the interventionist helps the parent identify either (a) a goal that they would like to see their child attain or (b) a daily routine that is problematic for their family. The interventionist helps the family break the goal or routine into smaller steps while identifying outcomes that reveal success. Next, the parent and professional collaborate to identify activities that can help the child move toward that first step of success. In the engagement phase, the parent is asked to interact with their child to implement the activity. During this phase, the interventionist coaches the parent to affirm their competence while providing developmental information and suggesting alternate approaches when appropriate. In the final phase, reflection, the interventionist reviews the strategies that were successful while helping the parent identify ways they could extend the activity into their daily routines throughout the week. Together, these three phases make up the Matrix Approach. More information on the Matix Approach is available at http://www.eiviprofessionals.com.
As suggested in the literature (i.e., Leko, 2014; Schwartz & Baer, 1991; Snodgrass et al., 2018; Wolf, 1978), and explicitly outlined in the quality indicators for single-case methodology (Horner et al., 2005), the first two authors designed a study on the Matrix Approach with careful attention to social validity. In this paper, we share the social validity data, gathered as part of the larger single-case study. Following Wolf’s framework, data on goals, procedures, and outcomes were gathered to evaluate the usefulness of the Matrix Approach. Four research questions were addressed: 1. Do participants value the goals of the study (i.e., building collaboration in home-based early intervention sessions and using online training and coaching to change professional practice)? 2. After participation in online training and coaching, do early intervention providers value this as a means to change their practice? 3. After using the Matrix Approach, do participants value the approach as a framework for organizing early intervention sessions? 4. What do participants report as impactful study outcomes?
Appropriate approvals from the Institutional Review Board of the University of Illinois Urbana-Champaign were obtained for the study reported here.
Methods
Qualitative measures were used to study social validity, including pre- and post-intervention interviews completed with parents and professionals. In the sections that follow, researcher beliefs and assumptions are described along with the participants and setting, data sources, and data analysis procedures utilized to answer the research questions.
Reflexivity
In qualitative research, it is important for researchers to reflect on personal characteristics and perspectives that influence their beliefs. All three authors are white females, and, at the time of the study, the first and third authors were graduate students at a Midwestern university, where the second author was on faculty. The first and second authors have experience working with young children with visual impairments, and they each have a family member who is blind or has low vision. The third author has experience working with children with a range of abilities not specific to visual impairments. All three authors strongly believe in the importance of the family unit as a source of expertise, support, and resources when an individual has a disability, and they believe that family-centered relationship–based practices are essential for building competence in parents so that they can better support their children. Further, they acknowledge that family-centered relationship–based practices have the potential to build strong connections between parents and their children by encouraging parents to take the lead and implement strategies with family members.
Participants and Setting
Four professional and family triads participated in this study. Professionals were trained teachers of students with visual impairments who worked in their state’s early intervention system. Each family included a parent and their child with visual impairment who received early intervention services from a teacher who participated in the study. To protect the anonymity of participants, pseudonyms were assigned to each triad (professional, parent, and child). The professionals were all women (“Polly,” “Penny,” “Pam,” and “Paige”), and family members were all mothers (“Mia,” “Mary,” “Mabel,” and “Molly”), with children (“Chastity,” “Chad,” “Charles,” and “Chip”), respectively.
Data Collection and Analysis
According to Wolf (1978), validation of a study should encompass the significance of goals, appropriateness of procedures, and importance of outcomes. Therefore, data sources were created to provide evidence of social validity in these areas including pre- and post-intervention interviews of parents and professionals. Three interview questions were asked of participants to allow for comparison of perspectives before and after the study. Parent and professional post-interviews contained additional questions that offered a way to understand their views about the study goals, procedures, and outcomes.
Data Collection
The four professionals and four parents participated in telephone interviews. Sample questions included, (a) How would you describe your role in the early intervention session, (b) From your perspective, describe what you noticed, if anything, about differences in your sessions once you began using the Matrix Approach, and (c) Given the three components of the approach (planning, activity, and reflection) was there a piece that you especially liked or disliked?
All pre-interviews were completed by the first author, while all post-interviews were conducted by an early childhood special education graduate student who was naïve to the project. This decision was made to avoid influencing participant responses during post-interviews since the first author had developed relationships with participants throughout the study. Interviews were recorded and transcribed by an early childhood special education graduate student prior to data analysis.
Data Analysis
Data were analyzed following methods described by Creswell (2013). First, data were organized by question and participant. Then the first and third authors studied the data to identify main points within responses. These became categories for coding. They coded the data independently before coming together to reach consensus on coding. The researchers looked for similarities in the data between categories and collapsed categories as appropriate. This process (i.e., independent coding, coming together to reach consensus, and collapsing categories) was repeated twice. Finally, major themes were identified by the researchers.
Results
Social validity: Pre- and post-intervention interview data.
A primary goal of the study was to foster parental involvement in sessions as described by family-centered relationship–based practices and believed appropriate for families when a child is visually impaired (DEC Recommended Practices, 2014; Ely & Ostrosky, 2018; Hatton, Chen, Snyder, Smyth, Greeley, 2018). Before the study began, two of the four professionals said that they would like to increase parental involvement during intervention sessions. At the end of the study, all four professionals indicated that they felt the approach was valuable and useful to their practice by inviting the parent to identify goals and guide sessions. Three of the parents indicated that they would like their early interventionist to continue using the Matrix Approach in future sessions. The parent of one child, who aged out of early intervention by the end of the study, expressed some reservations about using the approach were she to continue receiving services. A detailed description of the major themes is described next, using Wolf’s framework.
Goals
As recommended within family-centered relationship–based practices, the Matrix Approach was designed to encourage parental involvement during early intervention sessions. Data showed that prior to intervention, parents and professionals both saw the role of the professional as expert and the parent as learner in intervention sessions. Further, professionals were motivated to participate in the study because they saw it as a way to (a) increase parental involvement and (b) learn more about the Matrix Approach. These two factors are discussed next.
Parental Involvement
Both professionals and parents described a lack of involvement from parents during sessions before the study began. However, while two interventionists indicated that they wanted to increase parental involvement, both parents and professionals were fairly content with the role of professionals in planning sessions, giving information, and working directly with the children. Although professionals described the value of parent input, they held that it was the primary role of the interventionist to lead sessions. For example, Polly explained that the role of the early interventionist was “working with the child and seeing what they can do… [then] working with the parent and educating the parents on how to follow-through.” Mary noted that her role was to “…sit back; I watch and try to incorporate after [Penny] is gone. [Chad is] more whiny when I’m involved.” And, while Mabel described her sessions with Pam as collaborative, she still described Pam in a lead role. She said, “I feel that we work together…. If [Charles is] doing fine and engaging well with her, but … if he’s not participating, then I try to step in and refocus or redirect him.”
Participation
Recruitment materials did not identify use of the Matrix Approach as a study component, but rather described the intervention as one that might lead to improved outcomes for parents and children. Some information about the approach was available to the professional vision community through an online website, a publication (Ely, Guilifor, Hollinshead, 2017), and conference presentations. However, as the study progressed, it became evident that the professionals affiliated the Matrix Approach with the researchers and saw the study as an opportunity to learn more about it. Therefore, the professionals’ desire to learn about the approach served as a natural form of social validity.
Procedures
Online training was valuable to professionals, yet they noted the necessary role of coaching. Also, in implementing the Matrix Approach, professionals and parents liked the structure it provided and felt the planning segment was essential. These perceptions from parents and professionals about study procedures are detailed next.
Coaching
Four to six online coaching sessions were provided when the first author met with each early interventionist for approximately 30 minutes between home-based intervention sessions. During coaching, discussions often centered around viewing short video clips from the session. Professionals indicated that viewing these clips proved helpful in applying the approach. For example, Penny stated, I think the coaching was definitely very, very important. Just hearing her feedback, seeing those videos being played and seeing the areas and sections where certain things were being done or said . . . it’s really good to just be able to pull back and see that. . . . [H]er coaching was very beneficial in helping me realize what I was doing.
Structure
Participants indicated that the three segments of the approach (planning, activity, and reflection) were a valuable way to structure sessions. Mary explained that she did not particularly like planning, but felt it was an important piece of the session. Polly, who worked with Mia, reported that she found the reflection portion to be redundant with discussion that took place during planning and the activity segments. Interestingly, this reflection activity was identified by Mia as being important in helping her identify what she could work on throughout the week. Molly believed all three components were essential for the approach to be effective. She explained that the structure of the approach helped her see the steps necessary to reach important outcomes. Further, it allowed her to identify why an outcome worked or did not work so that strategies could be revised when necessary.
Planning
Each of the professionals found the planning portion of the Matrix Approach to be key to understanding what parents wanted out of the session while helping parents assume a lead role in the session. For example, Penny explained that she liked that: It was something that we decided, we planned together. We get there for that moment and we talk about what [Mary] wants to work on … versus what I think should be happening.
Paige explained that encouraging the parent to identify her needs has “… given me a different perspective, watching how I changed and seeing how the family progressed…. Chip’s mom became so much more confident in her ability to do things with him and expressing an opinion.” Pam said that the planning discussion brought to light struggles: “I wouldn’t have known if I didn’t ask [Mabel] those questions.” However, although planning was intended to provide shared leadership, Mabel felt the approach fostered too significant a shift in control. She explained, I can give goals … [but] I feel like it should be a joint planning [exercise] and it . . . felt like it was up to the parent. [If] this was brand new to us, I wouldn’t have known really what to suggest for a goal.... I think that’s hard for new parents of children with visual impairments to come up with stuff like that on their own.
Outcomes
Overall, participants felt that the Matrix Approach resulted in professional and personal growth. Professionals increased their understanding of parent needs and how to identify those needs. Parents gained confidence as they learned to identify priorities, work with their child to accomplish those goals, and advocate for the needs of their family. These changes describe growth toward family-centered relationship–based practices.
Another outcome of the approach appeared to be parent follow-through. The structure of sessions helped draw parents’ attention to how they could embed strategies into daily routines. When asked about continuation, participants indicated a desire to use the approach after the study ended; several participants described ways they would alter the approach to meet their individual needs.
Family-Centered
When professionals reflected on their personal growth as a result of using the Matrix Approach, they indicated a deeper understanding and awareness of family priorities. Pam attributed this awareness to asking questions about goals during planning. Penny explained that it caused her to take a “… step back and observe and listen. . . .”: … [Sometimes] you forget that you’re there to support the family and help them grow and learn. … You have to listen to what they really want instead of assuming … this is what they want and this is what we need to work on.
Paige described the growth in herself and in the family, I’ve realized that I need to value the input and opinions of my families more … to figure out what their genuine needs are and to help them formulate and identify what those needs are and really help them come up with a plan.… I think a lot of Chip’s progress was largely due to helping Mom realize what resources she already has. … And then taking all those things and building up her knowledge and confidence to then take what was already readily available to her and find new and creative ways to use it.
The parents echoed this sentiment as they described their own growth as a result of the intervention. Molly said her confidence grew, and both she and Mia echoed similar sentiments to Mary, who noted, “Penny guid[ed] me on what to do and what’s going to be best for him. So, I guess it helped me better understand my son and what is going to help him.”
Follow-Through
Both parents and professionals noted increased follow-through. The parents seemed to attribute this increase primarily to more distinctly articulated goals and noticeable results, while professionals believed increases in follow-through resulted from confidence and leadership in parents. For example, Mary explained that, “… during the reflection period at the end of the session, we would sit down and talk about … things that we could … incorporate … so we use[d] the strategies that we learn during therapy every day.” Similarly, Mia and Molly noted that the sessions provided such a clear understanding of how to use the strategies in daily routines that they were able to use them. The progress that they saw in their children made them want to continue using the strategies as much as possible.
Paige described the value of parent-identified goals, along with a clear outline of strategies to accomplish those goals. Polly attributed increases in follow-through to Mia, “having the confidence to realize that she could figure something out on her own or knowing which things to think about rather than relying on [Polly] to decide what we should be working on.” Penny also recognized the increased follow-through and attributed it to the fact that the goals were: …things that Mary wanted support on. She was the one deciding, “this is what I want to work on, what I want to do with Chad.” …[S]he took ownership, she was being responsible … taking that and running with it.
Continuation
When asked about continued use of the approach, all participants indicated value in its use, although several suggestions for adaptations were given. Polly and Penny felt that the approach could be used effectively without the paperwork; while Paige found the paperwork helpful in guiding her practice and left it for the family to use throughout the week. Pam found value in the approach, but preferred to use it occasionally as a planning tool rather than as a guide for weekly sessions. Mabel found the approach to be too parent-driven. Similarly, Mia liked the planning and reflection parts, but preferred that Polly directly work with Chastity during the activity segment so she could observe and learn how to implement strategies more effectively. In fact, maintenance data showed that Polly did return to directly working with Chastity; however, Mia was observed to be verbally involved and closer in proximity during maintenance than she had been before the Matrix Approach was implemented. Conversely, Paige, Mary, and Molly reported valuing the approach as designed and did not indicate a desire to see it altered.
All four professionals indicated that they would use the approach with other families, and that it would probably be easier to begin with new families rather than existing families where patterns of engagement were already established. Pam speculated that it might be difficult to use this approach with families if parents did not have concerns with daily routines.
Discussion
Evaluation of the social validity in single-case research is an important component in determining the usefulness of study outcomes. Several lessons were learned as we explored this critical aspect of our research.
Wolf’s framework (1978) for social validity proved valuable in understanding the significance of goals, appropriateness of procedures, and importance of outcomes. However, methods for gathering these data must be an intentional component of a study design, and planning is critical. For example, we felt it was important to ask about the significance of study goals before the participants entered baseline in order to avoid influencing their perceptions as a result of study participation. However, if measuring social validity was not a part of our initial research design, we would have missed the opportunity to gather pre-intervention data. In fact, a failure to think deeply about social validity when designing a study may be a contributing factor to why the social validity of study goals is seldom reported (Snodgrass et al., 2018). Further, gathering pre- and post-study intervention data allowed us to compare changes in perceptions, a component that proved highly beneficial in understanding our outcomes.
Further, intentionally and purposefully planning social validity data collection ensures that the needs of stakeholders remain a high priority. For example, articulating study goals in interview questions challenged us as researchers to deeply consider the practicality of our work. This also held true as we thought through the procedures. Although we did not ask participants about the procedures until after maintenance data were gathered, as researchers, we needed to think through these procedures from the standpoint of our participants as we crafted the interview questions. Such consideration for both goals and procedures is bound to impact the research-to-practice gap as researchers are forced to look at these components in a practical light.
Strain et al. (2012) highlight the benefits of unexpected outcomes that are often only visible through social validity data. They argue that such information serves a valuable purpose in allowing researchers to refine interventions for sustainable use. In the present study, it was through comparing pre- and post-intervention data that the nuances of observed changes were brought to light. Such insights provide important implications for practice.
Implications for Practitioners
Collaborative planning serves an essential but complex role in early intervention. Several participants believed that collaborative planning was the catalyst for building confidence in parents as they learned to identify and address their own needs. At the same time, one parent participant felt that too much responsibility was given to parents. These outcomes, while encouraging, highlight the complex nature of being collaborative when delivering home-based services. Further, this study provides an introductory glimpse into the usefulness of the Matrix Approach in practice. However, future study of the approach should include investigations of its use with families from different cultural backgrounds and with fathers and other caregivers.
Research-informed-practice and practice-informed-research is a linkage that many educational researchers strive to achieve when conducting applied studies. Social validity encourages and informs such partnerships when it is an intentional part of a study design. Social validity data gathered at various points throughout a study appear to be an effective approach when conceptualizing research that is designed using Wolf’s framework.
Footnotes
Authors’ Note
The work presented in this manuscript is a result of the dissertation completed by the first author in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Special Education through the University of Illinois Urbana-Champaign. Portions of the manuscript are reflective of the deposited dissertation and available at
. Special recognition is due the doctoral committee for their contributions to this work including Drs. Michaelene M. Ostrosky (chair), Rosa Milagros Santos Gilbertz, Hedda Meadan–Kaplasky, and Chris Clark–Bischke. Requests for research materials pertaining to this study are available from the first author.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
