Informant Screening Profiles of Social and Emotional Learning: Exploring Teacher and Student Agreement

Abstract

When conducting universal social and emotional learning (SEL) screening, schools need clear decision-making guidelines for selecting informants. The current study examined informant profiles for screening SEL functioning using latent profile analysis with the student and teacher forms of the SSIS SEL Brief Scales for 536 students in grades 3–7. Teacher and student models each had three profiles emerge with roughly similar meanings of the profiles (Developing, Competent, and Advanced profiles), although a larger percentage of students were identified in the developing profile for the student rater. Profile categories aligned for 42% of students, with the most disagreement according between the Competent and Advanced SEL categories. For the teacher and student combined model, five profiles emerged (Competent-Developing, Developing-Competent, Competent-High Competent, Competent-Competent, and Advanced-Competent), with one profile indicating informant agreement. We explore gender and grade setting covariates and discuss implications for multi-informant research and practice.

Keywords

social and emotional learning universal screening informant latent profile analysis

Social and emotional learning (SEL) practices have garnered both a growing implementation in schools (Yoder et al., 2020) and extensive empirical support for a range of favorable student outcomes (e.g., Durlak et al., 2011). SEL refers to the acquisition and application of social-emotional skills, which offer enduring positive effects across various domains of human development. The Collaborative for Academic, Social, and Emotional Learning (CASEL) defines SEL as “the process through which children and adults understand and manage emotions, set and achieve positive goals, feel and show empathy for others, establish and maintain positive relationships, and make responsible decisions” (Yoder et al., 2020, p. 2). CASEL (2022) identifies five foundational SEL competencies: self-awareness, self-management, social awareness, relationship skills, and responsible decision-making (CASEL, 2022). Consistent with these competencies, school SEL programs instruct students in socio-emotional skills such as behavior and emotion regulation, perspective-taking and compassion, the development of healthy relationships, and conflict resolution skills (Weissberg et al., 2003).

Schools are increasingly recognized as viable settings for universal screening to promote student access to early identification and intervention services for social-emotional concerns (Stiffler & Dever, 2015). Given the importance of screening, schools need sound screening measures and clear guidance on administrative decisions such as who should complete the SEL screening measure. Multi-informant behavior rating scales are the predominate approach to universal screening of students’ social-emotional behaviors (Kamphaus et al., 2014). Selecting an informant or rater for universal SEL screening, however, can be a challenge (Dowdy & Kim, 2012). Research suggests that parents, students, and teacher often vary from one another on ratings of social-emotional functioning (De Los Reyes et al., 2015). Informant differences can be due to differences in perception or actual differences in behavior in different social situations with varying expectations. Below we review considerations for SEL informant decisions.

Interpreting Multi-Informant Social and Emotional Learning Screening Data

While there has been research supporting that informant recommendations vary by type of emotional concern (Dowdy & Kim, 2012), there is less guidance on selecting an informant for SEL screening and how to integrate data collected across several informants. Teachers are the most commonly used informant on universal screening measures, particularly in elementary school (Romer et al., 2020). Teachers spend extensive time with students, which affords them a unique position to observe student SEL functioning as well as have knowledge of peer normative behavior and state SEL standards. However, student self-report can contribute information on their own SEL functioning that teachers may not have (Donovan & Nickerson, 2007). Student raters can be uniquely able to identify internalizing concerns that may relate to SEL functioning (Salavera et al., 2019). Yet, due to potential limitations with young students’ ability to accurately report on their behaviors, student ratings generally are not emphasized until students are older (Durlak et al., 2011).

To address the paucity of informant research regarding children’s SEL competencies, Gresham et al. (2018) investigated informant agreement among teachers, parents, and students across SEL domains measured by the SEL edition of the Social Skills Improvement System (SSIS-SEL; Gresham & Elliot, 2017). Though teacher-parent informants demonstrated higher agreement than teacher-student and parent-student pairs across each of the five SEL domains assessed, overall cross-informant agreement was fairly high (rs were .34, .29, and .27 for parent-teacher, parent-student, and teacher-student pairs, respectively) relative to prior values observed in the literature (Achenbach et al., 1987), especially for composite SEL scores. Weaker cross-informant agreement emerged for individual competencies, specifically self-awareness and social awareness domains (Gresham et al., 2018). Overall, results suggest that while some cross-informant discrepancy is expected likely due to the influence of varying social contexts on social behavior, teachers, students, and parents rate students’ SEL skills with relatively low disagreement but additional research is needed to further evaluate SEL rating agreement.

Still, further research is needed to develop guidelines for interpreting informant discrepancies in the context of educational decision-making regarding SEL competencies (Gresham et al., 2018). Consideration of multiple informants is recommended in SEL assessment to capture unique perspectives of students’ SEL skills (Gresham et al., 2018). However, reliance on self-reports generally is discouraged for younger children (Durlak et al., 2011) and reliability and validity of SEL self-report measures is an identified area for future SEL informant literature (O’Conner et al., 2017). Given assessment of SEL needs is a core component of effective and sustainable SEL programming (O’Conner et al., 2017), further research on informant selection, differences, and data integration is warranted to guide school-based SEL assessment and programming.

Person-Centered Approaches to Considering Multi-Informant Data

Recently, person-centered approaches (e.g., latent profile analysis [LPA]) have emerged as a viable method to elucidate risk profiles and patterns of informant agreement to understand how screening outcomes differ based on informant selection. For SEL screening, LPA can identify how raters align in their perception of SEL functioning in relation to the CASEL SEL competency model (CASEL, 2022) and standard conceptualizations of overall functioning such as the SSIS SEL model of SEL performance levels (Elliott et al., 2020). Through exploring informant SEL profiles, we can demonstrate how informant decisions impact which students are identified for SEL support and what SEL skills are targeted for intervention. Further, informant SEL profiles can assist in evaluating the value of using both teacher and student informants for SEL ratings and clarify potential discrepancies that may pose a challenge with a multi-informant approach. For instance, von der Embse et al. (2021) illustrated how combined screening with both teachers and students for social, emotional, and behavioral risk provides an enhanced understanding of functioning to identify students for intervention and found rating incongruence of 21%, which was most likely to occur in the lower profiles. Further, researchers can identify factors that predict profile membership and evaluate how profiles predict student outcomes. For instance, identified student profiles (ranging from low-mid- and high-SEL functioning) based on aggregated teacher and student-ratings may be useful for both predicting distal academic outcomes in reading and math as well as identifying groups of students requiring additional supports (von der Embse et al., 2021).

Purpose of the Present Study

Although much research literature has established the importance of gathering multi-informant data, there is a need for guidance on how to integrate and interpret such data, especially when gathered in the context of universal SEL screening. Recently prominent person-centered analytic approaches hold promise to address this need and add to the growing knowledge base about SEL identification and intervention planning. As such, the purpose of this study was to utilize person-centered approaches to evaluate latent profiles for multi-informant SEL screening data. The following two research questions guided our analyses:

1. What patterns of SEL functioning emerge for student and teacher screening ratings separately and combined?

2. What is the level of agreement between raters for person-centered risk profiles?

Similar to von der Embse et al. (2021), we anticipated that several disparate profiles would emerge and that profiles would differ when including single-informant and multi-informant data. Considering the perennial importance of gender for considering social development (Del Guidice, 2015), and the fact that profiles were related to gender in prior similar research (von der Embse et al., 2021), we also considered the following exploratory research question to provide further guidance for universal screening practice.

3. Does profile membership differ across gender?

Finally, considering the developmental aspects of SEL (Nagaoka et al., 2015) and the different educational environments typically experienced across different grade levels (Anthony et al., 2021a; 2021b) we also considered a fourth exploratory research question:

4. Does profile membership differ across age group? Anthony

To address these four questions, the research strategy documented in the next section was employed.

Method

Participants

Two suburban private schools participated, which included 536 students in grades 3rd–7th grade and 22 teachers. Schools were selected based on interest in conducting both teacher and student SEL universal screening. Teachers were primarily Caucasian and female. Participants were drawn from universal screening data in two Midwestern private schools. The first school was a Catholic school serving 1109 children in kindergarten through 7th grade in a suburban setting. The school determined based on their capacity and school goals to screen students in 4th–7th grade. Five students were excluded from screening and the study due to returning the opt-out form to the school psychologist, which led to a total of 464 students being included in the study (99% completion). A total of 17 teachers completed SEL ratings. Students in this school were roughly evenly distributed across grades and 45% of students from this school were girls, and 55% were boys. Students were primarily White (93.3%) with the rest being Black (.4%), Asian (1.3%), Hispanic (1.7%), or multiracial (3%). A total of eight percent of the sample received special education services.

The second school was a private college preparatory K-4 school with 184 students in a suburban setting situated within a preK-12 district. Only data from 3rd and 4th grade students were included in the study with five teachers completing the SEL ratings. Participants from the other private school (n = 73) were evenly distributed in 3rd and 4th grade with equal distribution of gender (36 boys; 36 girls). The majority of the students were White (64.2%) with the rest being Black or African American (12.3%), Asian (5.6), Hispanic/Latino (2.2%), or two or more races (15.6). Approximately 30% of students receive needs-based tuition assistance.

Measures

SSIS Social and Emotional Learning Brief Scales-Teacher Form

The SSIS SELb Scales-T (SSIS SELb -T; Anthony et al., 2021a, 2021b; Elliott et al., 2020) is a brief version of the SSIS SEL Edition Rating Form - Teacher (SSIS SEL RF-T; Gresham & Elliot, 2017) and the SSIS Rating Scales (Gresham & Elliott, 2008). The SSIS SELb-T is a behavior rating scale that allows teachers to evaluate the social and emotional skills of students as determined by the CASEL SEL framework. The SSIS SELb-T can be completed by a teacher in roughly 5 minutes per student and was developed for use with K-12 students. Items on the SSIS SELb-T are rated on a 4-point Likert scale (0 = never, 1 = seldom, 2 = often, and 3 = almost always) that results in 5 scores aligned with the 5 CASEL competence domains (Self-Awareness, Self-Management, Social Awareness, Relationship Skills, and Responsible Decision Making), and yields an SEL Composite score. IRT was used to identify and analyze items from the SSIS SEL RF-T that would be included in the SSIS SELb-T, of which only 20 items (39%) of the original items were included (Anthony et al., 2022).

Anthony et al. (2021a; 2021b) found evidence to support the SSIS SELb-T scores as being psychometrically sound. For example, Cronbach’s alpha values were .95 for the SEL Composite and ranged from .79 to .85 (median = .83) across SEL scales. For the current sample, Cronbach’s α was .96. Test-retest reliability coefficients were .78 for the SEL Composite and ranged from .75 to .83 (median = .77) across SEL scales. Finally, interrater reliability coefficients were .65 for the SEL Composite and ranged from .47 to .65 (median = .55) across SEL scales. In regards to validity, the intercorrelations between the scales and convergent validity with the SSIS SELb-T and the SSRS-T, BASC-2, and Vineland-II were all as expected (Anthony et al., 2021a; 2021b), although more evidence to support the reliability and validity of the scores is needed.

SSIS Social and Emotional Learning Brief Scales-Student Form

The SSIS SELb Scales-S (SSIS SELb-S; Anthony et al., 2020) as a brief version of the SSIS SEL Edition Rating Form - Student (SSIS SEL RF-S; Gresham & Elliot, 2017) and the SSIS Rating Scales (Gresham & Elliott, 2008), a self-report measure that provides information on how students understand their own social and emotional skills. The SSIS SELb-S can be used with students ages 8 to 18 and consists of 28 items rated on a 4-point Likert scale (0 = not true, 1 = a little true, 2 = a lot true, and 3 = very true). The SSIS SELb-S can be completed in 5 minutes and yields scores for each of the 5 CASEL domains, as well as a total SEL Composite score. Potential uses for the SSIS SELb-S as suggested by Anthony et al. (2020) include using the scores to inform intervention planning, progress monitoring, and as a universal screener for social and emotional skills. As with the SSIS SELb-T, IRT was used to select 20 items from the SSIS SEL-RF-S and Anthony et al. (2020) tested the reliability and validity of the SSIS SELb-S scores on data from the SSIS Rating Scale standardization.

Anthony et al. (2020) found that the SSIS SELb-S scores had sufficient evidence of reliability. Specifically, Cronbach’s alpha was .90 for the SEL Composite with scale values ranging from .64 to .71 (median = .69). Cronbach’s alpha was .86 for the current study. Also, test-retest reliability coefficients were .87 for the SEL Composite and ranged from .64 to .83 (median = .71) across SEL scales. Evidence for validity of the scores was supported by moderate inter-scale correlations, as well as by moderate and expected correlations between the SSIS SELb-S to the BASC-2 (Anthony et al., 2020).

Procedures

Data were collected as part of universal screening procedures at each school where a passive consent form and data collection description was sent home to caregivers who were able to opt their child out of screening if desired. Ratings were completed in October of the 2019–2020 school year. Teachers (N = 22) completed the SSIS SELb-T for all students in their classes (ratings ranged from 2 to 32, median = 22). Students in the 3rd through 7th grade (N = 530) rated their SEL functioning on the SSIS SELb-S. The school psychologist in each building led the data collection process, trained teachers, and collected data from students missing ratings from the first round. The school psychologists checked that 100% of students with passive consent had both teacher and student ratings. Surveys were all distributed via Qualtrics and the school psychologist then de-identified aggregated datasets. The first author received IRB approval to access secondary de-identified data. Students in 3rd through 8th grade were included in the current study to include students with both a teacher and student rating.

Analyses

Research questions were evaluated with a person-centered latent profile analysis (LPA) to determine profiles of student and teacher SEL functioning. Analyses were completed in Mplus version 8 and the number of profiles was determined by evaluating fit statistics and considering interpretability and clinical significance of profiles (Muthén & Muthén, 1998–2012). Subscale sum scores for the SSIS SELb-T and SSIS SELb-S were entered in the LPA model. The analyses used a full information maximum likelihood estimation with robust standard errors via the LMR estimator of MPlus, which assumes missing at random (MAR) as a missing data mechanism. Only one case was missing teacher ratings for the teacher and combined analyses (99.8% completion) and there was no missing data for the student analyses (100%) completion. For determining stronger fit models, the lowest Bayesian information criterion (BIC; Schwartz, 1978) and Akaike information criterion (AIC; Akaike, 1987) suggest the best fitting model. Further, the Lo–Mendell–Rubin sample-size adjusted likelihood ratio test (LMR-A; Lo et al., 2001) compares the K – 1 class model with the current model where significant p values indicate that the current model improves the fit significantly better compared to the K – 1 model. Last, entropy indicates classification precision where values near one indicate stronger model fit (Muthén, 2004).

For Research Question 1, separate LPAs were first conducted for student and teacher ratings to identify the optimal number of profiles for each informant. After identifying the optimal number of profiles for the teachers and students separately, we compared teacher and student profiles through saving most likely profiles from MPlus for each informant to evaluate congruence between informants SEL functioning. Next, we entered teacher and student subscale ratings together into MPlus to identify profiles of functioning considering student and teacher ratings together. For all LPAs, the standard assumption of equal variances across classes was imposed. To answer Research Question 2, we used SAVEDATA command in MPlus to save the most likely profile for the teacher- and student-only models. We then compared the percentage agreement between the teacher and student models. For Research Questions 3 and 4, latent logistic regressions were used to compare profiles to the predictors of gender (binary variable; dummy coded where 1 indicated female) and grade setting (binary variable; dummy coded where 0 indicated 3rd–5th grade and 1 indicated 6th–7th grade). p-values below .05 indicated significance. We chose this approach to evaluating grade because of unequal distribution of cases across individual grades, to facilitate comparison and interpretation, and because of the typical differences in educational environments experienced between upper elementary grades and middle school grades.

Results

Descriptive Statistics

Table 1 presents the descriptive statistics of student- and teacher-rated Self-Awareness, Self-Management, Social Awareness, Relationship Skills, and Responsible Decision Making.

Table 1.

Descriptive Statistics by Gender.

	SSIS SELb-T Mean (SD)		SSIS SELb-S Mean (SD)
	Females	Males	Females	Males
Self-Awareness	9.44 (2.59)	8.69 (2.55)	8.95 (1.98)	8.28 (2.02)
Self-Management	10.13 (2.15)	8.95 (2.43)	8.28 (2.09)	7.58 (2.18)
Responsible Decision Making	10.61 (1.93)	9.79 (2.03)	9.96 (1.61)	9.10 (1.82)
Social Awareness	10.47 (1.86)	9.73 (2.04)	9.74 (1.91)	8.94 (2.06)
Relationship Skills	10.17 (2.07)	9.54 (2.06)	9.67 (1.62)	9.25 (1.92)
SEL Composite	50.82 (9.68)	46.54 (10.32)	46.61 (7.16)	43.15 (7.76)
Total n	239	290	239	291

Bold indicates the selected model.

Note. SSIS SELb + MHS, social skills improvement system social and emotional learning brief; T, Teacher, and S, student.

Teacher and Student Separate Latent Profile Analyses

Teacher Rating Patterns of Social and Emotional Learning Functioning

Model fit indices for the one-through five-profile SSIS SELb-T only models are presented in Table 2. The BIC, AIC, and SSaBIC continue to drop meaningfully until the third profile. Additionally, entropy was maximized with three profiles and the LMR-A value was significant, suggesting an improvement compared to the two-profile model. Due to the significant LMR-A p-value and maximized entropy, the three-profile model was selected in relation to Research Question 1. In terms of clinical significance, the three-profile model (Figure 1) provides three distinctive profiles for intervention decision-making with one profile with developing SEL skills. Profile terminology aligned the SSIS SEL categorical levels with Profile 1 (8%) being labeled Developing SEL, Profile 2 (39%) Competent SEL, and Profile 3 (52%) Advanced.

Table 2.

Fit Statistics for Latent Profile Analyses.

Number of Profiles	BIC	SSaBIC	Entropy	AIC	LMR-A	LMR-A p-Value
Teacher ratings
1	11,757.355	11,725.612	—	11,714.627	—	—
2	9842.593	9791.804	.942	9774.227	1901.868	<.001
3	9361.216	9291.381	.953	9267.212	505.581	<.001
4	9028.274	8939.394	.938	8908.633	360.988	.024
5	8966.202	8858.276	.946	8820.924	97.128	.190
Student ratings
1	11,130.845	11,099.102	—	11,088.116	—	—
2	10,434.727	10,383.938	.811	10,366.361	714.764	<.001
3	10,262.182	10,192.348	.778	10,168.179	204.742	.003
4	10,233.973	10,145.093	.772	10,114.332	64.142	.366
5	10,229.389	10,121.463	.806	10,084.111	41.128	.263
Teacher and student ratings combined
1	22,888.200	22,824.714	—	22,802.742	—	—
2	20,956.522	20,858.119	.942	20,824.063	1972.099	<.001
3	20,483.690	20,350.370	.955	20,304.229	543.093	<.001
4	20,183.158	20,014.920	.933	19,956.695	364.255	.0414
5	19,947.612	19,744.458	.932	19,674.148	293.161	.0014
6	19,752.562	19,514.490	.911	19,432.096	260.280	.2408

Note. BIC, Bayesian information criterion; SSaBIC, sample-size adjusted Bayesian information criterion; AIC, Akaike information criterion; LMR-A, Lo–Mendell–Rubin sample-size-adjusted likelihood ratio test. For entropy, values near one indicate stronger model fit. A significant LMR-A p-value indicates that the model significantly improves the fit compared to the K −1 model. Bold indicates the selected model.

Figure 1.

Estimated teacher-rated three-profile model.

Student Rating Patterns of Social and Emotional Learning Functioning

Table 2 presents model fit indices for the one-through five-profile SSIS SELb-S only models. The BIC, AIC, and SSaBIC meaningfully drop with the three- and four-profile models and entropy is maximized with five profiles. However, the LMR-A value is only significant with the three-profile model. The three-profile model was selected due to acceptable entropy, LMR-A significance, and meaningful decrease in information criteria. Further, the three-profile model provides three distinct profiles that represent differences in SEL functioning. In terms of profiles for the three-profile student model (Figure 2), 16% of students are likely in the Developing SEL (Profile 1), profile, 45% Competent (Profile 2), and 40% Advanced (Profile 3).

Figure 2.

Estimated student-rated three-profile model.

Level of Agreement for Social and Emotional Learning Functioning

Table 3 presents profile comparisons between informants to evaluate the level of agreement for Research Question 2. When the most likely profile was compared for each informant, we found that 42% of students were estimated to be in a similar profile across informants with 2% of students being estimated to be in a developing-developing profiles, 17% in a competent-competent profiles, and 23% in an advanced-advanced profiles. The greatest proportion of disagreement occurred with the competent and advanced profiles (15% teacher competent-student advanced and 24% teacher advanced-student competent). There were also profile disagreements for students with Developing SEL skills for both teacher and student informants. Of the 35 students estimated to be in the Developing profile from the teacher-rated model, 49% were estimated in the Competent profile and 26% in the Advanced profile from the student-rated model. Similarly, 76 students were estimated to be in the Developing profile from the student-rated model and 59% were estimated in the Competent profile and 29% in the Advanced profile with the teacher-rated model.

Table 3.

Profile Comparisons Between Informants.

Teacher-Student Profiles	N (%)
Developing-Developing	13 (2.45%)
Developing-Competent	21 (3.96%)
Developing-Advanced	11 (2.08%)
Competent-Developing	41 (7.74%)
Competent-Competent	88 (16.60%)
Competent-Advanced	79 (14.91%)
Advanced-Developing	26 (4.91%)
Advanced-Competent	129 (23.02%)
Advanced-Advanced	122 (23.02%)

Note. Bold indicates profile congruence.

Teacher and Student Combined Latent Profile Analyses of Social and Emotional Learning Functioning

Fit statistics for the teacher and student combined model are presented in Table 2 in relation to Research Question 1. The BIC, AIC, and SSaBIC continued to meaningful decrease until the five-profile model. The LMR-A value and significance level suggests that the five-profile model offer significant improvement compared to the four-profile model and entropy indicates acceptable separation for all tested models. The five-profile model was selected due to the decrease in information criteria, significant LMR-A, and high entropy. In the five-profile model (Figure 3), one profile suggested agreement between informants: Profile 4 estimated with 20% of students (Teacher Competent-Student Competent). The other profiles relate to differences in SEL functioning from the teacher and student perspective. Specifically, 9% of students were estimated to be in Profile 1 (Teacher Competent-Student Developing). Students in this profile tended to underestimate their SEL skills in comparison to teachers. In comparison, students in Profiles 2 and 3 tended to overestimate SEL functioning in comparison with teachers with 8% of students rating themselves as Competent whereas teachers indicated Developing SEL functioning (Profile (2) and 20% of students rating their SEL functioning near the Advanced category with teacher ratings in the mid-to-low Competent SEL category (Profile 3). Finally, 43% of students were estimated to be in Profile 5 where teachers on average rated students with Advanced SEL functioning but students reported average Competent SEL functioning.

Figure 3.

Estimated student-rated and teacher-rated combined five-profile model.

Profile Covariates

To explore Research Questions 3 and 4, the relationship between profiles and the covariates of gender (Research Question 3) and grade setting (Research Question 4) were examined with latent logistic regression (see Figures 4 –6 for displays of estimated probability of profile membership for the covariates). First, we describe the results for the gender covariate. For the teacher-only model, female students were more likely to be in the Advanced profile compared to the Developing (odds ratio [OR] = 2.42, p = .01) and Competent (OR = 2.41, p < .01) profiles. For the student-only model, female students were more likely to be in the Competent (OR = 2.40, p = .01) and Advanced (OR = 4.17, p < .01) profiles compared to the Developing profile and Advanced profile compared to the Competent profile (OR = 1.74, p = .03). For the combined model, female students were more likely to be in the Teacher Advanced-Student Competent SEL Profile compared to the Teacher Competent-Student Developing (OR = 3.01, p < .01), Teacher Developing-Student Competent (OR = 2.48, p < .01), Teacher Competent-Student Advanced (OR = 4.11, p < .01), and Teacher Competent-Student Competent (OR = 2.21, p < .01) profiles. There was no significant difference in gender probability for the other profiles.

Figure 4.

Estimated probability of profile membership across teacher rated three-profile model by gender and grade level.

Figure 5.

Estimated probability of profile membership across student rated three-profile model by gender and grade level.

Figure 6.

Estimated probability of profile membership across combined five-profile model by gender and grade level.

In terms of grade level, 3rd–5th grade was compared to 6th–7th grade. For the teacher-only model, 6th–7th grade students were more likely to be in the Advanced profile compared to the Developing (OR = 4.35, p < .01) and Competent (OR = 2.35, p < .01) profiles. For the student-only model, there was no significant difference in grade setting probability for any profile comparisons. In terms of the combined model, 6th–7th grade students were more likely to be in the Teacher Competent-Student Competent profile compared to the Teacher Competent-Student Developing (OR = 2.43, p = .04) and Teacher Competent-Student Advanced (OR = 2.10, p < .01) profiles. 6th–7th grade students were also more likely to be in the Teacher Advanced-Student Competent compared to the Teacher Competent-Student Developing (OR = 7.90, p < .01), Teacher Developing-Student Competent (OR = 5.73, p = .02), Teacher Competent-Student Advanced (OR = 6.81, p < .01), and Teacher Competent-Student Competent (OR = 3.15, p = .01) profiles.

Discussion

The current study examined informant profiles for screening SEL functioning. When screening for SEL functioning, schools need clear decision-making guidelines for selecting informants. Existing practice most often uses teachers to rate social-emotional functioning (Romer et al., 2020) but adding a student rater can enhance understanding of functioning from multiple perspectives. Both teacher and student informants add value for social-emotional screening but using multiple informants requires additional screening administration resources and could lead to risk identification disagreements.

Patterns of Social and Emotional Learning Functioning and Level of Agreement

Regarding Research Question 1, teacher and student models each had similar numbers of profiles emerge with roughly similar meanings of the profiles (three profiles with one group that needs attention, one that is competent, and one that is advanced). The SSIS SELb scale patterns were similar and generally within the same category—further supporting the use of the SEL Composite for decision-making (Anthony, 2022).

In regards to Research Question 2, teacher and student profiles aligned for 42% of students. A majority of the disagreements were less consequential for screening as 39% differed on whether the student in Competent or Advanced on SEL. More consequential is the 19% that differed on whether the student is Developing or Competent/Advanced, suggesting that students in these profiles may warrant further attention to determine if intervention is warranted. These results were corroborated by the combined models where a similar percentage (17%) were in consequential “disagreeing” profiles that may impact identification for additional SEL support. Results differ from von der Embse et al. (2021) where rating incongruence was only found for 21% of the sample and was more likely to occur in the lower profiles where there is higher risk and more potential for intervention discrepancies.

The combined LPA for Research Question 1 further elucidated that there was the Teacher Competent-Student Developing Profile (9%) and the Teacher Advanced-Student Competent (43%) in which students estimated their SEL competencies to be lower in comparison to teachers. In contrast, the Teacher Developing-Student Competent Profile (8%) and the Teacher Competent-Student Advanced Profile (20%) consisted of teachers rating lower SEL competencies whereas students indicated higher functioning. In cases where students estimated more competent SEL functioning compared to teachers, teachers indicated lower self-awareness functioning. This indicates that the students may not have self-awareness about their behavior, which may translate to less accuracy about their SEL functioning, although further research is needed to verify that interpretation. Gresham et al. (2018) also found weaker cross-informant agreement for the self-awareness domain.

Informant differences on universal screening result in confusion when using results to inform interventions and require additional resources to determine true and false positives. If replicated, these profiles could be further studied to better inform what to do when they emerge in practice. When schools only screen with one informant, they are missing potential differences in perception, although ratings reflect the perception of the informant and context rather than a true rating of functioning (De Los Reyes et al., 2015). Thus, informant selection impacts consequential validity and is a critical decision point when planning screening efforts. Results align with previous research demonstrating informant profile differences (Herman et al., 2018; von der Embse, 2021).

Profile Covariates

The gender covariate analysis (Research Question 3) validated the meaningfulness of the profiles as female students were more likely to be in Advanced SEL profiles. Similarly, von der Embse et al. (2021) found that male students were more likely to be in the low teacher and student SEL profile and Romer et al. (2011) found higher perceptions of social-emotional competencies for female students. The grade setting covariate analysis (Research Question 4) found that students in typical middle school grade levels were more likely to be in the Advanced profile when rated by teachers but we failed to find significant differences between grade settings when students rated their own SEL functioning.

Results align with von der Embse et al. (2021)’s finding that younger students were less likely to be in the profiles with teacher- and student-rated high social-emotional functioning compared to 6th grade students. This pattern could be indicative of developmental patterns as older children would be expected, on average, to have greater mastery of SEL skills. The differences when considering combined profiles could also point to differences in the developmental sensitivity of measures rated by different informants based on differences in the overall sample of behaviors considered and different rating patterns and processes across teacher and student raters (Whitcomb & Merrell, 2013). More research is needed to further explore and replicate these gender- and grade-based findings and inform applied practice. Furthermore, additional research is needed to further validate informant profiles to evaluate if profiles predict meaningful educational outcomes such as academic functioning, behavioral referrals, attendance, and school nurse or counselor visits.

Practical Implications

Results suggest that different informants provide unique information on SEL functioning. For instance, 16% of students were identified in the Developing SEL profile for the student-only model compared to 8% for the teacher-only model. Further, teacher and student profiles aligned for 42% of students, although the majority of the disagreement occurred between the Competent and Advanced profiles. Using teacher and student raters can assist in efficiently identifying SEL strengths and deficits from multiple perspectives.

Although there may be benefits to adding an informant, schools should consider how this will increase screening administration time and data review. Schools using this approach should expect an increase in the number of students identified and disagreements between informants. To assist in decision-making, schools can use a follow-up process (e.g., brief interviews, review of existing data, and rating forms) to determine which students would benefit from intervention. One benefit of self-report screening is that schools can administer a screening tool at the beginning of the school year rather than waiting 4–6 weeks for teachers to become familiar with students in their classrooms.

Limitations and Directions for Future Research

Results contribute to the literature base on informant differences and screening for the whole social-emotional child (Elliott et al., 2020) but should be viewed within the context of their limitations. First, the sample was limited to two private schools in one region of the country and only included primarily White 3rd through 7th grade students, which limits the generalizability of findings. Second, we used both theory and fit statistics to select the number of profiles for each model but fit statistics did not clearly support selection and there is still uncertainty in the exact number of profiles. To assist with these limitations, future research should examine SEL profiles for teacher, student, and combined models with larger and more diverse samples.

Another limitation is that the current study lacks an indication of true SEL functioning level and outcome variables to further validate the profiles. Future research should include additional covariates (race/ethnicity, age, and socioeconomic status) and distal academic and social-emotional outcome measures to understand characteristics of students in profiles and outcomes of profile membership.

In addition to directions to address limitations, future research could conduct a cost-benefit analysis on using a multi-informant approach to screening for SEL functioning to clarify the predictive, usability, and social consequences. Additional research is also needed to further understand profiles with disagreement and if disagreement on SEL skills predicts academic difficulties. Finally, research could also examine how informant ratings of SEL and internalizing needs can be integrated to facilitate decision-making and examine the stability of profiles across years.

Conclusions

In the current study, we found three profiles of functioning for teacher and student SEL ratings with groups aligning with the SSIS SEL criterion framework categories of Developing, Competent, and Advanced. We also found moderate agreement between raters with the most disagreement occurring between Competent and Advanced profiles. Profiles with rater disagreement were associated with lower levels of self-awareness. Results overall highlight the benefits of screening with both teachers and students to understand SEL functioning from different perspectives. Considering the importance of early identification of social-emotional needs, additional research on screening informants would be valuable to both researchers and practitioners.

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The second and third authors of this manuscript are also authors of the SSIS SELb Scales and receive royalties for its sale.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Kristy L. Brann

Christopher J. Anthony

Courtney A. DiOrio

References

Achenbach

McConaughy

Howell

(1987). Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. https://doi.org/10.1037/0033-2909.101.2.213

Akaike

(1987). Factor analysis and AIC (pp. 371–386). Springer Series in Statistics. https://doi.org/10.1007/978-1-4612-1694-0_29

Anthony

C. J.

Brann

Elliott

S. N.

Garis

E. J.

(2022). Examining the structural validity of the SSIS SEL brief scales – teacher and student forms. Psychology in the Schools, 59(4), 260–280. https://doi.org/10.1002/pits.22607

Anthony

C. J.

Elliott

S. N.

DiPerna

J. C.

Lei

P.-W.

(2020). The SSIS SEL brief scales- student form: Initial development and validation. School Psychologist, 35(4), 277–283. https://doi.org/10.1037/spq0000390

Anthony

C. J.

Elliott

S. N.

DiPerna

J. C.

Lei

P.-W.

(2021a). Initial development and validation of the SSIS SEL brief scales - teacher form. Journal of Psychoeducational Assessment, 39(2), 166–181. https://doi.org/10.1177/0734282920953240

Anthony

C. J.

Ogg

Jenkins

L. N.

(2021b). Academic enablers as dynamic moderators: Exploring academic enablers and achievement across elementary school. Journal of School Psychology, 86(8), 15–31. https://doi.org/10.1016/j.jsp.2021.02.007

Collaborative for the Academic, Social Learning, Emotional . (2022). What is the CASEL Framework? https://casel.org/fundamentals-of-sel/what-is-the-casel-framework/#:∼:text=A-framework-creates-a-foundation,advance-students'-learning-and-development

Del Giudice

(2015). Gender differences in personality and social behavior. In Wright

J. D.

(Ed.), International encyclopedia of the social and behavioral sciences (2nd ed., pp. 750–756), Elsevier.

De Los Reyes

Augenstein

T. M.

Wang

Thomas

S. A.

Drabick

D. A. G.

Burgers

D. E.

Rabinowitz

(2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141(4), 858–900. https://doi.org/10.1037/a0038498

10.

Donovan

S. A.

Nickerson

(2007). Strength-based versus traditional social-emotional reports: Impact on multidisciplinary team members’. Behavioral Disorders, 32(4), 228–237. https://doi.org/10.1177/019874290703200401

11.

Dowdy

Kim

(2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6(4), 1–10.

12.

Durlak

J. A.

Weissberg

R. P.

Dymnicki

A. B.

Taylor

R. D.

Schellinger

K. B.

(2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x

13.

Elliott

S. N.

Anthony

C. J.

DiPerna

J. C.

Lei

P.-W.

Gresham

F. M.

(2020). SSIS SEL Brief+ mental health scales: User guide and technical manual. SAIL CoLab.

14.

Elliott

S. N.

Lei

P.-W.

Anthony

C. J.

DiPerna

J. C.

(2020). Screening the whole social-emotional child: Integrating emotional behavior concerns to expand the utility of the SSIS SEL brief scales, School Psychology Review, 52(1), 15–29. (in press). https://doi.org/10.1080/2372966X.2020.1857659

15.

Gresham

F. M.

Elliot

S. N.

(2017). Social skills improvement system social-emotional learning edition rating forms. Pearson Assessments.

16.

Gresham

F. M.

Elliott

S. N.

(2008). Social Skills improvement System-Rating Scales. Minneapolis, MN: Pearson Assessments.

17.

Gresham

F. M.

Elliott

S. N.

Metallo

Byrd

Wilson

Cassidy

(2018). Cross-informant agreement of children’s social-emotional skills: An investigation of ratings by teachers, parents, and students from a nationally representative sample. Psychology in the Schools, 55(2), 208–223. https://doi.org/10.1002/pits.22101

18.

Herman

K. C.

Cohen

Reinke

W. M.

Ostrander

Burrell

McFarlane

Duggan

A. K.

(2018). Using latent profile and transition analyses to understand patterns of informant ratings of child depressive symptoms. Journal of School Psychology, 69, 84–99. https://doi.org/10.1016/j.jsp.2018.05.004

19.

Kamphaus

R. W.

Reynolds

C. R.

Dever

B. V.

(2014). Behavioral and mental health screening. In Kettler

R. J.

Glover

T. A.

Albers

C. A.

Feeney-Kettler

K. A.

(Eds), Universal screening in educational settings: Evidence-based decision making for schools (Chapter 10, pp. 249–273). American Psychological Association.

20.

Mendell

Rubin

(2001). Testing the number of components in a normal mixture. Biometrika, 88(3), 767–778. https://doi.org/10.1093/biomet/88.3.767

21.

Muthén

(2004). Latent variable analysis growth mixture modeling and related techniques for longitudinal data. In Kaplan

(Ed.), Handbook of quantitative methodology for the social sciences (pp. 345–368). Sage. Scirp.org. https://www.scirp.org/(S(lz5mqp453edsnp55rrgjct55.))/reference/referencespapers.aspx?referenceid=2253225

22.

Muthén

L. K.

Muthén

B. O.

(1998-2012). Mplus user’s guide statistical analysis with latent variables (7th ed.). Sage. Scirp.org. https://www.scirp.org/(S(i43dyn45teexjx455qlt3d2q))/reference/ReferencesPapers.aspx?ReferenceID=1736196

23.

Nagaoka

Farrington

C. A.

Ehrlich

S. B.

Heath

(2015). Foundation for young adult success: A developmental framework. University of Chicago Consortium on School Research.

24.

O'Conner

De Feyter

Carr

Luo

Romm

(2017). A review of the literature on social and emotional learning for students ages 3-8: Outcomes for different student populations and settings (part 4 of 4). Retrieved December 18, 2019, from. https://files.eric.ed.gov/fulltext/ED572724.pdf

25.

Romer

Ravitch

Tom

Merrell

Wesley

(2011). Gender differences in positive social–emotional functioning. Psychology in the Schools, 48(10), 958–970. https://doi.org/10.1002/pits.20604

26.

Romer

von der Embse

Eklund

Kilgus

Perales

Splett

J. W.

Sudlo

Wheeler

(2020). Best practices in social, emotional, and behavioral screening: An implementation guide version 2.0. Retrieved from. https://www.smhcollaborative.org/universalscreening

27.

Salavera

Usán

Teruel

(2019). The relationship of internalizing problems with emotional intelligence and social skills in secondary education students: Gender differences. Psicologia: Reflexão e Crítica, 32(1), 1–9. https://doi.org/10.1186/s41155-018-0115-y

28.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136

29.

Stiffler

M. C.

Dever

B. V.

(2015). Mental health screening at school: Instrumentation, implementation, and critical issues. Springer International Publishing/Springer Nature. https://doi.org/10.1007/978-3-319-19171-3

30.

von der Embse

Kim

Jenkins

Sanchez

Kilgus

S. P.

Eklund

(2021). Profiles of rater dis/agreement within universal screening in predicting distal outcomes. Journal of Psychopathology and Behavioral Assessment, 43(3), 632–645. https://doi.org/10.1007/s10862-021-09869-0

31.

Weissberg

R. P.

Kumpfer

K. L.

Seligman

M. E. P.

(2003). Prevention that works for children and youth: An introduction. American Psychologist, 58(6-7), 425–432. https://doi.org/10.1037/0003-066X.58.6-7.425

32.

Whitcomb

S. A.

Merrell

K. W.

(2013). Behavioral, social, and emotional assessment of children and adolescents (4th ed.). Routledge.

33.

Yoder

Dusenbury

Martinez-Black

Weissberg

R. P.

(2020). From insights to action: Redefining state efforts to support social and emotional learning CSI Emerging Insights Brief. Collaborative for Academic, Social, and Emotional Learning. https://files.eric.ed.gov/fulltext/ED617380.pdf