Psychometric Properties of the Child and Adolescent PsychProfiler: Self-Report Form

Existing Psychometric Properties and Limitations of the CAPP-SRF

Each of the three versions of the CAPP (i.e., CAPP-SRF, CAPP-TRF, and CAPP-PRF) have 126 items that are rated on a 6-point Likert-type scale (never = 0, rarely = 1, sometimes = 2, regularly = 3, often = 4, and very often = 5). One item (#39) is not used for screening purposes. Rather, it is for rater reliability purposes and therefore not assigned to any factor. Therefore, 125 items for clinical screening are used within the CAPP. For all versions of the CAPP, the screening scales and the DSM-5 (APA, 2013) disorders with the same names are highly congruent, thereby indicating strong face validity for the different screening scales in all three versions of the CAPP.

As stated in the PsychProfiler Manual, the original three versions of the CAPP underwent a thorough and lengthy psychometric analysis and were found to “be reliable and valid” (Langsford et al., 2014, p. 51). Although the PsychProfiler Manual contains information supporting inter-rater reliability, clinical calibration, the use of a six-point scale for the ratings of the items, and suitable readability for use by children and adolescents, it fails to include information on other psychometric properties commonly reported, such as factor structure, discriminant and criterion validity, and internal consistency. While there is substantial data available for the psychometric properties of the CAPP-PRF (e.g., Langsford et al., 2024) there is minimal psychometric data existing for the CAPP-SRF. One unpublished study by Sadeghi (2009), which used the Rasch measurement model to assess the CAPP-SRF, appears to be the only independent study that has investigated the SRF’s psychometric qualities. Sadeghi (2009) positively concluded that the CAPP-SRF has adequate internal consistency, reliability, and construct validity, however, there remains no published data on the CAPP-SRF factor structure which would no doubt be valuable for providing support for the CAPP-SRF’s use in clinical practice.

Although the teacher-report (CAPP-TRF), self-report (CAPP-SRF), and parent-report (CAPP-PRF) versions are exceedingly comparable to each other, it cannot simply be assumed that the CAPP-TRF and CAPP-SRF will also show comparable acceptable psychometric properties as that found for the CAPP-PRF (see Langsford et al., 2024) and it is therefore prudent that these be independently demonstrated. Providing confirmation of their factor structure (e.g., the presumed 17-factor structure) and other psychometric properties such as reliability and validity would be helpful for their continuing development and clinical utility.

Consequently, this study set out to further investigate the 17-factor structural model for the CAPP-SRF, and relatedly, its validity (discriminant, criterion, and concurrent) and reliability (alpha and omega coefficients).

The American Educational Research Association, American Psychological Association, and National Association for Measurement in Education (AERA, 2014) Standards for Educational and Psychological Testing provide criteria for the development and evaluation of tests and testing practices. They also furnished guidelines for assessing the validity of interpretations of test scores for the intended test uses. With regards to test validation, this document does not focus on distinct types of validity but rather focuses on different aspects of validity. As Hawkins et al. (2020) summarized, these include: evidence-based on test content (i.e., the relationship of the item themes, wording and format with the intended construct, including administration process); response processes (the cognitive processes and interpretation of items by respondents and users, as measured against the intended construct); internal structure (the extent to which item interrelationships conform to the intended construct); relations to other variables (the pattern of relationships of test scores to external variables as predicted by the intended construct); and consequences of testing (intended and unintended consequences, as can be traced to a source of invalidity such as construct under-representation or construct-irrelevant variance) as necessary for interpreting and using test scores.

For the CAPP-SRF, the evidence base for test content and response processes (i.e., use of a six-point scale for rating items, readability levels for children and adolescents, and clinical calibration) have already been established to some degree (see the PsychProfiler Manual, Langsford et al., 2014, p. 51). In comparison, there is limited evidence for internal structure, relationship to other variables, and outcomes of testing. Accordingly, these need to be established for reliable interpretation and use of the scores from the CAPP-SRF. Establishing these is the primary objective of this paper.

Aims of the Study

This present paper presents two studies. For Study 1, the focus was on internal structure, the overall aim being to use the Independent Clusters Model of Confirmatory Factor Analysis (ICM-CFA) to examine support for the 17-factor structure of the CAPP-SRF among adolescents from the general community. The model factors were SSD, LD, ADHDHI, ADHDI, SLDR, SLDW, SLDM, CD, ODD, ASD, GAD, SAD, PTSD, PDD, OCD, AN, and BN, with each scale consistent to their respective symptoms of the DSM-5 (APA, 2013) and disorder of the same name. However, model complexity (i.e., 125 items being rated on a 6-point Likert-type scale, and these items loading on 17 factors), the concern was the model may not show an admissible solution. Subsequently, it was decided that if this was not discovered, the original 17-factor model output would be re-examined, and the model would be revised accordingly. Furthermore, the model factors had to show reliability (alpha and omega coefficients), discriminant and criterion validity, and clarity (salience and significance of the loadings on their designated factors). However, as the data set used for the CFA study did not include variables that could be used for examining criterion validity, we used another data set that included variables suitable for this purpose. For ease of understanding, we refer to this other data set as Sample 2, and the sample used in the CFA as Sample 1.

The validity of the clinical scales in the CAPP-SRF in a clinic-referred group of adolescents was investigated in Study 2. Therefore, the evidence for relationships to other variables as per that specified in the aforementioned Standards (AERA, 2014), and noted by Hawkins et al. (2020), was the focus of Study 2.

As part of Study 2, adolescents completed the Beck Youth Inventories, Second Edition (BYI-2; Beck et al., 2005), Conners-3-SR (Conners, 2008), the Wechsler Intelligence Scale for Children, Fifth Edition (Wechsler, 2016a) and the Wechsler Individual Achievement Test, Third Edition (Wechsler, 2016b). Accordingly, Study 2 had the ability to examine evidence for criterion-referenced, convergent, and discriminant validity. When taken together, the two studies examined the internal validity of the CAPP-SRF, and the reliability (alpha and omega coefficients), and discriminant and criterion validities of the factors in the CAPP-SRF model. Based primarily on the research involving the CAPP-PRF (Langsford et al., 2024), we expected there would be support for the 17-factor model, with the factors in the model showing internal consistency, validity (discriminant and criterion), and clarity (items loading significantly and saliently on their respective factors).

Participants

Sample 1 comprised 790 adolescents aged 12 to 18 years (mean age 14.48 years, SD = 1.71) who completed an online or paper version of the CAPP-SRF. Of these, 410 were males (51.9.1%, mean age = 14.44 years, SD = 1.73 years) and 369 were females (46.7%, mean age = 14.55 years, SD = 1.70 years). No gender information was provided by 11 (1.4%) adolescents. There was no significant difference in age across boys and girls, t (df = 777) = 0.777, ns.

Sample 2 comprised 173 clinic-referred adolescents who completed a paper version of the CAPP-SRF at the clinic as part of their overall assessment. Their ages ranged from 12 to 17 years (mean age = 14.50 years, SD = 2.14 years). Of these, 112 (64.7%) were boys, (mean age = 14.40 years, SD = 2.30 years) and 61 (35.3%) were girls (mean age = 14.69 years, SD = 1.80 years). No significant age difference across these groups, t (df = 171) = .842, ns was found.

There were no inclusion or exclusion criteria for either of the samples, nor was it known if they had any existing diagnoses.

Measures

For both Samples 1 and 2, adolescents completed the CAPP-SRF (Langsford et al., 2014, which was described earlier. In addition, for Sample 2, the adolescents completed self-ratings of the Beck Youth Inventories, Second Edition (BYI-2; Beck et al., 2005) and the Conners 3-SR (Conners, 2008). All adolescents in Sample 2 were also administered the Wechsler Intelligence Scale for Children–Fifth Edition (WISC-V; Wechsler, 2016a) and the Wechsler Individual Achievement Test, Third Edition (WIAT-III; Wechsler, 2016b).

Procedure

Members of the public and professionals such as psychologists, psychiatrists, pediatricians, and general practitioners (who are the primary users) can access the PsychProfiler (including the CAPP) via its website (www.psychprofiler.com) for the online screening of DSM-5 disorders using any of the PsychProfiler forms.

Data for Study 1 was obtained from participants who completed a Self-report PsychProfiler on the website. These participants were provided with an option to give consent to their data being used for instrument validation and research purposes. To be included in Study 1 participants were required to have provided CAPP-SRF ratings and consent.

Sample 2 comprised adolescents who attended a clinic setting in Perth, Western Australia, to complete an ADHD, SLD, and/or ASD assessment at the request of a medical specialist, school, or parents).

Adolescent participants in this sample completed the CAPP-SRF, Conners 3-SR, and the self-report BYI-2 checklists. The adolescents were also administered the WISC-V and WIAT-III test batteries by a qualified examiner when at the clinic. At this time, permission for the use of adolescents’ de-identified data for future research and instrument validation purposes was sought from parents via a consent form. Only information from adolescents whose parents had provided informed consent were permitted inclusion in Sample 2.

Statistical Analysis

As previously mentioned, there were two different samples involved in the current study. Sample 1 comprised 790 self-ratings provided by adolescents on the CAPP-SRF and the primary objective of the analysis for this sample was to evaluate the fit of the 17-factor CAPP using Confirmatory Factor Analysis (CFA). First, the descriptive module in Jeffreys’ Amazing Statistics Program (JASP Team, 2023) version 0.16.6.0 statistical software was used to calculate the M and SD scores, and also the dispersion statistics of the 125 items of the CAPP. According to Brown (2006) if skewness is between –3 to +3 and kurtosis is between –10 to +10, then data can be considered to have normal univariate distribution. Furthermore, non-normality can be seen as problematic if ≥ 80% of responses are at one end of the scale (Streiner and Norman, (1995).

To analyze the 17-factor CAPP-SRF model, Mplus Version 7 (Muthén & Muthén, 1998–2012) was utilized. ML extraction was used because each item was rated on a 6-point Likert-type scale. Robust ML was not used as the data were considered not to have non-normality problems (details presented below). Global fit was assessed using the chi-square test, However, as known, the chi-square statistic can be inflated by large sample sizes, and therefore several approximate fit indices have been proposed. These include the relative chi-square (relative χ²) (sometimes called normed χ²) which is a ratio of the chi-square statistic to the respective degrees of freedom (χ²/df), Root Mean Squared Error of Approximation (RMSEA), Comparative Fit Index (CFI), Tucker Lewis Index (TLI), and the Standardized Root Mean Residual (SRMR).

Kline (2005) proposed that for model fit, chi-square, RMSEA, CFI, and SRMR be examined in combination. Although RMSEA, CFI and TLI are frequently used to evaluate model fit, these were not used for interpreting the findings in the present study because their values are derived from the chi-square value, and as such would also be compromised (Shi et al., 2019). On the other hand, the SRMR is not derived from the chi-square value (Pavlov et al., 2021). Considering all of this, the current study combined relative χ² and SRMR to evaluate model fit. For relative χ², acceptable values have ranged from less than 2 (Ullman, 2001) to less than 5 (Schumacker & Lomax, 2004) to be deemed acceptable. Hu and Bentler (1999) have proposed that for SRMR, values ≤ .80 = acceptable. Regardless of not relying on the CFI, TLI and RMSEA values for evaluating model fit, these values are reported. For these indices, Hu and Bentler (1999) have proposed that for RMSEA, values < 0.06 = good fit, <0.08 = acceptable fit, and > 0.08 to .10 = marginal fit. For CFI and TLI, values ≥.95 = good fit, and ≥ .90 = acceptable fit.

For model acceptance in this study, it was necessary for the loadings of the indicators in the model to be significant and salient (>.30; Field, 2013), and for the factors to reveal adequate discriminant validity (r <. .85; Brown, 2006), and acceptable reliability omega coefficients (Zinbarg et al., 2005). Although there are no universally accepted guidelines for interpreting omega coefficients, it has been proposed that omega values should mirror the same standards as alpha coefficients (Watkins, 2017). Guidelines for acceptability for alpha coefficients recommend .70 and above (Kline, 1998) and, therefore, for this study we used omega values of at least .70 as acceptable. Alpha reliability coefficients of the 17 factors are also reported on. Finally, as previously mentioned, model complexity was a potential concern regarding the ability of the 17-factor CAPP model to exhibit an admissible solution and therefore it was decided the model would undergo revision to achieve a model that had an acceptable solution, if this occurred.

Sample 2 consisted of 173 CAPP-SRF self-ratings from clinic-referred adolescents. For this sample, the data used were the total scores for the 17 scales in the CAPP-SRF (Langsford et al., 2014). The primary analysis for this sample examined the criterion validity of these 17 scales and SPSS was used for this. Pearson’s correlations of the factors in this model were examined with the total scores in the BDI-Y, BAI-Y, BANI-Y, BDBI-Y and BSCI-Y (BYI-2; Beck et al., 2005); DSM-5 symptom scales (DSM-5 ADHD Predominantly Inattentive Presentation Symptom Scale, DSM-5 ADHD Predominantly Hyperactive-Impulsive Presentation Symptom Scale, DSM-5 Conduct Disorder Symptom Scale, and the DSM-5 Oppositional Defiant Disorder Symptom Scale) of the self-report version of Conners-3 (Conners, 2008); and the WISC-V (Wechsler, 2016a) composite scores for VCI, VSI, FRI, WMI, PSI, and FSIQ. Correlations were also calculated for the CAPP-SRF model factors with the diagnosis of specific learning disorders for reading, written expression, and mathematics. These correlation values were squared to illustrate the proportional relationships between variables as they are empirically sound for inferring magnitude of effect (Baguley, 2009).

Although it is generally recommended that EFA be conducted initially to examine the expected causal connections between variables (i.e., factor structure of a measure), it is not mandatory. As has been noted by researchers (e.g., Bandalos & Finney, 2010; Hurley et al., 1997; Kline, 2011), when there is strong theory underlying the measurement model or when the relationships of the factors and related items are known, the application of CFA prior to EFA is justified. Given that the CAPP-SRF was developed to align fully with the DSM-5 disorders, it can be considered to have a well-developed underlying model, and a clear expectation of the patterns of factor loadings (i.e., the relevant items for each of the separate disorders loading on its specific disorders)—a requirement to conduct a CFA. In addition, the factor structure has some support from a previous study involving a parallel version of the CAPP (i.e., CAPP-Parent Report Form, Langsford et al., 2024). Taking these points into consideration, we decided to progress straight to the examination of the factor structure of the CAPP-SRF using CFA. However, we also decided that if the CFA failed to establish at least an acceptable model, EFA would then be used to explore potential reasons for the misfit, such as cross-loadings or problematic items.

Sample Size Requirements

To calculate the required sample size for Sample 1 Soper’s (2022) software was used. With the anticipated effect size of 0.3, power of 0.8, 17 latent variables, 123 observed variables, and alpha of 0.05, a minimum sample size of 218 was recommended to detect the effect, and a minimum sample size of 462 for model structure. Therefore, with N = 790, our sample size was more than adequate for the CFA in the current study.

Data Availability Statement

For access to the data used in this study contact the corresponding author.

Study 1

Study 2

The alpha reliability for the Conners 3‑SR DSM‑Symptom scales and BYI‑2 scales were α = .82–.93; α = .86–.94, respectively.

Practical Implications

Overall, our findings show the CAPP-SRF has satisfactory psychometric properties for use with adolescents. Furthermore, because the 17 different screening scales in the CAPP-SRF correspond to the DSM-5 symptom disorders of the same name, it can be assumed that the CAPP-SRF screening scales are suitable for adolescent screening of the DSM-5 clinical disorders with the same name. Similar findings have been reported by Langsford et al. (2024) for the CAPP-PRF screening scales.

Notwithstanding the argument here that the CAPP-SRF is a useful scale for screening DSM-5 adolescent psychopathologies, there are limitations to keep in mind when using this measure. First, the discriminant validity between GAD and PDD, AN and BN, LD and SSD, and LD and SLDR was not supported. However, this absence of discriminant validity could be seen as somewhat expected given the high association between these pairs of correlations (e.g., Achenbach & Rescorla, 2001; Beitchman et al., 1989; Fairburn & Bohn, 2005; Snowling et al., 2019).

Second, despite the findings providing support for the criterion validity of the CAPP-SRF ADHDI, ADHDHI, CD, ODD, GAD, PDD, ASD, SSD and SLDM, SLDR, SLDW and SLDM, criterion validity for SAD and OCD was not demonstrated as these disorders were not involved within the external measures used in Study 2. Therefore, when using the SSD and OCD scales a degree of caution is warranted. Furthermore, for the clinical use of the CAPP, information on its predictive validity must be proven

Implications for Comorbidity

Given the study found significant correlations between all the factors in CAPP-SRF it can be assumed that the factors have some degree of shared variance. As the CAPP-SRF screening scales are aligned to the DSM-5 disorders with the same names, it could be argued that our findings indicated considerable support for a high level of comorbidity between the disorders that are screened by the CAPP-SRF.

Implications for the Alternate Use and Revisions of the CAPP-SRF

Considering our discriminant validity findings, we found support for a revised 14-factor model in which the indicators for GAD with PDD were merged into a single mixed anxiety/depression disorder (GAD/DEP) factor, the indicators for AN and BN were merged into a single eating disorder (ED) factor, and the indicators for SLDR and SLDW were merged into a single specific learning disorder (SLD) factor. On the one hand, this highlights the need for revisions of these scales. Conversely, they suggest that at present the current CAPP-SRF is better viewed as comprising 14 scales comprising ADHDHI, ADHDI, ODD, CD, SLD, SLDM, ASD, LD, SSD, GAD/PDD, SAD, OCD, PTSD, and ED.

Limitations

It should be borne in mind there are limitations with the present research that should be acknowledged. First, we used a cross‑sectional design, that precludes causal inferences; (b) our convenience sampling may limit generalizability, particularly to culturally and linguistically‑diverse populations; (c) our reliance on self‑report may be influenced by insight and response styles; and (d) there was an absence of test–retest stability analyses. These should be addressed in future research, especially through longitudinal and multi‑informant studies.

The CAPP-SRF subscales screen 17 common DSM-5 disorders in children and adolescents and therefore might be considered to have a 17-factor structure. This structure, or any other structure, is yet to be fully tested or confirmed for the CAPP-SRF, and thus the primary aim of Study 1 was to examine the level of support for the 17-factor structure of the CAPP-SRF. The reliability (alpha and omega coefficients), and discriminant validity of the factors in this model were also examined. In Study 1, data from 790 adolescents provided support for a slightly modified 17-factor model. All items in this model loaded significantly and saliently on their respective designated factors. Furthermore, good discriminant validity was shown by almost all factors (apart from GAD and PDD, AN and BN, LD and SSD, and LD and SLDR). Omega coefficients of .787 to .895 and their alpha coefficients of .748 to .888 provided evidence of good internal consistency reliability

Summary and Concluding Remarks

In summary, this study indicates the CAPP-SRF possesses acceptable psychometric properties. Given that a significant feature of the CAPP is that it screens for criteria that mimic the diagnostic criteria of the DSM-5 (APA, 2013), it could be presumed that the CAPP-SRF is suitable for clinical and research use for the screening of DSM-5 disorders among adolescents. Despite the utility of adolescent self-report information for clinical assessment and diagnosis being questioned (Pelham et al., 2005), our results support researchers who advocate adolescent self-reports as a valuable source of information that can contribute to better clinical assessment and diagnosis (Achenbach, 1991; Gadow et al., 2002; Goodman, 1997; Jensen et al., 1999).

With reference to psychometric properties, the main focus of the current study was internal structure (i.e., item interrelationships and the extent they conform to the intended construct), and relationships to other variables (i.e., pattern of relationships of test scores to external variables as predicted by the intended construct). Apart from these, the AERA (2014) Standards for Educational and Psychological Testing, suggest examination of other properties (Hawkins et al., 2020), for interpreting and using test scores. These include test content (i.e., relationship of item themes, wording and format with the intended construct, including administration process), response processes (cognitive processes and interpretation of items by respondents, as measured against the intended construct), and intended and unintended consequences of testing (traced to a source of invalidity such as construct under-representation or construct-irrelevant variance) for interpreting and using test scores. The current study did not include these, and it may be that although we conducted a comprehensive evaluation of the psychometric properties the entire requirements specified in the AERA (2014) Standards for Educational and Psychological Testing were not fulfilled. Therefore, further research is necessary to evaluate psychometric properties while concurrently addressing the limitations noted above.

Overall, the present findings from adolescents’ self-reports support the 17-factor model. In addition, acceptable reliability was obtained for all the factors in this model, therefore, the findings indicate the CAPP-SRF possesses acceptable psychometric properties. The CAPP-SRF screens for criteria that represent the diagnostic criteria of the current DSM-5-TR (APA, 2022), and it can assist in obtaining more reliable future diagnoses, including differential diagnosis and information about comorbidity (Langsford et al., 2014). Therefore, the CAPP-SRF is deemed appropriate for clinical and research use for the screening of DSM-5-TR disorders among adolescents.