The Impact of Risk Assessment on Juvenile Justice Decision-Making and New Adjudication: An Analysis of Usage and Outcome

Abstract

Juvenile risk and needs assessments (JRNAs) have been the focus of extensive research in the academic literature. Prior studies have primarily focused on the risk-recidivism relationship and establishing predictive validity with juvenile populations. Less investigated is the use of risk and need assessment in practice, including how such tools are used to inform decision-making. This study uses record data encompassing 3,034 youth from a multi-state study to examine dispositional and treatment decisions associated with the Ohio Youth Assessment System (OYAS). Specifically, mediation analyses were conducted to evaluate how current practices align with underlying logic and theory regarding the role of assessments in juvenile justice. Findings reveal varied and complex relationships between assessment scores, case decisions, and recidivism. While risk was generally associated with recidivism, our results suggest juvenile risk and need assessments are inconsistently used to inform case management and placement decisions. Implications for practice and future research are also discussed.

Keywords

Juvenile Justice Mediation Risk Assessment Risk-Needs-Responsivity Ohio Youth Assessment System (OYAS)Youth Delinquency Disposition Treatment

Every year, close to one million youths are processed through the contemporary juvenile justice system (Sickmund et al., 2019), and likely encounter some type of structured risk assessment (Wachter, 2015). Juvenile risk and needs assessments (JRNAs) are intended to classify individuals into groups based on the likelihood that they will reoffend (e.g., low, moderate, and high risk) and identify needs to target in treatment to reduce individual risk of continued delinquency (Bonta & Andrews, 2017). The success of this strategy hinges on both the validity of the assessment tools used and the integration of assessment information into case management. Prior studies suggest that when implemented correctly, risk assessment tools can increase practitioner ability to accurately anticipate recidivism outcomes in comparison to clinical judgments (Andrews et al., 2006; Hilterman et al., 2014; Olver et al., 2009).

Research involving juveniles indicates that modern risk assessments are capable of differentiating between those who will reoffend and those who will not (Duwe & Rocque, 2017). However, the extent to which information collected in JRNAs is used to guide case decision-making remains an understudied area of research. Assessments can be viewed as the first step in an appropriate juvenile justice response with subsequent steps involving the use of JRNA information to better inform decisions about placement and supervision levels in the disposition process and route youths to appropriate treatment. While the breadth of existing risk–recidivism studies suggest that assessments are regularly completed, there is a shortage of studies that investigate their impact as theorized. That theory suggests that disposition and service receipt should serve as a conduit for the relationship between risk and needs assessment scores and recidivism. In this study, we utilize data from two states, including more than 40 agencies and 3000 individual cases, to consider the degree to which JRNAs are integrated in the juvenile justice process.

Literature Review

Aligning routine operations in juvenile justice agencies with evidence-based practices has primarily involved implementing the risk–need–responsivity (RNR) model (Bonta & Andrews, 2017). The model includes three principles that, when fully adhered to in practice, results in the greatest reductions in recidivism (Smith et al., 2009). These are risk, need, and responsivity.

The risk principle directs that those services should be based on the individuals’ risk of reoffending. The need principle states that the most effective interventions will address one’s criminogenic needs or factors, that have been empirically linked with criminal and delinquent behavior. The responsivity principle stresses that intervention and treatment strategies should be delivered using cognitive behavioral and social learning methods to maximize their effectiveness and tailored to individual traits and barriers (specific responsivity) (Bonta & Andrews, 2017). Research suggests that classifying and assigning offenders to programming following the principles outlined in the RNR model can reduce the risk of recidivism (Andrews & Bonta, 2010; Bonta & Andrews, 2017; Latessa et al., 2014); however, the RNR approach cannot be put into practice without the ability to determine risk and needs in a valid and reliable way (Andrews & Bonta, 2010; Duwe & Rocque, 2017). JRNAs are meant to serve this role in juvenile justice practice.

Predictive Validity and Variability in the Risk–Recidivism Relationship

As JRNAs have been more widely adopted, studies have investigated the strength of their predictive validity. In his meta-analysis of 42 predictive validity studies including 28 risk assessments, Schwalbe (2007) indicated that validated JRNAs predict youths’ recidivism significantly better than chance (AUC = 0.64) and have a moderate effect size (r = .25) similar to those found with meta-analyses of adult risk assessments (e.g., Gendreau et al., 1996). Though estimates of JRNA effectiveness can vary across different risk assessment tools, as well as across studies with the same instrument (e.g., Rossegger et al., 2013), this suggests that there is room for improvement in JRNA practice.

There are several possible reasons for these findings including the stage of the assessment and agency involved in each study (e.g., intake, probation agency, and residential facility), agency resources available, types of youth served (e.g., girls vs. boys) (Andrews et al., 2011), and fidelity in conducting assessments correctly. Additionally, there are informal processes that can compromise the integrity of the JRNA tools, such as gender, racial, and ethnic disparities in the juvenile justice system (Schwalbe et al., 2006). While risk assessments strive to ameliorate disparities across different groups, the informal selection processes that occur prior to administering the tool (e.g., police contacts, neighborhood bias, and school policies) may inherently exacerbate a youth’s criminogenic risk (Campbell et al., 2020).

The variation observed across individual cases, agencies, juvenile justice personnel, and states highlights the essential need to look more closely at the entire process underlying risk and needs assessment usage. Among the most important questions for implementation and usage concerns the degree to which the information gleaned from JRNAs affects juvenile justice decision-making as intended, and whether the results of those decisions contribute to better outcomes. While adopting valid and reliable JRNAs might be a first step in more systematic juvenile justice system processing, effective implementation of this approach in practice is necessary for positive changes in case management (Latessa & Lovins, 2010; Vincent et al., 2012b). Risk assessments may be completed but not used to guide decision-making effectively (Shook & Sarri, 2007), or in some cases, used incorrectly. As a result, the full benefits of including JRNAs in decision-making and case planning cannot be realized (Vincent et al., 2012b) prompting ineffective juvenile justice intervention. It is, therefore, imperative to better understand usage of JRNAs to guide decision-making in the juvenile justice system (Peterson-Badali et al., 2015; Vincent et al., 2012a).

Juvenile Risk and Needs Assessments Implementation and Impact

Juvenile risk and needs assessments are frequently part of the implementation of evidence-based practices, having been touted as potentially improving case decision-making, youth outcomes, and recidivism rates (Nelson & Vincent, 2018). The results of JRNAs can be viewed as a “cornerstone of treatment” (Sullivan et al., 2019), with the potential to exert great influence on a youth’s path through the juvenile justice system due to its impact on case management—ranging from intake to case disposition and placement to treatment. The JRNA process illustrates how assessments can inform juvenile justice response in ways that reduce the risk for recidivism and potentially affect important developmental outcomes (National Research Council, 2013; Sullivan, 2019). Juvenile risk and needs assessments should also be administered to youths to inform key juvenile-justice decision points. This process starts with intake and detention decisions (Mears, 2012; Steinhart, 2006), and proceeds through later stages of the system (Baglivio et al., 2015). In addition to detailing background and personal characteristics (which could be useful in highlighting specific responsivity factors), results from each assessment should be used to gauge risk of re-offense and areas of criminogenic need (Duwe & Rocque, 2017). This information should then be used to match youths to appropriate services and supervision levels, as well as inform dispositional and case-management decisions (Andrews & Bonta, 2010; Vieira et al., 2009). Finally, assuming each youth completes the assigned supervision requirements or is placed appropriately, we should expect to see greater reductions in recidivism than if a systematic assessment was not used.

Contemporary research has examined some aspects of how agencies integrate JRNAs into their operations and the impacts of those efforts. For example, Viera and colleagues (2009) studied whether practitioners were matching youths to services appropriate for their criminogenic needs and responsivity factors as determined by JRNA, and if doing so led to reductions in risk and recidivism. Youths with fewer criminogenic needs addressed re-offended earlier and acquired a greater number of new convictions (Viera et al., 2009). Similarly, Luong and Wormith (2011) found that some agencies were using JRNAs to determine supervision levels and identify appropriate services, and that following the need principle was related to reductions in recidivism.

Vincent et al. (2012a; 2012b) also examined JRNA implementation and the RNR principles in practice. In their first study, Vincent and colleagues (2012a) found that following the implementation of a common JRNA residential placements, supervision, and total numbers of services offered decreased, while services offered to high-risk youths increased. Moreover, the greatest adherence to proper JRNA use was achieved after new case management forms were implemented and practitioners were trained on how to specifically use JRNAs to inform their decision-making (Vincent et al., 2012a). The second study by Vincent et al., (2012b) investigated changes in the attitudes and case management decisions of juvenile probation officers before and after the introduction of a JRNA tool. Findings indicated that implementation of the tool increased attention to youths’ risk and needs and the likelihood that officers would assign youths to services and supervision levels that aligned with their criminogenic needs.

More recently, research examining structured decision-making in the Florida Department of Juvenile Justice (FDJJ) reaffirmed the importance of matching youths’ risk levels to appropriate services. Baglivio and colleagues (2014) found that placements above or below the recommended range of graduated sanctions had increased rates of recidivism. Deviations were more pronounced for high-risk youths for whom the lowest adherence to the recommended sanctions was observed. Studies with Florida youth have also investigated the impact of treatment dosage in combination with matching. The researchers examined the effects of matching youth to services based on their criminogenic needs, and ensuring treatment met or exceeded empirically developed dosage and duration targets for those services (Baglivio et al., 2018). One key result included that among youth who received treatment that targeted their strongest criminogenic need, and who met or exceeded treatment dosage and duration requirements, experienced significantly lower recidivism upon release from residential placement. This combination also led to significant within-individual reductions in key areas of dynamic risk post-treatment. A third study with Florida youth further demonstrated that these findings held even among youth with extensive trauma histories (Baglivio et al., 2021).

Finally, research by Peterson-Badali et al. (2015) examined the implementation of JRNAs to identify usage gaps that could undermine efforts to properly use the resultant information. This study found that practitioners matched youths to services that addressed their criminogenic needs less than 43% of the time across all seven criminogenic domains (and in four domains, less than 25% of the time). Results from this study also confirmed that youths who were matched to appropriate services were significantly less likely to re-offend than youths who were not matched to appropriate services. Further, a 2016 study by Vincent et al. examined use of risk assessment in six juvenile probation sites and found that implementing the JRNA led to significant changes in all study sites except for one. Importantly, the degree and nature of the impact of applying the RNR approach was a function of each sites’ implementation quality rather than the type of risk assessment utilized. Sites with well-implemented practices were able to conserve resources, reduce costs, and provide services to a higher number of youths without jeopardizing the public (Vincent et al., 2016).

Current Study

This study extends the existing body of literature by estimating a series of mediation models to investigate the degree that information from JRNAs affects juvenile justice case decision-making and recidivism. We evaluated if measures of recidivism risk captured by the Ohio Youth Assessment System (OYAS) impact case dispositions and treatment assignments, leading to reductions in new adjudications. To accomplish this goal, we examined two primary research questions. First, we considered whether key disposition types mediate the relationship between the results of the OYAS risk assessment (i.e., risk level) and recidivism. Specifically, (1) Does matching youth to dispositions based on assessment results mediate the relationship between risk and recidivism? Second, we tested whether assessment domain scores drive youths’ assignment to relevant treatment types (e.g., substance use and cognitive behavioral therapy), and whether this treatment assignment affects recidivism. In other words, we ask: (2) Does matching to treatment type based on assessment information mediate the relationship between risk and recidivism? If JRNA information is being used as intended with our study sample, we expect that disposition and treatment referral should mediate the risk–recidivism relationship. Thus, we hypothesize that youth who receive dispositions and/or treatment types that align with their OYAS assessment results (i.e., risk and domain scores) will be less likely to recidivate, as theorized by the RNR model. Our research models relevant juvenile justice processes as potential mediators to better understand how JRNA usage works in practice, and if it is in line with theory—or not.

Methods

Sample

Data for this study come from juvenile justice systems in two Midwestern States and include 44 agencies (30 from State 1 and 14 from State 2). The sample is comprised of youths who were assessed using the OYAS in 2014 or 2015 pre- or post-adjudication. The multi-stage sampling process was developed based on assessment type (i.e., diversion, detention, disposition, residential, and re-entry) and county use. To ensure adequate representation of youths at varying stages of the juvenile justice process, a large sample of youths was selected through stratified random sampling techniques. Youths were eligible for selection if they were assessed in 2014 or 2015. The first stage of sampling involved selecting counties from which youth would be selected. Due to the large number of counties, the primary sample units were stratified based on assessment usage (e.g., low, moderate, or high) and were randomly selected from each stratum to participate in the study. Youths within the selected counties were then stratified and selected based on the type of assessment used. The total sample encompassed 3772 unique youths; however, this study excludes youths who were assessed using the diversion or detention screening tools due to the limited information these tools capture compared to others. More information about existing OYAS tool types is included below.

After excluding diversion and detention youths, the final sample included a total of 3039 unique individuals: 1656 (55%) youths from State 1 and 1383 (45%) youths from State 2. Data were collected from juvenile justice agencies with county- and state-wide jurisdictions. Age, race/ethnicity, and gender for each youth were obtained via the records from each research site. A breakdown of the sample’s demographic characteristics is outlined in Table 1. Most youths in the analytic sample were male (81%), non-white (55%), and ranged from 10 to 20 years old (M = 16.0; SD = 1.51).¹

Table 1.

Sample Descriptives.

	State 1	State 2	Total Sample
Characteristic	% (n)	% (n)	% (n)
Sex (1 = female)	17.6 (292)	20.0 (276)	18.7 (568)
Race/ethnicity (1 = non-white)	61.6 (1019)	47.2 (652)	55.1 (1675)
Felony focal offense (1 = yes)	66.7 (872)	54.3 (750)	60.3 (1622)
Offense history (1 = present)	83.2 (1376)	79.4 (1097)	81.5 (2473)
Recidivism (1 = yes)	26.0 (386)	21.1 (291)	23.7 (678)
	M(SD)	M(SD)	M(SD)
Age	16.3 (1.6)	15.8 (1.4)	16.0 (1.5)
Disposition/treatment received	% (n)	% (n)	% (n)
Most serious disposition = state commitment/residential	51.9 (672)	71.1 (946)	61.8 (1618)
Most serious disposition = probation	37.0 (479)	14.1 (187)	25.37 (666)
CBT^a (1 = yes)	24.7 (353)	7.4 (53)	18.9 (406)
Family therapy (1 = yes)	17.6 (252)	14.4 (103)	16.6 (355)
Education or employment services (1 = yes)	2.8 (46)	5.3 (73)	3.9 (119)
SA^b or MH^c treatment (1 = yes)	37.0 (530)	52.4 (373)	42.1 (903)
OYAS scores	% (n)	% (n)	% (n)
Low risk	41.0 (677)	39.4 (544)	40.2 (1221)
Moderate risk	36.5 (603)	47.4 (654)	41.4 (1257)
High or very high risk	22.4 (371)	13.3 (183)	18.3 (554)
	M(SD)	M(SD)	M(SD)
Total score	13.7 (6.9)	14.1 (5.7)	13.9 (6.4)
Juvenile justice history	2.1 (1.5)	2.0 (1.5)	2.1 (1.5)
Family and living arrangements	1.3 (1.3)	1.2 (1.2)	1.2 (1.3)
Peers and social support networks	2.8 (2.0)	3.0 (1.8)	2.9 (1.9)
Education and employment	1.4 (1.1)	1.5 (1.0)	1.4 (1.1)
Pro-social skills	2.0 (1.5)	1.9 (1.3)	2.0 (1.4)
SA^b, MH^c, and personality	2.6 (1.9)	2.9 (2.0)	2.7 (2.0)
Values, beliefs, and attitudes	1.7 (1.6)	1.5 (1.4)	1.7 (1.5)

^a Cognitive behavioral therapy.

^b Substance abuse.

^c Mental health.

Measures

The Ohio Youth Assessment System (OYAS)

The main independent variables come from the OYAS, which was developed by Latessa et al. (2009), and includes five tools that can be used to assess risk of recidivism across each stage of the juvenile justice process. Studies show that the various tools that comprise the OYAS predict recidivism relatively well at various stages of the juvenile justice system (Latessa et al., 2009; McCafferty, 2013) and among diverse populations (McCafferty, 2018). Each tool is administered via interview by trained practitioners (Lovins & Latessa, 2013). A specific set of scoring criteria is then used to assess each item, which can vary depending on the tool used. Items are generally scored as (0) no risk and (1) risk present, with a small number scored as (0) no risk, (1) some risk, and (2) definite risk to capture varying degrees of risk associated with those items. The assessor may use collateral information, such as parental reports or available court or probation records, as needed in the process of completing and scoring the OYAS as well.

Depending on the type of OYAS tool used, the total number of items and the breadth of criminogenic factors captured can vary. The sample in this study is restricted to individuals assessed with the disposition (32 items), residential (33 items), and re-entry (41 items) tools because they are the most comprehensive and all three include seven criminogenic domains categorized by topic: Juvenile Justice History; Family; Peers; Education/Employment; Prosocial Skills; Substance Abuse, Mental Health, and Personality²; and Values, Beliefs, and Attitudes. The disposition assessment process is completed prior to the juvenile court making decisions about sanctions and/or treatment referrals; the residential tool is typically administered upon youth intake into a residential facility; and reentry assessments take place just prior to a youth being released from a residential facility.

Ohio Youth Assessment System tools determine total risk of recidivism by summing each item score to obtain a total, with higher numbers indicating increased risk for recidivism. Youths are also assigned a risk level of low-, moderate-, or high-risk based on their total scores (Lovins & Latessa, 2013).³ Additionally, a total score for each OYAS domain is calculated, which is designed to identify criminogenic need areas. Depending on the specific research question, either OYAS risk levels or domain scores were used as independent variables in this study. Occasionally practitioners can override the OYAS risk level received; when an override occurred the final risk level was used.⁴ Table 1 summarizes the distribution of risk levels and scores for both states in the sample.

Offense History

Prior offense history was ascertained from items on the OYAS offense history domain, which includes indicators such as “any prior offenses” and “any prior probation.” For analysis, this measure was collapsed into a dummy variable with (0) no offense history and (1) offense history present. More than three-quarters (81.5%) of the final sample had an offense history.

Focal Offense Category

The offense associated with a youth’s selected assessment was deemed the “focal offense.” Information regarding the offense code, type, level, and category listed on the focal offense petition were collected. To illustrate the seriousness of each focal offense, offense category was collapsed into (1) felony and (0) non-felony (misdemeanor, status, or other offense classification less serious than a felony). In cases where a youth was associated with more than one type of offense, the most serious offense type was coded. About 60% of youths in the final sample had a felony focal offense.

Disposition Type

Disposition type captures the outcome of each youth’s disposition hearing for the focal case. Since this study focused on juvenile justice case processing, youth who were transferred to adult court (n = 5) were not included in our analyses. Specifically, disposition types included secure commitment, non-secure residential placement, house arrest or electronic monitoring, intensive and standard probation, diversion, restitution or fines, and dismissal or termination. Given the categorical nature of the variable, disposition type was collapsed into two binary indicators, which were then used as mediators in the multivariate statistical models that were estimated in this study. The first dichotomous mediator compared youths who received the most severe dispositions in our dataset, commitment to a state or residential facility (1), with youth who received a less serious disposition (0). The second binary disposition mediator compared youths who received intensive or standard probation (1), to youths with a less serious case outcome (0) based on scenarios where youths had a community-based sanction (e.g., diversion based on community service involvement) or their case was dismissed following formal system processing. If there was more than one disposition for the youth’s focal case, the most serious one was recorded and used for analysis.

Treatment Type

Treatment type refers to therapeutic-oriented case outcomes (e.g., cognitive behavioral therapy) and captures the category of services and/or programming assigned to each youth. Agencies were asked to specify the treatment provider, program name, and type of treatment for each youth in the sample. Type of treatment included counseling, educational and employment services, family services, sex offender treatment, religious services, substance abuse, anger management, and treatment for antisocial attitudes. Program names and providers were used to provide supplementary information whenever the treatment type was not clear. In cases where youths were assigned more than one treatment type, each category was recorded separately.

Four binary categories of treatment referral were examined in our analyses: cognitive behavioral therapy, education and/or employment services, family services, and mental health or substance abuse treatment. Programs were classified into these categories based on the content or topic of the services they provided, or the type of behavior or issue they targeted. A score of (0) indicated that the youth did not receive the type of treatment specified, while (1) meant they did receive the specified treatment. These specific treatment categories were selected for our analyses because they most closely align with the Values, Beliefs, and Attitudes; Education and Employment; Substance Abuse, Mental Health, and Personality; and Family domains represented on the OYAS. Therefore, referrals to services in these categories should best capture whether OYAS information is being used to guide treatment placement.

Official Recidivism

Official recidivism is the main dependent variable in this study. It was scored dichotomously, with (0) indicating no recidivism, and (1) indicating recidivism. Recidivism was defined as an adjudication for a new charge following each youth’s assessment date associated with their focal offense. The earliest assessments in the sample were conducted in 2014, and the latest recidivism occurred in 2018. Therefore, up to 4 years of new adjudication was captured for youths. For youth in residential placement, new offenses must also have occurred post-release to count as recidivism. A re-offense was recorded for approximately 24% of the total sample (n = 678). Most justice-involved youths with a new adjudication were male (81%, n = 545), non-white (57%, n = 387), had a non-felony focal offense (51%, n = 282), had offense history present (84%, n = 570), and were 15.4 years old on average (SD = 1.5). Table 1 illustrates the distribution of recidivism across each state in the sample.

Only re-offenses with juvenile court jurisdiction were captured; thus, adult offenses are not included. It is possible that older youths could have been classified as “non-recidivists” in our sample because they “aged out” of juvenile jurisdiction before re-offending. To address this, a control variable was added to our analyses denoting youths who were 16 years or older (the average age of the sample) at the time of their OYAS assessment. We selected this cut-off because individuals who were 16 years or older at the time of their assessment would have aged out of adult court within 2 years,⁵ and thus were most likely to have committed offenses in adult court that were not captured in our data.

Analytic Plan

We use controlled mediation analyses with bootstrapped standard errors to assess the assumed relationship from JRNA to case management decisions to outcome (Hayes, 2013; Vanderweele, 2015). These analyses were carried out using the paramed program (Valeri & Vanderweele, 2013) available in Stata, as well as Mplus software used for path modeling (Muthén & Muthén, 1996–2017). When examining disposition and placement type as a mediator, risk levels were utilized to operationalize risk for recidivism. This resulted in binary mediator variables, therefore paramed was used. Domain scores were used as the independent variable when treatment was examined as a mediator. Consequently, Mplus was used for the treatment mediation analyses because the domain scores included as independent variables are continuous.

This dual strategy was used because it allowed us to effectively model indirect relationships between risk/needs and recidivism while controlling for potential unmeasured effects on the risk and disposition/treatment relationships that could otherwise confound results. These modeling strategies also allow us to handle interaction assumptions in mediation analysis that cannot be dealt with in standard multivariate regression processes or other approaches to mediation (c.f., Baron & Kenny, 1986; Vanderweele, 2015). The interaction term is tested in each case to determine whether the relationship between the mediator and outcome is stronger or weaker depending on the initial risk level or domain risk score.

The control variables used in each equation vary somewhat depending on the models. The core control variables include age, sex (1 = female), race (1 = non-white), felony focal offense (1 = yes), and a dummy variable reflecting the state (reference = State 1). In analyses examining treatment type as a mediator, offense history (1 = present) was also included as a control variable to better account for the nature of static versus dynamic criminogenic factors that may also factor into decisions about disposition that could affect the likelihood of receipt of certain treatment (i.e., residential placement vs. community-based services). We utilized two binary indicators as controls for assessment tool type in analyses where the sample composition meant that youth could have been assessed with the disposition, residential, or reentry tool (e.g., when they were placed in residential facility). Finally, a control variable was added denoting youths who were 16 years or older at the time of their assessment for their focal offenses to account for youths who may have aged out of the juvenile system prior to re-offending.

Results

Assessment, Disposition Decisions, and Recidivism

State Commitment

Results of the mediation process model for risk level and recidivism via state commitment are presented in Table 2. There was no significant difference between moderate-to-high risk youths and low-risk youths in terms of the likelihood of residential commitment. This suggests that JRNA information is not associated with state commitment placement decisions. This may be due to the relatively few low-risk cases committed to these facilities (n = 41). Older youths, males, and those with felony offenses were significantly more likely to be committed, as were youths from State 2.

Table 2.

Results of Mediation Process Model for State Commitment.

	Mod/High versus Low Risk and State Commitment	High versus Mod Risk and State Commitment
	(n = 2318)	(n = 1463)
New adjudication on…	b (95% CI)	b (95% CI)
OYAS risk level	0.70*** (0.37, 1.03)	0.16 (−0.13, 0.46)
State commitment	−0.78** (−1.23, −0.32)	−1.12*** (−1.47, −0.76)
Risk level × state commitment	−0.40 (−0.85, 0.07)	-
Age	−0.35*** (−0.48, −0.21)	−0.37*** (−0.54, −0.20)
Sex (1 = female)	−0.11 (−0.39, 0.17)	−0.11 (−0.48, 0.25)
Race/ethnicity (1 = non-white)	0.12 (−0.09, 0.34)	0.04 (−0.22, 0.31)
Felony offense	−0.42*** (−0.65, −0.19)	−0.41** (−0.70, −0.13)
State 2	−0.14 (−0.37, 0.10)	−0.10 (−0.39, 0.20)
Any juvenile justice history	0.29 (−0.02, 0.60)	0.18 (−0.24, 0.60)
Aged out of juvenile system	0.03 (−0.36, 0.40)	0.004 (−0.47, 0.48)
Residential tool	−0.16 (−0.49, 0.17)	−0.27 (−0.66, 0.11)
Reentry tool	0.02 (−0.33, 0.37)	−0.07 (−0.49, 0.36)
Disposition on...
OYAS risk level	0.06 (−0.23, 0.35)	0.44* (0.09, 0.80)
Age	0.25** (0.09, 0.40)	0.23* (0.03, 0.43)
Sex (1 = female)	−0.65*** (−0.99, −0.31)	−0.81 (−1.24, −0.38)
Race/ethnicity (1 = non-white)	0.02 (−0.24, 0.29)	−0.04 (−0.36, 0.29)
Felony offense	1.04*** (0.76, 1.33)	1.09*** (0.73, 1.45)
State 2	2.07*** (1.74, 2.39)	2.34*** (1.92, 2.76)
Any juvenile justice history	1.06*** (0.71, 1.41)	1.01*** (0.52, 1.50)
Aged out of juvenile system	−0.28 (−0.75, 0.19)	−0.46 (−1.04, 0.12)
Residential tool	3.78*** (3.41, 4.13)	3.64*** (3.22, 4.08)
Reentry tool	3.75*** (3.39, 4.12)	3.60*** (3.11, 4.09)
Risk level to recidivism	Odds ratio (95% bootstrapped CI)	Odds ratio (95% bootstrapped CI)
Controlled direct effect	2.02*** (1.46, 2.79)	1.18 (0.88, 1.58)
Natural direct effect	1.72*** (1.34, 2.19)	1.18 (0.88, 1.58)
Natural indirect effect (thru disposition)	0.98 (0.90, 1.07)	0.89 (0.89, 1.11)
Marginal total effect	1.69*** (1.31, 2.17)	1.05 (0.73, 1.51)

* p<0.05, ** p<0.01, *** p<0.001.

Risk level (b = 0.70, p < 0.001) and state commitment were also significantly associated with new adjudication (b = −0.78, p < 0.01). Those in the higher risk group had a greater likelihood of recidivism while those who were committed to state facilities had a reduced likelihood of recidivism. Further, the interaction between risk level and the mediator suggests that among those committed to a state facility, youths assessed as moderate or high risk had lower chances of recidivism compared to low-risk youths. Although this estimate was nonsignificant, it was included in the model due to its relative size. The indirect effect of risk level on recidivism through state commitment was not statistically significant (OR = 0.98), and the overall decomposition of effects indicates only that the direct and total effects were statistically significant. These results suggest a lack of an indirect pathway between risk level and recidivism via state commitment.

The relationship between state commitment and new adjudication held when youths designated as high- and moderate-risk were compared, as well (b = −1.12, p < 0.001). Beyond that the results differed when this model was re-estimated comparing only those youths assessed as moderate- or high-risk (i.e., excluding low-risk youths). Ohio Youth Assessment System risk level significantly predicted state commitment or new adjudication directly in that model (b = 0.44, p < .05). No significant indirect effects were detected from risk to recidivism through state commitment. These results were obtained even though many of the other individual estimates remained the same as discussed above (see Table 2). It does, however, illustrate that appropriate dispositional placement based on risk level is related to recidivism, and that appropriately accounting for the risk level—even if it results in custody—can have comparative advantages in reducing recidivism.

Standard Probation versus Less Restrictive Sanctions

The mediation process model for risk level and recidivism via probation (intensive or standard) compared to less severe sanctions (e.g., diversion, restitution, fines, and community service) is shown in Table 3. Focusing first in the left-hand column, moderate and high-risk youths had significantly greater log odds of recidivism (b = 1.03, p < 0.01) compared to low-risk youths. Individuals assigned to probation were not significantly more likely to recidivate than those assigned to less serious dispositions, however (b = −0.04). This suggests that a youth’s assignment to community supervision or less serious community-based sanctions did not directly impact their chance of new adjudication. Compared to the low-risk group, youths assessed at moderate- or high-risk did not have a significantly higher likelihood of being assigned to probation rather than less intensive sanctions. Further exploration of this result was conducted by restricting the model to compare low-versus moderate-risk youths only; however, no statistically significant relationship between risk level and assignment to standard probation was detected (b = 0.44).

Table 3.

Results of Mediation Process Model for Probation versus Less Severe Sanctions.

	Mod/High versus Low Risk and Probation versus Less Severe (n = 416)	Mod versus Low Risk and Probation versus Less Severe (n = 378)
New adjudication on…	b (95% CI)	b (95% CI)
OYAS risk level	1.03** (0.58, 1.50)	1.10*** (0.62, 1.59)
Probation	−0.04 (−0.69, 0.63)	−0.04 (−0.71, 0.63)
Age	−0.21 (−0.49, 0.05)	−0.25 (−0.53, 0.03)
Sex (1 = female)	−0.11 (−0.62, 0.41)	−0.15 (−0.74, 0.19)
Race/ethnicity (1 = non-white)	0.08 (−0.38, 0.54)	0.25 (−0.23, 0.73)
Felony offense	−0.79** (−1.29, −0.27)	−0.93* (−1.48, −0.38)
State 2	−0.70** (−1.17, −0.23)	−0.65* (−1.14, −0.16)
Any juvenile justice history	0.63** (0.27, 1.20)	0.59* (0.06, 1.12)
Aged out of juvenile system	0.07 (−0.76, 0.90)	0.15 (−0.73, 1.04)
Disposition on…
OYAS risk level	0.59 (−0.03, 1.22)	0.44 (−0.19, 1.08)
Age	0.19 (−0.15, 0.53)	0.22 (−0.13, 0.56)
Sex (1 = female)	−0.20 (−0.60, 0.54)	−0.73 (−0.87, 0.40)
Race/ethnicity (1 = non-white)	−0.67* (−1.26, −0.08)	−0.65* (−1.24, −0.06)
Felony offense	0.42 (−0.24, 1.08)	0.34 (−0.33, 1.00)
State 2	−0.03 (−0.62, 0.57)	−0.02 (−0.62, 0.58)
Any juvenile justice history	0.19 (−0.44, 0.83)	0.22 (−0.43, 0.86)
Aged out of juvenile system	−0.49 (−1.58, 0.59)	−0.52 (−1.62, 0.59)
Risk level to recidivism	Odds ratio (95% bootstrapped CI)	Odds ratio (95% bootstrapped CI)
Controlled direct effect	2.83** (1.78, 4.48)	3.01** (1.72, 4.96)
Natural direct effect	2.83*** (1.78, 4.48)	3.01*** (1.72, 4.96)
Natural indirect effect (thru disposition)	0.99 (0.91, 1.04)	0.99 (0.93, 1.04)
Marginal total effect	2.82** (1.41, 5.65)	3.01** (1.76, 4.91)

* p < 0.05, ** p < 0.01, *** p < 0.001.

Consistent with risk assessment logic, the direct effect of OYAS risk level was significantly predictive of recidivism for moderate/high youths as compared to low-risk youths. The natural indirect effect of probation versus less serious sanctions on the risk level and recidivism relationship was also statistically significant (OR = 2.83, p < 0.001), suggesting elevated likelihood of a new adjudication. The interaction term between risk level and probation was also nonsignificant. The restricted model comparing low-versus moderate-risk youths produced similar estimates for the direct (OR = 3.01, p < 0.001) effect of probation on the risk–recidivism relationship; however, the indirect effect of this relationship operating through probation was not statistically significant (OR = 0.99). Though they support the conclusion that JRNAs can effectively predict recidivism, these results are inconsistent with respect to the indirect pathway between risk level and recidivism via standard probation or less severe sanctions. This is counter to expectations relevant to how JRNA is intended to operate, which would reveal more consistent indirect relationships between the result of the JRNA and recidivism via juvenile justice dispositions.

Assessment, Treatment Decisions, and Recidivism

We focused on four combinations of treatment type and OYAS subdomains in these analyses and estimated models treating relevant needs scores as independent variables and an indicator of treatment receipt as the mediator. We also estimated a version of each model that included an interaction between the independent variable and the treatment variable to ensure that the indirect effect estimates were not biased by its omission (see Vanderweele, 2015). The interactions were statistically significant or substantively noticeable in three of the models, so those were included in the final estimates presented here. The distribution of youth referred to each type of treatment is shown in Table 1. Few youths in the total sample were referred to CBT (19%), family services (17%), or Education/Employment services (4%), especially compared to substance abuse or mental health treatment (42%). This likely reflects an undervaluing or lack of resources for these types of interventions.

Cognitive Behavioral Treatment (CBT)

The results of the treatment mediation models examining the relationships between criminogenic needs, receiving CBT, and recidivism are shown in Table 4. The first column includes cases that were referred to community-based placements or diverted. The second column refers to the subsample of cases in residential facilities. The Values, Beliefs, and Attitudes and Prosocial Skills domain scores were included as predictors of receiving CBT and recidivism, as higher scores in these domains suggest that treatment targeting cognition may be appropriate. The Juvenile Justice History domain score was included in the treatment models as a control variable. Referral to CBT did not have a significant relationship with new adjudication for community-based cases. The Juvenile Justice History score had a statistically significant and small relationship with recidivism in the community-based sample suggesting that there was a slight increase in the likelihood of recidivism (b = 0.11, p < .01) for a one unit increase on that Juvenile Justice History score.⁶ One of the OYAS domain variables, Values, Beliefs, and Attitudes had a relationship to whether youths were referred to CBT in the community-based sample. Those with more pro-criminal values, beliefs, and attitudes were slightly more likely to receive CBT (b = 0.09, p < .01). The indirect effects estimate for the Values, Beliefs, and Attitudes and Prosocial Skills domain did not yield any significant effects on recidivism in either of the samples.⁷ There is a positive and statistically significant interaction (b = 0.39, p < .001) between Prosocial Skills risk and CBT for the sample of residential cases.

Table 4.

Results of Mediation Process Model for Cognitive Behavioral Treatment.

	Community-Based Cases (n = 959)	Residential Cases (n = 1531)
New adjudication on…	b (se)	b (se)
Cognitive behavioral treatment		0.04 (0.38)
Values, Beliefs, and Attitudes Domain Score	0.04 (0.04)	0.06 (0.30)
Prosocial Skills Domain Score	0.06 (0.04)	−0.03 (0.23)
Juvenile Justice History Domain Score	0.11 (0.04)**	0.01 (0.09)
Treatment × Values, Beliefs, Attitudes Score	—	−0.14 (0.08)
Treatment × Prosocial Skills Score	—	0.39 (0.08)***
Age	−0.20 (0.06)**	−0.02 (0.45)
Sex (1 = female)	−0.05 (0.12)	0.20 (1.60)
Race/ethnicity (1 = non-white)	0.04 (0.10)	0.06 (0.59)
Violent offense	−0.18 (0.10)	−0.26 (0.23)
State 2	−0.35 (0.12)*	0.85 (3.01)
Aged out of juvenile system	0.08 (0.18)	−0.02 (0.45)
Cognitive behavioral treatment on…	b (se)	b (se)
Values, Beliefs, and Attitudes Domain Score	0.09 (0.04)*	−0.25 (0.73)
Prosocial Skills Domain Score	0.01 (0.04)	−0.20 (0.57)
Juvenile Justice History Domain Score	0.02 (0.04)	0.15 (0.25)
Age	−0.07 (0.05)	−0.07 (0.24)
Sex (1 = female)	−0.37 (0.14)*	−0.47 (5.73)
Race/ethnicity (1 = non-white)	0.17 (0.10)	0.74 (0.86)
Violent offense	0.08 (0.10)	−0.32 (0.61)
State 2	−0.57 (0.13)***	−11.6 (7.45)
Aged out of juvenile system	0.07 (0.18)	−0.01 (1.43)
Values, beliefs, and attitudes to recidivism	Odds ratio (95% bootstrapped CI)	Odds ratio (95% bootstrapped CI)
Natural direct effect	1.13 (0.91–1.44)	1.13 (0.99–1.78)
Natural indirect effect	1.001 (0.98–1.02)	0.99 (0.48–1.001)
Total effect	1.13 (0.91–1.45)	1.11 (0.99–1.29)
Prosocial skills to recidivism	Odds ratio (95% bootstrapped CI)	Odds ratio (95% bootstrapped CI)
Natural direct effect	1.18 (0.96–1.54)	0.93 (0.74–1.20)
Natural indirect effect	1.00 (0.99–1.02)	0.99 (0.57–1.001)
Total effect	1.18 (0.96–1.54)	0.92 (0.74–1.03)

Notes: * p < 0.05; ** p < 0.01; *** p < 0.001 CI = confidence interval; b = partially standardized estimate; se = standard error of the estimate.

Education and Vocational Services

The mediation process model examining the relationships between criminogenic needs, referral to services targeting education or employment, and recidivism is shown in Table 5. The Education and Employment domain score was included as a key independent variable in these models, as these need areas are expected to be the most impacted by these types of interventions. The results indicate that receiving services aimed at education and employment is not a significant predictor of recidivism in either of the two models. The Education and Employment need/risk score is a statistically significant predictor of recidivism in both models, increasing the likelihood of recidivism in both instances (b = 0.13; b = 0.14, p < 0.01). Focusing on the “Natural Direct Effect” in the bottom panel of Table 5, for instance, shows roughly 39 and 44% greater odds of new adjudication for a unit increase in that domain score in the community and residential samples, respectively.⁸ Controlling for the other variables in the model, the Juvenile Justice History domain score is significantly related to new adjudication in the community-based sample (b = 0.12, p < 0.01), but not the sample of residential youths (b = 0.02). The Education and Employment domain score did not predict referral to those services in either subgroup, however. The only significant coefficient among the key variables in that aspect of the model was the Juvenile Justice History domain score for the community-based cases. This suggests that the likelihood that youths receive those services increases with each additional point on that domain score. Neither of the indirect effects was statistically significant, however, suggesting that the needs information for education and employment was not affecting recidivism through treatment referral (or not).

Table 5.

Results of Mediation Process Model for Education and Vocational Services.

	Community-Based Cases (n = 865)	State Residential Cases (n = 1530)
New adjudication on…	b (se)	b (se)
Education and vocational services	0.23 (0.10)*	−0.16 (0.13)
Education and Employment Domain Score	0.13 (0.04)**	0.14 (0.05)**
Juvenile Justice History Domain Score	0.12 (0.04)**	0.02 (0.03)
Age	−0.20 (0.06)**	−0.24 (0.06)***
Sex (1 = female)	−0.06 (0.13)	0.18 (0.15)
Race/ethnicity (1 = non-white)	0.01 (0.11)	0.08 (0.10)
Violent offense	−0.11 (0.11)	−0.27 (0.09)**
State 2	−0.50 (0.12)***	0.31 (0.16)
Aged out of juvenile system	0.18 (0.20)	0.05 (0.16)
Education and vocational services on…	b (se)	b (se)
Education and Employment Domain Score	0.002 (0.07)	0.01 (0.09)
Juvenile Justice History Domain Score	0.10 (0.05)*	−0.04 (0.06)
Age	0.08 (0.09)	−0.10 (0.13)
Sex (1 = female)	0.06 (0.18)	0.32 (0.36)
Race/ethnicity (1 = non-white)	0.18 (0.18)	0.13 (0.23)
Violent offense	−0.17 (0.16)	0.00 (0.19)
State 2	0.42 (0.16)**	0.90 (0.32)**
Aged out of juvenile system	−0.56 (0.28)*	0.24 (0.37)
Education and Employment Score to Recidivism	Odds ratio (95% bootstrapped CI)	Odds ratio (95% bootstrapped CI)
Natural direct effect	1.39* (1.12–1.91)	1.44* (1.14–2.03)
Natural indirect effect	1.00 (0.99–1.01)	0.89 (0.95–1.04)
Total effect	1.39* (1.12–1.91)	1.44* (1.14–2.03)

Notes: * p < 0.05; ** p < 0.01; *** p < 0.001 CI = confidence interval; b = partially standardized estimate; se = standard error of the estimate.

Family Services

The mediation process model examining the effects of four criminogenic needs areas on recidivism through treatment consisting of family services is shown in Table 6. The Family and Living Arrangements domain is included in the model as it most clearly maps onto the services of interest in this analysis. For residential cases, on average, youths who were referred to family services had a significantly higher likelihood of new adjudication relative to youths who were not referred to these interventions (b = 0.25, p < .01). That relationship was not statically significant in the community-based sample (b = 0.04). For the community-based sample, both OYAS domain scores were also found to increase the likelihood of a new adjudication: Family and Living Arrangements (b = 0.12, p < .01) and Juvenile Justice History (b = 0.14, p < .001). Scores in the Family and Living Arrangements domain were not significantly related to whether the youths would receive family services in either of the samples, suggesting that youths may not be matched to these services based on assessment information. Coupling this with the findings for the new adjudication equation suggests some potential untreated need among these samples (especially the community-based youths). None of the indirect effects for OYAS domain score through treatment was statistically significant in these models.

Table 6.

Results of Mediation Process Model for Family Treatment and Services.

	Community-Based Cases (n = 864)	State Residential Cases (n = 1534
New adjudication on…	b (se)	b (se)
Family treatment and services	0.04 (0.07)	0.25 (0.08)**
Family and Living Arrangements Domain Score	0.12 (0.03)**	0.09 (0.04)*
Juvenile Justice History Domain Score	0.14 (0.04)***	0.05 (0.03)
Age	−0.19 (0.06)**	−0.18 (0.06)
Sex (1 = female)	−0.07 (0.11)	0.03 (0.13)
Race/ethnicity (1 = non-white)	0.08 (0.10)	0.08 (0.09)
Violent offense	−0.15 (0.10)	−0.23 (0.10)*
State 2	−0.38 (0.12)**	0.07 (0.11)
Aged out of juvenile system	0.07 (0.17)	−0.03 (0.16)
Family treatment and services on…	b (se)	b (se)
Family and Living Situation Domain Score	0.04 (0.04)	0.003 (0.05)
Juvenile Justice History Domain Score	0.03 (0.04)	−0.12 (0.04)**
Age	−0.001 (0.06)	−0.19 (0.08)*
Sex (1 = female)	0.10 (0.12)	0.24 (0.20)
Race/ethnicity (1 = non-white)	0.14 (0.11)	−0.04 (0.12)
Violent offense	−0.03 (0.11)	−0.23 (0.13)
State 2	−0.75 (0.13)***	0.35 (0.14)*
Aged out of juvenile system	−0.03 (0.19)	0.15 (0.23)
Family and living situation score to recidivism	Odds ratio (95% bootstrapped CI)	Odds ratio (95% bootstrapped CI)
Natural direct effect	1.37* (1.15–1.87)	1.27* (1.05–1.72)
Natural indirect effect	1.002 (0.99–1.01)	1.001 (0.97–1.04)
Total effect	1.37* (1.15–1.86)	1.27* (1.05–1.72)

Notes: * p < 0.05; ** p < 0.01; *** p < 0.001 CI = confidence interval; b = partially standardized estimate; se = standard error of the estimate.

Mental Health and Substance Abuse Treatment

Given the integrated nature of the OYAS domain capturing mental health and substance needs, we created a potential mediator variable that reflects whether the youths were assigned to one of those treatment types. Treatment for mental health or substance use was the most recorded form of treatment in the sample with roughly 38% of youths referred to interventions in these categories. The model estimates for the relationship between this domain and recidivism through mental health or substance use treatment are shown in Table 7. This type of treatment was not associated with a greater likelihood of a new adjudication in either sample, and the Substance Use, Mental Health, and Personality domain score was not significantly related to the likelihood of a new adjudication for either subgroup. As in previous models, this suggests that unaddressed need might be contributing to elevated levels of recidivism. There is a significant treatment and domain score interaction term in the state residential sample that indicates that those who receive treatment and have higher domain scores tend to have a slightly higher likelihood of a new adjudication (b = 0.09, p < .01). Neither of the indirect effects was statistically significant, suggesting that the risk domain information was not affecting recidivism through treatment referral (or not).

Table 7.

Results of Mediation Process Model for Mental Health or Substance Abuse Treatment.

	Community-Based Cases (n = 796)	State Residential Cases (n = 868)
New adjudication on…	b (se)	b (se)
Mental health or substance abuse treatment	−0.10 (0.16)	0.36 (0.32)
SA, MH, and Personality Score	0.01 (0.04)	0.02 (0.08)
Treatment × SA, MH, and Personality Score	0.065 (0.03)	0.09 (0.03)**
Juvenile Justice History Domain Score	0.13 (0.05)**	−0.03 (0.08)
Age	−0.18 (0.03)**	−0.24 (0.12)*
Sex (1 = female)	−0.05 (0.13)	0.47 (0.64)
Race/ethnicity (1 = non-white)	0.07 (0.12)	−0.06 (0.24)
Violent offense	−0.14 (0.12)	−0.58 (0.23)*
State 2	−0.24 (0.13)*	0.26 (0.38)
Aged out of juvenile system	−0.01 (0.20)	0.12 (0.40)
MH or substance abuse treatment…	b (se)	b (se)
SA, MH, and Personality Score	−0.03 (0.07)	−0.08 (0.19)
Juvenile Justice History Domain Score	−0.002 (0.11)	0.09 (0.11)
Age	−0.01 (0.12)	−0.12 (0.19)
Sex (1 = female)	−0.03 (0.26)	−0.51 (1.24)
Race/ethnicity (1 = non-white)	0.31 (0.22)	0.13 (0.39)
Violent offense	0.26 (0.23)	0.34 (0.37)
State 2	−0.22 (0.22)	0.47 (0.56)
Aged out of juvenile justice system	−0.09 (0.42)	0.16 (0.72)
SA, MH, and Personality Score to Recidivism	Odds ratio (95% bootstrapped CI)	Odds ratio (95% bootstrapped CI)
Natural direct effect	1.04 (0.83–1.23)	1.04 (0.71–1.35)
Natural indirect effect	1.004 (0.98–1.07)	0.94 (0.75–1.03)
Total effect	1.04 (0.86–1.25)	0.98 (0.70–1.23)

Notes: * p < 0.05; ** p < 0.01; *** p < 0.001 CI = confidence interval; b = partially standardized estimate; se = standard error of the estimate.

Discussion

The past few decades have seen exponential growth in the use of JRNAs to reach conclusions about justice-involved youths and make decisions about their cases. Tremendous emphasis has been placed on using JRNAs to classify youths into categories based on their likelihood of reoffending and determining whether those classifications are valid and reliable through research. Far less attention has been given to understanding the use of risk and needs assessment to inform decision-making and case management, or the impact that it may have on the youths who are involved in the juvenile justice system. Implementation and usage of JRNA are multifaceted and depend on various sub-components to work effectively and efficiently. We used record data from a multi-state implementation, usage, and outcome study to evaluate how the OYAS is used in practice. We estimated several mediation models to specify the underlying logic of JRNA procedures more wholly from assessment to case decision to outcome.

Key Findings

The results of these mediation analyses suggest that the relationships between assessment, juvenile justice decisions, and recidivism are varied and complex when they are broken down into the various pieces that operate within the theorized implementation, usage, and outcome model. The addition of controlled mediation analysis (see Vanderweele, 2015) offers insight as to the match between disposition and risk level and how that may relate to recidivism. In the disposition models, risk level was generally associated with new adjudication as expected and consistent with predictive validity studies. Findings for the rest of the process were mixed, however.

The models focused on disposition show that risk level and juvenile court dispositions generally have an impact on youths’ recidivism (i.e., new adjudication) in ways anticipated by prior research and broader RNR principles relevant to JRNAs. For example, commitment to a residential facility may be appropriate for high/moderate youths—certainly more so than low-risk youths. We would expect that state commitment would raise chances of recidivism for low-risk youths, while for moderate- or high-risk youths, it could offer appropriate services and an element of short-term control that might lead to decreases in recidivism. The analysis here shows direct effects that support that assertion; however, this is likely conditioned by the quality of the programming in facilities (Lipsey, 2009) and is also affected by the possibility that youths who were committed to residential facilities might be more apt to eventually encounter the adult system where recidivism data for our sample are likely to be less reliable. Although there was a positive relationship between risk level and state commitment in the high-risk sample, the indirect effect estimates generally were not statistically significant and did not align with the process anticipated by JRNA logic.

The models for probation and lesser sanctions (e.g., diversion and community service) did not follow as closely from what would be anticipated in the logic of JRNA processes. The risk level estimate was in line with what would be anticipated for recidivism, but the relationships between risk levels and the probation disposition were mixed. A statistically significant positive direct relationship between OYAS risk level and whether a youth was placed on community supervision versus less serious community-based sanctions was not identified in these analyses, suggesting that the relationship between risk level and recidivism did not work through whether youths were placed on probation supervision or not.

The second group of mediation models examined processes in which risk assessment information is used to strategically match youths to treatment, which in turn is intended to reduce recidivism. The results of the analysis generally suggest that matching criminogenic needs to treatment does not consistently occur in the juvenile justice system as anticipated based on the underlying logic of JRNA. Additionally, the relatively small number of youths referred to many of the treatment program types studied here (apart from substance abuse or mental health services) could reflect a lack of understanding of the benefits of such referrals and/or a lack of availability in some locales. This conclusion is based on the absence of any indirect effects from OYAS subdomain scores through treatment/services to recidivism, as well as the limited findings of relationships between the various domain score covariates and referral indicators. Specifically, we observed no single instance where that relationship followed from theory across several models and two subgroups. In fact, offense history and the state dummy variable tended to have as consistent relationships with treatment referral, as did the OYAS domain score that would seemingly best correspond to that decision-making. Relationships between referral to treatment and recidivism were largely null or—in two cases—in the opposite direction of what would be expected if the referrals made to treatment programming had their intended effect (i.e., engaging in treatment increased the likelihood of recidivism). This may be a marker of the (in)effectiveness of the programming to which youths—even those who have a particular area of need—are being exposed (Lipsey, 2009), availability of programming to fit with need, as well as some inefficiencies in how youths are assigned to types of treatments.⁹ Nevertheless, this study’s formal modeling helps to better illustrate the way different elements of measured risk and needs might impact eventual case outcomes. Overall, it does not appear that needs information is being used to assign youths to treatment consistently. Even when it appears to be used, the relationships are not entirely consistent with the underlying logic of JRNA. The findings may also suggest a gravitation toward the use of the “risk” portion of the assessment without as much careful attention to the “needs” element that is essential in striking the balance anticipated by the mission of the juvenile justice system.

Implications

Given that the study sample comprises agencies that have actively implemented a JRNA process in recent years, its findings have some implications for practice. These are pertinent to training, oversight, and continuous quality improvement. First, one challenge in optimizing usage comes from the fact that the underlying model that supports JRNA requires buy-in from practitioners at multiple points and in multiple roles. As an example, Vincent and colleagues (2016) found that obtaining buy-in from judges was essential in fully realizing the implementation process. States and local agencies should train staff who are both administrators and non-administrators of JRNAs, as this may generate more buy-in from those who will not be responsible for administering the tool, but who will make decisions based on information from it. It is, however, essential that those promoting the new approach shows the full sequence expected from JRNA practices. Even if it is assumed that the assessment provides useful information on the needs of youths, agencies must have appropriate services in place to address need areas. This may come through services within the agency or via contractors, but it is essential that those in the field can visualize the entire JRNA logic model with their caseloads. More generally, Peterson-Badali et al. (2015) asserted that it is important that agencies identify and respond to barriers to effective matching of youths to services. This may require devoting greater resources to the programs and services suggested by JRNAs as well as greater training on how to map the assessment results to appropriate placement and intervention.

Training and other related documents should include in-depth modules to map assessment results to case decisions. This might include expectations for supervision levels (i.e., frequency and types of contact based on risk and needs level), for treatment referrals, and on how to conduct effective case planning (including good example case plans that can serve as a point of reference post-training). Training should also emphasize the consideration of additional idiographic factors (i.e., non-criminogenic needs) that are not inherently built into the risk assessment score that can impact responsivity to treatment and whether a youth fits the typical pattern of risk. In general, it is essential that agencies introducing and utilizing JRNAs do not simply assume that the risk-outcome association is the only basis on which they should determine effective usage of a tool. They must recognize the series of interrelated parts in information gathering, decision-making, and case management that are a part of this JRNA process. This is true for researchers as predictive validity studies that do not capture the decisions made based on the risk and needs assessment may miss an important piece of the puzzle in understanding the tool(s) in which they are interested. This problem may be compounded without appropriate reassessment as the “baseline” for predicting outcomes may change as a youth’s case proceeds through the justice system and treatment process.

Limitations

While this study offers useful insight and considers more complete models of the JRNA process, it does have several limitations that offer context around the key findings. The sampling strategies used for the case record and youth interview data were intended to generate representative samples of the justice-involved youths based on geographic region (which may influence the agency size and operations) and the various stages of the juvenile justice system where the OYAS was employed. Some counties in these states refused to participate, however (71% of those contacted in State 1 and 67% in State 2). We also analyzed the data without survey weights due to compatibility between those elements of our main models and available software. As a result, the estimates may not have fully captured the range of cases in these states. Further, our decision to drop cases using the OYAS-diversion and OYAS-detention tool reduced the number of cases included in our final sample that received less stringent penalties for their disposition (e.g., diversion and dismissal). While necessary due to the limited scope of risk and need factors captured by these two tools, this undoubtedly changed the makeup of cases included our sample and could have impacted our results.

The case record data presented some limitations with respect to measurement. First, data were gathered from numerous agencies, each of which adopted their own norms for recordkeeping. Although each agency was given the same instructions for completing the data request, some items were not routinely collected by all agencies, or there may have been variation in what and how data fields were documented. The research team discussed inconsistencies as they were identified and addressed them as each data file was cleaned and integrated. Those inconsistencies impact the sample sizes used in different analyses as measure coverage affected the degree to which cases could be included in statistical models. For example, a residential treatment provider with a wide variety of services may have been listed for a case, however, no additional information was provided (e.g., program name and treatment type) to accurately assess the specific treatment type received by the youths. Additionally, over a quarter of the sample (29%) were cases from counties that did not provide any treatment-related data. This is evident in shifting sample sizes for some of the multivariate models estimated here.

There were also limitations in the terms of the depth of data collected on each treatment type and aspects such as dosage, quality, or completion status were not available or not provided. Where the information provided was unclear or deviated from the provided instructions, efforts were made to contact the agency for clarification. While these strategies likely enhanced the consistency and accuracy of the data, it is possible that some differences in interpretation exist. Although evaluation of specific treatment types and dosages was beyond the scope of this study, that may have impacted some conclusions about the relationships between treatment/services and recidivism. We also did not account for the priority of needs in the analysis, which may miss the nuance that sometimes is necessary in matching treatment to youth. For example, a youth (especially those considered high risk) may have had several need areas requiring that one or more take priority—regardless of score. Future research should investigate the impact that program completion, treatment characteristics (e.g., quality, fidelity, and dosage), and complex treatment matching for multi-need youths have on the relationships studied here. It is possible that our inability to control for this information could help explain our findings that ran counter to expectations. Similarly, future studies should include measures of JRNA implementation and fidelity and assess the influence of such attributes on the JRNA process.

Similar issues arose in the officially recorded case details and justice outcome data. Agencies had different definitions for recording data about focal cases, dispositions, and recidivism and varied in the level of detail provided about each in response to data requests. Most agencies provided dichotomous recidivism indicators that amounted to tallies of new referrals, adjudications, and commitments that occurred within their jurisdiction. For these reasons, we are limited to just a single, officially recorded, outcome measure: new adjudication for a juvenile offense. Although we controlled for age and stratified based on different disposition and supervision types, it is likely that the officially recorded outcomes are at least partly affected by monitoring effects. Further, our inability to capture recidivism that youths may have engaged in outside their original jurisdiction or beyond the juvenile justice follow up window could have led to under-reporting recidivism occurrences for youth in this study. Based on age and recidivism patterns, most new adjudications were likely captured, but recidivism was at least slightly undercounted by the lack of coverage of new offenses in the adult system and/or self-reports.

Conclusion

Despite its limitations, this study specifies a full model of the JRNA process from the initial gathering and processing of information about youths to later individual case outcomes. Most studies of JRNA—especially those based on record data—do not move past a focus on assessed risk and later recidivism. Given the need to compare the logic model for JRNAs to its actual use, we sought to look more explicitly at the process that unfolds between the assessment of the youth and later follow-up. We considered the degree to which conclusions from the assessment are related to key juvenile justice decisions and, in turn, whether those dispositions or placements affect recidivism. The findings are mixed but suggest that the risk aspect of assessment is more closely linked to juvenile justice decisions than the specific criminogenic domains captured in the tool, which may suggest treatment referrals may not be aligned with criminogenic needs. In turn, the relationships between risk, case disposition, and later recidivism are stronger and more consistent with theory than are those related to treatment. It is important that juvenile justice agencies address different facets of the JRNA process if they wish to maximize its effective usage to better respond to youths’ risks and needs in a fair way. Similarly, researchers should look more broadly at the within-system processes that sometimes become a “black box” between structured assessment and later case outcomes.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice, under Grant 2014-MU-FX-0006. The opinions, findings, conclusions, or recommendations expressed are those of the authors and do not necessarily reflect those of the U.S. Department of Justice.

ORCID iD

Amber Petkus

Notes

Author biographies

Amber Petkus is a doctoral student in criminal justice at the University of Cincinnati. Her interests include correctional practice and policy, collateral consequences, rehabilitation, juvenile justice, child welfare, and family violence. She focuses especially on translational, implementation, and evaluation research. Her recent work has appeared in the Journal of Contemporary Criminal Justice, Journal of Ethnicity in Criminal Justice, Justice Evaluation Journal, and Criminal Justice and Behavior.

Christopher J. Sullivan is Professor and Director of the School of Criminal Justice and Criminology at Texas State University. He is the author of Taking Juvenile Justice Seriously: Developmental Insights and System Challenges (2019, Temple University Press) and co-author of Juvenile Risk and Needs Assessment: Theory, Research, Policy, and Practice (2021, Routledge). He is currently Co-Editor of the Journal of Research in Crime and Delinquency.

Melissa Lugo received her PhD from the University of Cincinnati and is a research and policy associate at the North Carolina Sentencing and Policy Advisory Commission. Her interests focus on the integration of developmental and life-course criminology with corrections, the evaluation of correctional interventions, and the application of data-driven approaches to inform sentencing practices. Her research has been published in Justice Quarterly, Crime & Delinquency, Journal of Contemporary Criminal Justice, and Criminology & Public Policy.

Jamie Newsome is the Research Coordinator for the University of Cincinnati Corrections Institute. She received her doctorate in Criminal Justice from the University of Cincinnati in 2013. Her interests include the design, implementation, and effectiveness of correctional interventions and translating research into practice. Her research has been featured in Criminal Justice and Behavior, Journal of Criminal Justice, and Development and Psychopathology.

References

Andrews

D. A.

Bonta

(2010). Rehabilitating criminal justice policy and practice. Psychology, Public Policy, & Law, 16(1), 39–55. http://doi.org/10.1037/a0018362

Andrews

D. A.

Bonta

Wormith

J. S.

(2006). The recent past and near future of risk and/or need assessment. Crime & Delinquency, 52(1), 7–27. http://doi.org/10.1177/0011128705281756

Andrews

D. A.

Bonta

Wormith

J. S.

Guzzo

Brews

Rettinger

Rowe

(2011). Sources of variability in estimates of predictive validity: A specification with level of service general risk and need. Criminal Justice and Behavior, 38(5), 413–432. http://doi.org/10.1177/0093854811401990

Baglivio

M. T.

Greenwald

M. A.

Russell

(2014). Assessing the implications of a structured decision-making tool for recidivism in a statewide analysis. Criminology & Public Policy, 14(1), 5–49. http://doi.org/10.1111/1745-9133.12108

Baglivio

M. T.

Wolff

K. T.

Howell

J. C.

Jackowski

, & Greenwald

M. A.

(2018). The search for the holy grail: Criminogenic needs matching, intervention dosage, and subsequent recidivism among serious juvenile offenders in residential placement. Journal of Criminal Justice, 55(March/April), 46–57. https://doi.org/10.1016/j.jcrimjus.2018.02.001.

Baglivio

M. T.

Wolff

K. T.

Jackowski

Greenwald

M.A.

(2015). A multilevel examination of risk/need change scores, community context, and successful re-entry of committed juvenile offenders. Youth Violence and Juvenile Justice, 15(1), 38–61. https://doi.org/10.1177/1541204015596052

Baglivio

M. T.

Zettler

Craig

J. M.

Wolff

K. T.

(2021). Evaluating RNR-based targeted treatment and intervention dosage in the context of traumatic exposure. Youth Violence and Juvenile Justice, 19(3), 251–276. https://doi.org/10.1177/1541204020988575

Baron

Kenny

(1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. https://doi.org/10.1037//0022-3514.51.6.1173

Bonta

Andrews

D. A.

(2017). The psychology of criminal conduct (6th ed.). Routledge.

10.

Campbell

C. A.

D’Amato

Papp

(2020). Validation of the Ohio youth assessment system dispositional tool (OYAS-DIS): An examination of race and gender differences. Youth Violence and Juvenile Justice, 18(2), 196–211. https://doi.org/10.1177/1541204019859938

11.

Duwe

Rocque

(2017). Effects of automatic recidivism risk assessment on reliability, predictive validity, and return on investment (ROI). Criminology & Public Policy, 16(1), 235–269. https://doi.org/10.1111/1745-9133.12270

12.

Gendreau

Little

Goggin

(1996). A meta-analysis of the predictors of adult offender recidivism: What works. Criminology, 34(4), 575–608. https://doi.org/10.1111/j.1745-9125.1996.tb01220.x

13.

Grattet

Lin

Petersilia

(2011). Supervision regimes, risk, and official reactions to parolee deviance. Criminology, 49(2), 371–399. https://doi.org/10.1111/j.1745-9125.2011.00229.x

14.

Hayes

A. F.

(2013). Introduction to mediation, moderation, and conditional process analysis: A regression based approach. The Guilford Press.

15.

Hilterman

Nicholls

van Nieuwenhuizen

(2014). Predictive validity of risk assessments in juvenile offenders: Comparing the SAVRY, PCL: YV, and YLS/CMI with the unstructured clinical assessments. Assessment, 21(3), 324–339. https://doi.org/10.1177/1073191113498113

16.

Latessa

Listwan

Koetzle

(2014). What works (and doesn’t) in reducing recidivism. Anderson Publishing.

17.

Latessa

E. J.

Lovins

(2010). The role of offender risk assessment: A policy maker guide. Victims & Offenders, 5(3), 203–219. https://doi.org/10.1080/15564886.2010.485900

18.

Latessa

Lovins

Ostrowski

(2009). The Ohio youth assessment system: Final report. University of Cincinnati.

19.

Lipsey

M. W.

(2009). The primary factors that characterize effective interventions with juvenile offenders: A meta-analytic overview. Victims & Offenders, 4(2), 124–147. https://doi.org/10.1080/15564880802612573.

20.

Lovins

Latessa

(2013). Creation and validation of the Ohio youth assessment system (OYAS) and strategies for successful implementation. Justice Research and Policy, 15(1), 67–93. https://doi.org/10.3818/jrp.15.1.2013.67

21.

Luong

Wormith

J. S.

(2011). Applying risk/need assessment to probation practice and its impact on the recidivism of young offenders. Criminal Justice and Behavior, 38(12), 1177–1199. https://doi.org/10.1177/0093854811421596

22.

Lurigio

A. J.

(2011). People with serious mental illness in the criminal justice system: Causes consequences and correctives. The Prison Journal, 91(3S), 66S–86S. https://doi.org/10.1177/0032885511415226

23.

McCafferty

J. T.

(2013). The predictive validity of the Ohio youth assessment system-disposition instrument: A revalidation study, doctoral dissertation. University of Cincinnati. https://etd.ohiolink.edu/

24.

McCafferty

J. T.

(2018). Unjust disparities? The impact of race on juvenile risk assessment outcomes. Criminal Justice Policy Review, 29(5), 423–442. https://doi.org/10.1177/0887403416634163

25.

Mears

(2012). The front end of the juvenile court: Intake and informal versus formal processing. In Bishop

D. M.

Feld

B.C.

(Eds), The Oxford Handbook of Juvenile Crime and Juvenile Justice (pp. 1–35). Oxford University Press.

26.

Muthén

L. K.

Muthén

B. O.

(1998-2017). Mplus user’s guide (8th ed.). Muthén & Muthén.

27.

National Research Council (2013). Reforming juvenile justice: A developmental approach. The National Academies Press.

28.

Nelson

R. J.

Vincent

G. M.

(2018). Matching services to criminogenic needs following comprehensive risk assessment implementation in juvenile probation. Criminal Justice and Behavior, 45(8), 1136–1153. https://doi.org/10.1177/0093854818780923

29.

Olver

M. E.

Stockdale

K. C.

Wormith

J. S.

(2009). Risk assessment with young offenders: A meta-analysis of three assessment measures. Criminal Justice and Behavior, 36(4), 329–353. https://doi.org/10.1177/0093854809331457

30.

Peterson-Badali

Skilling

Haqanee

(2015). Examining implementation of risk assessment in case management for youth in the justice system. Criminal Justice and Behavior, 42(3), 304–320. https://doi.org/10.1177/0093854814549595

31.

Rossegger

Gerth

Seewald

UrbaniokSingh

Singh

Endrass

(2013). Current obstacles in replicating risk assessment findings: A systematic review of commonly used actuarial instruments. Behavioral Sciences & the Law, 31(1), 154–164. https://doi.org/10.1002/bsl.2044

32.

Schwalbe

(2007). Risk assessment for juvenile justice: A meta-analysis. Law and Human Behavior, 31(5), 449–462. https://doi.org/10.1007/s10979-006-9071-7

33.

Schwalbe

C. S.

Fraser

M. W.

Day

S. H.

Cooley

(2006). Predictive validity, race/ethnicity, and gender. Criminal Justice and Behavior, 33(3), 305–324. https://doi.org/10.1177/0093854806286451

34.

Shook

J. J.

Sarri

(2007). Structured decision-making in juvenile justice: Judges’ and probation officers’ perceptions and use. Children and Youth Services Review, 29(1), 1335–1351. https://doi.org/10.1016/j.childyouth.2007.05.008

35.

Sickmund

Sladky

Kang

(2019). Easy access to juvenile court statistics 1985-2017. National Center for Juvenile Justice. https://www.ojjdp.gov/ezajcs

36.

Smith

Gendreau

Swartz

(2009). Validating the principles of effective intervention: A systematic review of the contributions of meta-analysis in the field of corrections. Victims & Offenders, 4(2), 148–169. https://doi.org/10.1080/15564880802612581

37.

Steinhart

(2006). A practice guide to juvenile detention reform: Juvenile detention risk assessment. The Annie E. Casey Foundation.

38.

Sullivan

C.J.

(2019). Taking juvenile justice seriously: Developmental insights and system challenges. Temple University.

39.

Sullivan

Christopher J.

Strange

C. Clare

Sullivan

Carrie

Newsome

Jamie

Lugo

Melissa

Mueller

Derek

Petkus

Amber

Holmes

Bryan

Lonergan

Holly

, & McCafferty

James

(2019). Multi-Method Study on Risk Assessment, Implementation, and Youth Outcomes in the Juvenile Justice System. Washington, D.C.: Office of Juvenile Justice and Delinquency Prevention. https://www.ojp.gov/ncjrs/virtual-library/abstracts/multi-method-study-risk-assessment-implementation-and-youth.

40.

Valeri

Vanderweele

(2013). Mediation analysis allowing for exposure-mediator interactions and causal interpretation: Theoretical assumptions and implementation with SAS and SPSS macros. Psychological Methods, 18(2), 137–150. https://doi.org/10.1037/a0031034

41.

Vanderweele

(2016). Mediation analysis: A practitioner’s guide. Annual Review of Public Health, 37, 17–32. https://doi.org/10.1146/annurev-publhealth-032315-021402

42.

Vieira

T. A.

Skilling

T. A.

Peterson-Badali

(2009). Matching court-ordered services with treatment needs: Predicting treatment success with young offenders. Criminal Justice & Behavior, 36(4), 385–401. https://doi.org/10.1177/0093854808331249

43.

Vincent

Guy

Fusco

Gershenson

(2012a). Field reliability of the SAVRY with juvenile probation officers: Implications for training. Law and Human Behavior, 36(3), 225–236. https://doi.org/10.1037/h0093974

44.

Vincent

Guy

Gershenson

McCabe

(2012b). Does risk assessment make a difference? Results of implementing the SAVRY in juvenile probation. Behavioral Sciences & the Law, 30(4), 384–405. https://doi.org/10.1002/bsl.2014

45.

Vincent

Guy

Perrault

Gershenson

(2016). Risk assessment matters, but only when implemented well: A multisite study in juvenile probation. Law and Human Behavior, 40(6), 683–696. https://doi.org/10.1037/lhb0000214

46.

Vogel

(2014). Mental illness and criminal behavior. Sociology Compass, 8(4), 337–346. . https://doi-org.uc.idm.oclc.org/10.1111/soc4.12140

47.

Wachter

(2015). Statewide risk assessment in juvenile probation. Juvenile justice geography, policy, practice, & statistics state scan. National Center for Juvenile Justice. http://www.ncjrs.gov/App/publications/abstract.aspx?ID=268893