Jurisdictional Resource Factors Influencing Adoption and Implementation of the Public Safety Assessment During Pretrial

Abstract

Criminal legal systems are increasingly adopting actuarial pretrial assessments which use statistical formulas to estimate individuals’ probability of adhering to pretrial requirements and guide the conditions of their supervision. A large gap in the study of pretrial assessments is due to inattention to implementation of these assessments in real-world settings. We conducted qualitative research examining personnel and data resource factors that influenced adoption and implementation of one pretrial assessment, the Public Safety Assessment (PSA) in seven counties in the United States. Qualitative interviews with legal and community actors were conducted and supplemented with implementation process data to elucidate personnel and data capacity factors impacting PSA adoption and implementation. Findings suggest that generally, jurisdictions with existing pretrial services programs were more likely to adopt the PSA and to encounter fewer barriers to implementation due to personnel and data infrastructure. Implications for PSA adoption and implementation in future pretrial settings are discussed.

Keywords

pretrial risk assessment Public Safety Assessment implementation qualitative

Introduction

Reforms are underway in many criminal legal systems across the United States focused on improving management of pretrial populations (Carroll, 2023; Hopkins & Doyle, 2018; Stevenson & Mayson, 2017; Van Brunt & Bowman, 2018). To improve pretrial outcomes (e.g., remaining arrest-free and appearing in court) and minimize potentially harmful consequences of pretrial detention, bail, and other methods of pretrial supervision, systems are increasingly adopting actuarial assessments which use statistical formulas to estimate individuals’ probability of adhering to pretrial requirements and guide the conditions of their supervision (Desmarais et al., 2022). Numerous studies have demonstrated the validity of specific pretrial instruments in predicting individuals’ behavior while released from jail (Desmarais et al., 2021).

A large gap in the study of pretrial assessments in the criminal legal system is due to inattention to their implementation in real-world settings. A tool that demonstrates validity and effectiveness in a research study may not do so in a different setting, and a main reason for lack of effect can be poor implementation (Damschroder et al., 2009). The Consolidated Framework for Implementation Research (CFIR) was first published in 2009 to synthesize constructs from multiple theories of innovation and organizational change and has since been widely used and refined to incorporate more recent literature and feedback from users (Damschroder et al., 2015, 2022). The framework provides a menu of constructs that can guide systematic assessment of barriers to and facilitators of implementation, which can help in tailoring and adapting implementation strategies and understanding outcomes. The CFIR has been widely used to understand implementation of innovations in real-world settings related to health care delivery, quality improvement, and health promotion and disease management, including health care in legal and correctional settings (Hanna et al., 2020; Kirk et al., 2015; Louie et al., 2021; Van Deinse et al., 2023). However, this useful framework has rarely been applied in criminal legal settings to understand non-health care innovations such as pretrial assessments, despite its potential for increasing understanding of implementation factors.

The CFIR specifies five major domains that influence intervention implementation: intervention characteristics (e.g., complexity, cost, components), outer setting (the “economic, political, and social context”), inner setting (the organization’s “structural characteristics, networks and communications, culture, climate, and readiness”), characteristics of the individuals involved in the intervention (e.g., their behaviors, “mindsets, norms, interests, and affiliations”), and the process of implementation (sequence and active promotion of implementation events) (Damschroder et al., 2009, p. 5). The inner setting is particularly relevant for studying implementation in criminal legal systems because local courtrooms and legal agencies in the United States present highly diverse settings for criminal legal processes and interventions, apparent in the heterogeneity they produce in outcomes such as pretrial detention rates, racial disparities, and punitiveness (Nardulli et al., 1988; Stemen & Escobar, 2018). An especially important inner setting factor in legal systems is available resources, particularly staffing and access to data (LeMasters et al., 2022; Metcalfe & Kuhns, 2023; Pruitt & Davies, 2022; Wilson & Grammich, 2024). With pretrial assessments being used more broadly across a variety of systems and influencing more individuals and communities, studies of inner setting factors influencing implementation are imperative.

We used qualitative methods guided by the concept of available resources, a facet of the CFIR “inner setting” domain, to elucidate how criminal legal agency factors impacted adoption and implementation of the Public Safety Assessment (PSA), a popular pretrial assessment, in seven diverse U.S. counties. This study addresses a gap in criminal justice research by examining how personnel and data capacity are associated with PSA adoption and implementation.

Pretrial Assessment and the PSA

Individuals who have been charged with crimes but not yet convicted pose a unique challenge to courts and law enforcement agencies. Although individuals are legally presumed innocent of criminal charges until they plead guilty or are convicted at trial, if they are not detained pretrial there is a chance that they could commit a crime while awaiting adjudication or not show up for court (Grodensky et al., 2022; Magnuson et al., 2023; Nam-Sonenstein, 2023). Traditionally, systems have used pretrial detention and monetary bail to incentivize individuals to adhere to pretrial requirements, but these strategies can contribute to overincarceration, negative individual and community outcomes, and socioeconomic inequities (Van Brunt & Bowman, 2018) and lack effectiveness in producing positive pretrial outcomes (Zottola & Desmarais, 2022). Pretrial assessments have been presented as tools to help systems make data-informed decisions regarding pretrial release to improve outcomes and minimize unwanted effects. The use of actuarial risk assessment instruments during pretrial and at other points in the criminal legal system has been controversial due to concerns that they may perpetuate racial biases and fail to accurately predict pretrial outcomes; however, numerous systems have viewed the use of valid pretrial instruments as preferable to relying on the discretion of legal actors which may lead to biased decisions and the overuse of detention (Pretrial Justice Institute, 2020).

One tool that has gained popularity in recent years is the PSA, developed by VanNostrand and Lowenkamp with funding from Arnold Ventures to expedite the release process by providing court actors with information on the likelihood that an individual will comply with pretrial conditions (Arnold Ventures, 2020; VanNostrand & Lowenkamp, 2013). PSA scores are calculated from nine factors which can be found in administrative databases (jail, court, criminal history repositories): age at current arrest, whether the current offense is violent, whether there is a pending charge at the time of arrest, prior conviction of misdemeanor, felony, or violent crime, prior failure to appear (FTA) in court in the past 2 years and older than 2 years, and prior sentence to at least 2 weeks of incarceration. These factors have been shown to be associated with the three PSA outcomes: FTA, new criminal arrest (NCA), and new violent criminal arrest (NCVA) (Brittain et al., 2021; DeMichele et al., 2020; DeMichele et al., 2023; Greiner et al., 2020). A common concern with the use of pretrial assessment instruments often centers around whether they exacerbate racial bias by scoring Black people as having higher risk than their true propensity for not appearing in court or committing a new crime (Desmarais et al., 2022). Although Black individuals face more attention from law enforcement and substantive disadvantages in criminal legal system processing (i.e., differential enforcement) and therefore may score higher than White individuals on the criminal history items in the assessment (Hamilton, 2015), validation and bias studies have found little evidence of predictive racial bias with the PSA. A recent multicounty study of the PSA found the instrument to be a valid and consistent predictor of pretrial outcomes across racial-ethnic and sex groups, with little evidence of predictive bias (DeMichele et al., 2023).

PSA scores are used with a jurisdiction-determined release conditions matrix (RCM) to link local pretrial supervision conditions to each score level to help judges and other court actors make decisions about whether and how to supervise individuals pretrial. The PSA is meant to be scored within 24 hours of booking so that it can inform early pretrial release decisions. Without such tools, court officials must rely on their own discretion in determining pretrial supervision, which is susceptible to the influence of extra-legal factors, implicit biases, and cognitive shortcuts which have likely contributed to racial bias in pretrial supervision decisions (Copp et al., 2022; Gouldin, 2020; Kahneman, 2011). Pretrial assessment tools like the PSA do not replace the discretion of judges and other legal actors, but rather add standardized, transparent information to support decision-making (Ludwig & Mullainathan, 2021). Although there is scant research examining the impact of the PSA on pretrial release, an analysis of data on all jail bookings before and after the PSA was adopted in Lucas County, Ohio, found that adoption was associated with decreases in rates of FTAs, NCAs, and NVCAs (Lowenkamp et al., 2020).

More jurisdictions are adopting pretrial assessment instruments across the United States. In a 2019 report, the Pretrial Justice Institute reported that approximately two-thirds of surveyed counties used a pretrial assessment instrument, and of those, over 30% reported using the PSA (Pretrial Justice Institute, 2019). Since then, PSA use has increased even more as additional evidence has been published demonstrating its predictive validity and as Arnold Ventures has worked with numerous localities to support implementation (Arnold Ventures, 2020; Lattimore et al., 2020). The PSA is now used by the states of Arizona, Kentucky, Utah, Rhode Island, and New Jersey, in multiple major cities, and in dozens of smaller jurisdictions (Arnold Ventures, 2020).

CFIR and PSA Implementation

Innovations that are implemented poorly, or implemented differently from the way they were initially designed and validated, may not yield the intended positive results. Poor implementation in criminal legal settings may consist of administering a tool incorrectly or with the wrong target population, drawing the wrong conclusions from the results, or neglecting to use the tool at all. In addition, agencies that could benefit from the use of innovations may face barriers such that they are never adopted, resulting in missed opportunities for improved outcomes. The study of implementation science is more well-developed in the health care field, but it is gaining attention in the area of criminal legal system research and was the focus of a 2024 funding solicitation from the National Institute of Justice (Escoffery et al., 2019; Gleicher, 2017; National Institute of Justice, 2024; Stirman et al., 2013; Taxman & Belenko, 2011; Zielinski et al., 2020). Implementation science has been used most frequently in criminal legal systems to inform the implementation of evidence-based practices in corrections. For example, researchers in the state of Oregon identified barriers to implementing evidence-based practices in community corrections agencies including internal policy alignment and data systems (Salisbury et al., 2019), and the Federal Probation and Pretrial Services System employed implementation science principles to improve local probation agencies’ fidelity to evidence-based supervision skills and techniques (Goldstein, 2020). However, these concepts have been used less frequently with pretrial interventions. With innovations such as the PSA being used more broadly and influencing more individuals and communities, studies of their implementation are imperative.

The CFIR describes important factors that impact implementation in five major domains. The inner setting, the organization in which an innovation is implemented, is particularly relevant in criminal legal settings. In a systematic review of studies using implementation science frameworks to assess health interventions in criminal legal settings, the factors impacting implementation were most commonly associated with the inner setting (e.g., jail, prison, community corrections) and were “most salient in highlighting the complexity of implementing health interventions within correctional health settings” (Van Deinse et al., 2023, p. 2). The resources available, especially staffing and access to valid data, are inner setting characteristics that are particularly relevant to the implementation of innovations such as the PSA (LeMasters et al., 2022; Metcalfe & Kuhns, 2023; Pruitt & Davies, 2022; Wilson & Grammich, 2024). Challenges related to these factors were exacerbated during the COVID-19 pandemic when many systems lost staff and experienced budget cuts (Haigh & Preston, 2020). PSA implementation requires resources, particularly staff time and access to data, that vary considerably across districts (Arnold Ventures, 2021).

A key component of rigorous implementation is fidelity to the innovation, or “The extent to which the program is implemented consistently across different settings, staff, and [clients]” (Damschroder et al., 2022, p. 5). Arnold Ventures, which funded the development of the PSA, has guided and promoted its use through an effort called Advancing Pretrial Policy and Research (APPR) and published a fidelity manual outlining seven phases of PSA implementation, which are listed and defined in Table 1 (Arnold Ventures, 2021). First, the jurisdiction assesses its readiness to implement the PSA (“Readiness”), which depends on the following, at a minimum: a commitment to use the PSA to inform decisions; the availability of data to score the nine PSA factors; systems to automatically calculate the PSA score; and staff to enter the necessary data to complete the PSA (Arnold Ventures, 2020).

Table 1:

Seven Phases of PSA Implementation

Phase	Description
Readiness	The jurisdiction assesses its readiness to implement pretrial improvements, including the PSA. Based on the experiences of other jurisdictions, readiness depends on the following, at a minimum: a commitment to use the PSA to inform decisions; the availability of data to score the nine PSA factors; the capacity to automate the PSA; and the availability of staff to complete the PSA
Engagement	The jurisdiction engages legal actors and community members in the pretrial improvement process, creating a policy team.
Automation	The policy team decides how to automate the PSA and incorporate it into its data system.
Assistance	The policy team examines state pretrial laws, studies the research on pretrial assessment and services, validates the PSA in the local setting, and develops a Release Conditions Matrix.
Assessment	The policy team tailors PSA-related materials for local use (the PSA Violent Offense List, the PSA Scoring Manual, and the Pretrial Assessment Report.).
Training	Legal actors, community partners, and others learn how the PSA and other advancements contribute to improved pretrial decision-making. Assessors are trained to score the PSA.
Fidelity	A quality assurance process is put in place to ensure the PSA is scored accurately and produces the expected results.

Next, in the “Engagement” phase, the jurisdiction engages legal system actors and community members to create a policy team to manage all aspects of the project and conduct a review of the jurisdiction’s pretrial system. A policy team may create subcommittees charged with specific tasks throughout the implementation phases. In the next three phases (“Automation,” “Assistance,” and “Assessment”), the policy team and its subcommittees determine how to automate the PSA and incorporate it into the jurisdiction’s data system; examine research and local policies; perform a local statistical validation of the PSA using historical data in the local jurisdiction; and tailor PSA-related materials for local use. The remaining phases involve educating legal actors and community partners about the PSA and training assessors to score the tool (“Training”) and establishing a quality assurance (QA) process to ensure accurate PSA scoring (“Fidelity”) (Arnold Ventures, 2021).

This article presents qualitative research examining personnel and data capacity factors that influenced PSA adoption implementation in seven U.S. jurisdictions as part of an APPR research project (Arnold Ventures, 2020). Qualitative methods can be particularly effective in implementation research; for example, in a study of the implementation of evidence-based assessments in probation agencies, qualitative data revealed that probation officers rarely used the assessments to inform decision-making as intended (Viglione, 2019). In this study, in general, jurisdictions with existing pretrial services programs were more likely to adopt the PSA and successfully implement it. In counties without pretrial services, it was necessary to build this capacity before adopting the PSA. The remainder of this article describes the APPR study within which this research was conducted, methods used to understand capacity factors, and results and a discussion within the context of current pretrial research and reform.

Method

The data for this study were collected as part of Arnold Venture’s APPR project (Arnold Ventures, 2020). APPR provides training and technical assistance to dozens of jurisdictions to support pretrial reform efforts and implementation of the PSA, and selected Research-Action Sites for the purpose of researching these efforts. Arnold Ventures, with support from teams of Training and Technical Assistance providers and evaluators, reviewed proposals from each jurisdiction detailing their history and capacity for pretrial reform, requested additional information as needed, and conducted interviews with site leadership. APPR Research-Action Sites were competitively selected in two waves: five counties were accepted in 2019 as Wave 1, and two counties were accepted in 2020 as Wave 2. Characteristics of the seven sites are summarized in Table 2. In 2021, one Wave 1 study site chose to discontinue participation in APPR, but it is included in this article along with the six other counties.

Table 2:

Characteristics of APPR Research Action Sites

Site	Approximate population size^a	Proportion non-White population	Proportion non-White bookings (2021)	Proportion living in poverty	Geographic location in United States	Date of PSA adoption
County A	1,051,550	56.6%	85.7%	13.4%	South Atlantic	December 2022
County B	891,862	28.6%	38.8%	9.5%	Pacific Northwest	October 2022
County C	284,698	20.5%	32.3%	9.8%	Pacific Northwest	January 2024
County D	158,507	21.5%	28.9% (2019)	12.4%	South Atlantic	October 2023
County E	393,078	45.4%	61.3%	15.8%	South	Not adopted as of May 2024
County F	226,451	65.3%	77.8%	18.8%	South	Not adopted as of May 2024
County G	546,598	35.5%	65.4%	13.4%	Midwest	Not adopted as of May 2024

Data from the U.S. Census Bureau, 2020 American Community Survey.

The seven APPR Research-Action Sites (referred to as “sites” henceforth) received technical and research assistance from Arnold Ventures and its contracting agencies during the seven PSA implementation phases. APPR’s external evaluator conducted multiple research activities with the sites focused on understanding the implementation, validity, and impact of the PSA. The statistical validation of the PSA using local data during the “Assistance” phase was conducted entirely by the external evaluator, who worked with representatives from each site to obtain and clean data from multiple agencies to create the validation analytic data set.

Data Sources

Throughout the study period, researchers from APPR’s external evaluator served as site liaisons for each county. Site liaisons met regularly with a point of contact, the policy team, and the technical assistance providers at each study site, and collected a variety of qualitative data related to the implementation of the PSA and the surrounding context of the local criminal legal system. In addition, each site was assigned an evaluation data manager who was responsible for working with local and state agency staff to identify, transfer, and clean the appropriate quantitative data needed for PSA validation and scoring. Data sources used for the current qualitative analysis include (a) memos and notes entered by site liaisons into an implementation database; (b) interviews conducted with points of contact and legal system actors at each site; and (c) notes compiled from each evaluation site data manager.

The study implementation database was a system for storing and categorizing notes from policy team and subcommittee meetings, communications with local actors, and local developments or media coverage related to the criminal legal system. Site liaisons and other study staff created entries into the database regularly throughout the study period to document developments related to PSA implementation and other pretrial reforms. Entries into the database were either short descriptive memos categorized according to the type of PSA implementation activity (e.g., “system map,” “Violent Offense List,” “PSA training”) or local development (e.g., “high-profile incidents,” “local government,” “laws/statutes”), or documents containing the notes taken from policy team meetings, sub-committee meetings, and site visits.

Semi-structured interviews were conducted by each site liaison with site points of contact and key policy team members annually throughout the study period during virtual or in-person site visits between the Fall of 2019 and March of 2024. The site point of contact also recommended additional individuals at each site visit who were involved in recent PSA activities, such as community members who were working on engagement or data managers supporting validation. Overall, 162 individuals were interviewed, with an average of eight individuals per site interviewed at each of the six site visits. Participants were interviewed an average of 2 times, and six were interviewed at all six time points. Most were local legal actors serving on policy teams (e.g., judges, pretrial services leadership, prosecutors, defense attorneys, and law enforcement). Community members were also interviewed at five sites.

All site liaisons were experienced interviewers who had conducted in-depth interviews with criminal legal system actors on previous research studies. They had also developed rapport with the site points of contact and gained in-depth knowledge of the site’s activities that allowed for thorough exploration of the concepts most relevant to the site and detailed probing. Before beginning each round of interviews, site liaisons reviewed each interview protocol as a group and discussed the intention behind each question. Interviews typically lasted an hour and were audio-recorded with interviewees’ permission and conducted on Zoom or in a private or semi-private location. Interviewers and note-takers also recorded brief notes and observations.

CFIR constructs informed the primary topics and domains for data collection in the implementation database and qualitative interviews. For instance, entries in the database were coded according to constructs related to inner setting (e.g., court procedures, laws/statutes), outer setting (e.g., local government, high-profile incidents in the community), characteristics of the intervention (e.g., scoring manual, RCM) and individuals (e.g., policy team), and the process of implementation (e.g., workgroup meetings, community engagement). Interview topics also mapped onto CFIR domains and how they related to the different phases of PSA implementation.

Data Analysis

For the first step of data analysis, the first author (CAG) reviewed all implementation database entries and qualitative interviews to identify the most significant factors impacting PSA implementation across sites. The most significant factors identified related most closely to the inner setting domain of the CFIR, and therefore CFIR inner setting constructs were selected to inform the creation of a qualitative codebook for thematic content analysis. These constructs included inner setting structural and relational characteristics, communications, culture, tension for change, compatibility and relative priority of the innovation, incentive systems, mission alignment, available resources, and access to knowledge and information. Audio-recorded interviews and database entries were reviewed by CAG and when excerpts were identified that reflected inner setting factors impacting PSA implementation, they were transcribed and assigned codes to indicate both the relevant CFIR inner setting construct and the relevant PSA implementation phase. Throughout the coding process, CAG held regular meetings with the site liaisons to discuss the coded material and ensure the validity of the interpretation. Coded excerpts were then reviewed and synthesized into the most salient and impactful inner setting factors that influenced PSA adoption and implementation across sites. In this article, we seek to answer the following research question: What were factors related to personnel and data capacity, two aspects of inner setting that were identified as highly salient themes across interviews, that influenced PSA adoption and implementation at APPR Research-Action Sites?

Results

The APPR timeline allowed 5 years for PSA planning and implementation. Four of the seven sites adopted and implemented the PSA during the study period (Counties A, B, C, D) and three did not (Counties E, F, G). Of the three sites that did not implement the PSA, two had plans to do so within 6 months after the study period concluded. One of the sites that implemented the PSA, County D, discontinued participation with the APPR project in 2021 but agreed to participate in interviews in 2023 to provide updates on their adoption and use of the PSA.

Personnel Capacity Factors Influencing PSA Adoption and Implementation

Personnel are needed to support (1) design and oversight of the PSA implementation process in a jurisdiction, (2) scoring of the PSA, and (3) performance of QA. Personnel (primarily local legal actors) were available at all sites to serve on a policy team dedicated to performing the duties related to design and oversight of the PSA. In sites with personnel available to score the PSA and perform QA, these personnel did not lack the skill or organizational infrastructure to complete these activities. The main personnel capacity factor that influenced PSA adoption and implementation was the availability of personnel to score the PSA and perform QA. Three of the sites that ultimately adopted the PSA had preexisting pretrial services staff available to complete PSA scoring and QA, and the fourth adopting site built a new pretrial services program of one staff member. Two sites (Counties E and F) that did not adopt the PSA lacked preexisting pretrial services staff to support PSA scoring and QA. The third site that did not adopt the PSA (County G) had sufficient preexisting pretrial services for the PSA but did not adopt the PSA during the study period for reasons other than capacity.

According to the APPR website, minimum staff capacity for PSA scoring entails that staff are available to review the relevant data sources, enter PSA factor responses into an application in a timely manner, and possibly generate pretrial assessment reports and disseminate them to legal actors. The seven sites varied considerably in the availability of personnel to score the PSA. Three did not have dedicated pretrial services programs at the time of the study, but rather provided limited services and supervision to pretrial individuals through their community corrections departments which also supervised individuals sentenced to probation. In those three counties (Counties D, E, F), it was unclear at the beginning of APPR who would score the PSA if it was adopted. In County F, a policy team member explained that even if the PSA was adopted, they did not think there would be staff capacity to score it. They explained,

It was just evident that we don’t have the personnel to do it . . . Someone has to come in early in the morning and run an individual who may be eligible for release through several law enforcement databases, and access to those databases is limited to very specific personnel, and those personnel already are busy. So we were talking through, how would you do this? What would the schedule be? Who would be the person? . . . We would need to hire somebody . . . There’s not people to do it.

Before any decision about adopting the PSA in County F could be made, the county would need to expand services provided by their community corrections program to include individuals being supervised pretrial, conduct a workload analysis to determine staffing requirements for a new pretrial unit, and identify key staff to transition from other county departments to a new pretrial unit. As of February 2024, the policy team was making progress on these steps.

Three of the four sites that adopted the PSA had preexisting pretrial staff available to administer it to all eligible pretrial individuals (Counties A, B, C). (In addition, County G, that did not implement, anticipated they would have capacity to do so.) In these sites, PSA scoring was incorporated into the duties of existing pretrial services staff. In County A, for example, pretrial intake staff, supervised by a pretrial intake manager, were already conducting one-on-one intake interviews with all individuals in the jail and accessing information on their complete criminal history that could be used in scoring the PSA. One site that adopted the PSA, County D, did not have pretrial services before the APPR project, but added pretrial services to the duties of a local nonprofit that had been providing other court services there for many years. The new pretrial services director was responsible for scoring the PSA and implementing release conditions including court notifications, phone calls and in-person check-ins, and monthly record checks. At the time of the last study interview, they were in the process of training a second person to help perform these duties. In spite of limited personnel capacity, County D was able to complete scoring and provide release conditions for most eligible individuals.

Availability of personnel also influenced PSA implementation at the sites where it was adopted. QA, the process of verifying that the PSA is scored accurately and consistently, was performed by pretrial staff in three of the four sites that adopted the PSA (Counties A, B, C), although sites varied in the proportion of cases that were checked. County A elected to QA 100% of their PSA scores, and County C conducted the APPR-recommended 30%. County B began by conducting QA on 35% of PSA scores at the beginning of implementation; however, due to the high volume of cases referred to pretrial services after PSA adoption, pretrial staff did not have time to review 35% and had to decrease to 15%. County D reported that they did not have any QA procedures in place because there was only one staff member who was scoring all PSAs and no other staff available to conduct QA. Because this site discontinued participation in the APPR project in 2021, they did not receive the same technical assistance as the other implementing sites, which may have contributed to QA being omitted.

Data Capacity Factors Influencing PSA Adoption and Implementation

The PSA relies on local criminal legal system data to predict an individual’s likelihood of court non-appearance and re-arrest, and therefore the quality and availability of data in a jurisdiction and state determines their ability to score the PSA accurately. Moreover, sufficient data capacity is critical in performing key steps in implementing the PSA: validating the PSA in the local population; automating and scoring the PSA; and generating performance metrics (Arnold Ventures, 2021). Data capacity varied widely across the seven sites. In a baseline assessment, site points of contact rated the accessibility of 55 different data elements such as “number of cases detained pretrial” and “average daily jail population.” Data elements rated 1 if they were “easy to pull,” 2 if “data is kept somewhere but difficult to pull,” 3 for “totals available but not disaggregated by race/ethnicity,” and 4 if “data are totally unavailable.” Scores varied from a mean of 1.3 for County A (most data points were readily available), to a mean of 2.8 for County D (data elements were generally difficult to pull and 23 were unavailable). Sites rating higher on data availability generally had an easier time implementing the data elements of the PSA. However, all sites experienced challenges related to data capacity that contributed to either lack of PSA adoption or delayed implementation.

Local PSA Validation

The first point in the PSA implementation process that requires local criminal legal system data is the local validation of the PSA. It is important to ensure that the PSA predicts FTA in court, new criminal arrest, and NVCA outcomes in the local population and that the tool is not biased in its risk estimation across race/ethnicity, sex, and age groups. PSA validation requires data on the demographics, pending charges, arrests, court appearance, and criminal history of individuals booked in jail pretrial in the district, and requires that complete records from multiple sources (e.g., jail, law enforcement, court system) be accessed and linked to create a single analytic data set. There was not a specific threshold of validity required to move forward with PSA adoption; rather, validation results were presented and discussed with the policy team at each site to guide site-specific decisions about whether to adopt the PSA.

APPR’s external evaluator led data validation processes at all sites and worked with local personnel to obtain the required multiagency legal system data. Five of the seven sites (Counties A, B, C, D, and G) were able to provide sufficient data to the external evaluator to allow for local PSA validation, and results showed that PSA scores were fair or good predictors of pretrial outcomes in each site. Counties E and F were unable to provide the data needed for local validation during the study period, which may have contributed to a lack of PSA adoption. All seven sites encountered barriers to data validation, but the nature and severity of the barriers varied across sites. The most significant data capacity challenges across sites that delayed PSA implementation included the lack of an indicator for pretrial bookings in jail databases, the lack of a cross-agency identifier to enable matching, and poor-quality court appearance data.

None of the sites’ jail information systems included an indicator for pretrial bookings, which required the external evaluator to develop a process to link each booking to its associated court case(s) in the state court data file to determine whether the case was still open. Linking jail bookings with their associated court cases is a straightforward process when common identifiers at the person-level (e.g., state identification number) and booking-level (e.g., court docket number) are found in both jail and court systems. However, most sites either did not have person-level or booking-level identifiers that allowed for high levels of accuracy in matching, which resulted in jail bookings that could not be reasonably linked to a court case. Because these bookings lacked a court disposition date, which was required to determine the pretrial release period and examine PSA outcomes, they were excluded from PSA validation files.

In addition to difficulties determining pretrial detention status and linking jail and state data, data quality issues impeded validation in some sites. Data related to the FTA outcome posed the most significant challenges. In County D, the quality of FTA in court data in the state court system compromised validation of FTA scores. When an individual missed court in that state, the state court data system recorded the missed date only until the individual showed up to court again for the case, at which point the missed date was overwritten with the new date and the record of missed court was eliminated. In County E, documentation of FTA was indistinguishable from other warrants for arrest. As one judge explained,

In [County E], and mostly for the state . . ., if a person has notice to be in my court, say today at nine o’clock, and they don’t show up, I don’t charge them with a FTA. I issue a warrant for FTA and they’re arrested. So there’s not a good way to determine FTA for the PSA.

At the time of the last interview, the sheriff’s office was exploring a system of documenting whether warrants resulted from FTA or other causes, which they hoped would provide a more accurate source of FTA data.

Access to data elements was particularly limited in Counties E and F. In County F, data systems were plagued by high levels of missingness on all pretrial outcomes, the sources of which were undetermined. Because of these high levels of missingness, the external evaluator and the policy team conducted a “data diagnostic” to determine whether various court actors (e.g., court staff, community corrections, prosecution, and defense attorneys) could access more accurate data for PSA scoring that would inform a solution for the validation. This diagnostic had not produced a solution by the time of the last site interview. In County E, the site was never granted permission from the state government to access criminal history data for use in PSA validation. The site was able to obtain data from the jail and from the state Administrative Office of the Courts to use in understanding the criminal legal system in the county, but the court data had many inaccuracies and ambiguities that made it unreliable for validation.

All sites experienced delays in PSA validation due to at least one of the aforementioned barriers, and in some sites these delays contributed to a sidelining of the PSA as the policy team moved on to other reform efforts. One County F policy team member reported in 2023, “There’s been no conversation about [the PSA] except at these monthly policy team meetings where there’s 30 seconds where they’re like ‘We’re still figuring out how to validate this tool!’” In the meantime, the County F policy team shifted focus to work on an executive order to reform bail in the district, a project which they ultimately prioritized over the PSA. In County G, policy team members felt that they needed to delay work on other implementation tasks (e.g., finalizing the Violent Offense List and RCM, training scorers and quality assessors) while they waited on PSA validation results, and they temporarily lost momentum. After validation was completed, the policy team was able to regain momentum in its plans to implement the PSA.

PSA Scoring, Automation, and Performance Metrics

The second point in the PSA implementation process that depends on jurisdictional data capacity is the scoring of the PSA. PSA scoring requires real-time access to data on criminal history, FTA in court, and pending charges, and the capacity to automate the scoring process to minimize the potential for human error. Three of the four sites that adopted the PSA were ultimately able to access the required data and automate PSA scoring but did encounter a few barriers that led to delay. The County A point of contact described encountering challenges in integrating their case management system with their automation spreadsheet and discovering errors in the programming that computed the PSA score, requiring multiple interactions with court and county IT departments that were short-staffed. Counties B and C depended on the external designers of their pretrial case management software to automate and integrate the PSA with the system, but they had to plan the integration around a software system overhaul and update that was repeatedly delayed. In County D, the county developed a new system to compute the PSA score automatically from information entered by the pretrial services manager.

Sites that did not adopt the PSA during the study period varied in their capacity for PSA scoring and automation. At the beginning of APPR, County G was already contracting with a pretrial services provider with experience scoring and automating the PSA in other jurisdictions, and they were unlikely to encounter major barriers in PSA scoring and automation. However, the individual responsible for their jail’s data management system had to develop a system for identifying individuals who were being held pretrial and eligible for PSA scoring. They described that working with an old data management system made identification difficult:

Our jail management system is a very old system. It was really poorly designed, and a lot of the data were not clearly defined . . . When people ask, why is that person being held in jail? You have to tell them, well, which one do you want? He is held because he is waiting to be formally charged, or he’s being held for violation of his parole conditions, or he’s violating his no contact order, and all that stuff. So it’s difficult to determine which reason why the person is being held in custody.

In Counties E and F, on the other hand, accessing criminal history data was a significant barrier to PSA scoring. In County F, criminal history and FTA in court data were available but had to be accessed on a case-by-case basis, which was time-consuming. When the site began assessing the staff time required to score the PSA, one policy team member described attempting to score the PSA on test cases and needing about 45 minutes per case, which they felt was not feasible for the number of cases requiring PSA scoring in the district. They described,

We don’t have a structured way to get out information . . . We’re kind of stuck with what we have from the courts and the system that everything is put into. And that is not user-friendly. And it doesn’t give you the data that you need. So like, I don’t know about FTA in court rates. Which is sad. And then you think about, well, that should be easy enough to calculate, but then you start thinking about it and you have to draw it from multiple places and you’re still not getting a true set . . . I’m really fearful of, how do we make this work long-term? . . . When you’re having to rely on people to extract it and put it in, then you’re already behind the ball. I’m appreciating that the sheriff’s office is giving us much more data as of late than they’ve given us in a long time, but what happens if they don’t? It’s the only source we can get it from, and you really have to beg and plead to get it.

Later in the process, the policy team determined that pretrial staff would be able to access more accurate data in a more timely manner if they were granted a higher level of permission to access data within the state court system, specifically the level of access available to judges. At the time of the last site interview, options for obtaining this level of access were being investigated.

In County E, the policy team engaged in lengthy discussions with individuals managing the state criminal history data repository to allow their pretrial services department access to criminal history data for PSA scoring. The pretrial services director was finally granted this access 1 month prior to a planned soft launch of the PSA. However, although the site had hoped to utilize data extraction processes developed by the external evaluator to automate some of the data retrieval for PSA scoring, they did not have personnel trained in the software required for their continued use. Therefore, the newly hired pretrial services director planned to score the PSA manually during the soft launch for only every 10th individual in only one courtroom.

For sites that adopted the PSA, a final data requirement was the routine generation of performance metrics to monitor whether the jurisdiction is meeting their goals and objectives related to their pretrial population and the PSA. Sites may choose to monitor a number of performance measures, but key measures relevant to PSA implementation are related to the county’s pretrial population (the total number of unconvicted people booked into jail and who are awaiting trial for a new crime) and the PSA assessment rate (the number of people arrested and booked into the jail who are assessed with the PSA, divided by all people arrested and booked into the jail) (Arnold Ventures, 2020). County A reported virtually no barriers to generating performance metrics because they already had strong capacity for producing and relying on metrics for routine decision-making. Regarding metrics recommended by APPR, the County A point of contact reported,

Yes, most of them we were already tracking and so we changed the [metric] names to the APPR names. We may have added on 8-9 that were different that we hadn’t been doing before. We have always tried to use data to make informed decisions.

At two of the other sites that implemented the PSA, the most significant barriers to generating performance metrics were related to exporting the data entered into their case management system. In Counties B and C, their case management software was able to track the required pretrial and PSA data elements and automate scoring but lacked the functionality to export usable data for producing metrics. The main County B site contact described the situation by saying, “We have really good data! We just can’t get it out!” The fourth site that adopted the PSA, County D, was not utilizing performance metrics at the time of the last site interview.

Discussion

It is widely accepted that implementing innovations such as the PSA in real-world settings requires attention to the implementation context. In criminal legal system settings, implementing new practices is often difficult due to a lack of resources and insufficient organizational capacity, aspects of what Damschroder and colleagues refer to as the “inner setting” of the innovation in the CFIR (Damschroder et al., 2015). Arnold Ventures provides a roadmap of seven phases to assist jurisdictions in implementing the PSA, but in spite of the PSA’s relative complexity and increasing ubiquity, few research studies have described or evaluated its implementation. Such research is critical, as the PSA influences decisions about pretrial supervision conditions that impact individuals’ lives and public safety. Barriers to PSA implementation are not well understood and need to be researched as it is adopted more widely.

This study examined capacity factors influencing PSA adoption and implementation in seven jurisdictions and found substantial variation in jurisdictional capacity that affected whether the PSA was adopted and how it was implemented. Findings suggest that, generally, jurisdictions with preexisting infrastructure for pretrial services had better personnel and data capacity for PSA implementation than those with no preexisting infrastructure. In addition, local validation of the PSA, an important step in determining whether it is a valid and unbiased assessment tool for the jurisdiction, was a difficult and lengthy process for all study sites due to challenges in accessing the required data from multiple agencies, missing elements, or poor data quality. PSA validation was completed for five of the seven sites, but each encountered unique data capacity barriers. Arnold Ventures recommends ongoing validation of the PSA postimplementation, which is likely to be difficult without improved data capacity and research partners to perform the analysis, as the APPR evaluator did for these sites.

In this study, four jurisdictions had preexisting pretrial services programs, three of which adopted the PSA during the 5-year study period. Three jurisdictions did not have preexisting pretrial services programs, and although one of those jurisdictions did adopt the PSA, its implementation lacked fidelity as it did not include the use of performance metrics or QA. The two jurisdictions without preexisting pretrial programs that did not adopt the PSA made substantial progress toward building the personnel and data capacity to provide pretrial services, but still faced considerable obstacles to full PSA adoption and implementation.

These findings point to the potential difficulty of implementing the PSA without preexisting pretrial services, perhaps the most fundamental characteristic of these jurisdictions’ inner settings in relation to the PSA. Practically speaking, pretrial services programs perform two important roles in criminal legal systems: (a) gathering information about newly arrested individuals and assessing the appropriateness of available release options, and (b) supervising individuals and monitoring their compliance with release conditions (Mahoney et al., 2001). The sites’ preexisting pretrial services programs had dedicated personnel who could incorporate the PSA into existing duties, identify individuals in jail pretrial, access data needed for PSA scoring and performance metrics, and carry out release conditions ordered by the court. Jurisdictions without this preexisting infrastructure needed to build these capacities to implement the PSA.

In addition to the infrastructure provided by existing pretrial services, it is also possible that jurisdictions that have been able to establish pretrial services have built more capacity and consensus to implement an innovation like the PSA. According to the CFIR, important inner setting constructs other than infrastructure include relational connections, communications, culture, and mission alignment, all of which are likely significantly strengthened in the presence of pretrial services to facilitate implementation of the PSA (Damschroder et al., 2022). Pretrial services programs are typically established as the result of a cooperative effort between multiple local agencies and policymakers, and a program’s continued funding can indicate that pretrial release is a local priority. This sort of cooperation and collective prioritization around pretrial release may provide a more conducive environment for adopting further pretrial innovations, a phenomenon referred to in the implementation science literature as “clustering of technologies” (Rogers et al., 2014; Taxman & Belenko, 2011). With the support of technical assistance through APPR, all seven sites worked toward implementing pretrial initiatives such as court notifications, diversion opportunities, and the use of citations in lieu of arrest. Jurisdictions without pretrial services may wish to consider establishing them as a first step before implementing the PSA.

The implementation science literature has extensively demonstrated that personnel capacity is critical to an organization’s willingness to adopt an innovation and its ability to implement with fidelity. Innovations have been more likely to be adopted in settings with more personnel, more personnel with advanced training or education, more supervisory capacity, and less staff turnover (Levin et al., 2016; McCarty & Chandler, 2009; Taxman & Belenko, 2011). In this study, supervisory capacity impacted sites’ ability to conduct QA and the number of staff available for pretrial processes impacted sites’ ability to score the PSA. Interestingly, the PSA likely requires less staff time to implement than other pretrial assessments because it does not require interviews, meaning that staff capacity limit the implementation of other pretrial assessments even more (Desmarais et al., 2021). However, the PSA does require pretrial officers to gather information from multiple agencies, track down additional information for individuals with extensive criminal histories, and read and interpret scores. The time required for PSA scoring depends on data accessibility, the amount of automation, and staff experience, but has typically been measured at 5 to 10 minutes per case (K. Bechtel, personal communication, July 25, 2024). Jurisdictions must consider these staffing demands in preparing to implement the PSA.

Findings from this study also illustrate the challenges of implementing pretrial assessments in settings with limited data capacity. In its recent report on a 20-state review of criminal legal system data, the nonprofit organization Measures for Justice succinctly summarizes the main challenges with U.S. criminal legal system data: “In some cases, the data aren’t there—no one collects them. In other cases, the data are there but are protected by law or administrative discretion. Finally, the data are sometimes just too incomplete or messy to use” (Measures for Justice, 2021, p. 3). Each of these problems arose in this project as sites attempted to use state and local criminal legal system data to validate the PSA tool in their local population, score the PSA, and generate performance metrics. Even sites that ranked high in early assessments of data availability encountered challenges that caused delays. These experiences point to broader work that is needed to improve the usability of criminal legal system data.

PSA validation and scoring rely on accessing data from multiple agencies on criminal history, pending charges, history of court appearance, and age. These efforts were impeded at every site by the lack of a common identifier for individuals across data sources, requiring tremendous effort from site personnel and research partners to match cases based on imprecise criteria or exclude records that could not be matched. A recommendation from the Measures for Justice report is to build capacity to link records across agencies, which enables a far more accurate and sophisticated understanding of individuals’ criminal legal system trajectories (Measures for Justice, 2021). Although most states assign a common identifier to individuals who are sentenced to prison that are associated with them for all subsequent legal encounters, this is often difficult to implement and not available for individuals not sentenced to prison. The availability of a common criminal legal system identifier across agencies would improve the accuracy of PSA validation and scoring by ensuring that all relevant information for an individual is able to be identified and incorporated from multiple agencies. It would also increase the feasibility of validation and scoring for jurisdictions that lack trained data analysts or resources to pay an external research partner for many hours of data cleaning.

Local jurisdictions would also benefit from improved tracking of reasons why individuals are being held in jail and of FTA. Identifying the appropriate population for PSA validation and scoring was often difficult for sites, as was determining whether an individual had missed court. In the absence of substantial advancements in how systems track this information, jurisdictions may be wise to devote time to understanding and adjusting how these elements are tracked locally before beginning PSA implementation to avoid delays and loss of momentum.

This study has several limitations. First, the seven sites selected for this study are not representative of all jurisdictions attempting to implement the PSA. Because they were selected based on their willingness and capacity to implement the PSA and other pretrial reforms, they may actually have higher capacity than many other jurisdictions. Further research is needed to understand capacity for PSA implementation more broadly in jurisdictions across the United States. Second, the sites in this study worked with a research partner who supported them with local PSA validation and received extensive technical assistance for the other aspects of PSA implementation. These resources are not universally available to other jurisdictions seeking to implement the PSA and may have created unique experiences for these sites. Third, the activities in this study were impacted by the COVID pandemic, which may have increased challenges in personnel capacity beyond normal levels in the criminal legal system but may have also increased receptivity to reducing jail populations and use of pretrial detention.

Finally, the study may have additional limitations related to the study sample and qualitative methods. The sample of interviewees was selected based on individuals who were involved with PSA implementation at each site and recommended by the site policy team. Selecting from this group of involved individuals could increase the risk of social desirability bias in their responses if they wished to convey that implementation was successful. However, risk of social desirability bias is likely low because interviewees reported many implementation concerns and challenges. Regarding analysis, only one researcher applied qualitative codes to database and interview data, whereas many qualitative studies employ multiple coders to increase coding sensitivity and to reduce the impact of a single coder’s biases. Instead of using multiple coders, we involved the interviewers themselves in discussions to confirm the validity of emergent themes. Because the interviewers were also site liaisons, they had knowledge and insight into the topics being discussed and the most important factors related to PSA implementation and were able to provide valuable feedback on the accuracy and completeness of the coding. Another potential limitation was the use of a different lead interviewer at each site, which could limit standardization. However, the use of each site’s site liaison was an added strength of the study due to that person’s rapport with site points of contact and familiarity with site PSA activities. To ensure standardization, before each round of interviews all interviewers reviewed interview guides together to understand the priority and purpose behind each question, and site liaisons also served as note-takers on each other’s interviews so that they would be exposed to each other’s styles of probing and to the themes emerging across sites.

Conclusion

As pretrial assessments and the PSA specifically are being more widely adopted in jurisdictions across the United States, it is critical that jurisdictional capacity for implementing these tools be understood and enhanced to maximize their benefits. Pretrial assessments have potential to improve pretrial decision-making and pretrial outcomes, but they require local capacity in personnel and data that are often lacking in each. Findings from this study reinforce other implementation science research suggesting that attention to the inner setting of criminal legal system agencies is an important aspect of innovating in these settings. As agencies are increasingly expected to innovate and deliver evidence-based practices with fidelity, policymakers should support agencies with resources and support to strengthen their infrastructure, and researchers should conduct implementation science studies to elucidate the essential inner setting characteristics required for specific innovations.

Jurisdictions that wish to adopt the PSA should consider strengthening their data capacity to allow for PSA validation and scoring, most importantly incorporating a pretrial indicator into jail databases, establishing a common identifier across agencies, and developing a valid indicator of court non-appearance. They also must ensure that they have personnel capacity to both score and conduct QA on PSAs. Those without preexisting pretrial services may benefit from building that capacity first before adopting and implementing a pretrial risk assessment tool. Although the PSA as a tool is uniform across districts, the processes and changes required to implement it are unique to each setting and vary in the time and resources required. Efforts to understand and strengthen system capacity across systems have the potential to improve fidelity and increase benefits of using pretrial assessments to inform pretrial release decisions.

Footnotes

Authors’ Note:

The manuscript is the culmination of a more than 5-year study that benefited from the work of dozens of individuals who we would like to thank. The study would not have been possible without the patience and dedication of local and state criminal legal system officials who supported this research by providing access to their data and operations. Besides the legal system officials, we thank several RTI colleagues who were responsible for data management and processing, Kristin Bechtel for her thoughtful comments on prior drafts, and site liaisons Sarah Cook, Venita Embry, Ashley Lowe, Yamanda Wright, Monica Sheppard, and Erin Kennedy. We appreciate Virginia Bersch of Arnold Ventures for her support and encouragement. The authors, however, are solely responsible for all statements within the manuscript. This project was funded by Arnold Ventures. All the conclusions derived from this study are of the authors alone. The authors have no conflicts of interest to report.

ORCID iDs

Catherine A. Grodensky

Matthew DeMichele

Catherine A. Grodensky, PhD, is a justice researcher with RTI International’s Court Systems Research Program. Her research focuses on understanding the drivers of incarceration in state and local criminal legal systems and on evaluating programs that can improve both health and legal outcomes. She conducts program evaluation and implementation science research using mixed research methods.

Megan Comfort, PhD, is a senior fellow with RTI International’s Transformative Research Unity for Equity (TRUE). Her work centers on understanding pathways to promote equity, justice, and wellbeing for people involved with the criminal legal system and their loved ones.

James Trudeau, PhD, is a Principal Scientist at RTI International’s Justice Practice Area. He conducts evaluations of large-scale initiatives to reduce violence, bolster well-being, and improve prevention and intervention programs in schools and communities across the nation.

Elizabeth Tibaduiza, MA, is a research analyst at RTI International’s Center for Public Safety and Resilience. Her research focuses on programs that support vulnerable populations, such as victims of crime and people involved with the criminal legal system. Her substantive interests include equity in our health and legal systems, homelessness and housing, and community safety.

Matthew DeMichele, PhD, is the director of the Center for Criminal Legal Systems Research and has conducted criminal justice research on correctional population trends, risk prediction, terrorism prevention, extremist pathways, and program evaluation and methods. His research has recently been published in several outlets including Crime & Delinquency, American Sociological Review, and Criminology & Public Policy.

References

Arnold Ventures. (2020). Advancing Pretrial Policy & Research. https://advancingpretrial.org/

Arnold Ventures. (2021). Responding to the COVID-19 pandemic in the research-action sites: Common challenges and key strengths. Advancing Pretrial Policy & Research.

Brittain

B. J.

Georges

Martin

(2021). Examining the predictive validity of the Public Safety Assessment. Criminal Justice and Behavior, 48(10), 1431–1449. https://doi.org/10.1177/00938548211005836

Carroll

J. E.

(2023). The ever-shifting ground of pretrial detention reform. Annual Review of Law and Social Science, 19, 75–91. https://doi.org/10.1146/annurev-lawsocsci-120122-092655

Copp

J. E.

Casey

Blomberg

T. G.

Pesta

(2022). Pretrial assessment instruments in practice: The role of judicial discretion in pretrial reform. Criminology & Public Policy, 21(2), 329–358. https://doi.org/10.1111/1745-9133.12575

Damschroder

Hall

Gillon

Reardon

Kelley

Sparks

Lowery

(2015). The Consolidated Framework for Implementation Research (CFIR): Progress to date, tools and resources, and plans for the future. Implementation Science, 10(Suppl. 1), A12. https://doi.org/10.1186/1748-5908-10-s1-a12

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice. Implementation Science, 4(1), 1–15. https://doi.org/10.1186/1748-5908-4-50

Damschroder

L. J.

Reardon

C. M.

Opra Widerquist

M. A.

Lowery

(2022). Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): The CFIR Outcomes Addendum. Implementation Science, 17(1), 1–10. https://doi.org/10.1186/s13012-021-01181-5

DeMichele

Baumgartner

Wenger

Barrick

Comfort

(2020). Public safety assessment: Predictive utility and differential prediction by race in Kentucky. Criminology & Public Policy, 19(2), 409–431. https://doi.org/10.1111/1745-9133.12481

10.

DeMichele

Silver

Labrecque

Dawes

Lattimore

P. K.

Tueller

(2023). Testing predictive biases at the intersection of race-ethnicity and sex: A multi-site evaluation of a pretrial risk assessment tool. Social Science Research Network. https://doi.org/10.2139/ssrn.4432449

11.

Desmarais

S. L.

Monahan

Austin

(2022). The empirical case for pretrial assessment instruments. Criminal Justice and Behavior, 49(6), 807–816. https://doi.org/10.1177/00938548211041651

12.

Desmarais

S. L.

Zottola

S. A.

Duhart Clarke

S. E.

Lowder

E. M.

(2021). Predictive validity of pretrial assessments: A systematic review of the literature. Criminal Justice and Behavior, 48(4), 398–420. https://doi.org/10.1177/0093854820932959

13.

Escoffery

Lebow-Skelley

Udelson

Böing

E. A.

Wood

Fernandez

M. E.

Mullen

P. D.

(2019). A scoping study of frameworks for adapting public health evidence-based interventions. Translational Behavioral Medicine, 9(1), 1–10. https://doi.org/10.1093/tbm/ibx067

14.

Gleicher

(2017). Implementation science in criminal justice: How implementation of evidence-based programs and practices affects outcomes. Illinois Criminal Justice Information Authority.

15.

Goldstein

(2020). Using implementation science to transform policy into practice. Federal Probation, 84(2), 44.

16.

Gouldin

L. P.

(2020). Reforming pretrial decision-making. Wake Forest Law Review, 55, 857–906.

17.

Greiner

D. J.

Stubenberg

Halen

(2020). Validation of the PSA in Harris County, TX. Access to Justice Lab, Harvard Law School.

18.

Grodensky

C. A.

Abija

Joyce

Ariturk

(2022). “I don’t miss court just for nothing”: Understanding barriers to court appearance among criminal defendants in NC. In: Fall Research Conference, November 18, 2022. Washington, DC. Presented at Association for Public Policy Analysis and Management.

19.

Haigh

Preston

(2020). The court system in a time of crisis: COVID-19 and issues in court administration. Osgoode Hall Law Journal, 57(3), 869–904. https://doi.org/10.60082/2817-5069.3608

20.

Hamilton

(2015). Back to the future: The influence of criminal history on risk assessments. Berkeley Journal of Criminal Law, 20, 75–133.

21.

Hanna

Kubiak

Pasman

Gaba

Andre

Smelson

Pinals

D. A.

(2020). Evaluating the implementation of a prisoner re-entry initiative for individuals with opioid use and mental health disorders: Application of the CFIR in a cross-system initiative. Journal of Substance Abuse Treatment, 108, 104–114. https://doi.org/10.1016/j.jsat.2019.06.012

22.

Hopkins

Doyle

(2018). The pathways of pretrial reform. The Judges’ Journal, 57, 31–35.

23.

Kahneman

(2011). Thinking, fast and slow. Palgrave Macmillan. https://doi.org/10.4324/9781912453207

24.

Kirk

M. A.

Kelley

Yankey

Birken

S. A.

Abadie

Damschroder

(2015). A systematic review of the use of the Consolidated Framework for Implementation Research. Implementation Science, 11(72), 1–13. https://doi.org/10.1186/s13012-016-0437-z

25.

Lattimore

P. K.

Tueller

Levin-Rector

Witwer

(2020). The prevalence of local criminal justice practices. Federal Probation, 84(1), 28–37.

26.

LeMasters

Maner

Peterson

Brinkley-Rubinstein

(2022, January 21). Staff shortages in prisons and jails highlight opportunities for decarceration. Health Affairs Forefront. https://www.healthaffairs.org/content/forefront/staff-shortages-prisons-and-jails-highlight-opportunities-decarceration

27.

Levin

S. K.

Nilsen

Bendtsen

Bulow

(2016). Structured risk assessment instruments: A systematic review of implementation determinants. Psychiatry, Psychology and Law, 23(4), 602–628. https://doi.org/10.1080/13218719.2015.1084661

28.

Louie

Barrett

E. L.

Baillie

Haber

Morley

K. C.

(2021). A systematic review of evidence-based practice implementation in drug and alcohol settings: Applying the CFIR framework. Implementation Science, 16, 1–29. https://doi.org/10.1186/s13012-021-01090-7

29.

Lowenkamp

DeMichele

Warren

L. K.

(2020). Replication and extension of the Lucas County PSA Project. Advancing Pretrial Policy & Research.

30.

Ludwig

Mullainathan

(2021). Fragile algorithms and fallible decision-makers: Lessons from the justice system. Journal of Economic Perspectives, 35(4), 71–96. https://doi.org/10.3386/w29267

31.

Magnuson

Dezember

Kuehmeier

Green

Gautschi

(2023). Understanding court absence and reframing “FTA.” MacArthur Foundation Safety & Justice Challenge.

32.

Mahoney

Beaudin

B. D.

Carver

J. A.

Ryan

D. B.

Hoffman

R. B.

(2001). Pretrial services programs: Responsibilities and potential. National Institute of Justice: Issues and Practices.

33.

McCarty

Chandler

R. K.

(2009). Understanding the importance of organizational and system variables on addiction treatment services within criminal justice settings. Drug and Alcohol Dependence, 103, S91–S93. https://doi.org/10.1016/j.drugalcdep.2009.03.001

34.

Measures for Justice. (2021). The power and problem of criminal justice data: A 20-state review. https://measuresforjustice.org/wp-content/uploads/2023/01/The_Power_And_Problem_Of_Criminal_Justice_Data.pdf

35.

Metcalfe

Kuhns

J. B.

(2023). Coping with limited prosecutorial resources: An assessment of the case processing and community impact from the perspective of prosecutors and staff in a southeastern county. Criminal Justice Policy Review, 34(4), 337–360. https://doi.org/10.1177/08874034231163070

36.

Nam-Sonenstein

(2023). High stakes mistakes: How courts respond to “FTA.” https://www.prisonpolicy.org/blog/2023/08/15/fta/

37.

Nardulli

P. F.

Eisenstein

Flemming

R. B.

(1988). The tenor of justice: Criminal courts and the guilty plea process. University of Illinois Press. https://doi.org/10.2307/1143772

38.

National Institute of Justice. (2024). NIJ FY24 evaluating strategies to advance the implementation of evidence-based policies and practices. https://nij.ojp.gov/funding/opportunities/o-nij-2024-172002

39.

Pretrial Justice Institute. (2019). Scan of pretrial practices. https://www.pretrial.org/resources/scan-of-pretrial-practices

40.

Pretrial Justice Institute. (2020). The case against pretrial risk assessment instruments. https://www.pretrial.org/resources/the-case-against-pretrial-risk-assessments

41.

Pruitt

Davies

(2022). Investigating access to justice, the rural lawyer shortage, and implications for civil and criminal legal systems.In Weisheit

R. A.

Peterson

J. R.

Pytlarz

(Eds.), Research methods for rural criminologists (pp. 67–78). Routledge. https://doi.org/10.4324/9781003118657-6

42.

Rogers

E. M.

Singhal

Quinlan

M. M.

(2014). Diffusion of innovations. In Pedersen

P. M.

(Ed.), An integrated approach to communication theory and research (pp. 432–448). Routledge.

43.

Salisbury

E. J.

Sundt

Boppre

(2019). Mapping the implementation landscape: Assessing the systemic capacity of statewide community corrections agencies to deliver evidence-based practices. Corrections, 4(1), 19–38. https://doi.org/10.1080/23774657.2018.1522279

44.

Stemen

Escobar

(2018). Whither the prosecutor? Prosecutor and county effects on guilty plea outcomes in Wisconsin. Justice Quarterly, 35(7), 1166–1194. https://doi.org/10.1080/07418825.2018.1548637

45.

Stevenson

Mayson

S. G.

(2017). Bail reform: New directions for pretrial detention and release (No. 17-18). Penn Law: Public Law and Legal Theory Research Paper Series.

46.

Stirman

S. W.

Miller

C. J.

Toder

Calloway

(2013). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science, 8, 1–12. https://doi.org/10.1186/1748-5908-8-65

47.

Taxman

F. S.

Belenko

(2011). Implementing evidence-based practices in community corrections and addiction treatment. Springer Science & Business Media. https://doi.org/10.1007/978-1-4614-0412-5

48.

Van Brunt

Bowman

L. E

. (2018). Toward a just model of pretrial release: A history of bail reform and a prescription for what’s next. Journal of Criminal Law & Criminology, 108(4), 701–774.

49.

Van Deinse

T. B.

Zielinski

M. J.

Holliday

S. B.

Rudd

B. N.

Crable

E. L

. (2023). The application of implementation science methods in correctional health intervention research: A systematic review. Implementation Science Communications, 4(1), 149. https://doi.org/10.1186/s43058-023-00521-4

50.

VanNostrand

Lowenkamp

C. T.

(2013). Assessing pretrial risk without a defendant interview. Laura and John Arnold Foundation.

51.

Viglione

(2019). The risk-need-responsivity model: How do probation officers implement the principles of effective intervention? Criminal Justice and Behavior, 46(5), 655–673. https://doi.org/10.1177/0093854818807505

52.

Wilson

J. M.

Grammich

C. A.

(2024). Reframing the police staffing challenge: A systems approach to workforce planning and managing workload demand. Policing, 18, 1–12. https://doi.org/10.1093/police/paae005

53.

Zielinski

M. J.

Allison

M. K.

Brinkley-Rubinstein

Curran

Zaller

N. D.

Kirchner

J. A. E.

(2020). Making change happen in criminal justice settings: Leveraging implementation science to improve mental health care. Health & Justice, 8(1), 1–10. https://doi.org/10.1186/s40352-020-00122-6

54.

Zottola

S. A.

Desmarais

S. L.

(2022). Comparing the relationships between money bail, pretrial risk scores, and pretrial outcomes. Law and Human Behavior, 46(4), 277–289. https://doi.org/10.1037/lhb0000487