Abstract
Youth violence is a preventable public health issue. Few hospital-based programs intentionally focus on youth violence prevention. This project aimed to describe the Systematic Screening and Assessment (SSA) methodology used to identify existing hospital-based youth violence prevention (HBYVP) programs ready for future rigorous evaluation. To identify promising HBYVP programs currently in use and assess readiness for evaluation, data from the 2017 American Hospital Association (AHA) Annual Survey of Hospitals was used to identify hospitals with Level I-III trauma centers with reported HBYVP programs. Information for each program was gathered via environmental scan and key informant interviews. A total of 383 hospital-based violence prevention programs were identified. Two review panels were conducted with violence prevention experts to identify characteristics of programs suitable for an evaluability assessment (EA). Fifteen programs focused on youth (10–24 years old) and were identified to be promising and evaluable. Three of the 15 programs were determined to have the infrastructure and readiness necessary for rigorous evaluation. Lessons learned and best practices for SSA project success included use of streamlined outreach efforts that provide program staff with informative and culturally tailored project materials outlining information about the problem, project goals, proposed SSA activities, and altruistic benefit to the community at the initial point of contact. In addition, success of review panels was attributed to use of software to streamline panelist review processes and use of evaluation and data analysis subject matter experts to serve as panel facilitators. Communities experiencing high youth violence burden and hospitals serving these communities can improve health outcomes among youth by implementing and evaluating tailored HBYVP programs.
Keywords
Violence among youth is a significant and preventable public health issue: Youth violence is the intentional use of physical force or power to threaten or harm others by young people ages 10-24. It typically involves young people hurting peers who are unrelated to them and who they may or may not know well. Youth violence can include fighting, bullying, threats with weapons, and gang-related violence. A young person can be involved with youth violence as a victim, offender, or witness. (David-Ferdon et al., 2016)
In 2019, homicide in the United States was the fourth leading cause of death among youth aged 10–14 years and the third leading cause of death among youth aged 15–24 years (Centers for Disease Control and Prevention [CDC], 2021). It has also been reported that youth present to the emergency department (ED) for violence-related injuries daily and will possibly experience assault-related re-injuries or perpetrate violence against others within a few years of the first time they present to the ED for treatment (Cunningham et al., 2015).
Since 2014, the American College of Surgeons has required that all Level I, II, and III trauma centers implement injury prevention strategies (Drake et al., 2020). Though not a traditional aim of conventional hospital models, violence prevention is aligned with the common mission to improve the health of individuals and communities. Thus, an increasing number of hospitals have implemented violence prevention programs for individuals presenting to the ED with violence-related injuries and have partnered with community organizations to provide critical services for individuals who are at higher risk for recidivism and re-injury after discharge (Aboutanos et al., 2011; Health Research and Educational Trust [HRET], 2015). In an example of the successful implementation of a hospital-based youth violence prevention (HBYVP) program, a retrospective case–control study found that youth who participated in a hospital-based peer intervention program following a hospitalization due to violent injuries were 70% less likely to be arrested for any offense 6 months post-injury when compared with youth in the control group (Becker et al., 2004).
These HBYVP programs typically conduct needs assessments to identify gaps in available resources, use motivational interviewing techniques to help modify risky behaviors, conduct training to develop youth skills and risk awareness, and facilitate the connection of youth to community-based prevention services, resulting in reduced violence-related injuries and decreased future involvement in the criminal justice system (Becker et al., 2004; Cunningham et al., 2015). While several violence prevention programs have been implemented in various cities around the United States, most are not widely known and/or lack the resources that are necessary to conduct rigorous evaluation (Cunningham et al., 2012; David-Ferdon et al., 2016).
The Systematic Screening and Assessment (SSA) Method identifies innovations that real-world practitioners have developed and then systematically assesses which innovations will offer the greatest payoff from further evaluation, that is, helping to identify innovations most likely to be effective and increasing the potential for beneficial outcome evaluation (Leviton & Gutman, 2010). Correspondingly, evaluability assessment (EA), also known as exploratory evaluation, is used to help determine the extent to which an activity or project can be evaluated in a reliable and credible fashion, resulting in useful information (Organisation for Economic Co-operation and Development-DAC [OECD-DAC], 2010). This project aimed to apply the SSA methodology to identify existing HBYVP programs and describe the results of a review that used EA to assess existing HBYVP programs’ readiness for future rigorous evaluation.
Methods and Results
Guided by Leviton and Gutman (2010), this project systematically sequences SSA methodology, EA activities, and a review panel consisting of violence prevention experts to help identify the programs that have the most potential to reduce youth violence, as well as the capacity to document program activities and evaluate program outcomes. While EAs provide useful information about the feasibility and scope of an evaluation, they are just one of six important steps in the SSA methodology. Figure 1 summarizes the six-step methodology that guided this project. Additional details related to each step of this project are outlined below.
Overview of the Systematic Screening and Assessment Approach (Adapted From Leviton & Gutman, 2010)
Step 1: Establish Priorities and Program Inclusion and Exclusion Criteria
Programs selected for review were divided into two priority groups (i.e., “first priority” or “second priority”) based on the degree to which program goals and characteristics aligned with project focus and the potential to withstand rigorous evaluation. Further details related to priority groups and inclusion/exclusion criteria are outlined below.
Programs classified as first priority for inclusion in the review met the following criteria:
• Primarily focused on reducing/preventing: ○ Youth violence-related re-injury ○ Youth violence-related morbidity and mortality rates ○ Youth involvement in criminal justice system
• Initial contact with youth, ages 10–24, occurs in the hospital
• Program fully implemented for at least 1 year
• Monitored and measured youth violence prevention outcomes
• Included community-based partnerships or wraparound services to prevent re-injury and negative outcomes of violence
• Addressed disparities, the social determinants of health, and other circumstances that may contribute to violence
• Documented that services originated in a hospital setting
Programs classified as second priority for inclusion in the review met the following criteria:
• Prevention program primarily focused on an area that is related to, but not directly aligned with youth violence-related injury (i.e., sexual health and teen dating violence)
• Reported little to no violence prevention-related data collection activities
• Focused on coordinated nurse response rather than prevention outcomes
• Documented potentially limited capacity to partner on a rigorous evaluation (e.g., small staff and data extraction challenges)
Programs excluded from review met the following criteria:
• Identified as a community-based program (e.g., services originated in the community)
• Services originated outside of the hospital but delivered in the hospital (e.g., referral programs from community partners)
• Program implemented for less than 12 months
• Program no longer exists
• Program currently or previously involved in an outcome evaluation
Step 2: Identify Programs for Further Consideration
To identify HBYVP programs for inclusion in the review, we conducted a search of the 2017 American Hospital Association (AHA) Annual Survey of Hospitals to identify potential HBYVP programs. The AHA Annual Survey of Hospitals database represents information that is directly provided by hospitals and health care systems and covers an array of data points, including demographics, operations, service lines, staffing, expenses, physician organization structures, beds, utilization, and population health (American Hospital Association [AHA], 2017). A research analyst from the project team extracted and organized program details for all programs. Additional information for each program was gathered via environmental scan and other EA activities (i.e., interviews with key program staff).
There were 6,282 hospital organizations (hospitals or hospital systems) in the 2017 AHA database. Of the 6,282 hospital organizations included in the AHA database, 1,200 hospitals included a Level I, II, or III trauma center. Nine hundred of the 1,200 hospital organizations with a trauma center reported having an affiliated violence prevention program operating in or around the hospital. Thus, the universe of programs under consideration was comprised of 900 hundred violence prevention programs.
Of the 900 violence prevention programs reported to be affiliated with a Level I-III trauma center, there were 465 violence prevention programs that reported specifically operating within a Level I-III trauma center. Of the 465 programs operating within a Level I-III trauma center, 82 were duplicates and excluded from review. The remaining 383 programs were reviewed and classified into priority groups as outlined by the SSA methodology.
From this review of the 383 non-duplicated programs operating within a Level I-III trauma center, 105 programs were selected for inclusion in the review. Of the 105 programs included in the review, 57 programs were classified as first priority for inclusion in review as they focused on youth and child violence and matched the inclusion criteria (i.e., implementation for at least 1 year, monitoring and measurement of youth violence prevention outcomes, and inclusion of community-based partnerships or wraparound services). Forty-eight additional programs were classified as second priority for inclusion in the review as they did not have a primary focus on violence prevention, reported little to no violence prevention-related data collection, and focused on a coordinated nurse response rather than prevention. Additional information was not available for the remaining 278 programs that were excluded.
The team was able to reach 31 of the 105 first and second-priority programs identified via email or phone to verify and collect additional information. Programs that were not reachable tended to have outdated contact information listed and/or were no longer in operation. After initial outreach activities, 15 programs were confirmed to have met the inclusion criteria and were recommended to be included in the panel review. The other 16 programs were not recommended as they were either community-based programs with services that did not originate in the hospital, were implemented for less than 12 months, were currently involved in an outcome evaluation, and/or are no longer in existence. Figure 2 outlines the screening process used to identify HBYVP programs included in the review.
Identifying Programs for Further Consideration: Hospital-Based Youth Violence Prevention Program SSA Environmental Scanning Process
Step 3: Convene Initial Review Panel and Ratings Meeting
In preparation for the review panel, the team populated a program description form using information collected during the environmental scan. The program description form included details about the program’s location, delivery setting, number of youths served, priority areas, implementation duration, violence prevention data, target population, a detailed description of the program, type of data collected, and data collection method. Additional information included on the program description forms was gathered from discussions conducted with hospital staff at each of the 15 programs. Discussions with hospital staff were guided by formative questions used to help confirm that the programs were considered active and were providing services to the target population at the time of the review. Additional questions were used to help identify additional and knowledgeable point(s) of contact for each program, guiding theories/theory of change used by the program, and other important background information related to program development and implementation. The program description forms served as the primary reference materials for the reviewers during the initial review panel.
To reduce the likelihood of bias during the review process, team analysts ensured that all programs were de-identified, assigned a color pseudonym, and all potential identifiable information was removed from the forms shared with reviewers (e.g., program city, name of affiliated programs, and partnerships). This study received an exemption from the Office of Management and Budget responsible for reviewing federal regulations and handling proposals and communications under the Paperwork Reduction Act of 1980.
Two review panels were conducted with 12 violence prevention experts experienced in various aspects of HBYVP (e.g., program administrators, designers, facilitators, and researchers) to identify characteristics of programs suitable for an EA. Reviewers assessed and provided feedback for each program described in the program description forms using an online platform known as SurveyMonkey Apply (SurveyMonkey Inc., San Mateo, California). Team analysts also recorded meeting minutes during the review panels. The scoring guide used to rank the programs included six categories (i.e., program design, potential impact, evaluation capacity, sustainability, replicability, and transferability). Reviewers were provided with the scoring criteria/description for each category and asked to rank the program using a point system. Cumulative scores were used to rank the programs. Thematic analysis of the panel discussion notes was used to identify common themes that were shared during the panel session. These themes were reviewed by the group of panelists as a part of the member-checking process. The panelist suggested the following summary of characteristics as those to consider when selecting programs for an evaluability assessment (EA).
Characteristics for Strong Consideration for an EA
Programs that received the most favorable assessments of their characteristics were those that shared the following:
• Components that address social determinants of health and included community wraparound services
• Adequate number of trained staff to implement the program, maintain the caseload, and provide an opportunity for scaling the program
• Aims and goals that align with the established goals of the CDC initiative to reduce youth violence-related mortality, morbidity, and recidivism
• Focus population within the desired age range (10–24 years)
• Strong capacity for or demonstrated a history of evaluation
• Systematic tracking of violence-related outcomes (e.g., rates of re-injury, community referrals, and service utilization)
Characteristics for Limited Consideration for an EA
Programs that were not suggested for consideration for evaluation were those that shared the following:
• Poorly described details about the intervention components
• Misaligned focus areas (i.e., not aimed at reducing youth violence-related morbidity and mortality rates)
• Focus population outside the ages of 10–24 years of age
• Lacking outcome data related to re-injury and recidivism or capacity to monitor data related to re-injury and recidivism
• Limited or no follow-up or tracking of participant outcomes
Of the fifteen programs considered by the review panel, two programs were not highly ranked by any of the panel members; three programs did not capture youth violence prevention outcomes (e.g., rates of re-injury, community referrals, and service utilization) and did not have the organizational capacity to sustain an evaluability assessment; and one program was not selected due to its inclusion in an existing evaluation project, resulting in six programs not selected for EA. The remaining nine programs were selected for inclusion in the EA.
Step 4: Conduct Evaluability Assessments (EAs)
EA is a pre-evaluation activity designed to maximize the chances that any subsequent evaluation of programs, practices, or policies will result in useful information (Leviton et al., 2010). Nine programs were contacted to conduct an EA site visit. One program did not respond to numerous outreach efforts, resulting in eight programs included in the EA process. While EAs are traditionally conducted in-person, travel restrictions and safety concerns related to the COVID-19 pandemic resulted in a minor limitation, causing the project team to adjust the original approach and develop a virtual alternative. As a result, the EA site visits for this project were conducted virtually using the Zoom web conferencing platform (Gray et al., 2020).
Prior to each virtual visit, program materials were reviewed by the site visit team, followed by virtual site visit interviews, and virtual site visit debrief. During each virtual visit, at least two members of the project team assessed program implementation, data collection, and outcomes. All virtual EA site visit sessions lasted approximately 1 hour. Semi-structured guides were developed by the project team during study development and the guides focused on program design, potential impact, evaluation potential, sustainability, replicability, and transferability. Questions for the key informant interview guides were grouped into the following categories and sub-categories: (a) History (Program Description, Implementation Feasibility, Replicability/Transferability); (b) HBYVP Program (Program Design, Potential Impact, Implementation Feasibility, Sustainability, Replicability/Transferability); (c) Community Support and Partnerships (Reach to Target Population, Implementation Feasibility, Sustainability, Replicability/Transferability, Staff/Organizational Capacity); (d) Evaluation (Potential Impact, Evaluation Capacity, Options for Further Evaluation, Staff/Organizational Capacity); (e) Funding (Implementation Feasibility, Sustainability, Replicability/Transferability). EA project team members interviewed key program staff, including lead program administrators, partners, program staff, evaluators, and other stakeholders.
During each virtual EA site visit, two EA project team members took detailed notes of the program’s responses. Thematic analysis was used to group the notes into major categories and themes. Team members compared the content of their notes and themes identified. The notes were synthesized into a report identifying any areas of ambiguity. This summary report was shared with the programs after each virtual site visit as a part of the collaborative member-checking process. A preliminary logic model (based on the formative data from environmental scan activities) was drafted prior to each virtual EA site visit to describe the program’s resources, activities, and outputs. This logic model was reviewed and further refined with the program during the virtual EA site visit and debrief. EA team members then drafted a summary report, describing the program’s approach, major elements as planned and implemented, observations and findings from the team members, and the program’s evaluation potential, impact, and recommendations. The report also included the detailed updated logic model of the program and an appendix of reviewed documents. Programs were instructed to review the notes and logic model and provide clarifying details to the EA team within 2 weeks following the virtual interview.
After the summary report was verified by the program staff, EA project team members were responsible for de-identifying all program documents including any available annual reports and classifying information for each program using a color-coding system to classify programs before they were shared with the panel reviewers. This de-identified information, along with the initial program information, was made available to the review panel for a second review during Step 5.
Characteristics of violence prevention programs ready for a rigorous evaluation included a program champion, a strong connection to the community, warm referrals/handoffs to community resources, robust or tailored case management, and some experience with violence prevention, either through training or past lived experience. If programs were built on a violence prevention theory, it was either the Cure Violence model (Butts et al., 2015), SNUG (guns spelled backward and an adaptation from Cure Violence), or some other strengths-based prevention model designed to highlight and build on existing resources and strengths of the community and individuals served. All programs had some qualitative or quantitative data that could be used or optimized for future evaluation efforts. Overall, each of the programs’ features was tailored to their respective city or county depending on the community or hospital needs.
Step 5: Reconvene Panel to Review Programs and Rate EA Findings
Nine of the original panelists returned to review the EA findings for the eight programs that participated in the EA and provided feedback using SurveyMonkey Apply during a second virtual panel review meeting. In preparation for the virtual panel review, panelists reviewed program materials and resources (i.e., program brochures and overview documents, intake forms, data collection forms, available annual reports, and other program documents as available). After reviewing the information, panelists were asked to score the programs using the scoring guide and criteria used in the initial review (i.e., program design, potential impact, evaluation capacity, sustainability, replicability, and transferability). Like the initial review panel meeting outlined in Step 3, panelists highlighted shared characteristics of the programs that should be considered for further review and potential selection as a program with potential to be recommended for rigorous evaluation. Panelists’ scores and feedback from the initial and second panel reviews were extracted, compiled for thematic analysis, and shared with panelists as a part of the member-checking process during the panel session.
Across the eight programs, major themes and concepts discussed included the target population (the total number of clients served within the target population [youth 10–24 years old]); program goals (the program’s alignment with priority areas and outcomes); data infrastructure (the program’s capacity to collect and measure youth prevention); sustainability (the funding mechanisms for longevity of the program); and transferability/replicability (the strategies that could or could not be duplicated in different and similar settings).
Major areas for discussion included the program’s primary goals, data accessibility, service availability, and program duration. Some of the programs’ priorities were not within the scope of this project’s specific aims. For example, two programs did not offer a holistic approach to violence prevention (e.g., referral and access to community services, complementary and alternative therapies, and outpatient clinical services). Instead, only one aspect of a promising violence prevention program, such as mental health, was highlighted. Strengths of the programs included access to multiple data sources, such as the hospital’s trauma registry and/or electronic health records, and the ability to share data among these systems. Although three programs had access to these data, the ability to merge data across systems did not appear feasible without major modifications or resource investments. Seven programs implemented a client-centered approach and provided wraparound services to their clients. However, the client’s duration from enrollment to completion of the program was not clear or indefinite for all programs.
Step 6: Disseminate Information and Identify Programs for Future Rigorous Evaluation
Findings from the SSA methodology, including the EA site visits and two virtual panel reviews, identified three hospital-based programs ready for a more rigorous outcome evaluation. The remaining four programs revealed there were factors that indicated a strong readiness for evaluation, even if the program was not yet prepared. The remaining hospital-based program did not include factors that made them ready for an evaluation. These factors included:
• Commitment to evidence-based violence prevention
• Strong champions of the program within the hospital and community
• Strong data infrastructure that can be replicated and shared across data systems
• Intensive case management and mentorship that addresses risk factors
• Well-developed client-centered treatment plans
• Various sources of local and state funding
• Adequately staffed, including a commitment to staff’s professional development
• Access to and/or collecting community-level data
• Tracking of violent re-injury and re-arrests
• Inclusion of preliminary data outcomes in reducing recidivism
• Efforts to address the target population that match current project priorities
• Broad selection of wraparound services
• Infrastructure and components are based on an existing model
Factors associated with limited readiness during the virtual panel review included:
• Lack of standards for selecting and training essential program staff
• Unimplemented program components and aspirational outcomes
• Lack of outcome data and the necessary evaluation staff
• Underdeveloped data infrastructure leading to inaccessible data and/or limited data sharing
• Program mission was not clearly in alignment with priority focus areas and aims of reducing youth violence-related injury, involvement in criminal justice system, and morbidity and mortality rates
• Unclear number of clients served within the target population (youth ages 10–24 years old)
• No access to baseline data or criminal justice data
• Highly individualized plan for participants causing a decrease in program replicability and transferability
Discussion
Key activities and desired outcomes varied across the identified violence prevention programs. Overall, panelists believed that programs with strong evaluation potential could be replicated, sustained financially, and presented clear goals and outcomes for their programs. They placed a high priority on data collection and management, including long-term outcomes. They also displayed strong partnerships within the hospital and community. Those programs also focused on clients’ needs, providing access to a wide selection of services within the hospital and community. They include credible messengers and/or dedicated staff that have trained experience, and/or have lived experience with violence. These programs often build on existing HBYVP program models and make improvements, including the use of violence prevention theories and addressing social determinants of health (e.g., transportation, housing, and food insecurity).
The use of existing databases, environmental scans, review panels, and in-depth interviews with program staff resulted in a set of criteria that can be used to determine future investments in HBYVP programs and activities. Panelists recognized that some programs not yet ready for rigorous evaluation could benefit from partnering with researchers to bolster their data capacity. The SSA methodology combined with the use of subject matter experts to objectively review and provide insights about related programs is a valuable approach that can be used to help outline, dissect, and compare complex programs in practice and further facilitate the uptake of evidence-based practice and research into regular use by other practitioners, researchers, and policymakers. In addition, using the implementation science as a guide, this methodology has the potential to not only inform the field of hospital-based violence prevention but also to improve the public health response to reducing violence-related injuries and decreasing future involvement of youth in the criminal justice system overall.
Lessons Learned
To ensure that outreach to HBYVP programs is successful, it is important to provide informative SSA project materials at the initial point of contact. Providing the program points of contact with a project factsheet that used accessible and culturally appropriate terminology to outline information about the problem, project goals, proposed activities, and altruistic benefit to the community was a successful strategy for recruiting promising HBYVP programs for this project.
Additionally, successful recruitment of blinded subject matter experts for the project review panels was attributed to (a) the use of SurveyMonkey Apply to streamline panelist review and rating of the materials, (b) the use of an experienced panel facilitator with knowledge of program evaluation, and (c) the accommodation of panelist schedules. Panel members agreed to participate in both the initial and final panel review to ensure consistency in review. This strategy helped ensure the reliability of both review panels. Due to the time between panel review sessions, there was some attrition due to organizational changes.
Implications for Practice
The capacity for future rigorous evaluation should be a significant consideration during program planning and design. Findings of this review indicate that there is a need to educate hospital-based program designers on the importance of including multi-level evaluation (i.e., formative, process, and outcome) at every stage of program design, from program planning to program implementation and dissemination of findings. The CDC’s Framework for Program Evaluation in Public Health states that program evaluation should be “practical, ongoing. . .and involve all program stakeholders, not just evaluation experts” (Milstein & Wetterhall, 1999). Programs that do not incorporate these elements, including the creation of evaluation plans at the beginning of program development, risk implementation of programs that do not have the infrastructure necessary to identify areas for improvement, or ability to demonstrate a return on investment.
Conclusion
The purpose of this project was to use the SSA methodology to identify innovative HBYVP programs focusing on youth ages 10–24 that were evaluable, scalable, and had the potential to improve youth health outcomes. Given that HBYVP programs exist in health care settings and few programs have been rigorously evaluated (Cunningham et al., 2012; David-Ferdon et al., 2016), the results of this review can be used to identify characteristics of the most promising HBYVP programs, as well as the performance measures and indicators necessary for the completion of rigorous evaluation to support continual program improvement and long-term sustainability.
Footnotes
Authors’ Note:
Theresa Armstead and Arielle Arzu who provided project guidance and technical support. Program staff from the 15 hospitals that provided their time to help the project team understand their violence prevention program. American Hospital Association for their provision to access their hospital survey database. Panel members who reviewed the programs and provided critical feedback. Study was funded by the National Center for Injury Prevention and Control, US Centers for Disease Control and Prevention (# GS35F393AA and task order #75D30119F06945).
Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
