A Process for Identifying Indicators With Public Data: An Example From Sexual Violence Prevention

Using Public Data for Program Evaluation

Public data, such as existing administrative and surveillance system data, may be a good option for health agencies and other organizations to examine trends in SV indicators for the purpose of program evaluation. We define public data as information previously collected and analyzed, and available online (e.g., in a published report) such as a raw number, percentage, proportion, or rate. Three main advantages of public data, which address some of the potential challenges that practitioners might face, are data are readily available online, there is no cost for data acquisition, and data are already analyzed in a usable format. Because public data can be easily accessed by health agencies, if they know it is available, the utility of using such data to evaluate the impact of public health programs and strategies is worth exploring.

Distinct from our definition of public data, publicly available data sets may be accessed by the public but require additional steps to obtain (e.g., through data use agreement) or utilize (e.g., analysis required). Publicly available data sets have been used to evaluate public health programs in areas such as child welfare and chronic disease. One example is the Mother and Infant Home Visiting Program Evaluation–Strong Start, a multiyear large-scale evaluation study using vital records and Medicaid data in combination with a program participant survey to assess the effects of home visiting programs on birth and maternal health outcomes (Lee, Crowne, Faucetta, & Hughes, 2016; Lee, Warren, & Gill, 2015). Researchers note that the use of publicly available data sets allows for access to a vast amount of rich data, ongoing data extraction without participant follow-up, reduction of cost related to data collection, minimization of bias (e.g., recall bias), and the ability to conduct analyses with statistical confidence due to the large sample size (Feeney, Bauman, Chabrier, Mehra, & Woodford, 2015; Green et al., 2015; Lee et al., 2015; Lorden et al., 2016). The use of publicly available data sets for research evaluation illustrates the potential for health departments to use public data for their program evaluations.

Another reason public data may be a viable option for resourced burdened organizations is the movement toward data transparency. Federal initiatives to make existing data available to the public have increased over the past decade. Since the Office of Management and Budget issued an Open Government Directive in 2009, departments and agencies have been working to increase their availability of data to the public (Orszag, 2009). In addition, the Evidence-Based Policymaking Commission Acts of 2016 and 2018 further this effort by recognizing the potential of using existing data to conduct evidence-based research (Evidence-Based Policymaking Commission Act of 2016; Evidence-Based Policymaking Commission Act of 2018). Anticipating a further increase in the availability of public data, identifying indicators and data sources that health agencies and other practitioners could use to meet certain program evaluation and surveillance needs may be a feasible option to addressing the previously mentioned data challenges.

In this article, we describe a framework and a process for identifying indicators with public data for the purpose of conducting program evaluation. Specifically, we present the SV Indicator Framework and the five steps of the indicator review process we conducted for CDC’s RPE program. We present the findings of the indicator review, which include descriptive information on indicators and characteristics of data sources included in the review. We also explain how the process could be used by evaluators and program planners within other developing topic areas to identify indicators with public data.

CDC’s SV Indicators Project

CDC funds all 50 state health departments as well as four territorial health departments and Washington, DC, through the RPE program, to address SV using a public health approach (CDC, n.d., 2004). The public health approach uses data to describe the problem and identify priority populations, develop and implement prevention strategies, and monitor and evaluate program impacts (Basile et al., 2016; Mercy, Rosenberg, Powell, Broome, & Roper, 1993). In addition to using a public health approach, CDC organizes SV risk and protective factors, as well as strategies to address such factors, within a social-ecological model (SEM) (Dahlberg & Krug, 2002; Sallis & Owen, 2015). An SEM explains the complex interplay of individual, relationship, community, and societal-level factors that put an individual at risk of SV perpetration or help to prevent SV victimization or perpetration (Dahlberg & Krug, 2002). RPE recipients use the SEM to guide their state-level prevention strategies and their program activities. Recipients fund local health departments, state coalitions, local rape crisis centers, and other community organizations to implement prevention strategies that address SV risk and protective factors across the SEM.

RPE started at a time when little was known about what works to prevent SV (DeGue, Simon, Basile, & Yee, 2012). Today, evidence linking prevention strategies to SV behavioral outcomes is still somewhat limited. DeGue et al. (2014) identified only a few primary prevention strategies with rigorous evidence of effectiveness for reducing SV behaviors in their systematic review of 30 years of research. More recent evaluation studies have added to the evidence for SV prevention strategies (e.g., Coker et al., 2017; Espelage, Low, Polanin, & Brown, 2015; Salazar, Vivolo-Kantor, Hardin, & Berkowitz, 2014); however, there are still considerable gaps in research on the effectiveness of community-level interventions, in part due to limitations in defined community-level measures of risk and protective factors (Armstead, Wilkins, & Doreson, 2018). In 2016, CDC published a technical package called STOP SV, which included strategies with research evidence on SV risk and protective factors, in addition to SV behavioral outcomes (Basile et al., 2016). A technical package is a core set of strategies to achieve and sustain substantial reductions in a particular outcome based on the best available evidence (Basile et al., 2016; Frieden, 2014). While STOP SV advanced what is known about what works to prevent SV, CDC needed to identify outcome indicators for STOP SV strategies that were feasible to measure in practice versus research contexts.

RPE recipients had consistently reported to the CDC program team that there were limited data sources for SV behaviors, particularly perpetration, that they did not know how to assess which state and local data sources had appropriate indicators, and they did not always have the capacity to collect data from program participants. In addition, both CDC and RPE recipients knew there were few known indicators to measure community factors associated with SV. Recipients needed to identify indicators with public data they could use, in addition to conducting primary data collection. CDC’s Division of Violence Prevention, therefore, initiated the SV Indicator Project to identify indicators RPE recipients could use for their program evaluation activities as well as CDC’s own evaluation of the RPE program.

Process for Reviewing Indicators With Public Data

We developed a five-step process to gather stakeholders, define search parameters and selection criteria, conduct the indicator search, select indicators, and apply findings to public health actions such as program evaluation (Figure 1, Table 1). Step 1, Gather Stakeholders, includes identifying stakeholders who can serve as experts on the topic area, the specific program or strategy involved, program evaluation, research, and surveillance, as well as key leaders and decision-makers. It also involves developing a plan and identifying resources needed to conduct the search. Step 2, Define the Search Parameters and Selection Criteria, includes defining inclusion and exclusion criteria for indicator selection and determining search parameters for the external review process. Step 3, Conduct the Indicator Search, starts with reviewing any internal program or other known data sources based on search criteria. After obtaining stakeholder input on existing gaps in known data sources, a search of external data sources is conducted based on the defined search criteria. Step 4, Select Indicators, consists of stakeholders reviewing indicators and data sources that met the defined inclusion criteria for overall quality and fit with defined outcomes (i.e., face validity), as well as the credibility of data sources. This step may also include developing additional criteria to prioritize “best” indicators, if the indicator search yields multiple indicators for a single outcome or if stakeholder reviews identify issues with some indicators or data sources, for example, data quality or credibility of data sources. Stakeholders can develop additional criteria to prioritize indicators for the end uses and audiences, illustrated as a feedback loop in Figure 1. Step 5, Apply Findings to Public Health Actions, can involve integrating findings into practice guidelines, program evaluation, surveillance activities, and research efforts. This step also can include an analysis of the extent to which indicators cover the defined outcomes (i.e., content validity), where there are gaps that need to be filled by other data sources and methods, and the range of indicators, for example, from least to most severe SV behaviors. Because this process is intended for practice settings and is developmental in nature, stakeholder reviews throughout the process provide some rigor in applying criteria and making selections. An organization, however, could incorporate additional checks or more rigorous methods to assess validity, reliability, and credibility, if this fits the organizational values and would ensure findings are used.

OPEN IN VIEWER

Figure 1.

Process for identifying indicators with public data.

OPEN IN VIEWER

Table 1.

Five Steps to Identifying Indicators With Public Data.

Step	Purpose	Key Actions	RPE Example
1. Gather Stakeholders	Convene a group of individuals who can provide input on indicator search parameters, selection and use, and feedback at each stage, as well as influence use of indicators for evaluation and planning	Identify stakeholders who can serve as experts on topic area, program/strategy, evaluation, research, and surveillance Identify key leaders and decision-makers Develop a work plan and identify resources needed to conduct the search	CDC formed an SV indicator work group with practice, research, evaluation, and surveillance staff who have SV expertise and experience with the RPE program CDC hired a fellow to conduct online searches, coordinate the process, and manage work group member reviews
2. Define the Search Parameters and Selection Criteria	Align the indicator search with the program/strategy priorities, goals, and outcomes	Define inclusion and exclusion criteria for indicator selection Determine search parameters and external review processes Develop prioritization criteria (optional)	The SV indicator work group defined inclusion criteria for indicators: must fit within the SEM, must have public data, must measure an outcome that is modifiable, and must have data for all or most states. The work group also identified exclusion criteria for data sources The work group determined that the end users would be RPE recipients and CDC staff and the primary end use would be to increase tracking of state-level SV indicators
3. Conduct the Indicator Search	Identify indicators with public data that meet search parameters and selection criteria	Identify and review available information, such as stakeholder expert knowledge on indicator data sources and internal organizational documents Conduct external search	The work group used their own knowledge and external SV experts to identify known data sources and search parameters The fellow conducted online searches for additional sources. After several iterative cycles, work group members decided they had reached saturation for public data sources
4. Select Indicators	Determine whether to use all indicators or prioritize “best” indicators	Use criteria to include or exclude indicators Develop additional criteria to prioritize Identify stakeholders to review indicators (validity check)	The fellow organized the list of indicators by SEM levels and applied inclusion criteria Work group members reviewed the indicators and data sources for completeness, data quality, and credibility
5. Apply Findings to Public Health Actions	Use indicators to measure program/strategy context and impact as well as planning efforts	Determine the strengths and weaknesses of available indicators as well as the extent to which they address program goals and outcomes Incorporate indicators in program evaluation and planning Identify additional uses such as surveillance and research	The fellow conducted an analysis to assess which risk and protective factors were most covered and gaps. The fellow developed a database and guidance documents to facilitate recipient use of indicators, and the program team conducted webinars and provided training to recipients. Indicators were also shared with the Division’s research and surveillance staff Twelve RPE recipients were funded to use state-level indicators with public data in their evaluation activities

Note. RPE = Rape Prevention and Education; CDC = Center for Disease Control and Prevention; SV = sexual violence; SEM = social-ecological model.

Step 1: Gather Stakeholders

CDC’s Division of Violence Prevention established an internal SV indicator work group consisting of practice, surveillance, research, and program evaluation experts across the Division. The work group initially was formed to address the needs of RPE program recipients and CDC’s program team to measure outcomes for RPE program strategies. The work group identified data needs, defined search criteria, and provided ongoing input and review throughout the process. Program and evaluation staff members provided input on the program priorities and goals and helped the group think strategically about what types of indicators would serve future program evaluation goals for the next funding cycles as well as addressing current needs. Research and surveillance staff provided their subject matter expertise on SV, shared their knowledge of measurement and data sources, and served as expert reviewers throughout the process, along with program staff. Although the work group was internal to CDC, its members considered recipients to be an important stakeholder group and the primary end user for the indicator review findings. Recipients were updated on the work group’s progress and provided feedback at different points in the process.

Step 2: Define the Search Parameters and Selection Criteria

The work group first examined two systematic reviews of SV behavioral outcomes and risk and protective factors (DeGue et al., 2014; Tharp et al., 2013). In addition, we reviewed the World Health Organization report on violence prevention, SV Chapter, which also includes a review of SV-related outcomes (Dahlberg & Krug, 2002). The work group identified 20 SV outcomes, which included SV behavioral outcomes and risk and protective factors. We used the SEM as an organizing framework for SV risk and protective factors for SV and to guide the review and selection process (Bronfenbrenner, 1977; Sallis & Owen, 2015). Figure 2 illustrates how SV risk and protective factors as well as SV behavioral outcomes were organized within the SEM. As stated earlier, CDC and the RPE program commonly use the SEM to help explain the complex factors that put an individual at risk of SV perpetration or victimization and create the conditions to prevent SV. Subject matter experts on the work group reviewed the framework to ensure that the risk and protective factors were organized appropriately and to assess whether there were any additional factors that needed to be added based on new empirical research. At the time of the review, there were no additional risk or protective factors identified. The resulting CDC SV Indicator Framework (Framework) was used to guide the search of public data sources and the selection of indicators.

OPEN IN VIEWER

Figure 2.

Centers for Disease Control and Prevention Sexual Violence Indicator Framework. Dashed border indicates the indicator across the social-ecological model levels. ⁺Framework informed by empirical work (Basile et al., 2016; Tharp et al., 2014).

Inclusion criteria for indicator selection were (1) fits within the Framework—that is, an identified risk or protective factor or SV behavioral outcome, (2) measures a modifiable factor (i.e., can demonstrate changes overtime), (3) data available publicly at the state level on an ongoing basis (e.g., annually), and (4) the data collected were clearly defined by the data collector. This last criterion was important for the work group to determine the extent to which an indicator measured the defined outcomes and the populations included in the sample.

Exclusion criteria for data sources, and by extension indicators from these data sources, were (1) did not provide public data, (2) did not provide data at the state level, and (3) did not have indicators that fit within the Framework. We also excluded data sources that had publicly accessible data sets but required steps for acquisition (e.g., data use agreement) or analysis (e.g., data file available for download), sources that did not conduct ongoing data collection (e.g., onetime survey data), and sources that only provided data for one or a few states or only collected non-U.S. data. This last criterion was important because RPE is a national program with 55 recipients, and CDC sought to identify indicators that could be measured across all recipients.

Step 3: Conduct Indicator Search

We conducted a search of data sources (e.g., administrative data, surveillance systems) from the fall of 2015 through spring 2016 based on the defined search parameters described in Step 1. As a starting point, we reviewed several resources to identify potential data sources including indicator databases (e.g., Health Indicator Warehouse), peer-reviewed literature, evaluation studies, government data systems, and other established indicator projects (e.g., Healthy People 2020). We also gathered input from work group members and external subject matter experts in SV on potential data sources. Because this was not a traditional literature review where key search words could be entered into defined search engines, we used an iterative process that combined expert input and review with online searches to increase our coverage of public data sources.

In total, 76 public data sources were reviewed and 26 were included in our indicator review. Sixteen data sources were excluded because they did not provide state-level data, five data sources were excluded because they required users to download or request a data file for analysis, and two sources did not have indicators that fit in the Framework. The SV indicators work group determined that another 27 excluded data sources would not meet the RPE programs’ data needs, for example, only had data for a few states.

Step 4: Select Indicators

We reviewed the 26 data sources to identify and select indicators that could be used as measures of SV. Indicators meeting the minimum inclusion criteria (described in Step 1) were reviewed by the SV indicators work group during the spring of 2016 for final selection. Work group members provided input on data quality, gaps in data sources, and additional considerations for RPE recipients’ indicator selection. Administrators of data sources were contacted, when necessary, to obtain additional information on data collection methods. The work group also decided that they did not have enough information to determine which indicators were the “best performing” or would best measure RPE program strategies given the early developmental stage of RPE’s evaluation activities. Work group members, therefore, did not create additional prioritization criteria to pare down the list of identified indicators.

Step 5: Apply Findings to Public Health Actions

CDC’s Division of Violence Prevention applied a number of public health actions primarily intended to accelerate RPE recipients’ ability to identify, select, and track indicators. We developed an MS Access database for RPE recipients with all indicators and data sources, as well as descriptions of each and search variables (e.g., SEM level, data collection intervals). We also developed guidance on selecting appropriate indicators that align with prevention strategies and outcomes for recipients’ program evaluation, informed by other indicator guidance (CDC, 2016b; MacDonald, 2013). The indicator database was shared with the research and surveillance units within the division to inform their work (e.g., research funding opportunities), as well as future pilot work to identify a core set of SV indicators that could potentially be tracked by all RPE recipients. In 2016, the RPE program initiated 2-year supplemental funding for 12 of its recipients to pilot the usability of the SV indicator database and guidance for the purpose of tracking their state SV indicators. The pilot also assessed how useful these resources were in measuring state-level outcomes recipients included in their evaluation plans (Ottley et al., 2019). In 2018, the indicators and data sources included in the database were reviewed and updated, and in 2019, CDC launched an online search tool as part of its existing online portal Veto Violence.

Limitations

Although this review resulted in numerous indicators, there are notable limitations to this approach. First, because the priority was for the work group to identify indicators that did not require primary data collection or analysis—that is, public data, our results were limited and in particular did not include additional indicators we identified but excluded from the reported results. Some sources that were excluded due to requiring additional steps to obtain data (e.g., downloading a data file) may be of value to health departments, particularly those that have the expertise to manage this process. There may be other valuable data sources for individual states that were not included in our findings, such as state-specific surveys or surveillance systems, because they did not meet the purpose of the SV indicator review. We also did not rate or score indicators using additional criteria such as content validity (e.g., coverage of the range of SV behaviors from less to more severe), frequency of data collection, sensitivity to change, reliability of the measures, and credibility to stakeholders. Although the work group considered applying additional criteria to identify the “best performing indicators,” we ultimately decided that this was a first and formative phase. The work group, therefore, decided to first examine the utility and feasibility of RPE recipients tracking indicators with public data to measure the outcomes of their selected strategies during the current funding cycle before determining the best indicators. As such, the list of indicators should be considered a preliminary list based on the best available information at this time. In practice, RPE recipients have been able to use the SV indicator database to identify an initial set of indicators and then, in some cases, identified some additional (e.g., state-specific) data sources they could access for similar indicators. For example, one state that does not participate in CDC’s YRBSS data collection conducts their own state-level “Health Youth Survey.” This search was limited to address a specific resource gap for the national RPE program and could have been expanded or narrowed for another purpose or gap. This points to the importance of Step 2 in the Framework—Define Search Parameters and Selection Criteria.

We also found that within the year of our review, some data collectors implemented or planned changes, which points to a need for ongoing monitoring of updates. In particular, changes in data collection methods and online locations for data reports may present challenges for practitioners that rely on public data for ongoing evaluation. Changes in any public data being used for program evaluation may result in methodological challenges while assessing program outcomes. Last, public data are only one part of demonstrating the impact of SV prevention strategies. Practitioners, therefore, would likely need to combine tracking indicator data with other evaluation methods to show overall the contribution of their programs to changes in rates of SV. We also recognize that, while the risk and protective factors included in the Framework have empirical support, selected indicators are not necessarily the same measures used in previous studies. Future work is needed to determine the extent to which these indicators are useful to measure the impact of prevention efforts.

Despite these limitations, this review underscores the importance of research and surveillance efforts for programmatic supports. Our findings suggest that a significant amount of data to measure indicators of SV is already available for use by public health agencies that may have limited capacity and resources for data collection. Perhaps, the most significant impact of the SV indicator review to date is that the review findings and related guidance were incorporated in the most recent funding cycle, which started in 2019. For the first time, all RPE recipients are implementing strategies from CDC’s STOP SV technical package and tracking state-level indicators using public data. This significant advance in the RPE program will likely influence the SV field overall and potentially contribute to practice-based evidence (Green, 2008) on the implementation of prevention strategies in state and local contexts. It also is an opportunity to examine how useful the identified indicators with public data are to state program evaluations.

The methods and process used here could also be replicated to identify indicators with public data to measure other health outcomes. Indicators with public data available could potentially be used for both programmatic activities (e.g., decision-making) and evaluation activities (e.g., developing a baseline). Current and future requirements for open access to data collected from the public could increase data-informed decision-making at the state and local level. It also has the potential to increase available evidence for what works to prevent various health outcomes by providing a low-cost, low-burden method to conduct evaluation of prevention strategies.