Abstract
We applied a three-step process, abstracting and analyzing program budgets to examine how Colorectal Cancer Control Program (CRCCP) awardees are structuring their programs and to assess the fidelity of program design to the CRCCP public health model. We reviewed 23 state, one tribal organization, and six university awardee budgets. We assessed resource allocations, staffing structures, and contracted partners and their activities. Awardees allocated 83% of all funds to contracts and personnel. Program managers were the most budgeted personnel type across three measures: number of people, full-time equivalency, and personnel costs. Awardees not only contracted with health care systems and clinics (39% of all contracts) but also contracted other partner types. Contractors were mainly funded to implement evidence-based interventions (25%) and conduct evaluation (24%). Program design varied among awardees in the number of staff (0–22), number of full-time equivalencies (0–5.4), and the number of contracts (1–11) budgeted. State awardees budgeted more resources to contracts, compared with university awardees (57% vs. 31%), while universities budgeted more for total personnel costs (41% vs. 30%). We learned that awardees designed their programs with fidelity to the CRCCP model. Although implementation approaches varied, overall results suggest implementation requires a combination of internal capacities and contracted partners. Budgets provide opportunities to use already existing program data to evaluate program design, partnerships, and planned activities.
This is a visual representation of the abstract.
Keywords
Background
In the United States, colorectal cancer (CRC) is the third leading cause of cancer-related deaths and the third most prevalent type of cancer, among cancers affecting both men and women (United States Cancer Statistics Working Group, 2019). Despite the availability of several screening tests proven effective in reducing CRC morbidity and mortality (Bibbins-Domingo et al., 2016), only 68.8% of adults aged 50 to 75 years are currently up to date with CRC screening recommendations (Division of Cancer Prevention and Control, Centers for Disease Control and Prevention [CDC], 2020).
In 2015, CDC funded the 5-year Colorectal Cancer Control Program (CRCCP) DP15-1502 (CDC, National Center for Chronic Disease Prevention and Health Promotion, and Division of Population Health, 2015) to increase clinic-level CRC screening rates. The CRCCP funds 23 state health departments, six universities, and one tribal organization. The CRCCP aims to increase CRC screening using a health systems change model (https://www.cdc.gov/cancer/crccp/). Awardees collaborate with health systems to implement at least two evidence-based interventions (EBIs) from the Community Guide (https://www.thecommunityguide.org/) in primary care clinics.
Purpose
CDC wanted to learn how awardees structured and managed their programs to achieve health systems change, as there is no prescription on resource allocation. CDC was also interested in exploring the use of budget data in program evaluation. Hence, we conducted a formative assessment to answer the following question:
How are CRCCP awardees structuring and managing their programs?
Results answering this question are presented herein. Moreover, this methodology presents a unique application of budget data for formative evaluations.
Method
We analyzed Program Year 2 (PY2) awardee budgets (July 2016 to June 2017). Budget data, rather than expenditure data, were used because budget data are detailed and readily available, whereas expenditure data are limited and unavailable until after a given program year. In PY1, CDC provided extensive technical assistance to help awardees understand the health systems change model. Thus, PY2 budgets better represent how CRCCP awardees are structuring and managing their programs under the new model. Analysis incorporated 29 of 30 awardee budgets; one budget was unavailable. Due to the nature of the data, no human subjects’ approval was necessary.
We used a three-step process to assess program structure and management: (1) data abstraction, (2) creation and assignment of predefined classification types for personnel, contractors, and contractor activities; and (3) descriptive analysis.
Step 1: Data Abstraction
Using a Microsoft Excel-based data abstraction tool (Microsoft Office Professional Plus 2016), one author abstracted funding amounts and narrative justifications for every line item in the budget by nine mandatory budget categories. Additional data abstracted for staff persons included job title, job description, and full-time equivalency (FTE) level. Additional data for contracts included the contractor name, contract amount, and the description of contract activities.
A second evaluator verified 79% of all data abstraction entries, identifying and correcting one error. The remaining data were validated using cross-checks built into the abstraction tool.
Step 2: Development and Assignment of Classifications
Next, we reviewed personnel job descriptions and created 10 distinct personnel types (see Personnel Types, Supplemental Material 1). Every person (excluding staff in contracts) was assigned the personnel type that most represented their efforts, as described in the budget. Applying a similar method, we created seven distinct contractor types (see Contractor Types, Supplemental Material 2). We assigned all contractors a contractor type that most represented the contractor based on the contract organization’s name. We conducted web searches to learn more about an organization for categorical assignment if the name was insufficient to assign a contractor type. Finally, we reviewed all contract descriptions to develop 11 distinct contract activity types (see Contract Activities, Supplemental Material 3). Because awardee budgets did not delineate the proportion of resources (time or funds) devoted to any single activity and because a single contract often identified multiple activities, we individually classified all activities in every contract.
Two evaluators worked independently to make all the classifications. Evaluators compared results, discussed any disagreements, and collaboratively determined final assignments. Interrater reliability scores for personnel, contactors, and contract activity assignments were 82%, 74%, and 62%, respectively. Evaluators applied a high-matching standard for contract activities such that assignments had to correspond exactly on all activities within each contract.
Step 3: Descriptive Analysis
To holistically assess staff resources, we combined personnel salary and fringe into a new category, total personnel cost. Descriptive statistics were conducted using two units of analysis, the CRCCP in aggregate (all 29 awardees) and the individual awardee level. Resource allocations (funds and personnel time) and categorical assignments were compared across awardees and to the CRCCP mean and median. At the CRCCP aggregate level, we also examined variations in resource allocations comparing state to university awardees.
Results
In PY2, CDC awarded $18,862,877 to 29 CRCCP awardees (average $650,444, median $702,774, range $403,265–$816,388). Most funds were budgeted for contracts and total personnel costs (Figure 1). Within total personnel costs, 73% of funds represented personnel salary and 27% were for fringe benefits. Ten percent of total funds were budgeted for indirect expenses, with approved indirect rates ranging from 0% to 50%, per awardee.
CRCCP Budget Allocations by Budget Category
In total, awardees budgeted for 202 staff representing 77.3 FTEs (rounded to the 10th decimal) with CRCCP resources. The number of staff and FTEs budgeted per awardee ranged from 0 to 22 and 0 to 5.4, respectively. Average FTE time per staff person was 0.4.
In aggregate, awardee staff were most often classified as program managers, directors, and evaluators (Table 1). All but one awardee (97%) funded a program manager and most awardees also budgeted a director and an evaluator (Table 1). Program managers were prominent across three staffing measures: number of FTEs, percentage of total FTE time, and percentage of total personnel costs (Table 1). In contrast, directors and evaluators played a less prominent role in number of FTEs and total personnel costs (Table 1). Awardees also frequently budgeted administrative support and fiscal operations staff (Table 1).
Grantee Staff by Personnel Type
Note. Source: CRCCP PY2 approved budgets, N = 29. FTE = full-time equivalency; PY = program year; CRCCP = Colorectal Cancer Control Program.
Every awardee also budgeted at least one contract. Nearly 80% of contract funds were budgeted among three contractor types: health care systems and clinics, clinical care support organizations, and public health-focused nonprofit organizations (Table 2). Twenty-three awardees budgeted for contracts directly with health care systems and clinics, 15 awardees did so with clinical care support organizations, and 14 with public health focused nonprofit organizations.
Number of Contracts and Consultants and Percentage of Total Contract and Consultant Funding by Grantee Type
Note. Source: CRCCP PY2 approved budgets; all CRCCP grantees N = 29, state grantees N = 22, university grantees N = 6. PY = program year; CRCCP = Colorectal Cancer Control Program.
There were a total of 258 contract activities. Four contract activity types represented nearly three quarters of all contract activities: EBI implementation and support (25%), evaluation and performance measurement (24%), program assessment and planning (12%), and patient navigation (12%). Twenty-three awardees (79%) had at least one contractor conducting EBI implementation and support, 24 awardees (86%) had at least one contractor conducting evaluation and performance measurement, and 16 awardees (55%) had at least one contractor conducting patient navigation.
Under the CRCCP model, EBIs must be implemented in health care systems and clinics. Results show that 65% of EBI implementation and support activities, 74% of program assessment and planning activities, and 74% of patient navigation activities were budgeted with contracts to health care system and clinic. In contrast, non-EBI related activities were budgeted among non–health care system and clinic contractor types, including 88% of professional development activities, 77% of small media, and 75% of community engagement activities. Interesting, 93% of EHR improvement activities were conducted by non–health care systems and clinics suggesting health care systems and clinics need outside technical expertise to improve EHRs.
Interesting differences were observed between university and state awardees. University awardees received approximately $42,500 less, on average, than states and differed in the proportion of funds budgeted at the aggregate CRCCP level (Figure 1). Among staff, while both state and university awardees budgeted, on average, an equal number of staff people (seven), universities budgeted more FTEs and more program managers than states (Table 1). Within contracts, university awardees budgeted a larger proportion of contract funds directly to health care systems and clinics compared with state awardees (Table 2). Similarly, university awardees had almost twice as many contract activities with health care systems and clinics compared with state awardees.
Discussion
This formative evaluation provided insight on how CRCCP awardees are structuring and managing their CRCCP programs. Results show that awardees are aligning their program structure with the CRCCP model by budgeting staff and contractors to conduct key program activities including assessment and planning, EBI implementation, and evaluation. Awardees consistently funded program managers to lead the day-to-day implementation and management of their CRCCP programs, with directors and evaluators also playing important roles.
We observed unique management models by examining resource allocations. First, there was a clear difference in the amount of resources allocated between personnel and contractors, especially between state and university awardees. Hiring or other challenges facing public agencies may explain states’ reliance on contractors (Rainey & Bozeman 2000). Given the importance of contracted partners in the CRCCP, awardees may benefit from innovative public management approaches such as network governance (Cristofoli et al., 2014) or collaborative governance (Lubell, 2015). Diverse partnerships require managers to address unique challenges, tensions, or competing interests. Program managers should remain cognizant of accountability issues (Ryan & Walsh, 2004) and consider using performance management techniques (Koppenjan, 2008) to increase the likelihood of achieving program outcomes.
We also identified differences in how awardees are staffing their programs. There was a wide range in the number of staff used and average FTE time, per awardee, allocated to the program, which may suggest different approaches toward implementation. University awardees relied more on internal staff, budgeting for nearly a third more FTEs (in aggregate) than state awardees, which were primarily program managers. Anecdotal evidence suggests that almost no additional in-kind university staff contributed to EBI implementation. For example, only one in-kind staff member was identified in university budgets, an administrative assistant. University awardees’ reliance on internal staff suggests they may have more capacity to support CRCCP implementation, while states need to engage contractors for some types of expertise.
There are limitations to this process. Budgets are fiscal allocations and not known expenditures. However, we are confident that budget data are sufficiently valid to address our question because CDC must approve resource reallocations greater than 25% of the total budget or from one budget category to another, which were accounted for in this analysis. Second, in-kind resources were excluded from the analysis because CRCCP awardees do not need to identify in-kind resources and there is no match requirement for funding. However, among budgets, nine awardees committed $247,558 in in-kind contributions or 1% of the overall CRCCP funds awarded, suggesting in-kind contributions were minimal. Finally, CRCCP awardees operate in diverse socioeconomic, geographic, cultural, and organizational environments. Considerable deviations in resource allocations may be influenced by these contextual factors.
Future research could validate the methods used here by exploring how resource allocations vary among programs of different scope, including those outside of public health. Additionally, research should be undertaken to understand what different skills are beneficial for successful implementation among different organizational types, depending on their resource allocation.
Implications for Policy and Practice
The methods used here allowed CDC to explore whether awardees adopted the CRCCP’s health systems change model. Results were disseminated to CDC program consultants to inform their technical assistance provided to CRCCP awardees. Results were also shared with awardees, which gave them the opportunity to consider their program structure and management approach and consider how resource allocation is related to achieving program outcomes. Other nonpublic health programs can apply this method to understand whether a program model is being generally adopted and assess the skills important to program implementers. State awardees reliance on contractors suggests some limits in their capacity and/or challenges securing some position types.
Historically, program budgets have been used as a fiscal planning and accountability tool (Melitski & Manoharan, 2014). We learned that budgets can also be effectively used in formative evaluation to understand how awardees are structuring and managing a new program like the CRCCP, identify areas for further investigation (e.g., outliers), and identify where technical assistance may be needed. Finally, we have learned that this methodology is easily replicable and has broad utility.
Supplemental Material
sj-docx-1-hpi-10.1177_15248399211028150 – Supplemental material for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program
Supplemental material, sj-docx-1-hpi-10.1177_15248399211028150 for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program by Justin Uhd, Amy DeGroff and Krishna Sharma in Health Promotion Practice
Supplemental Material
sj-docx-2-hpi-10.1177_15248399211028150 – Supplemental material for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program
Supplemental material, sj-docx-2-hpi-10.1177_15248399211028150 for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program by Justin Uhd, Amy DeGroff and Krishna Sharma in Health Promotion Practice
Supplemental Material
sj-docx-3-hpi-10.1177_15248399211028150 – Supplemental material for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program
Supplemental material, sj-docx-3-hpi-10.1177_15248399211028150 for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program by Justin Uhd, Amy DeGroff and Krishna Sharma in Health Promotion Practice
Supplemental Material
sj-pptx-1-hpi-10.1177_15248399211028150 – Supplemental material for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program
Supplemental material, sj-pptx-1-hpi-10.1177_15248399211028150 for Money Matters: A Three-Step Process for Using Budget Data in Program Evaluation to Assess the Design and Management of a Novel Public Health Program by Justin Uhd, Amy DeGroff and Krishna Sharma in Health Promotion Practice
Footnotes
Authors’ Note:
The authors would like to thank the following people for their invaluable support and assistance in this research: Cynthia French, MA, PPM, Lindsay Gressard, MPH, MEd, and Katherine Ross, MPH. This research was supported in part by an appointment to the Research Participation Program at the Centers for Disease Control and Prevention administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy and Centers for Disease Control and Prevention. The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
