Developing and Applying an Evaluative Rubric to Support Continuing Education for Disease Intervention Professionals

Developing the Evaluation Rubric

To identify high-quality resources, the project team developed a rubric to assess multiple attributes, including relevance of content, best practices in test taking, and accessibility. The team developed the rubric through a deliberative process between subject matter experts at IU with final input from ASPPH project managers beginning in March 2025. The IU team, with agreement from the ASPPH project team, finalized the rubric during a meeting on May 30, 2025.

Research and online learning experts each proposed criteria for the draft rubric based on their knowledge area. The research team inventoried the Centers for Disease Control and Prevention’s (CDC’s) Quality eLearning Checklist ⁸ domains (Analysis, Interactivity, Interface and Navigation, Content Development, Photographs and Graphics, and Learner Assessment) and criteria. Next, the research team used Claude AI (https://claude.ai) to identify additional aspects of online training for evaluation and entered results into the spreadsheet with CDC’s domains to compare, contrast, and merge evaluative domains and criteria from both sources.

The online team contributed to the rubric design based on their expertise in using Quality Matters ⁹ for online course evaluation. However, the framework could not be used in its entirety because of time and resource constraints and a misalignment in objectives. Specifically, the Quality Matters framework provides a comprehensive evaluation for individual courses with up to 9 substandards in each of 8 domains covering topics such as grading policies and learner interaction.

From the compilation of domains and subtopics, the research team began an iterative process to reduce the number and complexity of criteria by targeting goals of the DI repository project and by considering the time and resources needed to conduct evaluations. The research team submitted a draft rubric to the principal investigator on May 6, 2025, for further refinement. The resulting rubric went to the CDI project team at ASPPH for final vetting and approval on May 21, 2025.

The final rubric included 10 criteria: Course Overview and Introduction, Learning Objectives, Learning Assessment and Activities, Instructional Materials, Social Awareness, Scenario-Based Learning, Timeliness of Content, Course Technology, Learner Support, and Accessibility and Usability.

Applying the Rubric

In April 2025, the research team hired MPH and PhD students to apply the rubric using a data dictionary that contained each criterion, its definition, and an example as well as the inclusion and exclusion criteria. The research team applied the rubric (Table) and entered the results using REDCap ¹⁰ (Research Electronic Data Capture), a secure, web-based software platform designed to support research data capture.

OPEN IN VIEWER

Table.

Evidence-informed rubric developed to assess materials for the Certified in Disease Intervention training repository ^a

Criterion	Description
Course overview and introduction	• What to do first is clearly explained. • The course overview describes the purpose, format, and what to expect. • Key information such as schedule, communication, and grading is easy to find.
Learning objectives	• Learning objectives are shown at the beginning of the course and in each module. • Learning objectives use clear action words such as “explain” or “analyze.” • Learning objectives are written from the student’s perspective and describe what students will be able to do.
Learning assessment and activities	• Activities and assignments match the learning objectives. • Grading criteria (eg, rubrics or clear instructions) are provided. • It is clear how students will be evaluated.
Instructional materials	• Materials match the learning objectives. • Content fits the level of professional learners. • Materials are clear and helpful and keep learners engaged.
Social awareness	• Content prompts learners to consider various regional practices or local context when applying course content in practice. • Materials use current, respectful, and inclusive language.
Scenario-based learning	• Content uses scenarios or examples to integrate theory with practical application. • Content uses scenarios that reflect real-world situations that disease intervention professionals encounter.
Timeliness of content	• Content is consistently current, reflecting the latest medical practices, public health guidelines, and terminology.
Course technology	• Technology helps learners reach course objectives. • Tools make learning more interactive and engaging. • Tools support participation and real learning (not just watching).
Learner support	• It is easy to find out how to get technical help. • There are multiple ways to ask for help (eg, email, telephone). • Support responds in a helpful and timely way.
Accessibility and usability	• The course is easy to navigate. • Text is readable, with good layout and contrast. • Images, videos, and audio include helpful features such as alternative text, captions, or transcripts.

a

All criteria were rated using a 3-point scale, where 1 = good, 2 = okay, and 3 = poor.

An initial list of training resources collated by ASPPH during an initial assessment of the DI training landscape was provided to the IU project team. The project team created a systematic data collection framework to identify additional resources in collaboration with the IU Bloomington Libraries. Recognizing the evolving nature of online training, the research team revisited the resource sites listed on NBPHE’s compilation of CDI study resources from April through July 2025 to capture newly released trainings. Resources were also identified through listening sessions with content developers and frontline DI professionals. Finally, the research team searched links and references from the identified resources to identify other relevant courses and platforms. In August 2025, the research team observed that most search results were yielding duplicate resources and shifted its focus to prioritizing the evaluation of materials.

Starting in May through August 2025, the research team evaluated each training resource against the rubric criteria using a 3-point rubric scale: good (3 points), okay (2 points), or poor (1 point). The research team also had the option to select not applicable (0 points) if a rubric category did not pertain to the resource. For example, prerecorded webinars may lack learning assessments or learner support features. Similarly, a course focused on a specific virus might be highly science oriented—centered on laboratory results and medications—and may not address regional practices or local context. In such cases, scoring the resource on social awareness would not be appropriate. To ensure comparable outcomes, scores were converted into percentages using an automated formula that excluded any not applicable selections, preventing those categories from lowering the overall score.

The research team and the IU project director met weekly during the evaluation process to deliberate unclear findings, clarify data definitions, and refine processes. In addition, the research team prioritized materials for review in response to the project timeline and pace of the review process.

Resource Format

Most of the trainings to be included in the repository were online courses (n = 120; 64.2%). Approximately one-third were prerecorded webinars or videos (n = 64; 34.2%). Only 3 (1.6%) were resource materials (eg, factsheets, infographics, toolkits) that the research team included in the analysis because they had continuing education credits assigned to them.

Mapping to CDI Examination Domains

Using the DIS Certification Job Task Analysis Validation Survey developed by NBPHE and currently pending final approval, the educational resources were mapped to the CDI examination domains when applicable (Figure 2). Distribution across domains was relatively even, with Interviewing and Case Management slightly higher (n = 59; 31.6%), followed by Planning and Case Analysis (n = 43; 23.0%), Field Services and Outreach Activities (n = 44; 23.5%), Surveillance and Data Collection (n = 42; 22.5%), Collaboration (n = 39; 20.9%), and Outbreak Response and Emergency Preparedness (n = 40; 21.4%). A total of 49 trainings (26.2%) did not fall into any CDI examination category.

OPEN IN VIEWER

Figure 2.

Number of educational resources (N = 233) mapped across Certified in Disease Intervention (CDI) examination domains during assessment using an evidence‑informed rubric. Resources were evaluated from May through August 2025 by a research team at the Indiana University School of Public Health–Bloomington using a standardized approach to evaluate material for a CDI training repository. Counts across domains are not mutually exclusive and may exceed the total number of resources assessed.

Topic Area

While the CDI examination categories were based on skills, the project team also wanted to capture topic areas, because workforce members may be looking for trainings in specific diseases or topics that are not explicitly in the CDI examination. These were topics such as effective communication, public health policy and advocacy, basics of epidemiology, motivational interviewing, leadership skills, infectious diseases such as COVID-19 and mpox, and enteric disease investigations. HIV and STI trainings were the most common topic areas (HIV: n = 40 [21.4%]; STI: n = 49 [26.2%]), while tuberculosis (n = 20; 10.7%), hepatitis (n = 24; 12.8%), infectious outbreak investigation (n = 26; 13.9%), and emergency preparedness and response (n = 24; 12.8%) accounted for the rest of the topic areas.

Level of Difficulty

Most of the educational resources reviewed were introductory-level trainings: Introductory (n = 124; 66.3%), Intermediate (n = 20; 10.7%), Advanced (n = 3; 1.6%), and All Levels (n = 40; 21.4%).