Designing a Peer Observation of Teaching Form: A Modified Delphi Study

Abstract

An important aspect of conducting peer observation of teaching (POT) is the observational form that guides the observation process. Although several published POT forms are available for faculty to use, these forms tend to be unique to the academic discipline, with none originating from the perspective of faculty in the field of public health. As such, the purpose of this study was to create a new POT form grounded in the published literature that could be used for in-person classes for a newly established Department of Public Health. This study used a modified Delphi method with 10 faculty experts in the department, ranging in academic title and years of experience in teaching, observing others’ teaching, and being observed as a teacher. A foundation of 657 relevant items found in 33 published POT forms were used for six phases of anonymous surveys, group discussions, and pilot testing. The final POT form consisted of 22 new items that represented over 30 categories of established, observable teaching practices, and a dichotomous measurement scale with open ended text boxes. The study underscored previous research regarding the utility of the Delphi method, which included allowing for instructor buy-in on the items that would be used to evaluate their own teaching practices.

Keywords

Delphi method faculty development instructor evaluation or feedback pedagogy

Introduction

In 1994, the American Association of Higher Education (AAHE) made a noteworthy step toward promoting peer review of teaching at the college level. Specifically, the AAHE sponsored the project “From idea to prototype: The peer review of teaching” to promote peer review and enhance faculty members’ teaching skills (Hutchings, 1995). Twelve universities from across the United States participated in the project and included “teams” of two faculty members per department from a variety of academic disciplines (e.g., chemistry, mathematics, English, history, music, business, engineering, nursing) to develop various peer review practices (e.g., reviewing course syllabi, observing teaching; Gabaccia, 1996).

During that same timeframe, the University of North Carolina (UNC) System also made an important decision regarding peer review of teaching. Sparked by a controversial tenure decision at the University of North Carolina at Chapel Hill, the Board of Governors questioned how teaching was evaluated in the tenure process (Gabaccia, 1996). Based on recommendations from committee members to improve faculty teaching practices, the Board of Governors required that all non-tenured faculty be directly observed during classroom teaching (University of North Carolina System, 1993).

In the decades following the AAHE and UNC System’s promotion of peer observation of teaching (POT), research studies found benefits to teaching observations at the college level. Several reviews of the published literature indicated that POT can improve college instructors’ knowledge of teaching practices, confidence in teaching, adoption of new teaching skills, and critical self-reflection of teaching (Corcelles-Seuba et al., 2025; Cutroni & Paladino, 2023; Johnston et al., 2022; Thomas et al., 2014; Zeng, 2020). In addition, POT can provide benefits at the departmental level, including the establishment of minimum teaching standards, professional development of junior faculty, and improved collegiality among peers (Corcelles-Seuba et al., 2025; Cutroni & Paladino, 2023).

An important aspect of conducting POT is the observational form that guides the observation process. Based on the popular “collaborative model” of POT, a faculty member is observed by their peers in a non-judgmental manner to promote faculty discussion, reflection, and improvement of teaching (Gosling, 2014). The collaborative model of POT tends to consist of three phases (G. A. Martin & Double, 1998). First, during the pre-observation meeting, the faculty member being observed (i.e., the “observee”) and those conducting the observation (i.e., the “observers”) discuss the context of the course, the focus of the observation, the day and time of the class session to be observed, and which POT form will be used for the observation. Second, the POT takes place in which the observers unobtrusively watch the class and complete the POT form. Third, during the post-observation feedback meeting, the faculty observee and observers use the completed POT form to discuss what was observed and to provide constructive criticism of how the observee’s teaching practices could be improved (G. A. Martin & Double, 1998).

Given that observational forms are a vital component during the second and third phases of POT (G. A. Martin & Double, 1998), choosing a POT form that best fits the course that is being observed is an important step in the POT process (Siddiqui et al., 2007). The literature suggests that there are several POT forms that have been published and are available for faculty to use (Dillon et al., 2020; Otero-Saborido et al., 2024); however, these forms tend to be unique to academic disciplines, especially in the fields of science, technology, engineering, and mathematics (i.e., STEM fields; Dillon et al., 2020). Although public health borrows from a variety of disciplines, it is a unique field that is grounded in the teaching of others, which has warranted the need for its own dissemination of scholarship of teaching and learning (Gambescia, 2015), including a POT form for faculty in the field. The POT form developed by Jia et al. is perhaps most relatable to the field of public health, as it was developed for faculty at an inter-professional school of health professions; however, that POT form was specifically designed for evaluating faculty who teach online courses (Jia et al., 2025). Other POT forms have been developed to assess faculty who teach in any discipline, but those forms have been limited by their specific focus (e.g., inclusive teaching practices [Addy et al., 2023], teaching in classrooms physically designed to enhance active learning strategies [Birdwell & Harris, 2022], teaching techniques to elicit student participation [Nunn, 1996]), and their use of excessive assessment items (Murray, 1983), which could make POT a cumbersome task (Otero-Saborido et al., 2024).

As such, the purpose of this study was to develop a POT form based on the perspectives from faculty in the field of public health. In July of 2025, our institution formed a new Department of Public Health. As part of being a new department, the faculty were required to develop their POT protocols per UNC System requirements, which includes determining which form would be used for POT. Rather than using an existing POT form originating from a different academic discipline, or general POT forms with a narrow focus, the faculty aimed to create a new POT form for in-person classes that might be useful for those teaching in the field of public health, and yet also grounded in the published literature.

Methods

This project was conducted as a modified Delphi study. The Delphi methodology was developed during the Cold War by the United States Air Force in collaboration with the RAND Corporation, in which seven experts were asked to estimate the number of bombs that the Soviet Union needed to eliminate strategic targets in the United States (Dalkey & Helmer, 1963). Aiming to reach a consensus, several phases of anonymous surveys were used to gather the experts’ opinions, with the survey results provided to experts at each phase to help inform their answers based on the opinions of the other experts. By using anonymous surveys, rather than in-person discussion, the study avoided direct confrontation among experts, and allowed for independent thought by reducing the potential for experts to persuade each other’s opinions (Dalkey & Helmer, 1963).

The Delphi method, and modifications to the method, have since been used by scholars from several academic disciplines to determine a level of consensus among experts on a variety of issues. Researchers from a wide array of disciplines have made use of the Delphi method, including education (Green, 2014) and health sciences (Nasa et al., 2021; Schifano & Niederberger, 2025). Although the Delphi method has been frequently modified to fit the topic and purpose of a study, common elements of a Delphi study include selecting several experts on a topic, conducting iterative phases of anonymous surveys, providing experts with anonymous results from each survey phase, and reaching a determined level of consensus (Keeney et al., 2001; Nasa et al., 2021; Niederberger & Spranger, 2020; Schifano & Niederberger, 2025).

Selection of Experts

The Delphi “experts” for this project were the 10 current faculty for the new Department of Public Health, who ranged in academic title and years of teaching experience. Although additional research is needed to learn more about the optimal number of experts for Delphi studies, based on Shang’s (2023) narrative literature review on the topic, it is recommended that at least eight experts be included in a Delphi study. The faculty were considered experts based on their years of teaching at the college level, the number of times they have observed their peers’ teaching, and the number of times that their teaching was observed by their peers (Shang, 2023; Table 1). They were also considered to be representative experts, as they would eventually be using the POT form as observers and as observees.

Table 1.

Experts’ Academic Title, Years of College-Level Teaching Experience, Number of Times Serving as an Observer to Peers’ Teaching, and Number of Times Teaching was Observed by Peers.

Expert	Title	Years	Observer	Observee
1	Senior Lecturer	30	15	18
2	Teaching Assistant Professor	19	27	9
3	Associate Professor	15	11	7
4	Professor	13	4	2
5	Professor	12	14	10
6	Professor	10	7	1
7	Research Assistant Professor	8	4	8
8	Teaching Assistant Professor	5	8	6
9	Teaching Assistant Professor	5	1	2
10	Assistant Professor	3	1	1

Phase 1

The first phase of the Delphi study consisted of finding items that were previously used in POT forms. In January of 2025, the authors collected college-level POT forms that were published in the peer-reviewed literature. POT forms were located using the databases: PubMed, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), the Educational Resources Information Center (ERIC), and Google Scholar. To locate articles, the authors used the following search terms in various combinations: college, peer, observation, teaching, and form. Reference pages of relevant articles were searched to locate additional publications. POT forms were included in the study regardless of academic discipline or mode of instruction (e.g., in-person, online); however, forms were only included if they had been used at the college level, included items on instructor teaching practices, and published from scholars in the United States. A total of 33 published college-level POT forms were included in the project (Table 2).

Table 2.

Published POT Forms Used for the Modified Delphi Study.

1st author, year	POT form name	Discipline
Addy et al. (2023)	Protocol for advancing inclusive teaching efforts	Any
Barna and Reenan (2020)	Peer observation record	Music
Berk et al. (2004)	Peer observation scale	Nursing
Birdwell and Harris (2022)	Active learning classroom observation tool	Any
Blanco et al. (2011)	Peer feedback tool for lectures & small group teaching	Medicine
Blauvelt et al. (2012)	Clinical peer-review tool for nurse educators	Nursing
Bryan et al. (2018)	Supplemental teaching evaluation form	Library
Cobb et al. (2001)	Peer review of a web-based course	Nursing
Çoruh and Kritek (2017)	Large group teaching evaluation form	Medicine
Crabtree et al. (2016)	Peer observation and evaluation tool	Pharmacy
Davis (2011)	Report of classroom observation document	Pharmacy
Eddy et al. (2015)	Practical observation rubric to assess active learning	STEM
Fielden and Foster (2010)	Course integrated seminar teaching observation rubric	Library
Fletcher (2018)	Observation worksheet	STEM
Flick et al. (2009)	Oregon teacher observation protocol	STEM
Fox-Alvarez et al. (2022)	Formative assessment rubric for peer evaluation of teaching	Veterinary
Frankl et al. (2017)	Peer observation worksheet of case-based collaborative learning	Medicine
García et al. (2017)	Online classroom evaluation form	Library
Gaskamp and Kintner (2014)	Peer evaluation for formative assessment of online teaching	Nursing
A. H. Harris and Cox (2003)	VaNTH observation system global ratings category	STEM
Hassel et al. (2020)	Peer observation of small-group/clinical teaching	Veterinary
Hora (2015)	Teaching dimensions observation protocol	STEM
Jia et al. (2025)	Peer observation and evaluation tool-online	Health professions
Kumar et al. (2018)	Classroom observation evaluation rubric	Pharmacy
Litzelman et al. (1998)	Stanford faculty development program	Medicine
B. A. Martin and Martin (1989)	Behavioral Observation Scale	Business
Middleton (2002)	IS instruction & training checklist for observations	Library
Murray (1983)	Teacher behaviors inventory	Any
Newman et al. (2009)	Peer assessment of medical lecturing instrument	Medicine
Nunn (1996)	Teaching technique	Any
Sawada et al. (2002)	Reformed teaching observation protocol	STEM
Servilio et al. (2017)	Evaluation of classroom skills	Special Education
Smith et al. (2013)	Classroom observation protocol for undergraduate STEM	STEM

The items from each POT form were combined into a single document and placed into separate categories by the first author. Items were included in the document if they evaluated the instructor’s teaching practices and could be directly observable during a single class session. Items were excluded if they were meant to evaluate teaching practices outside of class (e.g., quality of syllabi, quality of homework assignments and rubrics, organization of online learning management systems [e.g., Blackboard, Moodle, Canvas]) or student behaviors during class (e.g., paying attention, taking notes, using cell phones, being distracted by others). A separate category included the different measurement scales that were used in the published POT forms. A total of 657 items were included in the study. The first author then grouped these items into a total of 60 different categories (Table 3) and were included in an anonymous, online survey. Approval from our institution’s IRB was then sought, and then deemed as “exempt” (study # HS-25-229).

Table 3.

POT Categories and Number of Items Selected Per Phase of the Delphi Study.

Categories	Items selected per phase
	1	2	3	4	5	6
Scale of observation form
Likert-based	18	11	5	3	†	*
Checklist/dichotomous	9	2	*	*	*	†
Open ended	6	1	1	1	†	†
Scale (i.e., 0%–100%)	1	0	*	*	*	*
Time management
Starting/ending class on time	4	4	4	3	†	†
Pacing instruction, dedicating time on task, etc.	14	11	7	4	†	†
Content
Giving reminders to students about dates/deadlines	2	2	1	1	†	*
Presenting learning objectives of class session	20	17	5	3	†	†
Presenting lecture outline of class session	9	5	2	1	†	†
Organizing lecture effectively	14	10	5	3	†	†
Transitioning clearly between topics	4	4	*	*	*	*
Defining key terms	5	4	2	*	*	*
Explaining major concepts	8	7	4	3	†	†
Explaining relevance of content	11	9	7	*	*	*
Connecting content to prior/future content	17	14	11	7	†	†
Using appropriate depth and breadth of content	13	9	5	5	†	†
Using accurate content	6	6	5	*	*	*
Using current content	5	5	3	1	†	*
Explaining content clearly	22	8	4	4	†	†
Using examples	22	19	10	5	†	†
Using quality visual aids	24	16	8	6	†	†
Promoting critical thinking, reflection	21	10	8	7	†	†
Preparing students for future exams	7	5	*	*	*	*
Providing additional resources for learning	8	8	3	2	†	†
Summarizing content during/concluding the class session	16	14	7	6	†	†
Questions and answers
Assessing students’ understanding at the start of class	10	6	3	*	*	*
Asking students questions throughout the class	36	13	7	5	†	†
Allowing enough time for students to answer questions	4	3	*	*	*	†
Responding to students’ answers	18	14	10	9	†	†
Asking if students have any questions	16	10	8	5	†	†
Listening to students’ questions	3	2	2	2	†	†
Responding to students’ questions	10	9	7	6	†	†
Incorporating students’ ideas into the class session	8	7	*	*	*	*
Being flexible with class session based on questions	3	2	*	*	*	*
Diverse teaching and active learning methods
Using diverse teaching strategies	13	10	6	5	†	†
Using active learning in the class session	41	30	14	11	†	†
Providing instructions for class activities	2	2	1	1	†	†
Interacting and providing feedback during active learning	9	5	*	*	*	*
Communication skills
Using quality speech (i.e., volume, pace, tone, rate)	31	24	13	10	†	†
Using clear articulation of speech	8	6	4	2	†	†
Moving around the room	7	6	*	*	*	*
Using eye contact	9	6	5	2	†	†
Being aware of students	3	2	*	*	*	*
Capturing students’ attention	3	3	*	*	*	*
Being enthusiastic	26	11	3	3	†	†
Being prepared, confident, competent	15	12	7	6	†	†
Rapport
Using students’ names	3	3	2	2	†	†
Praising or encouraging students	7	6	4	2	†	†
Conversing with students before, during, after class	11	11	8	6	†	*
Being respectful to students	13	9	8	6	†	†
Showing humility	2	1	*	*	*	*
Using humor	5	2	*	*	*	*
Being helpful	5	5	2	2	†	†
Being sensitive to differing student views or backgrounds	3	2	2	2	†	*
Having good rapport with students in general	4	3	*	*	*	*
Creating a positive classroom environment	9	8	7	6	†	†
Classroom management
Managing disruptions by students	11	7	*	*	*	*
Managing external disruptions	2	1	*	*	*	*
Using classroom technology effectively	20	5	5	3	†	†
Overall effectiveness
Rating of instructor’s overall teaching effectiveness	1	1	*	*	*	*
Total	657	448	235	161	31	24

Category did not reach 70% consensus.

†

Category was reflected in new items created by combining and editing items during Phases 5 and 6.

Phase 2

The Experts were then invited to complete the survey. The survey was administered anonymously online via Google Forms. The Experts were given 1 to 2 weeks to complete each Phase of surveys, with reminders given close to each survey deadline to achieve a full response rate. For Phase 2, Experts were instructed to select any number of items from any category that they believed would be relevant to evaluating good teaching practices for POT in the new department. If any item within a category was selected by an expert, then that category also received a selection. A total of 448 items from 59 categories were selected (Table 3). After completing the Phase 2 survey, the Experts met to discuss the cutoff level for consensus in future Phases. Based on Diamond et al. (2014) systematic review of Delphi studies, the Experts agreed to only include categories of items that had been selected by at least 70% of the Experts.

Phase 3

Items that were selected from Phase 2 were then used to create a Phase 3 survey. The items that were not selected from Phase 2 were eliminated from the study. Experts were given the same instructions from Phase 2 when completing the Phase 3 survey; however, the Experts were provided with the anonymous results from the Phase 2 survey that showed the number of Experts that selected each item from each category. The results were meant to help build consensus by informing Experts of their peers’ selections. Using 70% consensus, a total of 245 items from 43 categories were selected (Table 3).

Phase 4

Items that were selected from Phase 3 were then used to create a Phase 4 survey. The items from categories that did not achieve 70% consensus from Phase 3 were eliminated from the study. In addition, any categories, and their respective items, that did not achieve 70% consensus were eliminated from the study. Experts were given the same instructions from Phase 3 and were again provided with the anonymous results from the Phase 3 survey. A total of 161 items from 39 categories were selected (Table 3).

Phase 5

As a modification to the Delphi method, Experts then met to discuss and edit the remaining items. Allowing space for group discussion during the final phases of a Delphi study has been used as a modification to the method in previous healthcare research (Broder et al., 2022; Vonk Noordegraaf et al., 2011). At the onset of the study, it was determined that group discussion would likely be needed to edit items for the creation of an original POT form. Specifically, the Experts met to combine items, eliminate items based on the number of selections, and then edit each item for consistency in tense, word choice, and length. Phase 5 resulted in a POT form consisting of 31 new items that reflected 39 of the original categories (Table 3).

Phase 6

The POT form that resulted from Phase 5 was then pilot tested by each faculty expert. Given the difficulty in coordinating the Experts’ schedules to observe a peer’s instruction for an in-person course, the website YouTube was searched for a lecture that the Experts could observe and evaluate at their convenience using the POT form. Due to the lack of classroom lectures uploaded on YouTube, a faculty workshop hosted by the University of South Carolina’s Center for Teaching Excellence (2020) YouTube channel was used for the pilot test. The workshop was focused on implementing active learning strategies in large classrooms, and was presented by an Associate Professor at the university to colleagues. The workshop was selected for the pilot test because it: resembled a lecture for college students, was similar in length as a typical college lecture (i.e., 50 min), featured several items that the POT form was designed to assess, provided clear audio, and showed the workshop’s slides, presenter, and audience members.

After completing the pilot test, the Experts met to edit the POT form. Based on the pilot observation experience, the Experts discussed editing the form’s measurement scale to eliminate ambiguity, combining items, and removing items that seemed excessive (e.g., redundant, irrelevant; Otero-Saborido et al., 2024). Phase 6 resulted in a POT form consisting of 24 new items. As a group, the Experts discussed how each item signified the original categories, which were ultimately reflective of 36 categories (Table 3).

Results

Several items and categories were eliminated during the Phases that involved the Experts’ anonymous feedback (Phases 2, 3, and 4). From the original 657 items gathered during Phase 1, the number of items decreased to 448 items, 235 items, and 161 items, respectively (Table 3). Likewise, from the original 60 categories during Phase 1, the number of categories decreased to 59 categories, 43 categories, and 39 categories, respectively (Table 3). Specifically, by Phase 4, the following categories were eliminated from each overarching topic: Scale of observation form (checklist/dichotomous, scale), Content (giving reminders to students about dates/deadlines, transitioning clearly between topics, defining key terms, explaining relevance of content, using accurate content, preparing students for future exams), Questions and answers (assessing students’ understanding at the start of class, allowing enough time for students to answer questions, incorporating students’ ideas into the class session, being flexible with class session based on questions), Diverse teaching and active learning methods (interacting and providing feedback during active learning), Communication skills (moving around the room, being aware of students, capturing students’ attention), Rapport (showing humility, using humor, having good rapport with students in general), Classroom management (managing disruptions by students, managing external disruptions), and Overall effectiveness (rating of instructor’s overall teaching effectiveness; Table 3).

A major refinement of the items and POT form took place during Phases 5 and 6 as the modified aspect of the Delphi study that removed Expert anonymity (i.e., meeting in person to discuss and edit the remaining items). In these Phases, the Experts’ discussions effectively decreased the remaining 161 items to 24 items (including the form’s measurement scale), and the remaining 39 categories to 36 categories. A contributing factor to the reduction of items took place by Experts working together to combine items. For example, by including the word “clarity” into item #3 on the POT form (Figure 1) “spoke with effective volume, clarity, pace, and tone,” the Experts were able to combine a separate item that assessed if an instructor was speaking clearly.

Figure 1.

Final peer observation of teaching form.

During the modified Phases 5 and 6 of the study, there were two categories that initially met consensus, yet were ultimately eliminated after group discussion. The Experts decided to eliminate the category of conversing with students before, during, or after class, as it may have been too difficult to do so for several faculty who teach back-to-back classes. In addition, the Experts chose to eliminate the category of being sensitive to differing student views or backgrounds, as the Experts felt apprehensive about potentially violating the UNC System’s recent changes in the implementation of diversity, equity, and inclusion on campus (University of North Carolina System, 2024; Table 3).

Similarly, during Phases 5 and 6, there was one category that the Experts initially eliminated from the Delphi study, but then included back into the POT form after group discussion. Specifically, the Experts discussed how the category of allowing enough time for students to answer questions was valuable and could be included by editing item #12 of the POT form (i.e., asked questions during the class session and gave time for answers; Figure 1).

In addition, during Phase 6, the Experts decided to switch from using a Likert-based measurement scale to a dichotomous measure. Prior to the study’s pilot-test, the Experts reached consensus on using a Likert-based scale for measuring each item on the POT from; however, after the pilot-test, the Experts determined that a Likert-based measurement for each item was too ambiguous. After discussion, the Experts decided that a dichotomous measurement (i.e., no revisions vs. suggested revisions), with an option for providing open-ended comments per each item that needed revisions or deserved commendations, was best suited for providing observees with developmental feedback (Table 1 and Figure 1).

At the end of the study, the final number of items were categorized into a POT form based on the logical progression of time when observing a typical class session. Two items were used to assess the beginning of class, 15 items to assess during the class session, and five items to assess at the end of the class session (Figure 1). By structuring the POT form according to when observers would be assessing certain items during the flow of class, the form allowed for a more user-friendly experience during an observation session.

Discussion

This study used a modified Delphi method to create a new POT form based on the perspectives of experienced college instructors of public health using existing POT items found in the peer-reviewed literature. From a foundation of 657 relevant items found in 33 published POT forms, six phases of surveys, group discussions, and pilot testing resulted in a final POT form for in-person classes consisting of a dichotomous measurement scale and 22 new items. The items represented over 30 categories of established, observable teaching practices.

To the authors’ knowledge, this study was the most comprehensive analysis of common categories found in existing POT forms in higher education. After conducting a systematic review, Otero-Saborido et al. (2024) found 13 published POT forms used in higher education. Our study’s findings expanded upon that review by including 33 forms, and by categorizing the POT items into frequently observable teaching practices. The list of POT categories may be useful for faculty from other departments that are creating or adapting POT forms, or for scholars who are interested in studying POT.

The findings from this study reflect previous research regarding the utility of the Delphi method. Granted, the Delphi method has its share of critiques (Diamond et al., 2014; Keeney et al., 2001; Niederberger & Spranger, 2020); however, the method has been widely used, and was applied by two studies in creating POT forms in the field medical education (Frankl et al., 2017; Newman et al., 2009) that were included in the first phase for this project. Instructors of higher education may want to consider using the Delphi method, or a modification of the method, when creating their department’s own POT form. If a department opts for using the Delphi method, the items from the articles included in this study, or simply the categories listed in this article, may serve as a starting point for faculty to assess.

The process of creating a POT form through the Delphi method allowed for instructor buy-in on the items that would be used to evaluate their own teaching practices. Since faculty may view POT with mistrust and anxiety (Cutroni & Paladino, 2023), it can be assumed that involving the faculty in deciding which teaching practices should be evaluated, as both observers and observees, could help to improve the trustworthiness of a POT form. As K. L. Harris et al. (2008) stated, “. . .including staff in the decision-making process about how peer review of teaching will be conducted should significantly assist the ‘buy-in’ of staff to the program and, consequently, affect the degree to which they participate effectively and productively” (p. 21). As such, those creating, or changing, a POT form may want to involve all faculty in a department to provide input on the items that are being used to evaluate their teaching practices.

It is important to note that those who use this study’s POT from may want to adapt its measurement scale. Although the study’s Experts eventually reached the conclusion that a dichotomous scale (i.e., no revisions vs. suggested revisions) with optional open-ended comments would best serve observees’ teaching development, that measurement may not be suitable for a department’s summative evaluation of teaching, or for faculty members’ promotion dossiers. As such, faculty may want to opt for a more nuanced Likert-based scale found in previously published POT forms (e.g., Solid evidence, Partial evidence, None or insufficient evidence [Bryan et al., 2018], Commendable with no recommendation for improvement, Needs minor improvement, Adequate as the instructor attempted to do this but could use some development or revision, Below adequate levels as development is recommended, Missing as the instructor did not do this or should consider adding [Davis, 2011], Truly exemplary, Done well, Needs improvement [Gaskamp & Kintner, 2014]).

The results from this study have at least two major implications for future research. First, the POT form resulting from this study would benefit from being tested for validity and reliability. Although the items that served as a basis for this study were published in peer-reviewed journals, by combining and editing the pre-existing items, the POT form consists of newly created items, and is thus in need of testing for validity and reliability, similar to most POT forms (Otero-Saborido et al., 2024). Second, scholars may want to study the impact of the POT form toward enhancing teaching practices, as well as improvements in student learning. There is a real need in the research literature for examining if POT can lead to changes in the observee (i.e., implementing practices suggested by an observer) or the observer (i.e., trying new teaching practices that were used by the observee). In addition, research on POT is lacking in terms of changes in student outcomes (e.g., assessment scores, improved active learning) as a result of changes that are implemented by faculty due to POT (Cutroni & Paladino, 2023).

There are several limitations that should be considered when interpreting the findings from this study. First, given the search terms and databases used to retrieve POT forms in the peer-reviewed literature, it is possible that additional forms may have been unintentionally excluded in the study. Second, although there were strengths to the study’s selection of Experts, it is important to note that only one of the Experts had an advanced degree in the field of education. By not including additional educational experts in the study, it is possible that POT items that were excluded from the study may have been included by those with a more nuanced understanding of the study’s topic. This limitation reflects Keeney and colleagues’ statement in their critical review of Delphi studies, “. . .consensus from a Delphi process does not mean that the correct answer has been found” (Keeney et al., 2001, p. 198). Third, although the Experts’ answers to the survey phases were anonymous, complete anonymity could not be assured, since the Experts knew each other, and were able to see other Experts’ collective survey responses and feedback during in-person group discussions in the study’s final phases. Fourth, the collection of items from POT forms were initially grouped into categories by only the first author, and later by the group of Experts. A more robust method (i.e., factor analysis) could have been used to identify meaningful groupings. Fifth, the POT form was developed only by the faculty in the Department of Public Health at our institution, and is thus not generalizable, and may not reflect the perspectives of faculty in other departments of public health across the country. Sixth, the POT form was meant for in-person classes and may not be relevant to online courses. Finally, the POT form that resulted from this study has yet to be tested for validity or reliability.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Appalachian State University’s Center for Excellence in Teaching and Learning for Student Success’ 2025 Teaching Quality Framework Grant.

ORCID iDs

Christopher M. Seitz

Martie P. Thompson

Shenghui Wu

References

Addy

T. M.

Younas

Cetin

Cham

Rizk

Nwankpa

Borzone

(2023). The development of the protocol for advancing inclusive teaching efforts (PAITE). Journal of Educational Research and Practice, 12, 65–93. https://doi.org/10.5590/jerap.2022.12.0.05

Barna

Reenan

(2020). Graduate instructor peer observation in music theory pedagogy. Journal of Music Theory Pedagogy, 34, 1. https://doi.org/10.71156/2994-7073.1212

Berk

R. A.

Naumann

P. L.

Appling

S. E.

(2004). Beyond student ratings: Peer observation of classroom and clinical teaching. International Journal of Nursing Education Scholarship, 1(1), Article 10. https://doi.org/10.2202/1548-923X.1024

Birdwell

Harris

(2022). Active learning classroom observation tool: Improving classroom teaching and supporting institutional change. Journal of Learning Spaces, 11(1), 108–121. https://libjournal.uncg.edu/jls/article/view/2145

Blanco

Capello

Gusic

McCormack

Hafler

(2011). Peer feedback tool for lectures & small-group teaching. MedEdPORTAL, 7, 8416. https://doi.org/10.15766/mep_2374-8265.8416

Blauvelt

M. J.

Erickson

C. L.

Davenport

N. C.

Spath

M. L.

(2012). Say yes to peer review: A collaborative approach to faculty development. Nurse Educator, 37(3), 126–130. https://doi.org/10.1097/NNE.0b013e318250419f

Broder

M. S.

Gibbs

S. N.

Yermilov

(2022). An adaptation of the RAND/UCLA modified Delphi panel method in the time of COVID-19. Journal of Healthcare Leadership, 14, 63–70. https://doi.org/10.2147/jhl.s352500

Bryan

J. E.

Asher

Karshmer

E. D.

(2018). Assessing librarians’ teaching of one-shot sessions: A new model for evaluating instructional performance. College & Undergraduate Libraries, 25(4), 350–371. https://doi.org/10.1080/10691316.2018.1527268

Cobb

K. L.

Billings

D. M.

Mays

R. M.

Canty-Mitchell

(2001). Peer review of teaching in web-based courses in nursing. Nurse Educator, 26(6), 274–279.

10.

Corcelles-Seuba

Ortiz

O’Leary

Duran

(2025). A systematic review of reciprocal peer observation in higher education. Professional Development in Education, 1–18. https://doi.org/10.1080/19415257.2025.2480768

11.

Çoruh

Kritek

P. A.

(2017). Implementation of a coaching program for peer feedback on large-group teaching. Annals of the American Thoracic Society, 14(4), 601–603. https://doi.org/10.1513/AnnalsATS.201611-936LE

12.

Crabtree

J. L.

Scott

P. J.

Kuo

(2016). Peer Observation and Evaluation Tool (POET): A formative peer review supporting scholarly teaching. Open Journal of Occupational Therapy, 4(3), Article 9. https://doi.org/10.15453/2168-6408.1273

13.

Cutroni

Paladino

(2023). Peer-ing in: A systematic review and framework of peer review of teaching in higher education. Teaching and Teacher Education, 133, 104302. https://doi.org/10.1016/j.tate.2023.104302

14.

Dalkey

Helmer

(1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458–467. https://doi.org/10.1287/mnsc.9.3.458

15.

Davis

T. S.

(2011). Peer observation: A faculty initiative. Currents in Pharmacy Teaching and Learning, 3(2), 106–115. https://doi.org/10.1016/j.cptl.2011.01.009

16.

Diamond

I. R.

Grant

R. C.

Feldman

B. M.

Pencharz

P. B.

Ling

S. C.

Moore

A. M.

Wales

P. W.

(2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002

17.

Dillon

James

Prestholdt

Peterson

Salomone

Anctil

(2020). Development of a formative peer observation protocol for STEM faculty reflection. Assessment & Evaluation in Higher Education, 45(3), 387–400. https://doi.org/10.1080/02602938.2019.1645091

18.

Eddy

S. L.

Converse

Wenderoth

M. P.

(2015). PORTAAL: A classroom observation tool assessing evidence-based teaching practices for active learning in large science, technology, engineering, and mathematics classes. CBE Life Sciences Education, 14(2), 14:ar23. https://doi.org/10.1187/cbe.14-06-0095

19.

Fielden

Foster

(2010). Crossing the rubricon: Evaluating the information literacy instructor. Journal of Information Literacy, 4(2), 78–90. https://doi.org/10.11645/4.2.1511

20.

Fletcher

J. A.

(2018). Peer observation of teaching: A practical tool in higher education. The Journal of Faculty Development, 32(1), 51–64.

21.

Flick

L. B.

Sadri

Morrell

P. D.

Wainwright

Schepige

(2009). A cross discipline study of reformed teaching by university science and mathematics faculty. School Science and Mathematics, 109(4), 197–211. https://doi.org/10.1111/j.1949-8594.2009.tb18258.x

22.

Fox-Alvarez

S. A.

Hostnik

L. D.

Conner

Watson

J. S.

(2022). Development of a formative assessment rubric for peer evaluation of teaching (FARPET) and pilot use in veterinary online teaching. Journal of Veterinary Medical Education, 49(6), 693–698.

23.

Frankl

Newman

Burgin

Atasoylu

Fishman

Gooding

Kamin

Puig

Thomas

A. M.

Cohen

Schwartzstein

(2017). The case-based collaborative learning peer observation worksheet and compendium: An evaluation tool for flipped classroom facilitators. MedEdPORTAL, 13, 10583. https://doi.org/10.15766/mep_2374-8265.10583

24.

Gabaccia

D. R.

(1996, March 1). Thinking globally, acting locally: Peer review of history teaching at UNC Charlotte. American Historical Association Perspectives. https://www.historians.org/perspectives-article/thinking-globally-acting-locally/

25.

Gambescia

S. F.

(2015). Message from the editor in Chief. Pedagogy in Health Promotion, 1(1), 3–4. https://doi.org/10.1177/2373379914559220

26.

García

S. A. V.

Stacy-Bates

K. K.

Alger

Marupova

(2017). Peer evaluation of teaching in an online information literacy course. Libraries and the Academy, 17(3), 471–483. https://doi.org/10.1353/pla.2017.0030

27.

Gaskamp

C. D.

Kintner

(2014). Development, evaluation, and utility of a peer evaluation form for online teaching. Nurse Educator, 39(1), 22–25. https://doi.org/10.1097/nne.0000000000000007

28.

Gosling

(2014). Collaborative peer-supported review of teaching. In Sachs

J. M

. (Ed.), Peer review of learning and teaching in higher education: International perspectives (pp. 13–31). Dordrecht Springer.

29.

Green

R. A.

(2014). The Delphi technique in educational research. Sage Open, 4(2), 8. https://doi.org/10.1177/2158244014529773

30.

Harris

A. H.

Cox

M. F.

(2003). Developing an observation system to capture instructional differences in engineering classrooms. Journal of Engineering Education, 92(4), 329–336. https://doi.org/10.1002/j.2168-9830.2003.tb00777.x

31.

Harris

K. L.

Farrell

Bell

Devlin

James

(2008). Peer review of teaching in Australian higher education: A handbook to support institutions in developing effective policies and practices. Centre for the Study of Higher Education.

32.

Hassel

D. M.

Fahie

Löhr

C. V.

Halsey

R. L.

Vernau

Gorman

(2020). Inter-institutional collaboration for the development of a local peer observation process to enhance teaching. Journal of Veterinary Medical Education, 47(5), 555–569. https://doi.org/10.3138/jvme-2019-0093

33.

Hora

M. T.

(2015). Toward a descriptive science of teaching: How the TDOP illuminates the multidimensional nature of active learning in postsecondary classrooms. Science & Education, 99(5), 783–818. https://doi.org/10.1002/sce.21175

34.

Hutchings

(1995). From idea to prototype: The peer review of teaching (a project workbook). American Association for Higher Education.

35.

Jia

Spagnolo

A. B.

Barrett

Murphy

A. A.

Basto

P. M.

Rothpletz-Puglia

Luther

(2025). Validation of a peer observation and evaluation tool for online teaching in the US. Educational Technology Research and Development, 73, 615–639. https://doi.org/10.1007/s11423-024-10428-z

36.

Johnston

A. L.

Baik

Chester

(2022). Peer review of teaching in Australian higher education: A systematic review. Higher Education Research & Development, 41(2), 390–404. https://doi.org/10.1080/07294360.2020.1845124

37.

Keeney

Hasson

McKenna

H. P.

(2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195–200. https://doi.org/10.1016/S0020-7489(00)00044-4

38.

Kumar

Bostwick

J. R.

Klein

K. C.

(2018). A pilot program featuring formative peer review of faculty teaching at a college of pharmacy. Currents in Pharmacy Teaching and Learning, 10(9), 1280–1287. https://doi.org/10.1016/j.cptl.2018.06.009

39.

Litzelman

D. K.

Stratos

G. A.

Marriott

D. J.

Skeff

K. M.

(1998). Factorial validation of a widely disseminated educational framework for evaluating clinical teachers. Academic Medicine, 73(6), 688–695. https://doi.org/10.1097/00001888-199806000-00016

40.

Martin

B. A.

Martin

J. H.

(1989). Assessing the lecture performance of university faculty: A behavioral observation scale. Journal of Education for Business, 64(4), 157–160. https://doi.org/10.1080/08832323.1989.10117350

41.

Martin

G. A.

Double

J. M.

(1998). Developing higher education teaching skills through peer observation and collaborative reflection. Innovations in Education and Training International, 35(2), 161–170. https://doi.org/10.1080/1355800980350210

42.

Middleton

(2002). Evolution of peer evaluation of library instruction at Oregon State University libraries. portal Libraries and the Academy, 2(1), 69–78. https://doi.org/10.1353/pla.2002.0019

43.

Murray

H. G.

(1983). Low-inference classroom teaching behaviors and student ratings of college teaching effectiveness. Journal of Education & Psychology, 75(1), 138–149. https://doi.org/10.1037/0022-0663.75.1.138

44.

Nasa

Jain

Juneja

(2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116–129. https://pubmed.ncbi.nlm.nih.gov/34322364/

45.

Newman

L. R.

Lown

B. A.

Jones

R. N.

Johansson

Schwartzstein

R. M.

(2009). Developing a peer assessment of lecturing instrument: Lessons learned. Academic Medicine, 84(8), 1104–1110. https://doi.org/10.1097/acm.0b013e3181ad18f9

46.

Niederberger

Spranger

(2020). Delphi technique in health sciences: A map. Frontiers in Public Health, 8, 457. https://doi.org/10.3389/fpubh.2020.00457

47.

Nunn

C. E.

(1996). Discussion in the college classroom: Triangulating observational and survey results. Journal of Higher Education, 67(3), 243–266. https://doi.org/10.1080/00221546.1996.11780260

48.

Otero-Saborido

F. M.

Domínguez-Montes

J. A.

Cenizo-Benjumea

J. M.

González-Calvo

(2024). Peer observation of teaching in higher education: Systematic review of observation tools. Educational Process International Journal, 13(1), 84. https://doi.org/10.22521/edupij.2024.131.6

49.

Sawada

Piburn

M. D.

Judson

Turley

Falconer

Benford

Bloom

(2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics, 102(6), 245–253. https://doi.org/10.1111/j.1949-8594.2002.tb17883.x

50.

Schifano

Niederberger

(2025). How Delphi studies in the health sciences find consensus: A scoping review. Systematic Reviews, 14, 14. https://doi.org/10.1186/s13643-024-02738-3

51.

Servilio

K. L.

Hollingshead

Hott

B. L.

(2017). Partnerships enhancing practice: A preliminary model of technology-based peer-to-peer evaluations of teaching in higher education. Journal of Special Education Technology, 32(1), 23–35. https://doi.org/10.1177/0162643416681161

52.

Shang

(2023). Use of Delphi in health sciences research: A narrative review. Medicine, 102(7), e32829. https://doi.org/10.1097/md.0000000000032829

53.

Siddiqui

Z. S.

Jonas-Dwyer

Carr

S. E.

(2007). Twelve tips for peer observation of teaching. Medical Teacher, 29(4), 297–300. https://doi.org/10.1080/01421590701291451

54.

Smith

M. K.

Jones

F. H. M.

Gilbert

S. L.

Wieman

C. E.

(2013). The Classroom Observation Protocol for undergraduate STEM (COPUS): A new instrument to characterize university STEM classroom practices. CBE—Life Sciences Education, 12(4), 618–627. https://doi.org/10.1187/cbe.13-08-0154

55.

Thomas

Chie

Q. T.

Abraham

Jalarajan Raj

Beh

L. S.

(2014). A qualitative review of literature on peer review of teaching in higher education: An application of the SWOT framework. Review of Educational Research, 84(1), 112–159. https://www.jstor.org/stable/24434230

56.

University of North Carolina System. (1993). Guideline on tenure and teaching in the University of North Carolina. https://www.northcarolina.edu/apps/policy/doc.php?id=182

57.

University of North Carolina System. (2024, July 15). UNC System shares guidance regarding DEI. https://www.unc.edu/posts/2024/07/15/unc-system-shares-guidance-regarding-dei/

58.

University of South Carolina’s Center for Teaching Excellence. (2020, March 24). Active learning in large courses: What’s possible, what works? [Video]. YouTube. https://www.youtube.com/watch?v=tTKWaTXSmB8

59.

Vonk Noordegraaf

Huirne

J. A. F.

Brölmann

H. A. M.

van Mechelen

Anema

J. R

. (2011). Multidisciplinary convalescence recommendations after gynaecological surgery: A modified Delphi method among experts. BJOG: An International Journal of Obstetrics & Gynaecology, 118(13), 1557–1567. https://doi.org/10.1111/j.1471-0528.2011.03091.x

60.

Zeng

L. M.

(2020). Peer review of teaching in higher education: A systematic review of its impact on the professional development of university teachers from the teaching expertise perspective. Educational Research and Reviews, 31, 100333. https://doi.org/10.1016/j.edurev.2020.100333