Abstract
Epistaxis is a common problem faced by primary care physicians. Typically, first-hand experience with this problem is not obtained in medical school. A computer learning module was developed to address practical management of epistaxis. It was evaluated by use of third-year clerks and compared with an interactive seminar based on the same material. Fifty-eight students doing their required otolaryngology rotations were randomly separated into 3 groups: (1) doing a preinstruction test, (2) using the computer module, and (3) participating in a small-group seminar. All participants were tested with a short written test and a practical test that involved performing anterior nasal pack placement in a model patient. Percentage scores for the 2 groups were compared by use of t tests, and there was no significant difference between the written, practical, or combined scores at a level where P = 0.05. This study shows that basic patient management and a simple procedure can be taught as effectively with a computer module as with a small-group interactive seminar.
Computer-assisted instruction (CAI) has been explored as a method of instruction in medical education for more than 2 decades. However, despite extensive descriptive literature, exploding technology, and a fair degree of enthusiasm, the computer has not become a vital partner in the day-to-day delivery of curricular content in most medical schools. Few would doubt the potential of CAI as we approach the digital age, but there are many reasons CAI has not as yet lived up to its potential. Excitement over the novelty of the approach tends to overshadow issues such as the pedagogic quality of a CAI package, its implementability, and its appropriate role in a given curriculum. 1,2 Studies assessing CAI learning efficiency have been plagued with design flaws such as unequal instructional time between CAI groups and other methods, dissimilar instructional content between methods, and consideration as to what about the CAI accounts for its success or lack thereof. 3 There have been very few attempts to exploit the unique properties of CAI, especially Internet CAI; instead, old instructional formats like syllabus notes have simply been transferred to the computer, with little further exploitation of the medium.
This project was stimulated by a needs assessment performed to clarify which important otolaryngology topics family doctors and community otolaryngologists thought medical graduates should have mastered. 4 Management of epistaxis was ranked second by otolaryngologists; however, a group of family-practice resident volunteers showed an inadequate level of competence at managing this problem. Although half of them had already had to deal with epistaxis, their scores on a practical and written test averaged less than 30%. This accentuates the fact that this common medical problem receives little curricular time, and it supports a plan to create a self-study epistaxis module emphasizing the practical aspects of managing this problem and aimed at senior medical students, residents, and practicing primary care physicians. Because more curricular time (and fewer clinical demands to allow clinicians to take advantage of curricular time) is unlikely to become available, we elected to make this teaching package accessible for learners through the Internet. A survey of University of Toronto clerks indicated a high degree of computer competency, Internet access, and interest is use of medical CAI packages. 5
PURPOSE
The purpose of this project was to compare 2 methods of learning about a practical medical topic—specifically, an interactive software program and a small-group, student-centered seminar, both designed to use the same curriculum and method of instructional design. The interactive seminar was chosen as being representative of the ideal medical school or continuing medical education learning environment.
Composition of study groups
Test scores
Two types of knowledge were compared. The first was the medical content knowledge pertaining to treatment of a patient with this medical problem, and the second was demonstration of the ability to perform a simple procedure involved in this treatment.
METHOD
Administrative consent for this study was obtained from the third-year otolaryngology clerkship preceptors at each teaching hospital, and ethical approval was granted by the Ontario Institute for Studies in Education. This teaching project was used as a replacement for the scheduled teaching of the topic of epistaxis. A brief description of the study was given to third-year clerks doing their core rotation in otolaryngology, and each was invited to participate. A consent form specifying that scores were confidential and anonymous, but that students were required to learn the material as a part of their medical school courses, was signed by each participant. Preceptors did not have access to scores and were not involved in the study. A pilot test was initially run, allowing students to choose which instructional method they preferred, to determine any major pedagogic flaws in either intervention. Performance on both posttests was comparable for these 2 groups, so we were assured that participating students randomly assigned to interventions would not be disadvantaged with respect to their medical education, but would still have the same motivation to do well that they have with their medical courses in general. 6 Randomization was accomplished by having each student choose a file card placed facedown on the table. The number on the card represented inclusion in 1 of 3 groups (Table 1) and the student's code number for identification of results.
Curricular Material
Both the seminar and the computer module were based on the same instructional design, 7,8 and their content was identical. The computer module was developed with Toolbook II Instructor 5.01 and consisted of a 60-page document. Pages contained interactive elements such as fill-in-the-blank questions, multiple-choice questions, true-false questions, and animations. Pages were hyperlinked to some degree to allow easy navigability. The design was such that the module was Internet adaptable and did not use complicated graphics, sound, or videos because these currently require a significant amount of time to download before use in this format. Initial formative assessment was undertaken with 3 final-year medical clerks who worked through the module individually and participated in an unstructured interview. The seminar intervention involved 2 to 4 students and had an interactive format. Students were first invited to discuss their current knowledge of the topic. Information was then introduced in the context of specific representative cases requiring management decisions. All students were encouraged to participate. The same person who designed the computer module gave the seminar.
All students participated in their assigned instruction at the same time. The seminar took about 30 minutes, and the computer students were advised to spend this amount of time with the module so that learning time was roughly equivalent. Students were evaluated immediately after the intervention.
Evaluation
The written test was a 17-point, short-answer, paper-and-pencil examination covering practical aspects of epistaxis management. The practical test was done in a separate room, with the usual instruments needed to place an anterior ribbon gauze nasal pack and a model made especially for the purpose of demonstrating this procedure. Each student came into the room alone, and questions about the written test were answered. Then the student demonstrated the procedure, and a 16-point checklist was used to assess his or her performance. After this, errors were corrected, the finished product was examined, and questions were answered. An opportunity to repeat the procedure was given. On completion of the test, all students were given a handout containing the content relevant for examination purposes in the course. Students were invited to write comments about the teaching program they experienced on the back of the written test.
RESULTS
There were 58 subjects enrolled, 16 in group 1 (pretest only), 21 in group 2 (computer module), and 21 in group 3 (seminar instruction). Scores for each are shown in Table 2 and Fig 1. The average pretest written percentage score was 27.9 ± 13.9 (range 6.7 to 60), and the pretest practical score was 21.5 ± 5.7 (range 12.5 to 31.3). This demonstrates a dramatic lack of knowledge of epistaxis management in junior medical clerks.
Test scores for each group. Vertical lines show SD. There was a significant difference between the pretest group and both posttest groups (P < 0.05). There were no significant differences between the posttest scores of the groups using either the computer module or the seminar intervention.
Posttest written mean scores were 83.5% ± 10.7% (range 63.3% to 100%) for group 2 and 88.6% ± 5.5% (range 76.7% to 100%) for group 3. The mode for the written score was 73.3% for group 2 and 86.7% for group 3. Posttest practical scores were 92.7% ± 7.4% (range 71.9% to 100%) for group 2 and 86.8% ± 6.8% (range 71.9% to 100%) for group 3. The mode for the practical score was 100% for group 2 and 93.8% for group 3.
Postinstruction scores for groups 2 and 3 were compared with t tests, and there was no significant difference between the written, practical, or combined scores at a level at which the type I error was 0.05. There was a significant difference between group 1 and both groups 2 and 3 in all scores, showing that both interventions resulted in improved performance on these tests.
A post hoc power calculation—taking as a significant difference 10% of the written or practical test score, using the sample size and variance in the study—indicated that we should have been able to detect such a difference, if it existed, in 99% of trials. This difference of 10% was chosen to be educationally significant because it translated to 2 wrong answers on the written test (eg, not knowing an appropriate discharge medication and its dosage) or missing 2 steps in the procedure.
Students' comments were very positive overall. Most of these comments related to their participation in a study that allowed them to learn something that they perceived as practical and useful and their appreciation of the one-on-one teaching they received as a benefit of their participation. Students found the computer module easy and fun to use.
DISCUSSION
These results show that the cognitive and psychomotor aspects of epistaxis management can be taught as effectively with a simple computer module as they can with a labor-intensive seminar requiring the assembling of an appropriate content expert and a small group of learners.
Many researchers in the area of CAI believe that it is superior to conventional teaching. However, studies adduced in support of this claim have been confounded in various ways. An important aspect of the experimental protocol was the use of the same instructional design in both interventions. Although the seminar was different in each case, depending on participants' questions and responses and the group dynamic, the computer module was relatively unchangeable. It was important to design this module so that a variety of learners could use it effectively; each time a new person opened the file, experience was tailored to his or her own current knowledge base and learning goals.
Instructional design refers to how the skills were defined and broken down into subskills and the order in which these were presented. 7 Other learning principles 8 were followed to make both interventions efficacious. External sources of frustration and conflict were minimized by giving each group 2 student written instructions on how to access and navigate the computer program. Computers used for the study were in student computer laboratories closed to others except study participants. Seminars took place in students' usual seminar rooms, in all cases quiet environments without distractions. Repetition of concepts and key words was used in both interventions, and correct answers were positively reinforced. Erroneous responses were corrected in a nonjudgmental way. In both interventions active student participation was required to work through the content. This was easily achieved in the seminar group, but a computer module user could have potentially skipped answering the questions on the screen and thus missed information presented as feedback to these answers. In both interventions expression of individual differences was possible, and an attempt to relate the content to a typical clinical case was made to allow for contextual learning.
For the computer module, consideration of screen design was important because poor design has previously been shown to have deleterious effects on learning. 9 The software used to design the module allows the programmer to build modules using templates that follow many of the suggested guidelines for screen design. 10,11 Some of these guidelines include use of a simple screen with consistently located elements (such as navigation buttons), easily readable fonts, a uniform layout for text, and limited use of color (because the module was designed for Internet use and color can affect data transmission speed or some users may be color blind or have monochromatic screens). 11 As well, this module was intended for eventual Internet use, so there was no inclusion of sound or videos; this minimized download time and waiting time between screens. The novelty effect was also minimized by this simple design.
Most of the previous work in medical CAI concerns packages that teach cognitive skills, such as interpreting blood gases or reading chest radiographs. Psychomotor skills have not been frequently addressed. Examples include learning the “swinging flashlight test” 12 and pulmonary auscultation, 13 both of which are very low-level motor skills requiring a great deal of cognitive interpretation. Neither is analogous to a procedure like placing an anterior nasal pack. Both of these studies also used written tests as outcome measures. Neither of these packages would be appropriate for Internet instruction because one requires a special mannequin head 12 and the other uses audiotaped breath sounds. 13
This study has a number of limitations. The population was fairly homogeneous in medical training, prior exposure to otolaryngology procedures, and motivation, which may reduce the generalizability (ie, to other medical students or practicing physicians) of these findings for this particular instructional module. Testing took place immediately after instruction, so we cannot make assumptions about how well this information will be retained and recalled at a future date when real patients may need to be treated. There is also the issue of transferability; the model used to demonstrate nasal packing is a life-sized silicone half head, which is more manipulable and cooperative than a real patient may be.
What advantage does this teaching module have over currently available methods of instruction? Because most Canadian medical students do not have core curricular time in otolaryngology, they are unlikely to get adequate instruction in this procedure. If they do spend time on an otolaryngology service, they may receive seminar instruction in this topic, but they will be unlikely to see a patient treated first hand, unless they do a reasonably long rotation and have the opportunity to see patients in the emergency department. Most otolaryngology handbooks, intended to be kept in laboratory coat pockets and used in clinical situations, do address the topic of epistaxis treatment. A recent American handbook gives step-by-step instructions in managing this problem but skims over the procedure we tested by simply stating that the nose is packed. 14 There are no diagrams in this section of the book. A British publication addresses the whole issue more briefly but describes placing an anterior pack in one sentence and uses a diagram to illustrate the concept that the pack is laid down in layers. 15 Neither describes the procedure as it is currently performed in an academic otolaryngology unit. Both of these handbooks are readily available in medical bookstores, and each costs more than Can $56 (US $38).
The advantage of the CAI version is not an inherent strength of CAI or the medium in this case; the advantage lies in the accessibility of the module to medical clerks, practicing physicians, and medical trainees working in emergency departments. We could produce a paper version of the local method and reminders for general management of this problem, but distribution to everyone who could use it would be impossible. Chances of the individual finding the instructions at the appropriate time would also be low. This module has become part of the faculty of medicine Web sites at the 2 involved medical schools, so it will be easily accessible to medical trainees and to physicians who use the Internet. It is also hyperlinked to other potentially useful resources.
Is this an economically viable alternative to either seminar teaching or book publishing? Because the module is simple, it was inexpensive to produce; the software cost about Can $2000 (US $1354), and there was an additional cost of less than Can $50 (US $34) for film and disks to produce the images used in the module. The module was initially developed as part of a graduate course in instructional design for multimedia and was the first project of this nature for the author, so a somewhat heuristic approach was taken. Instructional design, programming, formative evaluation, and revisions took about 100 hours. Instructional design was the most time-consuming aspect but could be facilitated in another situation by employing an expert in this field as part of the team. Comparing this cost in time and money with the costs of a seminar, which may be more or less difficult to organize depending on the situation (contrast clerks doing a core otolaryngology rotation with primary care physicians in a rural area), or a print publication is difficult because there are a number of unknown variables in each of these endeavors.
CONCLUSION
In this study, both the cognitive and psychomotor knowledge important for treating a patient with epistaxis was effectively learned by medical clerks using either a simple computer module or a small-group, case-based, student-centered seminar.
Thanks to Glenn Regehr, PhD, for his advice and support throughout this project; to the undergraduate otolaryngology clerkship coordinators, Drs Jennifer Anderson, Jerry Chapnik, and Joseph Chen, for their assistance; and to the medical students of the University of Toronto Class of 1998.