Student evaluations of teaching (SET): implications for medical education in psychiatry and an approach to evaluating SET and student performance

SET: concurrent and consequent performance of students

It is well known that SET is correlated with performance on concurrent courses, and a number of reasons, including but not limited to grade inflation, have been proposed as factors. ⁴ However, there are a number of studies that demonstrate more positive SET in regard to specific university educators does not necessarily correspond to improved performance of students in subsequent courses. ⁴

In an excellent review, Stroebe ⁴ describes five studies that relate teaching ratings in a concurrent course to student performance in subsequent courses, which we summarise here. Johnson ⁵ found that some items of SET were negatively correlated with grades on subsequent courses. ⁴ Yunker and Yunker ⁶ studied teaching evaluations and student performance in introductory and intermediate accounting courses, with the result that there was a negative correlation between evaluation of the introductory course, and performance in the subsequent course. ⁴ Weinberg et al. ⁷ investigated students undertaking introductory and intermediate economics courses at Ohio State University from 1995 to 2004, finding course ratings were correlated with grades in the concurrent course, but not in subsequent courses. ⁴ Carrell and West ⁸ conducted a study at the US Airforce Academy, measuring student performance in mandatory follow-on classes, finding that professors who excelled on SET in concurrent courses taught in ways that improved SET, but at the cost of poorer performance of students in more advanced follow-on classes. In a study at the Italian private university, Bocconi, Braga et al. ⁹ found students’ performance in follow-on coursework was negatively correlated with the SET of the professors.

Consequently, there is emerging evidence that SET is inversely proportional to the performance of students on subsequent courses, i.e. the higher the SET ratings, the poorer the students perform in subsequent studies. It has been posited that more experienced educators who effectively teach a foundation for later study receive lower SET scores as a consequence of the degree of challenge that students experience. ⁸ Conversely, those educators that provide less challenging teaching, that perhaps reinforces the illusion of mastery of the course material, ¹⁰ while at the same time being potentially vulnerable to grade inflation due to easy assessments, are more likely to be rated highly on SET. In medicine, assessment drives learning. Accordingly students are likely to rate highly any educator who helps them to pass the next barrier exam and reach the next stage of training. ¹¹ Of course, this appears to be an easier process in medical specialties assessment, especially in more easily operationalised and circumscribed fields such as surgical anatomy. ¹¹ However, there is a lack of research specific to SET and ASP in medical fields, and such research would better inform future medical curriculum development.

Addressing SET concurrent versus consequent performance

On the basis of the above, we propose that SET must be accompanied by assessment of student performance (ASP) longitudinally, especially in medical education in so-called ‘Cinderella’ specialities such as psychiatry and general practice; in which the subjective component of SET, infused with negative sentiment, may, if assessed without ASP, be invalid. ² However, the very considerable challenge is that there is insufficient time and the lack of an appropriate follow-on course in which to assess performance within Australian postgraduate medical school programmes. For example, there is frequently no directly related subsequent course to psychiatry, as it is often taught with surgery, general medicine and obstetrics/gynaecology. While psychiatric skills could be assessed in medical interns or junior medical officers, this would require a separate assessment process that would be confounded by not directly following on from the medical school teaching and the presence of possible intervening factors.

A possible solution, which has the added benefit of enhancing learning, ¹² is the development of progressive self-assessment tasks during the course of the term, which are graded for formative purposes, but separate from the summative formal course performance examinations. For example, this may involve second weekly or mid-term self-assessment tasks, with the grading provided to students to enable them to calibrate their learning, during the course of a typical 8-week psychiatry term. This would yield at least one to two data points per student to assess performance within the term, a third at the formal summative examinations. Such self-assessment tasks should ideally be available on demand via computer marking, with detailed feedback to guide further learning. Additionally, a further assessment in the pre-internship program, would result in three or more data points for ASP. Similarly, SET should be assessed contemporaneously with ASP throughout the medical education process.

Progressive assessment has become the new norm in medical education and, although yet to be widely adopted, is gaining impetus to achieve more effective learning. ¹³ We are proposing a model that involves appropriately graded formative examinations that allow students to calibrate their knowledge against the final summative examinations for assessment of student performance combined with SET at the end of each teaching term (see below). In our model, past summative examination questions (multiple choice and extended matching, at our university) will initially be administered as a self-assessment task, with feedback and discussion of results midway in an 8-week term (of which there are four per year). The use of an examination that facilitates testing and reflection upon knowledge is consistent with the cognitive science of learning, enhanced understanding and consolidation of learning. ¹² The results of the combined/whole year students’ mid-term assessments may then be compared with the results from the end-of-year summative examination. Further extension of assessment of performance into the pre-internship period and pre-specialisation junior medical office postgraduate education will add real-world validity, but will necessarily be more temporally distant from, and thus less clearly linked to, SET.

Conclusion

Building contemporaneous SET as well as graded formative and summative ASP longitudinally into medical education may yield useful information on the relationship of the two types of feedback, more accurately informing on the effectiveness of teaching and student performance. In contrast to the cynical approach, we can thus assess the value as well as the price (or consumer sentiment) of medical education in psychiatry and related disciplines.