The Use of Student Evaluations of University Teaching in Different Settings: The Applicability Paradigm

Abstract

The applicability paradigm has been used in five studies to evaluate the applicability of items from two North American instruments designed to measure students' evaluations of teaching effectiveness. In the present investigation, the paradigm is used at the new University of Western Sydney, Macarthur (UWSM). Items from both instruments were seen as appropriate and important, and differentiated among lecturers chosen as good, average and poor teachers. A multitrait-multimethod analysis of responses from the two instruments supported their convergent and divergent validity. The pattern of items judged to be most important at UWSM was more similar to patterns found at two research universities than patterns in the technical and further education sector or at two other institutions. These results support the applicability of the instruments at UWSM and across a diversity of educational settings.

Keywords

comparative analysis student evaluation teacher evaluation of teacher performance higher education teacher effectiveness university environment

References

Braskamp

L.A.

Brandenburg

D.C.

Ory

J.C.

(1985). Evaluating teaching effectiveness: A practical guide. Beverly Hills, CA: Sage

Campbell

D.T.

Fiske

D.W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105

Centra

J.A.

(1979). Determining faculty effectiveness. San Francisco: Jossey-Bass

Clarkson

P.C.

(1984). Papua New Guinea students' perceptions of mathematics lecturers. Journal of Educational Psychology, 76, 1386–1395

Cohen

P.A.

(1980). Effectiveness of student-rating feedback for improving college instruction: A meta-analysis. Research in Higher Education, 13, 321–341

Cohen

P.A.

(1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51, 281–309

Doyle

K.O.

(1983). Evaluating teaching, Lexington, MA: Lexington Books

Dunkin

M.J.

(1990). Willingness to obtain student evaluations as a criterion of academic staff performance. Higher Education Research and Development, 9, 51–60

Feldman

K.A.

(1976). The superior college teacher from the student's view. Research in Higher Education, 5, 243–288

10.

Feldman

K.A.

(1983). The seniority and instructional experience of college teachers as related to the evaluations they receive from their students. Research in Higher Education, 18, 3–124

11.

Feldman

K.A.

(1984). Class size and students' evaluation of college teacher and courses: A closer look. Research in Higher Education, 21, 45–116

12.

Frey

P.W.

(1973). Student ratings of teaching: Validity of several factors. Science, 182, 83–85

13.

Frey

P.W.

(1978). A two-dimensional analysis of student ratings of instruction. Research in Higher Education, 9, 69–91

14.

Frey

P.W.

Leonard

D.W.

Beatty

W.W.

(1975). Student ratings of instruction: Validation research. American Educational Research Journal, 12, 327–336

15.

Harman

(1989). The Dawkins reconstruction of Australian higher education. Paper presented at the Annual Meeting of the American Educational Research Association, 27–31 March, San Francisco

16.

Hayton

G.E.

(1983). An investigation of the applicability in technical and further education of a student evaluation of teaching instrument. Unpublished thesis, Faculty of Education, University of Sydney

17.

Hildebrand

Wilson

R.C.

Dienst

E.R.

(1971). Evaluating university teaching. Berkeley, CA: University of California, Center for Research and Development in Higher Education

18.

Isaacs

(1989). Changes in ratings for staff who evaluated their teaching more than once. Assessment and Evaluation in Higher Education, 14, 1–10

19.

Marsh

H.W.

(1977). The validity of students' evaluations: Classroom evaluations of instructors independently nominated as best and worst teachers by graduating seniors. American Educational Research Journal, 14, 441–447

20.

Marsh

H.W.

(1980). The influence of student, course and instructor characteristics on evaluations of university teaching. American Educational Research Journal, 17, 219–237

21.

Marsh

H.W.

(1981a). Students' evaluations of tertiary instruction: Testing the applicability of American surveys in an Australian setting. Australian Journal of Education, 25, 177–192

22.

Marsh

H.W.

(1981b). The use of path analysis to estimate teacher and course effects in student ratings of instructional effectiveness. Applied Psychological Measurement, 6, 47–60

23.

Marsh

H.W.

(1982a). SEEQ: A reliable, valid, and useful instrument for collecting students' evaluations of university teaching. British Journal of Educational Psychology, 52, 77–95

24.

Marsh

H.W.

(1982b). Validity of students' evaluations of college teaching: A multitrait-multimethod analysis. Journal of Educational Psychology, 74, 264–279

25.

Marsh

H.W.

(1983). Multidimensional ratings of teaching effectiveness by students from different academic settings and their relation to student/course/instructor characteristics. Journal of Educational Psychology, 75, 150–166

26.

Marsh

H.W.

(1984). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology, 76, 707–754

27.

Marsh

H.W.

(1986). Applicability paradigm: Students' evaluations of teaching effectiveness in different countries. Journal of Educational Psychology, 78, 465–473

28.

Marsh

H.W.

(1987). Students' evaluations of university teaching: Research findings, methodological issues, and directions for future research [Special issue]. International Journal of Educational Research, 11, 253–388

29.

Marsh

H.W.

(1988). Multitrait-multimethod analyses. In Keeves

J.P.

(Ed.), Educational research methodology, measurement and evaluation: An international handbook (pp. 570–580. Oxford: Pergamon Press

30.

Marsh

H.W.

(1991). Multidimensional students' evaluations of teaching effectiveness: A test of alternative higher-order structures. Journal of Educational Psychology, 83, 285–296

31.

Marsh

H.W.

Bailey

(1991). Multidimensional students' evaluations of teaching effectiveness: A profile analysis. Manuscript submitted for publication

32.

Marsh

H.W.

Fleiner

Thomas

C.S.

(1975). Validity and usefulness of student evaluations of instructional quality. Journal of Educational Psychology, 67, 833–839

33.

Marsh

H.W.

Hocevar

(1991a). The multidimensionality of students' evaluations of teaching effectiveness: The generality of factor structure across academic discipline, instructor level, and course level. Teaching and Teacher Education, 7, 9–18

34.

Marsh

H.W.

Hocevar

(1991b). Students' evaluations of teaching effectiveness: The stability of mean ratings of the same teacher over a 13-year period. Teaching and Teacher Education, 7, 303–314

35.

Marsh

H.W.

Overall

J.U.

(1980). Validity of students' evaluations of teaching effectiveness: Cognitive and affective criteria. Journal of Educational Psychology, 72, 468–475

36.

Marsh

H.W.

Overall

J.U.

(1981). The relative influence of course level, course type, and instructor on students' evaluations of college teaching. American Educational Research Journal, 18, 103–112

37.

Marsh

H.W.

Overall

J.U.

Kesler

S.P.

(1979). Validity of students' evaluations of teaching effectiveness: A comparison of faculty self-evaluations and evaluations by their students. Journal of Educational Psychology, 71, 149–160

38.

Marsh

H.W.

Touron

Wheeler

(1985). Students' evaluations of university instructors: The applicability of American instruments in a Spanish setting. Teaching and Teacher Education: An International Journal of Research and Studies, 1, 123–138

39.

McKeachie

W.J.

(1979, O). Student ratings of faculty: A reprise. Academic, pp. 384–397

40.

Meek

V.L.

O'Neill

Organizational change in Australian higher education: Process and outcome. Australian Educational Researcher, 17, 1–23

41.

Moses

(1986a). Self and student evaluations of academic staff. Assessment and Evaluation in Higher Education, 11, 76–86

42.

Moses

(1986b). Student evaluation of teaching in an Australian university: Staff perceptions and reactions. Assessment and Evaluation in Higher Education, 11, 117–129

43.

Murray

H.G.

(1980). Evaluating university teaching: A review of research. Toronto: Ontario Confederation of University Faculty Associations

44.

Nunnally

J.C.

(1978). Psychometric theory (2nd ed). New York: McGraw-Hill

45.

Overall

J.U.

Marsh

H.W.

(1979). Midterm feedback from students: Its relationship to instructional improvement and students' cognitive and affective outcomes. Journal of Educational Psychology, 71, 856–865

46.

Overall

J.U.

Marsh

H.W.

(1980). Students' evaluations of instruction: A longitudinal study of their stability. Journal of Educational Psychology, 72, 321–325

47.

Prosser

Trigwell

(1990). Student evaluations of teaching and courses: Student study strategies as a criterion of validity. Higher Education, 20, 135–142

48.

SPSS. (1986). SPSS-x user's guide (2nd ed.). New York: McGraw-Hill

49.

Warrington

W.G.

(1973). Student evaluation of instruction at Michigan State University. In Sockloff

A.L.

(Ed.), Proceedings: The first invitational conference on faculty effectiveness as evaluated by students (pp. 164–182). Philadelphia: Temple University, Measurement and Research Center

50.

Watkins

Marsh

H.W.

Young

(1987). Evaluating tertiary teaching: A New Zealand perspective. Teaching and Teacher Education: An International Journal of Research and Studies, 2, 41–53