“Even measures with perfect validity can be rendered useless if they are interpreted incorrectly, and anecdotal evidence suggests that teaching evaluations are frequently the subject of unwarranted interpretations based on assumed levels of precision that they do not possess.” (p. 641) And now there’s some
Whatever philosophical and empirical issues college teachers may have with the Rate My Professor (RMP) website, there is no denying that the site in now a huge repository of information on college teachers. The website reports that it contains 15 million ratings for 1.4 million
If you’re looking for a way to improve your teaching, consider teaching squares. A teaching square consists of four faculty from different disciplines who visit each other’s classes within a two-to-three-week period. After the classroom visits, the four gather around coffee or a meal to
Five faculty, all belonging to the same interdisciplinary sociology department, decided that collectively they could improve student writing skills better than they could individually. “Our approach emphasizes that a collective effort need not be a department-wide, institutionalized one. Indeed, faculty can still collaborate and students
A thoroughly referenced article seeks to answer why science faculty members are slow to adopt evidence-based teaching practices, despite what the authors describe as “heroic dissemination” of information on these practices. The folks on the science side of the house have evidence that use of
When student ratings first started being widely used and the research associated with them was prolific, many faculty objected to the focus on teaching effectiveness. “It shouldn’t be about me! I’m a good teacher if my students are learning. Measure how much my students have
This article explains areas that instructors need to work on in order to be successful: knowledge base of teaching and learning, underprepared students, instructional management, teaching strategies, assessment, and passion.
How much instructional advice have you heard over the years? How often when you talk about an instructional issue are you given advice, whether you ask for it or not? Let’s say you’re a new teacher or you’re teaching a class you haven’t taught before
When we talk about teaching effectiveness, it’s usually in the context of evaluation. Student ratings are frequently described as measures of teaching effectiveness, and that makes our understanding of the term important. Researcher Leslie Layne wondered whether students and teachers define the term similarly. If
Magna Publications © 2023 All rights reserved
“Even measures with perfect validity can be rendered useless if they are interpreted incorrectly, and anecdotal evidence suggests that teaching evaluations are frequently the subject of unwarranted interpretations based on assumed levels of precision that they do not possess.” (p. 641) And now there's some research verifying that faculty and administrators do make unwarranted interpretations. “We investigated if differences in teaching evaluations that are small enough to be within the standard error of measurement would still have significant effects on judgments made about teachers.” (p. 641)
There's no question that teaching evaluations matter. Teachers and administrators, including department chairs, deans, and provosts, take them seriously and make decisions based on the results. The problem is that the quantitative nature of the ratings makes the data appear precise and objective. A score of 4.62 is higher than a score of 4.34. However, that does not automatically mean that the faculty member with the higher score is superior, more effective, or a better teacher than the faculty member with the lower score. “As is true of all measurements, the means produced by teaching evaluations are only an estimate of the true score; sources of error—such as small sample sizes, outliers, misinterpretation of questions, and less-than-perfect reliability—interfere with the measure of true scores.” (p. 643) So what looks precise and reliable provides no more than a “veneer of objectivity.” (p. 643)
The huge research exploration of student ratings that occurred during the '80s and '90s clearly delineated appropriate statistical standards for interpreting evaluations. Unfortunately, what has been established empirically is not always implemented in practice, as this research demonstrates.
From three separate but related studies, randomly selected faculty and administrators from randomly selected postsecondary institutions and in randomly selected disciplines read short scenarios that contained the same details except for the teacher ratings, which were slightly higher or lower. For example, in the first study, 57 faculty members considered hiring two potential candidates and allocating a travel award. In the second study, 80 department chairs assessed two untenured faculty members who had implemented an instructional innovation. In the case of the innovations, the overall course rating was 4.20 in one scenario and 3.92 in the second one.
The results of both studies were the same. The rating differences had significant effects on the judgments made about these fictional teachers. Writing about Study 1, the researchers note that “even without critical statistical information needed for interpretation, small increases in an overall teaching evaluation led participants to perceive teachers as significantly more deserving of a merit-based reward; this is a meaningful effect for a teaching evaluation of less than a third of a point on a five-point scale.” (p.647) In Study 2, “small changes in raw means led to statistically significant differences in judgments about teaching techniques.” (p. 649)
In Study 3, the research team decided to include both the means and the standard deviations needed to interpret them. In this case, 48 faculty were asked to imagine they were on a committee in charge of evaluating faculty for reappointment. The scenarios described two faculty members, again with virtually identical teaching assignments, course designs, and instructional methods. A table listed their student evaluation means and standard deviations. “The results of Study 3 provide the most convincing support yet that small differences in fictional teaching evaluations have a significant impact on judgments. Study 3 allowed participants to consider full and realistic information about teaching evaluations. Still, faculty in the study judged variations as small as 0.15 of a point as meaningful. The statistical tests of significance indicate that participants were providing reliably different ratings based on teaching evaluations that were . . . highly unlikely to be reliably different.” (p. 653)
“Despite the importance of teaching evaluations and the simplicity of the principles for their interpretation, the current studies illustrate the relative ease with which faculty members and department heads can be led to make inappropriate generalizations from limited data.” (p. 653) These are not encouraging findings, although for most of us they do not come as a surprise. Let them be a wake-up call, whether you're a teacher looking at your own results, assessing those of a peer, or conversing with the department chair about your own results. If you happen to be serving on a search committee or doing tenure reviews with colleagues, this would be a great article to review and discuss before looking at teaching evaluation results.
Reference: Boysen, G. A., Kelly, T. J., Raesly, H. N. & Casner, R. W. (2014). The (mis)interpretation of teaching evaluations by college faculty and administrators. Assessment & Evaluation in Higher Education, 39 (6), 641-656.
