Are you sitting down? Good. Because I have a shocking confession.
For the past few years, I’ve reserved 40 minutes of class time each semester to discuss bias in student evaluations of teaching (SETs) with my students. You know, the ones who are actually evaluating their teachers? I know. Scandalous.
Why? Well, because if I do, then others won’t have to (more on that later). And because I find student evaluations essential and want them to stick around. And because I had to do something in response to the overwhelming evidence of harmful bias, and the something I know how to do is teach. Here’s an FAQ.
Why involve students?
SCENE: INTERIOR—NOON—TUESDAY. “What are you reading right now, Professor DeWall?” Nearly a dozen students had squeezed into my office for our weekly Literary Interest Society meeting. “Well, most recently,” I gestured toward my laptop, still open to the Chronicle, “another article on bias in student evals.”
“Bias?” one asked. “Yeah,” I said, “higher scores for attractive instructors and lenient graders; race and gender may play a role.” Silence. Blank stares. I snapped my laptop shut like I’d been caught spilling state secrets.
“Why don’t we talk to students about this?” I texted a colleague later that day. “That’s a no-brainer,” he replied. “When the parents are fighting, they shouldn’t put their children in the middle.”
“Makes sense,” I thought. “But wait . . . Why would I treat my students like children who don’t belong at the grown-ups table? If both measurement and equity biases are as persistent as research suggests, then shouldn’t I be doing more than my perfunctory, ‘I value your feedback. Here’s some extra credit’ spiel? If I’m going to ask students—OK, relentlessly pester students—to complete SETs, shouldn’t I teach them how to do so thoughtfully, self-critically, and with attention to the larger context (or, at the very least, with more care than a Yelp review)? And what if the most elegant solution had been right in front of me all along? In the middle. Where my students are.”
So many questions. It was time for some research. Turns out, reactions to SETs’ bias range from “Meh, it’s inevitable” to “They’re flawed, but they’re all we’ve got” to “How can we sleep at night?” The debate’s hypercharged tenor didn’t surprise me, of course: of all the issues in higher ed, this one seems tailor-made to tug at the seams of how we imagine ourselves as professionals. And as people. We want students to have a voice but know their voices can sometimes wound. We’re eager for feedback but leery of how our institution may use it. We cherish students’ firsthand perspectives on our teaching but wonder—deep down—how accurate they really are. Scylla, meet Charybdis.
The most intriguing aspect of scholarship on SETs’ bias? Its peculiar hourglass shape: wide at both ends and not much in the middle. That is, we’ve largely focused on our choices before and after submission and neglected—(avoided?)—direct interventions with students themselves. Lead-up and aftermath; prologue and epilogue. Like if we could just get it right in pre- and postproduction, the movie would film itself.
On the front end, we wonder whether we should call them “student perceptions of learning” or “student experience questionnaires” instead (Kreitzer & Sweet-Cushman, 2021); offer inducements like extra credit (Jaquett, VanMaaren, & Williams, 2016); rely on qualitative or quantitative questions (Varel, 2022); administer them in person or online (Stanny & Arruda, 2017); worry about scale effects (Courey & Lee, 2021); or keep them anonymous (Fowler, 2019). On the back end, we offer methods for incorporating students’ feedback into our teaching (Boysen, 2016); advice on how to filter out abusive and unhelpful comments (Supiano, 2019); opinions on using SETs in tenure and promotion decisions (Lakeman et al., 2023); and suggestions on how many glasses of wine to drink before reading them (OK, that one’s mine). We’ve even created our own subgenre: the tips-on-how-to-handle-negative-evals essay, with subtitles like “Seven Ways to Soothe the Sting” (Artze-Vega, 2014) and “These 6 Strategies Can Help You Cope” (Allen et al., 2022).
I found only two studies on direct, student-centered interventions—anti-bias language on evaluation forms and email reminders—and these showed promising (Peterson et al., 2019), albeit mixed (Key & Ardoin, 2019), results. The most comprehensive meta-analysis of SETs’ bias to date has since identified this approach as “an area ripe for future research” (Kreitzer & Sweet-Cushman, 2021, p. 80).
If merely cutting and pasting boilerplate anti-bias language onto an evaluation form showed promise, maybe we’d gain even more traction if we discussed the issue with students directly?
It’s worth a try.
Isn’t having this conversation unprofessional?
I used to think so. Now I think it’s unprofessional not to.
Because here’s what’s really unprofessional: Inconsistency. Opacity. In the end, excluding students from this conversation felt misaligned with everything else that I valued in my teaching. Antithetical, even.
With everything else: “Take intellectual risks. Don’t avoid important conversations just because they’re uncomfortable.”
With SETs: “Discuss bias in student evaluations? No way. Too thorny.”
With everything else: “Think critically. Always. But especially about diversity, power, and privilege.”
With SETs: “Never mind the overwhelming body of evidence that suggests SETs’ bias may disadvantage faculty of color, women faculty, and other marginalized groups. Please submit your evaluations by the last day of classes.”
With everything else: “Giving meaningful feedback is a skill that you can learn. Let’s practice!”
With SETs: “I can’t waste class time preparing students for this writing task! I’ll focus on completion rates instead.”
With everything else: “Always consider the rhetorical contexts—audience, purpose—of the texts that you produce.”
With SETs: “You needn’t worry yourselves about who sees evaluations or how they’re used. Here’s some extra credit.”
With everything else: “You can trust me. I’ll be honest and transparent with you.”
With SETs: “Pay no attention to the bias behind the curtain. Here, have some cookies!”
OK, I’ve never done the cookies thing (anything I bake would work against me). But withholding the whole story from students became disconcerting. Then uncomfortable. Then intolerable. And just plain unnecessary. “What they don’t know won’t hurt them” just wasn’t going to cut it anymore.
Isn’t this a waste of class time?
It depends how you define waste.
For the record, I’m notoriously jealous of class time. I even nurse an irrational grudge against the tornado drill that gobbles up 20 minutes of my Tuesday morning class once per semester.
But talking with students about SETs’ bias is not unlike that drill: essential preparation for responsible membership in a college community. Bonus points if this conversation is embedded in a course that students typically take early on, like first-year seminar or English composition. After all, completing SETs is not an insignificant part of our students’ college experiences.
Let’s crunch the numbers: if my students average four classes per semester, they’ll complete at least 32 evaluations; tack on a few music lessons, labs, and internships, and we’re pushing 40. Students have no business being as obsessed with SETs as we are (sidebar: neither do we), but complete obliviousness shouldn’t really be the goal either. With anything.
For me, this conversation provides a valuable end-of-term cumulative assessment for the methods of persuasion, logical fallacies, and feedback strategies that we’ve covered during the semester. I won’t pretend to know how a psychology, sociology, or human resources class may incorporate the conversation, but fundamental attribution error, gender norms, and diversity in the workplace spring to mind.
And anyway, there are learning objectives, and then there are Learning Objectives. You know, of the college-should-prepare-students-to-be-good-human-beings variety. I suspect this conversation about SETs’ bias aligns with your institution’s overall mission.
Let’s check. Does your institution’s mission statement mention leadership? Mine too. Engagement? Indeed. Citizenship? Ditto. Lifelong learning? We’re on a roll! My discussions of SETs’ bias often support each of these outcomes even more directly than my discipline-specific course content. If contributing our skills, labor, and expertise in service of our employers’ missions is a waste of class time, then we may need to rethink our approach.
Isn’t this conversation awkward?
Come on, since when do we avoid awkward conversations? Anyone who’s sat through a few faculty meetings would think we gravitate toward them (ba-dum-ching!).
And besides, nothingcould be as awkward as the cloak-and-dagger routine I performed during my first few years of teaching. Me, circa 2004: slink sheepishly into class; distribute evals (on paper—how quaint!); mumble something about “teaching effectiveness”; get the hell out of there, fingers crossed behind my back. Or, worse yet, remind students via email so I’d avoid a face-to-face conversation altogether. As my 13-year-old would say, “So cringe.” Me, circa now: “OK, everyone. As I’ve mentioned a few times this semester, we’re going to spend 40 minutes today discussing bias in student evaluations. Here’s why I think this conversation is important.” Better.
I’ll admit that discussing SETs’ bias with students did feel unnatural at first. It was funny, really: I had no problem engaging with them on race in Othello; antisemitism in The Canterbury Tales; or feminism in Beowulf. But this felt different. Inappropriate. Unseemly. A how-the-sausage-gets-made conversation that didn’t belong in the classroom.
To be clear, I’m not suggesting that we disclose every aspect of our work lives with students; frankly, I’m often surprised by some instructors’ willingness to discuss what I consider to be need-to-know-only details with students. Then again, I descend from a long line of tight-lipped Midwesterners for whom “airing one’s dirty laundry” was the ultimate sin, so I won’t even complain about classroom technology glitches in front of students (“Shakespeare did just fine without the internet, and so can we!”).
But talking with students about SETs’ bias is not the same as complaining to them about budget issues or staffing crises; those are best left behind closed doors. This, however, is a controversial, complicated topic in which our students have a stake and over which they can actually exercise some control. Isn’t that the very definition of a teachable moment?
Isn’t this conversation coercive, like we’re trying to influence the outcome?
Good question. First, tone matters: you’re going for calibration, not coercion. Less “LET’S RAGE AGAINST THE MACHINE!” and more “Guess what? We all possess miserly human brains that are prone to bias. What an exciting opportunity to think about our own thinking!” Neither fire nor brimstone. If you use cognitive wrappers throughout the semester like I do, then engaging in another metacognitive activity isn’t a huge stretch for students.
And let me be very clear: I am absolutely trying to influence the outcome. If, that is, the outcome we seek is threefold: (1) students who are more practiced in thinking critically about their own potential biases, (2) workplaces that are safer and more equitable for all instructors, and (3) student evaluations that teachers can actually use to improve their teaching. Although some institutions have abandoned them altogether, SETs remain irresistibly cheap, easy, and built for mass distribution, so they’ll probably stay put (then again, The Lecture was too, so we’ll see). For now, we need to work within the system to find low-impact, commonsense solutions.
Still concerned? Here’s my hack: ask your students to complete evaluations for your class a little early, before you discuss bias with them. Our online assessment portal shows me who has submitted evaluations long before I can view what they wrote. That way, I have no skin in the game. And anyway, I’m not having this conversation for me. Or, really, for my students (although it’s usually quite productive).
Who, then? Well, here goes: My colleague who’s crying in her office because her pregnancy made students “uncomfortable.” My colleagues whose courses are inherently more controversial than mine. My women colleagues in STEM disciplines. My colleague—a brilliant teacher and self-declared “profound introvert”—who gets dinged for not exuding the “enthusiasm” that students equate with good teaching. My more vulnerable, untenured colleagues. My contingent colleagues whose already-tenuous employment rides almost exclusively on SET scores. My queer colleague whose student accused him of “brainwashing” them by “acting gay.” My colleague whose students’ comments about her accent were so vicious that she left the profession all together. To name a few.
And here’s the most critical piece: faculty members who stand to be most adversely affected by SETs’ bias should not be expected to have these conversations with students. Full stop. Unless they want to, of course. But talking to students may be more treacherous for them: haunted by the specter of backlash; tainted by the appearance of self-interest.
So, who should be putting themselves in the middle? Well, I should be. Tenured professors should be doing the heavy lifting, especially full professors. To be blunt: our privilege can absorb the shock. Imagine if a core group of tenured full professors on each campus committed to having direct, research-based conversations with their students about SETs’ bias. Would it turn the tide? I don’t know. But it’s a start. A ripple. A wave. A . . . sea change?
What does this conversation even look like?
Next month, I’ll share a fly-on-the-wall transcript of how it usually goes. Stay tuned.
References
Allen, K., Crawford, J., & Waters, L. (2022, September 13). Negative feedback is part of academic (and life)—These 6 strategies can help you cope. The Conversation. https://theconversation.com/negative-feedback-is-part-of-academia-and-life-these-6-strategies-can-help-you-cope-190069
Artze-Vega, I. (2014, December 8). Cruel student comments: Seven ways to soothe the sting. Faculty Focus. https://www.facultyfocus.com/articles/faculty-development/cruel-student-comments-seven-ways-soothe-sting
Boysen, G. A. (2016). Using student evaluations to improve teaching: Evidence-based recommendations. Scholarship of Teaching and Learning in Psychology, 2(4), 273–284. https://doi.org/10.1037/stl0000069
Courey, K. A., & Lee, M. D. (2021). A model-based examination of scale effects in student evaluations of seaching. AERA Open, 7. https://doi.org/10.1177/23328584211040083
Fowler, R. (2019, June 22). Protecting students or enabling trolls? Why faculty evaluations should not be totally anonymous. Scatterplot. https://scatter.wordpress.com/2019/06/22/protecting-students-or-enabling-trolls-why-faculty-evaluations-should-not-be-totally-anonymous
Jaquett, C. M., VanMaaren, V. G., & Williams, R. L. (2016). The effect of extra-credit incentives on student submission of end-of-course evaluations. Scholarship of Teaching and Learning in Psychology, 2(1), 49–61. https://doi.org/10.1037/stl0000052
Key, E., & Ardoin, P. (2019, August 20). Students rate male instructors more highly than female instructors. We tried to counter that hidden bias. The Washington Post. https://www.washingtonpost.com/politics/2019/08/20/students-rate-male-instructors-more-highly-than-female-instructors-we-tried-counter-that-hidden-bias
Kreitzer, R. J., & Sweet-Cushman, J. (2021). Evaluating student evaluations of teaching: A review of measurement and equity bias in SETs and recommendations for ethical reform. Journal of Academic Ethics, 20, 73–84. https://doi.org/10.1007/s10805-021-09400-w
Lakeman, R., Coutts, R., Hutchinson, M., Massey, D., Nasrawi, D., Fielden, J., & Lee, M. (2023). Playing the SET game: How teachers view the impact of student evaluation on the experience of teaching and learning, Assessment & Evaluation in Higher Education, 48(6), 749–759. https://doi.org/10.1080/02602938.2022.2126430
Peterson, D., Biederman L., Andersen, D., Ditonto T., & Roe K. (2019). Mitigating gender bias in student evaluations of teaching. PLoS ONE, 14(5): e0216241. https://doi.org/10.1371/journal.pone.0216241
Stanny, C. J., & Arruda, J. E. (2017). A comparison of student evaluations of teaching with online and paper-based administration. Scholarship of Teaching and Learning in Psychology, 3(3), 198–207. https://doi.org/10.1037/stl0000087
Supiano, B. (2019, March 14). One way to take the sting out of student feedback. The Chronicle of Higher Education. https://www.chronicle.com/newsletter/teaching/2019-03-14
Varel, D. (2022, February 2). How and why colleges should reform student evaluations. Academe Blog. https://academeblog.org/2022/02/02/how-and-why-colleges-should-reform-student-evaluations
Nichole DeWall, PhD, is a professor of English at McKendree University in Lebanon, Illinois, where she teaches Shakespeare, medieval and early modern literature, drama, and composition courses.
