Thursday
August 28, 2008
Advertisement
Advertisement
News, Views and Careers for All of Higher Education
April 25
Validation for RateMyProfessors.com?
You’ve heard the reasons why professors don’t trust RateMyProfessors.com, the Web site to which students flock. Students who don’t do the work have equal say with those who do. The best way to get good ratings is to be relatively easy on grades, good looking or both, and so forth.
Related stories
But what if the much derided Web site’s rankings have a high correlation with markers that are more widely accepted as measures of faculty performance? Last year, a scholarly study found a high correlation between RateMyProfessors.com and a university’s own system of student evaluations. Now, a new study is finding a high correlation between RateMyProfessors and a student evaluation system used nationally.
A new study is about to appear in the journal Assessment & Evaluation in Higher Education and it will argue that there are similarities in the rankings in RateMyProfessors.com and IDEA, a student evaluation system used at about 275 colleges nationally and run by a nonprofit group affiliated with Kansas State University.
What is notable is that while RateMyProfessors.com gives power to students, IDEA gives a lot of control over the process to faculty members. Professors identify the teaching objectives that are important to the class, and those are the measures that count the most. In addition, weighting is used so that adjustments are made for factors beyond professors’ control, such as class size, student work habits and so forth — all variables that RateMyProfessors doesn’t really account for (or try to account for).
The study looked at the rankings of 126 professors at Lander University, in South Carolina, and compared the two ratings systems. The findings:
- Student rankings on the ease of courses were consistent in both systems and correlated with grades.
- Professors’ rankings for “clarity” and “helpfulness” on RateMyProfessors.com correlated with overall rankings for course excellence on IDEA.
- The similarities were such that, the journal article says, they offer “preliminary support for the validity of the evaluations on RateMyProfessors.com.”
The study was conducted by Michael E. Stonntag, who formerly taught at Lander and who is now vice president for academic affairs at the University of Maine at Presque Isle, and by two psychology professors at Lander, Jonathan F. Bassett and Timothy Snyder.
Sonntag said that there are two ways to read the results: One is to say that RateMyProfessors.com is as good as an educationally devised system and the other would be to say that the latter is as poor as the former. But either way, he suggested, it should give pause to critics to know that the students’ Web site “does correlate with a respected tool.”
William H. Pallett, president of IDEA, said he was “surprised a bit” by the correlation between his organization’s rankings and those of RateMyProfessors.com. That’s because much of the criticism he has heard of the student oriented site is that rankings aren’t representative, while much of the effort at IDEA is based on assuring representative samples.
“I am surprised, given that we do attend to issues of reliability and validity and they acknowledge that they don’t,” he said.
Pallett cautioned, however, that IDEA is not intended to be a sole basis for evaluating a course or professor. He said that he would always advise departments to have professors evaluate on another, and to use student evaluations as just one part of that review.
Sonntag said that his current institution uses a home-grown student evaluation system, and that he has no plans to seek a change to IDEA or RateMyProfessors.com — and that the evaluation system is covered by a collective bargaining contract anyway. But he said that he hoped the study might prompt some to think about the online rankings in new ways.
For his part, Sonntag acknowledged that some RateMyProfessors.com reviews are “so mean-spirited” that they aren’t worth anyone’s time. But he said that if you cast those aside, there are valuable lessons to be learned. He said that he does check what the site says about his teaching — and has found reinforcement for some innovations and reason to question whether some of his tests were too difficult.
“I’ve been an instructor for 10 years. I look at it,” he said, adding that he has found insights “that weren’t on my teaching evaluations and I have thought: ‘Wow. I believe what the student has said is valid and perhaps I can change the way I teach.”
Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.
Advertisement
Comments
RateMyProfessor should be called RateAnyProfessor. One does not have to have taken a course to rate the professor. In fact, it is even possible to invent a course and rate the professor’s teaching in that non-existent course.
Richard, Chair at Virginia Polytechnic Institute and State University, at 8:10 am EDT on April 25, 2008
An IDEA Whose Time Has Come
Poll a 1000 college students and 900 will tell you they do not think their school or professors care much about their views. This is not merely sad, frustrating and demoralizing, but counterproductive to the ultimate mission of our universities to inspire our individual minds and collective culture toward a higher ideal for the human condition. No sustainable system exists without valid feedback and while the occasional student verbal trashing is inevitable, the broader pattern of issues raised, particularly if students believe it to be taken seriously, should be a welcome insight for improvement in a largely insular academic setting.
RJW, at 8:40 am EDT on April 25, 2008
What does this tell us?
I come down squarely in the “latter is as poor as the former” camp. It’s pathetic that the tools we have for evaluating teaching are so inaccurate. My good scores in my teaching evals and on ratemyprofessor in no way correlate to my lousy half-hearted teaching. *shrug*
I have yet to see any evidence that anyone has really given this issue any serious thought. There are plenty of people like me who make snarky comments at the coffee maker, but mostly we all accept that this is the system we have, and if we keep our heads down and pass out candy on evaluation day, we’ll do well enough to get promoted past caring.
The “education research” that shows up in my mailbox seems to go in circles before not concluding anything at all on this issue. (I love this quote in today’s NYTimes: “Dr. Kaminski and her colleagues Vladimir M. Sloutsky and Andrew F. Heckler did something relatively rare in education research: they performed a randomized, controlled experiment.” HA!)
Anyway, I have to run to the store and buy a bag of mini Snickers...the semester’s almost over...
Ric, Assoc Prof, at 9:30 am EDT on April 25, 2008
Although there are definitely flaws with a site like RateMyProfessors.com, it still serves as an indicator of what students think about certain professors. What I would like to see, though, is if students who get a lower grade in the class complain more than students who get a higher grade.
I get numerical and written feedback from my students through the standard evaluation form once a semester. The numerical feedback is all the department cares about; all I look at is the comments. The comments that I got from students last semester were so constructive that I passed out an anonymous comment sheet after a few weeks this semester.
I’m curious to see what sorts of feedback procedures work well for other TAs and professors. What are some of your best practices, higher ed land?
I’m off to teach an 8:50 AM econ class, without candy bars. Happy Friday, everyone!
Robert, PhD Student, at 9:45 am EDT on April 25, 2008
The Peer Factor
Does it make a difference that students using RateMyProfessors.com can read other students’ evaluations of the professor? To what extent are they engaging in a conversation with their peers, more than assessing the professional qualifications of their teachers?
tom, at 9:50 am EDT on April 25, 2008
This is surprising?
This is surprising? In 40 years of teaching, I never had a meaningful evaluation, nor have my many colleagues. Most colleges rely mainly on student evaluations, which is little better than a popularity vote, much like RateMyProf. Rather than a validation of RateMyProf, perhaps such results represent an invalidation of current college evaluation methods. And could such pathetic efforts to determine instructional efficacy have an influence on the fact that many college grads today can’t pass a 7th-grade-level general knowledge test?JAC
John A. Conners, at 10:15 am EDT on April 25, 2008
The Good News Is In
As long as Departments use their own, agreed-upon, well understand, and disciplinary based (or relevant?) measurements for evaluating one another, the whole dust-up over RateMyProfessors is meaningless.
But with one exception — while professors are eager, bold and expected to exclaim their opinions about public personalities, domestic and foreign policies and other assorted topic of the day, they seem remarkably edgy and nervous whenever anyone, especially students, turns the spotlight back on the professors.
RateMyProfessor does exactly that and for that reason alone it is a valuable and humorous site that should be visited often.
Chuck, at 10:25 am EDT on April 25, 2008
Abolish course evaluations!
So much the worse for course evaluations. One man’s modus ponens in another man’s modus tollens.
LogicGuru, at 12:25 pm EDT on April 25, 2008
Judith’s thought was my first thought too — what’s the surprise?!
My problem with RateMyProfessors is that I can recognize who’s who by their writing style, after a semester of essays & papers & presentations. Altho in-class evals also have a section for written comments, the students are usually in a hurry to get out of class and make very brief remarks, if any; on the website they take more time and then I can tell who is writing what. It’s not anonymous at that point and I can tell who is grinding what axe (or buttering which scone). Not sure why that matters to me, but it does. I haven’t looked at the site for a couple of years because of that. Feels like eavesdropping.
marya, at 1:00 pm EDT on April 25, 2008
RateMy Professor.com
As someone who have been praised as “the best professor I ever had,” and someone who “talks to much and needs a makeover,” I can’t really take this site seriously. No one has cared to mention the very volunteer nature of the sample here. You may correlate a lot of things. That doesn’t make a finding significant.
It’s pathetic that so many universities rely on the very subjective comments of students who are being evaluated (graded) by the very people they are evaluating.
I’ve taught for more than 10 years, and although I’ve generally received remarkably good evaluations for someone who needs a makeover, it wasn’t until I came to Hopkins that anyone ever observed my teaching, really looked at my syllabi, or expressed any genuine interest in the true quality and rigor of my teaching. Until then, I just had to keep the customers happy. That’s a sad comment on the state of higher education at some universities. —Jane Ph.D. The Johns Hopkins University
Jane L. Twomey, Program Coordinator at Johns Hopkins University, at 1:05 pm EDT on April 25, 2008
IDEA evaluations
My college uses the IDEA, and I can’t say I’m surprised by the correlation between that evaluation tool and ratemyprofessors.com. Both are inadequate and flawed systems for gathering and analyzing information about students’ experiences in a course.
The IDEA is designed to be completely generic so that it can be used in any college or university course in any discipline. While the IDEA designers would argue, I’m sure, that the results wind up being customized to the course because individual instructors must rank the essential, important, and not at all important elements of a class, the questions that appear on the actual evaluation are general to the point of being confusing. (Example: “This class helped me to communicate more effectively” is the sort of statement IDEA uses to determine whether students think a course improved their writing skills. Writing is a form of communication, but I suspect that professors who teach writing courses would get higher scores in a response to a question that asked specifically about writing.)
Since I’m not a statistician, I’m somewhat reluctant to point out other potential problems with the IDEA in detail. For now, though, I’ll say that someone in my college’s psychology department regularly meets with new faculty and explains how they should fill out the instructor form in order to raise evaluation scores. (It mostly involves only choosing one or two items as “essential.") Maybe someone with a statistics background can explain why this works? The larger point, of course, is that an evaluation that can be “gamed” in this way should not be taken too seriously.
Rather than being titled “Validation for ratemyprofessors.com,” this article could be labeled “Study demonstrates that the IDEA is no more reliable than much-ridiculed website.”
Erin, at 1:05 pm EDT on April 25, 2008
proof that SETs are invalid
This study was unnecessary. Research already shows that both Student Evaluations of Teaching (SETs) and RateMyProfessor ratings are completely invalid for the same reasons. They both correlated VERY strongly with measures of attractiveness, non-verbal dynamism and ease of grading on the other. Anyone who still believes that either of these measures is valid is either completely ignorant of the available research, an administrator still desperately seeking to justify a shortcut for the job of evaluating teaching, or a professor who has gotten high SET and/or RMP ratings and feels the need to validate themselves. Pathetic.
As for the supposedly humanitarian need to give students a chance to provide feedback, such feedback shouldn’t be encouraged if it ends in wrongfully harming some people and giving students the sense that they rule the classroom.
ex-prof, at 1:20 pm EDT on April 25, 2008
Article gets a D-
Ok Inside Higher Ed, if there is a valid correlation between a popular blog site and some global rating from IDEA, what is the coefficient—r = what? The implication in this shoddy article is that administrators could use Rate-my-Professor just as easily as IDEA and save a lot of money. I don’t see any research that shows Rate My Professor reflects any student learning or achievement. All I see here is that one general ratings form shows the same people are generally most popular on two ratings forms. The same would be true for all popular ratings forms.That any two forms based on popularity would show general agreement is exactly what is expected. Student ratings have an affective component. Ambady’s and Rosenthal’s “thin slices” studies determined in 1993 that students arrive at ratings for teachers after watching 30 seconds of silent content-free video that are highly consistent (r = 0.76) with end-of-semester ratings. Student ratings also reflect cognitive gains. The best meta-analyses show a positive general correlation between student learning and student ratings (r = 0.4 to 0.5). Of the two, it seems that the affective component is stronger. Students answer most general items through a gut feeling, but it’s not an empty gut feeling. It is a feeling informed by their experience and has some value as shown by the correlations with learning. In short, it’s worth doing student ratings as one measure—we want responsible student comments and thoughtful input.
Here, we should not confuse general trends in research with helping or making decisions about individuals. Psychological and educational research is about finding relationships and describing trends on large populations; evaluating professors is an evaluation for a personnel decision about an individual. The r-values of 0.4 and 0.5 don’t begin to carry the predictability to allow anyone to use a student rating form in the absence of a lot more solid data to make such a decision about an individual.
If posting anonymous comments seems good enough to some advocates as a basis on which to judge retaining professors, then there ought to be equivalent sites where faculty can anonymously rate their deans, provosts, and presidents. ("Pick-a-provost” has an equally nice ring, doesn’t it?). People who think professors being terrorized for their job security based on anonymous bloggers’ comments shouldn’t mind having the same standards applied to themselves.
Prof Ed, at 1:20 pm EDT on April 25, 2008
rate at will!
I’ve had quite a few students tell me that students tend to vote several times (there are ways of doing this) and either extra harshly or extra positively on Rate my Professor. When they make an effort to go to this site, it’s because they have an “extreme” rating or because they are rating a prof for a friend. So, it’s not a “one student-one vote” process, and the ratings are skewed. If it’s not something that they take seriously, why should we?
p.s. I, too, would like to know the size of the correlation between the two rating approaches.
Betty, professor, at 2:20 pm EDT on April 25, 2008
1. Reliability is not validity.
2. I was at a college that used the IDEA Form. I thought that it was a bad combination of fake “rigor” and touchy-feely Californism. I don’t particularly care what correlates with it in any case.
TBD, at 3:55 pm EDT on April 25, 2008
I would like to add to Prof. Ed’s comment about pick-a-provost.com. I fully agree that we all should be evaluated at the university under the same blogging format. But I do not want to stop there, how about “rate your students.com” this way as professors we can check out potential students and evaluations they have received from former professors.
Judith, Professor, at 4:55 pm EDT on April 25, 2008
Are you kidding?
Is there no value in polling students about their experiences in your classroom? Are you students unfair judges of how much they feel they have gained? I am appalled at the arrogance shown by the replies to this story. The sooner you hop off the high horse and start paying attention, the better a teacher you will be.
Kirk, at 4:55 pm EDT on April 25, 2008
Kirk mistakes cynicism for arrogance. Every experienced professor I know realizes that teaching evaluations mostly evaluate charisma. I say this as one who has often been the top ranked professor in his dept by in-class evaluations, and last he checked, third at his institution by ratemyprofessor.com. My secret? I’m tall, with classic WASPy looks. I wear tweed in the winter and linen in the summer. I tell engaging yarns and I listen sympathetically before saying “no". I match expectations. I think I do a pretty good job teaching too, but no better (likely worse) than a colleague who’s gruff, occasionally loses his temper, and never matches my evals.
To answer Kirk’s questions: “Is there no value in polling students about their experiences in your classroom?” There is indeed value in polling students about their experiences, and I have often used their suggestions, but polls are very different from _evaluations_. Why? see my next answer.
“Are you [sic] students unfair judges of how much they feel they have gained?"Yes, absolutely, by definition even. How can a student possibly know what he doesn’t know, and the extent to which it was their individual effort and ability (or lack thereof) that led to the outcome? These are things we only see with distance and perspective. How many times has a student come back to you 5 years later to say, “Thank you, I get it now.” It happened to me again on Monday. He probably didn’t say that on his course evaluation in 2003.
Ric, at 8:05 pm EDT on April 25, 2008
I’ll start caring what professors think of the tools used to evaluate them when professors start remembering that their students are the ones paying them, not the ones there to serve them and their high-minded notions of themselves.
Dave, at 5:00 am EDT on April 26, 2008
Ratings and life
Too bad we can’t rate those all who are paid to service and have those ratings used to deny raises and fire surly employees at the grocery store, the cable tv companies, etc. where our students will end up working. (This is at our institution the purpose of the teacher ratings, and students must be told so before we leave the room, though not in so many words. No wonder we have grade inflation.)LM
LM, at 10:20 am EDT on April 26, 2008
Ah ... “Assessment & Evaluation in Higher Education”
Let’s see ... the meaning of teaching excellence? ... the definition of teaching excellence? ... the reliability and validity of measurement instruments (for god knows what purposes)? ... the absurdity of the questions asked? ... the sloppy research methods used by those who seem to care about collecting the data (including randomization, control, and missing data)? ... the general simple-mindedness of those who use the data for decision-making? ... the interminable, sanctimonious harping of the assessment crowd? ... IDEA? ... RateMyProfessor? ... the evaluation forms we use at dear old New World State U, written by a committee of individuals who didn’t know beans about pedagogy or research methods and haven’t thought about instructional objectives during the past two decades?
So you’re telling me that we should be waiting with bated breath for a “new scholarly study” that is about to appear in “Assessment & Evaluation in Higher Education” ... and based on an “analysis” of the rankings of 126 professors at Lander University.
Omigod, please cut me some slack!!!
Not that I want to be caught endorsing the following article by Otto, Sanford, and Ross (October 2007), but I’d be willing to bet – sight unseen – that it tells us a Hell of a lot more about evaluation, assessment, and RateMyProfessor than the yet-to-be released article mentioned above (and be sure to spend some time pondering Figure 1: “Relationships between learning and clarity, helpfulness, and easiness”).
http://www.informaworld.com/smpp/...nt=a782926822&fulltext=713240928
In truth, I don’t think any of us respondents – including yours truly – have improved upon LogicGuru’s concise and insightful remark (see above).
Frizbane Manley, at 2:00 pm EDT on April 27, 2008
Opinions
Why are they known as “opinions’? Because one definition of ‘opinion’ is ‘a vague idea in which some confidence is placed.’ Opinions surveys—even when nationally normed against other schools—are just as meaningless when not normed. They do not measure learning. They measure customer satisfaction. The difference between the two is virtually nil today. My students can score ON THE AVERAGE ten percent ABOVE the national average on a standardized examination from the American Chemical Society after a two semester chemistry course sequence, while my opinion surveys can be anywhere from moderate to abysmal. Indeed, it is about time higher education began validation of what is valid and stopped selling its ‘customers’ down the creek. HLC assessment is not the answer. Ethics are.
David A. Boyles, Professor of Chemistry, at 12:15 am EDT on April 28, 2008
Opinions
Those who think a student opinion survey is necessary to discover what the students are experiencing are likely out-of-touch with the students in the first place. The function of education is to pull people up from where they happen to be. As Laura Bush herself knows “We believe that children develop a bigger vocabulary because people use bigger words with them.” Being pulled down to the level of common opinion does not serve education and holds even the most conscientious professoriate nothing but hostage.
David, Professor of Chemistry, at 12:15 am EDT on April 28, 2008
evaluations
Once a week I go to the site and rate myself very highly. Why don’t we all?
t, at 2:00 pm EDT on April 28, 2008
Ratemy student.com?
Fine. So a report demonstrates that as an assessment tool of professors, RateMy Professor overlaps with our “industry tools.” Please try to tell that to my wife when she comes home from work in tears because her department esteems the nasty comments of knuckleheaded freshman students in her writing class. My wife has sterling teacher observation reports from inside her department and is an experienced instructor of 12 years. Do you mean to tell me that any discontented student — oh and I love how you underplayed “student work habits” as an indicator — can have a serious impact on my wife’s chances for tenure or full time lecturer status? The corporate model of satisfied customers in the university community can only go so far. While I am far from subscribing to the classist “gate-keeper” mentality, I know from my experience teaching at the university and secondary level that most students do not want to do tasks that require effort. And so they are rewarded with an opportunity to savage a teaching professional on a whim? Ratemy Professor is no better than Facebook and is worthy of no more validity in the minds of university professionals.And people wonder why teachers at all levels of the educational system flee like rats from the sinking ship each year...
Sincerely, Robert B. Vellani, Ph.D.New York, New York
Robert Vellani, Classroom instructor at Preston HIgh School, Bronx NY, at 11:45 am EDT on April 30, 2008
I rate quite a few profs each week, mine too.
This Web site is such a crock of turds. There is not control. I go in at least each week and add reviews for people I like to make them seem stellar. The ratings are such BS but because some idiot might think they matter I’ll keep adding positive reviews. I suppose you ca write bad ones for enemies but who has time for the negative energy?Will.
Will B. Dunn, Curmudgeon at Hard Knocks, at 5:15 pm EDT on May 5, 2008
Five percent of students responded in my case — Valid?
Spread over five years and ten semesters,with a course with 250 to 300 students, or close to 600 a year, there were 51 responses. That is 51 responses out of approximately close to 3000 students taking the course.
This seems to correspond to the number of students who either failed the course or gota D.
There seems to be some self-selection favoring disgruntled students.
Statistically, the results seem not relevant. And, the ratings do not correspond to the anonymous ratings by the Department’s StudentEvalations.
Howard E. Zimmerman, Professor, at 8:15 pm EDT on August 17, 2008
Advertisement
Jobs Related to Validation for RateMyProfessors.com?
or search for jobs directly.
Systemwide Director of Training
Alliant International University
Alliant International University is looking for an Assoc. Provost for Pedagogy/Quality Assurance who will develop & maintain ... see job
DIRECTOR OF EVALUATION AND ASSESSMENT, Office of Educational Affairs
Tufts University
The Office of Educational Affairs (OEA) is responsible for the overall medical education program, including curricular ... see job
Academic Advisor
NC State University
Join the Pack! A community with nearly 8,000 faculty and staff, and 30,000 students. NC State is one of the largest employers ... see job
Director of Student Learning Assessment
Ithaca College
Job Description: Ithaca College invites applications for the position of Director of Student Learning ... see job
Academic Services Coordinator
Harvard University
The Academic Services Coordinator provides comprehensive, professional academic services in the Department of Epidemiology to ... see job
Transfer Advisor
University of Minnesota, Twin Cities
The University of Minnesota is a premier employer and a talent magnet attracting leading faculty and staff from around the ... see job
E-Learning Platform Administrative Analyst
DeVry Inc.
DeVry Inc. (NYSE: DV) is the parent organization of DeVry University, Advanced Academics, Ross University, Chamberlain ... see job
Senior Director of Student and Academic Support Services
Sinclair Community College
Sinclair is a comprehensive community college with an enrollment of over 24,000 students that offers career and transfer ... see job
Associate Dean for Faculty
University of Minnesota, Twin Cities
The University of Minnesota is a premier employer and a talent magnet attracting leading faculty and staff from around the ... see job
Assistant Director, Introduction to the Humanities (IHUM)
Stanford University
Stanford University’s Office of the Vice Provost for Undergraduate Education (VPUE) announces an opening for the position of ... see job
Home
| Sign up | Log in
| About us
| Work for Us
| Press Releases
| Rights and permissions
| Contact us
| Advertise
| Partners
| RSS 
| Podcasts
| Links
| Privacy
| My Account
© Copyright 2008 Inside Higher Ed
Students fill out an anonymous evaluation form in one place and fill out an anonymous evaluation form in another place and the results correlate. And you’re surprised? Am I missing something?
Judith, at 7:30 am EDT on April 25, 2008