Evaluating Community College Coaches

Coaches at Division I programs in the National Collegiate Athletic Association are judged not only by their players' performance on the field but also by the players' performance in the classroom. A win-loss record is an easy enough gauge of the former. To judge a coach’s effectiveness in encouraging the latter, presidents and athletic directors turn to the Academic Progress Rate, an NCAA-developed score that shows how a team's athletes are faring with regard to academic eligibility to play and progress toward graduation. Though it is rarely the sole reason for a coach's dismissal, a poor APR is becoming a factor in many institutions' hiring and firing decisions.

Community college administrators have no such formal metric to judge a coach's effectiveness on the academic side. In fact, because governing bodies like the National Junior College Athletic Association do not require their member colleges to track detailed data on their athletes, as the NCAA does for its members, very little is known about how community college athletes perform academically. Convinced that it can do more to track the academic progress of its athletes, Seminole State College, a two-year institution in Oklahoma, is testing a data-driven “accountability index” to gauge the job performance of its athletics coaches — one that objectively grades them on a scale from 1 to 10.

Thomas Mills, director of athletics and head men’s basketball coach at Seminole State, said he developed his “accountability index” about three years ago, when he was getting ready to evaluate his coaches at the end of the academic year. Since he had only just then been promoted to director of athletics, Mills said he “wanted to have an objective way of starting the [evaluation] process so that everything didn’t come out to be [his] subjective opinion.” He added that he was influenced at least in part by the NCAA’s development of the APR and its other relatively recent academic reform efforts. Still, he bemoaned that there was no such help or movement at the community college level.

“As a coach, no one ever talked to me about graduation rates,” Mills said. “People talked to me about developing good solid students and citizens, and I knew getting them to graduate was thought of as a good thing. But there was never a strict connection between my leadership of a program and how many guys graduate. I thought, as an athletics director, that that ought to be part of the equation.”

At the end of the academic year, Seminole State coaches are graded on Mills’s 10-point “accountability index,” which is determined using an equation that rates performance in five areas. Up to 4 points are awarded proportionally for a team’s regular season winning percentage (e.g., a coach whose team had a .500 record would earn two of the four possible points). Up to 2 points are awarded for a team’s postseason winning percentage and up to 2 more are awarded proportionally for its graduation rate for athletes who have exhausted their two years of eligibility (for example, if three of four players on a team who have exhausted their eligibility graduate, then a coach would earn 1.5 of the possible 2 points). Finally one point is awarded for winning a conference championship and another is awarded for making an appearance in a national tournament.

Only 2 of the 10 points on Mills's scale relate to academics, and focusing only on those athletes who complete eligibility is a rather favorable way of calculating graduation rates, in that it excludes players who stop participating in sports or drop out of the institution altogether. Though Mills said he believes his methods are sufficient for his purposes at Seminole State -- given that no one had considered graduation rates in evaluating a coach's performance before him — he noted that he is considering putting more emphasis on academics in the future.

Mills also awards coaches 0.25 “bonus points” for each athletic and academic All-American on their team and 0.50 “bonus points” for teams that have an overall graduation rate greater than 50 percent for entering scholarship athletes. (He provided Inside Higher Ed with a sample spreadsheet detailing how all of these different variables help determine an overall score for a coach on a scale of 1 to 10. The numbers used in the spreadsheet are real and document the recent performance of Seminole State’s women’s volleyball team, but the names have been changed to protect the athletes’ privacy.)

A coach’s score on the “accountability index” — along with his or her three-year average and career average score — is outlined on a detailed annual evaluation. In addition to providing this objective look at a coach’s performance, the evaluation has a section in which the athletics director can make subjective “performance observations.” For example, on a scale from 1 to 7, the athletics director determines how descriptive certain statements are of a coach’s performance — such as “monitors and vigorously promotes the academic success and graduation of student athletes” and “manages budget and resources of the program effectively.”

James Utterback, president of Seminole State, said he appreciates Mills's method of evaluating his head coaches. He added that it gives him more insight into their performance than he has ever had, which he acknowledged is helpful when determining whether to retain or dismiss a coach.

“I believe that there’s so much more to college athletics than winning and losing,” Utterback said. “So often how our sports are judged is simply based on whether or not a student can put a ball through a hoop or whether or not a team wins. But then there are the things that matter in life — like the retention rate of students and the graduation rate of students — things that are never looked at. That’s what I appreciate about what Dr. Mills has developed. It looks at all of the things that are important.”

Utterback also said he appreciated the “flexibility” of Mills’s evaluation system. For example, Mills said he believed other community colleges could use his method but alter the weight they give to variables like regular season winning percentage and graduation rate of players as they see fit. Mills said he likes there to be a balance between athletic and academic success when judging a coach’s performance.

“Athletics is about competition,” Mills said. “If you’re in a situation where you have the resources to be successful, then you should be successful. You should win championships and graduate kids. You do it in a high-character way. Both are important. There are people in athletics who are just about winning and there are people in athletics who are just about graduating kids. The purest form of athletics is for the people that are striving for both.”

Mills's coaches and others who have heard about the coaching evaluation system have mixed reactions. Some administrators who heard Mills and Utterback present on the system at the annual meeting of the American Association of Community Colleges, which took place last month in New Orleans, complained that the system would not be a fair judge of their part-time coaches, many of whom are unpaid and do not have other connections to the institution. (Seminole State is a rarity among community colleges in that many of its coaches are also full-time instructors or staff members.) Even Mills himself said that one of his coaches, upon hearing of the new “accountability index,” asked him, “What score do I have to get so that I’m not fired?”

“There’s not a magical score that will get them fired or have them keep their job,” Utterback said. “But, if their overall score was going down for a three-to-five-year period, then that might tell us something…. We want head coaches that desire what we desire, and that includes making sure students go to class and graduate.”

Mills admitted he is still “working out the kinks” in his system but said, ideally, he would like for his coaches to maintain an average score above 5 and “consistently in the 6 or 7 range.” This is the third year the accountability index is being used at Seminole State, and Mills said he hopes to look at the three-year averages of his coaches and tweak it for future use after this academic year is finished.

Mills said he has not yet calculated longitudinal “accountability index” scores for his coaches or graduation rates for his athletes. Still, he hopes that implementing this system improves student outcomes at Seminole State.

Mills and Utterback both said they would be in favor of governing bodies like the NJCAA requiring that community colleges track the academic progress of their athletes via their coaches. Though Seminole State’s method is attracting some attention from some national community college athletics officials, it is unclear whether such an accountability system would ever be mandated.

“I think Dr. Mills has come up with a very logical and effective approach to evaluating his athletic program,” Mary Ellen Leicht, executive director of the NJCAA, who was on hand for Mills’s presentation at AACC, wrote in an e-mail to Inside Higher Ed. “It takes into account not only the success of the coach/program at the competitive level but also factors in the academic achievements of the student-athlete.”

Still, Leicht added that every program has “its own definition of success.” And though she wrote that Mills's evaluation system “would be welcomed at other two-year institutions,” she noted that it would not be “fair to assume his model would fit perfectly at every two-year college.”

NJCAA membership has repeatedly voted down attempts for academic reporting requirements of its institutions and strengthening its eligibility requirements — suggestions that outside critics, including those at the four-year level who are wary of accepting community college transfer athletes, have long advocated. Leicht, however, noted that “the NJCAA is continually looking at the association’s academic and athletic requirement so as to provide the best possible access to higher education for our student-athletes.” But she also expressed some anxiety about top-down mandates from the NJCAA.

“At a two-year college … not every student-athlete who walks in the door is looking to graduate,” Leicht wrote. “Some are in certificate programs while others may be returning to school for retraining. Because the NJCAA does not have an age limit currently, we see many student-athletes who are not what would be considered traditional age. I think we need to be careful about labeling what is considered ‘successful’ at the two-year college level but do believe programs will be asked to justify their level of success (however they choose to define success) based upon objective data in the future.”