Nov. 3, 2006

Does ‘Value Added’ Add Value?

Listen to some critics of higher education, and you’ll hear constant calls for more accountability and assessment. Colleges don’t do enough to figure out whether students actually learn anything, the line goes, whether there is any “value added.” Several hundred academic administrators from around the country heard a version of this from Education Secretary Margaret Spellings, when she told their meeting: “In higher education, we’ve invested tens of billions of taxpayer dollars over the years and basically just hoped for the best.”

Looking for a job?

See all 231 new postings

Browse all job listings:

Faculty: 3,298
Administrative: 1,239
Executive: 160
FEATURED EMPLOYERS

The Spellings text was largely a rehash of statements she and others have made about why accountability and assessment matter. Earlier in the day, however, attendees at the National Symposium on Postsecondary Student Success heard a very different presentation about assessment. Three administrators of public universities — none of them flagships — spoke about how assessment actually works on their campuses. While their experiences vary, they agreed that in key respects, the policy question of “do colleges need assessment?” misses the point. Their message: There’s much more assessment going on than policy makers seem to realize, making assessment really matter is much more difficult than talking about why we should have assessment, and quantitative measures that purport to show “value added” may or may not do so.

“All these national calls would make you think nothing is happening,” but that’s not true, said Jon Young, senior associate vice chancellor for academic affairs at Fayetteville State University.

To illustrate the point that plenty of assessment already takes place, Howard Cohen, chancellor of Purdue University Calumet, described the major assessment efforts on his campus. The 9,300 students at Cohen’s campus are in many pre-professional programs — and those programs tend to have specialized accreditors, all of which have revamped standards in recent years to require evidence of student learning. At his institution, that means major studies on student learning in all of the top majors: engineering, management, nursing, education, technology and therapy.

Then there is Purdue’s regional accreditor, the North Central Association of Colleges and Schools. North Central offers members the chance for an alternative accreditation review through its Academic Quality Improvement Program, which replaces periodic reviews with more regular sessions, all focused on showing improvements in outcomes. Purdue Calumet participates in that. Plus the university maintains 20 advisory boards of business and community leaders who employ graduates in various fields. Each advisory board is regularly surveyed about the how the university’s alumni perform. All in all, Cohen stressed, there is a whole lot of assessment going on.

At Fayetteville State, a historically black university in North Carolina, “assessment and value added have been part of our culture for a long time,” said Young. He said that’s frequently the case at black colleges, which take pride in admitting students who may not have received great high school educations, but who are able to develop and thrive. The value of black colleges is evident, Young said, not by things like average test scores coming in, but by the growth that takes place during students’ time in college.

Among the assessment tools used at Fayetteville State — most of them for some time, Young said — are the National Survey of Student Engagement, institutional surveys of students and alumni, the Noel-Levitz Student Satisfaction Inventory, and a “rising junior exam” that students take at the end of their sophomore years to show that they have reached certain competency levels.

Young said that policy makers calling for the collection of more data need to slow down and realize how much data is already being collected. “We’re good at collecting data,” Young said. “But we can also collect data, nod our heads, and just move back to our business” as it was before.

That raised the issue that many said was far more crucial than whether one test or another was required: how to get buy-in, from students and faculty members, to take assessment seriously. The administrators argued for a mix of carrots and promises not to use certain sticks as the best way to do this.

John D. Welty, president of California State University at Fresno, said that his institution has been engaged in rigorous assessment since 1998, with the programs evolving along the way. At the beginning, professors were encouraged to participate with an offer from the administration: Five-year program reviews, which were “much hated” by professors, could be skipped if departments adopted systems to measure student learning. Faculty members also needed more assurances, he said. As a result, the administration pledged not to make the results of assessment measures part of the budgeting process, so departments didn’t need to worry that an honest (and potentially harsh) look at a department would just lead to a program being killed.

Looking back, Welty said this was a good move to make — and that departments are generally taking assessment seriously. “We allowed faculty to embrace assessment,” he said, rather than forcing it on them.

Now, Fresno is looking at whether students have the right motivations. For instance, the university wants to expand use of the Collegiate Learning Assessment, but also wants to make sure students take it seriously. So Fresno is looking at paying students to take the test, as some other campuses do, Welty said.

Assuming faculty and students are on board, there’s also the question of whether measuring student learning necessarily answers the questions people want to know about the value of higher education.

Young, for example, said that at Fayetteville State, the top conclusion of most assessment tools is “something we already knew,” namely that many students arrive who are poorly prepared. Other assessment tools show that students are learning, so from a “value added” perspective, the university should be praised. But what he thinks Fayetteville State really should be praised for is the many cases where it not only raises student knowledge levels, but raises them to appropriate levels, not just achieving some value added milestone.

“We cannot substitute value added for basic competencies,” he said.

Purdue’s Cohen said that there were other problems with “value added” as a measure to assess. If you have two sections of an introductory math course, populate them with similar students, and use different teaching techniques, you can probably make conclusions (if the samples are large enough) on which technique is more effective. What Cohen worries about, however, is the idea that you can measure the “value added” in college generally, when so much of the experience of students is beyond the control of colleges.

He noted, for example, that students transfer in and out — are all of their knowledge gains presumed to be “added” at any one institution or at all of them? If a student at your university takes one or more courses online from another institution, how is that impact measured? If — as is the case at Purdue Calumet — 80 percent of students work at jobs at least 20 hours a week, with many of those jobs related to their education, don’t students learn something at those jobs?

Cohen stressed that he wasn’t saying that colleges shouldn’t think about what they do, or what students learn. But unless “the scope is sufficiently narrow,” he said he doubted all of the assessments currently being used would really tell people what value is added in college.

— Scott Jaschik

Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Comments

Kudos for making this point, and making it well. My initial reaction to the whole Spellings/Commission on Future of Higher Ed. call for more assessment was that anyone actually familiar with just about any higher education institution would know that there’s constantly a whole range of assessment going on, even if it doesn’t mirror the No Child Left Behind approach. The implication that administrators and faculty aren’t interested in knowing whether their curricula and programming are showing results seems bizarre and offensive to me.

John Nugent, Connecticut College, at 9:35 am EST on November 3, 2006

Spellings and Assessment

Assessment is important, and accountability is important. If we are looking for that, can we actually look to the same place that administers NCLB? The conception of that law was admirable, but the law itself and its implementation don’t reflect that and many educators will talk about the waste, the inability to meet goals set in the program, and the inflexibility of the program.

Is this what we are looking for in Higher Ed?I don’t think so.

Thomas J Riley, Dean at North Dakota State University, at 9:45 am EST on November 3, 2006

Few notes

First of all, this is a wonderful article. It provides me examples of what colleges is doing in regard to their responsibility. I am sure there are other institutions that are doing the same.

The question asked:"whether measuring student learning necessarily answers the questions people want to know about the value of higher education” is a very good one. Young’s statement “We cannot substitute value added for basic competencies” is particularly thoughtful.

The variation in assessments will always be there. But making the result available to public is a big step toward answering people’s question about the value of higher education. Think about those product reviews you find on BestBuy or CircuitCity’s website. They are diverse and probably have no statistical value. But, I found them quite useful.

For institutions the care so much about the accuracy, it may not be a bad idea to provide official info to deter people from judging institutions based on anonymous reviews.

Duncan, at 10:01 am EST on November 3, 2006

accountability

What a joke! If this administration is really worried about accountability, why did they just termininate Iraq auditor Stuart Bowen, who uncovered considerable fraud and misconduct among Iraqi contractors and military personnel. As for assessment, perhaps a count of body bags will suffice...

JO, at 11:30 am EST on November 3, 2006

Fundamental contradiction

One of the difficulties of developing uniform, publicly accessible, and comparable assessment tools for student learning is that American higher education has never embraced the goal of attaining uniform and comparable education for all. It seems contradictory to laud individuality, freedom of choice, decentralized control, and diversity as the great strengths of our system on the one hand, and to insist on common assessments on the other.

If assessment is limited to those few skills that we’d like all students to attain, such as basic reading skills, isn’t the test so “dumbed down” that it becomes meaningless in most contexts? Such information might help potential students and their families discriminate among schools within a certain limited range of selectivity and emphasis, but not be applicable outside that range. For example, the test might provide some meaningful information to families deciding between Eastern State and Western State, but not between MIT and Cal Tech, or between two programs designed for first-generation college students or nonnative-English speakers, or between art schools and technical schools.

I wonder if those pressing for more accountability don’t wake up during the night and worry that they are undercutting the very values of individuality, freedom of choice, and diversity that are the foundation of our higher education system. How do they reconcile decentralized control and freedom of the marketplace with one-size-fits-all assessment? The assessments described in this article, which are tailored to the goals of various institutions, seem far better suited to our current needs than one that results in easy comparisons across all institutions.

Lee Griffin, at 11:46 am EST on November 3, 2006

Higher Ed assessment

I agree that US higher ed has never embraced the goal of attaining uniform and comparable education. But that does not make any right or wrong. To most American, they probably not concerned about MIT and Cal Tech at all. But they could well be interested in anything went below those elite institutions. The question here isn’t about how greate US higher ed is but about how to best serve the American — not just those brightest but the American in general.

It is clear that most the post here angered about the notion that US Higher Ed haven’t done anything about the accountability. But, this is the America, everyone got chances to present their cases. Let our dear American to judge.

Tailored assessment is good in fitting the goal of a institution but serve less to help American to distiguish claims from a diploma mill to that of a genuine institution — the example is extreme, but you got the idea.

If you have not so bright a kid and you are facing the choices. You will think about the good old consumer market that you know what you are paying for. This is what facing most of the American these days.

I certainly understand that there are millions of factors to address and there probably no single solution to fit all. But how about the professional certificates, aren’t they some kind of multi-institutional assessments? Wouldn’t those results be good information to answer public’s call for accountability?

Duncan, at 2:50 pm EST on November 3, 2006

article doesn’t go far enough

“Assessment” is just another word for “knowledge", specifically, knowledge about the university. Unfortunately, in my experience only one form of knowledge is deemed to be relevant or desired, despite the fact that there are various forms of knowledge in different disciplines. It is usually bad social science, with little philosophical reflection behind it. This knowledge may well capture some disciplines very well, and I have no problem when it does. But it does a very poor job of capturing learning in other disciplines, particularly the humanities and fine arts. Add to this that assessment usually has to be developed and administered by faculty in those disciplines, rarely trained as social scientists, and you have a recipe for poor results and no motivation. I am the program director and assessment coordinator for my program, and each year we have ever increasing requirements, clearly meant to demonstrate “progress” to politicians and bureaucrats who understand nothing about what we do, and do not care in the least to find out. The exercise has little to do with assessment, and everything to do with surveillance. I would, in fact, be completely in favour of real assessment, that is, real knowledge generated about my discipline, and in fact, we do that alongside our assessment obligations. We report the figures dutifully to administrators, and then ignore them, preferring to generate our own knowledge. We have revised our program extensively over the last few years, and been praised consistently for it, but all those revisions came from our own knowledge, not from assessment.

I have been told that we should try to incorporate our own ideas into the process, but the problem is that the process demands badly operationalized social science, administered by untrained statisticians like me. This is not merely a lack of training.

I think all this comes from certain assumptions about faculty being lazy, untrustworthy, and so forth, that circulate among many politicians and bureaucrats. These requirements are meant to keep us in line, to keep us busy, not to improve programs. If they were, they would respect that different disciplines might have different ways of generating knowledge, and make room for that.

Ben J, at 5:45 pm EST on November 3, 2006

Professional assessment

Ben J mentioned the need of professional statisticians to carry out good assessment idea. Personally, I totally agree and I think this give us a good reason for using Professional Services provider like ETS to design and carry out the assessment.

Again, I agree that there are differences between disciplines and we need different ways in measuring them and I don’t think these should prevent us from develop multi-institutional assessments using professional services.

Duncan, at 5:26 am EST on November 4, 2006

Which Value Added?

I like your insights Duncan. How would you deal with the “value added” differential between being admitted to elite schools, and lower-tier ones? This “value added” seems to escaped everyones notice.

Previously (here), John Lombardi pointed out that occupational placement rates (as a measure of value added) are largely dependent upon the economy and employment availability. Okay.

He also makes the point that students going to the lower tier schools, doing remedial work, clearly can make impressive “value added” gains, whereas those in the elite schools don’t. They may even lose ground in certain respects. Yet, the job market could care less — it wants the prestige degree over the less elite degree, and it values it more.

Clearly, here is a concept at war with itself. And the problem is the differing contexts — value added by the market place, and value added that is determined statistically using standardized tests.

It is sad to see well-meaning postsecondary folks leave out the former in favor of the latter. But it is also very poor policy analysis.

Glen McGhee, FHEAP, at 12:50 pm EST on November 4, 2006

Assessment embraced?

I quote from the article above:

Wish this were true at our place, at least. We were thrown a ton of assessment work, School of Education evaluations, language assessment forms and told to “conform to the way it will be from now on or risk the College’s accreditation.” So much for embracing — we were threatened. This stuff took hours to complete, two faculty members worked in the summer on it for ZERO pay (and I hear people gripe about Walmart workers being forced to work off the clock? PUH-LEASE!) and we still got slam dunked. No incentives offered, just the “it’s part of your job.” I was wondering what was wrong with our testing our students the way we have in the past — good old-fashioned oral and written tests and a grade for all of it combined. All of this assessment has been reinventing the wheel and no one seems to be satisfied with results. It just gets better — when a department is finished with all the assessment garbage and then the administrator in charge finds another job or resigns. Then it’s back to Square One with an new in-coming! To put a new twist on an old saying: if the US Administration in DC (and higher ups in higher ed administration) think they knows what’s going on, they are probably full of you-know-what.

Abby, assoc.prof., at 2:20 pm EST on November 4, 2006

What is Assessment?

In all of the assessment activities at my campus, the exams and other grade-related activities conducted by professors to assess learning in their classrooms are excluded from assessment. Why is that?

Nancy, at 12:35 pm EST on November 5, 2006

Glen McGhee

I do agree with you on that the job market could care less of value added but the final vaule. However, with enough information out there, I do expect students and parents take up their responsibility of making their own decision.

Personally, I haven’t been vocal on the Value Added aspect of higher education. I think more about the final value. A little thought is that if a student is admitted to a school, his expectation is the final value of that school. I don’t think I really favor either one of them. I am more interested in seeing information been made available to parents and students. Students and parents will then make their choices. As point out earlier, a comparable measure is of benefit to the general public.

Duncan, at 12:35 pm EST on November 5, 2006

Assessment in institution — Interesting ..

I am pleased to see Abby and Nancy’s post, so is that of Ben J. It seems accreditation assessment create huge workload for institutions. It is also interesting that, ya, why can’t professors’ assessment in classroom be included? Especially, if you are intereseted in tailored assessment.

Strickly speaking, accreditation is trying to set some kind of multi-institutional standard based on resources and others. So. Can they do better job than professional test services ? Professional test services may not be perfect, but they can be improved too, or add other type of evaluation.

The question remain: will public see those result?

Duncan, at 8:45 am EST on November 6, 2006

Value Added

The real culprit here is the detective story on TV, which leads us to believe that ANY problem can be solved in one hour [with time out for commercials]. Evaluation efforts tend to be measured in weeks or months, but the ultimate value of a course or a college education may not show up for 20-30 years — long after the student graduates — and it will show up NOT on some test, but rather on what the long-since-graduated student contributes to our civilization. Perhaps we should give some sort of test to students who graduated about 1975, which might show not only what they learned in college, what they now know, but what they have contributed or are contributing to the total culture, rather than to some successful venture now defunct. Maybe we need more emphasis on faith and less on test results.

Bobby Bruce, Reitred after 44 years, at 12:55 am EST on November 8, 2006

Value Added at K-12

We are beginning the process of focusing not only on student achievement but also student progress. In New York State this can be done with the current 3-8 testing required by NCLB and eventually we include our own local assessments(already being used as summative and formative assessments). So we will be able to bring value added assessment to Oxford without adding more tests.

Randall Squier, Supeintendent at Oxford Schools, at 1:10 pm EST on November 18, 2006

In an Article by Alan Jones with insidehighered back in June, he exposed the Higher Education Commission by Spellings as linked with Horowitz, his funder Michael Joyce, Bill Buckley, Irving Kristol etc...

How does this fit into the picture here? Also, why do people discuss the arguments put forward by the Spellings et al. without adressing the political intent that this whole network represents?

Michael, Michael Joyce?, at 1:20 pm EST on November 29, 2006

Jobs Related to Does 'Value Added' Add Value?

or search for jobs directly.

Part Time Faculty Art Appreciation
University of Texas, Brownsville

[ Position Number: P/T FY 08-4 ] Reports to: Chairperson of Fine Arts Scope: Part-time faculty for Fine Arts Department. ... see job

Post-Doctoral Associate
University of Minnesota, Twin Cities

The University of Minnesota is a premier employer and a talent magnet attracting leading faculty and staff from around the ... see job

Math Instructor
South Texas College

South Texas College Job Announcement # 2010 – 3034 Position pending Board Approval of FY10 Budget Please note that only ... see job

Vice Chair of Research
East Carolina University

East Carolina University, a constituent institution of the University of North Carolina, is a doctoral institution with an ... see job

Biology-Anatomy and Physiology
Concordia University Chicago

Located 10 miles west of Chicago in the suburb of River Forest, is a Christian liberal arts institution. It is a member ... see job

Lecturer or Visiting Assistant Professor
Yale University

Yale University expects to make a one-year visiting appointment for 2009-2010 in the area of Phonetics and Phonology, at the ... see job

Senior Systems Administrator
SUNY IT

The State University of New York Institute of Technology (SUNYIT) is seeking applicants for the following opening in ... see job

Assistant, Associate or Full Professor — Political Science, Political Economy and Int.Relations
Zayed University

Zayed University is seeking to hire qualified Political Science, Political Economy and International Relations faculty to ... see job

English as a Second Language Instructor
City College of San Francisco

SAN FRANCISCO COMMUNITY COLLEGE DISTRICT Position Title: English as a Second Language Instructor Posting Number: 0080041 ... see job

ISA — Organic Dairy Herd Manager
SUNY College of Technology at Alfred

Nestled in a beautiful valley in the western Southern Tier of New York State, Alfred State College has a long tradition of ... see job