Assessing the ACT Changes

Starting next spring, the ACT will see some major changes. It will be shorter, and the science section is becoming optional. This change is important for students who want to use the ACT as an admissions credential, and it also confronts us with some fundamental assessment questions, such as:

• What is worth testing?
• What role should marketing considerations play when designing tests?

First, let’s look at what’s different and why it matters.

The Changes and the Impact

The new test will have 44 fewer total questions and shorter English and reading passages. Even though the test will be significantly shorter, examinees will have more time per question because they’ve cut so many questions. Early indications suggest that the test questions are not going to get harder. As a result, the ACT will become substantially easier.

Traditionally, one of the biggest challenges of the ACT has been time pressure. You currently need to answer 75 (!) English questions in only 45 minutes, 60 math questions in 60 minutes, 40 reading questions in 35 minutes and 40 science questions in 35 minutes. (Reading and science are especially time-pressured on today’s ACT.)

The overwhelming majority of ACT questions just aren’t that hard, but you really need to move or you’ll run out of time. After the change, much of this time pressure will be gone. You’ll still need speed, but it won’t be as challenging to finish as it is now. So if nothing else changes, it will be easier to get a high percentage of questions correct. But that advantage could be an illusion. Scaled scores are supposed to account for differences in test difficulty, so if the test is easier now, you might need more correct answers to get to the same scaled score. Alternatively, if it’s easier to get to the same scaled score, the percentiles associated with those scaled scores will drop because more students will earn top scores.

That being said, the change will help some and hurt others. Students who really struggle with time will do better on the new ACT. Students who can handle the time pressure now will be hurt. While the speedy examinees will do no worse on a less time-pressured test, their slower competitors will do better. It’s bad for you when your competitors lose a disadvantage, even if you stay the same.

Making tests easier is like inflating grades. If a school gets more generous when giving out A’s, students who get higher grades as a result of that inflation have an advantage, but it hurts people who would have gotten A’s anyway and now look like everyone else. And if everyone gets A’s, then good grades don’t really help you, but bad grades will really hurt you. Similarly, an easier test makes it harder for applicants to stand out in a competitive field. This puts more pressure on students to earn top scores while making those scores less helpful for the students who earn them.

Making the science section optional will function similarly. Some students will be able to hide their weaknesses, which is good for them, but those who can handle the science reasoning section as it stands now will have the impact of their skill diluted.

Chances are, the impact of these changes will be minor. But the argument holds: Making the test easier creates losers as well as winners and doesn’t necessarily make the test better.

Should Speed Matter?

Historically, typical ACT questions have been easier than typical SAT questions, but the ACT has been much more time-pressured. This was always true, but the differences have been magnified since the SAT went digital and adaptive. The hardest SAT questions are harder, but you have more time to answer them. That poses a fundamental question: Should speed be a factor in either test? Why not give everyone unlimited time, especially since access to extended time accommodations is radically unequal? I’ve known plenty of students who had extra time and didn’t need it, and there are untold thousands of students who are at least as deserving of extra time and don’t have the resources to get it. So why not take timing out of the process altogether?

There are reasonable arguments on both sides here. When we think of the smartest people we know, the ones we really want to have on our side when the stakes are high, they tend to be quick thinkers and not plodders. But maybe that’s just our biases talking. Maybe we favor the quick because we’re irrationally favoring them, and the quick keep on getting rewarded because the systems they enter are designed in their favor. Maybe speed is like height: Taller people aren’t really smarter, but they make more money and get more promotions because of unfair favoritism that follows them everywhere they go. Or maybe the best resolution of the speed issue is somewhere in the middle. Maybe speed should be important, but not as important as it is on the current ACT, where you really need to sprint if you want to get to everything.

If I had to guess, I’d say that de-emphasizing speed on the ACT is a good thing. I’d just like to see it become more challenging in other ways to help skilled applicants prove themselves. Although I’m no fanboy of the SAT, the recent changes there are encouraging. The digital SAT has more questions that require:

• A sophisticated vocabulary and the ability to engage with challenging texts.
• The ability to parse complex language, texts written long ago and even poems.
• Critical thinking skills, such as evaluating experimental design, analyzing arguments and drawing sophisticated conclusions.

I teach both SAT and ACT students, and it is clear that the skills required to succeed on each test are very different. Successful students typically can succeed on either one, but students who can’t break down arguments and don’t have sophisticated vocabularies will crash and burn on the SAT but might do just fine on the ACT, which generally requires lower-level reading and reasoning skills.

A better test would measure more of these skills, and maybe that’s on the way. The decision to make the science section optional (along with writing, which has been optional since 2005) allows students to craft their test to fit their skills and interests. That could be a good thing. While there are some skills that everyone should have, students should also be able to show special expertise in particular areas. Everyone needs some reading, writing and math skills, but maybe there should be additional opportunities for students to show skill in advanced math skills, critical thinking, essay writing, historical knowledge and more. A better testing system could reward a core group of skills while giving specialists a chance to shine.

Wouldn’t a customized system be better than the current system, which tests future historians and future engineers on the same math skills? Future engineers should be able to show off their advanced math skills, and future historians should be able to show their knowledge and writing skills. To some extent, the current Advanced Placement system allows some of this customization, and the now-defunct SAT subject tests functioned similarly, but there’s more we could do.

Is This All About Marketing?

We’d like to believe that the ACT changes are all about improving education, but there is a strong alternative explanation. The ACT has been remarkably stable for years, so why is it changing now to become shorter and easier? Maybe it’s competitive pressure. When the SAT and the ACT were both three-hour tests, people dealt with it. Now, the ACT is still three hours, but the SAT is only a little more than two hours, and my students who try both really prefer the shorter time. So the changes in the ACT might be best understood as a way to appeal to the kids today instead of a way to make the test better.

These choices have consequences. A shorter test is more convenient and less draining, but eliminating questions tends to make tests less accurate, especially when tests are adaptive. In theory, an adaptive test like the SAT is supposed to be super-consistent and accurate, but the early experiences of my students suggest that it is more volatile than the either the previous SAT or the ACT. Students are getting scores they don’t deserve, in either direction, too high or too low. There aren’t enough questions to sort the pretty good from the amazing, and students who make a few too many mistakes early can’t recover, which is not the way a computer adaptive test is supposed to work. From what I’ve seen, if you make a few too many sloppy mistakes early, you get sent to the easier section, which in my experience means you won’t be able to score better than the high 500s even if you get the easier second section completely correct. One of my students had practice scores that went from 530 in math to 720 in consecutive weeks not because he got any better at math, but because he made fewer silly mistakes in the first section and got to the harder questions that make a top score possible. Jumps like this should be very rare on a good test, but I’m seeing them often.

People like a shorter test, but there are trade-offs, and giving customers what they want isn’t always best for them—or for higher education.