What Really Makes a ‘High-Impact' Practice High Impact?

The phrase “high-impact practice,” or HIP, found its way into the higher education lexicon more than a decade ago. The words signal the unusually positive benefits that accrue to students who participate in such an educational practice, including enhanced engagement in a variety of educationally purposeful tasks; gains in deep, integrative learning; salutary effects for students from historically underserved populations (that is, students get a boost in their performance); and higher persistence and graduation rates.

Most of the individual activities that appear on the HIPs list promulgated by the Association of American College and Universities are familiar to faculty and staff members, as almost all of the HIPs have been available on most college campuses in one form or another for decades.

Some -- such as study abroad -- are considered transformative and life changing, according to testimonials by those fortunate enough to have done them. Multiple studies of the effects of service learning over the past quarter century yielded empirical evidence of the positive effects on desired outcomes of these courses designed for students to have meaningful community service experiences that are integrated with instruction and induce them to apply what they are learning and to reflect on what they have learned and their performance in messy, unscripted situations.

And other HIPs, such as learning communities and internships, long have had enthusiastic champions. Indeed, there was enough evidence in 1999 to persuade the design team that created the first iteration of the National Survey of Student Engagement to include many of the 11 high-impact practices on its questionnaire, asking students in their first and last years of college, “Did you participate in this?”

In 2006, a systematic analysis of several years of NSSE data showed that students who reported doing one or more of these practices benefited in various desired ways. In fact, the differences between those who did an HIP and those who did not were so large that we reanalyzed the data to be sure the results were accurate. The same pattern of results advantaging students participating in HIPs over their peers emerged in subsequent analyses.

A few years later, Ashley Finley and Tia McNair affirmed that historically underserved students benefited significantly from engaging in HIPs, and that participating in multiple HIPs had cumulative, accentuating effects.

And then California State University Northridge reported that its Latino students were about 10 percent more likely to earn a baccalaureate degree in six years than their counterparts were if they did just one HIP. The cumulative effects were also evident, for Latino and other students.

These promising reports and numerous others from individual campuses along with a growing body of literature on service learning, college writing and undergraduate research, and additional research by AAC&U, NSSE and others, propelled HIPs into something of a national juggernaut.

HIPs’ work is featured at regional and national meetings of various associations. And the National Association of System Heads partnered with California State University Dominguez Hills to sponsor the first national convening of the HIPs in the States initiative.

So, imagine the surprise and perhaps dismay of enthusiasts of high-impact practices who saw the recent lead story in Inside Higher Ed “Maybe Not So ‘High Impact’?”

What? HIPs don’t matter to graduation rates at public universities? Apparently so, according to the results of a study published in the well-respected Journal of Higher Education.

As with many research studies, one could quibble with the quality of the data or the analytical methods used, and some of these challenges apply to this paper.

What is worth pondering is the study’s animating purpose: Is the mere availability of HIPs at public universities related to institutional graduation rates? There are many reasons why expecting positive findings from such an inquiry are unrealistic; central among them are that a student’s precollege academic preparation and family socioeconomic status account for the largest share of explained variance when predicting completion.

A more compelling and actionable approach to determining the value of participating in an HIP is whether the experience is linked to desired outcomes/performance/behavior (including persistence and graduation) of students who have actually done one or more HIPs compared with that of their peers who have not had such experiences.

The study featured in the article relied on aggregated institution-level data that do not match individual students, HIP participation and whether they graduated. The study used two HIP measures, one related to the extent of specific HIP offerings and one summing the measure across HIPs. This approach overlooks implementation fidelity and masks the accentuating effects of multiple HIPs on individual student outcomes.

These limitations could not be overcome by the researchers who used a small arsenal of standard statistical approaches to analyze the information accessible to them.

Indeed, the Inside Higher Ed article brings to the fore a most important but often overlooked consideration: simply offering and labeling an activity an HIP does not necessarily guarantee that students who participate in it will benefit in the ways much of the extant literature claims.

Over the past few years, we’ve emphasized that implementation quality is critical in terms of realizing the benefits of HIP participation. This is not a surprise as the caveat applies to every effort a college or university makes to engage students in meaningful, relevant learning experiences inside and outside the classroom, on and off the campus.

Campus practitioners know firsthand that some service-learning courses and internships are better designed and implemented than others. This holds for every type of HIP and just about any other college experience that matters to student learning and personal development.

For example, soon-to-be-published NSSE data about the effects of learning communities on engagement and self-reported gains show great variation between institutions. So institutional context and implementation quality matter.

Scaling HIPs effectively through curricular or graduation requirements is one way to induce widespread participation. Indiana University-Purdue University at Indianapolis uses this approach in its RISE initiative to broaden access to a quality educational experience by expecting students to participate in research, international experiences, service learning and experiential learning.

To this end, IUPUI faculty and administrators have thoughtfully crafted experiences, supported faculty development and studied the effects of RISE experiences. Requiring student participation in one or more HIPs should be an intentional, evidence-based decision and tailored to the institutional context and its students. Simply increasing the number of available HIPs is not an effective approach to scaling.

There is much more to learn about HIPs and other college experiences that could or should have similar positive effects. Especially welcome are efforts to confirm the conditions that are associated with the depth and range of desired effects.

We’ve described many of these features elsewhere. But it is possible that some of these features, such as peer interaction, are more or less important to a certain HIP, such as an internship or other type of field experience. And while the positive effects of HIPs hold for all students when aggregated at the national level, perhaps certain students will benefit more from particular HIPs compared with others in different campus contexts.

This emphasizes the importance of being equity minded when scaling HIP participation. Which students are experiencing HIPs, and who is left out? Are underrepresented students having high-quality experiences? Access to HIPs without equitable participation is a hollow achievement.

The most recent National Institute of Learning Outcomes Assessment survey of provosts found that hundreds of colleges and universities are working to scale currently existing HIPs and add others so more students can participate in an HIP.

We owe it to our students to ensure HIPs and other innovations intended to enhance the quality of undergraduate education are implemented equitably and with fidelity so that students realize the promised benefits.