Professors Can Learn to Be More Effective Instructors

Intuitively, it makes sense that professors who spend time developing their teaching skills will become more effective instructors -- and that that will eventually translate to better student outcomes. Practically speaking, though, the challenges of (and the variables involved in) tracing the effects of professional development on student learning are myriad. That’s probably why the research on the matter is patchy, relying largely on self-reported measures. But a new book based on data from two very different institutions purports to show that faculty members can learn to become more effective teachers.

“Broadly speaking, faculty development has measurable impacts on teaching,” the book says. “Existing research and the current project confirm that faculty consistently self-report learning gains aligned with workshop goals at the end of these experiences.” Moreover, it continues, faculty members’ accounts demonstrate that they can look back at past development opportunities and describe changes in their teaching aligned with these goals. An analysis of subjects’ syllabi, assignments, methods and grading scales backs up those claims -- as does a review of student work.

Faculty Development and Student Learning: Assessing the Connections (Indiana University Press) was written by William Condon, a professor of English at Washington State University; Ellen R. Iverson, director of evaluation at the Science Education Resource Center at Carleton College; Cathryn A. Manduca, director of Carleton’s Science Education Resource Center; Carol Rutz, director of the College Writing Program at Carleton; and Gudrun Willett, a cultural anthropologist at Ethnoscapes Global. The book is based on the Tracer Project, two parallel, multiyear studies of how faculty development impacted student learning at Carleton, a small liberal arts college in Minnesota, and at Washington State, a large land-grant institution. Despite their differences, the authors found similar outcomes at both institutions -- including that the benefits of faculty development are cumulative.

Different Landscapes, Similar Outcomes

“Participants who amass a more extensive faculty development history … show measurably larger changes in their teaching than faculty whose participation is slight, such as a single department workshop on the same topic," the book says. Faculty development experiences stemming from informal, self-directed improvement efforts matter as well, “probably because faculty engage in their own development as teachers over time, in recurring contexts ranging from rating portfolios, to evaluating colleagues’ performance in class or being evaluated themselves, to participating in departmental efforts to reform curricula for majors.”

While all development participants show changes in their teaching, those who are self-motivated to improve (versus motivated by a free lunch or outside pressures) show larger changes in their teaching, according to the study. And faculty members with a strong sense of self-efficacy who perceive little risk in trying out new teaching styles are more likely to make changes. Institutional teaching cultures that value experimentation and accept associated risk, such as those the authors observed at Carleton and among tenured faculty at Washington State, also support changes in practice.

Both Washington State and Carleton have Writing Across the Curriculum programs that include specific workshops on how to use writing as a pedagogy in multiple disciplines, as well as portfolio assessment of student writing (in the middle of their degree programs) by faculty readers. The researchers targeted portfolio readers as well as faculty members who’d participated in Writing Across the Curriculum and/or separate, institution-specific development programs. At Washington State, that was the faculty development aspect of the Critical Thinking Project, which from 1999-2003 aimed to blend assessment and instruction to promote critical-thinking skills in the general education curriculum. At Carleton College, it was Quantitative Inquiry, Reasoning and Knowledge (QuIRK). The ongoing program was designed to create a “curriculum and practice around the teaching of quantitative reasoning,” using a circular model of pedagogical reform. That is, assessing student writing to inform professional development, which in turn informs curricular reform and assessment to evaluate its effectiveness.

Broad Participation and Development ‘Spread’

The researchers got an immediate sense of how important faculty development was to their colleagues, due to their relative difficulty establishing a control group of professors who hadn’t participated in any faculty development (they acknowledge that some nonparticipants were “hidden” among nonresponders to their initial survey, but that didn’t account for the phenomenon entirely).

Even among the small group of faculty members who hadn’t participated in formal professional development, there was still evidence that they’d benefited from others’ professional development, as by osmosis. For example, 15 of 28 low-participating faculty members listed writing and/or critical-thinking outcomes on their course syllabi.

Interviewees also suggested that institutional culture encourages or inhibits such a “spread” effect. One senior faculty member said, for example, “There’s sort of a general climate at Carleton, especially when you’re untenured, there’s sort of a finding out what pedagogical best practice is -- [it] is quite important and is one of your responsibilities.”

Again, targeting those faculty members who had participated in formal development, the researchers applied a variety of methods, including: faculty interviews on both campuses; examination of course syllabi, assignments and student evaluations of teaching; examination of faculty evaluations of development workshops; portfolio assessment of student work; and rubric assessment of student work in the respective development programs.

At Washington State, researchers used a tool for rating assignments that had been aligned with the Critical Thinking Project. Fifty of the 350 faculty members who had participated in the four-year project between 1999 and 2003 sat for interviews and provided samples of teaching materials from one of each of their courses to determine whether the development continued to bear on their work. Faculty claimed in interviews, and their syllabi and teaching materials confirmed, the book says, “that not only was promoting critical thinking still a goal in their teaching almost a decade later, but their practices had continued to evolve.” These faculty members were using different, more sophisticated assignments over time, and had revised their evaluative tools to reflect newer practices. They’d used what they’d learned about critical thinking to incorporate other learning outcomes, such was Washington State’s Six Goals for the Baccalaureate, and project alumni were likelier than even their other, high-participating peers to attend development events (1.3 more per semester, on average). They were also more likely than not to have presented on teaching within their programs or departments.

Development Means Better Teaching

Fifty additional Washington State faculty members who’d participated in development for Writing Across the Curriculum shared similar views, backed up by their course materials. For example, whatever the focus of the program seminar, faculty members continued to work on that aspect of teaching over the years. These faculty members also described teaching as a data-driven process, informed by students’ performance.

At Carleton, interviews, surveys and various other analyses of faculty assignments indicate that faculty members were applying many of the practices promoted by Writing Across the Curriculum and the quantitative reasoning program. Eighteen of 47 interviewees who’d participated in such development in the past described making related changes to their practices. Dominant themes include encouraging and assigning multiple drafts as part of assignments, scaffolding assignments to build up skills over the term, and helping students use more data in their work.

In one case study included from an ethnographic mini study, a professor used 15 of 21 methods recommended by either development program. The scaffolding she described in interviews, for example, took the form of smaller papers that helped students practice their writing with quick feedback and revisions before a bigger assignment.

Participation in portfolio review also proved a valuable secondary model for faculty development, the book says, since “the design of the professional activity focuses on studying the relationship between assignment design and student learning.” Even raters who hadn’t participated in other formal professional development made self-reported changes to their teaching practices that were backed up by their syllabi and other materials.

But does this all reach students? The researchers say it does, and that proving it -- while difficult -- is not impossible.

Tracing Faculty Development to Student Learning

First, a bit of background. From 1999-2003, Washington State received funding to train 400 instructors -- both on and off the tenure track, as well graduate students -- in redesigning courses to improve students' critical-thinking skills. Participants provided sets of assignments and student work from a course prior to the training, as well as after. Participants also trained as course raters, allowing the Critical Thinking Project to use Washington State’s guide to rating critical thinking to gauge any improvement after the intervention, or between sections in which the guide was used to craft a course and those in which it wasn’t. Data concerning the multisection World Civilizations course show that students scored higher on every dimension of the rubric when instructors were using the guide. Average writing scores were 2.4 in the nonrubric section and 3.1 in the section where the instructor adapted the guide -- a statistically significant difference. An even more drastic difference was observed in student writing from Entomology 401, a senior-level course. Before-and-after results for Entomology 401 are similar.

In a related, longer-term analysis suggesting that development gains are long lasting and cumulative, students from 1999 to 2003 were found to have demonstrated an average improvement of 0.7 points (from 3.1 to 3.4, depending on the section), on a six-point scale measuring what they learned, conceived as part of the development program in sections of course in which the instructor used the critical-thinking methodology, as compared to students in courses taught by instructors who hadn’t participated in development. Samples collected in 2009-2010 from professors who’d participated in the program a decade ago showed similar gains, but the average score was now 4.1.

“Independent ratings of students’ learning outcomes demonstrate that when faculty learn and apply better ways of addressing desirable student learning outcomes, they translate their learning into course materials and assignments that actually do positively influence students’ learning,” the book says. “That result, in the end, constitutes a successful case, and that kind of design produces long-range outcomes.”

The researchers note that non-tenure-track faculty members were much more likely to attend professional development than their tenure-line peers, even though they in many cases were similarly credentialed (with Ph.D.s) and had more experience teaching. That’s not necessarily a good thing, though, as their participation was undercut by a reticence to experiment for fear of negatively impacting their chances at being renewed. The researchers call this “defensive” faculty development.

Carleton student papers were rated at Washington State as part of the study. Results indicated an alignment between students’ learning and faculty development around QuIRK and critical thinking. Student work from three first-year seminars showed higher critical-thinking ratings in three categories that aligned with Writing Across the Curriculum and QuIRK: identifying the problem or issue, supporting position with data or evidence and communication. Those criteria not part of faculty development, such as developing an original argument and integrating other perspectives, were rated lowest.

Seeking to develop even stronger evidence that would compare the goals of faculty development at Carleton with student learning outcomes in critical thinking, the researchers developed a tool that more clearly matched teaching development goals with changes in faculty practice and student learning. The multidimensional rubric aligned with faculty development goals identified individual improvement by rating dimensions of assignment prompts or student writing samples. Raters compared pairs of “before” and “after” prompts or writing samples along these dimensions on a scale to identify which of the pair was “better,” and whether the intervention had worked. The rubric was designed to be more specific than the holistic critical-thinking rubric, and measure things the older tool couldn’t register. The assignment rubric, for example, included such criteria as, “Provides opportunity to develop earlier assignments into a final product,” and “elicits higher-order thinking and writing.” The student writing rubric, meanwhile, included such criteria as, “Demonstrates communication through clear language, effective writing mechanics and strong organization,” and “differentiates correlation from causation.”

An analysis of the assignment prompts revealed that while the “after” ratings weren’t “greatly better,” those from high-participating faculty members’ classes were consistently higher. Raters selected the newer version of the assignment in a more varied sample of participation levels and disciplines 10 out of 12 times.

Ratings on students’ papers also displayed improvement, though less conclusively than the assignment sample, the study says. Of the 28 paper pairs from faculty members with a range of development participation, 57 percent of the newer papers from high-participating faculty members were rated as holistically “better.” In one case study of a high-participating faculty member’s students’ papers, raters agreed that the papers were generally better in argument, conclusions, higher-order thinking and student point of view.

‘Worth the Investment’

The authors conclude that their study, while limited to two institutions with specific teaching improvement goals, confirms that faculty development translates to student learning outcomes. In order to achieve this connection, they say, “development initiatives must design research methods into their processes,” whether individual workshops or long-term initiatives. Any institution can begin collecting samples of student learning and of faculty teaching materials, and from there design ways to collect faculty learning outcomes and student learning outcomes “so that they match up to reveal any effects.”

The key variables in any such study, the book says, are the type of faculty development to be studied, the methods used and the assessment instrument applied to any relevant data. Regarding the first variable, “institutions must promote the learning outcomes that faculty value,” and typically begin with broader outcomes such as writing, higher-order thinking, information literacy and quantitative reasoning.

Most importantly, the authors say, the attempt should be made, despite assumptions that making such links are impossible, or that faculty members are uninterested in improving their teaching. “Mixed methods and a willingness to begin with what one has and build over time yield results that faculty across campus can believe.” Perhaps the best step, the authors suggest, is to establish a baseline for future data -- collecting assignments, syllabi and student work products -- “and move forward, keeping track along the way.” And successful development initiatives aren’t one size fits all, the book notes, as evidenced by Carleton’s more centralized model and Washington State’s more diffuse set of initiatives.

Rutz, Carleton’s director of college writing, said via email that the integrated approaches are “worth the investment.”

“Trustees, administrators and faculty all want to know whether an institution's curriculum delivers on its learning goals,” she added. “We argue that learning goals for faculty -- presented through workshops allied with institutional objectives -- powerfully advance student learning when coupled with assessment.” But that assessment must “speak to alignment between what faculty are taught, how they apply what they learn and how students respond to faculty innovation.”

Condon, at Washington State, highlighted how the study underscores the importance of a culture of teaching and learning -- even at institutions as different as Carleton and his own.

“Every institution has that culture, of course, and our results showed that our two schools had developed to the point that we’d say we have a generative culture of teaching and learning,” he said. In other words, what institutions value, as enacted in various faculty development opportunities, continues to “spread,” even “beyond the set of faculty who were directly involved in the original effort or initiative.”