Recent advances in undergraduate education indicate that across science, technology, engineering, and mathematics (STEM) and food, agriculture, natural resources, and human sciences (FANH) disciplines, active-learning strategies (e.g., inquiry-based learning, course-based research projects) can improve learning gains (Ambrose et al., 2010; Dolan & Collins, 2015; Eddy & Hogan, 2014; Freeman et al., 2014; Haak et al., 2011; Kober, 2015; Prince, 2004; Ruiz-Primo et al., 2011). However, the implementation of these innovative instructional practices remains relatively limited, with a majority of teaching across STEM and FANH disciplines being lecture focused and employing traditional instructional methods (Brownell & Tanner, 2012; Handelsman et al., 2004; Stains et al., 2018; Tagg, 2012). There is much work yet to be done to transform undergraduate education across STEM and FANH fields.

A critical step in implementing best practices in undergraduate STEM education is to provide faculty with the resources and support necessary to integrate these practices into their teaching. Faculty development programs have the potential to address several barriers to implementing best practices in undergraduate education by providing active-learning curriculum design strategies for various types of learning environments, access to teaching resources, collaboration with other faculty, and time to transition their classroom from a predominantly lecture style (Derting et al., 2016; Ebert-May et al., 2015; Manduca et al., 2017).

One method of assessing the success of faculty development programs is to measure change in instructional methods, particularly from lecture-style to active-learning methods, after completing the program. It is important to assess changes in teaching style through unbiased measures, as self-report surveys are often skewed (Ebert-May et al., 2015; Kane et al., 2002; Stains et al., 2018; Tuckman & Harper, 2012; Williams et al., 2015). Assessment tools such as the Classroom Observation Protocol for Undergraduate STEM (COPUS; Ebert-May et al., 2015; Kane et al., 2002; Lund et al., 2015; Smith et al., 2013; Stains et al., 2018) observe changes in teaching style through an unbiased lens and thus are a more reliable measure of a program’s efficacy. This direct observational evidence is critical to characterizing the evolution of instruction over time and attributing such changes to a program.

Changes in student outcomes offer another lens for assessing changes in teaching methods. Systems thinking and decision-making are essential skills for students learning in interdisciplinary environments (Arnold & Wade, 2015; Arvai et al., 2004; Sommers et al., 2019; Velasquez & Hester, 2013). Decision-making, or multi-criteria decision-making (MCDM), is required to address complex issues with many interrelated components in which decisions are made by accounting for many perspectives and possibilities (Velasquez & Hester, 2013). Systems thinking is defined as an individual’s ability to reason about complex, interconnected, and often interdisciplinary issues (Arnold & Wade, 2015; Richmond, 1993). Together, systems thinking and decision-making provide a critical foundation for STEM and FANH literacy, defined here as an enhanced capacity, both at the individual and collective levels, to make effective decisions grounded in STEM-informed analyses of complex, real-world challenges (Arvai et al., 2004; Bybee et al., 2009; Feinstein, 2011; Rudolph, 2014). Fostering STEM and FANH literacy is critical for students’ success in interdisciplinary contexts, as these skills help students not only master content knowledge (i.e., learn science) but also apply scientific knowledge across disciplines and learning contexts (i.e., learn to use science; Bybee et al., 2009). As such, it is important to measure student gains in systems thinking and decision-making, as well as the role of instruction in fostering these skills.

The Food-Energy-Water Nexus (FEW Nexus; Food and Agriculture Organization of the United Nations, 2014) presents an interdisciplinary framework to assess teaching and learning, focusing on coupled and interrelated human-environmental systems. At the University of Nebraska-Lincoln, we have implemented an undergraduate minor designed to offer all students the experiences, knowledge, and skills necessary to analyze complex, real-world problems from a systems perspective so they can make informed decisions regarding current and emerging FEW issues (Sommers et al., 2019). To support faculty in integrating effective undergraduate instruction in the minor, we implemented a 2-year faculty development program for those teaching minor-affiliated courses. The faculty development program focuses on supporting faculty to incorporate best practices into undergraduate STEM education, including student-centered, active-learning instructional strategies such as group work, collaborative learning, and peer instruction; the program also emphasizes science-informed decision-making and systems thinking as student outcomes of teaching (Freeman et al., 2014; Lund et al., 2015). To assess the potential impact of the faculty development program, we asked the following research questions:

1. To what extent do instructors incorporate more active-learning instruction into their courses after completing the faculty development program?
2. To what extent do student outcomes (i.e., decision-making and systems thinking) improve in consecutive years as instructors complete the faculty development program?

Methods

Institutional context and program description

This study was conducted at a large, midwestern university, as part of a grant-funded project to support effective STEM teaching and learning practices in a set of undergraduate courses focused on the FEW Nexus that together form a comprehensive, interdisciplinary minor in food, energy, and water systems. The minor focuses on cultivating students’ systems thinking and decision-making about contemporary, real-world challenges grounded in the FEW Nexus and is intended to complement a variety of student majors. These courses consist of individual classes of differing lengths, either shorter than 50 minutes or longer than 50 minutes. The postsecondary faculty (n = 8) participating in this study are instructors for these minor-affiliated courses. The undergraduate students in this study come from various STEM, FANH, and other majors and educational backgrounds within a single college at the university. One course had a different instructor in the second year of the program, so that instructor did not experience the full benefits of the faculty development program (see Course 8 in Table 1). The project also supported learning assistants (LAs), each of whom completed a 1-semester seminar course offered by Jenny Dauer, to assist participating faculty with instruction in their courses in Year 2.

Instructors of these minor-affiliated courses participated in a 15-month faculty development program spanning two consecutive academic years. The program focused on three key sources of growth for faculty: (a) workshops and teaching resources, (b) a system of faculty peers, and (c) investigating data from their own classroom (instruction observations and data collection). A series of workshops were held between fall 2017 and spring 2019. The goal of these workshops was to foster effective instructional practices in participants’ undergraduate courses, including active learning and the effective use of LAs. Both of these practices have been shown to enhance undergraduate students’ learning (Freeman et al., 2014) and opportunities for them to engage in systems thinking and science-informed decision-making through evidence-based practices such as scientific teaching and backward design, formative assessment strategies, active-learning strategies for large classrooms, peer learning communities, adoption of an LA model, and application of theories about how learning works (Ambrose et al., 2010; Handelsman et al., 2004). Program resources were designed based, in part, on materials available through the ARISE (https://arise.unl.edu/) program at the University of Nebraska-Lincoln. We provided participants with video recordings of their classroom observations and collected student data in each year. During the first workshop, participants discussed their goals and developed strategies for creating student-centered classrooms. In the workshops that followed, participants explored evidence-based best practices in undergraduate STEM education; the nature of and strategies for helping foster students’ system thinking and decision-making skills; and, specifically, how to effectively use LAs in undergraduate classrooms to enhance student learning. Throughout the program, participants collaborated with other faculty and worked to apply these ideas to their own courses and ground their teaching in best practices.

Data collection and analysis

Data collection methods presented here are also discussed in detail in Sommers and colleagues (2019). During the academic years of 2017–2018 (Year 1) and 2018–2019 (Year 2), we conducted observations of courses associated with the minor (Table 1). A member of the project team attended each observed class section and recorded the class so that each observation was archived. We collected eight observations for each course as a representative sample of classroom instruction (excluding exam days, field trips, etc.; n = 64 class sessions per year, 128 total). Video-recorded observations were analyzed using the COPUS instrument to characterize the instructional practices in each course (Lund et al., 2015). This instrument involves the use of eight codes (incorporating teacher and student behaviors) to characterize instruction in 2-minute intervals. After data collection, we jointly coded 12.5% of the video-recorded observations, then discussed discrepancies in our scores until we reached agreement (n = 16 observations, interrater reliability = 92%). Coding was then conducted by individual project team members. After all videos were coded, we categorized each class period into the appropriate COPUS Profile—either Lecture, Socratic, Peer Instruction, or Collaborative Learning. Lecture and Socratic both involve approximately 80% or more of the 2-minute intervals of class time spent lecturing. However, Socratic involves increased student-instructor interactions, such as students asking and answering questions in class, which are not seen in the traditional lecture style. Peer Instruction involves the emergence of group work, clicker questions, and, again, increased student-instructor interaction. Collaborative Learning is characterized by extensive group work and far less time spent lecturing.

To assess students’ decision-making and systems thinking, we collected pre- and postcourse assessments of undergraduate students enrolled in each of the courses during both academic years (n₁ = 218, n₂ = 129; total n = 347). These assessments included both a decision-making task and a systems thinking Likert-scale task. The decision-making task and scoring rubric were modified from a decision-making assessment reported on by Eggert and Bögeholz (2009). This task is designed to elicit decision-making processes with purposefully designed prompts and supports that guide students through a stepwise decision-making process. Students are presented with a real-world, problem-based scenario that demands decisions to resolve a challenge. Systems thinking and decision-making are core concepts that are integrated into all FEW Nexus minor courses. Additionally, faculty received training in the program on supporting systems thinking and decision-making with their course content in an effort to ensure all students within these courses had experience using these skills. We scored the decision-making task based on a rubric modified from Eggert and Bögeholz (2009; interrater reliability = 85% based on 33% of the data). Our systems thinking assessment and rubric were modified from a published Likert-scale-style instrument developed by Davis and Stroink (2016). This systems thinking task measures undergraduate students’ ability to analyze related systems components, including understanding the relationship between humans and FEW systems. All statistical analyses were conducted in program R, version 3.5.2.

Results

Faculty development

To address our first research question (To what extent do instructors incorporate more active-learning instruction into their courses after completing the faculty development program?), we compared instructional styles (Lecture, Socratic, Peer Instruction, or Collaborative Learning) of participating faculty members in Year 1 and Year 2 of the project using an independent samples t-test. Active-learning strategies are those categorized as either Peer Instruction or Collaborative Learning, while Lecture and Socratic instructional styles are categorized as traditional, non-student-centered strategies. Our results indicate that teaching styles in Year 2 of the program (post–faculty development program) included significantly more active learning, as 50% (n = 32 observations) of courses were categorized as Peer Instruction (n = 16 observations) and Collaborative Learning (n = 16 observations), compared to Year 1, when only 37.5% (n = 24 observations) incorporated active-learning instructional styles (Peer Instruction: n = 16 observations; Collaborative Learning: n = 8 observations; t = -4.9, df = 325.1, p = 1.9e^-6, 95% CI = -0.7, -0.3, n = 64 observations; Table 1; Figure 1).

Interestingly, the increased incorporation of active learning was even more noticeable in courses with longer class periods (those that meet for longer than 50 minutes at a time). In both Year 1 and Year 2, instructors who taught courses with longer class periods incorporated significantly more active-learning strategies (Peer Instruction and Collaborative Learning) than instructors who taught courses with shorter class periods (independent samples t-test; Year 1: t = -11.4, df = 175.6, p = 2.2e^-16, 95% CI = -1.6, -1.1, n = 32 observations; Year 2: t = -20.3, df = 62. 37, p = 2.2e^-16, 95% CI = -1.8, -1.4 , n = 32 observations; Figure 1). In Year 1, 50% of courses with longer class periods were categorized as active learning (Collaborative Learning: n = 8 observations, Peer Instruction: n = 8 observations), compared to courses with shorter class periods, in which only 25% incorporated active-learning instructional styles (Peer Instruction: n = 8 observations). In Year 2, 75% of courses with longer class periods were categorized as active learning (Collaborative Learning: n = 16 observations, Peer Instruction: n = 8 observations), compared to courses with shorter class periods, of which only 25% were classified as active learning (Peer Instruction: n = 8 observations). While shorter classes in both Year 1 and Year 2 included less active learning compared to longer classes, Year 2 of shorter classes did not include any Lecture-categorized courses, as they shifted to only Socratic and Peer Instruction. Additionally, we did not observe Collaborative Learning in shorter classes in either Year 1 or Year 2.

Student outcomes

To address our second research question (To what extent do student outcomes [decision-making and systems thinking] improve in consecutive years as instructors complete the faculty development program?), we analyzed change scores for pre- and postcourse decision-making and systems thinking measures from both Year 1 and Year 2. Results of an independent samples t-test show no statistically significant differences between Year 1 and Year 2 change scores for decision-making (t = -0.3, df = 287.4, p = 0.8, 95% CI = -0.5, 0.4, n = 347) or systems thinking (t = 0.3, df = 287.1, p = 0.8, 95% CI = -2.1, 2.8, n = 347; Table 2). These findings suggest that student outcomes in each year were comparable, despite the increased use of active-learning instructional styles between Year 1 and Year 2.

Discussion

It is imperative to advance efforts to implement best teaching practices in undergraduate STEM education, particularly through interdisciplinary contexts such as the FEW Nexus (Ambrose et al., 2010; Dolan & Collins, 2015; Eddy & Hogan, 2014; Food and Agriculture Organization of the United Nations, 2014; Freeman et al., 2014; Haak et al., 2011; Kober, 2015; Prince, 2004; Ruiz-Primo et al., 2011). To do so, reformers and researchers must understand the impact of teaching methods on students’ learning, as well as how faculty development programs can positively impact both (National Research Council, 2012). In this study, we directly measured the impact of teaching methods in undergraduate courses on student outcomes, as well as how both changed over consecutive years in parallel with a faculty development program. First, our results indicate that instructors do implement more active-learning teaching methods over time in parallel with the faculty development program (Figure 1). This suggests that the resources, collaboration with peers, and time allocated by the program positively help faculty with overhauling courses that were previously primarily lecture based to make them active learning focused, as is observed by other programs (Derting et al., 2016; Ebert-May et al., 2015; Manduca et al., 2017). Interestingly, there was more active learning implemented in courses with longer class meeting times than in those that met for shorter durations (Figure 2). It is possible that more time per class period may allow instructors the freedom to include potentially time-consuming, student-centered learning activities, whereas shorter class periods are limited in the possible duration of each activity. Additionally, shorter class periods are often designated as lecture sections, whereas longer class periods are often designated lab or field-based sections, pre-biasing both instructors and students to the types of activities that could or should be included. Perhaps the resources provided during the faculty development program were better suited to courses with longer class periods, and future iterations of the program should be intentional about including examples and resources that are suited for shorter activities and alternatives to traditional short lecture course sections (as opposed to lab sections) and training instructors on how to transition between activities efficiently. Similarly, it is important for time-limited courses to occur in learning environments that are conducive to active learning, including those that have non-fixed seating, space for group activities, and layouts that support peer interaction (Michael, 2007; Miller & Metz, 2014). These considerations can facilitate efficient implementation of active learning and provide overall benefits for students and faculty in the classroom.

While we observed greater implementation of best practices in undergraduate STEM education after faculty completed the program, student outcomes (systems thinking and decision-making) did not increase between the two consecutive years (Table 2). These results are contrary to a large body of literature that shows student learning outcomes increase during active learning (e.g., Ambrose et al., 2010; Dolan & Collins, 2015; Eddy & Hogan, 2014; Freeman et al., 2014; Haak et al., 2011; Kober, 2015; Prince, 2004; Ruiz-Primo et al., 2011) and may indicate that while instructional approaches can impact student learning outcomes in decision-making and systems thinking, other factors are also important, such as learning environment and course content alignment. A recent study on self-reporting student outcomes in active-learning classrooms indicates that while students learn more in active-learning settings, they perceive fewer learning gains compared to students in lecture settings due to the increased cognitive effort they experience during active learning (Deslauriers et al., 2019). While our methods of measuring systems thinking and decision-making were conducted through validated assessment tools (Davis & Stroink, 2016; Eggert & Bögeholz, 2009), it is possible that these methods are limited in scope and may not be indicative of actual student gains represented within this study (other factors and limitations associated with our measure of student outcomes are discussed further in Sommers and colleagues [2019]). Additionally, future faculty development workshops should highlight this discrepancy in student perceptions of learning and provide resources for instructors to proactively prepare students for the perceived challenges of active learning compared to passive learning (Deslauriers et al., 2019). Another student learning outcome we did not directly measure (separate from the systems thinking and decision-making assessments) is student engagement in the classroom, as indicated by an increase in student voice and participation in discussion. We observed this outcome by measuring the transition away from Lecture to Socratic teaching style after faculty completed the workshop (Year 2), as the Socratic teaching style is differentiated from Lecture style in that it includes more student engagement in discussion (Lund et al., 2015). Although these observations were made while assessing teaching styles, not student outcomes, students may have been more receptive to the shift in environment and thus engaged more in discussion. Finally, other student outcomes that we did not consider within this study include student absences per semester, student-generated questions during class discussion, and whether students pursue more FEW courses or select FEW Nexus career paths after completing the course. These data could provide insight into student outcomes as a result of a shift to active learning and would benefit future investigations into teaching and learning in the FEW Nexus areas (Chávez et al., 2019).

Conclusion

While our study findings are limited by the relatively modest number of courses at one institution, they nonetheless contribute to a larger body of research regarding faculty development in undergraduate STEM education and student outcomes in light of active learning compared to traditional lecture methods and shed light on future areas of research. As such, our findings are of interest regarding the teaching and learning of science because they can inform faculty development in interdisciplinary undergraduate contexts, as well as course planning and design in other institutions.

Amie S. Sommers (asommers2@unl.edu) is a senior STEM education specialist in the Engineering and Computing Education Core, College of Engineering, at the University of Nebraska-Lincoln. Holly White (holly.white2@maine.edu) is a doctoral student in the Ecology and Environmental Sciences program at the University of Maine. Jenny M. Dauer (jenny.dauer@unl.edu) is an associate professor and associate director in the School of Natural Resources at the University of Nebraska-Lincoln. Cory Forbes (cory.forbes@uta.edu) is chair of the Department of Curriculum and Instruction, Fenton Wayne Robnett Endowed Professor of Science Education, and director of the STEM Education Research Collaboratorium and Resource Center in the College of Education at the University of Texas at Arlington.