National efforts to promote diversity, inclusion, and student perseverance in science, technology, engineering and mathematics (STEM) programs recommend including research as an integral part of the undergraduate experience (AAAS, 2011; Olson & Riordan, 2012). Studies show that traditional undergraduate research experiences (UREs) provide students with varied benefits, including increased self-efficacy and increased likelihood to persist in STEM (Ainscough et al., 2016; Auchincloss et al., 2014; Corwin et al., 2015; Hanauer et al., 2016; Jordan et al., 2014; Kloser et al., 2013; Mordacq et al., 2017; Olson & Riordan, 2012; Seymour et al., 2004). Unfortunately, inequities are associated with student access to UREs due to factors such as lack of awareness, lack of time, and financial constraints (Bangera & Brownell, 2014). Course-based undergraduate research experiences (CUREs) are gaining acceptance as equitable alternatives to traditional UREs, leading STEM departments nationwide to incorporate research into their undergraduate curricula (Auchincloss et al., 2014; Bangera & Brownell, 2014; Dolan, 2016; NASEM, 2015; Rodenbusch et al., 2016). CUREs allow students to perform broadly relevant and novel research within courses designed to include scientific practices, collaboration, and iteration (Auchincloss et al., 2014). Incorporating CUREs into the required curriculum affords research opportunities to a greater number and more diverse group of students, as they need not compete for spots, forego necessary employment, or be equipped with the confidence and know-how to obtain a position. Additionally, implementing research in required lower-division courses eliminates student barriers associated with class rank or lack of awareness (Bangera & Brownell, 2014).

The SIRIUS Project at California State University, Sacramento, has supported the implementation of CUREs or CURE modules (short investigations linked to other course CUREs) to replace cookbook and/or nonresearch-based curricula in 10 courses offered by the Department of Biological Sciences, which range from introductory to advanced levels (McDonald & Landerholm, 2018; McDonald et al., 2019). SIRIUS courses were designed by faculty participating in a four-year faculty learning community (FLC) and are coordinated around scientific threads related to human impacts on the impaired American River, which runs adjacent to campus. Biology students may take up to eight of these redesigned courses depending on their specific major concentrations. Because FLC faculty worked together to design, align, and implement the courses, they know which skills students acquire as they progress from the prerequisite to the advanced courses. SIRIUS program goals, course research descriptions, and faculty development activities are described in McDonald et al. (2019).

While this study assesses the impacts of the two redesigned introductory biology laboratory courses on students’ self-efficacy and future goals, it also represents the first step in our long-term, cross-sectional study because we are able to test protocols and pilot surveys for future use. Introductory courses often suffer from high dropout and failure rates, and students of color, first-generation students, and those experiencing poverty are particularly vulnerable (Freeman et al., 2007; Mintzes & Leonard, 2006; Peter et al., 2002; Slater et al., 2006). Evidence suggests that early participation in research leads to increased confidence, perseverance in STEM courses, and an increased likelihood of pursuing postgraduate STEM degrees and careers (Corwin et al., 2015). The literature further documents these and other benefits of research participation for students from historically underrepresented backgrounds (Corwin et al., 2015; Dirks & Cunningham, 2006; Espinosa, 2011; Freeman et al., 2007; Hernandez et al., 2013; Jones et al., 2010; Mintzes & Leonard, 2006; Villarejo et al., 2008). As a result, introductory courses are ideal venues for engaging all students in what is likely their first research experience (Bangera & Brownell, 2014; Corwin et al., 2015).

The SIRIUS Project is grounded in situated learning (Lave & Wenger, 1991) and social cognitive theories (Bandura, 1986), as faculty and students work in a community of practice (Allee, 2000) toward a common goal. Self-efficacy, a component of social cognitive theory (Bandura, 1977), has been shown to influence a number of student outcomes, including motivation, retention, and career decisions (Auchincloss et al., 2014; Chemers et al., 2001; Corwin et al., 2015; Estrada et al., 2016; Holland, 1997; Marra et al., 2009). Few studies have measured the impacts of CUREs on the self-efficacy of students in high-enrollment introductory courses (Mordacq et al., 2017; Olimpo et al., 2016), and still fewer report the impacts of introductory-level CUREs on students who are historically marginalized (Olimpo et al., 2016; Rodenbusch et al., 2016). The majority of studies indicating that CUREs provide students with similar benefits to UREs involve small-scale implementations of courses (Brownell et al., 2012; Caspers & Roberts-Kirchhoff, 2003; Drew & Triplett, 2008; Kloser et al., 2013) for self-selecting students (Brownell et al., 2012; Drew & Triplett, 2008; Rodenbusch et al., 2016), upper-division students (Brownell et al., 2015, 2012; Caspers & Roberts-Kirchhoff, 2003; Drew & Triplett, 2008; Kloser et al., 2013), and/or students from nationally competitive research institutions (Brownell et al., 2015, 2012; Drew & Triplett, 2008; Kloser et al., 2013; Olimpo et al., 2016; Rodenbusch et al., 2016; Staub et al., 2016). While these studies all report benefits to their study populations, to our knowledge only Olimpo et al. (2016) studied a population where the majority of students were from backgrounds traditionally underrepresented in STEM.

Our teaching university is a designated Hispanic-Serving Institution (HSI) and an Asian American, Native American, and Pacific Islander Serving Institution (AANAPISI), with over 70% of undergraduate students identifying as students of color. Thirty-two percent are first generation and over 50% qualify for federal Pell grants, providing an opportunity to examine the impacts of CUREs on both introductory students and those traditionally underrepresented in STEM. The specific aims of this study were to (1) validate a self-efficacy survey instrument for our population and (2) evaluate the effects of research-based curricula on students’ laboratory self-efficacy and future goals. We hypothesized that after CURE participation, students would report increased laboratory self-efficacy, which would vary for different demographic groups. We also hypothesized that students would report increased interest in pursuing future careers and academic pathways related to STEM research.

Methods

Course descriptions and student populations

Study participants included students enrolled in two introductory courses, “BIO1: Biodiversity, Evolution, and Ecology” and “BIO2: Introduction to Cells, Molecules and Genes” during the spring 2016 and spring 2017 semesters. Both lab courses meet once per week for three hours and are required for biology majors at our institution.

BIO1 serves around 200 students each semester and incorporates a CURE module assessing the effects of nitrogen fertilizer on macroinvertebrate biodiversity in the American River over seven nonconsecutive weeks (Figure 1). Students report their findings in a formal lab report at the conclusion of the semester, and data are passed to other SIRIUS courses for analysis. BIO2 serves 125–150 students per semester, and incorporates a three-module CURE spanning the 16-week semester (Figure 2). Modules two and three are repeated, providing opportunity for troubleshooting and iteration. Student groups report their findings through both a written and oral report at the conclusion of the semester and pass PCR products and results on to other SIRIUS courses.

Assessment instruments

A two-part survey was administered both pre- and postcourse in an online format operated by Campus Labs. Students were invited to participate during class time, but participation was voluntary. Sample sizes are smaller than typical enrollments for each semester due to eliminated responses. Four criteria were used for eliminating students from the study: (1) the student did not complete both the pre- and postcourse surveys, (2) portions of the survey were unanswered, (3) the student gave the same response for all items, which may indicate they did not actively participate, or (4) the student repeated the course and was in both analyzed semesters; therefore, they were removed from the second semester.

The Self-Efficacy Survey (SES) included nine Likert-like items (Table 1), adapted from Reeves et al. (2018), and was used to evaluate students’ confidence related to laboratory skills. Exploratory factor analysis was employed to validate the SES.

The second part of the survey, the Future Goals Survey (FGS) included five Likert-like items modified from the Louis Stokes Alliance for Minority Participation (LSAMP) survey, which ask students about their future STEM intentions (Table 1) (CSU-LSAMP, n.d.).

This study was approved by the California State University, Sacramento Institutional Review Board under protocol #13-14-148, and all students who participated in the surveys agreed to the use of their survey responses and demographic data.

Self-Efficacy Survey validation

Precourse responses from both BIO1 and BIO2 (N = 498) were analyzed to determine if they met the assumptions of factor analysis. The survey’s online format prevented missing data and six observations had a Mahalanobis distance below the cutoff (p < 0.001) indicating they were multivariate outliers. These observations were removed (N = 492). Inter-item correlation was evaluated using the psych package in R (Revelle, 2014), and all items had correlations above 0.30 (Appendix 1). The psych package (Revelle, 2014) was also used to calculate Kaiser’s measure of sampling adequacy, which showed the data had good factorability (0.91). Skewness and kurtosis for each item was below |1.0|; however, Shapiro-Wilk tests and Mardia’s multivariate normality tests, both calculated using the MVN package in R (Korkmax et al., 2015), indicated nonnormality for all items. Variance inflation factors for linear regressions of all items against one another were less than three, indicating no issues with multicollinearity. Based on these observations and recommendations from the literature (Knekta et al., 2019), the data were deemed suitable for an exploratory factor analysis (EFA).

Parallel analysis of the data was performed, and a scree plot was created, both using the psych package in R (R Core Team, 2016; Revelle, 2014). Both the parallel analysis and scree plot (Appendix 2) suggested three factors, so the pysch package in R (R Core Team, 2016; Revelle, 2014) was used to run a three-factor EFA. A 0.40 cutoff for standardized pattern coefficients was used (Table 2). This three-factor model explained a total of 62% of data variance and was used to divide the SES into three constructs: one factor for items 6–9, related to scientific literacy and data interpretation (SL-26% of data variance), a second factor for items 3–5, related to scientific communication and collaboration (CO-18% of data variance), and a third factor for items 1 and 2, related to experimental design (ED-18% of data variance). The Tucker-Lewis Index of factoring reliability (TLI) was high (0.986), and the Root Mean Square Error of Approximation (RMSEA) was low (0.042), suggesting a good model. Cronbach’s α for each factor was calculated using R’s psych package (R Core Team, 2016; Revelle, 2014) and all were > 0.75 (SL α = 0.86; CO α = 0.76; ED α = 0.80), showing high internal reliability. For each factor, removing an item either lowers the α (SL, ED) or leaves the raw α unchanged (CO-item 5), indicating that each item contributes to the overall meaning of the survey (Appendix 3).

Self-Efficacy Survey analysis

Likert-like item responses ranged from 1: “Strongly Disagree” to 4: “Strongly Agree” and were summed for each construct to create pre- and postcourse construct scores for each student. Pre- and postcourse responses for each construct were compared using Wilcoxon Signed Ranks tests. For each course, the two semesters (spring 2016, spring 2017) were combined after no statistical differences were found between precourse construct scores (BIO1 N = 173; BIO2 N = 149).

Summed construct scores were also used to explore relationships between students’ demographics and their self-efficacy pre- and postcourse. Demographic data including ethnicity, Pell grant eligibility (a measure of socioeconomic status), gender, and first-generation status were obtained through the Office of the Registrar. Ethnicity was broken into three categories: Asian, Underrepresented Minority (URM), and White. The URM category includes students who are identified as Hispanic, African American, Native American, and Pacific Islander. Students with an ethnicity reported as “other” or “two or more ethnicities” were not included in the ethnicity analyses. Our data violated assumptions of normality, so nonparametric tests were used in these analyses. Kruskal-Wallis tests were used for ethnicity data, and if necessary, multiple comparison tests for Kruskal-Wallis were performed posthoc. For all other demographic variables, Mann-Whitney U tests were performed. Our university only provides binary (male and female) genders at this time.

Future Goals Survey analysis

Items on the FGS were analyzed qualitatively using histograms that compared pre- and postcourse responses. A chi-square analysis was performed in Microsoft Excel comparing all BIO1 precourse responses and all BIO2 postcourse responses for each FGS item to see if distributions differed for student populations before and after the introductory biology series.

Results

Analysis of Self-Efficacy Survey

While medians are reported for nonparametric tests, it is possible to get significant differences between groups even when medians are identical. For this reason, we have also reported means in our tables. Wilcoxon Signed Ranks tests showed differences in pre- and postcourse self-efficacy for BIO1 students in all SES constructs (ED Mdn₁ = 6, Mdn₂ = 7, Z = -5.67, r = 0.43, p < 0.001; CO Mdn₁ = 9, Mdn₂ = 10, Z = -4.83, r = 0.37, p < 0.001; SL Mdn_{1 =} 7, Mdn_{2 =} 13, Z = -6.75, r = 0.51, p < 0.001; Table 3). BIO2 students’ only significant change in confidence was for SL (SL Mdn₁ = 12, Mdn₂ = 12, Z = -4.07, r = 0.33, p < 0.001; Table 3).

Table 4 summarizes the demographics for students in both analyzed courses. Table 5a summarizes the differences in construct scores for students of differing demographics both pre- and postcourse for each course. No significant differences were found between pre- and postcourse confidence for BIO1 students of different ethnicity or first-generation status. Mann Whitney U tests revealed confidence differences between BIO1 males and females both pre- and postcourse for the CO construct (CO_Pre: Mdn_Female = 9, Mdn_Male = 10, U = 2582, Â₁₂ = 0.37, p < 0.01, CO_Post: Mdn_Female = 10, Mdn_Male = 11, U = 2804, Â₁₂ = 0.41, p < 0.05, Table 5b). A precourse difference was also found between males and females for the SL construct, but this was not reported postcourse (SL_Pre: Mdn_Female = 12, Mdn_Male = 12, U = 2461.5, Â₁₂ = 0.36, p < 0.01, SL_Post: Mdn_Female = 12, Mdn_Male = 13, U = 3015, Â₁₂ = 0.44, p > 0.05; Table 5b). Despite no reported differences for genders for the ED construct precourse, a postcourse difference was reported (ED_Pre: Mdn_Female = 6, Mdn_Male = 6, U = 2903, Â₁₂ = 0.42, p = 0.05, ED_Post: Mdn_Female = 7, Mdn_Male = 7, U = 2806.5, Â₁₂ = 0.41, p < 0.05; Table 5b). Pell-eligible students in BIO1 reported differences in precourse confidence for the CO and SL constructs when compared to their noneligible peers, but these differences were not reported postcourse (CO_Pre: Mdn_Eligible = 9, Mdn_{Noneligible =} 10, U = 4672.5, Â₁₂ = 0.63, p < 0.01, CO_Post: Mdn_Eligible = 10, Mdn_Noneligible = 10, U = 2011.5, Â ₁₂ = 0.54, p > 0.05; SL_Pre: Mdn_Eligible = 11.5, Mdn_Noneligible = 12, U = 4607, Â_{12 =} 0.62, p < 0.01, SL_Post: Mdn_Eligible = 12.5, Mdn_Noneligible = 13, U = 3995.5, Â₁₂ = 0.53, p > 0.05 ; Table 5b).

No differences in pre- and postcourse confidence were found for BIO2 students of differing gender or first-generation status. BIO2 Pell-eligible students reported differences in precourse confidence for the SL construct when compared to their noneligible peers, but this difference was not reported postcourse (SL_Pre: Mdn_Eligible = 12, Mdn_Noneligible = 12, U = 3379, Â ₁₂ = 0.62, p < 0.05, SL_Post: Mdn_Eligible = 12, Mdn_Noneligible = 13, U = 2888.5, Â ₁₂ = 0.53, p > 0.05; Table 5c). Kruskal-Wallis tests revealed that BIO2 students of different ethnicities reported precourse confidence differences for all three constructs and postcourse differences for the SL construct only (ED: χ²_Pre = 8.23(2), p < 0.05, χ²_Post = 5.30(2), p > 0.05; CO: χ²_Pre = 13.00(2), p < 0.01, χ²_Post = 4.41(2), p > 0.05; SL: χ²_Pre = 14.35(2), p < 0.001, χ²_Post = 11.07(2), p < 0.01; Table 5c). Multiple comparison tests for Kruskal Wallis performed posthoc revealed that these differences were between URM students and their white peers for all constructs (ED_Pre: Mdn_URM = 6, Mdn_White = 7, CO_Pre: Mdn_URM = 9, Mdn_White = 10, SL_Pre: Mdn_URM = 11, Mdn_White = 13; Table 5c). Posthoc tests also revealed a difference between these two groups’ postcourse confidence in the SL construct (SL_Post: Mdn_URM = 12, Mdn_White = 14; Table 5c). An additional precourse difference for the CO construct was shown for white and Asian students (CO_Pre: Mdn_Asian = 9, Mdn_White = 10, Table 5c).

Future Goals Survey

For all Future Goals Survey items, distributions of Likert-like responses remained consistent from pre- to postcourse for both BIO1 and BIO2. For example, in both courses “undecided” was the most-often selected response pre- and postcourse when students were asked about their intentions to enroll in a graduate STEM program (Appendix 3a, 3b) or pursue a career in a STEM-related field after graduation (Figure 3a). The majority of students in both courses “strongly agreed” or “agreed” that they intended to enroll in a medical, dental, or other health professional school program (Figure 3b) and “strongly disagreed” that they intended to pursue a career teaching high school STEM (Appendix 3C). Chi-square analyses that compared all BIO1 students’ precourse responses to all BIO2 students’ postcourse responses showed no significant differences for any survey item.

Discussion

The SIRIUS Project was created to introduce students on our campus to high-quality laboratory experiences, with the goal of providing them with the documented benefits of traditional undergraduate research in the context of a community of students and faculty studying a shared problem. In this study, student survey data provided valuable insight about how our research-based curricula may be influencing introductory biology students’ attitudes about their scientific skills and future goals. Our hypothesis that students would report improved self-efficacy after participating in the redesigned courses was partially supported. Our survey validation methods resulted in three constructs from which to measure self-efficacy: experimental design (ED), communication/collaboration (CO), and scientific literacy (SL). While our BIO1 students reported significant increases from pre- to postcourse in all constructs, our BIO2 students only reported significant increases for the SL construct (Table 4). Corwin-Auchincloss et al. (2014) postulate that enhanced self-efficacy is a medium-term outcome associated with CUREs, meaning students can experience it at the end of a CURE, which is consistent with our data, particularly for BIO1 students. These BIO1 results are also consistent with Mordacq and colleagues (2017), who observed an increase in self-efficacy related to scientific inquiry skills in introductory biology students after only two, nine-week courses. Our BIO2 course provides a longer, more in-depth research project, so lack of changes in BIO2 students’ self-efficacy in the ED and CO constructs may reflect the new curriculum’s high level of rigor.

Because the SES measures confidence in three different scientific skill subsets, we were able to compare these data to known curricular elements to either confirm their effectiveness or highlight areas for improvement. For instance, both introductory courses placed particular emphasis on reading the scientific literature in their new curricula, so it is encouraging that both BIO1 and BIO2 students reported significant gains in confidence for the SL construct. Future qualitative work may reveal more about why BIO2 students did not report self-efficacy increases in the ED and CO constructs.

We hypothesized that self-efficacy would be influenced by demographic variables, including gender, ethnicity, socioeconomic status, and first-generation college student status. Gender-based self-efficacy differences are well-documented in STEM education research, and a misperception persists that the biology field does not suffer gender-based equity issues when compared with other sciences because undergraduate biology courses are comprised of a female majority (Eddy et al., 2014; Flanagan & Einarson, 2017; Meece, et al., 2006). Similar to national trends, our BIO1 course is predominantly female; however, females reported lower postcourse self-efficacy for both the ED and CO constructs when compared to their male peers. This is an important finding for further exploration because self-efficacy has been linked to retention, and an important goal of CUREs, and the SIRIUS Project in particular, is to retain a greater number and more diverse population of students in biology. Interestingly, no gender differences were found in BIO2, which raises the question about what may have changed from the end of one introductory course to the beginning of another. On our campus, most students do not enroll in BIO2 the semester immediately following BIO1 due to chemistry prerequisites and other structural factors. Perhaps increased exposure to experimental design and communication/collaboration within their chemistry courses leads to an overall decrease in our gender self-efficacy gap for BIO2 students. These ideas can be explored in future studies, particularly with qualitative data we have collected through interviews, focus groups, and open-ended survey questions not reported here.

Pell-eligible students in BIO1 reported lower precourse confidence in all constructs, but all of these differences were eliminated postcourse. Pell-eligible students in BIO2 only reported differences from their noneligible peers for the precourse SL construct, which was also eliminated postcourse. These data suggest that our curricula may actually bolster the confidence of our Pell grant recipients who start the semester with lower confidence. This is particularly important for our student population, which has a large proportion of students who qualify for Pell grants.

No relationships were found between ethnicity and self-efficacy in our BIO1 data. However, white BIO2 students reported higher precourse self-efficacy than their URM peers for all constructs, higher precourse self-efficacy than their Asian peers for the CO construct, and higher postcourse self-efficacy than their URM peers for the SL construct.

CUREs are proposed to overcome inequities inherent in traditional UREs (Bangera & Brownell, 2014; Estrada et al., 2016). Our data show that both the BIO1 and BIO2 curricula have beneficial impacts on student self-efficacy regardless of first-generation status or socioeconomic status; however, our curriculum may not entirely close the gap for female BIO1 students or BIO2 students from underrepresented ethnic groups. Our data regarding first-generation college students is not surprising considering historical retention data within the College of Natural Sciences and Mathematics shows that our first-generation students remain in STEM at equal rates compared to their peers.

Our hypothesis that students would demonstrate increased interest in entering a STEM career or postbaccalaureate academic path after participating in the introductory CURE curricula was not supported. Our data show few changes, and many students still indicate that they are “undecided” about their future goals. Our survey data are consistent with Seymour and colleagues (2004), who found that participation in research did not change students’ interest in pursuing STEM careers or postgraduate work. However, the Seymour study population consisted of students participating in traditional UREs, primarily nearing the end of their undergraduate education. The population studied by Kloser et al. (2013) is more similar to our own, and they report a similar change in self-efficacy, with no change in research interest, following an introductory biology CURE consisting of nonvolunteer students.

Conclusion

This manuscript presents findings in the first step of a long-term cross-sectional study that aims to assess student impacts associated with our department’s response to a problem that plagues STEM education in the United States: equal opportunity for students to benefit from UREs. With the SIRIUS Project, we have eliminated barriers to gaining research experience in our department, as students who are employed, have significant commutes, and/or family obligations participate in research during normal class time. Our CUREs also offer a solution to the observation reported by Lopatto (2007) that students recruited into traditional UREs outside of their home institutions are less likely to continue with research.

A limitation of this study is the lack of baseline data collected before implementation of the new curriculum, which makes it difficult to assess whether gains are due to the CUREs or increased awareness/exposure to scientific practices inherent to any biology course. To address this, we used a prepost design and will look to current and future studies for important insights into the effectiveness and impacts of the SIRIUS courses in their entirety. We are currently analyzing data across multiple semesters of the upper-division CUREs, as well as for students in faculty-mentored research programs. Unlike our introductory students, preliminary data on our upper-division students show different trends related to future goals, after multiple research exposures and experiences gained through the SIRIUS Project’s integrated and sequential curriculum. Additional studies should also indicate whether students are gaining actual skills, along with confidence. Anecdotally, conversations with FLC faculty reveal that students in the upper-division courses demonstrate more advanced laboratory skills and increased research interests than similar cohorts before implementation of the new curricula in the lower-division courses.

We also acknowledge that our CURE curriculum is only one intervention aimed at retention of underrepresented groups, and we cannot deny or control for the varied lived experiences of our students. Our campus has an abundance of resources and interventions, and it is difficult to disaggregate these data to find out if students are benefitting from more than just the CURE curriculum. This is one reason why our SES is framed around lab skills confidence in particular rather than general self-efficacy. In addition to demonstrating positive outcomes from our introductory CURE curriculum and insights for course revisions, the results we present here represent important baseline data for future studies and continued use of the SES. ■

Acknowledgments

We thank the National Science Foundation (award number DUE-1432299), W.M. Keck Foundation, and the California State University Program for Undergraduate Biotechnology Education (CSUPERB) for their generous support of this work. We also thank the many CSUS faculty and students who participated in the SIRIUS Project as well as Susanne Gnagy for her impeccable proofreading/editing skills.

Online Supplemental Materials

Appendices 1–4—https://bit.ly/2Mx07Fu

Allison Martin (allison.martin2@tccd.edu) was formerly a lecturer in the Department of Biological Sciences at California State University, Sacramento, and is currently an adjunct instructor of biology at Tarrant County College, Northeast Campus in Hurst, Texas, and Dallas College in Dallas, Texas. Adam Rechs is a professor of biological sciences, Thomas Landerholm is a professor emeritus of biological sciences, and Kelly McDonald is a professor of biological sciences and director of the Center for Science and Math Success, all at California State University, Sacramento, in Sacramento, California.