Metacognition, the ability to think about and regulate one’s thinking, is widely recognized as a factor in effective student learning (Hattie et al., 1999; Tanner, 2012). A number of studies have explored methods designed to enhance students’ metacognitive abilities (e.g., Cook et al., 2013; McGuire, 2015; Kaplan et al., 2013; Zhao et al., 2014). One popular intervention is the exam wrapper—an exercise completed after an exam in which students reflect upon their performance on and preparation for that test (Ambrose et al., 2010; Lovett, 2013). Many examples exist (Eberly Center, 2016), but common questions asked of students include: How did you prepare for this exam? (Instructors may provide a list of options.) What specific kinds of mistakes did you make? (A list of general errors may prompt student analysis.) Given these kinds of mistakes, how would you change your preparation for the next exam?

Exam wrappers are designed to promote reflection, which instructors hope will lead students to view exams as learning experiences, and not just assessment. The hope, too, is that students will develop the habit of evaluating their own performance and increase their ability to self-regulate (Schraw et al., 2006).

Results are mixed on the effect of exam wrapper use on student achievement in science, technology, engineering, and mathematics (STEM) courses. Lovett’s (2013) original description and informal study of exam wrapper use across introductory courses in biology, calculus, chemistry, and physics showed that students’ perceptions of which study strategies were effective changed most when they were exposed to exam wrappers in more than one course, making the exam wrapper effect, then, seemingly “dose”-dependent (p. 34). Certainly, studies of exam wrapper use in a single course appear to show little effect on student performance (Soicher & Gurung, 2017; Kustritz & Clarkson, 2017). One study in a large introductory food science course using exam wrappers compared students’ self-reported study practices with exam and final grades. In this case, the performance of students with a final grade of B showed a modest correlation with study practices (Gezer-Templeton et al., 2017). The number of study practices, rather than the specific kinds, varied for students with different final grades. This finding corresponds to other work showing, perhaps not surprisingly, that higher-achieving students in introductory biology reported using specific cognitive and metacognitive strategies more frequently than their lower-achieving peers (Sebesta & Speth, 2017). As noted in Kustritz and Clarkson (2017), one key factor in the effectiveness of exam wrappers may be if instructors actively teach metacognitive strategies in addition to asking students to engage in activities designed to foster them.

This paper reports on a case study in which exam wrappers were used to promote student metacognition across five large introductory or core science and math courses. The impact of exam wrappers on self-reported use of metacognitive strategies and course outcomes was systematically examined, recognizing that some students in the study had exposure to exam wrappers in multiple courses simultaneously, increasing their “dose.” Students in the exam wrapper courses, along with four physics courses that did not use them, were asked to take the Metacognitive Awareness Inventory (MAI) (Schraw & Dennison, 1994). Scores on the MAI have been shown to relate to test performance and grade point average (GPA) (Young & Fry, 2008). Exam wrapper use, course outcomes, and MAI results were tracked across courses. Although the data showed that exam wrappers are not a magic bullet in promoting metacognitive awareness, a modest positive relation between exam wrapper use and course grades of first-year students as well as students farther along in their studies was observed. There was also a positive relation with male students’ cumulative GPA when they used wrappers in multiple courses. Students’ responses on various subscales of the MAI and their subsequent use of exam wrappers have implications for how instructors construct and implement the wrappers to maximize their potential usefulness.

This study was conducted in Fall 2017 at a midsize public research university in the United States. Exam wrappers were used in three introductory gateway courses for science majors including Introductory Biology (first semester), General Chemistry (first semester), and Introductory Calculus and two upper-level, follow-on courses for majors: Organic Chemistry (first semester) and Genetics. Four introductory physics courses (both semesters of courses for physical sciences or engineering majors and those for life science majors) did not use exam wrappers, but were included for comparison. All courses were large sections (100–300 students per section). Instructors varied in their pedagogical approaches, though all included some active learning during class, such as clicker questions (Hodges et al., 2017) and/or group work. Although instructors implemented exam wrappers differently, they all adopted the same nine items on the wrapper and delivered them via the learning management system. Students in all courses were asked to complete the MAI at the beginning and end of term. This project was approved as exempt through our Institutional Review Board (Protocol Y18LH26015).

Exam wrappers addressed three aspects of the learning situation: exam preparation (e.g., strategies used, number of practice problems completed), exam performance (types of mistakes made), and planning for the future (help available, awareness of learning habits, adjustments needed). Questions were generally worded the same for each course: 1.Did you feel prepared for the exam? 2.What did you do to prepare for the exam? (read before class, come to class, thinking, answering questions or doing practice problems) 3.Did you use certain assignments or resources as learning tools or just get them done as fast as possible? 4.How many problems or questions did you complete before the exam? 5.Did you miss questions due to not reading carefully (or a subject-specific reason)? 6.What was the main reason for missing exam questions? 7.What type of question did you miss the most? 8.Did you know how to get help in the course? 9.Do you feel that you have made a good assessment of your learning habits and know how to adjust your approach?

This online version, created by author Tara Carpenter, used the test tool function in the learning management system that allowed instructors to provide automated feedback based on students’ responses to multiple-choice prompts (Carpenter et al., in press). Thus, even in these large classes, students received a minimum of feedback on their study choices. Instructors determined whether the exam wrappers would be required or optional, and whether points would be awarded for completion of the wrappers as described in Table 1. Incentives for completing the MAI and the exam wrappers did not exceed 1% of the total course grade.

The MAI allows learners to assess their knowledge and regulation of cognition. Metacognitive awareness is evidenced in inventory subscales that capture individuals’ knowledge of themselves and their behavior related to planning, monitoring, and evaluating learning (Table 2). Scoring options for the MAI vary and include the original 100-mm, bi-polar sliding scale (Schraw & Dennison, 1994) and a five-point Likert scale (Harrison & Vallin, 2018). A dichotomous version (Soicher & Gurung, 2017) used in the current study asked students to indicate whether “the statement is true or false as it generally applies to you when you are in the role of a learner.” The maximum score was 52, corresponding to the number of items on the scale.

A total of 1,894 students completed the courses and consented to the use of their data. First-year students accounted for 41% of the total, whereas 59% were transfer students or had earned more than 30 credits at this university. Students’ academic history (high school or transfer GPA), metacognitive awareness scores, exam wrapper completion, and course grades were used in analyses. Descriptive and inferential statistics were calculated using SPSS Version 25. Random missing data due to noncompliance were handled by pairwise deletion.

The results presented below address several key questions: What do students’ metacognitive awareness scores suggest about their use of strategies offered in a course? Which students used the exam wrappers? Is there a relation between exam wrappers, academic performance, and metacognitive awareness as measured by the MAI? How did use of exam wrappers in multiple courses affect students’ metacognitive awareness and academic achievement?

Given that students’ overall college experience might affect their self-awareness and, potentially, their use of metacognitive strategies, some results are presented separately for first-year students (direct entry ≤ 30 cumulative credits by the end of the term) and continuing students (>30 credits and all transfers). In general, Introductory Biology, General Chemistry, and Calculus I enrolled many more first-year students, whereas the four Physics courses, Genetics, and Organic Chemistry were primarily populated by continuing students. Due to differences in the nature of the exam wrapper intervention, some analyses were conducted at the course level.

Differences related to gender and ethnicity were examined. Ethnicity did not contribute to an understanding of students’ metacognitive behavior or academic outcomes. Gender effects are discussed below.

Scores on the MAI at the beginning (MAI-Pre) and end of the term (MAI-Post) ranged from 0 to 52, with a median score of 41 and 42 respectively. Scores of 0, 1, and 52 on the MAI-Pre suggested exaggerated scoring or inattention to the task. They were deemed outliers and removed from first-year student MAI data (n = 29). Outlying scores for continuing students were included so as to allow for their experience in higher education (Table 3). For students enrolled in more than one study course, a representative average of MAI scores across courses was calculated. Approximately 20% of the students were enrolled in more than one of the study courses (including Physics). Effects of exposure to exam wrappers in multiple courses are discussed separately.

Despite high scores on the MAI-Pre, certain items were identified less often as “generally true of me” by both first-year and continuing students. Such strategies involved planning and reflecting (before, during, and after a learning situation). In contrast, strategies used most often were those that occur during the learning process (i.e., in-time strategies), including approaches that have worked in the past, asking for help, and rereading when confused (Table 2).

Among students who completed both administrations of the MAI, total scores increased significantly by the end of the term. For continuing students, the increase occurred only among those enrolled in exam wrapper courses (Table 3). Mean subscale scores increased pre-to-post such that students in exam wrapper courses reported significantly greater use of the strategies in all four of the “least used” subscales, whereas only the mean score for the evaluation subscale was significantly higher at the end of the term in the nonexam wrapper courses.

Exam wrapper use varied widely between the courses in this study (Figure 1). In courses offering extra credit points and/or requiring the wrappers, completion rates were as high as 89%. However, when wrappers were optional and no points were awarded, response rates were much lower. Students were encouraged to use the wrappers as a remedy to an exam score that was lower than desired.

Across all nine courses, first-year students’ scores on the MAI-Pre were not related to prior academic history (high school GPA) or to performance in the current course (exam or course grades). Neither did they relate to students’ use of exam wrappers in the courses that offered them; that is, self-reported metacognitive strategy use did not translate into proactive behavior or better outcomes (Table 4). MAI-Post scores correlated significantly with exam grades, course grade, and prior GPA. Although students reported higher scores on the MAI-Post, it did not appear to be connected to their use of the exam wrappers.

Exam wrapper use was only weakly associated with high school GPA, but more strongly associated with early performance in the study course. First-year students with higher scores on exam #1 and #2 were more likely to use the first two wrappers than those with lower exam scores. Course grade was most strongly related to high school GPA and exam performance, although a positive correlation was also observed with exam wrappers.

Gender differences for first-year students were mixed; females reported higher MAI-Pre scores and had stronger high school GPAs, but there was not a difference in terms of exam and course performance. Females were somewhat more likely to use the exam wrappers (Table 4).

For students with more than 30 credits and transfers to the university, MAI-Pre scores (including outliers) did not correlate with prior academic performance (high school or prior college GPA), but were somewhat related to exam and course grades (Table 4). Students with higher MAI-Pre scores were not, however, more inclined to use the exam wrappers. As with first-year students, MAI-Post scores for continuing students correlated positively with grades in the course. Exam wrapper use did not correlate with MAI-Post scores. However, scores on the MAI increased significantly from pre-to-post for students in exam wrapper courses only.

Students’ use of exam wrappers was not related to their academic history (prior GPA) or to their performance on the first exam. By the second exam, however, a significant negative relation was found; lower exam #2 scores were associated with subsequent use of exam wrapper #2 as well as overall exam wrapper use. Course grade was most strongly related to exam scores and prior GPA, but also correlated positively with exam wrapper use.

Females had significantly higher prior GPAs, although males tended to perform better on exams in the study course. There were no gender differences in overall course grade. Females were more likely to use exam wrapper #1 and more of the total exam wrappers offered (Table 4).

Self-reported metacognitive awareness (as measured by the MAI-Pre) was not strongly related to actual behavior (use of exam wrappers) or outcomes in this study. Other variables did, however, help to explain students’ performance in the courses, including prior GPA, early exam performance, and exam wrappers. Differences in the administration of exam wrappers (Table 1) warranted course-level analyses among these variables and final grades. Multiple linear regression analyses were carried out in each exam wrapper course to predict course grade from prior GPA, exam #1, and exam wrappers. These equations were statistically significant, although differences resulted across courses and student level in the extent to which each variable contributed uniquely to the total variance in course grade (Table 5).

For first-year students, the grade on exam #1 was highly predictive of the final grade in each course. Prior GPA and exam wrapper use also contributed uniquely to the variance. Among continuing students, exam #1 was the strongest predictor, and prior GPA explained course grade in all but one course. After controlling for the effect of exam #1 and prior GPA, continuing students’ use of exam wrappers was a significant predictor, except in one course that had very low completion rates.

Physics courses (N = 780) were primarily composed of continuing students. No exam wrappers were offered. MAI-Pre scores were not significantly correlated with exam or final grades in any of the Physics courses. As in exam wrapper courses, MAI-Post scores correlated with exam and course grades, but only in the two Basic Physics courses. The variable most strongly related to exam and course grades was prior academic history (high school or transfer GPA). Although some gender differences were observed in prior GPA (higher for females in level I of each type of Physics course), no consistent pattern was found relating gender to grades.

There were 432 students enrolled in more than one of the courses in this study, with the most frequent combination being Introductory Biology and General Chemistry. These courses, along with Calculus, are often taken by students in their first year at the university (direct entry from high school or transfers). The combined number of exam wrappers offered to students enrolled in two of these courses ranged from five to seven; nine were available for those enrolled in all three courses. A cumulative benefit of exam wrappers would inform the literature on first-year academic initiatives.

Students enrolled in a combination of Introductory Biology, General Chemistry, and/or Calculus (N = 173) were exposed to the greatest number of exam wrappers, having on average, six opportunities. They were primarily direct entry (93%) and most were in their first or second term at this university (Table 6).

Cumulative GPA at the end of the term, incorporating grades from exam wrapper courses and any others completed at this university, was the relevant outcome variable. Correlational analyses (Table 7) showed that the variable most strongly related to cumulative GPA was prior GPA (high school or prior college). Exam wrapper use was also positively related to the outcome.

The results were most informative, however, when examined by gender. Neither the number of cumulative credits (college experience) nor the distribution of students across the three 100-level courses differed by gender. Females tended to have a stronger academic background and higher MAI-Pre scores, and they used more of the offered exam wrappers. However, their use of exam wrappers was not significantly correlated with cumulative GPA or with the final grade in any of the individual courses. For males, there was a significant positive relation between exam wrapper use and cumulative GPA.

Regression equations predicting cumulative GPA for students in multiple 100-level courses were significant (Table 8). For males, prior GPA and exam wrapper use both contributed uniquely to the variance, but for females, only prior GPA significantly predicted cumulative GPA.

Metacognitive Awareness Inventory scores may not reflect actual practice

In this study, the MAI was not necessarily a good indicator of students’ metacognitive knowledge and behavior. Scores on the MAI-Pre were not related to the use of exam wrappers, nor did they explain exam and course grades. Results suggest that many students exaggerated their scores on the first administration of the MAI.

The “least-used” MAI items and subscales, those which more than 25% of students indicated not using, tended to be of the type that requires a higher level of awareness and initiative, both before and after a learning task. For example, only 43% answered true for “I ask myself questions about the material before I begin,” and only 48% of students indicated knowing “how well I did once I finish a test.” Exam wrappers are designed to walk students through those types of strategic efforts to prepare for and improve upon performance. Responses on the MAI-Pre suggest that interventions like the exam wrapper are needed, but students will not necessarily use them.

Metacognitive awareness scores tended to increase over the term and the sub-scales identified as “least used” showed some improvement. Mean scores increased significantly for declarative knowledge, planning, and comprehension monitoring, but only among students in the exam wrapper courses. Evaluation sub-scale scores increased across all courses in the study. MAI-Post scores correlated significantly with exam and course grades, but not with exam wrapper use as might be expected. Overall, it appears that students’ end-of-term scoring of their metacognitive awareness was influenced most by how well they performed in the course. Alternatively, MAI-Post scores may have been influenced by the metacognitive messages received, even if the student did not take advantage of the intervention.

Exam wrappers were required in some courses but not others, earned points or not, were completed as a group or individually, and were supplemented by different types of metacognitive instruction and activities. Yet, even given this variety of scenarios, use of exam wrappers was significantly related to students’ course grades. The strongest effect was observed in courses that required and/or offered points for them. Although some students with lower exam #1 scores did not use the first exam wrapper, completion rates were much higher when incentives were offered.

Among continuing students, a relation between grades and use of exam wrappers was not evident until after the second exam. Students with a lower exam #2 score were more likely to use the second exam wrapper and use more wrappers overall. The grade on the first exam was the strongest predictor of course grade, but importantly, exam wrappers also contributed in all but one of the courses (General Chemistry). Completion rates were quite low in that course with only 14% of students using two or more exam wrappers.

In addition to exams, course grades encompassed in-class and out-of-class activities that may also have stimulated metacognitive awareness (e.g., clicker questions, study guides). Each of these courses, whether offering exam wrappers or not, featured additional metacognitive instruction and activities. The extent and frequency varied by course. For example, the Introductory Biology, Introductory Chemistry, Organic Chemistry, and Genetics instructors provided a metacognition lecture (McGuire, 2015).

First-year students enrolled in more than one exam wrapper course (Introductory Biology, Introductory Chemistry, and/or Calculus) appeared to benefit from the higher “dose” in several ways. The combined exam wrapper completion rate was higher for students in more than one course than it was for first-year students overall. Introductory Chemistry students, in particular, completed a higher percentage of the offered exam wrappers when they were exposed to multiple opportunities than they did when only enrolled in the Chemistry course. As noted, total exam wrapper completion was related to students’ cumulative GPA, which suggests that metacognitive instruction in more than one course has the potential to impact students’ performance beyond just the course in which exam wrappers are offered. There was some indication that the increased dose was particularly helpful to male students; further study of this question is warranted.

There are limitations to generalizing about the findings of this study. The number of exam wrappers offered varied across courses, and completion rates differed depending upon whether they were required or if points could be earned. A metacognition lecture was provided in four of the courses, and in at least one course, students completed the exam wrappers during a group discussion time. Standardization of class procedures might provide information on how specific implementation practices impact effectiveness.

The scoring for the MAI may also affect study conclusions. Each of the 52 items on the MAI was rated dichotomously for whether it was generally true or false for individuals in their role as a learner. Dichotomous scoring allows for a wider spectrum of “true” scenarios (always, usually, often) than “false” (never or rarely). Expanding the response options to a five- or seven-point scale might reduce the incidence of inflated metacognitive awareness scores and allow for a more fine-tuned assessment of growth over time.

These study results have important implications for practice, suggesting that instructors: encourage and promote metacognitive interventions heavily after the first exam. McGuire (2015) showed the utility of this practice, one that is further supported by the predictive power of the first exam on final grade as shown in our data. incentivize exam wrappers with extra points or require them as an assignment to enhance student participation. First-year students especially need additional enticement to undertake this reflection when their performance on the first exam does not meet expectations. Higher-performing students tend to take advantage of the intervention more often. focus exam wrappers particularly on those planning and evaluation practices on the MAI that students typically use least often, but that are essential to developing self-regulation.

The effect of exam wrappers is seemingly dose-dependent, so institutions would be wise to encourage instructors to use them across the range of introductory STEM courses to support first-year students’ achievement. Continuing the practice into upper-level courses has payoffs as well. An online survey with automated feedback, as described in this study, requires only a small investment of instructor time, but can yield real dividends for students.

Acknowledgments

Research reported here was supported in part by the National Institute of General Medical Sciences of the National Institutes of Health under Award Numbers TL4GM118989, UL1GM118988, and RL5GM118987. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Linda C. Hodges (lhodges@umbc.edu) is the associate vice provost for faculty affairs and director of the Faculty Development Center, Lisa C. Beall coordinates learning assessment and program evaluation in the Academic Engagement and Transition Programs, Eric C. Anderson is senior lecturer in the Department of Physics, Tara S. Carpenter is senior lecturer in the Department of Chemistry and Biochemistry, Lili Cui is senior lecturer in the Department of Physics, Elizabeth Feeser is senior lecturer in the Department of Biological Sciences, Tiffany Gierasch is principal lecturer in the Department of Chemistry and Biochemistry, Kalman M. Nanes is senior lecturer in the Department of Mathematics and Statistics, H. Mark Perks is senior lecturer in the Department of Chemistry and Biochemistry, and Cynthia Wagner is senior lecturer in the Department of Biological Sciences, all at the University of Maryland, Baltimore County in Baltimore, Maryland.