In 2013 The Franklin Institute, a leading science museum in Philadelphia, Pennsylvania, received a grant from the National Science Foundation (#1138911) to reach under-resourced urban communities by strengthening the capacity of afterschool centers to offer STEM programming. Integrating Science Into Afterschool: A Three-Dimensional Approach to Engaging Underserved Populations In Science, or “STEM 3D,” was a five-year project that aspired to integrate STEM learning across afterschool, home, and community as a strategy for building positive science identities and creating viable STEM pathways for Philadelphia children who might otherwise be disconnected from the city’s STEM-rich institutions and opportunities. The initiative had three major goals: first, to increase engagement in hands-on, inquiry-based science projects among upper elementary–age youth; second, to cultivate intergenerational support for science learning; and third, to evaluate the effectiveness of this 3D (afterschool, home, and community) approach in engaging children, families, afterschool educators, and community-based organizations in science learning and the project’s promotion of STEM professions. This article focuses on the professional development for implementing STEM activities that emerged as a result of the STEM 3D project and examines its impacts on educators and their students. STEM 3D worked on building educators’ skills and confidence in developing and implementing high-quality project-based STEM units that can help children see STEM learning as something embedded in their homes and neighborhoods. The article will discuss STEM 3D’s professional development approach for facilitating afterschool, project-based learning in STEM. An additional focus of this article is how STEM 3D ultimately shaped the strategies that the Franklin Institute now incorporates into its professional development initiatives with both formal and informal educators.

This article is collaboratively written by the STEM 3D team (Clark, Cox, and McCreedy [PI]) and the external evaluator (Blanc). Additional information about STEM 3D and the STEM 3D evaluation is available online at InformalScience.org and Creative-Evaluations.com.

The evolution of STEM 3D

The city of Philadelphia provides funding to over 200 distinct afterschool and summer programs that are operated by 75 different community-based providers around the city. At the time STEM 3D began, all programs were required to implement project-based learning (PBL) units. PBL presents students with “real-world, multidisciplinary problems that demand critical thinking, real-world engagement, and collaboration” (Schwalm and Tylek 2012). The citywide commitment to utilizing project-based learning, along with the city’s interest in afterschool activities involving science and science careers, seemed to be a natural fit with the inquiry-based pedagogy and curricula long provided by the Franklin Institute. With this philosophy as the backdrop, STEM 3D was launched.

STEM 3D’s original plan was to partner with a selection of out-of-school time (OST) sites between June 2013 and May 2015 to provide intensive professional development, technical assistance, and resources to their afterschool educators. After implementation began, the project extended an additional two years and provided professional development to afterschool educators from about 50 sites across Philadelphia. The inclusion of two additional cohorts was a fortunate but unforeseen consequence of other changes in the project that allowed for the professional development framework to be refined over time and implemented using different structures.

Cohort I consisted of four afterschool partners that were selected through an application and interview process. They participated in ongoing intensive professional development sessions and onsite support, including a minimum of four full-day professional development sessions per year for three years, and they continued to collaborate in a variety of ways with STEM 3D staff after this time. Cohort II, launched in 2015, consisted of one lead partner organization with five separate OST program sites. The most important elements of Cohort I’s professional development were refined and distilled for Cohort II, which received six sessions of half-day professional development over the course of six months. Cohort III represented 40 OST sites that participated in a one-day, six-hour professional development session that was held in 2017. This allowed professional development staff to offer the STEM 3D professional development in a condensed form, testing its usefulness in a format commonly used by the Philadelphia Health Management Corporation, which is the OST funding conduit for all programs that participated in STEM 3D.

Professional development activities were designed to increase afterschool educators’ ability to facilitate student learning about STEM content, STEM process skills, and STEM-related careers within the context of project-based learning units. Participants from all cohorts received bins containing complete sets of materials needed to implement each STEM PBL unit with their children. For example, they received a variety of batteries, wires, lights, and buzzers for a unit on electricity. The first two cohorts also received ongoing technical assistance from the STEM 3D staff. While all groups were expected to implement STEM PBL units in their programs, Cohort I and II participants were also expected to engage families in activities such as STEM open houses and family science nights. Eight out of nine sites from Cohorts I and II implemented family engagement events that were tailored to match the needs and configuration of each site. Similar to STEM 3D’s evolving approach to PBL units, the approach to family and community engagement shifted as the staff at The Franklin Institute learned about the capacities, needs, and goals of a range of OST sites (Blanc 2017). In addition, the professional development programs for Cohort I and Cohort II partners addressed barriers to family engagement and offered strategies for turnkey trainings on STEM activities. The turnkey training approach provided afterschool staff members with new skills, which those individuals then passed on to colleagues who did not attend the initial sessions. This enabled partners to scale up STEM PBL implementation at their sites by involving center and site staff who were unable to attend sessions at The Franklin Institute.

Over the course of the project, more than 70 OST facilitators, coordinators, directors, and other staff members participated in professional development sessions at The Franklin Institute. In addition, at least 50 OST staff participated in turnkey training at their program sites (Figure 1). During the same period, a total of 1,200 children in upper elementary grades participated in collaboratively developed STEM 3D activities at their afterschool and summer camp sites, and dozens more led activities for other students and families though public-facing events such as the Philadelphia Science Festival.

Approach to professional development

The Franklin Institute has a long, rich history of offering professional development to a wide range of educators, bringing inquiry-based approaches to STEM education into both informal and formal arenas. STEM 3D drew on this past experience, implementing evidence-based strategies as well as creating new ones to meet the needs of their afterschool audience. From the project’s inception in 2013, the Institute created a professional development approach that aimed to:

• emphasize a team-based approach to afterschool;
• provide participants with opportunities for hands-on STEM exploration;
• focus on project-based STEM learning units to be implemented following each round of professional development;
• integrate STEM themes that were relevant to students’ daily lives;
• highlight diversity in gender, race, ethnicity, and disciplines of STEM professionals;
• support afterschool staff in exploring STEM career connections with upper elementary school children; and
• build the capacity of sites to integrate and expand STEM learning beyond the period of NSF support.

During the first year of STEM 3D professional development, Cohort I participants explored science process skills, science content, and real-world applications. Observations of subsequent implementation of these units were conducted using the Dimensions of Success Observation Tool developed by Partnerships in Education and Resilience (PEAR). Observations at each partner site showed that youth were actively engaged in open-ended STEM activities during site visits, and site staff reported that youth looked forward to the STEM 3D PBL time. In addition, in discussions held during professional development sessions, several facilitators reported having very successful experiences with framing open-ended activities and encouraging deep exploration. For example, in discussing a unit about how to test designs, one facilitator noted that his students were excited when he asked them to create and test “water blasters.” Others reflected that their students were interested in experiments that included shooting balloons across the room. Despite these successes, project staff and the evaluator also noted implementation challenges, such as a hesitancy on the part of facilitators to let children explore materials or difficulties in helping children make conceptual connections between discrete activities. Facilitators sometimes noted that activities they anticipated to be engaging held little interest for youth and thus ended much more quickly than intended. For example, several facilitators commented that their children had learned about electricity in school, and that it was easy for their students to make bulbs light up. In contrast, in the same classrooms, museum educators asked children questions that encouraged exploration of electrical circuits such as, “How do you think we can make this circuit longer?” or “Let’s see what materials can act as a conductor.” This modeling by museum staff helped facilitators see that such challenges could be addressed with questions or additional challenges.

A critical shift occurred in the second year of project implementation. After a year in which participants were excited about the initiative, but did not implement activities as expected, the STEM 3D staff members took a step back to reconceptualize their approach. One key professional activity in the fall of 2014 was a dialogue about two different interpretations of the term “project-based learning (PBL) unit.” Through this dialogue, STEM 3D staff became aware of discrepancies between their own use of the term “PBL” and OST staff participants’ use of the term. During this dialogue about different approaches to PBL, afterschool educators said that they valued the STEM 3D approach to PBL and were especially motivated by the ways these PBL units pushed their students to higher levels of engagement and thinking. Yet these facilitators also said that there were some things that were more effective in their non-STEM 3D PBL units. While implementing non-STEM 3D PBL units during the first year of the project, these facilitators found it easier to connect content with students’ lives and to identify more options for student voice and choice.

This dialogue about different types of project-based learning began to open up the space for the afterschool educators to share their own knowledge and expertise about students and about pedagogical approaches. During year 1, staff excitement about STEM activities had been palpable among both afterschool educators and museum educators, but the learning and teaching had been unidirectional. The strengths and existing knowledge base of afterschool educators had not yet been visible in the project. In year 2, there was an important shift in dynamics that continued throughout the rest of the project and with subsequent cohorts. Members of each professional grouping (afterschool facilitators and Franklin Institute museum educators) began to display role fluidity, switching from being the expert to being the novice and vice versa.

In the fall of 2014, the Institute’s professional development staff began to distill STEM PBL curricula into three implementation phases as a framework for building pedagogical skills needed for STEM facilitation among OST staff. This approach was identified as very helpful by afterschool educators in their assessments of professional development activities conducted during this period. The three phases were:

1. Introductory activities (creating the spark),
2. Exploratory activities (questioning and building knowledge), and
3. Culminating events or projects that provided connections to community issues and real-world STEM careers.

The newly streamlined approach was reflected in the layout of activities provided to the sites. For example, the PBL unit entitled Circuit City included introductory activities that sparked interest and curiosity in electricity by allowing participants to engage with an energy ball (a device that lights up when you close a circuit using your body) or by going for a walk in the neighborhood to observe how electricity gets to homes. These activities were designed to simply introduce the concept of electricity through personally experiencing the phenomenon. Exploratory activities allowed students to dive deeper into the concepts of open, closed, and parallel versus series circuits through building simple power grids and switches that controlled the flow of electricity. Each activity was open-ended and allowed students to investigate the concept through exploration and play. All of these activities led to the culminating project: creating a model city powered by electricity. Here, students were able to apply the knowledge and skills gained through both the introductory and exploratory activities to a youth-driven design.

Over time, the pedagogical approach incorporated specific techniques designed to help afterschool educators gain the skills and confidence to implement the three-phase framework.

A particularly important aspect of helping afterschool facilitators become successful was to be explicit and specific about the pedagogy museum educators were using to guide the STEM learning experiences of the afterschool educators themselves. For example, in one professional development session observed by the evaluator, a museum educator led an activity with the “energy ball” mentioned above. She guided the participants through a series of activities: looking at the energy ball, talking about the energy ball, forming a circuit with the participants and the energy ball, and experimenting with actions that would make the ball light up. During the course of these activities, the participants shared their ideas about what was happening and why a circuit was or was not created at various times during the activity.

Following this series of activities, the museum educator opened up a discussion of the pedagogy underlying the experience of the participants. She asked questions such as, “What did you notice about this exploration?” and “Did you hear any statements?” She also explicitly named the steps that she had taken, such as, “I gave you some vocabulary. I let you figure it out, and I helped you reflect.” This segment of the workshop ended with a concrete discussion about how this activity could be brought into the classroom. The museum educator challenged the facilitators: “What can you do if kids don’t get it right away?” She also offered suggestions such as, “My favorite question is, ‘What do you notice?’”

Modeling activities with educators as learners so that they can first experience from the perspective of their youth is a particularly effective professional development strategy. The modeling activities are followed by shared reflection on their experiences with a discussion of the facilitation skills and problem-solving strategies used, with a final focus on how they would apply these techniques at their own sites. Signaling to facilitators when they were “switching hats” from learners to educators was particularly important, in order to emphasize key messages about pedagogical strategies as they are enacted by facilitators and experienced by learners. This approach is still used consistently in professional development sessions offered by the museum today.

Impacts on afterschool educators

Participating facilitators reported meaningful changes, including increases in their ability to facilitate project-based STEM activities, improvements in their attitudes toward science, and shifts in their understanding of who can pursue a career in STEM. Final interviews about PBL implementation were conducted with nine Cohort I staff and five Cohort II staff. These were semistructured interviews with open-ended questions about participants’ views of science and their experiences with STEM 3D.

Thematic coding of participant responses indicated that all educators who were interviewed in the final phase of the STEM 3D evaluation reported that their facilitation of activities changed in the following ways:

• Greater use of open-ended problem solving and collaborative learning approaches in STEM
• More youth-driven approaches
• More depth and breadth of STEM content

Cohorts I and II

Facilitators and program coordinators in Cohort I and Cohort II reported radical changes in their attitudes about science. The adults interviewed were asked about their attitudes toward science and whether these attitudes had changed over time. Many interviewees had been interested in some aspect of science as children, but as adolescents were faced with a variety of barriers in following through on these interests. Whether or not the facilitators had ever been excited about science earlier in their lives, they reported that through STEM 3D, they came to embrace science as something that involved fun, curiosity, persistence, risk-taking, teamwork, and testing and retesting, rather than as an out-of-reach discipline not accessible to people like themselves.

The following quotes from Cohort I participants are examples of how facilitators talked about the impacts of STEM 3D on their classroom practice:

• “The project…sparked a little more creativity and new ideas that influence the PBLs I’m doing. I want to keep working on using questioning to help the kids learn – ask the right questions so that the kids’ investigations will have to move forward.”
• “In the beginning, I was clueless. I thought that the final project was just something to do. It was something cute for the parents, but it didn’t have a purpose. But it became a lot more important as we developed it. Now we are trying to build a village and light it up.”
• “As a facilitator, I am very naturally a type A personality, so in working with kids, I’m very quick to be like, ‘Oh, just give it to me. I’ll do it myself.’ But since starting STEM 3D, I’ve been able to think as a facilitator and not as a teacher and to kind of let them have their freedom.”

Participants from Cohort II also described changes in their self-confidence, facilitation skills, and attitudes toward science and STEM. One described her leap forward in confidence about taking chances: “[One of the big things was] just boosting my confidence in science, and not being afraid to go into the unknown.” A second participant explained how the Institute’s staff modeled open-ended problem solving and design for her. “When I explained to Miss Tara what I wasn’t confident about, she just made the transition to doing [explorations] easier than I thought it would be. She was a facilitator with us, so it wasn’t like, ‘These are the steps, A, B, C.’ It was more flexible.”

At least one observation was conducted by the evaluator at all Cohort I and Cohort II sites during the implementation of each STEM 3D unit. For Cohort I, it was possible to see change in practice over several years, and by the second year of the project, observations indicated that adults and youth were indeed participating in challenging, creative activities. At three of the four Cohort I sites, facilitators structured introductory, exploratory, and culminating activities as planned. For example, in “tinkerbot” design challenges, which charged students with creating a simple robot, students were provided with a variety of materials and asked to use a small motor to make an object that moved. Youth in these sites came up with diverse and creative tinkerbots, such as a dancing flower and a giraffe with a spinning head. They often struggled constructively with design problems during the process and asked for assistance from adults. Adults were observed to ask open-ended questions as they had been prompted to by the STEM 3D staff, such as, “Why do you want to do that?” and “What do you think will happen if…?” Observations of Cohort II showed similar open-ended explorations and question asking. Similar challenges came up across both cohorts, with some children and adults struggling with time management, and some experiencing pedagogical and content issues. Discussion and reflection about both successes and challenges were an ongoing and important part of professional development sessions for these participants.

Cohort III

The Cohort III one-day professional development training session was offered in the spring of 2017 and included the Circuit City PBL. A number of indicators suggested that this training was a successful model that can meaningfully be scaled up to reach any sites where staff are able to attend an off-site six-hour session.

Fifty-three participants in Cohort III completed surveys that indicated the STEM 3D professional development program increased their confidence in all areas addressed by the session. These areas included: facilitating open-ended questions, integrating science process skills, asking questions in all stages of a STEM activity, testing and retesting, building reflection into STEM projects, keeping youth engaged in STEM projects, helping youth make choices in STEM activities, conveying STEM content, and introducing ideas about STEM careers.

The OST oversight organization cosponsoring this training reported that it was successfully implemented. This model also fit well with the needs of OST sites, many of which had limited days available for professional development.

Impacts on youth

Student surveys demonstrate statistically significant increases in science interest that are consistent with educators’ observations of growth in critical thinking and other skills needed for success in STEM. One Cohort I site and one Cohort II site were unable to distribute student surveys due to program conflicts. At the other sites, lead facilitators collected surveys from STEM 3D students in 4th grade and above who were in attendance the day the surveys were distributed. The Common Instrument created by PEAR was administered to 66 youth in three Cohort I program during the summer of 2015, and to 38 youth in four Cohort II programs during the spring of 2016. This represents 33% of all students who participated in STEM 3D activities during the 2016 program year. Completed data analyses were shared with CR&E. Analysis by PEAR researchers indicated that youth in all of these programs reported feeling significantly more interested in science at the end of the program than they had been before starting. The impact is measured using a pre-post retrospective 10-item scale that provides an overall measure of science interest and engagement.

Cohort I and Cohort II educators’ perceptions of student growth were consistent with student survey data. Educators noticed over time that their students became more comfortable with testing, failing, and asking questions as they moved through different challenges. One summarized how her students changed: “Before [STEM 3D], my kids didn’t really like to ask questions. I would be like, ‘Oh, well, I don’t know how to do this, so can you show me?’ Now I might ask, ‘If we take this apart, how would this work better?’ So I feel like they walked away … knowing that asking questions is okay … learning how to be better critical thinkers [and] to keep pushing themselves scientifically.” Other educators noticed changes in their student’s persistence, cooperation, and motivation to participate in STEM activities. One educator noted the broader effects it had on her students in the school day and beyond: “They are not as quick to give up. They have learned through these projects how to persevere and get over problems.” An educator at another site reinforced this observation: “It’s interesting to see how [the students] speak among themselves. We can watch them talk it out and work it out. When they do get it, they are more excited because they got it on their own.”

The combination of the investment of time, shared learning, and trust modeled through the STEM 3D professional development approach led to an openness when communicating concerns and tensions between museum and afterschool staff, in addition to sharing in discoveries and successes. Throughout the project, there was consistent staff turnover at the afterschool sites, leading to inconsistencies in participation and later, the creation of Cohort II after one site dropped out entirely. This is a common challenge in out-of-school-time programs, where staff turnover occurs at every level and may be frequent. While this was treated as an opportunity to test and refine concepts with new staff, turnover also contributed to challenges in implementing some of the more complicated program elements, including working with families and the communities surrounding the afterschool centers to inspire intergenerational STEM learning and to connect STEM to everyday life. Museum staff learned that the families whose children attended these programs were, in many cases, not connected to afterschool program activities in the way that was expected. Through trying multiple approaches, including training afterschool staff on hosting family STEM nights and creating student-led STEM activities for the Philadelphia Science Festival, the family and community engagement goals were eventually reached but not in the ways initially envisioned. For example, one major plan for family engagement with STEM was to develop simple, open-ended STEM activities that would be completed by children and adults at home. The STEM 3D team soon learned that this type of take-home activity was not part of the culture of afterschool program participation. On the other hand, open house–style family STEM nights were successful, and afterschool classrooms became sites of adult-child engagement with STEM as parents saw their children’s excitement with STEM 3D projects, lingered longer than intended at pickup time, and offered their own creative ideas about problem solving.

Broader impacts

The professional development approach that was refined through this project shaped the strategies now incorporated in The Franklin Institute’s work with educators both in and out of school. This comes at a time when many schools in the Greater Philadelphia area have been transitioning libraries and classroom spaces into maker spaces with the intention of providing students with increased opportunities for critical thinking, collaboration, and creativity. However, school-based educators have lacked the training and guidance to make pedagogical shifts to support student learning within the context of a maker space.

In March 2017 the Institute launched a professional development series for K–12 classroom teachers and school librarians called Demystifying Maker Spaces. This series is built on the lessons learned from STEM 3D, and supports school districts that are starting maker spaces and other less structured, hands-on learning STEM spaces in their schools. The goal of the series is to build understanding and skills aligned with both the “maker movement” and informal STEM learning, which correspond with the PBL format that was central to STEM 3D. The “Demystifying Maker Spaces” professional development series consists of three sessions that bring together teams of teachers, administrators, and librarians from across grades, subjects, and schools within a district. These sessions create a space for participants to experience, practice, and reflect upon hands-on STEM activities, many of which utilize PBL and interactive exhibits. Institute staff guide these educators in understanding the history of maker spaces and characteristics of a maker mindset, ultimately helping them to define their unique vision and approach for cultivating meaningful learning in these informal STEM spaces.

The multisession approach spaced out over several months helps promote a community of practice similar to the ones that developed among afterschool educators in STEM 3D as educators engage in collaborative learning opportunities that take place both at the museum and in their schools. Evidence from The Franklin Institute’s internal evaluation of Demystifying Maker Spaces suggests that teachers found the approach of experiencing activities as learners and from a student’s perspective to be highly effective. Participants described increased confidence in facilitating open-ended STEM activities in their program settings and noted that the series allowed them to reflect on their own practice in a way that they had not in the past. One teacher commented that participating in Demystifying Maker Spaces professional development changed her perspective on what qualified as STEM learning. Another teacher reported that she used to think these were “craft spaces for creativity more than learning,” but after experiencing the professional development program, she believes that a maker space is “the most important learning space a school can have.” These philosophical shifts are similar to those observed during STEM 3D, which enables museum staff to continue to share these ideas in new ways, with new audiences.

Conclusion

With the conclusion of NSF funding, the STEM 3D project officially ended. However, the learning from STEM 3D continues to influence the work of both museum educators at The Franklin Institute and the afterschool partners. At The Franklin Institute, elements of STEM 3D were absorbed into other initiatives, with the activities and professional development methods that were developed over the course of the project being integrated into many activities offered by the museum. By working collaboratively with afterschool educators and students in Philadelphia, museum educators were not only able to influence the STEM attitudes and skills of these partners, but they were able to become learners themselves. Museum staff developed an understanding of how educators can utilize STEM in their programs and how professional development can build their confidence in the classroom. This later enriched the museum educators’ pedagogical framework for influencing the informal learning that takes place in maker spaces in the K–12 setting. This experience continues to permeate The Franklin Institute’s approach to enhancing STEM facilitation for educators in a variety of settings including schools, afterschool programs, camps, libraries, and community centers.

At the conclusion of the project, all of the initial participating sites had grown in their ability to develop and offer project-based learning opportunities to their students. The teachers no longer need the level of support that Franklin Institute staff had been providing. However, today, many of the institutional relationships remain, and the museum continues to work with community partners in new ways. The success of STEM 3D and its approach to professional development demonstrates that collaborations between scientific institutions and afterschool centers can contribute valuable knowledge about pedagogical frameworks that support STEM educators who work inside of schools, as well as those who work in informal settings.

Suzanne Blanc (sblanc@creative-evaluations.com) is president of Creative Research and Evaluation LLC in Philadelphia, Pennsylvania. Tara Cox (tcox@fi.edu) is manager of professional development at The Franklin Institute in Philadelphia, Pennsylvania. Dale McCreedy (mccreedy@fi.edu) is vice president of audience and community engagement at the Discovery Center at Murfree Spring in Murfreesboro, Tennesee. Alexis Clark (aclark@fi.edu) is assistant director of research and evaluation at the The Franklin Institute in Philadelphia, Pennsylvania.