The Maryland Out of School Time (MOST) Network has worked since 2013 to build systems that strengthen out-of-school time (OST) STEM education. With support from STEMNext (formerly the Noyce Foundation), the C.S. Mott Foundation, and H.J. Weinberg Foundation, MOST developed the sySTEM learning community, a statewide cohort of schools, afterschool programs, and community sites recruited to increase their capacity to offer best practice OST STEM curricula. The Noyce Foundation and STEMNext have explicitly funded initiatives that connect informal learning experiences in communities that build systems of STEM education throughout the learning continuum including supporting more than 30 statewide afterschool network STEM system building including MOST.

STEMbassadors: A professional learning community

The intent of MOST’s STEMbassador learning community, a program within the sySTEM learning community, is to support the implementation of high-quality, relevant, impactful, and connected learning experiences that engage and excite young people. Afterschool and summer learning programs serve as a critical space for STEM learning opportunities, since K–12 students spend over 80% of their waking hours outside of the classroom. As the image below from the Learning in Informal and Formal Environments (LIFE) Center illustrates, the formal learning environment occupies less than 20% of a learner’s time from kindergarten to adulthood (Stevens, Bransford, and Stevens 2005). Students are in need of meaningful activity and engagement in the out-of-school-time hours that supports their growth and development.

High-quality STEM content fits perfectly into the OST environment, with its focus on hands-on learning, inquiry, team work, and experimentation. Students can have flexible time to complete projects and engage in experimentation without the pressure of grades and assessment.

In the early design stages of building the sySTEM learning community, MOST conducted a statewide landscape survey to get a better picture of STEM in OST programs. While 72% of respondents (n = 111), representing 84 programs, reported offering one or more STEM learning opportunities in their programs at the time of the survey, providers expressed the need for additional training, support, and help identifying appropriate curricula. The top three barriers to implementing STEM in OST programs identified from the survey results were funding; staff confidence in implementing STEM; and awareness of STEM resources, partners, and curricula.

MOST created the STEMbassador initiative to directly address these barriers. In its first year (2013–2014), MOST recruited 38 sites from 19 organizations throughout Maryland through a request for proposals (RFP). The RFP sought both sites that were already implementing STEM but were looking to strengthen their programming, and sites with no STEM experience that were willing to dedicate a minimum of an hour and half per week of their programming to STEM. An additional priority was statewide geographic representation among the selected programs. As part of the opening orientation session, MOST hosted a matchmaking session between the afterschool program providers in the learning community and STEM content specialists that provide curriculum and professional development opportunities. The STEM content specialists were selected by assessing their best practices, including incorporation of project research and evaluation, use of engineering and design, and use of hands-on and project-based learning techniques. Each organization selected a curriculum that it wished to incorporate in the program year. The following partners and curricula were used during the first cohort:

• Code in the Schools
• University of Maryland Extension/4-H (Growing Health Habits, Junk Drawer Robotics)
• Campfire
• Afterschool Math Plus
• Let’s GO Boys & Girls
• NPASS2
• NASA’s BEST
• Maryland Science Olympiad

MOST organized group and individualized training for each selected curriculum, ensuring that times and locations were convenient for both parties and supportive to the programs’ needs. MOST covered all training costs, including materials, and when necessary assisted in acquiring materials and building kits for program sites; however, staff were not paid to attend the training. Training costs and training dosage varied significantly between the models chosen by the sites. Average training time for sites was one full day. MOST also facilitated ongoing communication between STEM curricula providers and organizations when additional support and technical assistance was requested.

Data collection: Student engagement and program quality

In addition to coordinating professional development, MOST supported the sySTEM learning community by gauging student interest in STEM using the Common Instrument (CI) survey developed by the Project in Education and Resilience (PEAR) at Harvard University, as well as measuring program quality utilizing PEAR’s Dimensions of Success (DoS) observation tool. The CI was offered as a pre- and postsurvey that captured student demographic information and STEM interest using a Likert scale. Students were asked to rate their agreement with statements such as, “I get excited to find out that I will be doing a science activity” or “I take science only because it will help me in the future,” on a scale from “strongly disagree” to “strongly agree.” These surveys were MOST’s first foray into collecting student-level data as part of program evaluation, and they encountered several challenges common to the field in collecting, coding, and analyzing the pre- and posttests. To control data quality, MOST staff attempted to distribute and collect paper surveys in person at as many STEMbassador program sites as possible. Weather, specifically snow, became a major complicating factor during winter. As an alternative, surveys were mailed to the sites with instructions for program staff to distribute them, including an identifying code for each student. Program staff were generally less successful at matching pre-post surveys than MOST staff. In total, 409 surveys were collected (285 pretest and 126 posttest). 113 students took both the pretest and posttest.

At first glance, the results of the aggregate data suggested little change, with the possibility of a ceiling effect due to the sample reporting overall very high levels of science engagement at the beginning of their programs. Within specific groups, however, there were statistically significant changes in both positive and negative directions. Students that initially reported low levels of engagement and excitement showed a significant increase in interest. On the other hand, students that started at the highest level of engagement reported decreased interest, possibly demonstrating fatigue of exposure. Students at the median level of engagement stayed the same.

MOST staff also performed DoS observations of sySTEM sites to measure program quality. The DoS tool allows for discrete observations of 12 dimensions of STEM programming organized into four domains: Features of the Learning Environment, Activity Engagement, STEM Knowledge and Practices, and Youth Development in STEM.

Programs that received higher ratings on the DoS observations also showed an observable trend toward more positive change on the CI surveys compared to programs that scored lower. Overall, programs in the learning community scored high on the Features of the Learning Environment and Youth Development domains and lower on the Activity Engagement and STEM Knowledge and Practices Domains. The post-learning community survey of participating facilitators reflected that nearly 50% of members had not attended any previous STEM trainings, less than 40% of members had a four-year college degree, and of those that did, over 80% had not majored in a STEM discipline.

Continuous program quality improvement

The data collected in the first year contributed to program design improvements that were implemented in subsequent cohorts, including

• strengthening our data collection practices,
• prioritizing programs that serve students that have less prior access and engagement with STEM, and
• intensifying the amount of training and technical assistance provided to the STEMbassadors.

Additionally, during the first year of the learning community, MOST staff observed that the program sites that were most enthusiastic and receptive to the demands of participation were the ones that had designated a single liaison to MOST. As a result, in year 2 (2014–2015), the MOST Network restructured the sySTEM learning community to focus on individual leaders at STEM-focused afterschool and summer programs, ideally across multiple sites. In this way, the OST STEM program leadership would be able to diffuse resources—curriculum training, materials, and quality observations—to more than one site, and more students across Maryland would benefit from the learning community.

In 2014 MOST released a Request for Nominations to recruit new STEMbassadors for the second iteration of the learning community. The application process focused on finding coordinator-level staff who had served at least one full year with their organizations. Additional preference was given to organizations serving students traditionally underserved and unrepresented within STEM fields (students of color, young women, rural students, students with disabilities, and English language learners). No prior STEM experience was required. Eleven STEMbassadors were selected, with some continuing from the first year cohort, including the Boys and Girls Clubs of Harford County, the National Electronics Museum, Maryland Science Olympiad, Elev8, the Y of Central Maryland, Caroline County Recreation and Parks, Gateways to Success Learning Center, and the Wicomico County Public Libraries. By focusing on multisite leadership rather than individual organizations, MOST expanded the sySTEM reach to over 50 sites total in 2014–2015.

The opening orientation at the Baltimore City Robotics Center in September gave STEMbassadors an overview of the scope and schedule of activities for the year, as well as time to network and connect with fellow STEMbassadors. The selected STEMbassadors were supported in choosing and implementing STEM curricula, including many of the same curricular partners from the 2013–2014 cohort. MOST offered a menu of trainings instead of matchmaking for this cohort and STEMbassadors had the option to participate in as many trainings as they liked, with the requirement to attend at least two STEM curricular focused training throughout the year. All trainings were provided free of charge and each STEMbassador received a $500 stipend for materials.

MOST also offered several general youth development training opportunities not specifically related to STEM content as optional additional training experiences. Monthly check-in calls provided the STEMbassadors with the opportunity to debrief and share their experiences at trainings, as well as how they were applying training experiences in their programs and troubleshooting challenges. MOST augmented the check-in calls with activities from Click2Science, an online professional development resource focused on building STEM facilitation skills specifically for the OST setting. The STEMbassadors worked directly with Click2Science staff to develop and facilitate a Click2Science workshop at MOST’s annual statewide conference to share learning with the broader field. An overview of the complete 2014–2015 STEMbassador year of activities is included below.

STEMbassadors were also offered three opportunities at the beginning of the year to complete the two-day DoS training. The DoS training helps build skills in objective observation using the domains and dimensions as a framework for organizing the data collected from observation to provide detailed, specific, and practical feedback to a facilitator on the strengths of their approach and ways to further improve facilitation. A detailed rubric guides staff in learning to understand how to observe STEM in OST time settings. Video calibration, discussion, and scoring practice help the participants build reliability in their usage of the tool. Using the tool reliably means that observers are consistent across observations in their scoring and that their scores are consistent with other trained and certified observers. Certification is completed after submission of the scoring and scoring justification for two observations completed after training.

MOST examined postsurvey response data from STEMbassadors at the end of the second program year. Eighty percent of respondents indicated that the DoS training was the most helpful element of the program year. Feedback included “[DoS] has given me great insight into how I can create better STEM programming for students,” and “[DoS] was essential to improving the quality of our programs.” The DoS framework is also available as a planning tool, and in-depth familiarity with the dimensions helps programs design effective programming and intentionally incorporate best practices in advance of implementation.

The power of partnership: Digital Harbor Foundation

During the 2014–2015 year, the Digital Harbor Foundation (DHF) was added as a new curricular partner. DHF is a local and national leader in Maker Education, with a specialization in 3D printing and making through technology. One appeal of the Digital Harbor Center for Excellence trainings was their focus on building facilitators’ skills so that they could design and implement curricula that worked specifically for their program or classroom. The current catalog of trainings available to STEMbassadors includes Making for Educators, 3D Printing for Educators, Interactive Games for Educators, Arduino for Educators, and Internet of Things (IoT) for Educators. Each training includes two to three days of hands-on applied practice, where participants build skills to develop unique, tailor-made projects. In addition to participants receiving relevant materials to take back to their spaces (including the 3D printer), an online portal allows for ongoing access to resources, templates, and requests for advice and guidance.

For the year 3 (2015–2016) program, MOST recruited 22 STEMbassadors from eight counties that served nearly 5,000 youth. MOST partnered with DHF to develop a STEMbassador schedule that included training in 3D Printing, Interactive Gaming, and Making. As in previous years, STEMbassadors would also become certified in the DoS observation tool and leverage their site-based data through CI surveying, but the DHF trainings were now the cornerstone of the year. Having a single curriculum training provider streamlined the logistics of the learning community, allowing both MOST and the STEMbassadors to simplify the schedule of trainings, contacts, and resources to request additional assistance and support. The common training schedule also offered additional opportunities for the STEMbassadors to network with each other during their sessions at the Digital Harbor Tech Center in Baltimore. STEMbassadors who were located more than an hour away from Baltimore were offered hotel accommodations in addition to mileage reimbursement. Using a single curriculum provider has helped stabilize the cost model, as well as providing more equal training time and quality, which in turn has allowed for a more even assessment of impact on providers and programs.

Since 2015 MOST has recruited over 60 STEMbassadors representing over 100 sites in 16 Maryland counties. Our STEMbassadors have completed training in 3D printing, interactive games, and Arduino, with the expectation that they will pass on their knowledge to their sites. The results? In the words of one STEMbassador: “Usually I’m sprinting out of training as soon as it is finished, but this is the only one in which I’m actually still here working after the class is over!” The DHF trainings have proven to have greatly enhanced STEMbassadors’ knowhow and confidence to introduce exciting hands-on curricula such as 3D printing. “Outside of making new contacts with people who have unique ideas for OST programs, it also provides a sounding board and idea generator for people who have ideas but are struggling for resources, or can’t seem to overcome barriers they are facing,” another STEMbassador commented.

In the spring of 2016, MOST’s STEMbassadors participated in an 11-state study coordinated by PEAR and Texas Tech University’s Institute for Measurement, Methodology, Analysis & Policy (IMAPP). The study examined OST STEM learning, and how quality OST programs correlate to in-school success and career interest. In Maryland, 15 programs—the majority of which were STEMbassador sites—participated, and the results were encouraging: Maryland finished in the top three states for STEM Interest, STEM Identity, STEM Career Knowledge, and STEM Career Interest.

As a result of the DHF trainings, afterschool activities led by or hosted at the sites of STEMbassadors showed DoS ratings higher than the national average in nearly every domain with the three domains of Materials, Purposeful Activities, and STEM Content Learning ranging from a half point to an entire point (on a four-point scale) higher than the rest of the United States. These results suggesting that participating in the STEMBassador learning community may lead to higher quality OST STEM programming.

For our youth participants, the student-level data has been similarly promising. According to the 2016 STEM Ready national survey, students in OST programs led by or hosted at STEMbassador sites showed a marked increase in science interest and science-related identity. The survey, administered retrospectively, asked students to reflect on how participating in the program impacted different aspects of their STEM engagement. Of those surveyed, 71% responded that they were more excited about science and innovation, 77% reported higher science career knowledge, and 73% expressed greater interest in STEM careers. As such, initiatives like the sySTEM learning community are further evidence that the OST space is crucial to developing youth STEM identity and exploration. Students who participate in high-quality OST STEM programs are able to explore content and couple that learning with relevant career experiences, which, due to time and curricular obligations, may not be possible in the formal classroom setting.

MOST continues to operate in the design of the STEMbassadors program. During the 2017–2018 school year, MOST supported its first all-female cohort of STEMbassadors. MOST will be facilitating Techbridge trainings with our STEMbassador learning community. Techbridge is an organization that expands opportunities in STEM for girls and students from underrepresented communities. MOST recently played host and provided logistical support for a three-day Techbridge Train the Trainer Program offered to other statewide afterschool networks working on STEM systems building. The training provided the tools and resources for STEM educators to increase the quality, relevance, and culturally responsive nature of their programs, with emphasis on relationship development and positive youth engagement. The Techbridge resources, which focus on responsive and inclusive facilitation, complement the STEM content and approach of the DHF trainings.

Navigating sustainability

Sustainability of the STEMbassador program is another key issue. MOST is working to diversify funding support for the program, which costs close to $4,000 per STEMbassador for the year, through both grants and sponsorship. The $4,000 includes the cost of training and materials subcontracted to DHF, compensation for MOST staff time for coordination, and completion of the external observation by certified DoS observers. So far, two members of the current cohort have been underwritten by local community foundations that invested to ensure that their counties were represented within the learning community. With the encouraging data and continued refinement and innovation of the model, MOST looks forward to requesting nominations for our sixth cohort for 2018–2019. Recognizing the importance of informal STEM opportunities and how they connect to formal STEM offerings and industry, cities and regions across the country are developing STEM ecosystems. The coordination of the sySTEM learning community was an ideal entry into the design of a focused, citywide, ecosystem initiative for Baltimore as part of the national Communities of Practice cohort.

In 2016 MOST launched BmoreSTEM to connect the city’s nonprofit STEM programs, philanthropy that supports STEM, school district administrators, and STEM industry. Baltimore City Public Schools is now leading the Baltimore STEM Ecosystem initiative, in which MOST represents the OST STEM community and engages the school district in enhancing and expanding STEM learning in the classroom.

Through MOST’s role as a statewide AmeriCorps VISTA (Volunteers in Service to America) intermediary, MOST uses both VISTA Summer Associates and year-round VISTA Projects to support afterschool and summer programs to implement high-quality STEM learning to address barriers to opportunity created by poverty. During the summer of 2018, MOST placed 30 VISTA Summer Associates with summer programs offering STEM learning across the city of Baltimore for eight weeks. Summer Associates also helped support data collection with the CI and DoS tools as part of an updated national data collection effort, in collaboration with PEAR. MOST will support 14 year-round STEM VISTA projects, several of which are placed within our STEMbassador alumni network. For example, beginning in September 2018, MOST will sponsor a three-year VISTA project with STEMbassador alumni Anya and Vernon Myers of KidSTREAM, focused on increasing access to STEM learning in rural Charles County, Maryland.

Youth engagement in the STEM disciplines requires expanded and connected opportunities from a variety of stakeholders throughout the year. MOST is excited to continue supporting future STEMbassadors from across Maryland and, more locally, supporting a connected learning ecosystem in the city of Baltimore. By improving the quality and accessibility of STEM programming for youth in Maryland, MOST strives to open doors and create new pathways for students to learn and excel.

Ellie Mitchell (emitchell@mostnetwork.org) is director of the Maryland Out of School Time Network in Baltimore, Maryland. Paul Mincarelli (paul.mincarelli@baltimorecity.gov) is an administrative policy analyst at Baltimore Police Department in Baltimore, Maryland. Elyse Preston (epreston@mostnetwork.org) is associate director of the Maryland Out of School Time Network in Baltimore, Maryland.