Research to Practice, Practice to Research
Insight From a Successful Collaboration of Formal and Informal STEM Educators
Connected Science Learning November 2016-January 2017 (Volume 1, Issue 2)
By Dan Carpenter and Todd Witt
Sky Ranch Outdoor Education program shows how it collaborates with formal educators to enhance student test scores and achieve its program outcomes.
The Center for the Advancement of Informal Science Education defines informal STEM education as “lifelong learning in science, technology, engineering, and math (STEM) that takes place across a multitude of designed settings and experiences outside of the formal classroom.” The design of an informal experience can vary widely. On one end of the spectrum are free-choice learning experiences, where participants determine what they want to learn, when they want to do it, and how and with whom they want to study. On the opposite end of the spectrum is nonformal learning, which includes any organized educational activity outside the established formal system, whether operating separately or as an important feature of some broader activity that is intended to serve identified learning clienteles and learning objectives. Opinions on intentionality of design differ, with educators championing a variety of strategies on the informal education scale, from accidental learning to task-conscious and self-directed learning, and everything in between (Rogers 2014). However, while free-choice learning is valuable in its own right, it is on the task-conscious and self-directed end of the spectrum that formal and informal organizations have the greatest opportunity to collaborate and work toward achieving their shared goals for learning outcomes.
Informal STEM experiences, such as visits to informal STEM education facilities, can be a constructive design element of a formal classroom strategy and have a lasting impact on learning (Falk and Dierking 1997). However, not all informal experiences are the same, and coordination between informal and formal educators is needed to ensure that learners have the best possible experiences and subsequent outcomes in their informal STEM education experiences (Tal, Alon, and Morag 2014). Recent research indicates that the attitudes and capabilities of informal educators vary widely (Falk and Dierking 2012), and more efforts are needed to develop collaboration between formal and informal educators (Bevan et al. 2010). Additional research shows that collaboration efforts between formal and informal educators range from entire organizations joining forces to create educational programs (Kisiel 2012) to individual formal and informal educators working together on a single project (Weiland and Akerson 2013).
Some studies on collaboration aimed to find evidence that joint STEM education efforts could achieve a certain outcome—for example, enhancing student performance on standardized tests (Robertson 2007). However, those studies largely focused on the development of new collaborative efforts that have yet to mature and demonstrate significant impact on student performance in formal settings. More research is needed to identify established partnerships that positively impact learning outcomes, such as those affecting student scores on standardized tests. Regardless of one’s personal beliefs on standardized testing, learning how to help students improve test scores is an important goal because testing is the currency of school evaluations, and the failure of informal organizations to respect this reality can limit opportunities for collaboration and access to informal STEM education. In our case study–based investigation, we aimed to learn how an established program with a proven track record of enhancing student test scores functions as a collaborative.
The investigation focused on an established, 10-year STEM collaboration effort between an informal STEM organization, Sky Ranch, and multiple schools that participate in the Sky Ranch Outdoor Education program. The collaboration is unique because it has demonstrated positive impact on state standardized test scores. In this article and at Sky Ranch, collaboration is understood to mean “a shared relationship between two or more people intent on accomplishing a common goal. It is important to note that the intent of this article is not to look at test scores and measure the impact of the Sky Ranch program on improving those scores. Rather, we investigate how the culture and processes of an established partnership support the desired program outcomes (i.e., higher student achievement on standardized testing). We hope to provide information that may serve as a resource for individuals and organizations seeking to develop an effective partnership.
Sky Ranch Outdoor Education is an example of a formal/informal collaborative program that has positively impacted student performance on standardized test scores. Located in East Texas, the camp provides three-day, residential outdoor education experiences for fifth graders. Each year, the camp serves approximately 16,000 students from roughly 160 different public and private schools. The program aims to address the science standards of the Texas Essential Knowledge and Skills (TEKS). Each session consists of a series of experiential activities that cover approximately 90% of the TEKS fifth-grade science content standards over three classes, with a fourth class focused on team building and critical thinking. The science standards are addressed across three classes, titled “Forces in Nature” (Earth and physical science), “Expeditions” (terrestrial ecology), and Limnology (aquatic ecology).
Witt_PDFs
Class activities are facilitated by a Sky Ranch instructor. Each activity is designed to be interactive and socially engaging while addressing one or more state standards. Each class is three hours long, with students rotating through six locations on the ranch that each cover a different topic. There are six rotations in the Expeditions class, five in Limnology, and four in Forces in Nature. To accommodate the maximum number of students per session, topics covered at each location in a class are taught as independent units; students do not need to have visited other locations first. The images of the TEKS matrix illustrate the standards covered in each class and where they are addressed in the class’s rotation. The first and second columns are the standards reference number and specific content or skill area, respectively. The columns with the icons (snake, turtle, or rocket) represent the content rotation within the class. For example, on the Limnology matrix, TEKS 3.9a was addressed in three rotations (Turtle Barn, pH Lab, Sky Pond), while TEKS 4.8b was only addressed in the Water Cycle rotation.
The collaboration was of particular interest because the program has been linked to a positive impact on student passing rates for the State of Texas Assessment of Academic Readiness (STAAR), which is Texas’s standardized test for public schools and is based on the TEKS instrumental standards. The teachers are required to teach the content outlined by grade in the TEKS, and the exam questions on the STAAR test are derived from the content in the standards. While this article focuses primarily on the collaborative relationship between Sky Ranch and the schools participating in its program, it is important to have some understanding of the evidence linking the Sky Ranch experience to positive test outcomes, because test performance was a critical qualifier supporting the value of investigating the Sky Ranch program. An external evaluation of the program comparing students from a single Title 1 school indicated a 28% higher passing rate on the STAAR test by students who had attended the Sky Ranch program the week prior to the state test compared to the control group of students who received traditional formal preparation in school. The two groups of students were kept separate from each other just prior to the test so that they did not share knowledge with each other. This strategy limited other competing variables.
Following the single school study, a comparison was conducted of state test averages between the 85 public schools whose fifth graders attended the Sky Ranch program and the remaining public schools whose fifth graders did not attend the program. Upon analysis of the statewide test scores, schools with Sky Ranch participants demonstrated a significantly higher passing rate on the STAAR test than the state average, suggesting that the program’s impact may not be limited to the single school in the intital study. While it could be argued that other underlying factors were at play, the composite of the data from the single school evaluation and statewide comparison are sufficient to suggest that participation in the Sky Ranch program directly enhances student performance on standardized tests.
In an effort to discover the drivers that contributed to Sky Ranch participants’ success on tests, an online survey instrument was used to poll educators from schools participating in the Sky Ranch program. The responses were grouped into eight primary drivers: instructors, standards-based content, curriculum, instruction, facilities, experience, geographical location, and collaboration. Collaboration supported five of the eight drivers, because program design elements related to decisions on instructors, standards-based content, curriculum, instruction, and facilities were all directly impacted and influenced by the collaboration process. Hence, collaboration is a key driving variable in the Sky Ranch program and merits study. This article digs deeper into the Sky Ranch collaborative in an effort to gain a better understanding of how the formal and informal worlds can come together to influence positive outcomes on test scores.
Using a two-stage qualitative case study design, the partnership between Sky Ranch and participating schools was first investigated through observations of informal and formal educator interactions at collaborative meetings. At the end of each school’s trip, participating teachers from schools attending that week meet with Sky Ranch teachers and directors to discuss student and teacher experiences and the possibility of collaborating on future improvements. Observations and field notes were taken during three of these collaborative meetings over a two-week period. We also conducted interviews with the informal education director at Sky Ranch and two teachers from the participating schools. We matched our observations with interviews to help provide a framework of understanding for the informal and formal collaborative partnership and the perspectives of adult participants on the potential impact the experience has on students.
To understand the collaboration, it is best to think of the relationship between the formal and informal institutions in two generalized categories: culture and process. Culture refers to how the participants interact. What are their values? How do they engage each other within the process? Process refers to the formal structures that support the collaborative culture. How do the project leaders organize themselves? How is information collected, analyzed, and processed? We found that culture and process are the driving forces behind successful collaborative relationships.
When considering cultural elements that served as the primary drivers of the collaboration, three cultural themes were apparent.
For a collaboration to be sustainable, all entities involved must have their needs met. In the context of the collaborative partnership between schools and Sky Ranch, each had its own set of needs that were identified. The schools in this study needed a program with a standards-aligned curriculum and experiences that led to increases in state test scores. They also needed students to be in a safe environment with opportunities to get outdoors, as well as learning experiences that provided the most value at the lowest cost. Sky Ranch needed to positively impact students, teach students about nature, and generate financial resources to support the program.
The relationship, and therefore the collaboration, between the schools and Sky Ranch was based on a common appreciation of the needs of both sides and a willingness to work toward solutions that best served the needs of all involved. Subsequently, processes and plans evolved from those needs. Any strategies developed began and ended with a clear understanding of each group’s goals.
The primary goal of the program was to positively impact standardized test scores. While the outdoor education program had been in operation for almost 25 years, the collaboration between Sky Ranch and the schools had only existed for the previous 10 years. At the beginning of the partnership, schools were facing increased pressure to allocate resources (i.e., funding, seat time) to activities that directly improved test scores. The Sky Ranch program did not meet the standards threshold and was experiencing declining attendance, as school administrators discouraged the use of resources for activities that didn’t focus on test preparation. Working within the political realities facing both formal and informal entities required a change in the educational model. One of the educators we interviewed during our study said, “The program was different before the collaboration started. It almost felt like a weekend camp … Not that the classes were bad; they just weren’t geared to the test. It was more of a bonding time than a science time.” This comment reflects the changing national landscape between formal and informal needs over the past two decades. While multiple needs were addressed in the collaborative process, it was very clear that Sky Ranch needed to adapt its program to directly work toward improving student performance on the test.
There was a high degree of trust between Sky Ranch and the school representatives that had developed over time. An ongoing process of open communication and following through with plans had resulted in a trusting relationship between informal and formal educators. Trust was especially critical when working through problems. As stated by one teacher, trust was “everything, because if I don’t trust you and you don’t follow through on what you say, I am probably not going to come back.”
Previous discussion about issues and possible solutions was a unifying theme in conversations. Repeatedly, previous discussions related to the issue at hand were referenced, and those discussions were brought into current conversations linking what was learned in the past, what changes were made, and evaluations of the outcome. The group had a history of productivity, and it reflected an organic relationship between the schools and Sky Ranch that was growing and adapting to the new testing requirements. This is not to say that there were no failures in the collaborative relationship; rather, the positive response of both parties in those failures resulted in trust. The actions of both parties over time helped build the capacity of the group to work collaboratively and achieve their goals.
Both the informal and formal sides actively took responsibility for their roles in the collaborative relationship. Both evaluated their contributions to the problems that arose and took responsibility if warranted. Moreover, both the informal and formal partners actively evaluated their roles in the process and communicated on how each could improve the process. This was observed repeatedly in the teacher meetings when they discussed trip issues. For example, for one group of fifth graders, some of the necessary content did not get covered in one of the classes, and different sections of the class were taught different things. The school that brought up the issue was quick to take responsibility for creating the problem by not getting their students to the class on time. Sky Ranch quickly followed up by taking responsibility for not adapting the content to the reduced class time.
The same type of shared responsibility was evident when developing and enhancing the program. The formal teachers saw themselves as active observers and evaluators of the program and came to the meetings with both positive comments and constructive criticism. They saw themselves as an integral part of the process and were invested in the program. Sky Ranch was also invested in the quality of communication with the schools and held itself accountable for the outcomes of the program as they related to the schools’ needs, especially test outcomes. For example, the Forces of Nature class features an activity where students conduct an experiment that involves timing their schoolteacher traveling down a zip line. The students are given the task of modifying the teacher (e.g., add weight to increase mass, raise knees to reduce resistance, etc.) in a way that increases the teacher’s speed and then slowing the teacher down relative to the control run. The time has to be reported by Sky Ranch staff via radio due to the distance from the top of the zip line tower where students are located to the bottom of the run where the teachers finish the ride. The Sky Ranch staff reported using the term “point” to report the time (e.g., 15 “point” 13 seconds). The formal educators reported in the meeting that students did not understand the “point” reference because they were learning it in school as reported in hundredths. That was also the way it would be seen on the STAAR test. In response, Sky Ranch modified the reporting so that it was expressed in hundredths (e.g., 15 and 13 one-hundredths seconds) over the radio. This is an example of a slight modification to an existing program. A more drastic example is the actual existence of the Forces in Nature class. Prior to the collaboration, the class did not exist. The school expressed a need for more Earth and physical science content because they were experiencing the greatest lack of success in this area on the state test. As a result, a completely new class was created. Sky Ranch developed the class by identifying the Earth and physical science standards in the test and creating a learning experience that best used the ranch’s resources to teach the necessary curriculum.
The collaboration was creative in nature, with both sides looking to find new and innovative approaches to improve the program and meet the goals. The group was not afraid to discuss new ideas and drop the planned agenda to explore a new concept. Evaluation of a current situation or problem was grounded in a focus on the future. Both sides looked ahead at what was coming and were proactive about making changes. They defined tangible goals and generally focused on addressing changes in education strategies and standards. Sky Ranch was willing and committed to investing in change rather than holding on to the past. One teacher recounted her experience: “I just remember they were teaching the recycling class and that was really huge when it used to be TAKS [former Texas standardized test], but it wasn’t when it was STAAR [current Texas standardized test], and they just got rid of that. They changed the curriculum because of those meetings. I think it’s huge. I mean you have to. They are always keeping up with the changes, and who knows better about the changes than the teachers teaching it.”
When considering process elements that served as the primary drivers of the collaboration, three themes were apparent.
There is a singular/composite relationship between Sky Ranch and the schools that attend, meaning that Sky Ranch is the singular informal entity, while the schools are in a composite relationship with Sky Ranch because each school has an individual collaboration with Sky Ranch but form a collection of groups that cycle in and out of the collaboration as students attend the camp sessions. Thus, each school individually interacts with Sky Ranch, but their input is taken into account throughout Sky Ranch’s process for making changes. Changes were not made based on an individual need or issue but by considering the needs of all schools.
When the Sky Ranch administrator, Marquita Stottern, talked about the collaboration, she referred to the schools collectively as “they,” and throughout the conversation, her comments were based on a summary of all the conversations Sky Ranch educators had with schools. For example, when Stottern spoke about how Sky Ranch used discussions from multiple meetings to make changes, she uses they to refer to the collective thoughts of a rotating group of 4 to 20 completely different people twice a week for nine months, the formal educators that Sky Ranch meets with as the schools cycle through the year.
One of the key components for me is the fact that we can talk about the shifts and changes in education, including when the STAAR test came out. That was a big shift and change in education, period, because you went from more rote memorization to having to have it where the kids were educated in a way that they could apply the knowledge that they had learned in the classroom … for instance, right now we are looking at our expeditions class and the way education has shifted is requiring us to reexamine the way we do a few of the sections in that class to make it more student-led instead of instructor-led. And those are the things that they are coming in and saying, ‘Okay, what if you maybe did something like this or this or this?’ And we are looking at it and going, ‘Okay, can we apply that? Can we make that work? And how can we take that idea and then build on it and expand it and make it something that works within the way we operate and the way we teach classes?’
This singular/composite concept is a unique aspect of the collaborative effort between Sky Ranch and the schools it serves. The concept is a hybrid of a traditional collaborative, which is typically a representative subset of a larger population that meets on a regular basis to develop new concepts, and an end-of-experience follow-up review meeting with all schools. The design incorporates both elements. The meetings do provide individual interaction for each group to process its specific trip, but Sky Ranch uses the schools’ composite input over time, using the meetings to maintain a continuous processing environment. This singular entity viewing its role as supporting a composite conversation is what pushes the interaction toward a collaborative concept. The idea is that you don’t have to have the exact same formal educators in the room each meeting to support collaboration. One of the challenges when trying to develop collaboratives is getting people together, especially enough people to represent the population adequately. Sky Ranch offers a unique singular/composite approach to these challenges.
Collaborative meetings were designed to be the primary way formal educators and Sky Ranch communicated. The meetings served as action-oriented think tanks that addressed multiple problems. The conversations cycled through topics such as what happened on the current trip, how one of the Sky Ranch instructors was teaching and how they might improve, troubleshooting something that went wrong in a class, reviewing changes that were made to a class, and upcoming classes. While the conversation was fluid, participants primarily stuck to an established organizational process that supported the collaboration:
The first four items on the agenda would be the Sky Ranch classes. Once the classes were discussed, the conversation would move on to broader topics such as housing, meals, activities, and pretrip planning. The process generally took between one and two hours.
A continuous improvement cycle was evident as participants worked though the process of gathering feedback, analyzing feedback, planning for changes, and implementing changes. Informed decision-making based on discussions at the collaborative meetings resulted in continuing improvement in the Sky Ranch program. Of particular interest was the tenacity of the Sky Ranch representatives when digging for critique. They were not satisfied with the status quo. For example, when teachers weren’t quick to offer criticism or became complacent in the process, Stottern, the Sky Ranch administrator, dug deeper even when the teachers had no criticism. When asked why she needed to dig deeper, she explained that it was in an effort to continue to find areas for improvement even when the relationship between the groups had grown so close. She emphasized that people sometimes were hesitant to bring up an issue and explained, “if they said it was fine, maybe it was fine, but maybe not. I want to take it a bit further and hear more, so I’ll start probing to see if I can’t bring up any other needs. Sometimes fine is fine, but sometimes I’ll find that fine is not necessarily fine, but that they are just being nice and need something more for the children.”
The collaboration, as it was designed, was built on the practice of fostering a continuous and holistic conversation about the needs of teachers and students. The meeting agenda and focus of the conversations went well beyond academic content, extending to the entire experience for students. From the time they began preparing for the trip to the time they arrived back at school, every element of the experience was considered during these conversations because the entire process impacts the experience. One teacher said, “I think the classes are just one piece, but they make sure everything is okay. They are making the parents happy, and it makes a huge impact on everything, because if parents are complaining about things, teachers are the ones that are going to hear it.”
The collaborative partnership considered multiple perspectives on the experience from all educators involved on both the informal and formal sides. The formal school side perspective came from administrators, nonscience teachers, science teachers, counselors, and a school nurse. On the informal side, Sky Ranch’s perspective was sought from staff and administrators. While students and parents did not attend the meetings, both Sky Ranch and the schools made an effort to gather their feedback prior to the meeting and share those perspectives, too.
When gathering feedback from everyone involved, it is inevitable that some of it will conflict. For example, even with standards in place, there are considerable gray areas when it comes to what is to be taught, and often some content specifics are left open to interpretation. For example, in a fish sampling activity, there was conflicting input from the formal educators relative to the use of the term weight versus mass in the fifth-grade standards. Over several months of gathering feedback from the schools that cycled through the program, it became obvious there was no agreement on the issue among the schools. The problem was eventually resolved by looking at previous state tests to see which term was being used. The term mass was used in the past tests, so the issue was resolved until the next year, when the test switched to using weight. At this point, Sky Ranch contacted the state agency responsible for the exam to ask for clarification. The agency was unaware of the problem and made adjustments in their standards and testing to reflect the use of weight. This example illustrates how critical it is to gather multiple perspectives and remain flexible when updating the curriculum. It also illustrates how the standards can be used to resolve conflict.
It is important to realize that if the goal is to increase student performance on test scores, that goal has to be the primary focus of the effort. There was no doubt when observing or speaking to anyone involved in the project that they all knew the state standards were the focal point of the relationship and the driver for everything they did. Testing was the common thread that held them together. This reality did not distract from other secondary goals, such as exposing students to nature, conducting learning experiences in a fun manner, keeping students safe, or any other concern. However, everyone knew that the secondary goals needed to support the primary goal. We would not be surprised to hear concerns from others in the informal STEM community over whether this primary goal detracts from the informal experience; however, we believe that it does not. Quality informal STEM experiences can be delivered while also adhering to established state or national standards. The standards simply serve as common ground for content development between the formal and informal educators. Informal educators do not need to not fear standards or fear that they will lose their identity and mission when working to meet them.
Another takeaway from this study was that both parties involved in the collaboration effort were truly committed. Sky Ranch staff saw themselves as stakeholders in the lives of students, rather than as a vendor of services. They were invested in students, parents, and the formal education community. It was also apparent that the formal teachers knew Sky Ranch staff were committed to student success as they expressed a high degree of respect for them. Robertson (2007) advocates building trust and relationships between collaborators. While these concepts are promoted in his recommendations, there is a high degree of latitude in the level of trust and commitment that can be invested in a relationship. In the collaboration between Sky Ranch and the schools it serves, everyone involved personally owned the success and failure of each student taking the test. This was not a passive relationship. It was obvious that everyone held a stake in student academic success, and it is unlikely that the successes would have come without this level of collective ownership.
There were several key aspects of the collaboration that seemed to support high levels of trust and commitment. First, there was a positive track record and history of interaction between informal and formal partners. People talked, listened, acted, re-evaluated, and repeated the reflective process. Each partner did what they said they were going to do and did it to the best of their ability.
One final conclusion learned from this investigation was directly related to informal educators’ functionality. Robertson (2007) recommends an investment of adequate time to support the effort. Because time is a very limited resource in many informal settings where staffing is tight, this recommendation can seem overwhelming to an informal educator when considering the sheer number of schools and groups any one informal science organization or educator supports in a year. This collaboration provided an interesting model for a singular, yet collective collaboration. While Sky Ranch spent two to three hours a week on teacher meetings (Sky Ranch participates in the meetings twice a week as schools rotate through), each school invested only one to two hours per year (each school meets one time during the week they attend). Rather than focusing on a single school or a small subset of schools, Sky Ranch looked at feedback gathered from all schools to find common threads and identify areas for improvement. They invested resources in areas that would have the greatest impact for as many schools as possible. The role Sky Ranch played did not just impact the learning that took place in the informal setting. If the focus is on improving test scores, then the informal experience is much more likely to be blended into the classroom curriculum. During the collaborative meetings, the formal eduators shared how they were integrating the experience into their daily curriculum to enhance the impact of the informal program. As the Sky Ranch team learned about the ways different schools were integrating the experience into their daily routines, they gathered that information and shared it with other schools as they cycled through the collaboration. Hence, the informal institution became the clearinghouse for information on the incorporation of the program back into the classroom.
Informal science organizations seeking to replicate what has been observed in this study should understand the potential role they play in facilitating exchanges between schools and developing a high-quality program based on the collective feedback from multiple school teachers rather than drilling down on the needs of a few and hoping they scale to the larger population.
As mentioned earlier, this investigation validated recommendations made by Robertson and further added to the body of knowledge on the development of effective collaboration between formal and informal science organizations. The research was especially timely given its focus on a collaborative partnership that centered on improving student performance on test scores.
There are obvious limitations to this investigation that should be noted. First, there are multiple types of informal STEM organizations and each has different constituencies. In the case of this study, the primary focus of the Sky Ranch program was on fifth-grade science, so it was easy to cater to the needs of a narrow section of the student population. While this is a limitation in our case study, it is also an asset because it minimizes the potential variables involved.
It is logical to expect that student performance on standardized tests would be enhanced if there is a coordinated effort to maximize informal resources in the pedagogy of a formal classroom teacher. In fact, constructivist theory dictates the incorporation of such resources for adult learners of their practice. By actively engaging each other in a collaborative process, formal and informal educators are better positioned to design learning experiences that maximize the resources of both partners. For example, in an effort to build upon what was learned at Sky Ranch, the Texas Freshwater Fisheries Center (TFFC) has recently established a working collaboration with several seventh-grade science teachers to develop an experience targeting the seventh-grade standards on genetics. The TFFC is a part of the Texas Parks and Wildlife Department and serves as a working fish hatchery and home to the Texas ShareLunker selective bass breeding program. The TFFC is uniquely positioned to deliver an authentic informal learning experience tied to the standards, if the experience is correctly designed. By collaborating with schools, rather than just having students come in for a tradional hatchery tour, both formal and informal education organizations are bringing their resources to the table to create a standards-focused experience.
The work at Sky Ranch and its collaborative partner schools has provided us with valuable knowledge about how to create a successful collaborative partnership. The results from this work should provide insight and encouragement to those seeking to enhance the success of their students through formal/informal collaborations by creating better STEM learning opportunities that allow them to successfully demonstrate their knowledge on standardized tests.
Dan Carpenter (Daniel.carpenter@ttu.edu) is assistant professor of education in science education at Texas Tech University in Lubbock, Texas. Todd Witt (todd.witt@tpwd.texas.gov) is outreach and education manager at Texas Parks and Wildlife Texas Freshwater Fisheries Center in Athens, Texas.