What Works in Professional Development

I recently read an IES-funded study, called “The Effects of a Principal Professional Development Program Focused on Instructional Leadership.” The study, reported by a research team at Mathematica (Hermann et al., 2019), was a two-year evaluation of a Center for Educational Leadership (CEL) program in which elementary principals received 188 hours of PD, including a 28-hour summer institute at the beginning of the program, quarterly virtual professional learning community sessions in which principals met other principals and CEL coaches, and 50 hours per year of individual coaching in which principals worked with their CEL coaches to set goals, implement strategies, and analyze effects of strategies. Principals helped teachers improve instruction by observing teachers, giving feedback, and selecting curricula; sought to improve their recruitment, management, and retention strategies, held PD sessions for teachers; and focused on setting a school mission, improving school climate, and deploying resources effectively.

A total of 100 low-achieving schools were recruited. Half received the CEL program, and half served as controls. After one, two, and three years, there were no differences between experimental and control schools on standardized measures of student reading or mathematics achievement, no differences on school climate, and no differences on principal or teacher retention.

So what happened? First, it is important to note that previous studies of principal professional development have also found zero (e.g., Jacob et al., 2014) or very small and inconsistent effects (e.g., Nunnery et al., 2011, 2016). Second, numerous studies of certain types of professional development for teachers have also found very small or zero impacts. For example, a review of research on elementary mathematics programs by Pellegrini et al. (2019) identified 12 qualifying studies of professional development for mathematics content and pedagogy. The average effect size was essentially zero (ES=+0.04).

What does work in professional development?

In sharp contrast to these dismal findings, there are many forms of professional development that work very well. For example, in the Pellegrini et al. (2019) mathematics review, professional development designed to teach teachers to use specific instructional processes were very effective, averaging ES=+0.25. These included studies of cooperative learning, classroom management strategies, and individualized instruction. In fact, other than one-to-one and one-to-small group tutoring, no other type of approach was as effective. In a review of research on programs for elementary struggling readers by Inns et al. (2019), programs incorporating cooperative learning had an effect size of +0.29, more effective than any other programs except tutoring. A review of research on secondary reading programs by Baye et al. (2018) found that cooperative learning programs and whole-school models incorporating cooperative learning, along with writing-focused models also incorporating cooperative learning, had larger impacts than anything other than tutoring.

How can it be that professional development on cooperative learning and classroom management are so much more effective than professional development on content, pedagogy, and general teaching strategies?

One reason, I would submit, is that it is very difficult to teach someone to improve practices that they already know how to do. For example, if as an adult you took a course in tennis or golf or sailing or bridge, you probably noticed that you learned very rapidly, retained what you learned, and quickly improved your performance in that new skill. Contrast this with a course on dieting or parenting. The problem with improving your eating or parenting is that you already know very well how to eat, and if you already have kids, you know how to parent. You could probably stand some improvement in these areas, which is why you took the course, but no matter how motivated you are to improve, over time you are likely to fall back on well-established routines, or even bad habits. The same is true of teaching. Early in their careers teachers develop routine ways of performing each of the tasks of teaching: lecturing, planning, praising, dealing with misbehavior, and so on. Teachers know their content and settle into patterns of communicating that content to students. Then one day a professional developer shows up, who watches teachers teaching and gives them advice. The advice might take, but quite often teachers give it a try, run into difficulties, and then settle back into comfortable routines.

Now consider a more specific, concrete set of strategies that are distinctly different from what teachers typically do: cooperative learning. Teachers can readily learn the key components. They put their students in mixed groups of four or five. After an initial lesson, they give students opportunities to work together to make sure that everyone can succeed at the task. Teachers observe and assist students during team practice. They assess student learning, and celebrate student success. Every one of these components is a well-defined, easily learned, and easily observed step. Teachers need training and coaching to succeed at first, but after a while, cooperative learning itself becomes second nature. It helps that almost all kids love to be noisy and engaged, and love to work with each other, so they are rooting for the teacher to succeed. But for most teachers, structured cooperative learning is distinctly different from ordinary teaching, so it is easy to learn and maintain.

As another example, consider classroom management strategies used in many programs. Trainers show teachers how to use Popsicle sticks with kids’ names on them to call on students, so all kids have to pay attention in case they are called. To get students’ immediate attention, teachers may learn to raise their hands and have students raise theirs, or to ring a bell, or to say a phrase like “one, two, three, look at me.” Teachers may learn to give points to groups or individuals who are meeting class expectations. They may learn to give students or groups privileges, such as lining up first to go outside or having the privilege of selecting and leading their favorite team or class cheer. These and many other teacher behaviors are clear, distinct, easily learned, and immediately solve persistent problems of low-level disturbances.

The point is not that these cooperative learning or classroom management strategies are more important than content knowledge or pedagogy. However, they are easily learned, retained, and institutionalized ways of solving critical daily problems of teaching, and they are so well-defined and clear that when they have started working, teachers are likely to hold on to them indefinitely and are unlikely to fall back on other strategies that may be less effective but are already deeply ingrained.

I am not suggesting that only observable, structural classroom reforms such as cooperative learning or classroom management strategies are good uses of professional development resources. All aspects of teaching need successive improvement, of course. But I am using these examples to illustrate why certain types of professional development are very difficult to make effective. It may be that improving the content and pedagogy teachers use day in and day out may require more concrete, specific strategies. I hope developers and researchers will create and successfully evaluate such new approaches, so that teachers can continually improve their effectiveness in all areas. But there are whole categories of professional development that research repeatedly finds are just not working. Researchers and educators need to focus on why this is true, and then design new PD strategies that are less subtle, more observable, and deal more with actual teacher and student behavior.

References

Hermann, M., Clark, M., James-Burdumy, S., Tuttle, C., Kautz, T., Knechtel, V., Dotter, D., Wulsin, C.S., & Deke, J. (2019). The effects of a principal professional development program focused on instructional leadership (NCEE 2020-0002). Washington, DC: Naitonal Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Inns, A., Lake, C., Pellegrini, M., & Slavin, R. (2019). A synthesis of quantitative research on programs for struggling readers in elementary schools. Available at www.bestevidence.org. Manuscript submitted for publication.

Jacob, R., Goddard, K., Miller, R., & Goddard, Y. (2014). Exploring the causal impact of the McREL Balanced Leadership Program on leadership, principal efficacy, instructional climate, educator turnover, and student achievement. Educational Evaluation and Policy Analysis, 52 187-220.

Nunnery, J., Ross, S., Chappel, S., Pribesh, S., & Hoag-Carhart, E. (2011). The impact of the National Institute for School Leadership’s Executive Development Program on school performance trends in Massachusetts: Cohort 2 Results. Norfolk, VA: Center for Educational Partnerships, Old Dominion University.

Nunnery, J., Ross, S., & Reilly, J. (2016). An evaluation of the National Institute for School Leadership: Executive Development Program in Milwaukee Public Schools. Norfolk, VA: Center for Educational Partnerships, Old Dominion University.

Pellegrini, M., Inns, A., Lake, C., & Slavin, R. (2019). Effective programs in elementary mathematics: A best-evidence synthesis. Available at www.bestevidence.com. Manuscript submitted for publication.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Evidence-Based Reform and the Multi-Academy Trust

Recently, I was in England to visit Success for All (SFA) schools there. I saw two of the best SFA schools I’ve ever seen anywhere, Applegarth Primary School in Croyden, south of London, and Houldsworth Primary School in Sussex, southeast of London. Both are very high-poverty schools with histories of poor achievement, violence, and high staff turnover. Applegarth mostly serves the children of African immigrants, and Houldsworth mostly serves White students from very poor homes. Yet I saw every class in each school and in each one, children were highly engaged, excited, and learning like crazy. Both schools were once in the lowest one percent of achievement in England, yet both are now performing at or above national norms.

In my travels, I often see outstanding Success for All schools. However, in this case I learned about an important set of policies that goes beyond Success for All, but could have implications for evidence-based reform more broadly.

Both Applegarth and Houldsworth are in multi-academy trusts (MATs), the STEP Trust and the Unity Trust, respectively. Academies are much like charter schools in the U.S., and multi-academy trusts are organizations that run more than one academy. Academies are far more common in the U.K. than the U.S., constituting 22% of primary (i.e., elementary) schools and 68% of secondary schools. There are 1,170 multi-academy trusts, managing more than 5,000 of Britain’s 32,000 schools, or 16%. Multi-academy trusts can operate within a single local authority (school district) (like Success Academies in New York City) or may operate in many local authorities. Quite commonly, poorly-performing schools in a local authority, or stand-alone academies, may be offered to a successful and capable multi-academy trust, and these hand-overs explain much of the growth in multi-academy trusts in recent years.

What I saw in the STEP and Unity Trusts was something extraordinary. In each case, the exceptional schools I saw were serving as lead schools for the dissemination of Success for All. Staff in these schools had an explicit responsibility to train and mentor future principals, facilitators, and teachers, who spend a year at the lead school learning about SFA and their role in it, and then taking on their roles in a new SFA school elsewhere in the multi-academy trust. Over time, there are multiple lead schools, each of which takes responsibility to mentor new SFA schools other than their own. This cascading dissemination strategy, carried out in close partnership with the national SFA-UK non-profit organization, is likely to produce exceptional implementations.

I’m sure there must be problems with multi-academy trusts that I don’t know about, and in the absence of data on MATs throughout Britain, I would not take a position on them in general. But based on my limited experience with the STEP and Unity Trusts, this policy has particular potential as a means of disseminating very effective forms of programs proven effective in rigorous research.

First, multi-academy trusts have the opportunity and motivation to establish themselves as effective. Ordinary U.S. districts want to do well, of course, but they do not grow (or shrink) because of their success (or lack of it). In contrast, a multi-academy trust in the U.K. is more likely to seek out proven programs and implement them with care and competence, both to increase student success and to establish a “brand” based on their effective use of proven programs. Both STEP and Unity Trusts are building a reputation for succeeding with difficult schools using methods known to be effective. Using cascading professional developing and mentoring from established schools to new ones, a multi-academy trust can build effectiveness and reputation.

Although the schools I saw were using Success for All, any multi-academy trust could use any proven program or programs to create positive outcomes and expand its reach and influence. As other multi-academy trusts see what the pioneers are accomplishing, they may decide to emulate them. One major advantage possessed by multi-academy trusts is that much in contrast to U.S. school districts, especially large, urban ones, multi-academy trusts are likely to remain under consistent leadership for many years. Leaders of multi-academy trusts, and their staff and supporters, are likely to have time to transform practices gradually over time, knowing that they have the stable leadership needed for long-term change.

There is no magic in school governance arrangements, and no guarantee that many multi-academy trusts will use the available opportunities to implement and perfect proven strategies. Yet by their nature, multi-academy trusts have the opportunity to make a substantial difference in the education provided to all students, especially those serving disadvantaged students. I look forward to watching plans unfold in the STEP and Unity Trusts, and to learn more about how the academy movement in the U.K. might provide a path toward widespread and thoughtful use of proven programs, benefiting very large numbers of students. And I’d love to see more U.S. charter networks and traditional school districts use cascading replication to scale up proven, whole-school approaches likely to improve outcomes in disadvantaged schools.

Photo credit: Kindermel [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)]

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Achieving Audacious Goals in Education: Amundson and the Fram

On a recent trip to Norway, I visited the Fram Museum in Oslo. The Fram was Roald Amundson’s ship, used to transport a small crew to the South Pole in 1911. The museum is built around the Fram itself, and visitors can go aboard this amazing ship, surrounded by information and displays about polar exploration. What was most impressive about the Fram is the meticulous attention to detail in every aspect of the expedition. Amundson had undertaken other trips to the polar seas to prepare for his trip, and had carefully studied the experiences of other polar explorers. The ship’s hull was special built to withstand crushing from the shifting of polar ice. He carried many huskies to pull sleds over the ice, and trained them to work in teams.. Every possible problem was carefully anticipated in light of experience, and exact amounts of food for men and dogs were allocated and stored. Amundson said that forgetting “a single trouser button” could doom the effort. As it unfolded, everything worked as anticipated, and all the men and dogs returned safely after reaching the South Pole.

From At the South Pole by Roald Amundsen, 1913 [Public domain]

The story of Amundson and the Fram is an illustration of how to overcome major obstacles to achieve audacious goals. I’d like to build on it to return to a topic I’ve touched on in two previous blogs. The audacious goal: Overcoming the substantial gap in elementary reading achievement between students who qualify for free lunch and those who do not, between African American and White students, and between Hispanic and non-Hispanic students. According to the National Assessment of Educational Progress (NAEP), each of these gaps is about one half of a standard deviation, also known as an effect size of +0.50. This is a very large gap, but it has been overcome in a very small number of intensive programs. These programs were able to increase the achievement of disadvantaged students by an effect size of more than +0.50, but few were able to reproduce these gains under normal circumstances. Our goal is to enable thousands of ordinary schools serving disadvantaged students to achieve such outcomes, at a cost of no more than 5% beyond ordinary per-pupil costs.

Educational Reform and Audacious Goals

Researchers have long been creating and evaluating many different approaches to improving reading achievement. This is necessary in the research and development process to find “what works” and build up from there. However, each individual program or practice has a modest effect on key outcomes, and we rarely combine proven programs to achieve an effect large enough to, for example, overcome the achievement gap. This is not what Amundson, or the Wright Brothers, or the worldwide team that achieved eradication of smallpox did. Instead, they set audacious goals and kept at them systematically, using what works, until they were achieved.

I would argue that we should and could do the same in education. The reading achievement gap is the largest problem of educational practice and policy in the U.S. We need to use everything we know how to do to solve it. This means stating in advance that our goal is to find strategies capable of eliminating reading gaps at scale, and refusing to declare victory until this goal is achieved. We need to establish that the goal can be achieved, by ordinary teachers and principals in ordinary schools serving disadvantaged students.

Tutoring Our Way to the Goal

In a previous blog I proposed that the goal of +0.50 could be reached by providing disadvantaged, low-achieving students tutoring in small groups or, when necessary, one-to-one. As I argued there and elsewhere, there is no reading intervention as effective as tutoring. Recent reviews of research have found that well-qualified teaching assistants using proven methods can achieve outcomes as good as those achieved by certified teachers working as tutors, thereby making tutoring much less expensive and more replicable (Inns et al., 2019). Providing schools with significant numbers of well-trained tutors is one likely means of reaching ES=+0.50 for disadvantaged students. Inns et al. (2019) found an average effect size of +0.38 for tutoring by teaching assistants, but several programs had effect sizes of +0.40 to +0.47. This is not +0.50, but it is within striking distance of the goal. However, each school would need multiple tutors in order to provide high-quality tutoring to most students, to extend the known positive effects of tutoring to the whole school.

Combining Intensive Tutoring With Success for All

Tutoring may be sufficient by itself, but research on tutoring has rarely used tutoring schoolwide, to benefit all students in high-poverty schools. It may be more effective to combine widespread tutoring for students who most need it with other proven strategies designed for the whole school, rather than simply extending a program designed for individuals and small groups. One logical strategy to reach the goal of +0.50 in reading might be to combine intensive tutoring with our Success for All whole-school reform model.

Success for All adds to intensive tutoring in several ways. It provides teachers with professional development on proven reading strategies, as well as cooperative learning and classroom management strategies at all levels. Strengthening core reading instruction reduces the number of children at great risk, and even for students who are receiving tutoring, it provides a setting in which students can apply and extend their skills. For students who do not need tutoring, Success for All provides acceleration. In high-poverty schools, students who are meeting reading standards are likely to still be performing below their potential, and improving instruction for all is likely to help these students excel.

Success for All was created in the late 1980s in an attempt to achieve a goal similar to the +0.50 challenge. In its first major evaluation, a matched study in six high-poverty Baltimore elementary schools, Success for All achieved a schoolwide reading effect size of at least +0.50 schoolwide in grades 1-5 on individually administered reading measures. For students in the lowest 25% of the sample at pretest, the effect size averaged +0.75 (Madden et al., 1993). That experiment provided two to six certified teacher tutors per school, who worked one to one with the lowest-achieving first and second graders. The tutors supplemented a detailed reading program, which used cooperative learning, phonics, proven classroom management methods, parent involvement, frequent assessment, distributed leadership, and other elements (as Success for All still does).

An independent follow-up assessment found that the effect maintained to the eighth grade, and also showed a halving of retentions in grade and a halving of assignments to special education, compared to the control group (Borman & Hewes, 2002). Schools using Success for All since that time have rarely been able to afford so many tutors, instead averaging one or two tutors. Many schools using SFA have not been able to afford even one tutor. Still, across 28 qualifying studies, mostly by third parties, the Success for All effect size has averaged +0.27 (Cheung et al., in press). This is impressive, but it is not +0.50. For the lowest achievers, the mean effect size was +0.62, but again, our goal is +0.50 for all disadvantaged students, not just the lowest achievers.

Over a period of years, could schools using Success for All with five or more teaching assistant tutors reach the +0.50 goal? I’m certain of it. Could we go even further, perhaps creating a similar approach for secondary schools or adding in an emphasis on mathematics? That would be the next frontier.

The Policy Importance of +0.50

If we can routinely achieve an effect size of +0.50 in reading in most Title I schools, this would provide a real challenge for policy makers. Many policy makers argue that money does not make much difference in education, or that housing, employment, and other basic economic improvements are needed before major improvements in the education of disadvantaged students will be possible. But what if it became widely known that outcomes in high-poverty schools could be reliably and substantially improved at a modest cost, compared to the outcomes? Policy makers would hopefully focus on finding ways to provide the resources needed if they could be confident in the outcomes.

As Amundson knew, difficult goals can be attained with meticulous planning and high-quality implementation. Every element of his expedition had been tested extensively in real arctic conditions, and had been found to be effective and practical. We would propose taking a similar path to universal success in reading. Each component of a practical plan to reach an effect size of +0.50 or more must be proven to be effective in schools serving many disadvantaged students. Combining proven approaches, we can add sufficiently to the reading achievement of disadvantaged schools to enable them to perform as well as middle class students do. It just takes an audacious goal and the commitment and resources to accomplish it.

References

Borman, G., & Hewes, G. (2002).  Long-term effects and cost effectiveness of Success for All.  Educational Evaluation and Policy Analysis, 24 (2), 243-266.

Cheung, A., Xie, C., Zhang, T., & Slavin, R. E. (in press). Success for All: A quantitative synthesis of evaluations. Education Research Review.

Inns, A., Lake, C., Pellegrini, M., & Slavin, R. (2019). A synthesis of quantitative research on programs for struggling readers in elementary schools. Available at www.bestevidence.org. Manuscript submitted for publication.

Madden, N. A., Slavin, R. E., Karweit, N. L., Dolan, L., & Wasik, B. (1993). Success for All:  Longitudinal effects of a schoolwide elementary restructuring program. American Educational Reseach Journal, 30, 123-148.

Madden, N. A., & Slavin, R. E. (2017). Evaluations of technology-assisted small-group tutoring for struggling readers. Reading & Writing Quarterly, 1-8. http://dx.doi.org/10.1080/10573569.2016.1255577

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.