Keep Up the Good Work (To Keep Up the Good Outcomes)

I just read an outstanding study that contains a hard but crucially important lesson. The study, by Woodbridge et al. (2014), evaluated a behavior management program for students with behavior problems. The program, First Step to Success, has been successfully evaluated many times. In the Woodbridge et al. study, 200 children in grades 1 to 3 with serious behavior problems were randomly assigned to experimental or control groups. On behavior and achievement measures, students in the experimental group scored much higher, with effect sizes of +0.44 to +0.87. Very impressive.

The researchers came back a year later to see if the outcomes were still there. Despite the substantial impacts seen at posttest, none of three prosocial/adaptive behavior measures, only one of three problem/maladaptive behaviors, and none of four academic achievement measures showed positive outcomes.

These findings were distressing to the researchers, but they contain a message. In this study, students passed from teachers who had been trained in the First Step method to teachers who had not. The treatment is well-established and inexpensive. Why should it ever be seen as a one-year intervention with a follow-up? Instead, imagine that all teachers in the school learned the program and all continued to implement it for many years. In this circumstance, it would be highly likely that the first-year positive impacts would be sustained and most likely improved over time.

Follow-up assessments are always interesting, and for interventions that are very expensive it may be crucial to demonstrate lasting impacts. But so often in education effective treatments can be maintained for many years, creating more effective school-wide environments and lasting impacts over time. Much as we might like to have one-shot treatments with long-lasting impacts, this does not correspond to the nature of children. The personal, family, or community problems that led children to have problems at a given point in time are likely to lead to problems in the future, too. But the solution is clear. Keep up the good work to keep up the good outcomes!

Advertisements

What Schools in One Place Can Learn from Schools Elsewhere

In a recent blog, I responded to an article by Lisbeth Schorr and Srik Gopal about their concerns that the findings of randomized experiments will not generalize from one set of schools to another. I got a lot of supportive response to the blog, but I realize that I left out a key point.

The missing point was this: the idea that effective programs readily generalize from one place to another is not theoretical. It happens all the time. I try to avoid talking about our own programs, but in this case, it’s unavoidable. Our Success for All program started almost 30 years ago, working with African American students in Baltimore. We got terrific results with those first schools. But our first dissemination schools beyond Baltimore included a Philadelphia school primarily serving Cambodian immigrants, rural schools in the South, small town schools in the Midwest, and so on. We had to adapt and refine our approaches for these different circumstances, but we found positive effects across a very wide range of settings and circumstances. Over the years, some of our most successful schools have been ones serving a Native Americans, such as a school in the Arizona desert and a school in far northern Quebec. Another category of schools where we see outstanding success is ones serving Hispanic students, including English language learners, as in the Alhambra district in Phoenix and a charter school near Los Angeles. One of our most successful districts anywhere is in small-city Steubenville, Ohio. We have established a successful network of SFA schools in England and Wales, where we have extraordinary schools primarily serving Pakistani, African, and disadvantaged White students in a very different policy context from the one we face in the U.S. And yes, we continue to find great results in Baltimore and in cities that resemble our original home, such as Detroit.

The ability to generalize from one set of schools to others is not at all limited to Success for All. Reading Recovery, for example, has had success in every kind of school, in countries throughout the world. Direct Instruction has also been successful in a wide array of types of schools. In fact, I’d argue that it is rare to find programs that have been proven to be effective in rigorous research that then fail to generalize to other schools, even ones that are quite different. Of course, there is great variation in outcomes in any set of schools using any innovative program, but that variation has to do with leadership, local support, resources, and so on, not with a fundamental limitation on generalizability to additional populations.

How is it possible that programs initially designed for one setting and population so often generalize to others? My answer would be that in most fundamental regards, the closer you get to the classroom, the more schools begin to resemble each other. Individual students do not all learn the same way, but every classroom contains a range of students who have a predictable set of needs. Any effective program has to be able to meet those needs, wherever the school happens to be located. For example, every classroom has some number of kids who are confident, curious, and capable, some number who are struggling, some number who are shy and quiet, some number who are troublemakers. Most contain students who are not native speakers of English. Any effective program has to have a workable plan for each of these types of students, even if the proportions of each may vary from classroom to classroom and school to school.

There are reasonable adaptations necessary for different school contexts, of course. There are schools where attendance is a big issue and others where it can be assumed, schools where safety is a major concern and others where it is less so. Schools in rural areas have different needs from those in urban or suburban ones, and obviously schools with many recent immigrants have different needs from those in which all students are native speakers of English. Involving parents effectively looks different in different places, and there are schools in which eyeglasses and other health concerns can be assumed to be taken care of and others where they are major impediments to success. But after the necessary accommodations are made, you come down to a teacher and twenty to thirty children who need to be motivated, to be guided, to have their individual needs met, and to have their time used to greatest effect. You need to have an effective plan to manage diverse needs and to inspire kids to see their own possibilities. You need to fire children’s imaginations and help them use their minds well to write and solve problems and imagine their own futures. These needs exist equally in Peru and Poughkeepsie, in the Arizona desert or the valleys of Wales, in Detroit or Eastern Kentucky, in California or Maine.

Disregarding evidence from randomized experiments because it does not always replicate is a recipe for the status quo, as far as the eye can see. And the status quo is unacceptable. In my experience, the reason programs fail to replicate is that they were never all that successful in the first place, or because they attempt to replicate a form of a model much less robust than the one they researched.

Generalization can happen. It happens all the time. It has to be planned for, designed for, not just assumed, but it can and does happen. Rather than using failure to replicate as a stick to beat evidence-based policy, let’s agree that we can learn to replicate, and then use every tool at hand to do so. There are so many vulnerable children who need better educations, and we cannot be distracted by arguments that “nothing replicates” that are contradicted by many examples throughout the world.

Kumbaya

When I was a kid, my brothers and I used to go to a YMCA camp on the Chesapeake Bay for a month every summer. My mother said that it was cheaper than feeding us, which was my first exposure to cost-effectiveness analysis.

At the camp, we did all the usual camp things. One of those was evening campfires with singing. This was a YMCA camp in the early 1960s, so we sang a lot of folk songs about peace, love, and understanding. I was reminded of this because I now have a granddaughter who loves a Peter, Paul, and Mary disk with just those songs on it, including Kumbaya.

Skip forward a few decades from those long-ago campfires. Today, the very word Kumbaya is used as an insult of sorts. It means that the person being insulted is an unrealistic idealist, who expects that social progress can be made by sitting around the campfire and singing. As a data-minded social scientist who expects evidence from randomized studies for just about everything, I should be firmly in the anti-Kumbaya camp, so to speak. But I’m not.

Let me be clear: I do not think that singing around campfires causes important social change. Yet I’d argue that a lack of Kumbaya is just as much a problem. Kumbaya-fueled idealism is the very core of evidence-based reform, in fact.

Here’s why. The greatest danger to evidence-based reform is the widespread belief that doing well-intentioned things is good enough, even if we don’t know whether they work. An idealist should never accept this. Good intentions are nice, but they do not bring about real Kumbaya. That depends on good outcomes.

Sitting around campfires and singing about peace, love, and understanding should be good preparation for actually caring whether your actions make the difference you intend them to make. Sure, life teaches you that it takes toughness to insist that good intentions become good actions, but you have to start with the good intentions.

So here is another verse to that ageless song:

Someone’s experimenting, Lord
Kumbaya
Randomizing, Lord
Kumbaya
Someone’s analyzing, Lord
Kumbaya
Oh Lord,
Kumbaya

Remembering Al Shanker: Teachers and Professionalism

Back in the day, I knew Al Shanker, the founder of the American Federation of Teachers. No one has ever been more of an advocate for teachers’ rights – or for their professionalism. At the same time, no one was more of an advocate for evidence as a basis for teaching. He saw no conflict between evidence-based teaching and professionalism. In fact, he saw them as complementary. He argued that in fields in which professionals possess unique knowledge and skills, backed up by research, those professionals are well respected, well compensated, and play a leading role in the institutions in which they work.

If teachers want to be taken seriously, they must be seen to be using methods, technologies, and materials that not just anyone knows how to use, and that are known to be effective. Think physicians, engineers, and lawyers. Their positions in society depend on their possession of specialized and proven knowledge and skills.

Yet when I speak about evidence-based reform, I often get questions from teachers about whether using evidence-proven programs will take away their professionalism, creativity, or independence. I am sympathetic to this question, because I am aware that teachers have had to put up with quite a lot in recent years. Teaching is increasingly being seen by government and the public as something anyone can do.

But how can the teaching profession turn this around? I think Al Shanker had the right answer. If teachers (and teacher educators) can honestly present themselves to the public as people who can select and use proven programs and practices, ones that not just anyone could use effectively, that would go a long way, I think, to enhancing the public’s perception of the professionalism of the field. It would also be awfully good for students, parents, and the economy, of course.

Al Shanker knew that teachers were going to have to publicly and fervently embrace evidence, both to do their jobs better and to make it clear that being a teacher requires knowledge and skills than the general public can respect. I’m certain that he would be a big fan of the new Every Student Succeeds Act (ESSA) evidence standards, which will help educators, policy makers, and researchers identify and put to use proven programs and practices.

Evidence-based reform is essential for kids, but also for teachers. Al Shanker knew that 30 years ago, and his AFT has been a champion for evidence ever since.