Can Findings From One Set Of Schools Apply To Others?

Every person is unique. Yet that does not mean that research showing the effectiveness of medical treatments, for example, does not apply to people beyond the ones who were in a particular study. If nothing generalized from one circumstance to others, then science would be meaningless. I think every educated person understands this.

Yet for some reason, research in education is often criticized for trying to generalize from one set of schools to others. Whenever I speak about evidence-based reform in education, most recently in a talk at my alma mater, Reed College, someone raises this concern. In a recent article by Lisbeth Schorr and Srik Gopal, the authors wonder how anything can generalize “from Peru to Poughkeepsie.”

First, let me state the obvious. Every school is different, and findings from studies done elsewhere cannot be assumed to apply to a specific school or set of schools. However, it would be foolish to ignore the evidence from high-quality research, especially to the degree that a given school considering using a program or practice found effective elsewhere resembles the schools in the study that established that evidence. So Peru to Poughkeepsie might be a stretch, unless it is Peru, Illinois. And should Poughkeepsie ignore evidence from nearby Tarrytown and Nyack? Taking a position that generalization is never appropriate would be just as unjustified as taking a position that it is always justified.

There is an old saying to the effect that the race is not always to the swift nor the fight to the strong, but it’s best to bet that way. When responsible educators choose programs for their schools and districts, they are making a bet on behalf of their children. Why would they not take the evidence into account in making these important choices?

Determining when generalization is more or less likely is not too difficult. First, you’d want to consider the strength of the evidence. For example, a program proven effective in multiple studies done by multiple researchers with many diverse schools, with random assignment to experimental or control groups and measures not made by the experimenters or developers, should give potential adopters a lot of confidence. To the degree that those studies involved schools similar to yours, serving similar communities, that adds a lot. The consistency of the outcomes across different studies would be important to consider.

Schorr and Gopal are not opposed to randomized studies, but they warn about placing too much reliance on them. Yet the advantage of randomized studies is that they rule out bias. How can that be a bad thing? What we need is a lot more randomized studies, and other studies with rigorous designs, done in a lot of places, so that we can build up a large and diverse evidence base for programs that can be replicated. It so happens that the road to generalizability is precisely the one that Schorr and Gopal want us to diminish, because if fewer randomized studies are done, we will lack the quality, size, freedom from bias, and diversity of research needed to determine if a program is truly and broadly effective.

Discussions about when generalization is most likely to take place are healthy and welcome. But they are not academic. America’s schools are not getting better fast enough, and achievement gaps by race and class remain unacceptable. Identifying proven programs and practices and replicating them broadly is the best way I know of to make genuine, lasting progress. The evidence base is only now getting large and good enough to justify policies of evidence-based reform, as the recent ESSA legislation tentatively begins to do. We need to continue to expand that evidence base and to use what we do know while working to learn more. Pretending that no school can learn from what was done in any other does not move us forward, and forward is the direction we need to be moving as fast as we possibly can.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: