Do Clinical Trials Work in Education?

A recent article in The New York Times asked a provocative question: Do clinical trials work? The article was written about clinical trials in medicine, especially cancer research, where it often happens that promising medications and procedures found to be effective in small studies turn out to be ineffective in large ones.

As an advocate of clinical trials (randomized experiments) in education, I found the article distressing. In education, as in medicine, larger and better-controlled experiments frequently fail to find positive effects of programs found to be effective in smaller and less-well-controlled studies. In fact, there is a clear relationship between study sample sizes and outcomes: The larger the study, the lower the reported impact of the treatment.

Some in both medicine and education are wondering if different research methods are needed that are more likely to show positive effects. In my view, this is foolish. The problem is not in the research methods. What we need to do is to identify why so many experiments show no impacts, and solve those problems.

One common problem in randomized experiments in education, for example, is that before the experiments, both experimental and control teachers must not have been using the experimental method. If the treatment is difficult to learn to use, this may mean that in a study of one year or less, teachers using the new method did not get good at it until near the end of the experiment. There are numerous experiments in education in which there were no impacts in the first year but significant impacts in the second. Yet a one-year study would not find out about the second-year impacts, shortchanging the treatments’ reported effect.

As someone who does a lot of meta-analyses of educational treatments, another problem I routinely see is that many large, randomized evaluations assess weak treatments. For example, there are dozens of studies comparing a publisher’s new textbook in comparison to existing textbooks. Such studies invariably produce effect sizes near zero. This does not mean that the new texts are ineffective, but that they are no more effective than other texts. Technology studies also often evaluate ho-hum commercial software unlikely to make much difference. All too often, researchers carry out large and expensive evaluations of programs that are too poorly defined or too much like ordinary practice to show much impact. Wishful thinking runs up against harsh reality in large, well-controlled experiments.

The funding structure for research in education often leads experimenters to carry out large scale, randomized evaluations of programs that are not fully ready for large-scale evaluation. New and truly innovative methods often need to be piloted, evaluated on a modest scale, and only then subjected to large-scale evaluation, but funding for small-scale formative evaluation is hard to obtain.

In education, as in medicine, there is a problem of too many disappointing findings in clinical trials. Yet the solution is not to abandon clinical trials. It is to create more powerful and effective treatments.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Lessons from Innovators: Reading Recovery

The process of moving an educational innovation from a good idea to widespread effective implementation is far from straightforward, and no one has a magic formula for doing it. The William T. Grant and Spencer Foundations, with help from the Forum for Youth Investment, have created a community composed of grantees in the federal Investing in Innovation (i3) program to share ideas and best practices. Our Success for All program participates in this community. In this space, I, in partnership with the Forum for Youth Investment, highlight observations from the experiences of i3 grantees other than our own, in an attempt to share the thinking and experience of colleagues out on the front lines of evidence-based reform.

This blog is based on an interview between the Forum for Youth Investment and Jerry D’Agostino, Professor of Education at the Ohio State University and Director of Reading Recovery’s i3 project. A persistent challenge for programs that have scaled up is how to sustain for the long term. In this interview, D’Agostino shares how this long-standing literacy intervention has dealt with the challenge and how it has reinvented itself over the years in order to stay current.

Stay Fresh
Reading Recovery is a research-based, short-term intervention that involves one-to-one teaching for the lowest-achieving first graders. It began in New Zealand in the 1970’s but has been in operation in the United States for 30 years and has spread across the country. Over the years, Reading Recovery has expanded and contracted depending on funding, interest from school districts, and our capacity. Today there are training centers at 19 universities that equip teachers to deliver the intervention and the program has a presence in some 8,000 schools across 49 states. With that kind of scale and longevity, it can be easy to become complacent and assume the intervention speaks for itself. D’Agostino says just the opposite is true. “We know that being the old brand that has been around for a long time can be hard,” he notes. “You have to think about how to keep the brand fresh. Superintendents want the newest hot thing. Teachers have to know it will work with their kids in their classrooms. We have spent time focused on how to adjust the model to offer new features and respond to current education trends such as the Common Core. You always have to show teachers and administrators how the intervention addresses the issue of the day. For example, it isn’t enough that the intervention produces strong effect sizes. For teachers, that is a meaningless number. They want to know that the program will help their third graders achieve the literacy level now required in nearly 40 states to be promoted to 4th grade.”

Be Flexible but Maintain Your Core
Reading Recovery has taken seriously the idea of identifying the intervention’s core elements and also responding to the educational system’s current needs. They know that one-to-one instruction and 30-minute daily lessons are non-negotiable, but they also recognize that adaptations are needed. For example, innovations in the lesson framework have resulted in a design for classroom instruction (Literacy Collaborative), small groups (Comprehensive Intervention Model), and training for special education and ESL teachers (Literacy Lessons). “Our innovations have come as direct requests from schools,” says D’Agostino. “For example, a school says they need something for English Language Learners and we develop something new for that one school that then becomes a part of our overall product line. It allows growth for Reading Recovery and flexibility for schools.” Another non-negotiable is keeping training centralized. Although teacher leaders can receive training at one of the 19 partner universities, there are only a few places where trainers of teacher leaders can get certified. That allows Reading Recovery to maintain some quality control and fidelity over teacher leader training. “I’ve always been impressed with the fidelity of Reading Recovery instruction,” said D’Aogstino. “I’ve seen Reading Recovery lessons in Zanesville, Ohio and Dublin, Ireland. The framework is the same, but each lesson is different in terms of how the teacher interacts with the student to scaffold literacy learning.

Combine Historical Expertise with Fresh Perspective
D’Agostino is quick to note that one of Reading Recovery’s strengths and challenges is the longevity of its founders and senior leadership. Many of the original developers of the intervention are still in leadership positions. This allows for a historical perspective and continuity of purpose that are rare in education these days. It can also hinder innovation. That is why the organization also tries to find leadership positions for newer faculty and teachers with recent teaching and administrative experience who can bring fresh ideas and a willingness to push for some of the new adjustments to the model that schools are requesting.

Adapt, Adjust, and Meet Schools Where They Are
D’Agostino emphasizes that Reading Recovery’s current success and long history is no reason to sit back and relax. “We have survived a lot of changes over the years. We’ve grown, we’ve shrunk, we’ve survived major threats to our program from other national initiatives. Right now with our i3 grant, we are in a great position. We are going to reach our goal of training 3,700 teachers and producing good effects. But I don’t know that that will position us well for the future. In fact, I won’t be happy if we just reach our goals.” Sustaining an effective intervention and bringing it to more schools and students around the country means innovating, moving, pushing to the next level…and spreading the word. “Schools don’t necessarily hear about government funded initiatives that achieve high evidence standards according to the What Works Clearinghouse,” muses D’Agostino. “They hear from hundreds of vendors each year citing their effectiveness, so how do we distinguish ourselves? We can’t just assume success in our i3 grant will lead to sustainability. Sustainability is all about results. For example, we know that the outcomes are remarkable – most of the lowest-achieving first graders accelerate with Reading Recovery and reach the average of their cohort – but we also know from our annual evaluation that there’s a great deal of variation across schools and teachers. So right now we want to know, what do effective Reading Recovery teachers do and how is that different from less effective Reading Recovery teachers? Knowing more about that black box of teaching will help the intervention overall. And understanding how to foster local ownership will give the intervention its real staying power.”

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Preservice Education and School Reform

Recently, I heard a great story. A friend was recalling her first year as a teacher, many years ago. She’d learned all about cooperative learning in her preservice program, and she was eager to try it out in her inner-city elementary school. She assigned her students to teams and got off to a great start.

Very soon, however, my friend got a visit from her principal. He wanted to know what all that noise was coming from her class. He had a clear opinion that a quiet school was a good school, so he was very concerned.

My friend was well prepared. She explained about all the research on cooperative learning, pointed out that she was using cooperative learning with group goals and individual accountability, in accord with the latest research, and showed the principal how happy and productive the students were.

“It’s fine if you want to use cooperative learning in your class,” the principal said, “but you’re going to have to get your students to stop talking to each other when they do it!”

My friend’s story illustrates some of the difficulties inherent in expecting preservice programs to reform America’s schools by turning out teachers trained in the very latest research-proven methods. The problem is that a new 22-year-old teacher is the least powerful person in the school. If he or she has any sense at all, the new teacher will do what the school is doing.

This does not mean that preservice programs need not expose aspiring teachers to proven methods. They should definitely do so, because as teachers grow and develop as professionals, they need a language and background in proven programs. My friend might not have been able to use cooperative learning in that school that year, but she did so later on. Knowing about it gave her a tool to improve her classroom that no principal could ever take away from her. But we need to be realistic about how much preservice can do to reform schools.

Change happens best in education when it takes place at the school level, so that many teachers can work together with a supportive administrator to implement ambitious reforms. Principals working in collaboration with whole staffs or departments can make major changes, bring in professional development and materials, and implement proven approaches. A supportive district and networks of liked-minded schools can also provide crucial strength to schools adopting, implementing, and progressively improving proven methods. When the school has adopted a given method with strong evidence of effectiveness, new teachers can learn to use it, and all children benefit. If the new teacher learned about the method or others like it during his or her preservice program, all the better, but counting on that experience to lead to change on a scale that matters is unlikely.

Preservice programs need to do their best to turn out capable, intentional, reflective teachers who have knowledge and skills that will be useful in any setting. Aspiring teachers need to learn about various approaches to teaching and to understand why they work and how they can be effectively applied. They should have opportunities to try out various methods in real classrooms, with feedback from mentors and peers. But don’t count on this experience to gradually lead to reformed schools. For that, we need to reform the schools directly, as whole organizations, and then staff them with the best teachers we can find. As more schools use proven models, preservice programs will hopefully prepare their students to teach in schools using them.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin