Love, Hope, and Evidence in Secondary Reading

I am pleased to announce that our article reviewing research on effective secondary reading programs has just been posted on the Best Evidence Encyclopedia, aka the BEE. Written with my colleagues Ariane Baye, Cynthia Lake, and Amanda Inns, our review found 64 studies of 49 reading programs for students in grades 6 to 12, which had to meet very high standards of quality. For example, 55 of the studies used random assignment to conditions.

But before I get all nerdy about the technical standards of the review, I want to reflect on what we learned. I’ve already written about one thing we learned, that simply providing more instructional time made little difference in outcomes. In 22 of the studies, students got an extra period for reading beyond what control students got for at least an entire year, yet programs (other than tutoring) that provided extra time did no better than those that did not.

If time doesn’t help struggling readers, what does? I think I can summarize our findings with three words: love, hope, and evidence.

Love and hope are exactly what students who are reading below grade level are lacking. They are no longer naive. They know exactly what it means to be a poor reader in a high-poverty secondary school (almost all of the schools in our review served disadvantaged adolescents). If you can’t read well, college is out of the question. Decent jobs without a degree are scarce. If you have no hope, you cannot be motivated, or you may be motivated in antisocial directions that give you at least a chance for money and recognition. Every child needs love, but poor readers in secondary schools are too often looking for love in all the wrong places.

The successful programs in our review were ones that give adolescents a chance to earn the hope and love they crave. One category, all studies done in England, involved one-to-one and small group tutoring. How better to build close relationships between students and caring adults than to have individual or very small group time with them? And the one-to-one or small group setting allows tutors to personalize instruction, giving students a sense of hope that this time, their efforts will pay off (as the evidence says it will).

But the largest impacts in our review came from two related programs – The Reading Edge and Talent Development High School (TDHS). These both developed in our research center at Johns Hopkins University in the 1990s, so I have to be very modest here. But beyond these individual programs, I think there is a larger message.

Both The Reading Edge (for middle schools) and TDHS (for high schools) organize students into mixed-ability cooperative teams. The team members work on activities designed to build reading comprehension and related skills. Students are frequently assessed and on the basis of those assessments, they can earn recognition for their teams. Teachers introduce lessons, and then, as students work with each other on reading activities, teachers can cruise around the class looking in on students who need encouragement or help, solving problems, and building relationships. Students are on task, eager to learn, and seeing the progress they are making, but students and teachers are laughing together, sharing easy banter, and encouraging each other. Yes, this really happens. I’ve seen it hundreds of times in secondary schools throughout the U.S. and England.

Many of the most successful programs in our review also are based on principles of love and hope. BARR, a high school program, is an excellent example. It uses block scheduling to build positive relationships among a group of students and teachers, adding regular meetings between teachers and students to review their progress in all areas, social as well as academic. The program focuses on building positive social-emotional skills and behaviors, and helping students describe their desired futures, make plans to get there, and regularly review progress on their plans with their teachers and peers. Love and hope.

California’s Expository Reading and Writing Course helps 12th graders hoping to attend California State Universities prepare to pass the test used to determine whether students have to take remedial English (a key factor in college dropout). The students work in groups, helping each other to build reading, writing, and discussion skills, and helping students to visualize a future for themselves. Love and hope.

A few technology programs showed promising outcomes, especially Achieve3000 and Read 180. These do not replace teachers and peers with technology, but instead cycle students through small group, teacher-led, and computer-assisted activities. Pure technology programs did not work so well, but models taking advantage of relationships as well as personalization did best. Love and hope.

Of course, love and hope are not sufficient. We also need evidence that students are learning more than they might have been. To produce positive achievement effects requires outstanding teaching strategies, professional development, curricular approaches, assessments, and more. Love and hope may be necessary but they are not sufficient.

Our review applied the toughest evidence standards we have ever applied. Most of the studies we reviewed did not show positive impacts on reading achievement. But the ones that did so inspire that much more confidence. The very fact that we could apply these standards and still find plenty of studies that meet them shows how much our field is maturing. This in itself fills me with hope.

And love.

Apology

In a recent blog, I wrote about work we are doing to measure the impact on reading and math performance of a citywide campaign to provide assessments and eyeglasses to every child in Baltimore, from pre-k to grade 8. I forgot to mention the name of the project, Vision for Baltimore, and neglected to say that the project operates under the authority of the Baltimore City Health Department, which has been a strong supporter. I apologize for the omission.

Advertisements

Time Passes. Will You?

When I was in high school, one of my teachers posted a sign on her classroom wall under the clock:

Time passes. Will you?

Students spend a lot of time watching clocks, yearning for the period to be over. Yet educators and researchers often seem to believe that more time is of course beneficial to kids’ learning. Isn’t that obvious?

In a major review of secondary reading programs I am completing with my colleagues Ariane Baye, Cynthia Lake, and Amanda Inns, it turns out that the kids were right. More time, at least in remedial reading, may not be beneficial at all.

Our review identified 60 studies of extraordinary quality- mostly large-scale randomized experiments- evaluating reading programs for students in grades 6 to 12. In most of the studies, students reading 2 to 5 grade levels below expectations were randomly assigned to receive an extra class period of reading instruction every day all year, in some cases for two or three years. Students randomly assigned to the control group continued in classes such as art, music, or study hall. The strategies used in the remedial classes varied widely, including technology approaches, teaching focused on metacognitive skills (e.g., summarization, clarification, graphic organizers), teaching focused on phonics skills that should have been learned in elementary school, and other remedial approaches, all of which provided substantial additional time for reading instruction. It is also important to note that the extra-time classes were generally smaller than ordinary classes, in the range of 12 to 20 students.

In contrast, other studies provided whole class or whole school methods, many of which also focused on metacognitive skills, but none of which provided additional time.

Analyzing across all studies, setting aside five British tutoring studies, there was no effect of additional time in remedial reading. The effect size for the 22 extra-time studies was +0.08, while for 34 whole class/whole school studies, it was slightly higher, ES =+0.10. That’s an awful lot of additional teaching time for no additional learning benefit.

So what did work? Not surprisingly, one-to-one and small-group tutoring (up to one to four) were very effective. These are remedial and do usually provide additional teaching time, but in a much more intensive and personalized way.

Other approaches that showed particular promise simply made better use of existing class time. A program called The Reading Edge involves students in small mixed-ability teams where they are responsible for the reading success of all team members. A technology approach called Achieve3000 showed substantial gains for low-achieving students. A whole-school model called BARR focuses on social-emotional learning, building relationships between teachers and students, and carefully monitoring students’ progress in reading and math. Another model called ERWC prepares 12th graders to succeed on the tests used to determine whether students have to take remedial English at California State Universities.

What characterized these successful approaches? None were presented as remedial. All were exciting and personalized, and not at all like traditional instruction. All gave students social supports from peers and teachers, and reasons to hope that this time, they were going to be successful.

There is no magic to these approaches, and not every study of them found positive outcomes. But there was clearly no advantage of remedial approaches providing extra time.

In fact, according to the data, students would have done just as well to stay in art or music. And if you’d asked the kids, they’d probably agree.

Time is important, but motivation, caring, and personalization are what counts most in secondary reading, and surely in other subjects as well.

Time passes. Kids will pass, too, if we make such good use of our time with them that they won’t even notice the minutes going by.

Joy is a Basic Skill in Secondary Reading

I have a policy of not talking about studies I’m engaged in before they are done and available, but I have an observation to make that just won’t wait.

I’m working on a review of research on secondary reading programs with colleagues Ariane Baye (University of Liege in Belgium) and Cynthia Lake (Johns Hopkins University). We have found a large number of very high-quality studies evaluating a broad range of programs. Most are large, randomized experiments.

Mostly, our review is really depressing. The great majority of studies have found no effects on learning. In particular, programs that focus on teaching middle and high school students struggling in reading in classes of 12 to 20, emphasizing meta-cognitive strategies, phonics, fluency, and/or training for teachers in what they were already doing, show few impacts on learning. Most of the studies provided daily, extra reading classes to help struggling readers build their skills, while the control group got band or art. They should have stayed in band or art.

Yet all is not dismal. Two approaches did have markedly positive effects. One was tutoring students in groups of one to four, not every day but perhaps twice a week. The other was cooperative learning, where students worked in four-member teams to help each other learn and practice reading skills. How could these approaches be so much more effective than the others?

My answer begins with a consideration of the nature of struggling adolescent readers. They are bored out of their brains. They are likely to see school as demeaning, isolating, and unrewarding. All adolescents live for their friends. They crave mastery and respect. Remedial approaches have to be fantastic to overcome the negative aspects of having to be remediated in the first place.

Tutoring can make a big difference, because groups are small enough for students to make meaningful relationships with adults and with other kids, and instruction can be personalized to meet their unique needs, to give them a real shot at mastery.

Cooperative learning, however, had a larger average effect size than tutoring. Even though cooperative learning did not require smaller class sizes and extra daily instructional periods, it was much more effective than remedial instruction. Cooperative learning gives struggling adolescent readers opportunities to work with their peers, to teach each other, to tease each other, to laugh, to be active rather than passive. To them, it means joy. And joy is a basic skill.

Of course, joy is not enough. Kids must be learning joyfully, not just joyful. Yet in our national education system, so focused on testing and accountability, we have to keep remembering who we are teaching and what they need. More of the same, a little slower and a little louder, won’t do it. Adolescents need a reason to believe that things can be better, and that school need not cut them off from their peers. They need opportunities to teach and learn from each other. School must be joyful, or it is nothing at all, for so many adolescents.

The Wonderful Reputation of Educational Research

Back in 1993, Carl Kaestle memorably wrote about the “awful reputation of educational research.” At the time, he was right. But that was 23 years ago. In the interim, educational research has made extraordinary advances. It is now admired by researchers in many other fields and by policy makers in many areas of government. As indicated by the importance of evidence in the Every Student Succeeds Act (ESSA), evidence is starting to make more of a difference in policy and practice. There is still a long, long way to go, but the trend is hugely positive.

In a recent article for the Brookings Institution, Ruth Curran Neild, acting director of the Institute of Education Sciences (IES), argued that educational research is on the right track. The one thing it lacks, she says, is adequate funding. I totally agree. Of course there are improvements that could be made to education policies and practices, but the part of the education field working on using science to improve outcomes for children is very much going in the right direction. Many are frustrated that it is not getting there fast enough, but we need more wind in our sails, not a change of course.

I was listening recently to an NPR broadcast about a new center for research on immunological treatments for cancer. The interviewer asked how their center could possibly make much difference with a grant of only $250 million. The director sheepishly agreed this was a problem, but hoped they could nevertheless make a contribution. If only we in education had conversations like this – ever!

What has radically changed over the past 15 years is that there is now far more support than there once was for randomized evaluations of replicable programs and practices, and as a result we are collectively building a strong set of studies that use the kinds of designs common in medicine and agriculture but not, until recently, in education. My colleagues and I constantly update reviews of research on educational interventions in the main areas of practice at the Best Evidence Encyclopedia website. Where once randomized studies were rare, they are becoming the norm. We recently published a review of research on early childhood programs, in which we located 32 studies of 22 different programs. Twenty-nine of the studies used randomized designs, thanks primarily to funding and leadership from a federal investment called Preschool Curriculum Evaluation Research (PCER). We are working on a review of research on secondary reading programs. Due to the federal Striving Readers program, which invested in evaluations of a wide variety of school interventions, our review is now dominated by randomized studies. Studies of programs for struggling elementary readers are now overwhelmingly randomized. The Investing in Innovation (i3) program requires randomized evaluations in its validation and scale-up grants and encourages them in its development grants, and this is increasing the prevalence of randomized studies across all studies of programs for students from grades pre-K to 12. The National Science Foundation has begun to fund scale-up projects that require random assignment, as have a few private foundations.

Random assignment is the hallmark of rigorous science. From a methodological standpoint, random assignment is crucial because only when students, teachers, or schools are randomly assigned to treatment or control conditions can readers be sure that any differences observed at posttest are truly the result of the treatments, and not of self-selection or other bias. But more than this, use of random assignment establishes a field as serious about its science. Studies that use random assignment are called “gold standard,” because there is no better design in existence. Yes, there are better and worse randomized studies, better and worse measures, and so on. Mixed methods studies can usefully add insight to the numbers. Replication is very important in establishing effectiveness. And there are certainly circumstances in which randomization is impossible or impractical, and a well-done quasi-experiment will do. But all this being said, the use of randomization moves the science of education forward and gives educational leaders reliable information on which to make decisions.

The most telling criticism of randomized experiments is that they are expensive. Yes, they can be. Encouragement and funding from IES and the Laura and John Arnold Foundation is increasing the use of inexpensive experiments in situations in which treatments and (usually) measures are already being paid for by government or other sources, so only funding for the evaluation is needed. But these experiments are only possible in special circumstances. In others, someone has to come up with serious funding to support randomized designs.

This brings us back to Ruth Neild’s main point. We know what needs to be done in educational research. We need to develop a wide variety of promising innovations, subject them to rigorous, ultimately randomized experiments, and then disseminate those programs found to be effective. We have systems in place to do all of these things. We just need a lot more funding to do them faster and better.

I don’t know if the increases in the quality of research in education are understood by policy makers, or how much this quality matters for funding. But education now has a case to make that it deserves much greater funding. Educational research is no longer just of interest to the academics who do it. It is producing answers that matter for children, and that should justify funding in line with our field’s new, wonderful reputation.

To Pluto and Beyond

Like many others, I was thrilled to see The New Horizons spacecraft reach and photograph Pluto. After being banished from the League of Planets shortly after New Horizons was launched, I’ll bet Pluto felt much better with all the attention.

Those who read this blog are probably expecting me to go into a rant at this point about how much we are willing to spend to send a spacecraft to take pictures and how little we are willing to spend on finding out how to help our nation’s children learn to read the newspaper or understand the math or the space science around this marvelous event. Well, consider it ranted. It does not make me feel any better that funding for NASA itself is being cut. We are a hugely wealthy country, and we can afford to go to Pluto and to educate our children to a much higher standard than we do. In fact, the way we became a hugely wealthy country, and the only way we can maintain our wealth into the future, is by investing in education, science, technology and invention.

My colleagues and I recently completed reviews of research on elementary and then secondary science education. You can find them here. The reviews find very similar outcomes at the different grade levels. Instructional methods emphasizing professional development for teachers on well-defined teaching strategies, such as cooperative learning and science-reading integration, have solid effects on science learning outcomes. Moving from one textbook to another almost never makes a difference, and use of science kits does not improve science learning. Technology-focused programs have a great deal of promise, but the studies are few and of limited quality, at least so far.

However, the most depressing finding is that there were far too few studies, across all science teaching approaches, that met even modest standards of rigor. Using our standards (which just require a control group, initial equality, fair measures, and a duration of 12 weeks), there were just 21 secondary studies in the past quarter-century. The number was the same for elementary studies. This is shameful. Science teaching is widely acknowledged to be a key to our nation’s future, yet our investment in high-quality studies and innovation is so low that we really know very little about how to do it better.

To explore the universe, to cure diseases, to engineer new solutions of all kinds, requires a population that is proficient in science, technology and mathematics. Is there anyone on the (still recognized) planet Earth who does not know this? Yet if we were serious about going boldly where no nation has gone before, would we continue to invest so little in understanding how to engage and excite our students in science, math, and technology?

Today, we rely on an extraordinary but tiny elite for the scientific progress we do make. We need to extend far beyond this, as more and more occupations come to require deep understanding of science and math. We need to enable teachers in elementary and secondary schools to democratize science knowledge and skill. There is no question that we can design better teaching methods and technologies, evaluate them, and scale them up. I wonder when we will get serious about doing so?

Congratulations to NASA, the Johns Hopkins Applied Physics Lab, and the American taxpayer for the New Horizons trip to Pluto. But consider this. The next generation of scientists and engineers who will perform the marvels of the future are in elementary and secondary classes right now. Improving science learning for these precious future scientists and engineers is essential for our nation’s future.