Out to Launch

Dear Reader,

Every Thursday for the past nine years, except for major holidays, I have produced my blog on educational research and practice.  However, this week, I am running my blog on Monday.  Why?  Because on Monday, we are launching our tutoring website, the one I’ve been hinting about for weeks.  And for some technical reasons I do not even understand, the website has to go out on the same day as my blog.   Yes, I should have scheduled the launch for a Thursday, but here we are.  

I can guarantee you that this website will completely change your perception of life, the universe, and everything, or at the very least about tutoring.  

After Monday, my blogs will return to normal.  Perhaps even better than normal.

Thank you for your patience and understanding.  

-Bob

Lessons for Educational Research from the COVID-19 Vaccines

Since the beginning of the COVID-19 pandemic, more than 130 biotech companies have launched major efforts to develop and test vaccines. Only four have been approved so far (Pfizer, Moderna, Johnson & Johnson, and AstraZeneca). Among the others, many have outright failed, and others are considered highly unlikely. Some of the failed vaccines are small, fringe companies, but they also include some of the largest and most successful drug companies in the world: Merck (U.S.), Glaxo-Smith-Kline (U.K.), and Sanofi (France).

Kamala Harris gets her vaccine.

Photo courtesy of NIH

If no further companies succeed, the score is something like 4 successes and 126 failures.  Based on this, is the COVID vaccine a triumph of science, or a failure? Obviously, if you believe that even one of the successful programs is truly effective, you would have to agree that this is one of the most extraordinary successes in the history of medicine. In less than one year, companies were able to create, evaluate, and roll out successful vaccines, already saving hundreds of thousands of lives worldwide.

Meanwhile, Back in Education . . .

The example of COVID vaccines contrasts sharply with the way research findings are treated in education. As one example, Borman et al. (2003) reviewed research on 33 comprehensive school reform programs. Only three of these had solid evidence of effectiveness, according to the authors (one of these was our program, Success for All; see Cheung et al., in press). Actually, few of the programs failed; most had just not been evaluated adequately. Yet the response from government and educational leaders was “comprehensive school reform doesn’t work” rather than, “How wonderful! Let’s use the programs proven to work.” As a result, a federal program supporting comprehensive school reform was canceled, use of comprehensive school reform plummeted, and most CSR programs went out of operation (we survived, just barely, but the other two successful programs soon disappeared).

Similarly, the What Works Clearinghouse, and our Evidence for ESSA website (www.evidenceforessa.org), are often criticized because so few of the programs we review turn out to have significant positive outcomes in rigorous studies.

The reality is that in any field in which rigorous experiments are used to evaluate innovations, most of the innovations fail. Mature science-focused fields, like medicine and agriculture, expect this and honor it, because the only way to prevent failures is to do no experiments at all, or only flawed experiments. Without rigorous experiments, we would have no reliable successes.  Also, we learn from failures, as scientists are learning from the findings of the evaluations of all 130 of the COVID vaccines.

Unfortunately, education is not a mature science-focused field, and in our field, failure to show positive effects in rigorous experiments leads to cover-ups, despair, abandonment of proven and promising approaches, or abandonment of rigorous research itself. About 20 years ago, a popular federally-funded education program was found to be ineffective in a large, randomized experiment. Supporters of this program actually got Congress to enact legislation that forbade the use of randomized experiments to evaluate this program!

Research has improved in the past two decades, and acceptance of research has improved as well. Yet we are a long way from medicine, for example, which accepts both success and failure as part of a process of using science to improve health. In our field, we need to commit to broad scale, rigorous evaluations of promising approaches, wide dissemination of programs that work, and learning from experiments that do not (yet) show positive outcomes. In this way, we could achieve the astonishing gains that take place in medicine, and learn how to produce these gains even faster using all the knowledge acquired in experiments, successful or not.

References

Borman, G. D., Hews, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 12(2), 125-230.

Cheung, A., Xie, C., Zhang, T. & Slavin, R. E. (in press). Success for All: A quantitative synthesis of evaluations. Journal of Research on Educational Effectiveness.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just enter your email address here.

Message From NorthBay: All Students Can Learn to Read

Last July, I wrote a blog about a Year 6 (fifth grade) student in England, “Richard,” who just happened to transfer to one of our Success for All schools in spring, 2020. The school staff tested him. He had no reading skills at all. None.

Because they only had a few months to prepare him to go to secondary school, the staff decided to use our Tutoring With the Lightning Squad program with Richard for 90 minutes a day for three weeks (it’s usually used 30 minutes a day for at least 60 days). He gained 2.2 grade levels. To his delight, he could read The Hodgeheg, and was looking forward to reading Harry Potter books!

Photo credit: NorthBay Media Arts

Recently, I discovered that the gains Richard made were not unique to him.

Last summer, we were approached by John Erickson, the founder of NorthBay Education, which had a campus called NorthBay at the top of the Chesapeake. For many years, NorthBay has provided week-long outdoor education and social and emotional learning experiences for sixth graders across Maryland. However, due to Covid, this was impossible, so Erickson and NorthBay Director of Education, Rick Garber, wanted to use their staff and campus to provide an extended educational experience for students who were particularly vulnerable due to their life circumstances. Erickson and Garber wanted us (at Johns Hopkins University and the Success for All Foundation) to provide daily tutoring in reading to these students. We were delighted to agree, so in October, the program got under way. The students had to participate in remote instruction, like other Baltimore students, for six hours a day, and then had a half-hour tutoring session taught by NorthBay staff, trained by SFA coaches. The students stayed all week at the camp, and then went home over each weekend.

I can’t say that all went smoothly, but after a while NorthBay was operating well.

There were two sessions, October to January and February to June. Some students stayed for only the first session and were replaced by others. Others dropped out for a variety of reasons along the way.

I just received the test scores for the 31 students who were pretested in October and posttested on March 2, on the Gray Silent Reading Test (GSRT). Because of all the coming and going, we cannot say anything scientific about the data. We do not know if the gains were representative of all students who attended, and there was no control group. However, something extraordinary happened, and I wanted to share it.

At pretest, eight of the sixth grade students tested at the beginning first grade level (1.0), and one at 1.2. The average for these nine former non-readers at posttest was 4.1. That’s a gain of 3 grade levels in four months. Every non-reader but one reached a grade equivalent of at least 3.0. The exception got to 2.8. One got to 5.8, and one to 9.2!

I had been impressed that Richard, in England, went from zero to Hodgeheg in three weeks, but he was getting the equivalent of three tutoring sessions a day. In four months, with one session a day, every one of the nine non-reading NorthBay students went from zero to Hodgeheg, and some from zero to Harry Potter.

Other students also made astonishing gains. One went from 1.8 to 5.8. One, from 2.0 to 5.0. One from 3.0 to 7.5. One from 3.5 to 9.8.

There were many things going on in this experience, of course. The students were living at a beautiful, peaceful place for four months, with caring staff, good food, and great outdoor activities. They were experiencing the Baltimore City online curriculum, with good computer linkages and plenty of on-site assistance. Tutoring is not all they were getting.

The most important conclusion from the NorthBay experience is that these kids, mostly from very difficult backgrounds, could learn to read. The NorthBay experience suggests that they always could have learned to read under the right circumstances. Had they had the opportunity to make these gains earlier in elementary school, and reached the third grade level or better by, say, third grade, perhaps they would now be at grade level in sixth grade.

The message of NorthBay is that the problems kids face in learning to read are not usually due to anything wrong with the kids. It is not due to anything wrong with the capable and heroic teachers who have done their very best. It is the system, far beyond Baltimore, that does not allocate the funding needed to provide every struggling reader whatever is necessary to learn to read.

Today, Baltimore City Public Schools is about to receive significant funding from the American Rescue Plan Act and other sources. They cannot send everyone to NorthBay, though that would be wonderful. But they can use the new funds to create the teaching and tutoring resources that made such a difference. And so can any school or district in America.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, send your email address to thebee@bestevidence.org

Prioritize Tutoring for Low-Achieving Readers

In speaking and writing about tutoring, I am often asked about where limited tutoring resources should be concentrated.  My answer is this: “Make certain that every single child in America who needs it gets enough tutoring to be proficient in phonics. All other priorities are tied for second place.”

There are many pragmatic reasons why early reading should be the focus. First, the largest number of proven tutoring programs focus on grades K-2 or K-3 reading, and these programs tend to have spectacular impacts on measures of reading comprehension as well as phonics. Second, reading is fundamental. Very few students end up in special education, or are held back in grade, due to failure in any other subject, for example. Most other subjects depend on reading, of course. But also, a student’s academic self-esteem depends primarily on his or her self-perception as a reader.

Note that “early reading” in this case does not only mean reading in grades K-3. The great majority of students who are failing in reading in grades 4-12 do not have solid phonics skills, typically taught in grades K-2. There are no proven middle school reading programs in the U.S., but there are two in England, and both were adapted from K-2 programs, with a strong focus on phonics.

My point about focusing on phonics first is influenced by the fact that very poor reading skills (defined here as scoring “below basic” on NAEP) are very widespread, especially among disadvantaged and minority students. On the 2019 4th grade NAEP, 34% of all students, but 52% of Black students and 45% of Hispanic students, scored below basic. At 8th grade, it’s 27% below basic for all, 46% for Black students, and 37% for Hispanic students. Using these numbers, I’d estimate that 9.5 million elementary and 7.3 million secondary students are scoring below basic on NAEP reading, and surely need tutoring. While we wait for someone to create and evaluate secondary reading tutoring programs, could we start with the 9.5 million elementary students? This is a mighty big job in itself. We could also use proven upper-elementary tutoring programs to work with some proportion of secondary students reading far below grade level.

Even leaving aside the importance of immediate capacity, demonstrated impact, and the obvious importance of reading in elementary schools, consider the huge role of the reading gap on the most important social problem in our nation: Inequality by race and social class. In large part, racial and social class disparities cause the reading gap, but they are also caused by reading gaps. And it is possible to close the reading gaps, alongside other efforts focusing on closing economic, housing, criminal justice, and other gaps. If we made sure that every American child could read at the fifth grade level by fifth grade, the reading gap would be no longer be a major problem.

The numbers I have been citing are from 2019, before the pandemic. Now things are much worse, and there are surely a lot more than 9.5 million elementary students and 7.3 million secondary students who would score below “basic” on NAEP. Fortunately, the American Rescue Plan Act is providing substantial resources to combat Covid learning losses, and low achievement in general. Covid or no Covid, we have an opportunity and a clear pressing need to close the reading gap and provide a strong foundation for reading for today’s students.

Note that I do not mean to minimize the importance of mathematics, or of secondary school reading and math. These are crucial as well, and we need to solve these problems too. But first out of the gate, in 2021-2022, let’s make sure we use the proven tools we have at hand to solve the problem we happen to be best placed to solve, which happens to be the most important educational problem we face at this moment.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org

Getting Proven Tutoring Programs Into Widespread Practice

Over the past 20 years, there has been a major increase in the number of educational programs that have been developed, evaluated in rigorous (usually randomized) experiments, found to make a substantial difference in achievement, and then offered to schools by non-profit or for-profit organizations. Educators can easily find out about these proven programs in the federal What Works Clearinghouse, our own Evidence for ESSA website, and other sources. Yet very few of these, no matter how effective, have been widely adopted by schools. In 2015, the Every Student Succeeds Act (ESSA) defined what it means for a program to have strong, moderate, or promising evidence of effectiveness, and encouraged or even (in some cases) incentivized use of these programs. Yet even with this, few of the roughly 120,000 U.S. elementary and secondary schools regularly use any of the more than 120 proven reading or mathematics programs that meet the requirements of Evidence for ESSA and show significant positive effects.

The evidence-to-practice connection in education is much in contrast to that in, say, medicine, where medications and treatments of all kinds that have proven in rigorous experiments that they can cure or prevent diseases are usually widely adopted by medical practitioners. The recent, rapid development and successful evaluations of Covid-19 vaccines are much in the news right now, but dozens of new drugs and other treatments are validated every year, and are then adopted widely. It is certainly true that vastly more money is invested in the whole evidence-to-practice process in medicine than is true in education, but even when educational programs are found effective in randomized experiments like those required in medicine, these programs rarely enter large-scale use. Further, evidence-to-practice is common in many other fields, such as agriculture and technology. But not education.

How Tutoring Might Change Evidence-to-Practice in Education

One of the problems of evidence-to-practice in education is that we lack clear examples where programs were proven effective and then universally applied and found effective at scale. For example, evidence-to-practice was haphazard in medicine until the 20th century, when sulfa drugs, penicillin, morphine, and cures for polio, among many others, solved massive societal problems, and established the idea that research in medicine could truly bring about major change. These breakthroughs were explicitly engineered to solve health problems of great concern to the public, just as the Covid-19 vaccines were explicitly engineered to solve the pandemic.

In education, we face similar problems in the post-Covid period. Millions of students have fallen far behind due to the closure of their schools. All sorts of solutions have been proposed, but only one, tutoring, has both a solid and substantial research base and a significant number of proven, practical, cost-effective solutions.

Perhaps this is our penicillin/polio/Covid moment. We face a problem that no one can deny, of a desperate need to enable millions of students who have lost ground in the pandemic to rapidly gain in reading and math achievement. The American Rescue Plan Act (ARPA) has provided billions of dollars to solve the problem, so for once, lack of money is not an obstacle. Due to developments in educational research over the past 20 years, we now have a set of tutoring models that have been proven effective in randomized experiments. If these programs can be rapidly scaled up and applied to enough students to make a meaningful difference among struggling students nationwide, then this may serve as the example we need to establish that development, research, and dissemination can solve societal problems in education.

ProvenTutoring

My colleagues and I are organizing a demonstration exactly along these lines. Fourteen proven tutoring programs for reading and mathematics have formed a coalition, which we call ProvenTutoring. The purpose of the coalition is to make the case for proven tutoring programs, and then provide schools, districts, and states a choice of proven programs. Whichever programs are selected will then provide tutors with first-class professional development to ensure that tutoring is done well and that students receive the maximum benefit possible.

The fourteen programs had to be proven to succeed with college-educated teaching assistants as tutors (because requiring certified teachers would be impossible in a time of teacher shortages, and because evidence finds that well-supported teaching assistants get results as good as those obtained by certified teachers). Each program had to have capacity to go to substantial scale. We estimate that we can collectively serve 100,000 tutors who can serve about four million children, if all goes well.

ProvenTutoring will soon launch a website (which we will call ProvenTutoring.org) and a nationwide communications campaign to focus schools on proven tutoring as a key part of their post Covid-19 plans to combat learning losses. We will offer school districts webinars on proven tutoring, and allow users to explore specific programs to make informed choices among them. We will maintain rigorous standards of training and implementation, to make sure that the quality of implementation in practice, at scale, will be no less than the quality of implementation in the controlled experiments that established their impact.

How Successful Tutoring Could Lead to Support for Evidence-to-Practice

When districts, states, and the nation evaluate reading and math outcomes of struggling students in schools that adopted proven tutoring programs, in comparison to schools that used their ARPA money on other approaches, the outcomes should be clear. If they are, this will be a wonderful development for struggling students, and a major boost for tutoring as an intervention. However, it may also provide the example educational research needs to establish its capacity to solve big practical problems.

If ProvenTutoring is as effective as we expect, perhaps it will occur to our profession that this same strategy could apply wherever we want. This could lead to accelerated investment by government in development and evaluation of replicable programs in every crucial area of education, where robust solutions are needed. Someday, could there be ProvenAlgebra.org? ProvenScience.org? ProvenGraduation.org? Proven programs for English learners? ProvenPreschool.org? ProvenClassroomManagement.org? ProvenCivics.org? Whatever problems most need to be solved, there is no reason we cannot solve them using the same evidence-to-practice strategies that medicine, agriculture, and other fields have used so successfully for many decades.

Photo courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org

Another Way to Understand Effect Sizes

Whenever I talk to educators and mention effect sizes, someone inevitably complains. “We don’t understand effect sizes,” they say. I always explain that you don’t have to understand exactly what effect sizes are, but if you do know that more of them are good and less of them are bad, assuming that the research from which they came is of equal quality, then why do you have to know precisely what they are? Sometimes I mention the car reliability rating system Consumer Reports uses, with full red circles at the top and full black circles at the bottom. Does anyone understand how they arrived at those ratings? I don’t even know, but I don’t care, because like everyone else, what I do know is that I don’t want a car with a reliability rating in the black.

People always tell me that they would like it better if I’d use “additional months of gain.” I do this when I have to, but I really do not like it, because these “months of gain” do not really mean very much, and they work very differently at the early elementary grades than they do in high schools.

So here is an idea that some people might find useful. The National Assessment of Educational Progress (NAEP) uses reading and math scales that have a theoretical standard deviation of 50. So an effect size of, say, +0.20 can be expressed as a gain equivalent to a NAEP score gain of +10 (0.20 x 50 = 10) points.  That’s not really interesting yet, because most people also don’t know what NAEP scores mean.

But here’s another way to use such data that might be more fun and easier to understand. I think people could understand and care about their state’s rank on NAEP scores. So for example, the highest-scoring state on 4th grade reading is Massachusetts, with a NAEP reading score of 231 in 2019. What if the 13th state, Nebraska (222), adopted a great reading program statewide, and it gained an average effect size of +0.20. That’s equivalent to 10 NAEP points. Such a gain in effect size would make Nebraska score one point ahead of Massachusetts (if Massachusetts didn’t change). Number 1!

If we learned to speak in terms of how many ranks states would gain if they gained a given effect size, I wonder if that would give educators more understanding and respect for the findings of experiments. Even fairly small effect sizes, if replicated across a whole state, could propel a state past its traditional rivals. For example, 26th ranked Wisconsin (220) could equal neighboring 12th ranked Minnesota (222) with a statewide reading effect size gain of only +0.04. As a practical matter, Wisconsin could increase its fourth grade test scores by an effect size of +0.04, perhaps by using a program with an effect size of +0.20 with (say) the lowest-achieving fifth of its fourth graders.

If only one could get states thinking this way, the meaning and importance of effect sizes would soon become clear. And as a side benefit, perhaps if Wisconsin invested its enthusiasm and money in a “Beat Minnesota” reading campaign, as it does to try to beat the University of Minnesota’s football team, Wisconsin’s students might actually benefit. I can hear it now:

            On Wisconsin, On Wisconsin,

            Raise effect size high!

            We are not such lazy loafers

            We can beat the Golden Gophers

            Point-oh-four or point-oh-eight

            We’ll surpass them, just you wait!

           

Well, a nerd can dream, can’t he?

_______

            Note:  No states were harmed in the writing of this blog.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org

The American Rescue Plan Can Rescue Education, If We Use It to Fund What Works

The American Rescue Plan was passed in the U.S. Congress this week. This $1.9 trillion bill provides funding for a lot of things I care about as a citizen, but as an educator, I’d like to focus on the portion of it allocated to healing Covid learning loss. This is $29 billion, or roughly double the usual amount spent annually on Title I. This is a major investment in the students whose educations were harmed the most by Covid school closures. These are mostly disadvantaged students and rural students who could not gain access to remote teaching, or who did not have assistance at home to take advantage of remote instruction. Data from all over the country is showing the educational damage these children have sustained.

Clearly, the new money in the ARP could make a substantial difference in the achievement and adjustment of all students returning to in-person schooling. But if educational research tells us anything at all, it tells us these two things:

  1. Making a big difference in educational outcomes costs money.
  2. However, lots of well-meaning uses of money do not make any perceptible difference in outcomes.

Of course, the only way to tell effective uses of new funds from ineffective uses is through rigorous research.

One of the unusual aspects of the ARP education funding is that the legislation is not very specific about how the money is to be used. This is due in part to the fact the ARP was passed using a reconciliation procedure that does not allow for much specificity. The U.S. Department of Education will be drafting guidelines for the money soon, but these guidelines are likely to be relatively flexible, because the legislation itself was not very specific.

This flexibility is likely to allow anything from very good uses of money to very poor uses. My guess is that state and district leaders, and individual principals, will have plenty of freedom to use plenty of money. How novel!

I hope states and districts will use this opportunity to clearly define what is most important to accomplish in their post-Covid planning and then insist on choosing programs, practices, and policies based on the best evidence available. This time, educators will have the opportunity to use research-proven programs not because Congress or the U.S. Department of Education tells them to, but because they care about the learning and emotional well-being of their students.

In the period following the passage of the 2015 Every Student Succeeds Act (ESSA), state, district, and building leaders learned how to use services such as the What Works Clearinghouse and our www.evidenceforessa.org website to find out the strength of the evidence supporting various programs. I hope schools will continue to use these resources to select programs that have been proven effective. I’ve written many times about the importance of using proven tutoring programs, and this is indeed the most effective strategy by far for students who are far behind in reading or math. But there are many other approaches proven to be effective, especially for disadvantaged students. There is good evidence of effectiveness not only for classroom approaches to reading and math, but also programs for creative writing, science, social-emotional learning, early childhood education, and much more. The ARP funding allows schools to invest in proven programs and find out for themselves whether they work. ARP money will not be around forever, but wouldn’t it be a great use of the money to find out what works, so that when things return to normal, school and district leaders will know more than ever before what works and what doesn’t for their particular students and their particular schools?

In the first months after all schools open for in-person learning, schools are sure to be thinking in emergency mode, about investments in tutoring and other relatively expensive but highly effective strategies. But the damage Covid has done will have long-lasting impacts, and even if schools use proven tutoring methods to help the students at the greatest risk, it is also important to build for the long haul for all students, using proven programs of all kinds. Wouldn’t it be wonderful if the terrible experience we have all been through leads to a more rational, evidence-driven approach to schooling, creating a lasting benefit not only for today’s children, but for future generations who will receive better educations than they would have before Covid?

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org

Summer 2021 Re-Imagined: A Grand Opening to a Successful Year

If you follow my blogs, you’ll note that I have been writing recently about the ineffectiveness of summer school (here, here, and here). Along with colleagues, I wrote a review of research on summer school, which is summarized here. The reason for the ineffectiveness of summer school, I proposed, is that when summer school resembles regular school, it can be boring. Kids are sitting in school while their friends are playing outside. As a result, attendance in summer school programs intended to help struggling students can be very poor, and the motivation of those who do attend may also be poor.

However, there are two major exceptions to the otherwise dismal outcomes of studies of summer school. One is a Los Angeles study by Schacter & Jo (2005), and the other is a study by Zvoch & Stevens (2013), done in a small city in the Northwest.

Both of these studies focused on disadvantaged students in grades 1 or K-1. Both provided small-group tutoring interventions. Schacter & Jo (2005) gave students phonics instruction in groups of 15, followed by small-group tutoring. The Gates-MacGinitie reading effect size was +1.16. Zvoch & Stevens (2013) also provided group phonics instruction followed by tutoring to groups of 3 to 5. The effect size on DIBELS measures was +0.69.

The large effect sizes seen in these two studies contrast sharply with all the other studies of summer classroom programs, which had a mean effect size near zero. What this suggests is that the best instructional use of summer may be to provide one-to-one or small-group tutoring to struggling students.

In summer, 2021, the rationale for summertime tutoring is particularly strong. If current trends maintain, most teachers will have received Covid vaccines by summer, and increasing numbers of schools will open by the end of the current semester. To close schools that could be open for summer vacation seems a waste. Also, assuming the American Rescue Plan is passed (as expected), it will make a great deal of money available to serve students who have lost ground due to Covid school closures, so schools will be able to afford to pay for tutoring during the summer.

The problem with summer school is that it cannot be made mandatory, and many students will not want to attend. However, in summer 2021, providing tutoring during the summer for students who do choose to attend (and keep attending regularly) could be of great value, even if most students who need tutoring do not attend. The reason is that there are so many students who will need tutoring in September, 2021, that not all of them can be tutored right away. Providing tutoring in the summer gives some students a full dose of tutoring before school officially opens, so that schools will not be under pressure to tutor more students than they are able to serve in fall, 2021.

How Can Summer Tutoring Work?

Summertime allows schools to provide more hours of tutoring each day than would be possible during the school year. For example, teaching and tutoring were provided 2 hours a day for 7 weeks in the Schacter & Jo (2005) study, and 3½ hours per day for 5 weeks in the Zvoch & Stevens (2013) study. If tutoring were alternated with sports or music or other fun activities, one might imagine providing two or three tutoring sessions each day, for as many as 8 weeks during the summer.

These sessions might be offered during a half day, so teachers and teaching assistants might teach one morning and one afternoon session each day. In fact, tutors might provide three two-hour sessions, and reach even more students.

The tutoring methods should be ones proven effective in rigorous experiments. While any whole-class teaching should be done by teachers, teaching assistants can be trained to be excellent tutors. They will need extensive training and in-class coaching, but this is worthwhile, especially because most of these tutors will continue working with additional students during the school day starting in the fall.

Tutoring in summer 2021 will provide a pilot opportunity for schools and districts to hit the ground running in September. It will provide time and resources for providers of tutoring to greatly increase their scale of operations. And it may attract students who have been out of school for many months by offering small group, supportive tutoring with caring tutors, to help ease the transition back into school.

Summertime need not be a time for summertime blues. Instead, it can serve as a “grand opening” for a successful re-entry to school for millions of students.

References

Schacter, J., & Jo, B. (2005). Learning when school is not in session: A reading summer day-camp intervention to improve the achievement of exiting first-grade students who are economically disadvantaged. Journal of Research in Reading, 28, 158-169. Doi:10.111/j.1467-9817.2005.00260.x

Zvoch, K., & Stevens, J. J. (2013). Summer school effects in a randomized field trial. Early Childhood Research Quarterly, 28(1), 24-32. Doi:10.1016/j.ecresq.2012,05.002

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org

The Role of Research and Development in Post-Covid Education

Everyone knows that during World War II, the pace of innovation greatly accelerated. Computers, rocketry, jets, sonar, radar, microwaves, aerosol cans, penicillin, and morphine were among the many wartime developments. What unites these innovations, of course, is that each was developed to solve an urgent problem important to the war effort, and all of them later tuned out to have revolutionary benefits for civilian use. Yet these advances could not have taken place so quickly if not for the urgent need for innovations and the massive resources devoted to them.

Crisis can be the catalyst for innovation.

Today, we face Covid, a dire medical crisis, and investments of massive resources have produced vaccines in record time. However, the Covid pandemic has also created an emergency in education, as millions of children are experiencing educational losses due to school closures. The Institute for Education Sciences (IES) has announced a grants program to respond to the Covid crisis, but at the usual pace, the grants will only lead to practical solutions in many years when (we fervently hope) the crisis will be over.

I would argue that in this perilous time, research in education should focus on urgent practical problems that could have a significant impact within, say, the next year or two on the problems of students who are far below grade level in essential skills because of Covid school closures, or for other reasons:

1. Tutoring. Yes, of course I was going to start with tutoring. The biggest problem in tutoring is that while we have many proven programs for elementary reading, especially for grades K-3, we have far fewer proven programs ready for prime time in the upper elementary grades, and none at all in middle or high school reading. Studies in England have found positive effects of tutoring in their equivalent of middle school, but none of these exist in the U.S. In mathematics, there are few proven tutoring programs in elementary school, and just one I know of for middle school, and one for high school.

How could research funding produce new tutoring programs for middle and high school reading, and for math at all grade levels, in such a short time?  Simple. First, there are already tutoring programs for reading and math at all grade levels, but few have been successfully evaluated, or (in most cases) ever evaluated at all in rigorous experiments. So it would be important to fund evaluations of particularly promising programs that are already working at significant scale.

Another means of rapidly discovering effective tutoring programs would be to fund programs that have been successful in certain grade levels to quickly create programs for adjacent grades. For example, a program proven effective in grades 2-3 should be able to be significantly modified to work in grades 4-5. One that works in grades 4-5 could be modified for use in middle school. Programs proven effective in reading might be modified for use in mathematics at the same grade level, or vice versa. Many programs with successful programs in some grade levels have the staff and experience to quickly create programs in adjacent grade levels.

Also, it might be possible for developers of successful classwide technology programs to create and pilot tutoring models using similar software, but adding the assistance of a tutor for groups of one to four students, perhaps in collaboration with experts on tutoring.

2. Approaches other than tutoring.  There are many effective reading and math programs of all kinds, not just tutoring, that have proven their effectiveness (see www.evidenceforessa.org). Such programs might be ready to go as they are, and others could be evaluated in a form appropriate to the current emergency. Very few programs other than tutoring obtain effect sizes like those typical of the best tutoring programs, but classwide programs with modest effect sizes serve many more students than tutoring programs do. Also, classroom programs might be evaluated for their capacity to maintain gains made due to tutoring.

Tutoring or non-tutoring programs that already exist at scale, or that could be quickly adapted from proven programs, might be ready for rigorous, third-party evaluations as soon as fall, 2021. These programs should be evaluated using rigorous, third-party evaluations, with all programs at a given grade level using identical procedures and measures. In this way, it should be possible to have many new, proven programs by the end of the 2021-2022 school year, ready for dissemination in fall, 2022. This would be in time to greatly add capacity to serve the millions of students who need proven programs to help them make rapid progress toward grade level.

A research program of this kind could be expensive, and it may not provide theoretical breakthroughs. However, given the substantial and obvious need, and the apparent willingness of government to provide major resources to combat Covid learning losses, such a research effort might be feasible. If it were to take place, it might build excitement about R & D as a practical means of enhancing student achievement. And if even a quarter of the experiments found sizable positive impacts, this would add substantially to our armamentarium of proven strategies for struggling students.

There is an old saying in social work: “Never let a good crisis go to waste.” As in World War II, the educational impacts of the Covid pandemic present educational research with a crisis that we must solve, but if we can solve any portion of this problem, this will create benefits for generations of children long after Covid has faded into a distant memory.

Photo credit: User Messybeast on en.wikipedia, CC BY-SA 3.0 <http://creativecommons.org/licenses/by-sa/3.0/>, via Wikimedia Commons

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org

Getting Below the Surface to Understand Disappointing Outcomes

Long ago, I toured West Germany, visiting some family friends near Hanover. They suggested I go see Duderstadt, a picturesque town nearby (see picture of it below).

My wife, Nancy, and I drove into Duderstadt and walked around. It was indeed gorgeous, but very strange. Not a person was in sight. Every shop was closed. In the center of the town was a beautiful church. We reasoned that churches are always open. We walked to the church door, I stretched out my hand to open it, but inches away the door burst open. An entire wedding party streamed out into the street. The church was packed to the rafters with happy people, now following the bride and groom out of the church. Mystery solved.

If social scientists came to Duderstadt when we did but failed to see the wedding, they might make all sorts of false conclusions. An economist might see the empty shops and conclude that the economy of rural Germany is doomed, due to low productivity. A demographer might agree and blame this on the obviously declining workforce. But looking just the thickness of a church door beneath the surface, all could immediately understand what was happening.

My point here is a simple one. I am a quant. I believe in numbers and rigorous research designs. But at the same time, I also want to understand what is really going on, and the main numbers rarely tell the whole story.

I was thinking about this when I read the rather remarkable study by Carolyn Heinrich and her colleagues (2010), cited in my two previous blogs. Like many other researchers, she and her colleagues found near-zero impacts for Supplemental Educational Services. At the time this study took place, this was a surprise. How could all that additional instructional time after school not make a meaningful difference?

But instead of just presenting the overall (bad) findings, she poked around town, so to speak, to find out what was going on.

What she found was appalling, but also perfectly logical. Most eligible middle and high school students in Milwaukee who were offered after-school programs either failed to sign up, or if they did sign up, did not attend even a single day, or if they did attend a single day, they attended irregularly, thereafter. And why did they not sign up or attend? Most programs offered attractive incentives, such as iPods, very popular at the time, so about half of the eligible students did sign up, at least. But after the first day, when they got their incentives, students faced drudgery. Heinrich et al. cite evidence that most instruction was either teachers teaching immobile students, or students doing unsupervised worksheets. Heinrich et al.’s technical report had a sentence (dropped in the published report), which I quoted previously, but will quote again here: “One might also speculate that parents and students are, in fact, choosing rationally in not registering for or attending SES.”

A study of summer school by Borman & Dowling (2006) made a similar observation. K-1 students in Baltimore were randomly assigned to have an opportunity to attend three years of summer school. The summer school sessions included 7 weeks of 6-hour a day activities, including 2 ½ hours of reading and writing instruction, plus sports, art, and other enrichment activities. Most eligible students (79%) signed up and attended in the first summer, but fewer did so in the second summer (69%) and even fewer in the third summer (42%). The analyses focused on the students who were eligible for the first and second summers, and found no impact on reading achievement. There was a positive effect for the students who did show up and attended for two summers.

Many studies of summer school, after school, and SES programs overall (including both) have just reported the disappointing outcomes without exploring why they occurred. Such reports are important, if well done, but they offer little understanding of why. Could after school or summer school programs work better if we took into account the evidence on why they usually fail? Perhaps. For example, in my previous blog, I suggested that extended-time programs might do better if they provided one-to-one, or small-group tutoring. However, there is only suggestive evidence that this might be true, and there are good reasons that it might not be, because of the same attendance and motivation problems that may doom any program, no matter how good, when struggling students go to school during times when their friends are outside playing.

Econometric production function models predicting that more instruction leads to more learning are useless unless we take into account what students are actually being provided in extended-time programs and what their motivational state is likely to be. We have to look a bit below the surface to explain why disappointing outcomes are so often achieved, so we can avoid mistakes and repeat successes, rather than making the same mistakes over and over again.

Correction

My recent blog, “Avoiding the Errors of Supplemental Educational Services,” started with a summary of the progress of the Learning Recovery Act.  It was brought to my attention that my summary was not correct.  In fact, the Learning Recovery Act has been introduced in Congress, but is not part of the current reconciliation proposal moving through Congress and has not become law. The Congressional action cited in my last blog was referring to a non-binding budget resolution, the recent passage of which facilitated the creation of the $1.9 trillion reconciliation bill that is currently moving through Congress. Finally, while there is expected to be some amount of funding within that current reconciliation bill to address the issues discussed within my blog, reconciliation rules will prevent the Learning Recovery Act from being included in the current legislation as introduced. I apologize for this error.

References

Borman, G. D., & Dowling, N. M. (2006). Longitudinal achievement effects of multiyear summer school: Evidence from the Teach Baltimore randomized field trial. Educational Evaluation and Policy Analysis, 28(1), 25–48. https://doi.org/10.3102/01623737028001025

Heinrich, C. J., Meyer, R., H., & Whitten, G. W. (2010). Supplemental Education Services under No Child Left Behind: Who signs up and what do they gain? Education Evaluation and Policy Analysis, 32, 273-298.

Photo credit: Amrhingar, CC BY-SA 3.0, via Wikimedia Commons

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org