Getting Proven Tutoring Programs Into Widespread Practice

Over the past 20 years, there has been a major increase in the number of educational programs that have been developed, evaluated in rigorous (usually randomized) experiments, found to make a substantial difference in achievement, and then offered to schools by non-profit or for-profit organizations. Educators can easily find out about these proven programs in the federal What Works Clearinghouse, our own Evidence for ESSA website, and other sources. Yet very few of these, no matter how effective, have been widely adopted by schools. In 2015, the Every Student Succeeds Act (ESSA) defined what it means for a program to have strong, moderate, or promising evidence of effectiveness, and encouraged or even (in some cases) incentivized use of these programs. Yet even with this, few of the roughly 120,000 U.S. elementary and secondary schools regularly use any of the more than 120 proven reading or mathematics programs that meet the requirements of Evidence for ESSA and show significant positive effects.

The evidence-to-practice connection in education is much in contrast to that in, say, medicine, where medications and treatments of all kinds that have proven in rigorous experiments that they can cure or prevent diseases are usually widely adopted by medical practitioners. The recent, rapid development and successful evaluations of Covid-19 vaccines are much in the news right now, but dozens of new drugs and other treatments are validated every year, and are then adopted widely. It is certainly true that vastly more money is invested in the whole evidence-to-practice process in medicine than is true in education, but even when educational programs are found effective in randomized experiments like those required in medicine, these programs rarely enter large-scale use. Further, evidence-to-practice is common in many other fields, such as agriculture and technology. But not education.

How Tutoring Might Change Evidence-to-Practice in Education

One of the problems of evidence-to-practice in education is that we lack clear examples where programs were proven effective and then universally applied and found effective at scale. For example, evidence-to-practice was haphazard in medicine until the 20^th century, when sulfa drugs, penicillin, morphine, and cures for polio, among many others, solved massive societal problems, and established the idea that research in medicine could truly bring about major change. These breakthroughs were explicitly engineered to solve health problems of great concern to the public, just as the Covid-19 vaccines were explicitly engineered to solve the pandemic.

In education, we face similar problems in the post-Covid period. Millions of students have fallen far behind due to the closure of their schools. All sorts of solutions have been proposed, but only one, tutoring, has both a solid and substantial research base and a significant number of proven, practical, cost-effective solutions.

Perhaps this is our penicillin/polio/Covid moment. We face a problem that no one can deny, of a desperate need to enable millions of students who have lost ground in the pandemic to rapidly gain in reading and math achievement. The American Rescue Plan Act (ARPA) has provided billions of dollars to solve the problem, so for once, lack of money is not an obstacle. Due to developments in educational research over the past 20 years, we now have a set of tutoring models that have been proven effective in randomized experiments. If these programs can be rapidly scaled up and applied to enough students to make a meaningful difference among struggling students nationwide, then this may serve as the example we need to establish that development, research, and dissemination can solve societal problems in education.

ProvenTutoring

My colleagues and I are organizing a demonstration exactly along these lines. Fourteen proven tutoring programs for reading and mathematics have formed a coalition, which we call ProvenTutoring. The purpose of the coalition is to make the case for proven tutoring programs, and then provide schools, districts, and states a choice of proven programs. Whichever programs are selected will then provide tutors with first-class professional development to ensure that tutoring is done well and that students receive the maximum benefit possible.

The fourteen programs had to be proven to succeed with college-educated teaching assistants as tutors (because requiring certified teachers would be impossible in a time of teacher shortages, and because evidence finds that well-supported teaching assistants get results as good as those obtained by certified teachers). Each program had to have capacity to go to substantial scale. We estimate that we can collectively serve 100,000 tutors who can serve about four million children, if all goes well.

ProvenTutoring will soon launch a website (which we will call ProvenTutoring.org) and a nationwide communications campaign to focus schools on proven tutoring as a key part of their post Covid-19 plans to combat learning losses. We will offer school districts webinars on proven tutoring, and allow users to explore specific programs to make informed choices among them. We will maintain rigorous standards of training and implementation, to make sure that the quality of implementation in practice, at scale, will be no less than the quality of implementation in the controlled experiments that established their impact.

How Successful Tutoring Could Lead to Support for Evidence-to-Practice

When districts, states, and the nation evaluate reading and math outcomes of struggling students in schools that adopted proven tutoring programs, in comparison to schools that used their ARPA money on other approaches, the outcomes should be clear. If they are, this will be a wonderful development for struggling students, and a major boost for tutoring as an intervention. However, it may also provide the example educational research needs to establish its capacity to solve big practical problems.

If ProvenTutoring is as effective as we expect, perhaps it will occur to our profession that this same strategy could apply wherever we want. This could lead to accelerated investment by government in development and evaluation of replicable programs in every crucial area of education, where robust solutions are needed. Someday, could there be ProvenAlgebra.org? ProvenScience.org? ProvenGraduation.org? Proven programs for English learners? ProvenPreschool.org? ProvenClassroomManagement.org? ProvenCivics.org? Whatever problems most need to be solved, there is no reason we cannot solve them using the same evidence-to-practice strategies that medicine, agriculture, and other fields have used so successfully for many decades.

Photo courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

Another Way to Understand Effect Sizes

Whenever I talk to educators and mention effect sizes, someone inevitably complains. “We don’t understand effect sizes,” they say. I always explain that you don’t have to understand exactly what effect sizes are, but if you do know that more of them are good and less of them are bad, assuming that the research from which they came is of equal quality, then why do you have to know precisely what they are? Sometimes I mention the car reliability rating system Consumer Reports uses, with full red circles at the top and full black circles at the bottom. Does anyone understand how they arrived at those ratings? I don’t even know, but I don’t care, because like everyone else, what I do know is that I don’t want a car with a reliability rating in the black.

People always tell me that they would like it better if I’d use “additional months of gain.” I do this when I have to, but I really do not like it, because these “months of gain” do not really mean very much, and they work very differently at the early elementary grades than they do in high schools.

So here is an idea that some people might find useful. The National Assessment of Educational Progress (NAEP) uses reading and math scales that have a theoretical standard deviation of 50. So an effect size of, say, +0.20 can be expressed as a gain equivalent to a NAEP score gain of +10 (0.20 x 50 = 10) points.  That’s not really interesting yet, because most people also don’t know what NAEP scores mean.

But here’s another way to use such data that might be more fun and easier to understand. I think people could understand and care about their state’s rank on NAEP scores. So for example, the highest-scoring state on 4^th grade reading is Massachusetts, with a NAEP reading score of 231 in 2019. What if the 13^th state, Nebraska (222), adopted a great reading program statewide, and it gained an average effect size of +0.20. That’s equivalent to 10 NAEP points. Such a gain in effect size would make Nebraska score one point ahead of Massachusetts (if Massachusetts didn’t change). Number 1!

If we learned to speak in terms of how many ranks states would gain if they gained a given effect size, I wonder if that would give educators more understanding and respect for the findings of experiments. Even fairly small effect sizes, if replicated across a whole state, could propel a state past its traditional rivals. For example, 26^th ranked Wisconsin (220) could equal neighboring 12^th ranked Minnesota (222) with a statewide reading effect size gain of only +0.04. As a practical matter, Wisconsin could increase its fourth grade test scores by an effect size of +0.04, perhaps by using a program with an effect size of +0.20 with (say) the lowest-achieving fifth of its fourth graders.

If only one could get states thinking this way, the meaning and importance of effect sizes would soon become clear. And as a side benefit, perhaps if Wisconsin invested its enthusiasm and money in a “Beat Minnesota” reading campaign, as it does to try to beat the University of Minnesota’s football team, Wisconsin’s students might actually benefit. I can hear it now:

            On Wisconsin, On Wisconsin,

            Raise effect size high!

            We are not such lazy loafers

            We can beat the Golden Gophers

            Point-oh-four or point-oh-eight

            We’ll surpass them, just you wait!

           

Well, a nerd can dream, can’t he?

_______

            Note:  No states were harmed in the writing of this blog.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

The American Rescue Plan Can Rescue Education, If We Use It to Fund What Works

The American Rescue Plan was passed in the U.S. Congress this week. This $1.9 trillion bill provides funding for a lot of things I care about as a citizen, but as an educator, I’d like to focus on the portion of it allocated to healing Covid learning loss. This is $29 billion, or roughly double the usual amount spent annually on Title I. This is a major investment in the students whose educations were harmed the most by Covid school closures. These are mostly disadvantaged students and rural students who could not gain access to remote teaching, or who did not have assistance at home to take advantage of remote instruction. Data from all over the country is showing the educational damage these children have sustained.

Clearly, the new money in the ARP could make a substantial difference in the achievement and adjustment of all students returning to in-person schooling. But if educational research tells us anything at all, it tells us these two things:

1. Making a big difference in educational outcomes costs money.
2. However, lots of well-meaning uses of money do not make any perceptible difference in outcomes.

Of course, the only way to tell effective uses of new funds from ineffective uses is through rigorous research.

One of the unusual aspects of the ARP education funding is that the legislation is not very specific about how the money is to be used. This is due in part to the fact the ARP was passed using a reconciliation procedure that does not allow for much specificity. The U.S. Department of Education will be drafting guidelines for the money soon, but these guidelines are likely to be relatively flexible, because the legislation itself was not very specific.

This flexibility is likely to allow anything from very good uses of money to very poor uses. My guess is that state and district leaders, and individual principals, will have plenty of freedom to use plenty of money. How novel!

I hope states and districts will use this opportunity to clearly define what is most important to accomplish in their post-Covid planning and then insist on choosing programs, practices, and policies based on the best evidence available. This time, educators will have the opportunity to use research-proven programs not because Congress or the U.S. Department of Education tells them to, but because they care about the learning and emotional well-being of their students.

In the period following the passage of the 2015 Every Student Succeeds Act (ESSA), state, district, and building leaders learned how to use services such as the What Works Clearinghouse and our www.evidenceforessa.org website to find out the strength of the evidence supporting various programs. I hope schools will continue to use these resources to select programs that have been proven effective. I’ve written many times about the importance of using proven tutoring programs, and this is indeed the most effective strategy by far for students who are far behind in reading or math. But there are many other approaches proven to be effective, especially for disadvantaged students. There is good evidence of effectiveness not only for classroom approaches to reading and math, but also programs for creative writing, science, social-emotional learning, early childhood education, and much more. The ARP funding allows schools to invest in proven programs and find out for themselves whether they work. ARP money will not be around forever, but wouldn’t it be a great use of the money to find out what works, so that when things return to normal, school and district leaders will know more than ever before what works and what doesn’t for their particular students and their particular schools?

In the first months after all schools open for in-person learning, schools are sure to be thinking in emergency mode, about investments in tutoring and other relatively expensive but highly effective strategies. But the damage Covid has done will have long-lasting impacts, and even if schools use proven tutoring methods to help the students at the greatest risk, it is also important to build for the long haul for all students, using proven programs of all kinds. Wouldn’t it be wonderful if the terrible experience we have all been through leads to a more rational, evidence-driven approach to schooling, creating a lasting benefit not only for today’s children, but for future generations who will receive better educations than they would have before Covid?

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

Summer 2021 Re-Imagined: A Grand Opening to a Successful Year

If you follow my blogs, you’ll note that I have been writing recently about the ineffectiveness of summer school (here, here, and here). Along with colleagues, I wrote a review of research on summer school, which is summarized here. The reason for the ineffectiveness of summer school, I proposed, is that when summer school resembles regular school, it can be boring. Kids are sitting in school while their friends are playing outside. As a result, attendance in summer school programs intended to help struggling students can be very poor, and the motivation of those who do attend may also be poor.

However, there are two major exceptions to the otherwise dismal outcomes of studies of summer school. One is a Los Angeles study by Schacter & Jo (2005), and the other is a study by Zvoch & Stevens (2013), done in a small city in the Northwest.

Both of these studies focused on disadvantaged students in grades 1 or K-1. Both provided small-group tutoring interventions. Schacter & Jo (2005) gave students phonics instruction in groups of 15, followed by small-group tutoring. The Gates-MacGinitie reading effect size was +1.16. Zvoch & Stevens (2013) also provided group phonics instruction followed by tutoring to groups of 3 to 5. The effect size on DIBELS measures was +0.69.

The large effect sizes seen in these two studies contrast sharply with all the other studies of summer classroom programs, which had a mean effect size near zero. What this suggests is that the best instructional use of summer may be to provide one-to-one or small-group tutoring to struggling students.

In summer, 2021, the rationale for summertime tutoring is particularly strong. If current trends maintain, most teachers will have received Covid vaccines by summer, and increasing numbers of schools will open by the end of the current semester. To close schools that could be open for summer vacation seems a waste. Also, assuming the American Rescue Plan is passed (as expected), it will make a great deal of money available to serve students who have lost ground due to Covid school closures, so schools will be able to afford to pay for tutoring during the summer.

The problem with summer school is that it cannot be made mandatory, and many students will not want to attend. However, in summer 2021, providing tutoring during the summer for students who do choose to attend (and keep attending regularly) could be of great value, even if most students who need tutoring do not attend. The reason is that there are so many students who will need tutoring in September, 2021, that not all of them can be tutored right away. Providing tutoring in the summer gives some students a full dose of tutoring before school officially opens, so that schools will not be under pressure to tutor more students than they are able to serve in fall, 2021.

How Can Summer Tutoring Work?

Summertime allows schools to provide more hours of tutoring each day than would be possible during the school year. For example, teaching and tutoring were provided 2 hours a day for 7 weeks in the Schacter & Jo (2005) study, and 3½ hours per day for 5 weeks in the Zvoch & Stevens (2013) study. If tutoring were alternated with sports or music or other fun activities, one might imagine providing two or three tutoring sessions each day, for as many as 8 weeks during the summer.

These sessions might be offered during a half day, so teachers and teaching assistants might teach one morning and one afternoon session each day. In fact, tutors might provide three two-hour sessions, and reach even more students.

The tutoring methods should be ones proven effective in rigorous experiments. While any whole-class teaching should be done by teachers, teaching assistants can be trained to be excellent tutors. They will need extensive training and in-class coaching, but this is worthwhile, especially because most of these tutors will continue working with additional students during the school day starting in the fall.

Tutoring in summer 2021 will provide a pilot opportunity for schools and districts to hit the ground running in September. It will provide time and resources for providers of tutoring to greatly increase their scale of operations. And it may attract students who have been out of school for many months by offering small group, supportive tutoring with caring tutors, to help ease the transition back into school.

Summertime need not be a time for summertime blues. Instead, it can serve as a “grand opening” for a successful re-entry to school for millions of students.

References

Schacter, J., & Jo, B. (2005). Learning when school is not in session: A reading summer day-camp intervention to improve the achievement of exiting first-grade students who are economically disadvantaged. Journal of Research in Reading, 28, 158-169. Doi:10.111/j.1467-9817.2005.00260.x

Zvoch, K., & Stevens, J. J. (2013). Summer school effects in a randomized field trial. Early Childhood Research Quarterly, 28(1), 24-32. Doi:10.1016/j.ecresq.2012,05.002

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org.