A Marshall Plan for Post-COVID-19 Recovery

In World War II, my father was in the U.S. Navy.  In 1945, he was serving on a specially outfitted destroyer preparing for the invasion of Japan.  He always claimed that had the invasion gone forward, he would have been doomed.  He was in charge of his ship’s “radio-radar countermeasures,” new technology that would have been able to blind the radio and radar of the Japanese Navy so that there would have been only one ship they could detect: his.  Fortunately, the Japanese surrendered on October 14, before the invasion was set to begin.

I’m sure you’ve seen the famous picture of jubilant crowds in New York celebrating the surrender.  My father’s experience was different.  He was landed in Tokyo as part of the occupation forces.  He described Tokyo as a city whose former industrial and military areas had not one stone standing on another.  Many others have described similar scenes in Europe and Asia.  Like all servicemen, he was relieved that the war had ended, that he had survived.  But the extent of the destruction was horrifying, even to the victors.  How could a normal country grow back from this desert?

But it did.  Even the countries that suffered the greatest destruction were able, with American and other help, to rebuild, and ultimately to prosper.  The U.S. Marshall Plan, in particular, was a far-sighted investment in reconstruction that led the way in enabling destroyed countries to rebuild their societies and their economies.

Now we face another challenge, the COVID-19 pandemic.  I write from Baltimore at the point of inflection, when new cases of the disease have started to decline.  But it will still take a long time for everything to return to normal.  Compared to the death and destruction of World War II, COVID-19 is far less of a challenge, but day to day, it does not feel that way.  And unlike VJ Day, there will not be a day when it all ends, when everyone knows they are safe.

For Americans, World War II was awful, but it was far away.  Life went on.  Schools and universities were open.  COVID-19 is different, because it profoundly affects the daily life of every American.  Most relevant to the readers of this blog, COVID-19 is severely interrupting the education of a generation.  This is a particular problem, of course, for disadvantaged students, whose parents are more likely to get the virus, who are less likely to have technology at home, and who were often already having difficulties in school.  How will we rebuild?  How will we help students regain the learning and the sense of security they once had?  And can we use this sobering experience to make lasting improvements in education?

Educational leaders are starting to think about what comes next.  Most are overwhelmed with the present, trying to figure out how, for example, to use distance learning to substitute for in-person school.  But anyone who has a child, or knows a child, or has ever been a child or parent, knows that distance education is not going to be enough, certainly not for most children, even in areas where students have plenty of computers, access to the Internet, excellent support from teachers teaching online, and parents who are willing and able to fill in to make sure that students are taking full advantage of whatever the school is providing their children. There will be happy exceptions, but there is a reason that homeschooling is rare.  When the schools open, hopefully next September, there will be a huge job to be done to repair the damage COVID-19 will have done to the educational futures of the 50 million U.S. children in grades PK to 12, as well as hundreds of millions more throughout the world.

One thing that seems highly likely is that when schools do open, they will open into an economic recession.  Currently, there is much concern for people who have lost their jobs, and initial efforts by the federal government have focused on propping up businesses and helping people who were employed, but happened to work for companies that had to close due to the pandemic.  This is essential, of course.  However, there is another problem that also needs attention: people who are just entering the workforce.  Since the Great Depression, economists have known how to respond to such crises: invest massively in people, to jump start the economy.

I would propose a solution that could help both with the schools and the recession. Schools should hire, train, and deploy large numbers of recent (and not so recent) college graduates as tutors, and in other essential roles in schools.

There is no intervention known that has an impact larger than that of tutoring.  One-to- one is most effective, but one-to-small group can also make a substantial difference in reading and mathematics performance in elementary and middle schools, and reaches many more students at a much lower cost per student.  Our recent research reviews (Baye et al., 2019; Neitzel et al., 2020; Pellegrini et al., 2020) tell us that teaching assistants, with proven materials and expert professional development, can obtain outcomes as good as those obtained by certified teachers working as tutors.

Imagine that every school could receive up to five well-trained, well-supported teaching assistant tutors, with the number of tutors determined by the school’s needs. This tutor corps could work with the students who are struggling in reading and/or mathematics, for as long as they need the assistance.  Our experience with small-group tutoring of this kind suggests that the cost per student tutored would be around $600 per year (Madden & Slavin, 2017).  Title I schools, especially those serving the most disadvantaged students, should be first in line for this assistance.  $600 per pupil per year is serious money, but well worth it in light of the need.  (Note: there are people suggesting that all students who missed school should repeat their most recent grade.  At an average per-pupil cost of $12,000 to do this, $600 per year sounds awfully reasonable as an alternative).   There are tutoring programs operating right now that can routinely obtain effect sizes of 0.40, or roughly 5 additional months of learning.  This  could go a very long way to not only solve the problems of students whose progress was interrupted by the COVID-19 pandemic, but also help the many students who had problems before, which now need to be urgently addressed).

College graduates could also be trained as health aides, to use proven strategies to ensure that students who need them receive and use eyeglasses, or receive needed medications for asthma and other chronic illnesses that affect children’s school success as well as their long-term health).  They might also be trained and deployed to work with parents on issues such as attendance, social-emotional development, and mental health.

The problems of schools after the COVID-19 health crisis has passed must be addressed, with sufficient power and intensity to ensure that they get solved.  A return to normal is not sufficient.

We may never have a V-COVID Day, as we did a V-J Day after World War II.  But we must have a Marshall Plan for schools.  Universal access to tutoring and other essential services for students who need them would be a feasible, cost-effective start to a plan to reconstruct our schools.

Photo: National Archives at College Park / Public domain

References

Baye, A., Lake, C., Inns, A., & Slavin, R. (2019). Effective reading programs for secondary students. Reading Research Quarterly, 54 (2), 133-166.

Madden, N. A., & Slavin, R. E. (2017). Evaluations of technology-assisted small-group tutoring for struggling readers. Reading & Writing Quarterly, 1-8.

Neitzel, A., Lake, C., Pellegrini, M., & Slavin, R. (2020). A synthesis of quantitative research on programs for struggling readers in elementary schools. Available at www.bestevidence.org. Manuscript submitted for publication.

Pellegrini, M., Neitzel, A., Lake, C., & Slavin, R. (2020). Effective programs in elementary mathematics: A best-evidence synthesis. Available at www.bestevidence.com. Manuscript submitted for publication.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

COVID-19 and School Closures: Could Summer Help?

If there is one educational benefit of the otherwise dismal experience of closing virtually all of America’s schools in response to the COVID-19 pandemic, it is this: I’ll bet parents are developing a lot of respect for teachers. I’m hearing a lot about parents finding out that online lessons are no substitute for capable, in-person teachers.

Because of the essential health need to reduce contacts among students and school personnel, schools all over the U.S. have closed. School leaders are scrambling to provide on-line coursework. It is difficult everywhere to go from zero to online in a very short time, but in schools in high-poverty areas, where many or most students lack home computers or reliable internet access, it is well-nigh impossible. But even if every student had a working computer and internet access, there seems to be widespread use of computerized worksheets, and other uninspiring content. In some schools and districts, in which online work is already well used and computers are universally available, the situation is surely better, but even there, online all day every day is no substitute for in-person teaching. Very conscientious and self-motivated students, the kind who already use Khan Academy just for fun, are probably thriving, but such students constitute a small minority, even in the finest schools.

School closures are likely to extend into May, leaving little if any of the regular school year for things to return to normal. Two states, Kansas and Virginia, have already announced that schools will not re-open before the end of the year, and others will surely follow.

The Summer Solution

In light of the realities we face, I think most schools are struggling to teach all of their children during the school closures. Parents are doing their best, as are some students, but nationwide, trying to keep schools going as they always have, except online, is not a satisfying solution.

I have an alternative solution. It has two simple steps.

1. As soon as feasible, declare schools to be on break. Instant vacation.
2. When it is safe to open schools, do so. Hold an in-person two-month session, starting (let’s say) on June 1 and running through the end of July.

During the instant vacation, provide parents and students with a menu of engaging activities that are fun, engage students’ energies and curiosity, and optional. These could focus on science, social studies, writing, art, music, and other subjects often given short shrift during the school year.  These would be facilitated by teachers; in my experience, every school and district has many teachers who are crazy about one or more topics that they rarely get to talk about in school.  Teachers may be Civil War reenactors, world travelers, art experts, amateur musicians, or published writers, even if those are not the topics they teach.  In three days, max, any school district could find extraordinary people with fierce passions for something they want to share with kids. Students might be given a choice of activities, and they might choose to do none at all. It’s vacation, after all. The reason to have these activities is to give students shut in at home useful and interesting things to do. I’m sure there are loads of great online activities already out there that are rarely used because of the lack of time for such activities in the regular school year. Imagine any of the following, facilitated by teachers who love these topics:

• Online trips to faraway places or to periods of history
• Online book clubs in which students could choose topics they’d like to read about and then discuss age-appropriate books on them with others from all over their school, district, or state.
• Science clubs, in which students could explore topics of their choice in groups from all over. One interesting topic: epidemiology.  Science clubs could find out everything there is to know about space travel, or the science of music, or the science of sports.
• Writer’s workshops, in which kids from all over could enroll in groups working on writing their own mystery stories, fantasy stories, sports stories, or biographies of famous people.  That’s how the Bronte sisters learned to write, shut in in small-town Yorkshire, surrounded by poverty and disease.  They wrote stories with and for each other, throughout their childhoods.
• Art or music appreciation, history, or techniques
• How students can get jobs and internships (in normal times)
• Post-secondary options for secondary students

I think you get the idea. Trying to cover all the usual school subjects in the usual way, but online, is sure to be boring and ineffective for most students. But on vacation, shut in students could select learning activities to do not for a grade, not under pressure from parents or teachers, but to satisfy their own curiosity.

When the crisis is over, presumably in the summer, students could return to school and resume their usual lessons, with in-person teachers.  I’m sure there would be practical difficulties, but I’m willing to bet that this could work, perhaps in some places, perhaps in many. At least it seems worth a try!

Photo credit: zhizhou deng / CC BY (https://creativecommons.org/licenses/by/2.0)

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

Nobel Experiments

The world of evidence-based policy just got some terrific news. Abhijit Banerjee and Esther Duflo, of MIT, and Michael Kremer of Harvard, were recently awarded the Nobel Prize in economics.

This award honors extraordinary people doing extraordinary work to alleviate poverty in developing countries. I heard Esther Duflo speak at the Society for Research on Effective Education, and saw her amazing Ted Talk on the research that won the Nobel (delivered before they knew this was going to happen). I strongly suggest you view her speech, at https://www.ted.com/talks/esther_duflo_social_experiments_to_fight_poverty?language=en

But the importance of this award goes far beyond its recognition of the scholars who received it. It celebrates the same movement toward evidence-based policy represented by the Institute for Education Sciences, Education Innovation Research, the Arnold Foundation, and others in the U.S., the Education Endowment Foundation in the U.K., and this blog. It also celebrates the work of researchers in education, psychology, sociology, as well as economics, who are committed to using rigorous research to advance human progress. The Nobel awardees represent the international development wing of this movement, largely funded by the World Bank, the InterAmerica Development Bank, and other international aid organizations.

In her Ted Talk, Esther Duflo explains the grand strategy she and her colleagues pursue. They take major societal problems in developing countries, break them down into solvable parts, and then use randomized experiments to test solutions to those parts. Along with Dr. Banerjee (her husband) and Michael Kremer, she first did a study that found that ensuring that students in India had textbooks made no difference in learning. They then successfully tested a plan to provide inexpensive tutors and, later, computers, to help struggling readers in India (Banerjee, Cole, Duflo, & Linden, 2007). One fascinating series of studies tested the cost-effectiveness of various educational treatments in developing countries. The winner? Curing children of intestinal worms. Based on this and other research, the Carter Foundation embarked on a campaign that has virtually eradicated Guinea worm worldwide.

Dr. Duflo and her colleagues later tested variations in programs to provide malaria-inhibiting bed nets in developing countries in which malaria is the number one killer of children, especially those less than five years old. Were outcomes best if bed nets (retail cost= $3) were free, or only discounted to varying degrees? Many economists and policy makers worried that people who paid nothing for bed nets would not value them, or might use them for other purposes. But the randomized study found that without question, free bed nets were more often obtained and used than were discounted ones, potentially saving thousands of children’s lives.

For those of us who work in evidence-based education, the types of experiments being done by the Nobel laureates are entirely familiar, even though they have practical aspects quite different from the ones we encounter when we work in the U.S. or the U.K., for example. However, we are far from a majority among researchers in our own countries, and we face major struggles to continue to insist on randomized experiments as the criterion of effectiveness. I’m sure people working in international development face equal challenges. This is why this Nobel Prize in economics means a lot to all of us. People pay a lot of attention to Nobel Prizes, and there isn’t one in educational research, so having a Nobel shared by economists whose main contribution is in the use of randomized experiments to solve questions of great practical and policy importance, including studies in education itself, may be the closest we’ll ever get to Nobel recognition for the principle espoused by many in applied research in psychology, sociology, and education, as it is by many economists.

Nobel Prizes are often used to send a message, to support important new developments in research as well as to recognize deserving researchers who are leaders in this area. This was clearly the case with this award. The Nobel announcement makes it clear how the work of the Nobel laureates has transformed their field, to the point that “their experimental research methodologies entirely dominate developmental economics.”  I hope this event will add further credibility and awareness to the idea that rigorous evidence is a key lever for change that matters in the lives of people

 

Reference

Banerjee, A., Cole, S., Duflo, E., & Linden, L. (2007). Remedying education: Evidence from two randomized experiments in India. The Quarterly Journal of Economics, 122 (3), 1235-1264.

 

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Do School Districts Really Have Difficulty Meeting ESSA Evidence Standards?

The Center for Educational Policy recently released a report on how school districts are responding to the Every Student Succeeds Act (ESSA) requirement that schools seeking school improvement grants select programs that meet ESSA’s strong, moderate, or promising standards of evidence. Education Week ran a story on the CEP report.

The report noted that many states, districts, and schools are taking the evidence requirements seriously, and are looking at websites and consulting with researchers to help them identify programs that meet the standards. This is all to the good.

However, the report also notes continuing problems districts and schools are having finding out “what works.” Two particular problems were cited. One was that districts and schools were not equipped to review research to find out what works. The other was that rural districts and schools found few programs proven effective in rural schools.

I find these concerns astounding. The same concerns were expressed when ESSA was first passed, in 2015. But that was almost four years ago. Since 2015, the What Works Clearinghouse has added information to help schools identify programs that meet the top two ESSA evidence categories, strong and moderate. Our own Evidence for ESSA, launched in February, 2017, has up-to-date information on virtually all PK-12 reading and math programs currently in dissemination. Among hundreds of programs examined, 113 meet ESSA standards for strong, moderate, or promising evidence of effectiveness. WWC, Evidence for ESSA, and other sources are available online at no cost. The contents of the entire Evidence for ESSA website were imported into Ohio’s own website on this topic, and dozens of states, perhaps all of them, have informed their districts and schools about these sources.

The idea that districts and schools could not find information on proven programs if they wanted to do so is difficult to believe, especially among schools eligible for school improvement grants. Such schools, and the districts in which they are located, write a lot of grant proposals for federal and state funding. The application forms for school improvement grants always explain the evidence requirements, because that is the law. Someone in every state involved with federal funding knows about the WWC and Evidence for ESSA websites. More than 90,000 unique users have used Evidence for ESSA, and more than 800 more sign on each week.

As to rural schools, it is true that many studies of educational programs have taken place in urban areas. However, 47 of the 113 programs qualified by Evidence for ESSA were validated in at least one rural study, or a study including a large enough rural sample to enable researchers to separately report program impacts for rural students. Also, almost all widely disseminated programs have been used in many rural schools. So rural districts and schools that care about evidence can find programs that have been evaluated in rural locations, or at least that were evaluated in urban or suburban schools but widely disseminated in rural schools.

Also, it is important to note that if a program was successfully evaluated only in urban or suburban schools, the program still meets the ESSA evidence standards. If no studies of a given outcome were done in rural locations, a rural school in need of better outcomes could, in effect, be asked to choose between a program proven to work somewhere and probably used in dissemination in rural schools, or they could choose a program not proven to work anywhere. Every school and district has to make the best choices for their kids, but if I were a rural superintendent or principal, I’d read up on proven programs, and then go visit some rural schools using that program nearby. Wouldn’t you?

I have no reason to suspect that the CEP survey is incorrect. There are many indications that district and school leaders often do feel that the ESSA evidence rules are too difficult to meet. So what is really going on?

My guess is that there are many district and school leaders who do not want to know about evidence on proven programs. For example, they may have longstanding, positive relationships with representatives of publishers or software developers, or they may be comfortable and happy with the materials and services they are already using, evidence-proven or not. If they do not have evidence of effectiveness that would pass muster with WWC or Evidence for ESSA, the publishers and software developers may push hard on state and district officials, put forward dubious claims for evidence (such as studies with no control groups), and do their best to get by in a system that increasingly demands evidence that they lack. In my experience, district and state officials often complain about having inadequate staff to review evidence of effectiveness, but their concern may be less often finding out what works as it is defending themselves from publishers, software developers, or current district or school users of programs, who maintain that they have been unfairly rated by WWC, Evidence for ESSA, or other reviews. State and district leaders who stand up to this pressure may have to spend a lot of time reviewing evidence or hearing arguments.

On the plus side, at the same time that publishers and software producers may be seeking recognition for their current products, many are also sponsoring evaluations of some of their products that they feel are mostly likely to perform well in rigorous evaluations. Some may be creating new programs that resemble programs that have met evidence standards. If the federal ESSA law continues to demand evidence for certain federal funding purposes, or even to expand this requirement to additional parts of federal grant-making, then over time the ESSA law will have its desired effect, rewarding the creation and evaluation of programs that do meet standards by making it easier to disseminate such programs. The difficulties the evidence movement is experiencing are likely to diminish over time as more proven programs appear, and as federal, state, district, and school leaders get comfortable with evidence.

Evidence-based reform was always going to be difficult, because of the amount of change it entails and the stakes involved. But sooner or later, it is the right thing to do, and leaders who insist on evidence will see increasing levels of learning among their students, at minimal cost beyond what they already spend on untested or ineffective approaches. Medicine went through a similar transition in 1962, when the U.S. Congress first required that medicines be rigorously evaluated for effectiveness and safety. At first, many leaders in the medical profession resisted the changes, but after a while, they came to insist on them. The key is political leadership willing to support the evidence requirement strongly and permanently, so that educators and vendors alike will see that the best way forward is to embrace evidence and make it work for kids.

Photo courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Hummingbirds and Horses: On Research Reviews

Once upon a time, there was a very famous restaurant, called The Hummingbird.   It was known the world over for its unique specialty: Hummingbird Stew.  It was expensive, but customers were amazed that it wasn’t more expensive. How much meat could be on a tiny hummingbird?  You’d have to catch dozens of them just for one bowl of stew.

One day, an experienced restauranteur came to The Hummingbird, and asked to speak to the owner.  When they were alone, the visitor said, “You have quite an operation here!  But I have been in the restaurant business for many years, and I have always wondered how you do it.  No one can make money selling Hummingbird Stew!  Tell me how you make it work, and I promise on my honor to keep your secret to my grave.  Do you…mix just a little bit?”

The Hummingbird’s owner looked around to be sure no one was listening.   “You look honest,” he said. “I will trust you with my secret.  We do mix in a bit of horsemeat.”

“I knew it!,” said the visitor.  “So tell me, what is the ratio?”

“One to one.”

“Really!,” said the visitor.  “Even that seems amazingly generous!”

“I think you misunderstand,” said the owner.  “I meant one hummingbird to one horse!”

In education, we write a lot of reviews of research.  These are often very widely cited, and can be very influential.  Because of the work my colleagues and I do, we have occasion to read a lot of reviews.  Some of them go to great pains to use rigorous, consistent methods, to minimize bias, to establish clear inclusion guidelines, and to follow them systematically.  Well- done reviews can reveal patterns of findings that can be of great value to both researchers and educators.  They can serve as a form of scientific inquiry in themselves, and can make it easy for readers to understand and verify the review’s findings.

However, all too many reviews are deeply flawed.  Frequently, reviews of research make it impossible to check the validity of the findings of the original studies.  As was going on at The Hummingbird, it is all too easy to mix unequal ingredients in an appealing-looking stew.   Today, most reviews use quantitative syntheses, such as meta-analyses, which apply mathematical procedures to synthesize findings of many studies.  If the individual studies are of good quality, this is wonderfully useful.  But if they are not, readers often have no easy way to tell, without looking up and carefully reading many of the key articles.  Few readers are willing to do this.

Recently, I have been looking at a lot of recent reviews, all of them published, often in top journals.  One published review only used pre-post gains.  Presumably, if the reviewers found a study with a control group, they would have ignored the control group data!  Not surprisingly, pre-post gains produce effect sizes far larger than experimental-control, because pre-post analyses ascribe to the programs being evaluated all of the gains that students would have made without any particular treatment.

I have also recently seen reviews that include studies with and without control groups (i.e., pre-post gains), and those with and without pretests.  Without pretests, experimental and control groups may have started at very different points, and these differences just carry over to the posttests.  Accepting this jumble of experimental designs, a review makes no sense.  Treatments evaluated using pre-post designs will almost always look far more effective than those that use experimental-control comparisons.

Many published reviews include results from measures that were made up by program developers.  We have documented that analyses using such measures produce outcomes that are two, three, or sometimes four times those involving independent measures, even within the very same studies (see Cheung & Slavin, 2016). We have also found far larger effect sizes from small studies than from large studies, from very brief studies rather than longer ones, and from published studies rather than, for example, technical reports.

The biggest problem is that in many reviews, the designs of the individual studies are never described sufficiently to know how much of the (purported) stew is hummingbirds and how much is horsemeat, so to speak. As noted earlier, readers often have to obtain and analyze each cited study to find out whether the review’s conclusions are based on rigorous research and how many are not. Many years ago, I looked into a widely cited review of research on achievement effects of class size.  Study details were lacking, so I had to find and read the original studies.   It turned out that the entire substantial effect of reducing class size was due to studies of one-to-one or very small group tutoring, and even more to a single study of tennis!   The studies that reduced class size within the usual range (e.g., comparing reductions from 24 to 12) had very small achievement  impacts, but averaging in studies of tennis and one-to-one tutoring made the class size effect appear to be huge. Funny how averaging in a horse or two can make a lot of hummingbirds look impressive.

It would be great if all reviews excluded studies that used procedures known to inflate effect sizes, but at bare minimum, reviewers should be routinely required to include tables showing critical details, and then analyzed to see if the reported outcomes might be due to studies that used procedures suspected to inflate effects. Then readers could easily find out how much of that lovely-looking hummingbird stew is really made from hummingbirds, and how much it owes to a horse or two.

References

Cheung, A., & Slavin, R. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45 (5), 283-292.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Is ES=+0.50 Achievable?: Schoolwide Approaches That Might Meet This Standard

In a recent blog, “Make No Small Plans,” I proposed a system innovators could use to create very effective schoolwide programs.  I defined these as programs capable of making a difference in student achievement large enough to bring entire schools serving disadvantaged students to the levels typical of middle class schools.  On average, that would mean creating school models that could routinely add an effect size of +0.50 for entire disadvantaged schools.  +0.50, or half a standard deviation, is roughly the average difference between students who qualify for free lunch and those who do not, between African American and White students, and between Hispanic and non-Hispanic White students.

Today, I wanted to give some examples of approaches intended to meet the +0.50 goal. From prior work, my colleagues and I already have created a successful schoolwide reform model, Success for All, which, with adequate numbers of tutors (as many as six per school) achieved reading effect sizes in high-poverty Baltimore elementary schools of over +0.50 for all students and +0.75 for the lowest-achieving quarter of students (Madden et al, 1993).   These outcomes maintained through eighth grade, and showed substantial reductions in grade retentions and special education placements (Borman & Hewes, 2003).  Steubenville, in Ohio’s Rust Belt, uses Success for All in all of its Title I elementary schools, providing several tutors in each.  Each year, Steubenville schools score among the highest in Ohio on state tests, exceeding most wealthy suburban schools.  Other SFA schools with sufficient tutors are also exemplary in achievement gains.  Yet these schools face a dilemma.  Most cannot afford significant numbers of tutors.  They still get excellent results, but less than those typical of SFA schools that do have sufficient tutors.

We are now planning another approach, also intended to produce schoolwide effect sizes of at least +0.50 in schools serving disadvantaged students.   However, in this case our emphasis is on tutoring, the most effective strategy known for improving the achievement of struggling readers (Inns et al., 2019).  We are calling this approach the Reading Safety Net.  Main components of this plan are as follows:

Tutoring

Like the most successful forms of Success for All, the Reading Safety Net places a substantial emphasis on tutoring.  Tutors will be well-qualified teaching assistants with BAs but not teaching certificates, extensively trained to provide one-to-four tutoring.   Tutors will use a proven computer-assisted model in which students do a lot of pair teaching.  This is what we now call our Tutoring With the Lightning Squad model, which achieved outcomes of +0.40 and +0.46 in two studies in the Baltimore City Public Schools (Madden & Slavin, 2017).  A high-poverty school of 500 students might engage about five tutors, providing extensive tutoring to the majority of students, for as many years as necessary.  One additional tutor or teacher will supervise the tutors and personally work with students having the most serious problems.   We will provide significant training and follow-up coaching to ensure that all tutors are effective.

Attendance and Health

Many students fail in reading or other outcomes because they have attendance problems or certain common health problems. We propose to provide a health aide to help solve these problems.

Attendance

Many students, especially those in high-poverty schools, fail because they do not attend school regularly. Yet there are several proven approaches for increasing attendance, and reducing chronic truancy (Shi, Inns, Lake, and Slavin, 2019).  Health aides will help teachers and other staff organize and manage effective attendance improvement approaches.

Vision Services

My colleagues and I have designed strategies to help ensure that all students who need eyeglasses receive them. A key problem in this work is ensuring that students who receive glasses use them, keep them safe, and replace them if they are lost or broken. Health aides will coordinate use of proven strategies to increase regular use of needed eyeglasses.

Asthma and other health problems

Many students in high-poverty schools suffer from chronic illnesses.  Cures or prevention are known for these, but the cures may not work if medications are not taken daily.   For example, asthma is common in high-poverty schools, where it is the top cause of hospital referrals and a leading cause of death for school-age children.  Inexpensive inhalers can substantially improve children’s health, yet many children do not regularly take their medicine. Studies suggest that having trained staff ensure that students take their medicine, and watch them doing so, can make a meaningful difference.  The same may be true of other chronic, easily treated diseases common among children but often not consistently treated in inner-city schools.  Health aides with special supplemental training may be able to play a key on-the-ground role in helping ensure effective treatment for asthma and other diseases.

Potential Impact

The Reading Safety Net is only a concept at present.  We are seeking funding to support its further development and evaluation.  As we work with front line educators, colleagues, and others to further develop this model, we are sure to find ways to make the approach more effective and cost-effective, and perhaps extend it to solve other key problems.

We cannot yet claim that the Reading Safety Net has been proven effective, although many of its components have been.  But we intend to do a series of pilots and component evaluations to progressively increase the impact, until that impact attains or surpasses the goal of ES=+0.50.  We hope that many other research teams will mobilize and obtain resources to find their own ways to +0.50.  A wide variety of approaches, each of which would be proven to meet this ambitious goal, would provide a range of effective choices for educational leaders and policy makers.  Each would be a powerful, replicable tool, capable of solving the core problems of education.

We know that with sufficient investment and encouragement from funders, this goal is attainable.  If it is in fact attainable, how could we accept anything less?

References

Borman, G., & Hewes, G. (2003).  Long-term effects and cost effectiveness of Success for All.  Educational Evaluation and Policy Analysis, 24 (2), 243-266.

Inns, A., Lake, C., Pellegrini, M., & Slavin, R. (2019). A synthesis of quantitative research on programs for struggling readers in elementary schools. Manuscript submitted for publication.

Madden, N. A., & Slavin, R. E. (2017). Evaluations of Technology-Assisted Small-Group Tutoring for Struggling Readers. Reading & Writing Quarterly, 1-8.

Madden, N. A., Slavin, R. E., Karweit, N. L., Dolan, L., & Wasik, B. (1993). Success for All:  Longitudinal effects of a schoolwide elementary restructuring program. American Educational Reseach Journal, 30, 123-148.

Shi, C., Inns, A., Lake, C., & Slavin, R. E. (2019). Effective school-based programs for K-12 students’ attendance: A best-evidence synthesis. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

 

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Why Do Some Educators Push Back Against Evidence?

In December, 2015, the U.S. Congress passed the Every Student Succeeds Act, or ESSA. Among many other provisions, ESSA defined levels of evidence supporting educational programs: Strong (at least one randomized experiment with positive outcomes), moderate (at least one quasi-experimental study with positive outcomes), and promising (at least one correlational study with positive outcomes). For various forms of federal funding, schools are required (in school improvement) or encouraged (in seven other funding streams) to use programs falling into one of these top three categories. There is also a fourth category, “demonstrates a rationale,” but this one has few practical consequences.

3 ½  years later, the ESSA evidence standards are increasing interest in evidence of effectiveness for educational programs, especially among schools applying for school improvement funding and in state departments of education, which are responsible for managing the school improvement grant process. All of this is to the good, in my view.

On the other hand, evidence is not yet transforming educational practice. Even in portions of ESSA that encourage or require use of proven programs among schools seeking federal funding, schools and districts often try to find ways around the evidence requirements rather than truly embracing them. Even when schools do say they used evidence in their proposals, they may have just accepted assurances from publishers or developers stating that their programs meet ESSA standards, even when this is clearly not so.

Why are these children in India pushing back on a car?  And why do many educators in our country push back on evidence?

Educators care a great deal about their children’s achievement, and they work hard to ensure their success. Implementing proven, effective programs does not guarantee success, but it greatly increases the chances. So why has evidence of effectiveness played such a limited role in program selection and implementation, even when ESSA, the national education law, defines evidence and requires use of proven programs under certain circumstances?

The Center on Education Policy Report

Not long ago, the Center on Education Policy (CEP) at George Washington University published a report of telephone interviews of state leaders in seven states. The interviews focused on problems states and districts were having with implementation of the ESSA evidence standards. Six themes emerged:

1. Educational leaders are not comfortable with educational research methods.
2. State leaders feel overwhelmed serving large numbers of schools qualifying for school improvement.
3. Districts have to seriously re-evaluate longstanding relationships with vendors of education products.
4. State and district staff are confused about the prohibition on using Title I school improvement funds on “Tier 4” programs (ones that demonstrate a rationale, but have not been successfully evaluated in a rigorous study).
5. Some state officials complained that the U.S. Department of Education had not been sufficiently helpful with implementation of ESSA evidence standards.
6. State leaders had suggestions to make education research more accessible to educators.

What is the Reality?

I’m sure that the concerns expressed by the state and district leaders in the CEP report are sincerely felt. But most of them raise issues that have already been solved at the federal, state, and/or district levels. If these concerns are as widespread as they appear to be, then we have serious problems of communication.

1. The first theme in the CEP report is one I hear all the time. I find it astonishing, in light of the reality.

No educator needs to be a research expert to find evidence of effectiveness for educational programs. The federal What Works Clearinghouse (https://ies.ed.gov/ncee/wwc/) and our Evidence for ESSA (www.evidenceforessa.org) provide free information on the outcomes of programs, at least in reading and mathematics, that is easy to understand and interpret. Evidence for ESSA provides information on programs that do meet ESSA standards as well as those that do not. We are constantly scouring the literature for studies of replicable programs, and when asked, we review entire state and district lists of adopted programs and textbooks, at no cost. The What Works Clearinghouse is not as up-to-date and has little information on programs lacking positive findings, but it also provides easily interpreted information on what works in education.

In fact, few educational leaders anywhere are evaluating the effectiveness of individual programs by reading research reports one at a time. The What Works Clearinghouse and Evidence for ESSA employ experts who know how to find and evaluate outcomes of valid research and to describe the findings clearly. Why would every state and district re-do this job for themselves? It would be like having every state do its own version of Consumer Reports, or its own reviews of medical treatments. It just makes no sense. In fact, at least in the case of Evidence for ESSA, we know that more than 80,000 unique readers have used Evidence for ESSA since it launched in 2017. I’m sure even larger numbers have used the What Works Clearinghouse and other reviews. The State of Ohio took our entire Evidence for ESSA website and put it on its own state servers with some other information. Several other states have strongly promoted the site. The bottom line is that educational leaders do not have to be research mavens to know what works, and tens of thousands of them know where to find fair and useful information.

2. State leaders are overwhelmed. I’m sure this is true, but most state departments of education have long been understaffed. This problem is not unique to ESSA.
3. Districts have to seriously re-evaluate longstanding relationships with vendors. I suspect that this concern is at the core of the problem on evidence. The fact is that most commercial programs do not have adequate evidence of effectiveness. Either they have no qualifying studies (by far the largest number), or they do have qualifying evidence that is not significantly positive. A vendor with programs that do not meet ESSA standards is not going to be a big fan of evidence, or ESSA. These are often powerful organizations with deep personal relationships with state and district leaders. When state officials adhere to a strict definition of evidence, defined in ESSA, local vendors push back hard. Understaffed state departments are poorly placed to fight with vendors and their friends in district offices, so they may be forced to accept weak or no evidence.
4. Confusions about Tier 4 evidence. ESSA is clear that to receive certain federal funds schools must use programs with evidence in Tiers 1, 2, or 3, but not 4. The reality is that definitions of Tier 4 are so weak that any program on Earth can meet this standard. What program anywhere does not have a rationale? The problem is that districts, states, and vendors have used confusion about Tier 4 to justify any program they wish. Some states are more sophisticated than others and do not allow this, but the very existence of Tier 4 in ESSA language creates a loophole that any clever sales rep or educator can use, or at least try to get away with.
5. The U. S. Department of Education is not helpful enough. In reality, USDoE is understaffed and overwhelmed on many fronts. In any case, ESSA puts a lot of emphasis on state autonomy, so the feds feel unwelcome in performing oversight.

The Future of Evidence in Education

Despite the serious problems in implementation of ESSA, I still think it is a giant step forward. Every successful field, such as medicine, agriculture, and technology, has started its own evidence revolution fighting entrenched interests and anxious stakeholders. As late as the 1920s, surgeons refused to wash their hands before operations, despite substantial evidence going back to the 1800s that handwashing was essential. Evidence eventually triumphs, though it often takes many years. Education is just at the beginning of its evidence revolution, and it will take many years to prevail. But I am unaware of any field that embraced evidence, only to retreat in the face of opposition. Evidence eventually prevails because it is focused on improving outcomes for people, and people vote. Sooner or later, evidence will transform the practice of education, as it has in so many other fields.

Photo credit: Roger Price from Hong Kong, Hong Kong [CC BY 2.0 (https://creativecommons.org/licenses/by/2.0)]

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Mislabeled as Disabled

Kenny is a 10^th grader in the Baltimore City Public Schools. He is an African American from a disadvantaged neighborhood, attending a high school that requires high grades and test scores. He has good attendance, and has never had any behavior problems. A good kid, by all accounts but one.

Kenny reads at the kindergarten level.

Kenny has spent most of his time in school in special education. He received extensive and expensive services, following an Individual Education Program (IEP) made and updated over time just for him, tailored to his needs.

Yet despite all of this, he is still reading at the kindergarten level in 10^th grade.

Kenny’s story starts off a remarkable book, Mislabeled as Disabled, by my friend Kalman (Buzzy) Hettleman. A lawyer by training, Hettleman has spent many years volunteering in Baltimore City schools to help children being considered for special education obtain the targeted assistance they need to either avoid special education or succeed in it. What he has seen, and describes in detail in his book, is nothing short of heartbreaking. In fact, it makes you furious. Here is a system designed to improve the lives of vulnerable children, spending vast amounts of money to enable talented and hard-working teachers to work with children. Yet the outcomes are appalling. It’s not just Kenny. Thousands of students in Baltimore, and in every other city and state, are failing. These are mostly children with specific learning disabilities or other mild, “high-incidence” categories. Or they are struggling readers not in special education who are not doing much better. Many of the students who are categorized as having mild disabilities are not disabled, and would have done at least as well with appropriate services in the regular classroom. Instead, what they get is an IEP. Such children are “mislabeled as disabled,” and obtain little benefit from the experience.

Buzzy has worked at many levels of this system. He was on the Baltimore school board for many years. He taught social work at the University of Maryland. He has been an activist, fighting relentlessly for the rights of struggling students (and at 84 years of age still is). Most recently, he has served on the Kirwan Commission, appointed to advise the state legislature on reform policies and new funding formulas for the state’s schools. Buzzy has seen it all, from every angle. His book is deeply perceptive and informed, and makes many recommendations for policy and practice. But his message is infuriating. What he documents is a misguided system that is obsessed with rules and policies but pays little attention to what actually works for struggling learners.

What most struggling readers need is proven, well-implemented programs in a Response to Intervention (RTI) framework. Mostly, this boils down to tutoring. Most struggling students can benefit enormously from one-to-small group tutoring by well-qualified teaching assistants (paraprofessionals), so tutoring need not be terribly expensive. Others may need certified teachers or one-to-one. Some struggling readers can succeed with well-implemented proven, strategies in the regular classroom (Tier 1). Those who do not succeed in Tier 1 should receive proven one-to-small group tutoring approaches (Tier 2). If that is not sufficient, a small number of students may need one-to-one tutoring, although research tells us that one-to-small group is almost as effective as one-to-one, and is a lot less expensive.

Tutoring is the missing dynamic in the special education system for struggling readers, whether or not they have IEPs. Yes, some districts do provide tutoring to struggling readers, and if the tutoring model they implement is proven in rigorous research it is generally effective. The problem is that there are few schools or districts that provide enough tutoring to enough struggling readers to move the needle.

Buzzy described a policy he devised with Baltimore’s then-superintendent, Andres Alonso. They called it “one year plus.” It was designed to ensure that all students with high-incidence disabilities, such as those with specific learning disabilities, must receive instruction sufficient to enable them to make one year’s progress or more every 12 months.  If students could do this, they would, over time, close the gap between their reading level and their grade level. This was a radical idea, and its implementation it fell far short. But the concept is exactly right. Students with mild disabilities, who are the majority of those with IEPs, can surely make such gains. In recent years, research has identified a variety of tutoring approaches that can ensure one year or more of progress in a year for most students with IEPs, at a cost a state like Maryland could surely afford.

            Mislabeled as Disabled is written about Buzzy’s personal experience in Baltimore. However, what he describes is happening in districts and states throughout the U.S., rich as well as poor. This dismal cycle can stop anywhere we choose to stop it. Buzzy Hettleman describes in plain, powerful language how this could happen, and most importantly, why it must.

Reference

Hettleman, K. R. (2019). Mislabeled as disabled. New York: Radius.

 This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Moneyball for Education

When I was a kid, growing up in the Maryland suburbs of Washington, DC, everyone I knew rooted for the hapless Washington Senators, one of the worst baseball teams ever. At that time, however, the Baltimore Orioles were one of the best teams in baseball, and every once in a while a classmate would snap. He (always “he”) would decide to become an Orioles fan. This would cause him to be shamed and ostracized for the rest of his life by all true Senators fans.

I’ve now lived in Baltimore for most of my life. I wonder if I came here in part because of my youthful impression of Baltimore as a winning franchise?

Skipping forward in time to now, I recently saw in the New York Times an article about the collapse of the Baltimore Orioles. In 2018, they had the worst record of any team in history. Worse than even the Washington Senators ever were. Why did this happen? According to the NYT, the Orioles are one of the last teams to embrace analytics, which means using evidence to decide which players to recruit or drop, to put on the field or on the bench. Some teams have analytics departments of 15. The Orioles? Zero, although they have just started one.

It’s not as though the benefits of analytics are a secret. A 2003 book by Michael Lewis, Moneyball, explained how the underfunded Oakland As used analytics to turn themselves around. A hugely popular 2011 movie told the same story.

In case anyone missed the obvious linkage of analytics in baseball to analytics in education, Results for America (RfA), a group that promotes the use of evidence in government social programs, issued a 2015 book called, you guessed it, Moneyball for Government (Nussle & Orszag, 2015). This Moneyball focused on success stories and ideas from key thinkers and practitioners in government and education. RfA was instrumental in encouraging the U.S. Congress to include in ESSA definitions of strong, moderate, and promising evidence of effectiveness, and to specify a few areas of federal funding that require or incentivize use of proven programs.

The ESSA evidence standards are a giant leap forward in supporting the use of evidence in education. Yet, like the Baltimore Orioles, the once-admired U.S. education system has been less than swept away by the idea that using proven programs and practices could improve outcomes for children. Yes, the situation is better than it was, but things are going very slowly. I’m worried that because of this, the whole evidence movement in education will someday be dismissed: “Evidence? Yeah, we tried that. Didn’t work.”

There are still good reasons for hope. The amount of high-quality evidence continues to grow at an unprecedented pace. The ESSA evidence standards have at least encouraged federal, state, and local leaders to pay some attention to evidence, though moving to action based on this evidence is a big lift.

Perhaps I’m just impatient. It took the Baltimore Orioles a book, a movie, and 16 years to arrive at the conclusion that maybe, just maybe, it was time to use evidence, as winning teams have been doing for a long time. Education is much bigger, and its survival does not depend on its success (as baseball teams do). Education will require visionary leadership to embrace the use of evidence. But I am confident that when it does, we will be overwhelmed by visits from educators from Finland, Singapore, China, and other countries that currently clobber us in international comparisons. They’ll want to know how the U.S. education system became the best in the world. Perhaps we’ll have to write a book and a movie to explain it all.  I’d suggest we call it . . . “Learnball.”

References

Nussle, J., & Orszag, P. (2015). Moneyball for Government (2^nd Ed.). Washington, DC: Disruption Books.

Photo credit: Keith Allison [CC BY-SA 2.0 (https://creativecommons.org/licenses/by-sa/2.0)]

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

On Keeping the Main Thing the Main Thing

When I finished college, I had an Oregon teaching certificate to teach secondary social studies. However, since the only sport I could coach was chess, I took a job teaching in a self-contained school for children with intellectual disabilities. I happened to know what I was doing, because I’d taught such children in summer jobs throughout college, but I had a strong feeling that my teaching certificate, plus the ability to fog a mirror, was all my school required.

In mid-year, however, my school moved all of its children into local elementary schools. I refused to go. I had the lowest-functioning kids, and I had to change diapers. The school I was slated to go to could only offer the boys’ bathroom, whose only source of water was a fountain in the middle that gave out an unenthusiastic mist.

My principal agreed, so I was left with the only class still operating in my original school. The school phone was put in my classroom.

Answering the school phone gave me an unexpected insight into one aspect of schooling I had not known about. Every day, people would call to offer to donate things to the school. Things like used shoes or clothing, or old toys. Once someone offered fresh flowers. I accepted these, thinking they might be nice. But when hundreds of lilies appeared (they turned out to be from a funeral), my kids immediately pounced on them and started eating them.

In addition to useless stuff, I had regular visits from people sent by our county, such as an occupational therapist and a physical therapist. These were all very nice people, but I just met with them an hour or two each month only to discover, each time, that there was nothing they could do for my kids.

Skip forward many years, and I’m still astounded by how much of the time of principals is taken up nurse-maiding well-meaning people who want to help disadvantaged children. Recently, I heard from some Maryland principals that the State Department of Education sent them dozens of coaches and experts of all sorts. They were not there enough to provide any useful services, but each visit required valuable time from the principal or other staff. Further, they said these coaches never had the remotest idea what the other coaches were doing, and rarely if ever coordinated with them, so redundant or contradictory services were offered.

I remember being in a school in Memphis where a local men’s service club had some of its members reading to kids. These men were very nice, and the kids and teachers were glad to see them. But their presence in the building tied up the principal and much other staff just helping the men find classrooms and materials. Our staff were in the building to discuss specific, proven strategies for helping students learn to read, but this activity was greatly disrupted by these well-meaning men.

Keeping the Main Thing the Main Thing.

The well-meaning chaos I’ve observed many times speaks volumes about one of the structural reasons that schools for disadvantaged students have so much trouble making progress. Such schools are in communities that care deeply about them and want to help. Some of this help is helpful, but much of it interferes with the school’s core mission, which is to ensure that all children are succeeding in all academic subjects, especially reading and math.

In schools serving disadvantaged students, the mission is not just a way to improve school test scores. Success in reading and math is a survival skill for students whose only way out of poverty may be success in school. Principals should have the right, indeed the mandate, to decide (with their staff) which external services support their core mission, and which are unlikely to do so in any meaningful way. External services that provide frequent, targeted, reliable assistance that is well-aligned with the school’s efforts to advance its core mission, such as volunteer tutors who commit to substantial time working with children using methods and materials closely aligned with the school’s curriculum, can be a godsend. Occasional tutors who are unlikely to show up on all scheduled days or are unable or unwilling to use proven methods aligned with the school’s needs? Not so much.

Knowing when to encourage and even solicit help and when not to do so is an important part of school leadership. Principals and their staffs need to have clear goals and plans for how to achieve them. That has to be job one for the whole staff. External assistance that helps schools achieve their goals is great. And assistance to help achieve other valued goals, such as after school art or music or theater programs, may be fine if it does not interfere with job one. But external assistance that takes time and adds complexity? Maybe later.

Every superintendent, principal, and school staff has to have a laser focus on the core mission of their school. They have only so many hours in the day to accomplish this core mission. The government and private entities who support schools have to learn that schools must focus their energies on the main thing, academic achievement, and then help school staff to accomplish this goal if they can, and help schools keep the main thing the main thing.

Photo credit: Courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.