Why Isn’t Achievement Whirled Enough by Time? (Why Summer School, After School, and Extended Day Do Not Work Very Well)

“Had we but world enough and time…” wrote Andrew Marvell, an English poet in the late 1600s (He also had another job, highly relevant to this blog, which I will reveal at the end. No peeking!).

Marvell’s poem was about making the most of the limited time we have on Earth. In education, we understand this sentiment. Time is a key resource for teaching, not to be wasted under any circumstances.

In fact, educators have long tried to improve students’ achievement by increasing their time in school. In particular, struggling students have been invited or required to attend after school or summer school classes.

Many school reformers have advocated expanded opportunities for extra-time instruction, as solutions to the learning losses due to Covid-19 school closures. In fact, the current draft of the Democrats’ relief bill emphasizes investments in after school and summer school programs to help these students catch up. Yet these very expensive efforts have not had much impact on reading or math learning in studies done before Covid, and are not likely to have much impact now (see my previous blog on this topic).

How can this be? Summer school, for example, offers several weeks of extra teaching in small classes tailored to the learning levels of the students. Yet summer school for reading has been completely ineffective, except for tutoring phonics in K-1. Math summer school studies involving disadvantaged and low-achieving students also found effect sizes near zero (Xie et al., 2020).

With respect to after-school programs, a review by Kidron & Lindsay (2014) found average effect sizes near zero.

A study in Milwaukee by Heinrich et al. (2009) of after school programs provided under Supplemental Education Services (SES) funding found effect sizes near zero for middle and high school students. The authors investigated the reasons for these disappointing findings. Among eligible students, 57% registered in the first year, dropping to 48% by the fourth year. Yet the bigger problem was attendance. As a percent of registered students, attendance dropped from 90% in the first year to 34% in the fourth, meaning that among all eligible students, only 16% attended in the final year. This abysmal attendance rate should not be surprising in light of the observation in the study that most of the after-school time was spent on worksheets, with little or no instruction. The Heinrich et al. (2009) paper contained the following depressing sentence:

“…one might also speculate that parents and students are, in fact, choosing rationally in not registering for or attending SES.” (p. 296).

Reviews of research on the impacts of all approaches to SES find average effects that are appalling (e.g., Chappell et al., 2011). I will write more about SES as a cautionary tale in a later blog, but one conclusion important to the blog is clear: Providing educational programs to struggling students after school or in the summer is unlikely to improve student achievement.

The reasons that additional time after school or in the summer does not enhance achievement is obvious, if you’ve ever been a teacher or a student. No one wants to be sitting in school while their friends are out playing. Extra time approaches that simply provide more of the same are probably boring, tedious, and soul-sapping. Imagine kids watching the clock, quietly cheering for every click. It is no wonder that students fail to register or fail to attend after school or summer school sessions, and learn little in them if they do.

The poet Andrew Marvell had it right. What is important is to make effective use of the time we have, rather than adding time. And his profession, other than being a poet? He was a tutor.

References

Chappell, S., Nunnery, J., Pribesh, S., & Hager, J. (2011). A meta-analysis of Supplemental Education Services (SES) provider effects on student achievement. Journal of Education for Students Placed at Risk, 16 (1), 1-23.

Heinrich, C. J., Meyer, R., H., & Whitten, G. W. (2010). Supplemental Education Services under No Child Left Behind: Who signs up and what do they gain? Education Evaluation and Policy Analysis, 32, 273-298.

Kidron, Y., & Lindsay, J. (2014). The effects of increased learning time on student academic and nonacademic outcomes: Findings from a meta‑analytic review (REL 2014-015). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Appalachia.

Xie, C., Neitzel, A., Cheung, A., & Slavin, R. E. (2020). The effects of summer programs on K-12 students’ reading and mathematics achievement: A meta-analysis. Manuscript submitted for publication.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org.

Healing Covid-19’s Educational Losses: What is the Evidence?

I’ve written several blogs (here, here, here, here, here, and here) on what schools can do when they finally open permanently, to remedy what will surely be serious harm to the educational progress of millions of students. Without doubt, the students who are suffering the most from lengthy school closures are disadvantaged students, who are most likely to lack access to remote technology or regular support when their schools have been closed.

Recently, there have been several articles circulated in the education press (e.g., Sawchuk, 2020) and newsletters laying out the options schools might consider to greatly improve the achievement of students who lost the most, and are performing far behind grade level.

The basic problem is that if schools simply start off with usual teaching for each grade level, this may be fine for students at or just below grade level, but for those who are far below level, this is likely to add catastrophe to catastrophe. Students who cannot read the material they are being taught, or who lack the prerequisite skills for their grade level, will experience failure and frustration. So the challenge is to provide students who are far behind with intensive, additional services likely to quickly accelerate their progress, so that they can then profit from ordinary, at-grade-level lessons.

In the publications I’ve seen, there have been several solutions frequently put forward. I thought this might be a good time to review the most common prescriptions in terms of their evidence basis in rigorous experimental or quasi-experimental research.

Extra Time

One proposal is to extend the school day or school year to provide additional time for instruction. This sounds logical; if the problem is time out of school, let’s add time in school.

The effects of extra time depend, of course, on what schools provide during that additional time. Simply providing more clock hours in which typical instruction is provided makes little difference. For example, in a large Florida study (Figlio, Holden, & Ozek, 2018), high-poverty schools were given a whole hour every day for a year, for additional reading instruction. This had a small impact on reading achievement (ES=+0.09) at a cost of about $800 per student, or $300,000-$400,000 per school. Also, in a review of research on secondary reading programs by Baye, Lake, Inns & Slavin (2019), my colleagues and I examined whether remedial programs were more effective if they were provided during additional time (one class period a day more than what the control group received for one or more years) or if they were provided during regular class time (the same amount of time the control group also received). The difference was essentially zero. The extra time did not matter. What did matter was what the schools provided (here and here).

After-School Programs

Some sources suggest providing after-school programs for students experiencing difficulties. A review of research on this topic by Kidron & Lindsay (2014) examined effects of after-school programs on student achievement in reading and mathematics. The effects were essentially zero. One problem is that students often did not attend regularly, or were poorly motivated when they did attend.

Summer School

As noted in a recent blog, positive effects of summer school were found only when intensive phonics instruction was provided in grades K or 1, but even in these cases, positive effects did not last to the following spring. Summer school is also very expensive.

Tutoring

By far the most effective approach for students struggling in reading or mathematics is tutoring (see blogs here, here, and here). Outcomes for one-to-one or one-to-small group tutoring average +0.20 to +0.30 in both reading and mathematics, and there are several particular programs that routinely report outcomes of +0.40 or more. Using teaching assistants with college degrees as tutors can make tutoring very cost-effective, especially in small-group programs.

Whole-School Reforms

There are a few whole-school reforms that can have substantial impacts on reading and mathematics achievement. A recent review of our elementary school reform model, Success for All (Cheung et al., 2020), found an average effect size of +0.24 for all students across 17 studies, and an average of +0.54 for low achievers.

A secondary reform model called BARR has reported positive reading and mathematics outcomes for ninth graders (T. Borman et al., 2017)

Conclusion

Clearly, something needs to be done about students returning to in-person education who are behind grade level in reading and/or mathematics. But resources devoted to helping these students need to be focused on approaches proven to work. This is not the time to invest in plausible but unproven programs. Students need the best we have that has been repeatedly shown to work.

References

Baye, A., Lake, C., Inns, A., & Slavin, R. (2019). Effective reading programs for secondary students. Reading Research Quarterly, 54 (2), 133-166.

Borman, T., Bos, H., O’Brien, B. C., Park, S. J., & Liu, F. (2017). i3 BARR validation study impact findings: Cohorts 1 and 2. Washington, DC: American Institutes for Research.

Cheung, A., Xie, C., Zhang, T., Neitzel, A., & Slavin, R. E. (2020). Success for All: A quantitative synthesis of evaluations. Manuscript submitted for publication. (Contact us for a copy.)

Figlio, D. N., Holden, K. L., & Ozek, U. (2018). Do students benefit from longer school days? Regression discontinuity evidence from Florida’s additional hour of literacy instruction. Economics of Education Review, 67, 171-183. https://doi.org/10.1016/j.econedurev.2018.06.003

Sawchuk, S. (2020, August 26). Overcoming Covid-19 learning loss. Education Week, 40 (2), 6.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org.

On Keeping the Main Thing the Main Thing

When I finished college, I had an Oregon teaching certificate to teach secondary social studies. However, since the only sport I could coach was chess, I took a job teaching in a self-contained school for children with intellectual disabilities. I happened to know what I was doing, because I’d taught such children in summer jobs throughout college, but I had a strong feeling that my teaching certificate, plus the ability to fog a mirror, was all my school required.

In mid-year, however, my school moved all of its children into local elementary schools. I refused to go. I had the lowest-functioning kids, and I had to change diapers. The school I was slated to go to could only offer the boys’ bathroom, whose only source of water was a fountain in the middle that gave out an unenthusiastic mist.

My principal agreed, so I was left with the only class still operating in my original school. The school phone was put in my classroom.

Answering the school phone gave me an unexpected insight into one aspect of schooling I had not known about. Every day, people would call to offer to donate things to the school. Things like used shoes or clothing, or old toys. Once someone offered fresh flowers. I accepted these, thinking they might be nice. But when hundreds of lilies appeared (they turned out to be from a funeral), my kids immediately pounced on them and started eating them.

In addition to useless stuff, I had regular visits from people sent by our county, such as an occupational therapist and a physical therapist. These were all very nice people, but I just met with them an hour or two each month only to discover, each time, that there was nothing they could do for my kids.

Skip forward many years, and I’m still astounded by how much of the time of principals is taken up nurse-maiding well-meaning people who want to help disadvantaged children. Recently, I heard from some Maryland principals that the State Department of Education sent them dozens of coaches and experts of all sorts. They were not there enough to provide any useful services, but each visit required valuable time from the principal or other staff. Further, they said these coaches never had the remotest idea what the other coaches were doing, and rarely if ever coordinated with them, so redundant or contradictory services were offered.

I remember being in a school in Memphis where a local men’s service club had some of its members reading to kids. These men were very nice, and the kids and teachers were glad to see them. But their presence in the building tied up the principal and much other staff just helping the men find classrooms and materials. Our staff were in the building to discuss specific, proven strategies for helping students learn to read, but this activity was greatly disrupted by these well-meaning men.

Keeping the Main Thing the Main Thing.

The well-meaning chaos I’ve observed many times speaks volumes about one of the structural reasons that schools for disadvantaged students have so much trouble making progress. Such schools are in communities that care deeply about them and want to help. Some of this help is helpful, but much of it interferes with the school’s core mission, which is to ensure that all children are succeeding in all academic subjects, especially reading and math.

In schools serving disadvantaged students, the mission is not just a way to improve school test scores. Success in reading and math is a survival skill for students whose only way out of poverty may be success in school. Principals should have the right, indeed the mandate, to decide (with their staff) which external services support their core mission, and which are unlikely to do so in any meaningful way. External services that provide frequent, targeted, reliable assistance that is well-aligned with the school’s efforts to advance its core mission, such as volunteer tutors who commit to substantial time working with children using methods and materials closely aligned with the school’s curriculum, can be a godsend. Occasional tutors who are unlikely to show up on all scheduled days or are unable or unwilling to use proven methods aligned with the school’s needs? Not so much.

Knowing when to encourage and even solicit help and when not to do so is an important part of school leadership. Principals and their staffs need to have clear goals and plans for how to achieve them. That has to be job one for the whole staff. External assistance that helps schools achieve their goals is great. And assistance to help achieve other valued goals, such as after school art or music or theater programs, may be fine if it does not interfere with job one. But external assistance that takes time and adds complexity? Maybe later.

Every superintendent, principal, and school staff has to have a laser focus on the core mission of their school. They have only so many hours in the day to accomplish this core mission. The government and private entities who support schools have to learn that schools must focus their energies on the main thing, academic achievement, and then help school staff to accomplish this goal if they can, and help schools keep the main thing the main thing.

Photo credit: Courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Systems

What came first? The can or the can opener?

The answer to this age-old question is that the modern can and can opener were invented at exactly the same moment. This had to be true because a can without a can opener (yes, they existed) is of very little value, and a can opener without a can is the sound of one hand clapping (i.e., less than worthless).

The can and the can opener are together a system. Between them, they make it possible to preserve, transport, and distribute foods.

In educational innovation, we frequently talk as though individual variables are sufficient to improve student achievement. You hear things like “more time-good,” “more technology-good,” and so on. Any of these factors can be effective as part of a system of innovations, or useless or harmful without other aligned components. As one example, consider time. A recent Florida study provided an extra hour each day for reading instruction, 180 hours over the course of a year, at a cost per student of $800 per student, or $300,000-$400,000 per school. The effect on reading performance, compared to schools that did not receive additional time, was very small (effect size =+0.09). In contrast, time used for one-to-one or one-to-small group tutoring by teaching assistants for example, can have a much larger impact on reading in elementary schools (effect size=+0.29), at about half the cost. As a system, cost-effective tutoring requires a coordinated combination of time, training for teaching assistants, use of proven materials, and monitoring of progress. Separately, each of these factors is nowhere near as effective as all of them taken together in a coordinated system. Each is a can with no can opener, or a can opener with no can: The sound of one hand clapping. Together, they can be very effective.

The importance of systems explains why programs are so important. Programs invariably combine individual elements to attempt to improve student outcomes. Not all programs are effective, of course, but those that have been proven to work have hit upon a balanced combination of instructional methods, classroom organization, professional development, technology, and supportive materials that, if implemented together with care and attention, have been proven to work. The opposite of a program is a “variable,” such as “time” or “technology,” that educators try to use with few consistent, proven links to other elements.

All successful human enterprises, such as schools, involve many individual variables. Moving these enterprises forward in effectiveness can rarely be done by changing one variable. Instead, we have to design coordinated plans to improve outcomes. A can opener can’t, a can can’t, but together, a can opener and a can can.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Time Passes. Will You?

When I was in high school, one of my teachers posted a sign on her classroom wall under the clock:

Time passes. Will you?

Students spend a lot of time watching clocks, yearning for the period to be over. Yet educators and researchers often seem to believe that more time is of course beneficial to kids’ learning. Isn’t that obvious?

In a major review of secondary reading programs I am completing with my colleagues Ariane Baye, Cynthia Lake, and Amanda Inns, it turns out that the kids were right. More time, at least in remedial reading, may not be beneficial at all.

Our review identified 60 studies of extraordinary quality- mostly large-scale randomized experiments- evaluating reading programs for students in grades 6 to 12. In most of the studies, students reading 2 to 5 grade levels below expectations were randomly assigned to receive an extra class period of reading instruction every day all year, in some cases for two or three years. Students randomly assigned to the control group continued in classes such as art, music, or study hall. The strategies used in the remedial classes varied widely, including technology approaches, teaching focused on metacognitive skills (e.g., summarization, clarification, graphic organizers), teaching focused on phonics skills that should have been learned in elementary school, and other remedial approaches, all of which provided substantial additional time for reading instruction. It is also important to note that the extra-time classes were generally smaller than ordinary classes, in the range of 12 to 20 students.

In contrast, other studies provided whole class or whole school methods, many of which also focused on metacognitive skills, but none of which provided additional time.

Analyzing across all studies, setting aside five British tutoring studies, there was no effect of additional time in remedial reading. The effect size for the 22 extra-time studies was +0.08, while for 34 whole class/whole school studies, it was slightly higher, ES =+0.10. That’s an awful lot of additional teaching time for no additional learning benefit.

So what did work? Not surprisingly, one-to-one and small-group tutoring (up to one to four) were very effective. These are remedial and do usually provide additional teaching time, but in a much more intensive and personalized way.

Other approaches that showed particular promise simply made better use of existing class time. A program called The Reading Edge involves students in small mixed-ability teams where they are responsible for the reading success of all team members. A technology approach called Achieve3000 showed substantial gains for low-achieving students. A whole-school model called BARR focuses on social-emotional learning, building relationships between teachers and students, and carefully monitoring students’ progress in reading and math. Another model called ERWC prepares 12th graders to succeed on the tests used to determine whether students have to take remedial English at California State Universities.

What characterized these successful approaches? None were presented as remedial. All were exciting and personalized, and not at all like traditional instruction. All gave students social supports from peers and teachers, and reasons to hope that this time, they were going to be successful.

There is no magic to these approaches, and not every study of them found positive outcomes. But there was clearly no advantage of remedial approaches providing extra time.

In fact, according to the data, students would have done just as well to stay in art or music. And if you’d asked the kids, they’d probably agree.

Time is important, but motivation, caring, and personalization are what counts most in secondary reading, and surely in other subjects as well.

Time passes. Kids will pass, too, if we make such good use of our time with them that they won’t even notice the minutes going by.