Why Isn’t Achievement Whirled Enough by Time? (Why Summer School, After School, and Extended Day Do Not Work Very Well)

“Had we but world enough and time…” wrote Andrew Marvell, an English poet in the late 1600s (He also had another job, highly relevant to this blog, which I will reveal at the end. No peeking!).

Marvell’s poem was about making the most of the limited time we have on Earth. In education, we understand this sentiment. Time is a key resource for teaching, not to be wasted under any circumstances.

In fact, educators have long tried to improve students’ achievement by increasing their time in school. In particular, struggling students have been invited or required to attend after school or summer school classes.

 Many school reformers have advocated expanded opportunities for extra-time instruction, as solutions to the learning losses due to Covid-19 school closures. In fact, the current draft of the Democrats’ relief bill emphasizes investments in after school and summer school programs to help these students catch up. Yet these very expensive efforts have not had much impact on reading or math learning in studies done before Covid, and are not likely to have much impact now (see my previous blog on this topic).

How can this be? Summer school, for example, offers several weeks of extra teaching in small classes tailored to the learning levels of the students. Yet summer school for reading has been completely ineffective, except for tutoring phonics in K-1. Math summer school studies involving disadvantaged and low-achieving students also found effect sizes near zero (Xie et al., 2020).

With respect to after-school programs, a review by Kidron & Lindsay (2014) found average effect sizes near zero.

A study in Milwaukee by Heinrich et al. (2009) of after school programs provided under Supplemental Education Services (SES) funding found effect sizes near zero for middle and high school students. The authors investigated the reasons for these disappointing findings. Among eligible students, 57% registered in the first year, dropping to 48% by the fourth year. Yet the bigger problem was attendance. As a percent of registered students, attendance dropped from 90% in the first year to 34% in the fourth, meaning that among all eligible students, only 16% attended in the final year. This abysmal attendance rate should not be surprising in light of the observation in the study that most of the after-school time was spent on worksheets, with little or no instruction. The Heinrich et al. (2009) paper contained the following depressing sentence:

“…one might also speculate that parents and students are, in fact, choosing rationally in not registering for or attending SES.” (p. 296).

Reviews of research on the impacts of all approaches to SES find average effects that are appalling (e.g., Chappell et al., 2011). I will write more about SES as a cautionary tale in a later blog, but one conclusion important to the blog is clear: Providing educational programs to struggling students after school or in the summer is unlikely to improve student achievement.

The reasons that additional time after school or in the summer does not enhance achievement is obvious, if you’ve ever been a teacher or a student. No one wants to be sitting in school while their friends are out playing. Extra time approaches that simply provide more of the same are probably boring, tedious, and soul-sapping. Imagine kids watching the clock, quietly cheering for every click. It is no wonder that students fail to register or fail to attend after school or summer school sessions, and learn little in them if they do.

The poet Andrew Marvell had it right. What is important is to make effective use of the time we have, rather than adding time. And his profession, other than being a poet? He was a tutor.

References

Chappell, S., Nunnery, J., Pribesh, S., & Hager, J. (2011). A meta-analysis of Supplemental Education Services (SES) provider effects on student achievement. Journal of Education for Students Placed at Risk, 16 (1), 1-23.

Heinrich, C. J., Meyer, R., H., & Whitten, G. W. (2010). Supplemental Education Services under No Child Left Behind: Who signs up and what do they gain? Education Evaluation and Policy Analysis, 32, 273-298.

Kidron, Y., & Lindsay, J. (2014). The effects of increased learning time on student academic and nonacademic outcomes: Findings from a meta‑analytic review (REL 2014-015). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Appalachia.

Xie, C., Neitzel, A., Cheung, A., & Slavin, R. E. (2020). The effects of summer programs on K-12 students’ reading and mathematics achievement: A meta-analysis. Manuscript submitted for publication.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

Thorough Implementation Saves Lives

In an article in the May 23 Washington Post, Dr. John Barry, a professor of Public Health and Tropical Medicine at Tulane, wrote about lessons from the 1918 influenza epidemic.  Dr. Barry is the author of a book about that long-ago precursor to the epidemic we face today.  I found the article chilling, in light of what is happening right now in the Covid-19 pandemic.

In particular, he wrote about a study of Army training camps in 1918.  Army leaders prescribed strict isolation and quarantine measures, and most camps followed this guidance.  However, some did not.  Most camps that did follow the guidance did so rigorously for a few weeks, but then gradually loosened up.  The study compared the camps that never did anything to the camps that followed the guidelines for a while.  There were no differences in the rates of sickness or death.  However, a third set of camps continued to follow the guidance for a much longer time. These camps saw greatly reduced rates of sickness and death.

Camp Funston at Fort Riley, Kansas, during the 1918 flu pandemic, Armed Forces Institute of Pathology/National Museum of Health and Medicine, distributed via the Associated Press / Public domain

Dr. Barry also gave an example from the SARS epidemic in the early 2000’s.  President George W. Bush wanted to honor the one hospital in the world with the lowest rate of SARS infection among staff.  A study of that hospital found that they were doing exactly what all hospitals were doing, making sure that staff maintained sterile procedures.  The difference was that in this hospital, the hospital administration made sure that these rules were being rigorously followed.  This reminds me of a story by Atul Gawande about the most successful hospital in the world for treating cystic fibrosis. Researchers studied this hospital, an ordinary, non-research hospital in a Minneapolis suburb.  The physician in charge of cystic fibrosis was found to be using the very same procedures and equipment that every other hospital used.  The difference was that he frequently called all of his patients to make sure they were using the equipment and procedures properly.  His patients had markedly higher survival rates than did patients in similar hospitals doing exactly the same (medical) things with less attention to fidelity.

Now consider what is happening in the U.S. in our current pandemic.  Given our late start, we have done a pretty good job reducing rates of disease and death, compared to what might have been.  However, all fifty states are now opening up, to one degree or another.  The basic message: “We have been careful long enough.  Now let’s get sloppy.”

Epidemiologists are watching all of this with horror.  They know full well what is coming.  Leana Wen, Baltimore’s former Health Commissioner, explained the consequences of the choices we are making in a deeply disturbing article in the May 13 Post.

The entire story of what has happened in the Covid-19 crisis, and what is likely to happen now, has a substantial resonance with problems we experience in educational reform.  Our field is full of really good ideas for improving educational outcomes.  However, we have relatively few examples of programs that have been successful even in one-year evaluations, much less over extended time periods at large scale.  The problem is not usually that the ideas turn out not to be so good after all, but that they are rarely implemented with consistency or rigor. Or they are implemented well for a while, but get sloppy over time, or stop altogether.  I am often asked how long innovators must stay connected with schools using their research-proven programs with success.  My answer is, “forever.”  The connection need not be frequent in successful implementers, but someone who knows what the program is supposed to look like needs to check in from time to time to see how things are going, to cheer the school on in its efforts to maintain and constantly improve their implementation, and to help the school identify and solve any problems that have cropped up.

Another thing I am frequently asked is how I can base my argument for evidence-based education on the examples of medicine and other evidence-based fields.  “Taking a pill is not like changing a school.”  This is true.  However, the examples of epidemiology, cystic fibrosis (before the recent cure was announced), dealing with obesity and drug abuse, and many other problems of medicine and public health, actually look quite a bit like the problems of education reform.  In medicine, there is a robust interest in “implementation science,” focused on, among other things, getting people to take their medicine or follow a proven protocol (e.g., “eat more veggies”).  There is growing interest in implementation science in education, too.  Similar problems, similar solutions, in many cases.

Education, public health, and medicine have a lot to learn from each other.  In each case, we are trying to make important differences in whole populations.  It is never easy, but in each of our fields, we are learning how to cost-effectively increase health and education outcomes at scale.  In the current pandemic, I hope science will prevail in both reducing the impact of the disease and in using proven practices, with consistency and rigor, to help schools repair the educational damage children have suffered.

References

Barry, J.M. (2020, May 23).  How to avoid a second wave of infections.  Washington Post.

Wen, L.S. (2020, May 13).  We are retreating to a new strategy on covid-19.  Let’s call it what it is.  Washington Post.

This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

 

Would Your School or District Like to Participate in Research?

As research becomes more influential in educational practice, it becomes important that studies take place in all kinds of schools. However, this does not happen. In particular, the large-scale quantitative research evaluating practical solutions for schools tends to take place in large, urban districts near major research universities. Sometimes they take place in large, suburban districts near major research universities. This is not terribly surprising, because in order to meet the highest standards of the What Works Clearinghouse or Evidence for ESSA, a study of a school-level program will need 40 to 50 schools willing to be assigned at random to either use a new program or to serve as a control group.

Naturally, researchers want to have to deal with a small number of districts (to avoid having to deal with many different district-level rules and leaders), so they try to sign up districts in which they might find 40 or 50 schools willing to participate, or perhaps split between two or three districts at most. But there are not that many districts with that number of schools. Further, researchers do not want to spend their time or money flying around to visit schools, so they usually try to find schools close to home.

As a result of these dynamics, of course, it is easy to predict where high-quality quantitative research on innovative programs is not going to take place very often. Small districts (even urban ones) can be hard to serve, but the main category of schools left out of big studies are ones in rural districts. This is not only unfair, but it deprives rural schools of a robust evidence base for practice. Also, it can be a good thing for schools and districts anywhere to participate in research. Typically, schools are paired and assigned at random to treatment or control groups. Treatment groups get the treatment, and control schools usually get some incentive, such as money, or an opportunity to use the innovative treatment a year after the experiment is over. So why should some places get all this attention and opportunity, while others complain that they never get to participate and that there are few programs evaluated in districts like theirs?

I have a solution to propose for this problem: A “Registry of Districts and Schools Seeking Research Opportunities.” The idea is that district leaders or principals could list information about themselves and the kinds of research they might be willing to host in their schools or districts. Researchers seeking district or school partners for proposals or funded projects could post invitations for participation. In this way, researchers could find out about districts they might never have otherwise considered, and district and school leaders could find out about research opportunities. Sort of like a dating site, but adapted to the interests of researchers and potential research partners (i.e., no photos would be required).

Scientists consulting a registry of volunteer participants.

If this idea interests you, or if you would like to participate, please write to Susan Davis at sdavi168@jh.edu . If you wish, you can share any opinions and ideas about how such a registry might best accomplish its goals. If you represent a district or school and are interested in participating in research, tell us, and I’ll see what I can do.

If I get lots of encouragement, we might create such a directory and operate it on behalf of all districts, schools, and researchers, to benefit students. I’ll look forward to hearing from you!

 This blog was developed with support from Arnold Ventures. The views expressed here do not necessarily reflect those of Arnold Ventures.

Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

Preschool: A Step, Not a Journey

“A journey of a thousand miles begins with a single step.”

So said Lau Tzi (or Lau Tzu), the great Chinese scholar who lived in the 6^th century BC.

For many years, especially since the extraordinary long-term outcomes of the Perry Preschool became known, many educators have seen high-quality preschool as an essential “first step” in a quality education. Truly, a first step in a journey of a thousand miles. Further, due to the Perry Preschool findings, educators, researchers, and policy makers have maintained that quality preschool is not only the first step in a quality education, but it is the most important, capable of making substantial differences in the lives of disadvantaged students.

I believe, based on the evidence, that high-quality preschool helps students enter kindergarten and, perhaps, first grade, with important advantages in academic and social skills. It is clear that quality preschool can provide a good start, and for this reason, I’d support investments in providing the best preschool experiences we can afford.

But the claims of most preschool advocates go far beyond benefits through kindergarten. We have been led to expect benefits that last throughout children’s lives.

Would that this were so, but it is not. The problem is that randomized studies rarely find long-term impacts. In such studies, children are randomly assigned to receive specific, high-quality preschool services or to serve in a control group, in which children may remain at home or may receive various daycare or preschool experiences of varying quality. In randomized long-term studies comparing students randomly assigned to preschool or business as usual, the usual pattern of findings shows positive effects on many measures at the end of the preschool year, fading effects at the end of kindergarten, and no differences in later years. One outstanding example is the Tennessee Voluntary Prekindergarten Program (Lipsey, Farran, & Durkin, 2018). A national study of Head Start by Puma, Bell, Cook, & Heid (2010) found the same pattern, as did randomized studies in England (Melhuish et al., 2010) and Australia (Claessens & Garrett, 2014). Reviews of research routinely identify this consistent pattern (Chambers, Cheung, & Slavin, 2017; Camilli et al., 2009; Melhuish et al., 2010).

So why do so many researchers and educators believe that there are long-term positive effects of preschool? There are two answers. One is the Perry Preschool, and the other is the use of matched rather than randomized study designs.

The Perry Preschool study (Schweinhart & Weikart, 1997) did use a randomized design, but it had many features that made it an interesting pilot rather than a conclusive demonstration of powerful and scalable impacts. First, the Perry Preschool study had a very small sample (initially, 123 students in a single school in Ypsilanti, Michigan). It allowed deviations from random assignment, such as assigning children whose mothers worked to the control group. It provided an extraordinary level of services, never intended to be broadly replicable. Further, the long-term effects were never seen on elementary achievement, but only appeared when students were in secondary school. It seems unlikely that powerful impacts could be seen after there were no detectable impacts in all of elementary school. No one can fully explain what happened, but it is important to note that no one has replicated anything like what the Perry Preschool did, in all the years since the program was implemented in 1962-1967.

With respect to matched study designs, which do sometimes find positive longitudinal effects, a likely explanation is that with preschool children, matching fails to adequately control for initial differences. Families that enroll their four-year-olds in preschool tend, on average, to be more positively oriented toward learning and more eager to promote their children’s academic success. Well-implemented matched designs in the elementary and secondary grades invariably control for prior achievement, and this usually does a good job of equalizing matched samples. With four-year-olds, however, early achievement or IQ tests are not very reliable or well-correlated with outcomes, so it is impossible to know how much matching has equalized the groups on key variables.

Preparing for a Journey

Lao Tzi’s observation reminds us that any great accomplishment is composed of many small, simple activities. Representing a student’s educational career as a journey, this fits. One grand intervention at one point in that journey may be necessary, but it is not sufficient to ensure the success of the journey. In the journey of education, it is surely important to begin with a positive experience, one that provides children with a positive orientation toward school, skills needed to get along with teachers and classmates, knowledge about how the world works, a love for books, stories, and drama, early mathematical ideas, and much more. This is the importance of preschool. Yet it is not enough. Major make-or-break objectives lie in the future. In the years after preschool, students must learn to read proficiently, they must learn basic concepts of mathematics, and they must continue to build social-emotional skills for the formal classroom setting. In the upper elementary grades, they must learn to use their reading and math skills to learn to write effectively, and to learn science and social studies. Then they must make a successful transition to master the challenges of secondary school, leading to successful graduation and entry into valued careers or post-secondary education. Each of these accomplishments, along with many others, requires the best teaching possible, and each is as important and as difficult to achieve for every child as is success in preschool.

A journey of a thousand miles may begin with a single step, but what matters is how the traveler negotiates all the challenges between the first step and the last one. This is true of education. We need to find effective and replicable methods to maximize the possibility that every student will succeed at every stage of the learning process. This can be done, and every year our profession finds more and better ways to improve outcomes at every grade level, in every subject. Preschool is only the first of a series of opportunities to enable all children to reach challenging goals. An important step, to be sure, but not the whole journey.

Photo courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action.

References

Camilli, G., Vargas, S., Ryan, S., & Barnett, S. (2009). Meta-analysis of the effects of early education interventions on cognitive and social development. Teachers College Record, 112 (3), 579-620.

Chambers, B., Cheung, A., & Slavin, R.E. (2016) Literacy and language outcomes of comprehensive and developmental-constructivist approaches to early childhood education: A systematic review. Educational Research Review, 18, 88-111..

Claessens, A., & Garrett, R. (2014). The role of early childhood settings for 4-5 year old children in early academic skills and later achievement in Australia. Early Childhood Research Quarterly, 29, (4), 550-561.

Lipsey, M., Farran, D., & Durkin, K. (2018). Effects of the Tennessee Prekindergarten Program on children’s achievement and behavior through third grade. Early Childhood Research Quarterly, 45 (4), 155-176.

Melhuish, E., Belsky, J., & Leyland, R. (2010). The impact of Sure Start local programmes on five year olds and their families. London: Department for Education.

Puma, M., Bell, S., Cook, R., & Heid, C. (2010). Head Start impact study: Final report. Washington, DC: U.S. Department of Health and Human Services.

Schweinhart, L. J., & Weikart, D. P. (1997). Lasting differences: The High/Scope Preschool curriculum comparison study through age 23 (Monographs of the High/Scope Educational Research Foundation No. 12) Ypsilanti, MI: High/Scope Press.

 Note: If you would like to subscribe to Robert Slavin’s weekly blogs, just send your email address to thebee@bestevidence.org. 

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.