Supplemental Educational Services: Noble Ideas + Unreasonable Expectations = Disappointing Results

NOTE: This is a guest post by Steven Ross, Professor in the Center for Research and Reform in Education at Johns Hopkins University

With the upcoming reauthorization of ESEA pending, and the future of Supplemental Educational Services (SES) in question, it is due time to reflect on the research and implementation lessons of this program.

Making tutoring available to increase the academic performance of low-achieving and disadvantaged students is a noble idea. After all, one-on-one and small-group tutoring have been supported by extensive research evidence, while having universal appeal as a teaching strategy. However, expectations that tutoring can be delivered efficiently and effectively when filtered through multiple layers of administrative requirements and processes are unrealistic.

First created under the last ESEA reauthorization in 2001, SES mandates school districts to offer free tutoring to disadvantaged students who attend low-performing schools. While noble in intentions, SES has turned out to be quite costly. To fund SES along with transportation for students who opt to transfer to better schools, districts must set aside 20 percent of their Title I allocations, a cost currently approximating $800 million each year.

If SES could accomplish what was originally hoped–raising student achievement sufficiently to move schools out of improvement status–the cost would be worth every penny. Unfortunately, results from numerous evaluation studies indicate much more modest effects and sometimes none at all. When serving as principal investigator of over 15 state-sponsored evaluations of SES, my impression was that the vast majority of SES providers offered quality tutoring services that were helping students both academically and socio-emotionally. But the road from the noble idea to the tutoring session is long and bumpy. Guided by lengthy federal compliance regulations, the process filters down first to the states, which are charged with approving, monitoring, and evaluating providers. Next in line are the school districts, which must fund and roll out the program locally. Parents who are low-income (and not expert about evidence-based practices) are then charged with choosing their children’s tutoring providers. The providers, in turn, must hire tutors, find tutoring space, market their services with parents, and grapple with all the federal, state, and local regulations. Ironically, those least involved with SES are the classroom teachers who deal with the children every day and best know their needs.

Research shows that, on average, SES raises participants’ reading and math scores by only .05 to .08 of a standard deviation compared to matched control groups. For example, a student participating in SES could advance from the 25th to 27nd or 28th percentile, while a comparable non-SES student would remain in the 25th percentile. For an intervention lasting only 30 to 60 hours per student, some might view such effects as a reasonable return. Reasonable or not, small gains by the relatively small subgroup of tutored students can’t do much to remove schools from “improvement status” and being required to follow the same “intervention pathway” another year.

Although the efficacy of the existing SES program needs to be questioned, after-school tutoring remains a viable intervention for boosting student achievement. Judging from the SES experience, rolling out tutoring in a large-scale, top-down, one-size-fits-all manner is not the most efficient way to expend limited Title I resources. A noble idea with much more reasonable expectations for success is to provide schools and districts freedom, with appropriate vetting, to adopt the evidence-based interventions (including tutoring, improved reading and math programs, practice-based professional development, etc.) that most directly address site-based their improvement needs.

-Steven Ross

NOTE: The Center for Research and Reform in Education at Johns Hopkins University develops the Best Evidence Encyclopedia which provides unbiased, practical reviews about the strength of evidence supporting a range of education programs. Robert Slavin is the director of CRRE.

Advertisements

Evidence of Evidence in Senate ESEA Draft

The Senate draft language for the reauthorization of the Elementary and Secondary Education Act (ESEA), released yesterday, gives me hope for evidence-based reform in education. Busy policy shops and newsrooms are still digesting the 860-page draft, and will surely provide thorough analysis in the coming days. In the meantime, I would like to highlight three critical developments for evidence-based reform:

1) The Investing in Innovation Fund (i3), only funded through appropriations in the past, would be codified in law. Already in its second round of competition, this program is no longer “new,” but a permanent authorization would be a real game changer. i3 provides funding to scale up programs that already meet a high standard of evidence and scalability and to develop and evaluate programs that appear capable of eventually reaching this standard. If authorized, i3 would produce an ever-expanding array of proven, replicable programs capable of solving all sorts of enduring problems of educational practice.

2) The draft language gives persistently low-achieving schools the option of implementing proven whole-school reform designs, defined as those found to be effective in at least two large, rigorous randomized or quasi-experimental studies. This is huge, on two counts. First, it reintroduces the idea that schools in difficulty should consider coherent, well-designed approaches, rather than employing consultants to help them start from scratch. Perhaps even more importantly, the mention of specific, relatively high standards of evidence for these programs is the first time in ESEA history that such evidence has been central to options offered to struggling schools. This is not the squishy “based on scientifically-based evidence” of NCLB. This is the real McCoy. If this provision survives into the final bill, it will lead to an acceleration of research, development, and dissemination of whole-school turnaround strategies.

3) Though not part of the draft, Senator Bennet plans to introduce an amendment authorizing ARPA-ED as a part of ESEA. Modeled after the Defense Department’s Advanced Research Projects Agency (DARPA), ARPA-ED would encourage and fund development of break-the-mold new technologies and approaches, which would need to show evidence that they improve learning substantially in comparison to current methods. New approaches could find solutions to enduring problems as diverse as the teaching of algebra, simulating experiments in science, treating reading disabilities, and helping English learners become proficient. What ARPA would add to i3 is a proactive outreach to non-traditional innovators perhaps capable of creating astonishing leaps forward in educational practice.

I have yet to read the full 860 pages released yesterday, but it is already apparent that those at the negotiating table in the Senate managed to stick to their guns on the importance of evidence in school innovation. I hope the support for innovation and evidence survives the political process. The future progress of our schools serving disadvantaged children depends on it.

For more comprehensive analysis of the draft bill, be sure to read this Education Week article, along with reactions from the field covered by PoliticsK12.

Leveraging ESEA Innovation for Impact

In federal education policy, the Elementary and Secondary Education Act (ESEA) is the big kahuna, the 800-pound gorilla, the nec plus ultra.

As the Senate HELP committee prepares to take action on the long-stalled ESEA renewal next week, it is timely to consider the law’s role in innovation. ESEA is the bedrock of educational innovation, at least for high-poverty schools. It’s been around since 1965, and it may be renamed (No Child Left Behind) or repurposed, but it’s unlikely to ever go away. Including over $14 billion a year for Title I alone, ESEA is a big chunk of change, but its influence is magnified because the Department of Education can threaten to withhold ESEA funds to state and local districts if they don’t follow federal policies.

The importance of Title I for innovation lies in the fact that schools have always been forbidden to use these funds for ordinary expenses, such as salaries, buildings, or buses. Instead it is intended to help struggling children, either through additional services (such as remedial teachers or technology) or, especially in school wide projects, professional development and program adoption. Most Title I funds are spent on instructional aides, technology, and other services with little if any evidence of effectiveness, and studies of the overall impact of Title I nationally find very small impacts. However, when effective programs are used at scale in high-poverty schools, it is almost always Title I that pays for them.

Making sure Title I has more impact on achievement should be a simple matter: Find out which replicable strategies designed for Title I schools are effective, and then encourage their use. That would be sensible, but to date, the law has not done this. What happens instead is endless debate about the rules under which Title I funds can be spent. Rules do provide some necessary guardrails, but if they do not improve what happens to children every day in school, and if they do not result in improved outcomes for Title I schools, then they do not matter.

The solution to a seemingly complex problem could be quite straightforward: Devote, say, 5 percent of ESEA funds to creating and evaluating programs that improve outcomes in Title I schools, and then offer incentives for schools to choose them. Could Congress take an innovative approach this year?