What’s the Evidence that Evidence Works?

I recently gave a couple of speeches on evidence-based reform in education in Barcelona.  In preparing for them, one of the organizers asked me an interesting question: “What is your evidence that evidence works?”

At one level, this is a trivial question. If schools select proven programs and practices aligned with their needs and implement them with fidelity and intelligence, with levels of resources similar to those used in the original successful research, then of course they’ll work, right? And if a school district adopts proven programs, encourages and funds them, and monitors their implementation and outcomes, then of course the appropriate use of all these programs is sure to enhance achievement district-wide, right?

Although logic suggests that a policy of encouraging and funding proven programs is sure to increase achievement on a broad scale, I like to be held to a higher standard: Evidence. And, it so happens, I happen to have some evidence on this very topic. This evidence came from a large-scale evaluation of an ambitious, national effort to increase use of proven and promising schoolwide programs in elementary and middle schools, in a research center funded by the Institute for Education Sciences (IES) called the Center for Data-Driven Reform in Education, or CDDRE (see Slavin, Cheung, Holmes, Madden, & Chamberlain, 2013). The name of the program the experimental schools used was Raising the Bar.

How Raising the Bar Raised the Bar

The idea behind Raising the Bar was to help schools analyze their own needs and strengths, and then select whole-school reform models likely to help them meet their achievement goals. CDDRE consultants provided about 30 days of on-site professional development to each district over a 2-year period. The PD focused on review of data, effective use of benchmark assessments, school walk-throughs by district leaders to see the degree to which schools were already using the programs they claimed to be using, and then exposing district and school leaders to information and data on schoolwide programs available to them, from several providers. If districts selected a program to implement, their district and school received PD on ensuring effective implementation and principals and teachers received PD on the programs they chose.

Evaluating Raising the Bar

In the study of Raising the Bar we recruited a total of 397 elementary and 225 middle schools in 59 districts in 7 states (AL, AZ, IN, MS, OH, TN). All schools were Title I schools in rural and mid-sized urban districts. Overall, 30% of students were African-American, 20% were Hispanic, and 47% were White. Across three cohorts, starting in 2005, 2006, or 2007, schools were randomly assigned to either use Raising the Bar, or to continue with what they were doing. The study ended in 2009, so schools could have been in the Raising the Bar group for two, three, or four years.

Did We Raise the Bar?

State test scores were obtained from all schools and transformed to z-scores so they could be combined across states. The analyses focused on grades 5 and 8, as these were the only grades tested in some states at the time. Hierarchical linear modeling, with schools nested within districts, were used for analysis.

For reading in fifth grade, outcomes were very good. By Year 3, the effect sizes were significant, with significant individual-level effect sizes of +0.10 in Year 3 and +0.19 in Year 4. In middle school reading, effect sizes reached an effect size of +0.10 by Year 4.

Effects were also very good in fifth grade math, with significant effects of +0.10 in Year 3 and +0.13 in Year 4. Effect sizes in middle school math were also significant in Year 4 (ES=+0.12).

Note that these effects are for all schools, whether they adopted a program or not. Non-experimental analyses found that by Year 4, elementary schools that had chosen and implemented a reading program (33% of schools by Year 3, 42% by Year 4) scored better than matched controls in reading. Schools that chose any reading program usually chose our Success for All reading program, but some chose other models. Even in schools that did not adopt reading or math programs, scores were always higher, on average, (though not always significantly higher) than for schools that did not choose programs.

How Much Did We Raise the Bar?

The CDDRE project was exceptional because of its size and scope. The 622 schools, in 59 districts in 7 states, were collectively equivalent to a medium-sized state. So if anyone asks what evidence-based reform could do to help an entire state, this study provides one estimate. The student-level outcome in elementary reading, an effect size of +0.19, applied to NAEP scores, would be enough to move 43 states to the scores now only attained by the top 10. If applied successfully to schools serving mostly African American and Hispanic students or to students receiving free- or reduced-price lunches regardless of ethnicity, it would reduce the achievement gap between these and White or middle-class students by about 38%. All in four years, at very modest cost.

Actually, implementing something like Raising the Bar could be done much more easily and effectively today than it could in 2005-2009. First, there are a lot more proven programs to choose from than there were then. Second, the U.S. Congress, in the Every Student Succeeds Act (ESSA), now has definitions of strong, moderate, and promising levels of evidence, and restricts school improvement grants to schools that choose such programs. The reason only 42% of Raising the Bar schools selected a program is that they had to pay for it, and many could not afford to do so. Today, there are resources to help with this.

The evidence is both logical and clear: Evidence works.

Reference

Slavin, R. E., Cheung, A., Holmes, G., Madden, N. A., & Chamberlain, A. (2013). Effects of a data-driven district reform model on state assessment outcomes. American Educational Research Journal, 50 (2), 371-396.

Photo by Sebastian Mary/Gio JL [CC BY-SA 2.0  (https://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

First There Must be Love. Then There Must be Technique.

I recently went to Barcelona. This was my third time in this wonderful city, and for the third time I visited La Sagrada Familia, Antoni Gaudi’s breathtaking church. It was begun in the 1880s, and Gaudi worked on it from the time he was 31 until he died in 1926 at 74. It is due to be completed in 2026.

Every time I go, La Sagrada Familia has grown even more astonishing. In the nave, massive columns branching into tree shapes hold up the spectacular roof. The architecture is extremely creative, and wonders lie around every corner.

I visited a new museum under the church. At the entrance, it had a Gaudi quote:

First there must be love.

Then there must be technique.

This quote sums up La Sagrada Familia. Gaudi used complex mathematics to plan his constructions. He was a master of technique. But he knew that it all meant nothing without love.

In writing about educational research, I try to remind my readers of this from time to time. There is much technique to master in creating educational programs, evaluating them, and fairly summarizing their effects. There is even more technique in implementing proven programs in schools and classrooms, and in creating policies to support use of proven programs. But what Gaudi reminds us of is just as essential in our field as it was in his. We must care about technique because we care about children. Caring about technique just for its own sake is of little value. Too many children in our schools are failing to learn adequately. We cannot say, “That’s not my problem, I’m a statistician,” or “that’s not my problem, I’m a policymaker,” or “that’s not my problem, I’m an economist.” If we love children and we know that our research can help them, then it’s all of our problems. All of us go into education to solve real problems in real classrooms. That’s the structure we are all building together over many years. Building this structure takes technique, and the skilled efforts of many researchers, developers, statisticians, superintendents, principals, and teachers.

Each of us brings his or her own skills and efforts to this task. None of us will live to see our structure completed, because education keeps growing in techniques and capability. But as Gaudi reminds us, it’s useful to stop from time to time and remember why we do what we do, and for whom.

Photo credit: By Txllxt TxllxT [CC BY-SA 4.0  (https://creativecommons.org/licenses/by-sa/4.0)], from Wikimedia Commons

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Teachers’ Crucial Role in School Health

These days, many schools are becoming “community schools,” or centering on the “whole child.”  A focus of these efforts is often on physical health, especially vision and hearing. As readers of this blog may know, our group at Johns Hopkins School of Education is collaborating on creating, evaluating, and disseminating a strategy, called Vision for Baltimore, to provide every Baltimore child in grades PK to 8 who needs them with eyeglasses. Our partners are the Johns Hopkins Hospital’s Wilmer Eye Institute, the Baltimore City Schools, the Baltimore City Health Department, Vision to Learn, a philanthropy that provides vision vans to do vision testing, and Warby Parker, which provides children with free eyeglasses.  We are two years into the project, and we are learning a lot.

The most important news is this:  you cannot hope to solve the substantial problem of disadvantaged children who need glasses but don’t have them by just providing vision testing and glasses.  There are too many steps to the solution.  If things go wrong and are not corrected at any step, children end up without glasses.   Children need to be screened, then (in most places) they need parent permission, then they need to be tested, then they need to get correct and attractive glasses, then they have to wear them every day as needed, and then they have to replace them if they are lost or broken.

School of Education graduate student Amanda Inns is writing her dissertation on many of these issues. By listening to teachers, learning from our partners, and trial and error, Amanda, ophthalmologist Megan Collins, Lead School Vision Advocate Christine Levy, and others involved in the project came up with practical solutions to a number of problems. I’ll tell you about them in a moment, but here’s the overarching discovery that Amanda reports:

Nothing can stop a motivated and informed teacher.

What Amanda found out is that all aspects of the vision program tended to work when teachers and principals were engaged and empowered to try to ensure that all children who needed glasses got them, and they tended not to work if teachers were not involved.  A moment’s reflection would tell you why.  Unlike anyone else, teachers see kids every day. Unlike anyone else, they are likely to have good relationships with parents.  In a city like Baltimore, parents may not answer calls from most people in their child’s school, and even less anyone in any other city agency.  But they will answer their own child’s teacher. In fact, the teacher may be the only person in the school, or indeed in the entire city, who knows the parents’ real, current phone number.

In places like Baltimore, many parents do not trust most city institutions. But if they trust anyone, it’s their own child’s teacher.

Cost-effective citywide systems are needed to solve widespread health problems, such as vision (about 30% of Baltimore children need glasses, and few had them before Vision for Baltimore began).  Building such systems should start, we’ve learned, with the question of how teachers can be enabled and supported to ensure that services are actually reaching children and parents. Then you have to work backward to fill in the rest of the system.

Obviously, teachers cannot do it alone. In Vision for Baltimore, the Health Department expanded its screening to include all grades, not just state-mandated grades (PK, 1, 8, and new entrants).  Vision to Learn’s vision vans and vision professionals are extremely effective at providing testing. Free eyeglasses, or ones paid for by Medicaid, are essential. The fact that kids (and teachers) love Warby Parker glasses is of no small consequence. School nurses and parent coordinators play a key role across all health issues, not just vision. But even with these services, universal provision of eyeglasses to students who need them, long-term use, and replacement of lost glasses, are still not guaranteed.

Our basic approach is to provide services and incentives to help teachers be as effective as possible in supporting vision health, using their special position to solve key problems. We hired three School Vision Advocates (SVAs) to work with about 43 schools entering the vision system each year. The SVAs work with teachers and principals to jointly plan how to ensure that all students who need them will get, use, and maintain their glasses. They interact with principals and staff members to build excitement about eyeglasses, offering posters to place around each school. They organize data to make it easy for teachers to know which students need glasses, and which students still need parent permission forms. They provide fun prizes, $20 worth of school supplies, to all teachers in schools in which 80% of parents sent in their forms.  They get to know school office staff, also critical to such efforts.  They listen to teachers and get their ideas about how to adapt their approaches to the school’s needs.  They do observations in classes to see what percentage of students are wearing their glasses. They arrange to replace lost or broken glasses.  They advocate for the program in the district office, and find ways to get the superintendent and other district leaders to show support for the teachers’ activities directed toward vision.

Amanda’s research found that introducing SVAs into schools had substantial impacts on rates of parent permission, and adding the school prizes added another substantial amount. Many students are wearing their glasses. There is still more to do to get to 100%, but the schools have made unprecedented gains.

Urban teachers are very busy, and adding vision to their list of responsibilities can only work if the teachers see the value, feel respected and engaged, and have help in doing their part. What Amanda’s research illustrates is how modest investments in friendly and capable people targeting high-leverage activities can make a big difference across an entire city.

Ensuring that all students have good vision is critical, and a hit-or-miss strategy is not sufficient. Schools need systems to bring cost-effective services to thousands of kids who need help with health issues that affect very large numbers, such as vision, hearing, and asthma. Teachers still must focus first on their roles as instructors, but with help, they can also provide essential assistance to build the health of their students.  No system to solve health problems that require daily, long-term monitoring of children’s behavior can work at this scale in urban schools without engaging teachers.

Photo credit: By SSgt Sara Csurilla [Public domain], via Wikimedia Commons

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.