Sunset for Textbook Adoption in California

There is great news from California. The State Board of Education has announced that it will drop its longstanding textbook adoption standards, which for many years have only allowed California schools to use state textbook funds on a limited set of choices. Many states have approved lists of textbooks, but California was relatively unique in limiting options to a very short list. For example, in recent years, California allowed only two basal reading series, Open Court and Houghton Mifflin.

For many years, California and the other large textbook adoption state, Texas, have had an outsized impact on textbooks everywhere, because publishers have had to dance to their tune. This has had a negative effect nationally. It has restricted innovation, and kept smaller competitors out of the textbook market, since it is hugely expensive to go through the adoption process.

For an advocate of evidence-based reform, the California and Texas textbook adoption policies are particularly galling. While California had rules against mentioning junk food in texts, for example, no textbook adoption state has ever paid any attention to the evidence base supporting the use of particular texts. This meant that in California, districts could use their textbook funds on texts found in high-quality research to be ineffective, or ones that had never been evaluated, while programs with strong evidence of effectiveness were effectively banned from the state.

I often hear the objection to evidence-based reform that schools should not be required to use proven models. I happen to agree, and have always argued that the proper role of government is to provide encouragement and incentives to use proven programs, not to mandate them. Yet it strikes me as absurd that California was requiring the use of particular programs based on no evidence at all, and Texas and other textbook adoption states continue to do so.

State textbook adoption lists have got to go. They should be replaced with solid information on the outcomes of various programs, perhaps with incentive funding to choose the best-evaluated. But schools and districts should be free to use that information to make whatever choices they believe to be best for their kids, knowing that they will ultimately be held accountable for the outcomes.

Don’t forget to follow me on Facebook and Twitter.

What the Presidential Election Might Mean for Evidence-Based Reform

Watching the presidential debates, I wasn’t terribly surprised to see that evidence-based reform in education was not mentioned. In a rational world it would have been, but maybe that is just my own irrational world view.

Still, it is possible to anticipate what the future might be for evidence-based reform in Obama or Romney administrations. Arne Duncan says he’s staying, so a second Obama administration will surely build on the first. This is mostly good news. The current administration, especially the Office of Management and Budget (OMB), has spoken strongly in favor of evidence-based policies. The current administration began i3, which is the most important concrete step ever taken toward evidence-based reform, and it favors ARPA-ED, which would also apply evidence to accelerate technology innovation in education.

Where the Obama administration has not yet gone is toward using evidence-proven approaches to improve outcomes in the main federal investments in education: Title I, School Improvement Grants, IDEA, and so on. As the economy improves and schools have more breathing space, this may change, but moving evidence into policy still has a ways to go.

What would a Romney administration do? In his education positions, Romney sounds more moderate than he does in other areas, and I think he genuinely wants to improve education and make it more cost-effective. In the United Kingdom, the conservative government of David Cameron has been far more favorable to evidence-based reform than was the previous Labor government. So one could hope that a Romney administration might similarly focus on finding out what works and supporting it. On the other hand, while it is true that more money does not automatically improve outcomes, it is also true that things that do improve outcomes will cost something in the short term. If a Romney presidency means substantial cuts in education funding, schools will be hunkering down to protect staff, not looking for proven approaches.

Whoever wins the election is likely to face a divided and polarized Congress and a raft of politically charged problems. But wouldn’t it be great if the next administration embraces evidence as a way forward through the difficulties our schools face?

Don’t forget to follow me on Facebook and Twitter.

More OMG From OMB

Imagine that education leaders began to encourage or provide incentives for schools to use proven programs and practices. Imagine that instead of a confused patchwork of policies and grants, government had a simple rule: if it works, we’ll help you adopt it. If it hasn’t yet been proven to work, we’ll help you evaluate it. If it’s just a good idea, we’ll help you move it forward. But the purpose of education policy is to find out what works and then help scale it up.

In a speech this summer, Robert Gordon from the OMB laid out such a vision. He was among friends, speaking to recipients of Investing in Innovation (i3) grants designed to build up our nation’s “shelf” of proven programs. He spoke about OMB’s memo to all government departments encouraging policies to favor use of proven programs, which I’ve written about before.

Gordon’s speech provided an opportunity to reflect on how far evidence-based reform has come in recent years. No previous leader from OMB has ever shown any interest in educational research and development, although Robert Shay, OMB Director under President George W. Bush, has recently expressed support for investing in programs with evidence of effectiveness from rigorous evaluations. Until fairly recently, research in education was mostly done by and for academics, and no one expected it to make much of a difference. Today, as I noted in a recent blog, we have the distinguished Lisbeth Schorr worrying that evidence from rigorous experiments might soon make too much of a difference. I think her concern is premature, but isn’t it cool to even have such a conversation?

Don’t forget to follow me on Facebook and Twitter!

Won’t Back Down: Union-Bashing Goes Hollywood

At the recent Education Nation meetings, I saw the opening of “Won’t Back Down.” If you’ve seen the movie or the reviews, you’ll know that it’s about a plucky parent, played by Maggie Gyllenhaal, who organizes parents and teachers in a terrible school to invoke a “parent trigger” law to take over the school. The movie is controversial in large part because it presents the teachers’ union, which tries to prevent the takeover, as 100 percent evil, in a time when teachers and their unions are very much under assault.

The movie does a good job of painting viewers into an emotional corner. Gyllenhaal’s second-grade daughter is not learning to read, and her incompetent, nearly comatose teacher (and her corrupt principal and uncaring district) do nothing to solve the problem, or even pretend to care. The mom, working two jobs with little money, does her best to change her daughter to a better class, then a better school, and finally tries and fails to get her into a wonderful-sounding charter school. Given all of this, what’s a mother to do?

In the movie version (this is not true in real life), the approval of half of the school’s teachers is needed to invoke the parent trigger. The mom teams up with a great teacher, played by Viola Davis. If you’ve seen any David vs. Goliath movie, you can predict the rest, except that it’s the union, not the school administration that is cast as Goliath. This is set up by the artificial device of making teacher approval a requirement for parent trigger. (This isn’t otherwise necessary since the school administration does a sufficient job of playing Goliath).

So going beyond the movie to real life, are unions part of the problem or part of the solution in school reform? Both. At their best, unions can mobilize the efforts of their teachers around real reform. In the 1990s, New York City partnered with its UFT to transform a large set of failing elementary and middle schools, in what it called the Chancellor’s District. An independent evaluation found substantial positive effects on students’ reading achievement. Further, the national AFT has been a strong advocate for evidence-based school reform. It eagerly supported the Chancellor’s District, as well as other efforts in big cities along similar lines.

Beyond their direct involvement in reform, the unions (and almost no one else) fights to increase teachers’ salaries and improve their working conditions and effectiveness. To get the most talented people to go into teaching and stay there, teachers need to be paid adequately and treated like professionals.

I’ve experienced situations in which union rules create difficulties in staffing and running schools trying to implement reforms, but I’ve never seen situations in which unions are unwilling to support reforms if they get a place at the table in planning them. Ultimately, unions are teachers, and teachers care about kids. They have to protect their members, but need to be involved in fair ways of evaluating teachers and removing incompetents from their own ranks. At Education Nation, in a panel on the movie, Randi Weingarten, President of the AFT, said that the awful teacher portrayed in “Won’t Back Down” should have been fired. I think any union leader would say the same thing.

Ultimately, what makes a difference is good teachers using proven strategies. The movie gives little sense about what the newly liberated teachers will do to improve outcomes for children. The movie’s examples of great teaching involved a teacher playing a ukulele to his kids. So the new school is sure to be more fun and more musical, but will it teach kids to read? Wouldn’t it be more sensible to offer teachers proven methods (with or without ukuleles), and then deal with the few who won’t or can’t teach? Failing schools are indeed unacceptable, but blowing them up is not the only solution.

Union bashing is not a path to school reform. Unions can and should be invited into the reform conversation. The focus of reform needs to be on improving what teachers and administrators do every day, and on attracting the best and brightest into teaching. I don’t think any union anywhere disagrees with these goals.

Don’t forget to follow me on Facebook and Twitter!

What Constitutes Strong Evidence of Program Effectiveness?

Note: This is a guest post by Jon Baron, President of the Coalition for Evidence-Based Policy, and former Chairman of the National Board for Education Sciences

Bob Slavin’s recent blog posts on Bad Measures and Brief, Small, and Artificial Studies provide a valuable discussion of how evaluation studies – even those using random assignment – can often fall well short of “rigorous.” This post seeks to address a related question: what constitutes strong evidence of effectiveness? By strong evidence, I mean evidence that provides confidence that a program would improve important educational outcomes if implemented faithfully in a similar population.

Evidence standards articulated by the Institute of Education Sciences (IES), National Academy of Sciences, and other respected scientific bodies underscore that strong evidence usually requires well-conducted randomized controlled trials. Whether the findings from these trials constitute strong evidence, however, also depends on factors such as the following.

The studies demonstrate effects on final, policy-important outcomes and not just intermediate outcomes that may or may not lead to final outcomes.

• Example – welfare/employment. The Labor Department’s New Chance demonstration – a program providing educational, job training, and other services to young welfare mothers – was found in a large randomized trial to produce a sizable increase of 12 percentage points in the mothers’ receipt of a GED (an intermediate outcome). By contrast, the study found no effects on employment, earnings, welfare dependency, or ability to read – i.e., any of the final, policy-important outcomes that were hoped for.

• Example – preschool education. HHS’s randomized trial of Head Start found a sizable effect on the program goal of increasing preschoolers’ ability to identify letters and words (an intermediate outcome), but no significant effects on their actual reading ability or other educational outcomes at the end of first grade (the more final, important outcomes).

The studies show that effects are sustained long enough to constitute meaningful improvement in educational or other key outcomes.

• Example – workforce development. The federal Job Corps program – which provides education and job training to disadvantaged youth – was prematurely declared a success based results from a large, randomized trial showing an 8% increase in earnings at the three-year follow-up, and a projection that such effects would continue over time. A later follow-up found that the earnings effects did not persist, and instead had faded to zero at the five-year point and thereafter. As a result, the program’s cost was found to greatly exceed its benefits.

The studies show that the effects are sizable (and not just statistically significant).

• Example – early childhood home visiting. Healthy Steps – which provides home visiting services to families with a newborn child – was found in a large randomized trial to produce a statistically-significant increase in the percent of mothers bringing their child to a well-child doctor visit at one month of age. However, the effect size was small: 97% of mothers receiving Healthy Steps did a one-month doctor visit, versus 95% of control group mothers. The effect reached statistical significance only because the study had a very large sample – over 2100 mothers. (Large samples are capable of detecting small effects that may not be of practical importance.)
The effects have been replicated across different studies and/or study sites, and in real-world educational settings.

• Example – K-12 education. Project CRISS – a teacher professional development program for improving adolescent reading – was considered highly promising based on a small randomized trial that met What Works Clearinghouse standards, and was therefore selected by IES for a replication trial. The initial study found a large increase in students’ reading comprehension as measured on a researcher-designed test (36 percentile points). By contrast, the more definitive, IES-sponsored replication trial – conducted in 38 high-poverty public schools, with a student sample 10 times as large as the initial study – found no effect on reading comprehension as measured on a well-established, standardized test. (This example is one of many in which IES-sponsored trials have overturned initial findings of effectiveness in small randomized trials or quasi-experiments.)

Of the educational programs often described as “evidence-based,” only a small subset meet the above conditions for strong evidence (an earlier post lists websites where they can be found). Although few in number, these programs – if implemented effectively on a large scale – could produce important gains in areas such as reading achievement, college attendance, and workforce earnings.

Programs with preliminary or moderate – as opposed to strong – evidence, offer substantially less confidence that they produce meaningful effects compared to schools’ usual practices. However, in many areas of education – such as dropout prevention – they may be the best option available to education officials, because programs with strong evidence do not yet exist. In these cases, we believe that program implementation should include a rigorous evaluation wherever feasible, to determine whether the program really works and build a more reliable foundation for future decision-making.

-Jon Baron

The Coalition for Evidence-Based Policy is a nonprofit, nonpartisan organization whose mission is to increase government effectiveness through the use of rigorous evidence about “what works.”