Accountability and Evidence

Illustration by James Bravo


At some level, just about everyone involved in education is in favor of “using what works.” There are plenty of healthy arguments about how we find out what works and how evidence gets translated into practice, but it’s hard to support a position that we shouldn’t use what works under at least some definition of evidence.

However, the dominant idea among policy makers about how we find out what works seems to be “Set up accountability systems and then learn from successful teachers, schools, systems, or states.” This sounds sensible, but in fact it is extremely difficult to do.

This point is made in a recent blog post by Tom Kane. Here’s a key section of his argument:

[In education] we tend to roll out reforms broadly, with no comparison group in mind, and hope for the best. Just imagine if we did that in health care. Suppose drug companies had not been required to systematically test drugs, such as statins, before they were marketed. Suppose drugs were freely marketed and the medical community simply stood back and monitored rates of heart disease in the population to judge their efficacy. Some doctors would begin prescribing them. Most would not. Even if the drugs were working, heart disease could have gone up or down, depending on other trends such as smoking and obesity. Two decades later, cardiologists would still be debating their efficacy. And age-adjusted death rates for heart disease would not have fallen by 60 percent [as they have] since 1980.

Kane was writing about big federal policies, such as Reading First and Race to the Top, which cannot be evaluated because they are national before their impact is known. But the same is true of smaller programs and practices. It is very difficult to look at, for example, more and less successful schools (on accountability measures) and figure out what they did that made the difference. Was it a particular program or practice that other schools could also adopt? Or was it that better-scoring schools were lucky in having better principals and teachers, or that the school’s intake or neighborhood is changing, or any number of other factors that may not even be stable for more than a year or two?

Accountability is necessary for communities to find out how students are doing. All countries have some test-based accountability (though none test every year, as we do from grades 3 through 8), but anyone who imagines that we can just look at test scores to find what works and what doesn’t is not being realistic.

The way we can find out what works is to compare schools or classrooms assigned to use any given program with those that continue current practices. Ideally, schools and classrooms are assigned at random to experimental or control groups. That’s how we find out what works in medicine, agriculture, technology, and other areas.

I know I’ve pointed this out in previous blog posts, and I’ll point it out in many to come. Sooner or later, it has to occur to our leaders that in education, too, we can use experiments to test good ideas before we subject millions of kids to something that will probably fail to improve their achievement. Again.


The Role of Research in Limited Government

Long ago Henry Kissinger said, “We can all have our own opinions, but we can’t all have our own facts.” I thought of this when I read a recent article on education reform by former Florida Gov. Jeb Bush. In it Bush argued that most education decisions should be made at the state and local levels. The same is also argued by U.S. Sen. Lamar Alexander (R-Tennessee) and U.S. Rep. John Kline (R-Minnesota), who are leading the attempt to reauthorize the Elementary and Secondary Education Act (ESEA).

What is important in Bush’s article is that while he states his belief that the federal role in education should be limited, he also discusses what it should continue to do:

[The federal government] should work to create transparency so that parents can see how their local schools measure up; it should support policies that have a proven record; and it should make sure states can’t ignore students who need extra help. That’s it.

I’m glad to see Gov. Bush say that if there are only three federal functions in education, “[supporting] polices that have a proven record” is one of them. However, if government at any level is to “support polices that have a proven record,” then it makes sense that someone has to find out what those proven policies are. Research, development, and evaluation of potential policies and practices in education are a national responsibility, not a local one. It would be inefficient to have each state developing and evaluating its own education programs. How often do states do their own cancer research, for example? States and localities depend primarily on federally funded research to help them make wise decisions at their levels.

Yet just as Bush and others in his party seem to be recognizing that research and development are among the few education-related activities that should remain at the federal level, the Republican-controlled Congress is proposing to eliminate the Investing in Innovation (i3) program and cut back other federal investments in research and development.

As I’ve said before in this space, I support the idea that the federal government should stop trying to micromanage schools. Yet states and localities still need to know which policies have a proven record. We can and should all have our own local opinions, to paraphrase Kissinger, but I hope we will first have the facts we need to make good local decisions. For these we need reliable, rigorous research. Innovations in education should be welcomed from every source — local, state, federal, or non-governmental — but evaluating these innovations and communicating their findings is fundamentally a federal responsibility.

Photo courtesy of the Library of Congress

Show Me the Evidence


This year, evidence-based reform in education got off to a great start with an article by Ron Haskins in the New York Times on December 31 explaining why evidence of effectiveness must become an expected part of the process by which policy ideas are adopted (or not). More recently, I received a book Haskins wrote with colleague Greg Margolis on this topic, Show Me the Evidence. Both the article and the book mentioned our Success for All program as an example of what “proven” looks like in education, but they are a lot more important than that.

Haskins makes a powerful argument for putting all social programs to the test. Those that work should be expanded. Those that don’t should be replaced by other approaches that work better.

The need for evidence should be obvious, but very few federal programs have evidence of effectiveness. Few even have a process for finding out what works and encouraging grantees to use proven approaches, instead of approaches with the same desired outcomes that do not work or whose effects are unknown. Haskins estimates the 75 percent of programs and practices intended to help people do better at school or work have little or no impact. Such programs are well-meaning, but they need to be improved or replaced with equally well-meaning approaches known under well-defined circumstances to have positive impacts.

The importance of Haskins’ article and book lies especially in the importance of Haskins himself. He knows whereof he speaks. Advisor to House Republicans in the 1990s and then an advisor to President George W. Bush on social policy, he was a key architect of the 1996 welfare overhaul. Welfare programs that worked improved peoples’ lives and saved federal and state governments billions of dollars. Those that didn’t were replaced. To this day, those innovations represent the best example of evidence-driven policy.

Haskins is a proud Republican. He wants every dollar of federal expenditure to do what it is intended to do. Is there anyone, of any political persuasion, who does not want the same? This is not a question of ideology. It’s a question of sound governance.

When Ron Haskins and others were starting out, evidence was a pretty risky idea. Today, evidence is showing up throughout government — still not nearly as much as it should, but far more than it did. Sooner or later, government will become more competent and cost-effective at achieving goals we all share. Ron Haskins was there when it mattered. He still is, and it still does.