Accountability and Evidence

Illustration by James Bravo

At some level, just about everyone involved in education is in favor of “using what works.” There are plenty of healthy arguments about how we find out what works and how evidence gets translated into practice, but it’s hard to support a position that we shouldn’t use what works under at least some definition of evidence.

However, the dominant idea among policy makers about how we find out what works seems to be “Set up accountability systems and then learn from successful teachers, schools, systems, or states.” This sounds sensible, but in fact it is extremely difficult to do.

This point is made in a recent blog post by Tom Kane. Here’s a key section of his argument:

[In education] we tend to roll out reforms broadly, with no comparison group in mind, and hope for the best. Just imagine if we did that in health care. Suppose drug companies had not been required to systematically test drugs, such as statins, before they were marketed. Suppose drugs were freely marketed and the medical community simply stood back and monitored rates of heart disease in the population to judge their efficacy. Some doctors would begin prescribing them. Most would not. Even if the drugs were working, heart disease could have gone up or down, depending on other trends such as smoking and obesity. Two decades later, cardiologists would still be debating their efficacy. And age-adjusted death rates for heart disease would not have fallen by 60 percent [as they have] since 1980.

Kane was writing about big federal policies, such as Reading First and Race to the Top, which cannot be evaluated because they are national before their impact is known. But the same is true of smaller programs and practices. It is very difficult to look at, for example, more and less successful schools (on accountability measures) and figure out what they did that made the difference. Was it a particular program or practice that other schools could also adopt? Or was it that better-scoring schools were lucky in having better principals and teachers, or that the school’s intake or neighborhood is changing, or any number of other factors that may not even be stable for more than a year or two?

Accountability is necessary for communities to find out how students are doing. All countries have some test-based accountability (though none test every year, as we do from grades 3 through 8), but anyone who imagines that we can just look at test scores to find what works and what doesn’t is not being realistic.

The way we can find out what works is to compare schools or classrooms assigned to use any given program with those that continue current practices. Ideally, schools and classrooms are assigned at random to experimental or control groups. That’s how we find out what works in medicine, agriculture, technology, and other areas.

I know I’ve pointed this out in previous blog posts, and I’ll point it out in many to come. Sooner or later, it has to occur to our leaders that in education, too, we can use experiments to test good ideas before we subject millions of kids to something that will probably fail to improve their achievement. Again.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Governing From the Outside In and Back Out Again

“If a man empties his purse into his head, no one can take it from him. An investment in knowledge always pays the highest return.” – Benjamin Franklin

“We have a saying in Congress, when decisions are being made: ‘If you’re not at the table, you’re on the menu.’” – Senator Tom Harkin

When it comes to innovation, federal government bureaucrats usually do not sit around in their offices coming up with lots of brilliant and not-so-brilliant ideas. The majority of innovative ideas seized upon by the government originate elsewhere: a small program run by a local community leader; a researcher at a state university; or an engineer in a private sector company. Yet few of these people or programs, if left to their own devices, have the knowledge to conduct a moderately sized RCT or quasi-experimental study to determine the effectiveness of the program, much less any incentive or funding to do so. If told to go big or go home, most would never make it out the front door. Most of these people would be content to remain in their backyards running a program based on anecdotal evidence of accomplishment in their local community. No collaboration, no learning from other people’s mistakes, and no benefiting from other communities’ successes. Their ideas may be terrific, but local educators are not likely to evaluate or disseminate their ideas nationally.

The Fiscal Year 2016 Budget Resolution released by the majority in the U.S. House of Representatives this week asserts that the federal government works to “smother” and “stifle” innovation. It claims instead that this budget “promotes innovation” and reduces “ineffective” and “duplicative” programs. One of the ways it does this is by putting policy decisions in the hands of states and localities, which it states will lead to more “choices and opportunities for Americans.” Unfortunately, this is contrary to the intended result. States and localities do not have the capability to expand innovative ideas and disseminate the results the way the federal government can. Moreover, this would lead to greater duplication of efforts. As I’ve stated before, states do not do their own cancer research or defense technology research because it would be inefficient and uneconomical. The same is true for other areas of research and development.

Also released this week was a bipartisan paper entitled Moneyball for Education written by Frederick Hess and Bethany Little. In this paper, Hess and Little find common ground to suggest ideas for improving federal education policies using evidence and evaluation to spend taxpayer dollars more effectively and improve student outcomes. They assert that growing the database of innovative programs with evidence of positive outcomes and disseminating this information is in the public interest. It will provide greater flexibility and choice to state and local decision makers with an increased return on their investment thanks to the evidence behind the programs.

State and local governments should play a key role in the education process in education. Promising ideas should be welcome from everywhere — state and local governments, charter schools, universities, non-profits, and for profits. Innovators should be encouraged and given development funding to make their ideas practical and ready for evaluation. If the evaluations show positive effects, the programs should be scaled up nationally and offered as alternatives (not mandated) for local and state educators to use.

What I’ve just described is very similar to what the Investing in Innovation (i3) program already does. i3 is an outstanding example of how the federal government can support innovation no matter what its source and then help evaluate and disseminate information that state and local educators can use to make wise decisions. No matter which party initiated it or what it is called, this process is the way federal, state, and local government can best work together to introduce innovation without imposing mandates or restricting choice. Yet the current majority funding proposals have taken i3 out of the budget.

A healthy system of evidence-driven innovation has to involve all levels of government and many actors outside of government. It is both outside-in and inside-out, with each part of the system playing the role to which it is best suited.

Let a thousand (local) flowers bloom, and then send sacks of proven flower seeds back to the locals to use as they see fit. But there is a key step in the middle of this process that only the federal government can play: evaluation, and communicating the results of the evaluations. So it should be in education.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

The Role of Research in Limited Government

Long ago Henry Kissinger said, “We can all have our own opinions, but we can’t all have our own facts.” I thought of this when I read a recent article on education reform by former Florida Gov. Jeb Bush. In it Bush argued that most education decisions should be made at the state and local levels. The same is also argued by U.S. Sen. Lamar Alexander (R-Tennessee) and U.S. Rep. John Kline (R-Minnesota), who are leading the attempt to reauthorize the Elementary and Secondary Education Act (ESEA).

What is important in Bush’s article is that while he states his belief that the federal role in education should be limited, he also discusses what it should continue to do:

[The federal government] should work to create transparency so that parents can see how their local schools measure up; it should support policies that have a proven record; and it should make sure states can’t ignore students who need extra help. That’s it.

I’m glad to see Gov. Bush say that if there are only three federal functions in education, “[supporting] polices that have a proven record” is one of them. However, if government at any level is to “support polices that have a proven record,” then it makes sense that someone has to find out what those proven policies are. Research, development, and evaluation of potential policies and practices in education are a national responsibility, not a local one. It would be inefficient to have each state developing and evaluating its own education programs. How often do states do their own cancer research, for example? States and localities depend primarily on federally funded research to help them make wise decisions at their levels.

Yet just as Bush and others in his party seem to be recognizing that research and development are among the few education-related activities that should remain at the federal level, the Republican-controlled Congress is proposing to eliminate the Investing in Innovation (i3) program and cut back other federal investments in research and development.

As I’ve said before in this space, I support the idea that the federal government should stop trying to micromanage schools. Yet states and localities still need to know which policies have a proven record. We can and should all have our own local opinions, to paraphrase Kissinger, but I hope we will first have the facts we need to make good local decisions. For these we need reliable, rigorous research. Innovations in education should be welcomed from every source — local, state, federal, or non-governmental — but evaluating these innovations and communicating their findings is fundamentally a federal responsibility.

Photo courtesy of the Library of Congress

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Eat Your Pets

Don’t worry, Fluffy. This isn’t about you. I’m talking about Chia Pets.

Many years ago, someone invented little terracotta animals on which you could plant chia seeds that would grow into a coat for the animal. These were all the rage a while back and still exist, as a last-ditch birthday gift for six-year-olds, for example. More than a half million Chia Pets are still sold each year.

More recently, health food enthusiasts have been touting the benefits of eating chia seeds and chia plants. Thus, eat your pets.

So what does this have to do with evidence-based reform in education? I could mention that more is probably spent on maintaining our essential chia reserves than on all research seeking technological breakthroughs to teach, say, algebra. But I would never stoop to making such a comparison.

Instead, the point I want to make is this. In education, people always want simple answers, when only slightly less simple answers are actually useful. If “eat your pets” were proclaimed in education, you’d immediately have a small faction in favor of eating pets and a really big one opposed, with almost no one saying “eat your Chia Pet, but not other pets.” Much too reasonable, and no one likes independent clauses in education pronouncements.

We see this insistence on super-simplification all the time. Is technology good or bad? Cooperative learning? Accountability? We have pretty good answers to each of these, and many more, but they each require an independent clause or two that explains “for whom” and “under what circumstances.”

Education will never make progress until we can educate the public to tolerate a little bit of explanation. Otherwise, we’ll be eating our pets until the cows come home.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin