Lighthouses

Visiting coastal lighthouses is one of the highlights of any beach vacation. Mostly relics of the 18th and 19th century, lighthouses played a vital role in helping harbor pilots and ship captains find channels and avoid hazards such as rocks and sandbars. They saved many ships and crews that would otherwise have been lost at night or in storms or fog.

In education, the idea of “lighthouse schools” comes up frequently. The notion here is to identify schools that are making impressive progress, especially in challenging circumstances, and then publicize their successes. The metaphor, of course, is that these “lighthouses” guide others to reach their goals.

Taken as a form of journalism, rather than a form of science, a policy of identifying and publicizing lighthouse schools in high-poverty neighborhoods makes a statement that success is actually possible. With stories of tough but loving principals, caring and hard-working staff and ringing praise from parents and politicians, lighthouse stories are irresistible feel-good press.

However, if what we want is evidence-based reform, identifying and describing lighthouse schools is less useful. The process is riddled with problems. First, you have to make sure that the lighthouse school is truly a shining example. In the 1980s, everyone was abuzz about District 2 in New York City, which was rapidly rising in academic performance using a variety of innovative methods and visionary leadership. I asked my New York friends at the time what was going on in District 2, and they fell on the floor laughing. It turned out that District 2 was quickly gentrifying. A former high-poverty area was now attracting many upper-middle class families eager for quality schools that did not charge tuition. Lighthouse schools often have such explanations.

Lighthouse schools are sometimes identified based on a single outstanding year, which may be in the past. This may mean that a temporarily-outstanding school is now rather ordinary.

Even if a given school has made major, sustained gains serving the same population it always served, it is rarely clear what caused the change. Was it the principal? New teachers? Additional resources? Innovative programs? In a single school, it is impossible to pick out what made the difference because everything is intertwined with everything else. A journalistic approach simply cannot account for this.

Further, even if we have strong suspicions about what makes the lighthouse shine, that factor may not be replicable. What if it is the principal? Another school might hire away that principal, but that hardly moves the system forward. What if a local foundation gave the school a pile of money? What if the school managed to attract volunteer tutors from across the city? Such advantages are possible in some schools or even districts, but not in others.

The whole lighthouse school idea is undercut by the observation that, whatever the “secret sauce” might be, it does not travel well. If it did, we would find not only lighthouse schools but lighthouse districts and states, in which great ideas spread outward using replicable methods. The very fact that one school stands out from its neighbors should give us pause about whether the neighbors have the capacity and the willingness to imitate success.

Lighthouse schools can certainly contribute ideas or inspiration to evidence-based reform, but before a program can be considered effective and replicable, it needs to be clearly defined and then evaluated by rigorous methods. Such evaluation would normally compare at least 20 schools assigned at random to implement the program to 20 similar schools that continue with their practice as usual. If such a study finds that the schools using the innovative program did better, then we really know something worthwhile. In this scenario, we could have confidence because factors other than the innovation balance out. The experimental and control schools are likely to have equal numbers of good principals, equal funding (on average), equally qualified teachers (on average) and so on. The only difference between the experimental and control schools is the “secret sauce” itself, which, if it can work in 20 schools or more, is probably replicable.

Lighthouses once guided ships to safe harbors, but in education, policies limited to finding and celebrating lighthouse schools are less likely to improve outcomes more broadly. They may lead policy in a good direction, but they may just as likely guide us onto the rocks.

The Evidence or The Morgue

Many years ago, when I was a special education teacher, I had a summer job at a residential school for emotionally disturbed children. The school happened to be located in a former tuberculosis sanitarium. Later, I heard from other teachers elsewhere about having worked in schools in one-time sanitaria as well.

How did it come about, one might ask, that tuberculosis sanitaria across the country became available for use as schools? The answer is that researchers cured the disease. The sanitaria were no longer needed for their original purpose, so they were turned into schools.

One feature of the former sanitaria is that they all had morgues. We used ours to store curriculum materials, because it had very sturdy and useful sliding horizontal cabinets. This arrangement led to a certain amount of macabre humor, but the morgue reminded us that what the sanitaria had once done was deadly serious indeed.

I was recalling my summer in the sanitarium after reading about the latest developments in the reauthorization of ESEA. Both the House and the Senate have now passed bills that eliminate the Investing in Innovation (i3) program and cut funding for the Institute of Education Sciences. In their place, the bills have a lot of language about state and local control, and about identifying and publicizing individual schools that are doing a particularly good job so their good works can help inspire and influence other schools. None of this would bother me if the legislation contained a clear commitment to rigorous research, development, and dissemination, but this may or may not be the case.

The Senate bill, which passed with bipartisan support last week, does authorize an evidence-based innovation fund. Modeled on the successful Small Business Innovation Research (SBIR) program, which funds innovation and evaluation in 11 different government agencies, this initiative would provide flexible funding for a broad range of field-driven projects and allow schools, districts, non-profits, and small businesses to develop and grow innovative programs to improve student achievement. Grants would be awarded using a tiered evidence framework based on an applicant’s proven effectiveness. The provision was initially offered and accepted as a bipartisan amendment during the Senate HELP Committee markup of its bill. However, the House bill has no comparable provision, and I have to wonder if the Senate provision will survive the grueling conference process and make it into the final bill.

Try to imagine what would have happened if tuberculosis research had been treated the way education research has been treated in the House version of the ESEA reauthorization bill. Individual sanitaria with lower death rates might be recognized. States and localities might try out ideas to make the sanitaria more effective, but few if any states or localities would be large enough to do the necessary sustained R & D. “Best practices” would be constrained by the current system, so they might involve better ways for sanitarium staff to do exercises with patients, for example, rather than experimenting with medications or other treatments. The disease would never have been cured. The morgues would still be used for unfortunate patients, not for curriculum materials.

The U.S. spends hundreds of billions of dollars every year on education. What student, parent, teacher, administrator, or policy maker does not want those billions used to make as much of a difference as possible? The pursuit of knowledge about how to improve educational outcomes is obviously important, but it is rarely very high on anyone’s priority list.

Fortunately, medicine and other fields long ago decided that research was in the national interest, and that investments in research were the most reliable way forward in improving important outcomes. In medicine, the choice is stark: either the evidence prevails or the morgue does. Yet in education, anyone with eyes to see knows what happens when children fail to learn. Most of the children who cannot read end up unemployed. Many end up in prison, and all too many in the morgue. We know enough now to be able to say that the great majority of reading failure, for example, is preventable. Yet we choose not to prevent it. What does this say about us as a people, as a society, as a political system?

I hope our leaders in Congress approve the Senate language on evidence, or something similar, and reinstate and fund programs that have the greatest promise in identifying and disseminating effective approaches to key problems. The lives of a generation of vulnerable children depend on their wisdom and courage at this critical juncture.

To Pluto and Beyond

Like many others, I was thrilled to see The New Horizons spacecraft reach and photograph Pluto. After being banished from the League of Planets shortly after New Horizons was launched, I’ll bet Pluto felt much better with all the attention.

Those who read this blog are probably expecting me to go into a rant at this point about how much we are willing to spend to send a spacecraft to take pictures and how little we are willing to spend on finding out how to help our nation’s children learn to read the newspaper or understand the math or the space science around this marvelous event. Well, consider it ranted. It does not make me feel any better that funding for NASA itself is being cut. We are a hugely wealthy country, and we can afford to go to Pluto and to educate our children to a much higher standard than we do. In fact, the way we became a hugely wealthy country, and the only way we can maintain our wealth into the future, is by investing in education, science, technology and invention.

My colleagues and I recently completed reviews of research on elementary and then secondary science education. You can find them here. The reviews find very similar outcomes at the different grade levels. Instructional methods emphasizing professional development for teachers on well-defined teaching strategies, such as cooperative learning and science-reading integration, have solid effects on science learning outcomes. Moving from one textbook to another almost never makes a difference, and use of science kits does not improve science learning. Technology-focused programs have a great deal of promise, but the studies are few and of limited quality, at least so far.

However, the most depressing finding is that there were far too few studies, across all science teaching approaches, that met even modest standards of rigor. Using our standards (which just require a control group, initial equality, fair measures, and a duration of 12 weeks), there were just 21 secondary studies in the past quarter-century. The number was the same for elementary studies. This is shameful. Science teaching is widely acknowledged to be a key to our nation’s future, yet our investment in high-quality studies and innovation is so low that we really know very little about how to do it better.

To explore the universe, to cure diseases, to engineer new solutions of all kinds, requires a population that is proficient in science, technology and mathematics. Is there anyone on the (still recognized) planet Earth who does not know this? Yet if we were serious about going boldly where no nation has gone before, would we continue to invest so little in understanding how to engage and excite our students in science, math, and technology?

Today, we rely on an extraordinary but tiny elite for the scientific progress we do make. We need to extend far beyond this, as more and more occupations come to require deep understanding of science and math. We need to enable teachers in elementary and secondary schools to democratize science knowledge and skill. There is no question that we can design better teaching methods and technologies, evaluate them, and scale them up. I wonder when we will get serious about doing so?

Congratulations to NASA, the Johns Hopkins Applied Physics Lab, and the American taxpayer for the New Horizons trip to Pluto. But consider this. The next generation of scientists and engineers who will perform the marvels of the future are in elementary and secondary classes right now. Improving science learning for these precious future scientists and engineers is essential for our nation’s future.

Good Failure/Bad Failure

Evidence junkies (like me) are reacting to the disappointing news on the evaluation of the Adolescent Behavioral Learning Experience (ABLE), a program implemented at Rikers Island to reduce recidivism among adolescent prisoners. Bottom line: The rigorous independent evaluation of the program failed to find any benefits. What makes this experiment especially interesting is that it is the first U.S. application of social impact bonds. Goldman Sachs put up a $7.2 million loan, and Bloomberg Philanthropies committed to a $6 million loan guarantee. Since the program did not produce the expected outcomes, Goldman Sachs lost $1.2 million.

Ironically, New York City administrators are delighted about the outcome because they do not have to pay for the program. They think they learned a great deal from the experience, for free.

It’s unclear what this will do to the social impact bond movement, currently in its infancy. However, I wanted to extend from this fascinating case to a broader issue in evidence-based reform.

The developers and advocates for the ABLE program who expected positive outcomes turned out to be wrong, at least in this implementation. The investors were wrong in expecting to make a profit. But I’d argue that they are all better off because of this experience, just as the N.Y.C. administrators said.

The distinction I want to make is between wrong and wrong-headed. Wrong, as I’m defining it in this context, means that a given outcome was not achieved, but it was entirely reasonable to expect that it might have been achieved. In contrast, wrong-headed means that not only was the desired outcome not achieved, but it was extremely unlikely that it could have been achieved. In many cases, a key component of wrong-headed actions is that the actor does not even know whether the action was effective or ineffective, right or wrong, and therefore continues with the same or similar actions indefinitely.

Wrong, I’d argue, is an honorable and useful outcome. In a recent interview, former White House advisor Gene Sperling noted that when a few cancer drugs fail to cure cancer, you don’t close down NIH. Instead, you take that information and use it to continue the research and development process. “Wrong,” in this view, can be defined as “good failure,” because it is a step on the path to progress.

“Wrong-headed,” on the other hand, is “bad failure.” When you do something wrong-headed, you learn nothing, or you learn the wrong lessons. Wrong-headed decisions tend to lead to more wrong-headed decisions, as you have no systematic guide to what is working and what is not.

The issue of wrong vs. wrong-headed comes up in the current discussions in Congress about continuing the Investing in Innovation (i3) program. By now, committees in both the House and the Senate have recommended ending i3. But this would be the very essence of wrong-headed policy. Sure, it is probable that many i3 programs funded so far will fail to make a difference in achievement, or will fail to go to scale. This just means that these programs have not yet found success. Some of these may still have evidence of promise, and some will not. However, all i3 programs are rigorously evaluated, so we will know a lot about which worked, which did not, and which still seem promising even if they did not work this time. That’s huge progress. The programs that are already showing success can have immediate impact in hundreds or thousands of schools while others greatly enrich understanding of what needs to be done.

Abandoning i3, in contrast, would be wrong-headed, a sure path to bad failure. A tiny slice of education funding, i3 tells us what works and what does not, so we can continually move towards effective strategies and policies. Without i3 and other research and development investments, education policy is just guesswork, and it gets no smarter over time.

No one can honestly argue that American education is as successful as it should be. Our kids, our economy, and our society deserve much better. Policies that seek a mixture of proven success and “good failure” will get us to solid advances in educational practice and policy. Abandoning or cutting programs like i3 is not just wrong. It’s wrong-headed.

Innovation Nation

As we celebrate our nation’s birthday, I thought I’d once again remind those few of you who, instead of having picnics, are reading blogs, how the U.S. came to be, and how innovation has been baked into its DNA since Day 1.

First, our nation began as a huge experiment. Can a country truly govern itself? Few had, and those examples (notably Athens) were small and long ago. Could a sprawling nation elect its own leaders, write its own laws, solve its own problems? It was an extraordinary experiment, with the whole world (especially Europe) as a control group. Notably, two foreign veterans of the U.S. experiment, the Marquis de la Fayette and Tadeusz Kosciuszko, went back to their home countries (France and Poland) to try to establish similar forms of government. Both failed miserably, as did most other countries throughout the 19th Century and into the 20th.

Why did the American experiment succeed? There are many reasons, but I’ll put forward my own. It cannot be irrelevant that the leaders of the American Revolution were innovators, and not just in government. Benjamin Franklin, with his stove, bifocals, lightning rod, and other inventions, comes immediately to mind. Immediately after the U.S. Constitution was ratified, his first order of business was to establish a patent office, to encourage and protect innovators in every field.

Thomas Jefferson was also an innovator and scientist, of course. If you visit his home in Monticello, you can’t help but wonder at his innovative mind. If you visit Mount Vernon, you learn that George Washington, in between rescuing and then leading the country, was also an agricultural innovator.

Our founders disagreed on a great deal, and their politics was as ferocious and divided as our politics today. Yet they deeply agreed on the importance of innovation and evidence in improving the agricultural and commercial success of a new nation.

What would they think about evidence-based reform in education? All were passionate advocates of education; Jefferson, Franklin, and Washington all helped to found great universities, as did many others. As early as 1862, the Homestead Act set aside funding in new territories for schools and land-grant universities. Clearly, our founders valued education and they valued innovation and evidence.

What would our founders say if they came back today and found that our far larger, richer, and more stable country was spending little of its wealth on educational research and innovation, and that the Congress they founded was proposing to substantially cut the little that is spent?

Have a great holiday. When everyone gets back to work, I hope we’ll all redouble our efforts to make the education we provide our students second to none. It’s our patriotic duty.