Visiting coastal lighthouses is one of the highlights of any beach vacation. Mostly relics of the 18th and 19th century, lighthouses played a vital role in helping harbor pilots and ship captains find channels and avoid hazards such as rocks and sandbars. They saved many ships and crews that would otherwise have been lost at night or in storms or fog.

In education, the idea of “lighthouse schools” comes up frequently. The notion here is to identify schools that are making impressive progress, especially in challenging circumstances, and then publicize their successes. The metaphor, of course, is that these “lighthouses” guide others to reach their goals.

Taken as a form of journalism, rather than a form of science, a policy of identifying and publicizing lighthouse schools in high-poverty neighborhoods makes a statement that success is actually possible. With stories of tough but loving principals, caring and hard-working staff and ringing praise from parents and politicians, lighthouse stories are irresistible feel-good press.

However, if what we want is evidence-based reform, identifying and describing lighthouse schools is less useful. The process is riddled with problems. First, you have to make sure that the lighthouse school is truly a shining example. In the 1980s, everyone was abuzz about District 2 in New York City, which was rapidly rising in academic performance using a variety of innovative methods and visionary leadership. I asked my New York friends at the time what was going on in District 2, and they fell on the floor laughing. It turned out that District 2 was quickly gentrifying. A former high-poverty area was now attracting many upper-middle class families eager for quality schools that did not charge tuition. Lighthouse schools often have such explanations.

Lighthouse schools are sometimes identified based on a single outstanding year, which may be in the past. This may mean that a temporarily-outstanding school is now rather ordinary.

Even if a given school has made major, sustained gains serving the same population it always served, it is rarely clear what caused the change. Was it the principal? New teachers? Additional resources? Innovative programs? In a single school, it is impossible to pick out what made the difference because everything is intertwined with everything else. A journalistic approach simply cannot account for this.

Further, even if we have strong suspicions about what makes the lighthouse shine, that factor may not be replicable. What if it is the principal? Another school might hire away that principal, but that hardly moves the system forward. What if a local foundation gave the school a pile of money? What if the school managed to attract volunteer tutors from across the city? Such advantages are possible in some schools or even districts, but not in others.

The whole lighthouse school idea is undercut by the observation that, whatever the “secret sauce” might be, it does not travel well. If it did, we would find not only lighthouse schools but lighthouse districts and states, in which great ideas spread outward using replicable methods. The very fact that one school stands out from its neighbors should give us pause about whether the neighbors have the capacity and the willingness to imitate success.

Lighthouse schools can certainly contribute ideas or inspiration to evidence-based reform, but before a program can be considered effective and replicable, it needs to be clearly defined and then evaluated by rigorous methods. Such evaluation would normally compare at least 20 schools assigned at random to implement the program to 20 similar schools that continue with their practice as usual. If such a study finds that the schools using the innovative program did better, then we really know something worthwhile. In this scenario, we could have confidence because factors other than the innovation balance out. The experimental and control schools are likely to have equal numbers of good principals, equal funding (on average), equally qualified teachers (on average) and so on. The only difference between the experimental and control schools is the “secret sauce” itself, which, if it can work in 20 schools or more, is probably replicable.

Lighthouses once guided ships to safe harbors, but in education, policies limited to finding and celebrating lighthouse schools are less likely to improve outcomes more broadly. They may lead policy in a good direction, but they may just as likely guide us onto the rocks.