I was recently in Oslo, Norway’s capital, and visited a wonderful museum displaying three Viking ships that had been buried with important people. The museum had all sorts of displays focused on the amazing exploits of Viking ships, always including the Viking landings in Newfoundland, about 500 years before Columbus. Since the 1960s, most people have known that Vikings, not Columbus, were the first Europeans to land in America. So why do we celebrate Columbus Day, not Viking Day?
Given the bloodthirsty actions of Columbus, easily rivaling those of the Vikings, we surely don’t prefer one to the other based on their charming personalities. Instead, we celebrate Columbus Day because what Columbus did was far more important. The Vikings knew how to get back to Newfoundland, but they were secretive about it. Columbus was eager to publicize and repeat his discovery. It was this focus on replication that opened the door to regular exchanges. The Vikings brought back salted cod. Columbus brought back a new world.
In educational research, academics often imagine that if they establish new theories or demonstrate new methods on a small scale, and then publish their results in reputable journals, their job is done. Call this the Viking model: they got what they wanted (promotions or salt cod), and who cares if ordinary people found out about it? Even if the Vikings had published their findings in the Viking Journal of Exploration, this would have had roughly the same effect as educational researchers publishing in their own research journals.
Columbus, in contrast, told everyone about his voyages, and very publicly repeated and extended them. His brutal leadership ended with him being sent back to Spain in chains, but his discoveries had resounding impacts that long outlived him.
Educational researchers only want to do good, but they are unlikely to have any impact at all unless they can make their ideas useful to educators. Many educational researchers would love to make their ideas into replicable programs, evaluate these programs in schools, and if they are found to be effective, disseminate them broadly. However, resources for the early stages of development and research are scarce. Yes, the Institute of Education Sciences (IES) and Education Innovation Research (EIR) fund a lot of development projects, and Small Business Innovation Research (SBIR) provides small grants for this purpose to for-profit companies. Yet these funders support only a tiny proportion of the proposals they receive. In England, the Education Endowment Foundation (EEF) spends a lot on randomized evaluations of promising programs, but very little on development or early-stage research. Innovations that are funded by government or other funding very rarely end up being evaluated in large experiments, fewer still are found to be effective, and vanishingly few eventually enter widespread use. The exceptions are generally programs crated by large for-profit companies, large and entrepreneurial non-profits, or other entities with proven capacity to develop, evaluate, support, and disseminate programs at scale. Even the most brilliant developers and researchers rarely have the interest, time, capital, business expertise, or infrastructure to nurture effective programs through all the steps necessary to bring a practical and effective program to market. As a result, most educational products introduced at scale to schools come from commercial publishers or software companies, who have the capital and expertise to create and disseminate educational programs, but serve a market that primarily wants attractive, inexpensive, easy-to-use materials, software, and professional development, and is not (yet) willing to pay for programs proven to be effective. I discussed this problem in a recent blog on technology, but the same dynamics apply to all innovations, tech and non-tech alike.
How Government Can Promote Proven, Replicable Programs
There is an old saying that Columbus personified the spirit of research. He didn’t know where he was going, he didn’t know where he was when he got there, and he did it all on government funding. The relevant part of this is the government funding. In Columbus’ time, only royalty could afford to support his voyage, and his grant from Queen Isabella was essential to his success. Yet Isabella was not interested in pure research. She was hoping that Columbus might open rich trade routes to the (east) Indies or China, or might find gold or silver, or might acquire valuable new lands for the crown (all of these things did eventually happen). Educational research, development, and dissemination face a similar situation. Because education is virtually a government monopoly, only government is capable of sustained, sizable funding of research, development, and dissemination, and only the U.S. government has the acknowledged responsibility to improve outcomes for the 50 million American children ages 4-18 in its care. So what can government do to accelerate the research-development-dissemination process?
- Contract with “seed bed” organizations capable of identifying and supporting innovators with ideas likely to make a difference in student learning. These organizations might be rewarded, in part, based on the number of proven programs they are able to help create, support, and (if effective) ultimately disseminate.
- Contract with independent third-party evaluators capable of doing rigorous evaluations of promising programs. These organizations would evaluate promising programs from any source, not just from seed bed companies, as they do now in IES, EIR, and EEF grants.
- Provide funding for innovators with demonstrated capacity to create programs likely to be effective and funding to disseminate them if they are proven effective. Developers may also contract with “seed bed” organizations to help program developers succeed with development and dissemination.
- Provide information and incentive funding to schools to encourage them to adopt proven programs, as described in a recent blog on technology. Incentives should be available on a competitive basis to a broad set of schools, such as all Title I schools, to engage many schools in adoption of proven programs.
Evidence-based reform in education has made considerable progress in the past 15 years, both in finding positive examples that are in use today and in finding out what is not likely to make substantial differences. It is time for this movement to go beyond its early achievements to enter a new phase of professionalism, in which collaborations among developers, researchers, and disseminators can sustain a much faster and more reliable process of research, development, and dissemination. It’s time to move beyond the Viking stage of exploration to embrace the good parts of the collaboration between Columbus and Queen Isabella that made a substantial and lasting change in the whole world.
This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.
Robert Slavin's Blog 
Great blog.
I’ve got a foot in both camps now. My first 16 years were RCT-driven interventions and my heart remains there. The last few years, though, I’ve been in and around the for-profit camp.
For-profit has 3 skill sets that non-profits typically don’t: sales, marketing (to them, efficacy research is perhaps 0% to 20% of a marketing plan), and “product-izing” (making the intervention easier to use, more likeable to the end user).
For-profits spend most of their money on these 3 things. Non-profit leaders just can’t bear to do so. “We” want to create valuable things and reluctantly spend a tiny bit on “dissemination. (We don’t even like the word “sales.”)
Your gov’t ideas are good ones, but I wonder if some version of “commercializing the evidence proven programs” — a handoff — needs to be part of the vision.
LikeLike
Thanks for your note. I did not in any way mean to imply that commercial companies should not be part of the solution. They indeed have capabilities essential for bringing proven programs to market at scale. My argument, however, was that real change toward evidence-based reform requires a market eager to use proven programs, and that only government is likely to be able to create such markets. Once the demand is there, capable organizations, for-profit as well as non-profit, will be needed to provide proven programs in a form that maintains their effectiveness.
LikeLike
I agree with Mike G that a lot of great education methods go unshared. But I think Prof Slavin was right the first time and shouldn’t back down. For-Profit and Education don’t mix. (Though a B-Corp dedicated to more than profit in it’s by-laws might be an exception.)
“research is perhaps 0% to 20% of a marketing plan”
This sentence sums up the problem with for-profit education companies. There’s no point in having a marketing plan unless you have a methodology/curriculum that is classroom-proven to produce a significant effect-size. And even when you have a promising method it will take years of patience and integrity to refine it to the point where any teacher or school in the country can benefit from it with only minimal training. This is not the same process as commercializing a product. The timelines are different, the measures of success are different.
The argument that commercial companies can be part of the solution also assumes that Ed Tech companies care about selling effective education tools. But it’s pretty clear to me from watching them for decades that they don’t want to be judged by their results. They are much happier relying on access and marketing; and the appearance of effectiveness is enough if that is your intention: hire a Rent-a-PhD as your “Research Director” to publish worthless “White Papers”; reverse engineer your content or features from the latest standards (which you hopefully helped rewrite for a large fee) or a research “guru,” then trumpet kids “progress” moving through your internal curriculum, and put your money on meeting superintendents and other decision makers who can force your dubious curriculum or ed tech down the throats of unwilling principals and classroom educators.) Dishonesty is their achilles heel.
I don’t know how how else you explain the long long history of terrible for-profit math curriculums.
On the other hand, a short review of innovative and effective math curriculums or apps shows that only non-profits, people with other jobs (usually academics), the occasional small developer with classroom teaching experience or just a crazy hunch, are likely to discover effective approaches in the first place, go the distance to develop and refine them, keep the classroom connection, and make truly effective curriculums.
A few examples: ST Math (academic at UC Irvine, took years to refine the original method (first it was piano lab + math!) and put together an effective team)—last time I looked at their 990 they had something like $20million in revenue and $25million in expenses—they raised the rest through donations); Saxon Math (a frustrated classroom teacher who had to mortgage his house to grow the company); and DuoLingo, not math but… an example of the education renaissance made possible by the App store revolution (computer science professor, admittedly didn’t know much about language instruction, used A/B testing with goal of getting people to actually use his program. It took years and years of experimentation to arrive at the highly effective addictive program they have now).
But I agree that a lot of great education methods go unshared. A truly effective teacher is handicapped by (a) their unwillingness to “experiment on children” (a control = depriving kids of the curriculum you’re pretty damn sure works best—which is unethical) and (b) their unwillingness to stop improving the curriculum/solution alone long enough to test it for 3 years, the minimum for a rigrorous effect size study!
Perhaps a mythical Benefit Corp dedicated to something besides marketshare and profits could be successful in identifying and developing the best methods to the point where other other teachers and schools could share them.
—–
LikeLike