What Makes Educational Technology Programs Work?

While everyone else is having a lot more fun, my colleagues and I sit up late at night writing a free website, the Best Evidence Encyclopedia (www.bestevidence.org), which reviews evaluations of educational programs in reading, math, and science.

The recent reports reinforce an observation I’ve made previously. When programs are found to have little or no impact on student learning, it is often the case that they provide very little professional development to teachers. Giving teachers lots of professional development does not guarantee positive effects, but failing to do so seems to virtually guarantee disappointing impacts.

This observation takes on new importance as technology comes to play an increasing role in educational innovation. Numerous high-quality studies of traditional computer-assisted instruction programs, in which students walk down the hall or to the back of the classroom to work on technology largely disconnected from teachers’ instruction, find few positive effects on learning. Many technology applications appearing in schools today have learned nothing from this sad history and are offering free or low-cost apps that students work on individually, with little professional development for teachers or even any connection to their (non-technology) lessons. In light of the prior research, it would be astonishing if these apps made any difference in student learning, no matter how appealing or well-designed they are.

Alongside the thousands of free apps going into schools, there has also developed an entirely different approach to technology, one that integrates technology with teacher lessons and provides teachers with extensive professional development and coaching. Studies of such programs do find significant positive effects. As one example, I recently saw an evaluation of a reading and math program called Time to Know. In Time to Know, teachers use computers and their own non-computer lessons to start a lesson. Students then do activities on their individual devices, personalized to their needs and learning histories. Student learning is continuously assessed and fed back to the teacher to use in informing further lessons and guiding interventions with individual students.

Time to Know provides teachers with significant professional development and coaching, so they can use it flexibly and effectively. Perhaps as a result, the program showed very good outcomes in a small but high-quality study, with an effect size of +0.32 in reading and +0.29 in math.

There are many other studies of classroom programs that improve student learning, in particular studies of forms of cooperative learning in many subjects and grade levels. As a group, the outcomes reported in these studies are always far higher than those seen in studies of traditional technology applications, in all subjects and grade levels. What is interesting about the study of Time to Know is that here is an unusually positive outcome for a technology application in a rigorous experiment. What is unique about the intervention is that it embeds technology in the classroom and provides teachers with extensive PD. Perhaps classroom-embedded technology with adequate professional development is the wave of the future, and perhaps it will finally achieve the long-awaited breakthroughs that technology has been promising for the past 40 years.

Show Me the Evidence

This year, evidence-based reform in education got off to a great start with an article by Ron Haskins in the New York Times on December 31 explaining why evidence of effectiveness must become an expected part of the process by which policy ideas are adopted (or not). More recently, I received a book Haskins wrote with colleague Greg Margolis on this topic, Show Me the Evidence. Both the article and the book mentioned our Success for All program as an example of what “proven” looks like in education, but they are a lot more important than that.

Haskins makes a powerful argument for putting all social programs to the test. Those that work should be expanded. Those that don’t should be replaced by other approaches that work better.

The need for evidence should be obvious, but very few federal programs have evidence of effectiveness. Few even have a process for finding out what works and encouraging grantees to use proven approaches, instead of approaches with the same desired outcomes that do not work or whose effects are unknown. Haskins estimates the 75 percent of programs and practices intended to help people do better at school or work have little or no impact. Such programs are well-meaning, but they need to be improved or replaced with equally well-meaning approaches known under well-defined circumstances to have positive impacts.

The importance of Haskins’ article and book lies especially in the importance of Haskins himself. He knows whereof he speaks. Advisor to House Republicans in the 1990s and then an advisor to President George W. Bush on social policy, he was a key architect of the 1996 welfare overhaul. Welfare programs that worked improved peoples’ lives and saved federal and state governments billions of dollars. Those that didn’t were replaced. To this day, those innovations represent the best example of evidence-driven policy.

Haskins is a proud Republican. He wants every dollar of federal expenditure to do what it is intended to do. Is there anyone, of any political persuasion, who does not want the same? This is not a question of ideology. It’s a question of sound governance.

When Ron Haskins and others were starting out, evidence was a pretty risky idea. Today, evidence is showing up throughout government — still not nearly as much as it should, but far more than it did. Sooner or later, government will become more competent and cost-effective at achieving goals we all share. Ron Haskins was there when it mattered. He still is, and it still does.

Seeds, Bricks, and Sand: Stages of School-Reform Readiness

Every school, no matter how effective at improving student outcomes, could probably be even more effective, and some schools have a particularly long way to go. Various proven reform models for whole schools, particular subjects, or specific purposes stand ready to help all of these schools improve. Yet schools vary a great deal in terms of readiness for particular approaches to reform.

A metaphor for three types of schools in terms of readiness for reform is seeds, bricks, sand. The “seeds” metaphor implies an environment so conducive to reform that anything can grow there. The staff and leadership of the school are capable, aware of research, participating in professional development, well-coordinated, cohesive, and unafraid of change. Such a school may be able to create and evaluate its own reform methods and sustain and improve them over time, perhaps with general advice from consultants. “Bricks” schools are also positively oriented toward change, but are unlikely to invent effective reforms themselves. Such schools have committed and hard-working teachers and leaders who have not had the time or resources to become reform experts themselves, but are welcoming to proven models. The “bricks” metaphor implies that if someone brings the bricks and a set of plans to the site, a durable edifice can be built and maintained.

A “sand” school, on the other hand, is one that is not ready for reform, and building on this site is like building a sand castle, which will wash away with the next tide. In such schools the staff and leadership may be at odds with each other, may not believe that children can learn any more than they do now, or may have experienced failure with previous reforms. These schools may need serious restructuring.

The usefulness of the “seeds-bricks-sand” categories is in understanding how to help schools adopt and sustain proven programs. The great majority of Title I schools, in my experience, are “bricks” schools, ready, willing, and able to implement well-defined, research-proven programs, but unlikely to have the inclination to invent their own school-wide approach. Others are clearly in the “sand” category. Yet Title I schools in trouble are frequently given “seeds” advice. Until now, schools receiving substantial funding under the current School Improvement Grants (SIG) have been routinely given consultants to help them work out their own school-wide reform designs, rather than being helped to adopt proven programs. There are schools that can benefit from such strategies, but they are rarely the ones that are persistently low achieving, as all SIG schools are. Recent proposed regulations would offer SIG schools the option of adopting proven, whole-school reform models, a welcome and long overdue change.

Use of proven, well-structured reforms needs to be expanded in all schools. For those in the greatest difficulty, this need is urgent. The new SIG regulations would allow schools capable of implementing (but not inventing) proven, effective reforms a chance to turn themselves around.

Whole-school reform is difficult and expensive, and when it fails, the consequences for children as well as educators can be dire and long-lasting. Failed initiatives not only waste money, but they undermine the belief that high-poverty schools can be saved. We need to get smarter about targeting interventions to specific types of schools to increase the likelihood that investments in reform truly pay off for kids and for our society.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

America’s Strength: An Innovation Economy

In a 2012 article in The New York Times called “China’s Rise Isn’t Our Demise,” Vice President Joe Biden wrote a cogent summary of America’s advantage in the world economy that has enormous implications for innovation in education.

“The United States is hard-wired for innovation. Competition is in the very fabric of our society. It has enabled each generation of Americans to give life to world-changing ideas – from the cotton gin to the airplane, the microchip, the Internet.

We owe our strength to our political and economic system and to the way we educate our children – not merely to accept established orthodoxy but to challenge and improve it… Our universities remain the ultimate destination for the world’s students and scholars.”

Nothing in Biden’s article was new or surprising. Every American understands that our success in the world economy depends on education and innovation.

So why do we devote so little attention to innovation in education? The very orientations and investments Vice President Biden cited as the basis of our success in other fields are rarely applied to improving education itself. Instead of inventing our way to success, as we do in so many other fields, we keep trying to improve education through changes in governance, regulations, and rules, which never produce change in core classroom practices and outcomes. Every state’s textbook adoption requirements specify paperweight, but never mention the weight of evidence behind the use of the book. Special education regulations specify that children be placed in the “least restrictive environment” but never the “most effective environment.” Title I has reams of regulations about how funds can or can’t be spent, but hardly a word suggesting that they be spent on programs proven to work.

The shelf of proven programs is steadily growing, due to investments at the Institute for Education Sciences, Investing in Innovation (i3), the National Science Foundation, and other government funders, as well as private foundation funders. Yet evidence and innovation continue to play an extremely small role in Title I, Title II, special education, and other federal programs, much less in state and local programs. A movement toward giving schools and districts more freedom in choosing how to use federal funding is a positive development, but local educators will need reliable information about proven, replicable programs to translate their new freedom into solid benefits for their children. Innovation based on research and development is what America does best. Isn’t it time to dedicate ourselves to innovating our way to solutions of our longstanding educational problems?

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin