Use Proven Programs or Manage Using Data? Two Approaches to Evidence-Based Reform


In last week’s blog, I welcomed the good work Results for America (RfA) is doing to promote policy support for evidence-based reform. In reality, RfA is promoting two quite different approaches to evidence-based reform, both equally valid but worth discussing separately.

The first evidence-based strategy is “use proven programs (UPP),” which is what most academics and reformers think of when they think of evidence-based reform. This strategy depends on creation, evaluation, and widespread dissemination of proven programs capable of, for example, teaching beginning reading, algebra, or biology better than current methods do. If you see “programs” as being like the individual baseball players that Oakland A’s manager Billy Beane chose based on statistics, then RfA’s “Moneyball” campaign could be seen as consistent in part with the “use proven programs” approach.

The second strategy might be called “manage using data (MUD).” The idea is for leaders of complex organizations, such as mayors or superintendents, to use data systematically to identify and understand problems and then to test out the solutions, expanding solutions that work and scaling back or abandoning others. This is the approach used in “Geek Cities“ celebrated by RfA.

“Use proven programs” and “manage using data” have many similarities, of course. Both emphasize hard-headed, sophisticated use of data. Advocates of both approaches would be comfortable with the adage, “In God we trust. All others bring data.”

However, there are key differences between the UPP and MUD approaches that have important consequences for policy and practice. UPP emphasizes the creation of relatively universal solutions to widespread problems. In this, they draw from a deep well of experience in medicine, agriculture, technology, and other fields. When an innovator develops a new heart valve, a cow that produces more milk, or a new cell phone, and proves that it produces better outcomes than current solutions, that solution can be used with confidence in a broad range of circumstances, and may have immediate and profound impacts on practice.

In contrast, when a given school district succeeds with the MUD approach (for example, analyzing where school violence is concentrated, placing additional security guards in those areas, and then noting the changes in violence), this success is likely to be valued and acted upon by district leaders, because the data come from a context they understand and are collected by people they employ and trust. However, the success is unlikely to spread to or be easily replicated by other school districts. The MUD district may tout its success, but district leaders are not particularly motivated to tell outsiders about their successes, and usually lack sufficient staff to even write them up. Further, since MUD approaches are not designed for replication, they may or may not work in other places with different contexts. The difficulty of replicating success in a different context also applies to UPP strategies, but after several program evaluations in different contexts, program developers are likely to be able to say where their approach is most and least likely to work.

From a policy perspective, MUD and UPP approaches can and should work together. A district, city, or state that proactively uses data to analyze all aspects of its own functioning and to test out its own innovations or variations in services should also be eager to adopt programs proven to be effective elsewhere, perhaps doing their own evaluations and/or adaptations to local circumstances. If the bottom line is what’s best for children, for example, then a mix of solutions “made and proven here” and those “made and proven elsewhere and replicated or tested here” seems optimal.

For federal policy, however, the two approaches lead in somewhat different directions. The federal government cannot do a great deal to encourage local governments to use their own data wisely. In areas such as education, federal and state governments use accountability schemes of various kinds as a means of motivating districts to use data-driven management, but looking at NAEP scores since accountability took off in the early 1980s, this strategy is not going very well. The federal government could identify well-defined, proven, and replicable “manage using data” methods, but if it did, those MUD models would just become a special case of “use proven programs” (and in fact, the great majority of education programs proven to work use data-driven management in some form as part of their approach).

In contrast, the federal government can do a great deal to promote “use proven programs.” In education, of course, it is doing so with Investing in Innovation (i3), the What Works Clearinghouse (WWC), and most of the research at the Institute of Education Sciences (IES). All of these are building up the number of proven programs ready to be used broadly, and some are helping programs to start or accelerate scale-up. The existence of the proven programs coming from these and other sources creates enormous potential, but is not yet having much impact on federal policies relating, for example, to Title I or School Improvement Grants, but this could be coming.

Ultimately, “use proven programs” and “manage using data” should become a seamless whole, using every tool of policy and practice to see that children are succeeding in school, whatever that takes. But the federal government is wisely taking the lead in building up capacity for replicable “use proven programs” strategies to provide new visions and practical guides to help schools improve themselves.

Results for America: A Welcome Addition to Evidence-Based Reform


Advocating for evidence-based reform in education is often a lonely activity, but over the past couple of years, it’s gotten a lot less lonely due to the arrival of Results for America(RfA), an initiative dedicated to encouraging government at all levels to make more and better use of evidence to make important decisions. RfA is not limited to education, but covers all areas of human services for children and families. RfA is led by Michele Jolin and is part of a broader organization called America Achieves, led by Jon Schnur. Both Jon and Michele have had extensive experience working in government.One constant meme adopted by RfA is “Moneyball for Government,” based on the movie “Moneyball” about legendary Oakland A’s manager Billy Beane. Constrained by limited resources to improve the team, unlike the deep pockets of the Yankees who were able to offer high-salaried contracts to the best players, Beane engaged a statistician to help him find less desirable or lower-cost players who did not fit all his scouts’ requirements but somehow managed to get on base a lot.

As applied to government, Moneyball means using data to identify programs and practices that have been proven to produce valued outcomes. In education, for example, a leader who played Moneyball would actively seek reading, math, science, and other programs that had been rigorously evaluated and found to be effective. A Moneyball approach would also include using data to identify programs or approaches already in place within the district and expand the ones that are working. RfA has even identified “Geek Cities,” where the local government does a particularly good job of using data to make effective and efficient use of resources to serve children and families.

RfA is strictly and wisely nonpartisan, arguing that everyone, liberal or conservative, has an interest in seeing that resources are used sensibly and well. As time goes on, Results for America is making significant inroads in the policy world, helping political leaders of all stripes get comfortable with the idea of heightening the role of evidence in policymaking.