Theory Is Not Enough

“The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge.” — Stephen Hawking

Readers of this blog are aware that I am enthusiastic about the EDGAR definitions of “strong” and “moderate” evidence of effectiveness for educational programs. However, EDGAR also has two categories below “moderate.” These are called “promising” and “based on strong theory.” I never mention these, but I thought it’s time to do so, because they come up in current policy debates.

Advocates for evidence in education want federal policies to seek to increase the use of proven programs, and I have frequently written in support of encouraging schools to use these programs when they exist and building them when they don’t. But some would expand the definition of “evidence-based” to make it too broad. In particular, in areas in which very few programs meet at least the “moderate” level of EDGAR, they want to allow “promising” or “based on strong theory” as alternatives. “Promising” means that a program has correlational evidence supporting it. With the right definition, this may not be so bad. However, “based on strong theory” just means that there is some explanation for why the program might work.

In a moment I’ll share my thoughts about why encouraging use of programs that only have “strong theory” is a bad idea, but first I’d like to tell you a story about why this discussion is far from theoretical for me personally.

When I was in college, I had a summer job at a school for children with intellectual disabilities in Washington, DC. Another college sophomore and I were in charge of the lowest-performing class. We overlapped by a week with the outgoing teacher, who showed us in detail what she did with the children every day. This consisted of two activities. One involved getting children to close their eyes and smell small jars of substances, such as garlic, cinnamon, and cloves, and say what they were. The other involved having children string beads in alternating patterns. These activities were directly from a theory called Psychomotor Patterning, or Doman-Delcato, which was extremely popular in the 1960s and early 1970s. In addition to sensory stimulation, advocates of Doman-Delcato had children with intellectual disabilities crawl and do other stylized exercises on the theory that these children had skipped developmental steps and could catch up to their peers by going back and repeating those steps.

In our school, my partner and I started off dutifully continuing what the teacher had shown us, but after a few days we looked at each other and said, “This is stupid.” We knew that our kids, aged perhaps 11-15, had two potential futures. If they were lucky, they could stay at home and perhaps get a job in a sheltered workshop. Otherwise, they were likely to end up in the state hospital, a terrible fate. We decided to drop the patterning, and teach our kids to tie their shoes, to sweep, to take care of themselves. We began to take them on walks and to a local McDonalds to teach them how to behave in public.

One of our children was a sweet, beautiful girl named Sarah, about 12 years old. Sarah was extremely proud of her long, blond hair, which she would stroke and say, “Sarah’s so pretty,” which I’m sure she’d heard countless times.

I was working especially hard with Sarah, and she learned quickly. I taught her to sweep, for example, starting with balled-up paper and moving to smaller and smaller things to sweep up.

One day, Sarah was gone. We heard that her parents had taken her to the state hospital.

For some reason, the parents brought Sarah back for a visit about a month later. Her beautiful hair was gone, as was the sparkle that had once been in her eyes. She stared at the floor.

A few years later, in another school, I saw teachers working with teenagers with Down Syndrome, having them crawl around the classroom every day. Like Sarah, these kids had two potential futures. This school had a sheltered workshop housed in it, and if they could qualify to work there, their futures were bright. Instead, they were wasting their best chances crawling like babies.

“Based on strong theory” may sound academic or policy-wonky to many, but to me, it means that it is okay to subject children to treatments with no conclusive evidence of effectiveness when better treatments exist or could exist. In particular, “based on strong theory” all too often just means “what’s fashionable right now.” Doman-Delcato Psychomotor Patterning was a hugely popular fad because it gave parents and teachers hope that intellectual disabilities could be reversible. When I was a special education teacher, “based on strong theory” meant that my kids had received years of useless sensory stimulation instead of learning anything potentially useful. Perhaps Sarah was going to end up in the state hospital no matter what my school did. But I cannot set aside my memory of her when I hear people say that “strong theory” might be enough when actual evidence is lacking.

From a policy perspective, it would be useful to have federal and state funding support programs with strong or moderate evidence of effectiveness. In areas lacking such evidence-proven programs, government might fund research and development, perhaps, but should not encourage use of programs that are only supported by “strong theory.” Allowing weak categories into the discussion waters down the entire idea of evidence-based reform, as all programs could probably meet such definitions. Even worse, encouraging use of programs based on strong theory could lead schools to use the current fad. And if you doubt this, ask Dr. Doman and Dr. Delcato.

Leveraging What Works

In my blog from two weeks ago, I discussed several exciting proposals in President Obama’s recent budget relating to increasing the role of evidence in education policy and practice. Today, I want to say more about one of these proposals, Leveraging What Works (LWW).

Leveraging What Works is deceptively simple. It offers grants totaling $100 million nationwide to school districts willing to use the grant, along with a portion of its formula funds — such as Title I and IDEA — to adopt proven programs that meet the “strong” or “moderate” level of evidence of effectiveness as defined in EDGAR.

Simple though it appears, Leveraging What Works would be revolutionary. Here’s why.

First, the program would generate a huge amount of interest. Winning LWW funding would be sought after avidly not only for the money itself but as a feather in the cap of innovative thought-leader districts. These districts will be eager to win the money and tell their stories. The whole process will create a positive “buzz” around the use of proven programs.

Because of the money and the positive buzz, many more districts will apply for LWW funding than can be funded. Yet having looked at the range of proven programs available to them, many of these districts will choose to adopt proven programs using their formula funding even without the LWW grant. This is exactly what happened with the Obey-Porter Comprehensive School Reform Demonstration Act (CSR) of the late 1990’s. Thousands of schools applied for modest grants to help them adopt whole-school models, and each year, hundreds of schools that were turned down for grant funding adopted CSR models anyway, using other funding.

Leveraging What Works could revive the idea that formula funding can be the fuel for innovation rather than just a mainstay of the status quo. Let’s be honest: It’s been a long time since Title I has been considered sexy. LWW could energize Title I advocates and those who want schools to have the freedom to choose what works to improve outcomes for children. Title I needs to move from a compliance mindset to an innovation mindset, and LWW could help make this happen. It could help establish Title I schools as the places where up-and-coming teachers and administrators want to be, because those are the schools that get the first crack at the latest proven innovations.

Leveraging What Works would also energize the world of research and development, and the funders of R&D within and outside government. They would see programs proven in rigorous research being eagerly adopted by schools nationwide, and seeing the clear connection between research, development, and practice, they would redouble their efforts to create and evaluate promising, replicable programs of all kinds.

Until recently, it would have been difficult to justify an initiative like Leveraging What Works, but thanks to Investing in Innovation (i3), IES, NSF, and other funders, the number of proven programs is growing. For example, I recently counted 28 elementary reading approaches, from tutoring to whole-school reform, that should meet the EDGAR standards, and more are qualifying every year. Every one of these is actively disseminating its methods and is ready to grow.

One curious aspect of the Leveraging What Works proposal is that it provides incentives for the use of formula funding to adopt proven programs but does not provide similar incentives for adopting proven programs using competitive grants. When competitive grants are offered to schools, districts, or states, it would be easy to incentivize the use of proven programs by giving preference points to proposals that commit to using them. For example, proposals might get four extra points for choosing a program that meets the EDGAR “strong” definition, and two points for choosing a program meeting the EDGAR “moderate” definition, as I’ve argued before. It may be that this strategy was left out of the budget proposal because it does not really cost anything, so I hope it will be part of the administration’s plans whatever happens with LWW.

The Greek mathematician Archimedes said, “Give me a lever long enough and a fulcrum on which to place it and I’ll move the Earth.” Leveraging What Works could be such a lever, a modest investment with potential to make a meaningful difference in the lives of millions of children.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Fund What Works!

President Obama’s recently released budget may or may not make its way into law, but it has already made its way into the hearts of those who believe that government programs must be held to account for producing the outcomes they are intended to produce. Red or blue, everyone should want government dollars to make a difference.

For PK-12 education, evidence appears in a starring role in the president’s proposal. To me, the most exciting innovation in the budget is something called Leveraging What Works. This is a proposal for a competitive grant program in which the U.S. Department of Education would reward school districts that use their existing federal formula funds for activities that have a strong or moderate evidence base as defined in the recent EDGAR changes, a proposal I have advocated on this blog. The total amount requested, $100 million per year, is tiny compared with current expenditures, but it would be a start on the idea that federal dollars should increasingly go to proven programs and practices. Hopefully, this idea would go along with policies encouraging adding preference points to competitive proposals that commit to using proven approaches.

The proposal would add $50 million for School Improvement Grants (SIG) for states to use to encourage school districts to implement proven strategies. If the standards for “proven” are rigorous, this would be another way to use modest funding to spur effective use of much larger existing funding.

The Investing in Innovation (i3) program is showing its value in moving along the pipeline of educational innovations from development to validation to scale-up. The budget would more than double annual funding for this crucial program. It would also double funding for early education research and evaluation and would invest $20 million a year to test strategies to improve transitions from preschool to third grade. It adds $70 million to the Social Innovation Fund (SIF) to test promising solutions to a broad array of problems relating to children and youth.

There is much more in the remarkable budget. I hope our Congress can put aside partisanship and agree on a compelling new direction for government: Fund What Works!

Are Proven Educational Innovations Ready for Prime Time?

These are good times for evidence-based reform in education. Due in particular to Investing in Innovation (i3) and the Institute of Education Sciences (IES), the number of proven programs in all subjects and grade levels in increasing, and as i3 programs come to the ends of their evaluations, the number of proven programs should accelerate (even though there are also sure to be many disappointments).

The increasing numbers of programs proven effective in rigorous research creates new opportunities for policy and practice in education. Already, School Improvement Grants (SIG) are newly introducing an option for schools to choose proven, comprehensive reform models. Other areas of policy may also soon begin to encourage or incentivize use of programs with strong evidence.

If these changes in policy begin to happen, it will begin to matter whether educational programs proven to be effective are in fact ready for prime time, meaning that they are ready to be disseminated and supported in the form they existed when they were successfully evaluated. It would be catastrophic if educators and policy makers began looking on the What Works Clearinghouse, for example, or looking for programs that meet the EDGAR standards for strong or moderate evidence of effectiveness, and found many program that were unavailable, or unrealistic, or impractical.

Much as providing evidence of effectiveness is an advance in education, there is a real need for a determination of the degree to which programs are also ready for widespread implementation.

Some indicators of readiness for prime time would be easy to assess. For example, programs that lack a web site, do not offer materials or training, or otherwise do not exist in anything like the form in which they were evaluated cannot be considered ready for implementation. Some programs used procedures in their evaluation that could never be replicated, such as science programs that provide each experimental class with enough graduate students to monitor and assist every lab group. Some proven technology products run on hardware that no longer exists.

Many studies use measures of learning outcomes that are closely aligned with what was taught in the experimental but not the control group. Such studies might be excluded on the basis that the overaligned measure does not have meaning beyond the experiment itself.

Educators who choose to use proven programs have a right to be confident that the programs they have selected are, if implemented well, likely to result in enhanced performance on measures they care about. Finding a lot of programs that cannot be implemented under ordinary circumstances and with meaningful measures will diminish interest in evidence-based reform.

Evidence-based reform itself is ready for prime time in education, but its future depends on whether it is perceived to produce genuine benefits for children. We need to make sure that the proven programs we offer to educators meet their needs and those of the students, not just scientific standards.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Promoting Proven Programs in Title I: The Bully Pulpit

Title I, the 800-pound gorilla of federal education policy, spends $15 billion a year to help high-poverty schools enhance outcomes for their students. The greatest victory for evidence-based reform would be for the roughly 51,500 Title I schools to make far greater use of programs known to enhance student learning, benefitting millions of at-risk children throughout the U.S. Yet because Title I is a formula grant, it is difficult for federal policy to increase use of proven approaches. Competitive grants can provide preference points for using proven models, as I’ve argued before, but in formula grants, it is more difficult to nudge educational leaders toward use of proven programs, since they will receive their money as long as they follow the rules.

One solution to this problem might be to borrow President Theodore Roosevelt’s conception of the presidency as a “bully pulpit.” In other words, even during a time of congressional gridlock, it is possible for the administration to promote the use of proven approaches, even in formula grants, at little or no cost.

The first thing the U.S. Department of Education would have to do is to review all the programs in the What Works Clearinghouse according to the simpler, clearer standards in the EDGAR regulations. Someone would then have to prune the resulting lists of programs, identifying programs that meet the EDGAR standards for “strong” and “moderate” levels of evidence to remove programs that no longer exist or that do not have anyone providing training and materials similar to those provided in the successful studies. The remaining programs would represent a good starting list of programs that, if implemented well, would be likely to have positive impacts on student achievement.

Department officials could then publicize this list in many ways. Certainly, they could create a web site showing the programs and the evidence behind them and linking to the programs’ web sites. They might sponsor “effective methods fairs” around the U.S. to demonstrate programs available for schools and districts to choose. They might distribute certificates to schools that adopt proven programs and then implement them with fidelity, as certified by the developers.

These strategies and others could arouse widespread interest in proven programs, and help school leaders make a wide array of choices of programs appropriate to their needs.

If funds became available, the Department might provide modest incentive grants to help schools supplement the start-up costs of proven programs. But even without special incentive funding, schools should be able to make choices from among programs known to be likely to help them succeed with their children, using their existing Title I funds.

A creative “bully pulpit” policy might begin a process of expanding use of existing proven programs, encouraging creation and evaluation of new ones, and increasing sophistication in choosing how to spend federal resources. All of this could be accomplished for nearly nothing, while gradually moving the $15 billion in Title I toward more effective uses. Over time, such a policy would also encourage developers and researchers to create and evaluate programs likely to meet EDGAR standards, and it could help build political support for investments in R&D that ultimately result in better outcomes for children on a broad scale.

A “bully pulpit” strategy would still need to be accompanied by policies of providing incentives to adopt proven programs in competitive grants, and with continued support for the R&D pipeline, such as that provided by Investing in Innovation (i3), the Institute of Education Sciences (IES), and the National Science Foundation (NSF). However, development and research in education have to go beyond R&D; they need to be seen as a routine, growing part of the world of educational practice and innovation.

*Photo courtesy of the Library of Congress

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

On Teacher Evaluation

Could evidence provide a solution to the continuing controversy about teacher evaluation? In a recent blog, I discussed low-cost and free ways to use proven programs to substantially improve outcomes in America’s schools. One of the most promising of these is based on providing alternatives to federal and state policies mandating new forms of teacher evaluation that combine extensive principal observations with value-added scores from students’ state reading and math tests.

Current teacher evaluation schemes are among the most contentious of the current administration’s policies. While states have long held schools accountable for their students’ achievement, teachers are now being increasingly and individually held accountable, based on some combination of frequent, structured principal observations and value-added scores from state achievement tests. States that received giant Race to the Top grants have had to have teacher evaluation plans as a part of their applications, as have states seeking waivers from onerous requirements of No Child Left Behind.

In concept, evaluating teachers makes perfect sense. In what private company are employees not evaluated and held accountable for their contribution to their company’s bottom line? Why should teachers be exempt from assessments of their job performance? In fact, teachers have been evaluated by their principals since long before Willa Cather was a first-year teacher, and these observations have long identified inadequate teachers.

In practice, evaluating teachers is not so easy. For a long time, principals have evaluated teachers based on formal observations. The problem is that principals give the great majority of their teachers the highest possible ratings, so they really only differentiate for teachers they perceive to be very poor. This is not unique to education, but is common in any business where metrics for success are subjective.

The new evaluation systems involve much more frequent and structured observations, and districts are paying a great deal of money to train their principals in detailed observation strategies. But guess what? Despite putting in many long hours learning and using the new methods, principals still end up giving all but their very least effective teachers very high scores. Further, even when trained researchers use these forms, they cannot make reliable differentiations between teachers from below average to outstanding (though, like the principals, they can reliably identify very poor teachers).

If teacher ratings are difficult to do reliably and tend to produce overwhelmingly high ratings, then overall evaluations of teachers will largely depend on value-added measures based on the reading and math scores of children in the grades tested, 3-8, plus one grade in high school (usually 11). Right off the bat, there’s an obvious problem: what about teachers of grades below 3, and of subjects other than reading and math? Middle and high schools do not usually even teach reading as a separate course. So how fair or accurate is it to judge preschool, kindergarten, grade 1-2, art, music, PE, and secondary English, science, and social studies teachers based on students’ reading and math gains?

There are many other technical problems of value added, mostly having to do with the difficulties of separating the effects teachers have from the effects of poverty, home environments, other teachers in the school, and so on.

Further, let’s be realistic about what teacher evaluations can do. They may help identify teachers who are doing a very poor job, and this information might be used to direct them toward assistance or toward other professions. However, it is not possible to fire a large proportion of teachers. There is not a great army of terrific teachers waiting for opportunities to teach, especially in high-poverty urban and rural schools. The small proportion of teachers who do need to leave the profession was, in general, already being identified by principals long before the current enthusiasm for teacher evaluation.

So if firing more teachers is not the main goal of current teacher evaluation systems, what is? The hope seems to be that evaluations will improve outcomes for whole schools by providing feedback and incentives for teachers to do their best.

Here at last we come to a testable hypothesis. If teacher evaluations help all teachers in a school get to the top of their game, then schools should show improvements in student test scores, right?

This might perhaps be true, but I have not yet seen a convincing study demonstrating such an effect. You might imagine that a school improvement approach that costs a lot in principal time and training, not to mention teacher angst and confrontations, would have been tested out in large-scale, randomized experiments, before it was required in schools across our nation. As one counterexample, all programs receiving i3 funding have to be subjected to third-party evaluations far more stringent than any that have evaluated student outcomes of recent teacher evaluation policies, yet the successfully evaluated programs are rare in practice while the unevaluated teacher evaluation schemes are nationally mandated. There are many programs for improving reading and math performance in grades K-12 that have already been found to be effective in rigorous evaluations, and many more proven programs are emerging from i3 and other sources. If the goal of teacher evaluation systems is to improve student outcomes, why not encourage use of all programs that are known to improve outcomes?

So here is my modest proposal for improving America’s elementary and secondary schools, at minimal cost.

In all states required to use the new teacher evaluation schemes (extensive principal observation plus value-added scores) under Race to the Top, NCLB waivers, or other policy initiatives, allow schools to apply to implement proven programs instead of the new teacher evaluation schemes. These programs could be chosen from among those that meet current EDGAR standards for strong or moderate evidence of effectiveness. Principals would be expected to continue to use teacher evaluations to identify incompetent teachers.
In order for schools to participate, 80% of their staffs would have to agree by secret ballot to implement the proven program with integrity and fidelity, using resources currently devoted to teacher evaluation.
Schools selecting this option would then have three years to implement their chosen program or programs. Their students’ state test scores over the three-year period would be compared to those of a group of schools using the state’s teacher evaluation systems (extensive principal evaluation plus value added) and serving similar students.
After three years, schools scoring no better than their comparison group would have to return to using the state’s teacher evaluation plan.
During the time this is going on, the federal government and other funders would fund the development and evaluation of whole-school reforms and reading and math programs that might be added into the set of proven options schools might adopt over time, as this activity progresses.

If teacher evaluation schemes are intended to improve the performance of whole schools, then it is certainly fair to compare them to alternative strategies. Teachers and principals might be powerfully motivated to implement proven models well because their success keeps them out of the new teacher evaluation systems that are, let’s face it, not terrifically popular among educators. Kids would benefit today from proven programs, and knowledge would grow about how to unite schools around an enthusiastic embrace of proven strategies.If the proven strategies cost no more than the teacher evaluation plans, which seems likely, this could all be done at little or no cost.

Higher-achieving kids, happier teachers, happier principals, more knowledge about schoolwide reform, all at little or no cost to anyone. Does this sound good to anyone?

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Six Low-Cost or Free Ways to Make American Education the Best in the World

It does not take a political genius to know that for the foreseeable future, American education is not going to be rescued by a grand influx of new money. Certainly in the near term, the slow economic recovery, gridlock in Washington, and other factors mean that the path to substantial improvement in outcomes is going to be paved not with new gold, but with better use of the gold that’s already there.

No problem.

We already spend a lot of money on education. The task right now is to change how we spend federal, state, and local resources so that more money is spent on programs and practices known to make a difference rather than on investments with zero or unknown impacts on learning. Here are my top six suggestions for how to spend our education resources more effectively. (I’ll go into more details on these in future blogs).

1. Provide incentives for schools and districts to implement programs with strong evidence of effectiveness in competitive grants. In competitive grants in all parts of federal and state government, offer “competitive preference points” for applicants who promise to adopt and effectively implement programs proven to be effective. For example, schools proposing to implement programs identified as having “strong evidence of effectiveness” under the new EDGAR definitions might receive four extra points on a 100-point scale, while those meeting the criteria for “moderate evidence of effectiveness” might receive two points. Readers of this blog have seen me make this recommendation many times. Perfect example: School Improvement Grants for low-achieving schools. Cost: zero.

2. Provide incentives for schools and districts to implement programs with strong evidence of effectiveness in formula grants. The big money in federal and state education funding is in formula grants that go to districts and schools based on, for example, levels of poverty, rather than competitive applications. The classic example is Title I. Schools have great freedom in how they use these funds, so how can they be encouraged to use them in more effective ways? The answer is to provide additional incentive funding if schools or districts commit to using proven programs with their allotted formula funds. For example, if schools agree to use a portion of their (formula-driven) Title I funds on a proven program, they may qualify for additional funds (not from the formula pot). This was the idea behind the Obey-Porter Comprehensive School Reform initiative of the late 1990s, which encouraged thousands of Title I schools to adopt whole-school reform models. Cost: This strategy could be done at a cost of perhaps 1% of the current $15 billion annual Title I budget.

3. Offer commitment to proven programs as an alternative to use of value-added teacher evaluation models. A central part of the current administration’s policies is incentivizing states and districts to adopt teacher evaluation plans that combine principal ratings of teachers with value-added scores based on students’ state reading and math tests. This is a required part of Race to the Top in those states that received this funding, and it is a required element of state applications for a waiver of elements of No Child Left Behind.

In practice, current teacher evaluation policies are intended to do two things. First, they insist that schools identify extremely ineffective teachers and help them find other futures. If done fairly and consistently, few oppose this aspect of teacher evaluation. Principals have evaluated teachers and identified those with serious deficits forever, and I am not arguing against continuing this type of evaluation.

The second purpose of the teacher evaluation policies is to improve teaching and learning for all teachers. This is the expensive and contentious part of the policies; in most states it requires a combination of frequent, structured observation by principals and “value-added” assessments of a given teacher’s students. The technical difficulties of both are substantial, and no study has yet shown any benefit to student learning as a result of going through the whole ordeal.

If the goal is better teaching and learning, why not require that all reform approaches meet the same evidence standards? If a school proposes to use a schoolwide strategy that (unlike current teacher evaluation policies) has strong evidence of effectiveness, the school should be permitted, even encouraged, to suspend aspects of the new model as long as it is implementing proven alternatives with fidelity and good outcomes. Cost: Modest, assuming proven programs are similar in cost to the expensive new teacher evaluation strategies.

4. Train and equip paraprofessionals as tutors. The most common expenditure of Title I funds is on paraprofessionals or aides, educators who do not usually have teaching degrees but perform all sorts of functions within schools other than class teaching. Paraprofessionals can be wonderful and capable people, but evidence in the U.S. and U.K. consistently finds that as they are most commonly used, they make little difference in student learning.

Yet there is also extensive evidence that paraprofessionals can be very effective if
they are trained to provide well-structured one-to-one or one-to-small group tutoring to students who are struggling in reading and math. Paraprofessionals are a multi-billion dollar army eager and capable of making more of a difference. Let’s empower them to do so. Cost: Minimal (just training and materials).

5. Encourage schools to use Supplemental Educational Services (SES) funding on proven programs. As part of No Child Left Behind, Title I schools had to use substantial portions of their Title I dollars to provide Supplemental Educational Services (SES) to students in schools failing to meet standards. Study after study has found SES to be ineffective, and expenditures on SES are waning, yet they remain as a significant element of Title I funding, even in states with waivers. If districts could be encouraged to use SES funds on programs with evidence of effectiveness in improving achievement (such as training paraprofessionals and teachers to be tutors in reading and/or math), outcomes are sure to improve. Cost: Minimal.

6. Invest in research and development to identify effective uses of universal access to tablets or computers. Despite economic and political hard times, schools everywhere are moving rapidly toward providing universal, all-student access to tablets or computers. There is a lot of talk about blended learning, flipped learning, and so on, but little actual research and development is going on that is likely to identify effective and replicable classroom strategies likely to make good use of these powerful tools. As it has done many times before, American education is about to spend billions on technology without first knowing which applications actually work. Setting aside a tiny percentage of the costs of the hardware and software, we could fund many innovators to create and rigorously evaluate approaches using all-student technology access, before we get stuck on ineffective solutions (again). Cost: modest.

* * *

I’m sure there are many more ways we could shift existing funds to advance
American education, but they all come down to one common recommendation: use what works. Collectively, the six strategies I’ve outlined, and others like them, could catapult American education to the top on international comparisons, greatly reduce education gaps, and prepare our students for the demands of a technological economy, all at little or no net cost, if we’re willing to also stop making ineffective investments. Moreover, all of these six prescriptions could be substantially underway in the next two years, during the remainder of the current administration. All could be done by the Department of Education alone, without congressional action. And again, I’m sure that others have many other examples of low-cost and no-cost solutions that I haven’t thought of or haven’t addressed here.

A revolution in American education does not necessarily require money, but it does require courage, leadership, and resolve. Those are resources our nation has in abundance. Let’s put them to work.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

Many Programs Meet New Evidence Standards

One of the most common objections to evidence-based reform is that there are too few programs with strong evidence of effectiveness to start encouraging schools to use proven programs. The concern is that it looks bad if a policy of “use what works” leads educators to look for proven programs, only to find that there are very few such programs in a given area, or that there are none at all.

The lack of proven programs is indeed a problem in some areas, such as science and writing, but it is not a problem in others, such as reading and math. There is no reason to hold back on encouraging evidence where it exists.

The U.S. Department of Education has proposed changes to its EDGAR regulations to define “strong” and “moderate” levels of evidence supporting educational programs. These standards use information from the What Works Clearinghouse (WWC), and are very similar to those used in the federal Investing in Innovation (i3) program to designate programs eligible for “scale-up” or “validation” grants, respectively.

As an exercise, my colleagues and I checked to see how many elementary reading programs currently exist that qualify as “strong” or “moderate” according to the new EDGAR standards. This necessitated excluding WWC-approved programs that are not actively disseminated and those that would not meet current WWC standards (2.1 or 3.0), and adding programs not yet reviewed by WWC but that appear likely to meet its standards.

Here’s a breakdown of what we found.

Beginning Reading (K-1)
Total programs                              26
School/classroom programs      16
Small-group tutoring                   4
1-1 tutoring                                   6

Upper Elementary Reading (2-6)
Total programs                               17
School/classroom programs       12
Small-group tutoring                    4
1-1 tutoring                                     1

The total number of unique programs is 35 (many of the programs covered both beginning and upper-elementary reading). Of these, only four met the EDGAR “strong” criterion, but the “moderate” category, which requires a single rigorous study with positive impacts, had 31 programs.

We’ll soon be looking at secondary reading and elementary and secondary math, but the pattern is clear. While few programs will meet the highest EDGAR standard, many will meet the “moderate” standard.

Here’s why this matters. The EDGAR definitions can be referenced in any competitive request for proposals to encourage and/or incentivize the use of proven programs, perhaps offering two competitive preference points for proposals to implement programs meeting the “moderate” standard and three points for proposals to adopt programs meeting the “strong” standard.

Since there are many programs to choose from, educators will not feel constrained by this process. In fact, many may be happy to learn about the many offerings available, and to obtain objective information on their effectiveness. If none of the programs fit their needs, they can choose something unevaluated and forgo the extra points, but even then, they will have considered evidence as a basis for their decisions. And that would be a huge step forward.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin

EDGAR and the Two-Point Conversion

Once upon a time, there was a football player named EDGAR. His team was in the state championship. It was the fourth quarter, and they were down by seven points. But just as time ran out, EDGAR ran around the opposing line and scored a touchdown.

EDGAR’s coach now had a dilemma. Should he try a safe kick for one point, putting the game into overtime, or go for a much more difficult two-point conversion, one chance to score from the five-yard line?

Evidence-based reform faces a similar dilemma. The U.S. Department of Education proposed several months ago some additions to EDGAR, not a football player but a stultifyingly boring doorstopper of a book of regulations for grants and contracts. These new regulations, as I noted in an earlier blog, are really exciting, at least to evidence wonks. They define four levels of evidence for educational programs: Strong evidence of effectiveness, moderate evidence of effectiveness, evidence of promise, and strong-theory. These definitions are similar to those used in Investing in Innovation (i3) to qualify proposals for scale-up (strong), validation (moderate), or development grants (evidence of promise or strong theory).

Here’s where the two-point conversion comes in. Readers of this blog may recall that I have long advocated the provision of, say, two competitive preference points in discretionary grants for proposals promising to use proven programs when such programs are available. The idea is that two points on a scale of 100 would greatly increase the interest of grant writers in proven programs without heavy handedly requiring their use. No grant writer ignores two points, but school leaders may have good reasons to prefer programs that have not yet been successfully evaluated. In those cases, the schools would be free to forego the competitive preference points. Still, the two points would telegraph the government’s support for the use of proven programs without undermining local control. Requests for proposals routinely include competitive preference points for criteria a lot less important than whether the program schools are proposing to use have been proven to work. Why not provide at least this much encouragement for potential innovators, at no additional cost to the government?

Offering two points for proposing to use proven programs would bring about a major conversion of education reform. It would focus attention on the evidence and make school and district leaders aware that effective options are available to them and are ok, even encouraged, by the powers that be.

As in my fictitious football game, it’s EDGAR who sets up the two-point conversion. The new definitions in EDGAR make it relatively easy for the Department of Education to put competitive preference points or other encouragements for use of proven programs into requests for proposals. EDGARS’s end run does not win the game, but it creates a condition in which the game can be won – by a two point conversion.

Follow Robert E. Slavin on Twitter: www.twitter.com/RobertSlavin