The Sweet Land of Carrots: Promoting Evidence with Incentives

Results for America (RFA) released a report in July analyzing the first 17 Every Student Succeeds Act (ESSA) plans submitted by states. RFA was particularly interested in the degree to which evidence of effectiveness was represented in the plans, and the news is generally good. All states discussed evidence (it’s in the law), but many went much further, proposing to award competitive funding to districts to the degree that they propose to adopt programs proven to be effective according to the ESSA evidence standards. This was particularly true of school improvement grants, where the ESSA law requires evidence, but many state plans extended this principle beyond school improvement into other areas.

As an incurable optimist, this all looks very good to me. If state leaders are clear about what qualifies as “proven” under ESSA, and clear about how proper supports are also needed (e.g. needs assessments, high-quality implementation), then this creates an environment in which evidence will, at long last, play an important role in education policy. This was always the intent of the ESSA evidence standards, which were designed to make it easy for states and districts to identify proven programs so that they could incentivize and assist schools in using such programs.

The focus on encouragement, incentives, and high-quality implementation is a hallmark of the evidence elements of ESSA. To greatly oversimplify, ESSA moves education policy from the frightening land of sticks to the sweet land of carrots. Even though ESSA specifies that schools performing in the lowest 5% of their states must select proven programs, schools still have a wide range of choices that meet ESSA evidence standards. Beyond school improvement, Title II, Striving Readers, and other federal programs already provide funds to schools promising to adopt proven programs, or at least provide competitive preference to applicants promising to implement qualifying programs. Instead of the top-down, over-specific mandates of NCLB, ESSA provides incentives to use proven programs, but leaves it up to schools to pick the ones that are most appropriate to their needs.

There’s an old (and surely apocryphal) story about two approaches to introduce innovations. After the potato was introduced to Europe from the New World, the aristocracy realized that potatoes were great peasant food, rich in calories, easy to grow, and capable of thriving in otherwise non-arable land. The problem was, the peasants didn’t want to have anything to do with potatoes.

Catherine the Great of Russia approached the problem by capturing a few peasants, tying them up, and force-feeding them potatoes. “See?” said her minsters. “They ate potatoes and didn’t die.”

Louis XIV of France had a better idea. His minsters planted a large garden with potatoes, just outside of Paris, and posted a very sleepy guard over it. The wily peasants watched the whole process, and when the guard was asleep, they dug up the potatoes, ate them with great pleasure, and told all their friends how great they were. The word spread like wildfire, and soon peasants all over France were planting and eating potatoes.

The potato story is not precisely carrots and sticks, but it contains the core message. No matter how beneficial an innovation may be, there is always a risk and/or a cost in being the first on your block to adopt it. That risk/cost can be overcome if the innovation is super cool, or if early innovators gain status (as in Louis XIV’s potato strategy). Alternatively, or in addition, providing incentives to prime the pump, to get early adopters out promoting innovations to their friends, is a key part of a strategy to spread proven innovations.

What isn’t part of any effective dissemination plan is sticks. If people feel they must adopt particular policies from above, they are likely to be resentful, and to reason that if the government has to force you to do something, there must be something wrong with it. The moment the government stops monitoring compliance or policies change, the old innovations are dropped like, well, hot potatoes. That was the Catherine the Great strategy. The ESSA rules for school improvement do require that schools use proven programs but this is very different from being told which specific programs they must use, since they have a lot of proven programs to choose from. If schools still can choose which program to implement, then those who do make the choice will put all their energy into high-quality implementation. This is why, in our Success for All program, we require a vote of 80% of school staff in favor of program adoption.

My more cynical friends tell me that once again, I’m being overly optimistic. States, districts, and schools will pretend to adopt proven programs to get their money, they say, but won’t actually implement anything, or will do so leaving out key components, such as adequate professional development. I’m realistic enough to know that this will in fact happen in some places. Enthusiastic and informed federal, state, and district leadership will help avoid this problem, but it cannot be avoided entirely.

However, America is a very big country. If just a few states, for example, wholeheartedly adopted pro-evidence policies and provided technical assistance in selecting, implementing, evaluating, and continuously improving proven programs, they would surely have a substantial impact on their students. And other states would start to notice. Pretty soon, proven programs would be spreading like French fries.

I hope the age of the stick is over, and the age of the sweet carrot has arrived. ESSA has contributed to this possibility, but visionary state and district leaders will have to embrace the idea that helping and incentivizing schools to use proven programs is the best way to rapidly expand their use. And expanding well-implemented proven programs is the way to improve student achievement on a state or national scale. The innovations will be adopted, thoughtfully implemented, and sustained for the right reason – because they work for kids.

This blog is sponsored by the Laura and John Arnold Foundation

Advertisements

Strong Evidence Meets Local Control

At the political level, the burgeoning evidence movement in education is running smack dab into a growing focus on state and local control of education. The Every Student Succeeds Act (ESSA) emphasizes state and local control and it’s easy to see why. Educators, local and state policy makers, and the general public are tired of being micromanaged from Washington. But where does local control leave evidence? Does it make sense to have each state and locality select its own standards of evidence?

First, it is important to note that evidence-based reform and local control need not be in conflict. In fact, ESSA balances its emphasis on state and local control with an emphasis on evidence-based reform. It establishes criteria for strong, moderate, and promising evidence of effectiveness.

If there are established lists of proven programs as defined by ESSA standards, then state and local education leaders can choose to use the proven programs they think best for their schools and students, hopefully highlighting them when they make decisions. If this is how evidence is applied, then there is no conflict at all between evidence-based reform and local control.

While ultimate decisions are up to state and local leaders, it is entirely appropriate for the federal government to incentivize use of proven programs, as I’ve often urged in this space. For example, I’ve argued that in federal competitive grants, applicants could receive competitive preference points if they promise to adopt proven programs. This would involve a federal definition of “proven programs” for schools and districts that want the extra points. ESSA does exactly this with at least seven competitive programs. Locals need not seek the preference points, and they need not apply for discretionary grants at all.

Ideally, incentives should not be necessary to motivate state and local leaders to adopt proven programs. They care about their students and want them to learn, so why should they not actively seek the best approaches? Yet evidence has played so small a role in educational policy and practice up to the present that federal incentives may be necessary for a period of time to raise state and local awareness of approaches found to be effective in rigorous evaluations.

The Department of Agriculture has long sponsored research and development to identify productive farming methods, seeds, and so on, and proactively disseminates information about the findings, but has never, to my knowledge, required farmers to use these methods. However, farmers eagerly seek proven methods because they increase yields and efficiency. In the same way, the Department of Education can continue to sponsor research and development and proactively disseminate information on what works. Then it’s up to local educators to put that information to work in service of improving outcomes for children.

When Will We Reach Our 1962 Moment in Education?

2015-04-23-1429791233-4460932-HP69_57Chevy.jpg

When I was in college, I had an ancient 1957 Chevy. What a great car. Stylish, dependable, indestructible.

My 1957 Chevy was beautiful, but it had no seatbelts, no airbags, and no recourse if the brakes went out. It got about 13 miles to the gallon, polluted the atmosphere, and was not expected to last more than 100,000 miles. Due to development, evaluation, and public-spirited policy, all these problems have been solved. Automotive design has been revolutionized by embracing policies based on innovation and evidence.

Not that I remember 1957 very well, but I was thinking about it as a model for where we are today in evidence-based reform in education, as distinct from medicine. In 1957, drug companies could make any claims they liked about medications. There was research, but physicians routinely ignored it. However, change was on the way. In 1962, the Kefauver-Harris Amendment required that all drug applications to the Food and Drug Administration (FDA, established in 1927) demonstrate “substantial evidence” of safety and effectiveness. These standards continue to evolve, but today it is unthinkable that drug companies could make misleading claims about unproven medicines.

In 1957, the progress toward evidence-based reform in medicine would have been clear, but the policy world was not yet ready. For one thing, the American Medical Association fought tooth and nail against the evidence standards, as did most drug companies. Yet evidence prevailed because despite the power and money of the AMA and the drug companies, millions of ordinary citizens, not to mention the majority of physicians, knew that prescribing medications of unknown safety and effectiveness was just plain wrong. Everyone takes medicine, or we have relatives who do, and we want to know what works and what doesn’t. Specifically, a European drug called Thalidomide taken by pregnant mothers caused massive and widespread birth defects, and this swept away the opposition to drug testing standards.

In education, we have not reached our 1962 moment. Publishers and software developers are free to make any claims they like about the effectiveness of their products, and educators have difficulty sorting effective from ineffective products. Yet the handwriting is on the wall. Rigorous evaluations of educational programs are becoming more and more common. Many of these evaluations are being paid for by the companies themselves, who want to be on the right side of history when and if our 1962 moment arrives.

In education, our 1962 will probably not involve an equivalent of the FDA or a prohibition on the use of untested products. Unlike medicine, few educational products are likely to be harmful, so experimentation with new approaches is a lot safer. What is more likely, I believe, is that there will be incentives and encouragement from various levels of government for schools to adopt proven programs. In particular, I think it is very likely that Title I and other federal programs will begin insisting on a strong evidence base for investments of federal dollars.

To reach our 1962 moment will require sustained investment in development, evaluation, and scale-up of proven programs in all subjects and grade levels, and a change of policies to encourage the use of proven programs.

I hope our 1962 moment is coming soon. To bring it closer, we have a lot of work to do, in innovation, evaluation, policy, and practice. Government, foundations, innovators, researchers, and anyone who knows the transformative potential of education should be working toward the day when we no longer have to guess what works and what doesn’t. This is the time to build up our stock of proven, replicable programs of all kinds. It is also the time to try policy experiments such as Investing in Innovation (i3)SIG evidence-proven whole-school models, and Leveraging What Works, because when our 1962 comes, we will need to know how to build support for the whole evidence movement. Like my beloved 1957 Chevy, I hope we’re driving confidently toward our 1962 and beyond, confident that every new year will bring better outcomes for all.

Leveraging What Works

2015-02-19-HP6319_2015.jpg

In my blog from two weeks ago, I discussed several exciting proposals in President Obama’s recent budget relating to increasing the role of evidence in education policy and practice. Today, I want to say more about one of these proposals, Leveraging What Works (LWW).

Leveraging What Works is deceptively simple. It offers grants totaling $100 million nationwide to school districts willing to use the grant, along with a portion of its formula funds — such as Title I and IDEA — to adopt proven programs that meet the “strong” or “moderate” level of evidence of effectiveness as defined in EDGAR.

Simple though it appears, Leveraging What Works would be revolutionary. Here’s why.

First, the program would generate a huge amount of interest. Winning LWW funding would be sought after avidly not only for the money itself but as a feather in the cap of innovative thought-leader districts. These districts will be eager to win the money and tell their stories. The whole process will create a positive “buzz” around the use of proven programs.

Because of the money and the positive buzz, many more districts will apply for LWW funding than can be funded. Yet having looked at the range of proven programs available to them, many of these districts will choose to adopt proven programs using their formula funding even without the LWW grant. This is exactly what happened with the Obey-Porter Comprehensive School Reform Demonstration Act (CSR) of the late 1990’s. Thousands of schools applied for modest grants to help them adopt whole-school models, and each year, hundreds of schools that were turned down for grant funding adopted CSR models anyway, using other funding.

Leveraging What Works could revive the idea that formula funding can be the fuel for innovation rather than just a mainstay of the status quo. Let’s be honest: It’s been a long time since Title I has been considered sexy. LWW could energize Title I advocates and those who want schools to have the freedom to choose what works to improve outcomes for children. Title I needs to move from a compliance mindset to an innovation mindset, and LWW could help make this happen. It could help establish Title I schools as the places where up-and-coming teachers and administrators want to be, because those are the schools that get the first crack at the latest proven innovations.

Leveraging What Works would also energize the world of research and development, and the funders of R&D within and outside government. They would see programs proven in rigorous research being eagerly adopted by schools nationwide, and seeing the clear connection between research, development, and practice, they would redouble their efforts to create and evaluate promising, replicable programs of all kinds.

Until recently, it would have been difficult to justify an initiative like Leveraging What Works, but thanks to Investing in Innovation (i3), IES, NSF, and other funders, the number of proven programs is growing. For example, I recently counted 28 elementary reading approaches, from tutoring to whole-school reform, that should meet the EDGAR standards, and more are qualifying every year. Every one of these is actively disseminating its methods and is ready to grow.

One curious aspect of the Leveraging What Works proposal is that it provides incentives for the use of formula funding to adopt proven programs but does not provide similar incentives for adopting proven programs using competitive grants. When competitive grants are offered to schools, districts, or states, it would be easy to incentivize the use of proven programs by giving preference points to proposals that commit to using them. For example, proposals might get four extra points for choosing a program that meets the EDGAR “strong” definition, and two points for choosing a program meeting the EDGAR “moderate” definition, as I’ve argued before. It may be that this strategy was left out of the budget proposal because it does not really cost anything, so I hope it will be part of the administration’s plans whatever happens with LWW.

The Greek mathematician Archimedes said, “Give me a lever long enough and a fulcrum on which to place it and I’ll move the Earth.” Leveraging What Works could be such a lever, a modest investment with potential to make a meaningful difference in the lives of millions of children.

To the New Congress

With the results of the mid-term elections just behind us, it is time to think about the opportunities and challenges for education policy in the next two years and beyond. At the top level gridlock is sure to continue, but much progress remains possible if the administration and congressional leaders can cooperate in areas where they fundamentally agree. These include a shared belief that American education cannot be complacent, but must continue to advance by helping teachers, districts and school leaders raise standards, improve teaching and learning, and make wise choices for children based on the best available evidence. Here are some specific actions I’d suggest to accomplish these goals.

1. Maintain the Investing in Innovation (i3) Program and other sources of evidence
Investing in Innovation (i3) is a U. S. Department of Education program that funds development, evaluation, and dissemination of proven programs for all of grades pre-k to 12. In a policy environment emphasizing local schools’ right to choose their path to success, i3 offers information on “what works” that is essential in a system moving from government mandates to local control. Just as Department of Agriculture-funded research has long provided information but not direction to farmers, i3, the Institute of Education Sciences (IES), and other agencies are providing information for informed decision-making in education.

2. Encourage use of proven programs
Where the federal government continues to fund programs of assistance to schools such as Title I, educators should be encouraged to use programs and practices with strong evidence of effectiveness. This encouragement could include modest incentives in competitive grants (such as a few preference points) or modest additional funds in formula grants if grantees agree to use a portion of their formula grants (such as Title I) on proven programs.

3. Maintain and upgrade the What Works Clearinghouse
The What Works Clearinghouse provides information on the strength of evidence supporting various educational programs. It is not as user-friendly as it might be, and it needs to be revamped to focus on pragmatic programs with impacts on measures that matter. However, for evidence to matter in a system of informed local choice, something much like the WWC is needed.

While there are many issues on which the Congress and the administration will disagree, I hope and expect that they will agree that every federal dollar spent on education should make the largest possible difference for children. Investments designed to create, evaluate, and disseminate effective approaches have to be central to any strategy intended to help educators improve outcomes among all schools. Our children can’t wait for better educational programs and practices. They need them now. This is something that all people of good will can agree on.

* Illustration by James Bravo

Six Low-Cost or Free Ways to Make American Education the Best in the World

2014-06-12-HP38Image_06_12_2014.jpg

It does not take a political genius to know that for the foreseeable future, American education is not going to be rescued by a grand influx of new money. Certainly in the near term, the slow economic recovery, gridlock in Washington, and other factors mean that the path to substantial improvement in outcomes is going to be paved not with new gold, but with better use of the gold that’s already there.

No problem.

We already spend a lot of money on education. The task right now is to change how we spend federal, state, and local resources so that more money is spent on programs and practices known to make a difference rather than on investments with zero or unknown impacts on learning. Here are my top six suggestions for how to spend our education resources more effectively. (I’ll go into more details on these in future blogs).

1. Provide incentives for schools and districts to implement programs with strong evidence of effectiveness in competitive grants. In competitive grants in all parts of federal and state government, offer “competitive preference points” for applicants who promise to adopt and effectively implement programs proven to be effective. For example, schools proposing to implement programs identified as having “strong evidence of effectiveness” under the new EDGAR definitions might receive four extra points on a 100-point scale, while those meeting the criteria for “moderate evidence of effectiveness” might receive two points. Readers of this blog have seen me make this recommendation many times. Perfect example: School Improvement Grants for low-achieving schools. Cost: zero.

2. Provide incentives for schools and districts to implement programs with strong evidence of effectiveness in formula grants. The big money in federal and state education funding is in formula grants that go to districts and schools based on, for example, levels of poverty, rather than competitive applications. The classic example is Title I. Schools have great freedom in how they use these funds, so how can they be encouraged to use them in more effective ways? The answer is to provide additional incentive funding if schools or districts commit to using proven programs with their allotted formula funds. For example, if schools agree to use a portion of their (formula-driven) Title I funds on a proven program, they may qualify for additional funds (not from the formula pot). This was the idea behind the Obey-Porter Comprehensive School Reform initiative of the late 1990s, which encouraged thousands of Title I schools to adopt whole-school reform models. Cost: This strategy could be done at a cost of perhaps 1% of the current $15 billion annual Title I budget.

3. Offer commitment to proven programs as an alternative to use of value-added teacher evaluation models. A central part of the current administration’s policies is incentivizing states and districts to adopt teacher evaluation plans that combine principal ratings of teachers with value-added scores based on students’ state reading and math tests. This is a required part of Race to the Top in those states that received this funding, and it is a required element of state applications for a waiver of elements of No Child Left Behind.

In practice, current teacher evaluation policies are intended to do two things. First, they insist that schools identify extremely ineffective teachers and help them find other futures. If done fairly and consistently, few oppose this aspect of teacher evaluation. Principals have evaluated teachers and identified those with serious deficits forever, and I am not arguing against continuing this type of evaluation.

The second purpose of the teacher evaluation policies is to improve teaching and learning for all teachers. This is the expensive and contentious part of the policies; in most states it requires a combination of frequent, structured observation by principals and “value-added” assessments of a given teacher’s students. The technical difficulties of both are substantial, and no study has yet shown any benefit to student learning as a result of going through the whole ordeal.

If the goal is better teaching and learning, why not require that all reform approaches meet the same evidence standards? If a school proposes to use a schoolwide strategy that (unlike current teacher evaluation policies) has strong evidence of effectiveness, the school should be permitted, even encouraged, to suspend aspects of the new model as long as it is implementing proven alternatives with fidelity and good outcomes. Cost: Modest, assuming proven programs are similar in cost to the expensive new teacher evaluation strategies.

4. Train and equip paraprofessionals as tutors. The most common expenditure of Title I funds is on paraprofessionals or aides, educators who do not usually have teaching degrees but perform all sorts of functions within schools other than class teaching. Paraprofessionals can be wonderful and capable people, but evidence in the U.S. and U.K. consistently finds that as they are most commonly used, they make little difference in student learning.

Yet there is also extensive evidence that paraprofessionals can be very effective if 
they are trained to provide well-structured one-to-one or one-to-small group tutoring to students who are struggling in reading and math
. Paraprofessionals are a multi-billion dollar army eager and capable of making more of a difference. Let’s empower them to do so. Cost: Minimal (just training and materials).

5. Encourage schools to use Supplemental Educational Services (SES) funding on proven programs. As part of No Child Left Behind, Title I schools had to use substantial portions of their Title I dollars to provide Supplemental Educational Services (SES) to students in schools failing to meet standards. Study after study has found SES to be ineffective, and expenditures on SES are waning, yet they remain as a significant element of Title I funding, even in states with waivers. If districts could be encouraged to use SES funds on programs with evidence of effectiveness in improving achievement (such as training paraprofessionals and teachers to be tutors in reading and/or math), outcomes are sure to improve. Cost: Minimal.

6. Invest in research and development to identify effective uses of universal access to tablets or computers. Despite economic and political hard times, schools everywhere are moving rapidly toward providing universal, all-student access to tablets or computers. There is a lot of talk about blended learning, flipped learning, and so on, but little actual research and development is going on that is likely to identify effective and replicable classroom strategies likely to make good use of these powerful tools. As it has done many times before, American education is about to spend billions on technology without first knowing which applications actually work. Setting aside a tiny percentage of the costs of the hardware and software, we could fund many innovators to create and rigorously evaluate approaches using all-student technology access, before we get stuck on ineffective solutions (again). Cost: modest.

* * *

I’m sure there are many more ways we could shift existing funds to advance
American education, but they all come down to one common recommendation: use what works. Collectively, the six strategies I’ve outlined, and others like them, could catapult American education to the top on international comparisons, greatly reduce education gaps, and prepare our students for the demands of a technological economy, all at little or no net cost, if we’re willing to also stop making ineffective investments. Moreover, all of these six prescriptions could be substantially underway in the next two years, during the remainder of the current administration. All could be done by the Department of Education alone, without congressional action. And again, I’m sure that others have many other examples of low-cost and no-cost solutions that I haven’t thought of or haven’t addressed here.

A revolution in American education does not necessarily require money, but it does require courage, leadership, and resolve. Those are resources our nation has in abundance. Let’s put them to work.

More OMG From OMB

Imagine that education leaders began to encourage or provide incentives for schools to use proven programs and practices. Imagine that instead of a confused patchwork of policies and grants, government had a simple rule: if it works, we’ll help you adopt it. If it hasn’t yet been proven to work, we’ll help you evaluate it. If it’s just a good idea, we’ll help you move it forward. But the purpose of education policy is to find out what works and then help scale it up.

In a speech this summer, Robert Gordon from the OMB laid out such a vision. He was among friends, speaking to recipients of Investing in Innovation (i3) grants designed to build up our nation’s “shelf” of proven programs. He spoke about OMB’s memo to all government departments encouraging policies to favor use of proven programs, which I’ve written about before.

Gordon’s speech provided an opportunity to reflect on how far evidence-based reform has come in recent years. No previous leader from OMB has ever shown any interest in educational research and development, although Robert Shay, OMB Director under President George W. Bush, has recently expressed support for investing in programs with evidence of effectiveness from rigorous evaluations. Until fairly recently, research in education was mostly done by and for academics, and no one expected it to make much of a difference. Today, as I noted in a recent blog, we have the distinguished Lisbeth Schorr worrying that evidence from rigorous experiments might soon make too much of a difference. I think her concern is premature, but isn’t it cool to even have such a conversation?

Don’t forget to follow me on Facebook and Twitter!