Seeds, Bricks, and Sand: Stages of School-Reform Readiness

2015-01-15-HP57Seeds_Sands.jpg

Every school, no matter how effective at improving student outcomes, could probably be even more effective, and some schools have a particularly long way to go. Various proven reform models for whole schools, particular subjects, or specific purposes stand ready to help all of these schools improve. Yet schools vary a great deal in terms of readiness for particular approaches to reform.

A metaphor for three types of schools in terms of readiness for reform is seeds, bricks, sand. The “seeds” metaphor implies an environment so conducive to reform that anything can grow there. The staff and leadership of the school are capable, aware of research, participating in professional development, well-coordinated, cohesive, and unafraid of change. Such a school may be able to create and evaluate its own reform methods and sustain and improve them over time, perhaps with general advice from consultants. “Bricks” schools are also positively oriented toward change, but are unlikely to invent effective reforms themselves. Such schools have committed and hard-working teachers and leaders who have not had the time or resources to become reform experts themselves, but are welcoming to proven models. The “bricks” metaphor implies that if someone brings the bricks and a set of plans to the site, a durable edifice can be built and maintained.

A “sand” school, on the other hand, is one that is not ready for reform, and building on this site is like building a sand castle, which will wash away with the next tide. In such schools the staff and leadership may be at odds with each other, may not believe that children can learn any more than they do now, or may have experienced failure with previous reforms. These schools may need serious restructuring.

The usefulness of the “seeds-bricks-sand” categories is in understanding how to help schools adopt and sustain proven programs. The great majority of Title I schools, in my experience, are “bricks” schools, ready, willing, and able to implement well-defined, research-proven programs, but unlikely to have the inclination to invent their own school-wide approach. Others are clearly in the “sand” category. Yet Title I schools in trouble are frequently given “seeds” advice. Until now, schools receiving substantial funding under the current School Improvement Grants (SIG) have been routinely given consultants to help them work out their own school-wide reform designs, rather than being helped to adopt proven programs. There are schools that can benefit from such strategies, but they are rarely the ones that are persistently low achieving, as all SIG schools are. Recent proposed regulations would offer SIG schools the option of adopting proven, whole-school reform models, a welcome and long overdue change.

Use of proven, well-structured reforms needs to be expanded in all schools. For those in the greatest difficulty, this need is urgent. The new SIG regulations would allow schools capable of implementing (but not inventing) proven, effective reforms a chance to turn themselves around.

Whole-school reform is difficult and expensive, and when it fails, the consequences for children as well as educators can be dire and long-lasting. Failed initiatives not only waste money, but they undermine the belief that high-poverty schools can be saved. We need to get smarter about targeting interventions to specific types of schools to increase the likelihood that investments in reform truly pay off for kids and for our society.

Advertisements

Are Proven Educational Innovations Ready for Prime Time?

These are good times for evidence-based reform in education. Due in particular to Investing in Innovation (i3) and the Institute of Education Sciences (IES), the number of proven programs in all subjects and grade levels in increasing, and as i3 programs come to the ends of their evaluations, the number of proven programs should accelerate (even though there are also sure to be many disappointments).

The increasing numbers of programs proven effective in rigorous research creates new opportunities for policy and practice in education. Already, School Improvement Grants (SIG) are newly introducing an option for schools to choose proven, comprehensive reform models. Other areas of policy may also soon begin to encourage or incentivize use of programs with strong evidence.

If these changes in policy begin to happen, it will begin to matter whether educational programs proven to be effective are in fact ready for prime time, meaning that they are ready to be disseminated and supported in the form they existed when they were successfully evaluated. It would be catastrophic if educators and policy makers began looking on the What Works Clearinghouse, for example, or looking for programs that meet the EDGAR standards for strong or moderate evidence of effectiveness, and found many program that were unavailable, or unrealistic, or impractical.

Much as providing evidence of effectiveness is an advance in education, there is a real need for a determination of the degree to which programs are also ready for widespread implementation.

Some indicators of readiness for prime time would be easy to assess. For example, programs that lack a web site, do not offer materials or training, or otherwise do not exist in anything like the form in which they were evaluated cannot be considered ready for implementation. Some programs used procedures in their evaluation that could never be replicated, such as science programs that provide each experimental class with enough graduate students to monitor and assist every lab group. Some proven technology products run on hardware that no longer exists.

Many studies use measures of learning outcomes that are closely aligned with what was taught in the experimental but not the control group. Such studies might be excluded on the basis that the overaligned measure does not have meaning beyond the experiment itself.

Educators who choose to use proven programs have a right to be confident that the programs they have selected are, if implemented well, likely to result in enhanced performance on measures they care about. Finding a lot of programs that cannot be implemented under ordinary circumstances and with meaningful measures will diminish interest in evidence-based reform.

Evidence-based reform itself is ready for prime time in education, but its future depends on whether it is perceived to produce genuine benefits for children. We need to make sure that the proven programs we offer to educators meet their needs and those of the students, not just scientific standards.

Thank You, Jim Shelton

2014-10-16-HP5010_16_2014.jpg

Everyone who works to advance evidence-based reform in education was saddened to learn that Jim Shelton will be leaving the U.S. Department of Education. Jim is currently the Department’s Deputy Secretary, and before that he was the Assistant Deputy Secretary for the Office of Innovation and Improvement (OII), the home of the Investing in Innovation (i3) program. Under his leadership, i3 created a unique approach to speeding up innovation in education, supporting development of new programs, evaluation of promising programs, and scale-up of already proven programs. Jim has provided the intellectual leadership for i3, understanding better than anyone the process by which proven and replicable programs could gradually infuse government-funded programs in education, starting with competitive programs (such as School Improvement Grants) and then moving on to formula programs (such as Title I).

Jim has been a tireless advocate for evidence, but even more so for kids. He understands that the children who need the best schools need the best programs. Better teachers? Smaller classes? Better parent support? Sure, they need these too, but until core classroom practices are better in every class, millions of children will continue to fail each year.

Jim is a powerful speaker and a powerful advocate for his ideas. I don’t know where he’s going next, but I’m certain that wherever it is, he will carry on the fight for children and for evidence so that the programs we provide children actually work.

Evidence to the Rescue! Whole School Option for SIG

2014-09-11-HP48.jpg

The U. S. Department of Education has just released draft regulations that will substantially change School Improvement Grants (SIG). Within the document are two pages that could change the world. Really.

Up to now, the very low achieving schools that qualified for SIG funding had to choose among four options. Two of these, school closure and restart, are rarely used. Turnaround requires firing the principal and at least 50% of the staff. So most SIG schools have opted for transformation, which still requires firing the principal in most cases and implementing a lengthy list of changes.

The new regs keep these four options, with a few tweaks, and they add three more. One is a “state-determined” model to be devised by state education departments, which requires approval by the U.S. Secretary of Education, and another emphasizes pre-kindergarten programs as part of elementary school reform. These are good ideas, but not revolutionary.

The revolutionary option is an “evidence-based whole-school reform option.” Schools that choose it have to implement whole-school reform models created and disseminated by external organizations. That’s good, but not quite revolutionary in itself.

Here comes the revolutionary part. Be sure you’re sitting down before reading on.

The whole-school reform models SIG schools may choose must have a high level of evidence of effectiveness. At least two studies that meet tough What Works Clearinghouse (WWC) standards must have documented positive effects on major learning outcomes. The Department will review programs in advance and will determine if they meet the standards. Full disclosure: Our Success for All program will most likely qualify. But this is a lot bigger than a good thing for Success for All.

This is the first time in all of education history, going back to Socrates, when evidence of effectiveness has been a requirement for anything in practice. Yes, the equally revolutionary Investing in Innovation (i3) program requires evidence of effectiveness for its large validation and scale-up grants, and the scale-up grants, in particular, have collectively benefitted thousands of schools. However, the focus of i3 is on building capacity among developers, not on direct service. In contrast, the SIG whole-school option moves evidence-based reform into a new arena. For the first time, a major federal program that does provide direct service to schools is offering them an option that requires substantial evidence of effectiveness.

This gives schools that apply for and receive SIG funding a new set of possibilities to take on a proven whole-school strategy without having to fire their principals or others. Assuming whole-school models were proven in schools like theirs, and assuming the organizations that made and evaluated them have a lot of experience working with similar schools, their students are likely to benefit, and that’s all good.

If all goes well, the whole-school option will demonstrate the power of evidence-based reform. Proven models will show well in evaluations of SIG, because their capacity to show positive outcomes is what got them on the list. If this happens, one might expect that other parts of federal, state, and local education would also begin demanding evidence of effectiveness for innovative programs. Seeing this, perhaps government and private funders will ramp up their investments in research and development, to create and validate additional models for a wide variety of purposes. Imagine, dare I say it, that Title I might begin to encourage schools to use proven approaches. Or Title II. Or IDEA. Or technology applications. Or STEM. Or. . . just about any area of education in which outcomes can be measured.

If we start with the SIG whole-school option, the evidence-based reform movement could get completely out of hand. Wouldn’t that be great?

Six Low-Cost or Free Ways to Make American Education the Best in the World

2014-06-12-HP38Image_06_12_2014.jpg

It does not take a political genius to know that for the foreseeable future, American education is not going to be rescued by a grand influx of new money. Certainly in the near term, the slow economic recovery, gridlock in Washington, and other factors mean that the path to substantial improvement in outcomes is going to be paved not with new gold, but with better use of the gold that’s already there.

No problem.

We already spend a lot of money on education. The task right now is to change how we spend federal, state, and local resources so that more money is spent on programs and practices known to make a difference rather than on investments with zero or unknown impacts on learning. Here are my top six suggestions for how to spend our education resources more effectively. (I’ll go into more details on these in future blogs).

1. Provide incentives for schools and districts to implement programs with strong evidence of effectiveness in competitive grants. In competitive grants in all parts of federal and state government, offer “competitive preference points” for applicants who promise to adopt and effectively implement programs proven to be effective. For example, schools proposing to implement programs identified as having “strong evidence of effectiveness” under the new EDGAR definitions might receive four extra points on a 100-point scale, while those meeting the criteria for “moderate evidence of effectiveness” might receive two points. Readers of this blog have seen me make this recommendation many times. Perfect example: School Improvement Grants for low-achieving schools. Cost: zero.

2. Provide incentives for schools and districts to implement programs with strong evidence of effectiveness in formula grants. The big money in federal and state education funding is in formula grants that go to districts and schools based on, for example, levels of poverty, rather than competitive applications. The classic example is Title I. Schools have great freedom in how they use these funds, so how can they be encouraged to use them in more effective ways? The answer is to provide additional incentive funding if schools or districts commit to using proven programs with their allotted formula funds. For example, if schools agree to use a portion of their (formula-driven) Title I funds on a proven program, they may qualify for additional funds (not from the formula pot). This was the idea behind the Obey-Porter Comprehensive School Reform initiative of the late 1990s, which encouraged thousands of Title I schools to adopt whole-school reform models. Cost: This strategy could be done at a cost of perhaps 1% of the current $15 billion annual Title I budget.

3. Offer commitment to proven programs as an alternative to use of value-added teacher evaluation models. A central part of the current administration’s policies is incentivizing states and districts to adopt teacher evaluation plans that combine principal ratings of teachers with value-added scores based on students’ state reading and math tests. This is a required part of Race to the Top in those states that received this funding, and it is a required element of state applications for a waiver of elements of No Child Left Behind.

In practice, current teacher evaluation policies are intended to do two things. First, they insist that schools identify extremely ineffective teachers and help them find other futures. If done fairly and consistently, few oppose this aspect of teacher evaluation. Principals have evaluated teachers and identified those with serious deficits forever, and I am not arguing against continuing this type of evaluation.

The second purpose of the teacher evaluation policies is to improve teaching and learning for all teachers. This is the expensive and contentious part of the policies; in most states it requires a combination of frequent, structured observation by principals and “value-added” assessments of a given teacher’s students. The technical difficulties of both are substantial, and no study has yet shown any benefit to student learning as a result of going through the whole ordeal.

If the goal is better teaching and learning, why not require that all reform approaches meet the same evidence standards? If a school proposes to use a schoolwide strategy that (unlike current teacher evaluation policies) has strong evidence of effectiveness, the school should be permitted, even encouraged, to suspend aspects of the new model as long as it is implementing proven alternatives with fidelity and good outcomes. Cost: Modest, assuming proven programs are similar in cost to the expensive new teacher evaluation strategies.

4. Train and equip paraprofessionals as tutors. The most common expenditure of Title I funds is on paraprofessionals or aides, educators who do not usually have teaching degrees but perform all sorts of functions within schools other than class teaching. Paraprofessionals can be wonderful and capable people, but evidence in the U.S. and U.K. consistently finds that as they are most commonly used, they make little difference in student learning.

Yet there is also extensive evidence that paraprofessionals can be very effective if 
they are trained to provide well-structured one-to-one or one-to-small group tutoring to students who are struggling in reading and math. Paraprofessionals are a multi-billion dollar army eager and capable of making more of a difference. Let’s empower them to do so. Cost: Minimal (just training and materials).

5. Encourage schools to use Supplemental Educational Services (SES) funding on proven programs. As part of No Child Left Behind, Title I schools had to use substantial portions of their Title I dollars to provide Supplemental Educational Services (SES) to students in schools failing to meet standards. Study after study has found SES to be ineffective, and expenditures on SES are waning, yet they remain as a significant element of Title I funding, even in states with waivers. If districts could be encouraged to use SES funds on programs with evidence of effectiveness in improving achievement (such as training paraprofessionals and teachers to be tutors in reading and/or math), outcomes are sure to improve. Cost: Minimal.

6. Invest in research and development to identify effective uses of universal access to tablets or computers. Despite economic and political hard times, schools everywhere are moving rapidly toward providing universal, all-student access to tablets or computers. There is a lot of talk about blended learning, flipped learning, and so on, but little actual research and development is going on that is likely to identify effective and replicable classroom strategies likely to make good use of these powerful tools. As it has done many times before, American education is about to spend billions on technology without first knowing which applications actually work. Setting aside a tiny percentage of the costs of the hardware and software, we could fund many innovators to create and rigorously evaluate approaches using all-student technology access, before we get stuck on ineffective solutions (again). Cost: modest.

* * *

I’m sure there are many more ways we could shift existing funds to advance
American education, but they all come down to one common recommendation: use what works. Collectively, the six strategies I’ve outlined, and others like them, could catapult American education to the top on international comparisons, greatly reduce education gaps, and prepare our students for the demands of a technological economy, all at little or no net cost, if we’re willing to also stop making ineffective investments. Moreover, all of these six prescriptions could be substantially underway in the next two years, during the remainder of the current administration. All could be done by the Department of Education alone, without congressional action. And again, I’m sure that others have many other examples of low-cost and no-cost solutions that I haven’t thought of or haven’t addressed here.

A revolution in American education does not necessarily require money, but it does require courage, leadership, and resolve. Those are resources our nation has in abundance. Let’s put them to work.

School Improvement Grants Embrace Evidence of Effectiveness

2014-02-20-HPImage022014.jpg

Despite all of the exciting gains made by evidence-based reform in recent years, all of the progress so far has been limited to development, evaluation, and scale-up of proven programs outside of mainline education policy or funding. Title I, Title II, Race to the Top, School Improvement Grants, and other large, influential funding sources for reform have hardly been touched by the growth of proven, replicable programs sponsored primarily by the Institute of Education Sciences and Investing in Innovation (i3). Until the evidence movement crosses over from R&D to the real world of policy and practice, it will remain the domain of academics and policy wonks, not a real force for change.

In the recently passed Omnibus budget, however, appears a first modest step over the R&D/policy divide. This is a new provision in congressional authorization of School Improvement Grants (SIG). Up until now, SIG schools (ones that have suffered from very low achievement levels for many years) had to choose among four models, all of which require major changes in staffing. Each SIG school is expected to develop its own model of reform, usually with the help of consultants. The problem has been that each of the hundreds of schools receiving (substantial) SIG funding has to create its own never-before-tested path to reform, and then try to implement it with quality in a school that has just experienced a substantial turnover of its leadership and staff.

The “Fifth Option” recently introduced by Congress adds a new alternative. SIG schools can choose to adopt a “proven whole-school reform model” that meets at least a moderate level of evidence support, which includes having been tested against a control group in at least one rigorous experiment. The fifth option will let schools keep their leaders and staffs, but adopt a schoolwide approach that has been used in many similar schools and found to be effective.

The Omnibus bill was passed too late in the year to apply this fifth option to the 2014-2015 school year, and the U. S. Department of Education, as well as individual states, have a lot of work to do to prepare new regulations and supports for schools applying for SIG funds under this new option in 2015-2016.

However, the fifth option makes an important statement that has not been made previously. In a major school improvement (not R&D) funding program, the fifth option says “use what works.” Wisely, it does not mandate the use of any specific programs, but by highlighting evidence-proven approaches, it puts the government behind the idea that federal funding should whenever possible be used to help educators use programs with strong evidence of effectiveness. This could be the start of something beautiful.

How To Do Lots of High-Quality Educational Evaluations for Peanuts

2013-09-19-Peanuts09192013.jpg

One of the greatest impediments to evidence-based reform in education is the difficulty and expense of doing large-scale randomized experiments. These are essential for several reasons. Large-scale experiments are important because when treatments are at the level of classrooms and schools, you need a lot of classrooms and schools to avoid having just a few unusual sites influence the results too much. Also, research finds that small-scale studies produce inflated effects, particularly because researchers can create special conditions on a small scale that they could not sustain on a large scale. Large experiments simulate the situations likely to exist when programs are used in the real world, not under optimal, hothouse conditions.

Randomized experiments are important because when schools or classrooms are assigned at random to use a program or continue to serve as a control group (doing what they were doing before), we can be confident that there are no special factors that favor the experimental group other than the program itself. Non-randomized, matched studies that are well designed can also be valid, but they have more potential for bias.

Most quantitative researchers would agree that large-scale randomized studies are preferable, but in the real world such studies done well can cost a lot – more than $10 million per study in some cases. That may be chump change in medicine, but in education, we can’t afford many such studies.

How could we do high-quality studies far less expensively? The answer is to attach studies to funding being offered by the U. S. Department of Education. That is, when the Department is about to hand out a lot of money, it should commission large-scale randomized studies to evaluate specific ways of spending these resources.

To understand what I’m proposing, consider what the Department might have done when No Child Left Behind (NCLB) required that low-performing schools offer after-school tutoring to low-achieving students, in its Supplemental Educational Services (SES) initiative. The Department might have invited proposals from established providers of tutoring services, which would have had to participate in research as a condition of special funding. It might have then chosen a set of after-school tutoring providers (I’m making these up):

Program A provides structured one-to-three tutoring.
Program B rotates children through computer, small-group, and individualized activities.
Program C provides computer-assisted instruction.
Program D offers small-group tutoring in which children who make progress get treats or free time for sports.

Now imagine that for each program, 60 qualifying schools were recruited for the studies. For the Program A study, half get Program A and half get the same funding to do whatever they wanted to do (except Programs A to D) consistent with the national rules. The assignment to Program A or its control group would be at random. Program B, C, and D would be evaluated in the same way.

Here’s why such a study would have cost peanuts. The costs of offering the program to the schools that got Programs A, B, C, or D would have been covered by Title I, as was true of all NCLB after-school tutoring programs. Further, state achievement tests, routinely collected in every state in grades 3-8, could have been obtained at pre- and posttest at little cost for data collection. The only costs would be for data management, analysis, and reporting, plus some amount of questionnaires and/or observations to see what was actually happening in the participating classes. Peanuts.

Any time money is going out from the Department, such designs might be used. For example, in recent years a lot of money has been spent on School Improvement Grants (SIG), now called School Turnaround Grants. Imagine that various whole-school reform models were invited to work with many of the very low-achieving schools that received SIG grants. Schools would have been assigned at random to use Programs A, B, C, or D, or to control groups able to use the same amount of money however they wished. Again, various models could be evaluated. The costs of implementing the programs would have been provided by SIG (which was going to spend this money anyway), and the cost of data collection would have been minimal because test scores and graduation rates already being collected could have been used. Again, the costs of this evaluation would have just involved data management, analysis, and reporting. More peanuts.

Note that in such evaluations, no school gets nothing. All of them get the money. Only schools that want to sign up for the studies would be randomly assigned. Modest incentives might be necessary to get schools to participate in the research, such as a few competitive preference points in competitive proposals (such as SIG) or somewhat higher funding levels in formula grants (such as after-school tutoring). Schools that do not want to participate in the research could do what they would have done if the study had never existed.

Against the minimal cost, however, weigh the potential gain. Each U. S. Department of Education program that lends itself to this type of evaluation would produce information about how the funds could best be used. Over time, not only would we learn about specific effective programs, we’d also learn about types of programs most likely to work. Also, additional promising programs could enter into the evaluation over time, ultimately expanding the range of options for schools. Funding from the Institute of Education Sciences (IES) or Investing in Innovation (i3) might be used specifically to build up the set of promising programs for use in such federal programs and evaluations.

Ideally, the Department might continuously commission evaluations of this kind alongside any funding it provides for schools to adopt programs capable of being evaluated on existing measures. Perhaps the Department might designate an evaluation expert to sit in on early meetings to identify such opportunities, or perhaps it might fund an external “Center for Cost-Effective Evaluations in Education.”

There are many circumstances in which expensive evaluations of promising programs still need to be done, but constantly doing inexpensive studies where they are feasible might free up resources to do necessarily expensive research and development. It might also accelerate the trend toward evidence-based reform by adding a lot of evidence quickly to support (or not) programs of immediate importance to educators, to government, and to taxpayers.

Because of the central role government plays in education, and because government routinely collects a lot of data on student achievement, we could be learning a lot more from government initiatives and innovative programs. For just a little more investment, we could learn a lot about how to make the billions we spend on providing educational services a lot more effective. Very important peanuts, if you ask me.