Nevada Places Its Bets on Evidence

blog_3-29-18_HooverDam_500x375In Nevada, known as the land of big bets, taking risks is what they do. The Nevada State Department of Education (NDE) is showing this in its approach to ESSA evidence standards .  Of course, many states are planning policies to encourage use of programs that meet the ESSA evidence standards, but to my knowledge, no state department of education has taken as proactive a stance in this direction as Nevada.

 

Under the leadership of their state superintendent, Steve Canavero, Deputy Superintendent Brett Barley, and Director of the Office of Student and School Supports Seng-Dao Keo, Nevada has taken a strong stand: Evidence is essential for our schools, they maintain, because our kids deserve the best programs we can give them.

All states are asked by ESSA to require strong, moderate, or promising programs (defined in the law) for low-achieving schools seeking school improvement funding. Nevada has made it clear to its local districts that it will enforce the federal definitions rigorously, and only approve school improvement funding for schools proposing to implement proven programs appropriate to their needs. The federal ESSA law also provides bonus points on various other applications for federal funding, and Nevada will support these provisions as well.

However, Nevada will go beyond these policies, reasoning that if evidence from rigorous evaluations is good for federal funding, why shouldn’t it be good for state funding too? For example, Nevada will require ESSA-type evidence for its own funding program for very high-poverty schools, and for schools serving many English learners. The state has a reading-by-third-grade initiative that will also require use of programs proven to be effective under the ESSA regulations. For all of the discretionary programs offered by the state, NDE will create lists of ESSA-proven supplementary programs in each area in which evidence exists.

Nevada has even taken on the holy grail: Textbook adoption. It is not politically possible for the state to require that textbooks have rigorous evidence of effectiveness to be considered state approved. As in the past, texts will be state adopted if they align with state standards. However, on the state list of aligned programs, two key pieces of information will be added: the ESSA evidence level and the average effect size. Districts will not be required to take this information into account, but by listing it on the state adoption lists the state leaders hope to alert district leaders to pay attention to the evidence in making their selections of textbooks.

The Nevada focus on evidence takes courage. NDE has been deluged with concern from districts, from vendors, and from providers of professional development services. To each, NDE has made the same response: we need to move our state toward use of programs known to work. This is worth undergoing the difficult changes to new partnerships and new materials, if it provides Nevada’s children better programs, which will translate into better achievement and a chance at a better life. Seng-Dao Keo describes the evidence movement in Nevada as a moral imperative, delivering proven programs to Nevada’s children and then working to see that they are well implemented and actually produce the outcomes Nevada expects.

Perhaps other states are making similar plans. I certainly hope so, but it is heartening to see one state, at least, willing to use the ESSA standards as they were intended to be used, as a rationale for state and local educators not just to meet federal mandates, but to move toward use of proven programs. If other states also do this, it could drive publishers, software producers, and providers of professional development to invest in innovation and rigorous evaluation of promising approaches, as it increases use of approaches known to be effective now.

NDE is not just rolling the dice and hoping for the best. It is actively educating its district and school leaders on the benefits of evidence-based reform, and helping them make wise choices. With a proper focus on assessments of needs, facilitating access to information, and assistance with ensuring high quality implementation, really promoting use of proven programs should be more like Nevada’s Hoover Dam: A sure thing.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Photo by: Michael Karavanov [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
Advertisements

Publishers and Evidence

High above the Avenue of the Americas on prime real estate in midtown Manhattan towers the 51-story McGraw-Hill building. When in New York, I always find time to go look at that building and reflect on the quixotic quest I and my colleagues are on to get educational decision makers to choose rigorously evaluated, proven programs. These programs are often made by small non-profit organizations and universities, like the ones I work in. Looking up at that mighty building in New York, I always wonder, are we fooling ourselves? Who are we to take on some of the most powerful companies in the world?

Education publishing is dominated by three giant, multi-billion dollar publishers, plus another three or four even bigger technology companies. These behemoths are not worried about us, not one bit. Instead, they are worried about each other.

From my experience, there are very good people who work in publishing and technology companies, people who genuinely hope that their products will improve learning for students. They would love to create innovative programs, evaluate them rigorously, and disseminate those found to be effective. However, big as they are, the major publishers face severe constraints in offering proven programs. Because they are in ferocious competition with each other, publishers cannot easily invest in expensive development and evaluation, or insist on extensive professional development, a crucial element of virtually all programs that have been shown to improve student achievement. Doing so would raise their costs, making them vulnerable to lower-cost competitors.

In recent years, many big publishers and technology companies have begun to commission third-party evaluations of their major textbooks, software, and other products. If the evaluations show positive outcomes, they can use this information in their marketing, and having rigorous evidence showing positive impacts helps protect them from the possibility that government might begin to favor programs, software, or other products with proven outcomes in rigorous research. This is exactly what did happen with the enactment of the ESSA evidence standards, though the impact of these standards has not yet been strongly felt.

However, publishers and technology companies cannot get too far out ahead of their market. If superintendents, central office leaders, and others who select textbooks and technology get on board the evidence train, then publishers will greatly expand their efforts in research and development. If the market continues to place little value on evidence, so will the big publishers.

In contrast to commercial publishers and technology companies, non-profit organizations play a disproportionate role in the evidence movement. They are often funded by government or philanthropies to create and evaluate innovations, as big commercial companies almost never are. Non-profits have the freedom to experiment, and to disseminate what works. However, non-profits, universities, and tiny for-profit start-ups are small, under-capitalized, and have little capacity or experience in marketing. Their main, and perhaps only, competitive advantage is that they have evidence of effectiveness. If no one cares about evidence, our programs will not last long.

One problem publishers face is that evaluations of traditional textbooks usually do not show any achievement benefits compared to control groups. The reason is that one publisher’s textbook is just not that different from another’s, which is what the control group is using. Publishers rarely provide much professional development, which makes it difficult for them to introduce anything truly innovative. The half-day August in-service that comes with most textbooks is barely enough to get teachers familiar with the features in the most traditional book. The same is true of technology approaches, which also rarely make much difference in student outcomes, perhaps because they typically provide little professional development beyond what is necessary to run the software.

The strategy emphasized by government and philanthropy for many years has been to fund innovators to create and evaluate programs. Those that succeed are then encouraged or funded to “scale up” their proven programs. Some are able to grow to impressive scale, but never so much as to worry big companies. An occasional David can surprise an occasional Goliath, but in the long run, the big guys win, and they’ll keep winning until someone changes the rules. To oversimplify a bit, what we have are massive publishers and technology companies with few proven innovations, and small non-profits with proven programs but little money or marketing expertise. This is not a recipe for progress.

The solution lays with government. National, state, and/or local governments have to adopt policies that favor the use of programs and software that have been proven in rigorous experiments to be effective in improving student achievement. At the federal level, the ESSA evidence standards are showing the way, and if they truly catch hold, this may be enough. But imagine if a few large states or even big districts started announcing that they were henceforth going to require evidence of effectiveness when they adopt programs and software. The effect could be electric.

For non-profits, such policies could greatly expand access to schools, and perhaps to funding. But most non-profits are so small that it would take them years to scale up substantially while maintaining quality and effectiveness.

For publishers and technology companies, the effect could be even more dramatic. If effectiveness begins to matter, even if just in a few key places, then it becomes worthwhile for them to create, partner with, or acquire effective innovations that provide sufficient professional development. In states and districts with pro-evidence policies, publishers would not have to worry about being undercut by competitors, because all vendors would have to meet evidence standards.

Publishers have tried to acquire proven programs in the past, but this usually comes to smash, because they tend to strip out the professional development and other elements that made the program work in the first place. However, in a pro-evidence environment, publishers would be motivated to maintain the quality, effectiveness, and “brand” of any programs they acquire.

In medicine, most research on practical medications is funded by drug companies and carefully monitored and certified by government. Could such a thing happen in education?

Publishers and technology companies have the capital and expertise to take effective programs to scale. Partnering with creators of proven programs, or creating and evaluating their own, big companies can make a real difference, as long as government ensures that the programs they are disseminating are in fact of the same quality and effectiveness as the versions that were found to be effective.

Publishers and technology companies are a key part of the education landscape. They need to be welcomed into evidence-based reform, and incentivized to engage in innovation and evaluation. Otherwise, educational innovation will remain a marginal activity, benefitting thousands of students when millions are in need.

This blog is sponsored by the Laura and John Arnold Foundation

‘We Don’t Do Programs’

When speaking with educational leaders, I frequently hear them say, “We don’t do programs.” They lament that teachers and principals are often too driven by programs, and when outcomes are not what they would like, they drop their program and bring in another, learning little in the process.

I can sympathize with the sentiment. The sad secret known to quantitative researchers is that most programs don’t make any difference in achievement. In particular, commercial textbooks almost never make a difference. It’s not that textbooks are worthless, but they are usually so similar to each other that few textbooks produce better outcomes than any others. So the experience of school and district leaders is a cycle of adopting new textbooks, implementing them with enthusiasm, and then gradually being disappointed in the outcomes. The schools and districts are then stuck with the old texts until the books wear out, and then the whole cycle starts again.

A very similar cycle of adoption, enthusiasm, frustration, and abandonment, also happens with technology, though it may happen faster.

Another part of the negative experience with programs that many school leaders share is an observation that educators often believe that they are on the right track just because they have adopted a new program. This may be particularly true if the school or district has adopted a program that is seen as innovative or “in,” such as the technology of the moment.

What’s left out of the “we don’t do programs” conversation is, of course, evidence. Would people who say “We don’t do programs” also say “we don’t do effective programs?” I certainly hope not. Yet the categorical rejection of all programs makes no sense, so I’m going to assume that since most educators are sensible people, those who say “we don’t do programs” must just not be aware that there are in fact effective and ineffective programs.

Of course, simply adopting a program is not a guarantee of positive outcomes. Programs must be implemented with fidelity, thoughtfulness, and appropriate adaptations to local needs and resources. What evidence of effectiveness provides is not certainty, but rather a valid reason to believe that if teachers and principals put in the time, effort, and resources to implement a program well, outcomes will be positive.

If you follow my blogs, you are aware that there are many proven programs and many programs that lack evidence of effectiveness. I’ll consider my life goal to have been achieved when I start hearing educational leaders saying “we don’t do ineffective programs. We do effective ones.” Or more succinctly, “Show me the evidence!”

Do Textbooks Matter?

Recently, some colleagues and I were speaking with some superintendents about how they use evidence to select educational programs. Although they had many useful insights, it quickly became clear that when we said programs, they thought we meant textbooks.

But a textbook is not a program.

A program is a set of coordinated strategies designed to improve student achievement. A hallmark of programs is that they almost invariably include a lot of professional development. Textbooks almost invariably do not. A half day inservice is typical of textbooks, while programs generally provide many days of inservice, plus on-site coaching and feedback, on line or in-school discussions, and so on. Programs may also include textbooks or other curriculum or software, but they are focused on changing teachers’ behaviors in the classroom, not just changing content.

Content is important, of course, but changing textbooks almost never changes outcomes on achievement tests. My colleagues and I have published reviews of research on elementary and secondary reading, math, and science. In every one of these reviews, changing textbooks is one category of interventions that has been studied, often in very large, randomized experiments. Yet textbooks never make much of a difference on average, and it is rare that they show significant differences in even a single qualifying study. These studies usually use standardized tests as the outcome measures, and a focus of many textbook innovations is on closer alignment with current standards and assessments. Yet that strategy has been tried and evaluated many times, and it almost never works.

What does work, in contrast, are programs, ones that provide a great deal of professional development on well-defined models of teaching, such as cooperative learning and teaching of metacognitive skills.

Not every study of professional development approaches shows increases in achievement, and there are other factors that underlie more and less effective innovations. But on average, the difference between professional development and textbook approaches is crystal clear, and applies to all subjects and grade levels.

So when your textbooks are worn out, or you are tired of them, go ahead and replace them with a shiny new textbook or digital textbook. It won’t make any difference in students’ learning, but no one wants students to have shabby or outdated material. But when you decide to do something to improve student learning, do not follow your textbook adoption cycle. Instead, find proven programs with outstanding and sufficient professional development. Your kids, parents, and colleagues will be glad you did.

Sunset for Textbook Adoption in California

There is great news from California. The State Board of Education has announced that it will drop its longstanding textbook adoption standards, which for many years have only allowed California schools to use state textbook funds on a limited set of choices. Many states have approved lists of textbooks, but California was relatively unique in limiting options to a very short list. For example, in recent years, California allowed only two basal reading series, Open Court and Houghton Mifflin.

For many years, California and the other large textbook adoption state, Texas, have had an outsized impact on textbooks everywhere, because publishers have had to dance to their tune. This has had a negative effect nationally. It has restricted innovation, and kept smaller competitors out of the textbook market, since it is hugely expensive to go through the adoption process.

For an advocate of evidence-based reform, the California and Texas textbook adoption policies are particularly galling. While California had rules against mentioning junk food in texts, for example, no textbook adoption state has ever paid any attention to the evidence base supporting the use of particular texts. This meant that in California, districts could use their textbook funds on texts found in high-quality research to be ineffective, or ones that had never been evaluated, while programs with strong evidence of effectiveness were effectively banned from the state.

I often hear the objection to evidence-based reform that schools should not be required to use proven models. I happen to agree, and have always argued that the proper role of government is to provide encouragement and incentives to use proven programs, not to mandate them. Yet it strikes me as absurd that California was requiring the use of particular programs based on no evidence at all, and Texas and other textbook adoption states continue to do so.

State textbook adoption lists have got to go. They should be replaced with solid information on the outcomes of various programs, perhaps with incentive funding to choose the best-evaluated. But schools and districts should be free to use that information to make whatever choices they believe to be best for their kids, knowing that they will ultimately be held accountable for the outcomes.

Don’t forget to follow me on Facebook and Twitter.

Kiss Your Textbook Goodbye

When I was a kid, I loved my textbooks. I loved their heft, their musty smell, and the long list of names of previous users in the back. I loved the confident, definitive prose that led into new worlds of thought and experience. When I grew up, I even wrote some textbooks myself.

So it is with mixed emotions that I’m witnessing the demise of the textbook as we know it. Apple recently announced that it will be partnering with major publishers to create online textbooks, and the Obama Administration set the goal of having e-textbooks in the hands of every student by 2017, It’s only a matter of time before the textbook goes the way of the slide rule, the typewriter, and the chalkboard.

The end of the paper textbook will lessen the weight in students’ backpacks and perhaps reduce schools’ costs, but will it be beneficial to students’ learning? It’s our job to make it so. If electronic textbooks are just like paper ones, there is little reason to expect them to be more effective. But e-texts offer many possibilities for innovation. Electronic texts can be linked to videos, including tutoring or alternative and more in-depth explanations. They can provide study aids, such as outlines, summaries, self-assessments, and embedded definitions. They can connect students in online study groups to jointly prepare each other for assessments. They can continuously assess students’ understanding and prescribe either remedial work to fill gaps or offer extensions for students willing and able to go beyond the ordinary. Digital textbooks may be linked to content shown by teachers on interactive whiteboards, tablets, or other electronic devices used in classrooms to supplement teachers’ instruction. They may communicate to teachers students’ current levels of knowledge and skill so that teachers can adapt their class lessons to meet the needs of their class and identify individual students who need additional assistance.

Using video, games, online study groups, and other means, electronic homework might actually become something students want to do, perhaps even replacing some of the vast wasteland of mindless television and shoot-’em-up video gaming that currently occupy a huge proportion of children’s days.

The move toward electronic textbooks will soon require that every student has secure, reliable access to technology at home that connects to school-approved networks. When teachers can count on the idea that every student (and parent) has access to technology, the possibilities for home-school collaboration will be limitless.

Of course, all of these possibilities are just that – possibilities. When electronic textbooks become the norm, it will also be possible to rigorously evaluate each of hundreds of variations in how they are used. Even if electronic textbooks are no better than paper ones at first, e-textbooks can be rapidly and continuously improved in a way that paper textbooks never could. A hundred years ago, cars were not much better than horses, as they were expensive, difficult to maintain, and prone to breakdowns. However, it was easy to see that eventually, the car would prevail. Horses had reached their limit, while cars could be progressively improved.

Electronic textbooks will provide opportunities for researchers and developers to create exciting, astonishingly effective learning opportunities for students. They will also provide opportunities to create “killer apps” that turn out to be ineffective or even harmful. As we cross this digital bridge, the onus is on us to test the numerous applications and know which are beneficial and which are not, not just which are popular.

I, for one, will miss the old-fashioned textbook, but I welcome the great potential of its electronic successor. Now let’s make sure that this potential is realized.