Whadja Do In School Today?

Every parent of a four or five year old knows the drill. Your child comes home after pre-kindergarten or kindergarten. “Whadja do today?” you  say with eager anticipation, thinking of all the friends your child must have made, the stories your child heard, the songs your child sang, the projects or dress-up or phonics or math, or…well, anything.

“Nuffin,” your child says, wandering out of range to avoid further interrogation.

You know your child did a whole lot more than “nuffin.” But how can you find out so that you can build on what the teacher did each day?

One answer is something our group at the Success for All Foundation created utilizing Investing in Innovation (i3) funding with partners at Sesame Workshop, Sirius Thinking, and Johns Hopkins University. We call it Home Links. Home Links are 10-15 minute videos, akin to short television shows, that parents and children watch together, 4 evenings a week. Each show uses content from Sesame Street and animations we have made with Sirius Thinking, so they are a bit like Sesame Street shows themselves, with one huge difference: the content of the shows reflects the activities that children and teachers were doing that day in school.

The Home Links give kids reinforcement and extension of vocabulary and skills they learned that day, and that’s important. But more important, they tell parents what’s happening in school. When a show contains skits about fall, the letter V, counting to five, and singing traditional songs, the parents know that all of these things are happening in school. Our surveys found that 96% of the time, a parent, grandparent, or other relative watches with the child. At the end of each show there is music and movement, and parents tell us they dance with their children, and they love the closeness and fun. But parents also now know how to support their children’s learning. If the topic is markets, they know to point out interesting things when they next are at the market with their child. If the letter is T, they know to point out things that begin with T. If the math segment is on shapes, parents know to ask children about shapes they see in daily life. Home should not be another classroom, but it’s the ideal place for a child to learn that the things he or she is learning in school are important to his or her parents and exist in his or her community. It also helps children understand that knowing about and learning about those things brings pride and builds curiosity.

Home Links are sent home on DVDs each day. We are now looking for funding to make an online version so families can download Home Links to digital devices such as phones and tablets.

Right now, Home Links are being used in approximately 300 preschool and kindergarten classes already using our proven Success for All whole-school approach. In the future, we hope to disseminate Home Links to preschools and kindergartens whether or not they use Success for All.

When this happens, more and more parents won’t have to ask, “Whadja do in school today?” They’ll know. And they’ll know how to build on what they find out.

And that ain’t nuffin’.

 

The Investing in Innovation (i3) program is a federal competitive grant program at the U.S. Department of Education, within the Office of Innovation and Improvement (OII). It provides funding to support local education agencies or nonprofit organizations in partnership with LEAs and/or schools to expand and develop innovative practices that can serve as models of best practices and to identify and document best practices that can be shared and taken to scale in the areas of improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates.

More information on the i3 program can be found here.

More information on Success for All Foundation’s grant Around the Corner: A Technology-Enhanced Approach to Early Literacy can be found here.

Advertisements

Good News from i3 Projects

In ancient Sparta, when a young man finished his military training, the city elders held a ceremony in which they removed a stone from the city wall. The idea was that the defense of Sparta was in its people, not its walls.

Just a few days ago, I was privileged to witness an event in which five more programs funded by the Investing in Innovation (i3) program presented their methods and findings on Capitol Hill. Our government might have used this as an opportunity to remove some of the walls that impede progress in education. As more and more programs demonstrate that they can reliably improve student achievement in high-poverty schools, can we slowly back away from the idea that evidence has little role to play in educational policy? Not all of the programs supported by i3, the Institute of Education Sciences (IES), and other funders, are achieving immediate positive outcomes in comparison to control groups, though they may do so in the future. But based just on the programs we know to be effective, why can’t we begin to base educational policies on what works while simultaneously continuing to build up our army of proven approaches?

The findings from the first set of programs funded by i3 are starting to become available. As most readers of this blog know, i3 provides grants according to the levels of evidence programs already have. In 2010, scale-up proposals had to have strong, replicated evidence of effectiveness, and could receive up to $50 million over 5 years. Validation grants needed at least one supportive study, and could receive up to $30 million, and development grants needed only a good idea, and could receive up to $5 million.

Not surprisingly, all four of the 2010 scale-up grants found positive effects in rigorous third-party evaluations. These programs – Reading Recovery, Teach for America, KIPP, and our own Success for All – reported on their impacts in a briefing on Capitol Hill in September, 2014. This year, a new crop of validation and development grantees had good news to share, and did so in a Capitol Hill briefing on October 29. Sarah Sparks of Education Week served as moderator. The Annie E. Casey Foundation, represented by Ilene Berman, funded the session.

Five programs were presented. Ruth Schoenbach of WestEd spoke for Reading Apprenticeship Improving Secondary Literacy (RAISE), which provides professional development for high school and middle school content teachers (e.g., science, history, English) to help them engage students in productive ways of engaging with text.

Nancy Brynelson, of the California State University system, described the Expository Reading and Writing Course (ERWC), an approach used in 800 high schools across California to attempt to reduce the number of students who need remedial literacy courses when they go on to post-secondary education. Like RAISE, ERWC provides professional development to high school teachers to help them help students to engage deeply with text, using discussions, extended writing, and development of critical thinking skills.

Building Assets Reducing Risks (BARR)
also focuses on high schools, but uses a very different approach. Represented at the briefing by its evaluator, Dr. Maryann Corsello, BARR focuses on social-emotional as well as academic learning. It provides professional development to teachers focused on building relationships among students and staff, effective communication, and risk reviews for low-performing students.

Two of the programs focused on early reading. The Children’s Literacy Initiative (CLI), represented by Joel Zarrow, works in grades pre-K to 3 in high-poverty schools. The program provides one-to-one coaching to teachers, as well as group and leadership coaching, to help teachers with creating and managing the learning environment, using data to guide instruction, and using best practices for early literacy.

SPARK is a tutoring program for grades K-3 that is provided by Boys & Girls Clubs of Greater Milwaukee. Pat Marcus spoke for the program. It provides one-to-one tutoring for struggling readers, primarily using AmeriCorps members. The tutoring strategies are patterned on those of Reading Recovery, an i3 scale-up program used nationally. The SPARK model also includes a family engagement component to increase parents’ skills in supporting their children’s education.

It was wonderful to learn about each of the five programs, but to me, it was particularly exciting to reflect on the larger message: Investing in Innovation is doing what it was intended to do. Every one of the programs at the briefing had been evaluated by third-party evaluators and found to be effective in rigorous experiments. The programs were highly diverse in approaches and intentions, but each adds significantly to our armamentarium of effective models, ready to serve large numbers of students throughout the U.S. Add the proven i3 scale-up programs and many more i3 programs sure to produce similar impacts, not to mention many more already proven programs being funded by other sources, as well as many sure to be proven effective in the next few years, and you can see the potential for dramatic change in how we collectively improve educational outcomes.

The i3 process has had a huge impact on our understanding of educational change. Programs that have shown positive impacts provide solutions that are ready to go. Those that did not are sure to be a majority (in all areas of evidence-based research, most experiments do not show positive effects). However, a great deal is being learned from those studies as well, and the entire field is moving forward at a pace once thought unimaginable. The recent Hill briefing is the second of what I am sure will be many celebrations of proven programs for an ever-expanding set of subjects, grade levels, and situations.

As we gain confidence in each of these approaches, I hope we can learn to rely on our people, our ingenuity, and our science to improve our schools. Like the Spartans of long ago, let’s recognize that these are the essential assets for our security and our progress.

Columbus and Replicability

Happy Columbus Day!

Columbus is revered among researchers because:

  1. He didn’t know where he was going;
  2. He didn’t know where he was when he got there; and
  3. He did it all on government money.

Columbus gets a lot of abuse these days, and for good reason. He was a terrible person. However, people also say that he didn’t actually discover America. Leif Erikson had been here earlier, they say, and of course the Indians were already here.

What Columbus did discover was not America per se, but a replicable and openly published route to America. And that’s what made him justifiably famous. In research, as in discovery, what matters is replicability, the ability to show that you can do something again, and to tell others how they can do the same. Columbus was indisputably the first to do that (Leif Erikson kept his voyage secret).

Replicability is the hallmark of science. In science, if you can’t do it again, it didn’t happen. In fact, there is a popular science humor magazine called the Journal of Irreproducible Results, named for this principle.

As important as replication is in all of science, it is rare in educational research. It’s difficult to get funding to do replications, and if you manage to replicate a finding, journal editors are likely to dismiss it (“What does this add to the literature?” they say). Yet as evidence-based reform in education advances, the need for replication increases. This is a problem because, for example, the majority of programs with at least one study that met What Works Clearinghouse standards had exactly one study that did so.

Soon, results will become available for the first and largest cohort of projects funded by the Investing in Innovation (i3) program. Some of these will show positive effects and some will show outcomes close enough to significance to be worth trying again. I hope there will be opportunities for these programs to replicate and hopefully improve their outcomes, so we can expand our armamentarium of replicable and effective approaches to enduring problems of education.

We really should celebrate Columbus Day on November 3rd when Columbus returned to the New World. The day he reached the New World was a significant event, but it wasn’t really important until he showed that he (and anyone else) could do it again.

It’s the Only Gum My Mom Lets Me Chew

For many years, there was a series of ads for Trident Sugarless Gum that always followed the same pattern. First there were statements about all the wonderful things about the gum, including, “4 out of 5 dentists…” This part was completely boring, perhaps deliberately so. But at the end, there would always be a very cute kid with a sheepish expression who’d say, “Besides, it’s the only gum my mom lets me chew.”

The ad was clearly directed to the parents, not the kids, and I think it was brilliant. What it was trying to do, I’d assume, is to play on parents’ sense of responsibility. Every parent knows that sugared gum is bad for kids’ teeth. The ad subtly said, “You care about your kids and their health. Take a stand to defend your child from the evils of Juicy Fruit.”

Evidence-based reform in education needs to occupy a similar place in the culture of education. Someday, teachers need to expect each other to use proven programs, and to take it as a point of pride that they know about what works and put that knowledge to work in the classroom every day. Teachers care about their kids and their profession, and therefore they see the value of using programs known to work.

Government can play a role in establishing such a norm. For example, government agencies can provide preference points on competitive grants to schools that commit to using proven programs. They can establish criteria for levels of evidence required for a program to be considered proven, and disseminate information about those programs. They can support developers and researchers in creating, evaluating, and disseminating proven programs, as Investing in Innovation currently does. All of these strategies, and more, could help educators learn about and use proven programs to accomplish their goals, and this in turn could build a sense of professionalism, an optimism in the profession that solutions are readily available.

By putting a child’s face and parents’ love and care in front of the statistics, Trident made a place for sugarless gum in the marketplace. In the same way, proven programs have to become desirable to educators for all the right reasons — not just the effect sizes, but kids who are successful and excited about learning.

R&D That Makes a Difference

Over the course of my career, I’ve written a lot of proposals. I’ve also reviewed a lot, and mostly, I’ve seen many funded projects crash and burn, or produce a scholarly article or two that are never heard of again.

As evidence becomes more important in educational policy and practice, I think it’s time to rethink the whole process of funding for development, evaluation, and dissemination.

Here’s how the process works now at the federal level. The feds put out a Request for Proposals (RFP) in the Federal Register. It specifies the purpose of the grant, who is eligible, funding available, deadlines, and most importantly, the criteria on which the proposals will be judged. Proposal writers know that they must follow those criteria very carefully to make it easy for readers to know that each criterion has been satisfied.

The problem with the whole proposal system lies in the perception that each proposal starts with a perfect score (usually 100), and is then marked down for any deficiencies. To oversimplify, reviewers nitpick, and if there is much left after the nits have been picked, the proposal wins.

What this system rewards is enormous care and OCD-level attention to detail. It does not reward creativity, risk, insight, or actual utility for schools. Yet funding grants that do not move forward practice at any significant scale do not do much good in an applied field like education (in related fields such as psychology, purely basic research might justify such approaches, but in education this is a hard argument to make). Maybe our collective inability to do research that affects practice on a broad scale explains some of the lack of enthusiasm our political leadership has for research.

So what would I propose as an alternative? I’m so glad you asked. I’d propose that RFPs be explicitly structured to ask not, “Why shouldn’t we fund this proposal,” but, “Why should we?” That is, proposal writers should be asked to make a case for the potential importance of their work. Here’s a model set of evaluation standards to illustrate what I mean.

A. Significance
1. What are you planning to create?
2. What national problem does your proposed program potentially solve?
3. What outcomes do you expect to achieve, and why are these important?
4. Based on prior research by yourself and others, what is the likelihood that your program will produce the outcomes you expect?
5. What is the likelihood that, if your program is successful, it will work on a significant scale? What is your experience with working at scale or scaling up proven programs in educational settings?
6. In what way is your program creative or distinctive? How might it spark new thinking or development to solve longstanding problems in education?

B. Capabilities
1. Describe the organizational capabilities of the partners to this proposal, as well as the capabilities of the project leadership. Consider capabilities in the following areas:
a. development
b. roll-out, piloting
c. evaluation
d. reporting
e. scale-up
f. communications, marketing
2. Timelines, milestones

C. Evaluation
1. Research questions
2. Design, analysis

D. Impact
Given all you’ve written so far, summarize in one page why this project will make a substantial difference in educational practice and policy.

If we want research and development to produce useful solutions to educational problems, we have to ask the field for just that, and reward those able to produce, evaluate, and disseminate such solutions. Ironically, the federal funding stream closest to the ideal I’ve described is the Investing in Innovation (i3) program, which Congress may be about to shut down. i3 is at least focused on pragmatic solutions rather than theory-building and it has high standards of evidence. But if i3 survives or if it is replaced by another initiative to support innovation, development, evaluation, and scale-up of proven programs, I’d argue that it needs to focus even more on pragmatic issues of effectiveness and scale. Reviewers should be exclaiming, “I get it!” rather than “I gotcha!”

The Evidence or The Morgue

Many years ago, when I was a special education teacher, I had a summer job at a residential school for emotionally disturbed children. The school happened to be located in a former tuberculosis sanitarium. Later, I heard from other teachers elsewhere about having worked in schools in one-time sanitaria as well.

How did it come about, one might ask, that tuberculosis sanitaria across the country became available for use as schools? The answer is that researchers cured the disease. The sanitaria were no longer needed for their original purpose, so they were turned into schools.

One feature of the former sanitaria is that they all had morgues. We used ours to store curriculum materials, because it had very sturdy and useful sliding horizontal cabinets. This arrangement led to a certain amount of macabre humor, but the morgue reminded us that what the sanitaria had once done was deadly serious indeed.

I was recalling my summer in the sanitarium after reading about the latest developments in the reauthorization of ESEA. Both the House and the Senate have now passed bills that eliminate the Investing in Innovation (i3) program and cut funding for the Institute of Education Sciences. In their place, the bills have a lot of language about state and local control, and about identifying and publicizing individual schools that are doing a particularly good job so their good works can help inspire and influence other schools. None of this would bother me if the legislation contained a clear commitment to rigorous research, development, and dissemination, but this may or may not be the case.

The Senate bill, which passed with bipartisan support last week, does authorize an evidence-based innovation fund. Modeled on the successful Small Business Innovation Research (SBIR) program, which funds innovation and evaluation in 11 different government agencies, this initiative would provide flexible funding for a broad range of field-driven projects and allow schools, districts, non-profits, and small businesses to develop and grow innovative programs to improve student achievement. Grants would be awarded using a tiered evidence framework based on an applicant’s proven effectiveness. The provision was initially offered and accepted as a bipartisan amendment during the Senate HELP Committee markup of its bill. However, the House bill has no comparable provision, and I have to wonder if the Senate provision will survive the grueling conference process and make it into the final bill.

Try to imagine what would have happened if tuberculosis research had been treated the way education research has been treated in the House version of the ESEA reauthorization bill. Individual sanitaria with lower death rates might be recognized. States and localities might try out ideas to make the sanitaria more effective, but few if any states or localities would be large enough to do the necessary sustained R & D. “Best practices” would be constrained by the current system, so they might involve better ways for sanitarium staff to do exercises with patients, for example, rather than experimenting with medications or other treatments. The disease would never have been cured. The morgues would still be used for unfortunate patients, not for curriculum materials.

The U.S. spends hundreds of billions of dollars every year on education. What student, parent, teacher, administrator, or policy maker does not want those billions used to make as much of a difference as possible? The pursuit of knowledge about how to improve educational outcomes is obviously important, but it is rarely very high on anyone’s priority list.

Fortunately, medicine and other fields long ago decided that research was in the national interest, and that investments in research were the most reliable way forward in improving important outcomes. In medicine, the choice is stark: either the evidence prevails or the morgue does. Yet in education, anyone with eyes to see knows what happens when children fail to learn. Most of the children who cannot read end up unemployed. Many end up in prison, and all too many in the morgue. We know enough now to be able to say that the great majority of reading failure, for example, is preventable. Yet we choose not to prevent it. What does this say about us as a people, as a society, as a political system?

I hope our leaders in Congress approve the Senate language on evidence, or something similar, and reinstate and fund programs that have the greatest promise in identifying and disseminating effective approaches to key problems. The lives of a generation of vulnerable children depend on their wisdom and courage at this critical juncture.

Good Failure/Bad Failure

Evidence junkies (like me) are reacting to the disappointing news on the evaluation of the Adolescent Behavioral Learning Experience (ABLE), a program implemented at Rikers Island to reduce recidivism among adolescent prisoners. Bottom line: The rigorous independent evaluation of the program failed to find any benefits. What makes this experiment especially interesting is that it is the first U.S. application of social impact bonds. Goldman Sachs put up a $7.2 million loan, and Bloomberg Philanthropies committed to a $6 million loan guarantee. Since the program did not produce the expected outcomes, Goldman Sachs lost $1.2 million.

Ironically, New York City administrators are delighted about the outcome because they do not have to pay for the program. They think they learned a great deal from the experience, for free.

It’s unclear what this will do to the social impact bond movement, currently in its infancy. However, I wanted to extend from this fascinating case to a broader issue in evidence-based reform.

The developers and advocates for the ABLE program who expected positive outcomes turned out to be wrong, at least in this implementation. The investors were wrong in expecting to make a profit. But I’d argue that they are all better off because of this experience, just as the N.Y.C. administrators said.

The distinction I want to make is between wrong and wrong-headed. Wrong, as I’m defining it in this context, means that a given outcome was not achieved, but it was entirely reasonable to expect that it might have been achieved. In contrast, wrong-headed means that not only was the desired outcome not achieved, but it was extremely unlikely that it could have been achieved. In many cases, a key component of wrong-headed actions is that the actor does not even know whether the action was effective or ineffective, right or wrong, and therefore continues with the same or similar actions indefinitely.

Wrong, I’d argue, is an honorable and useful outcome. In a recent interview, former White House advisor Gene Sperling noted that when a few cancer drugs fail to cure cancer, you don’t close down NIH. Instead, you take that information and use it to continue the research and development process. “Wrong,” in this view, can be defined as “good failure,” because it is a step on the path to progress.

“Wrong-headed,” on the other hand, is “bad failure.” When you do something wrong-headed, you learn nothing, or you learn the wrong lessons. Wrong-headed decisions tend to lead to more wrong-headed decisions, as you have no systematic guide to what is working and what is not.

The issue of wrong vs. wrong-headed comes up in the current discussions in Congress about continuing the Investing in Innovation (i3) program. By now, committees in both the House and the Senate have recommended ending i3. But this would be the very essence of wrong-headed policy. Sure, it is probable that many i3 programs funded so far will fail to make a difference in achievement, or will fail to go to scale. This just means that these programs have not yet found success. Some of these may still have evidence of promise, and some will not. However, all i3 programs are rigorously evaluated, so we will know a lot about which worked, which did not, and which still seem promising even if they did not work this time. That’s huge progress. The programs that are already showing success can have immediate impact in hundreds or thousands of schools while others greatly enrich understanding of what needs to be done.

Abandoning i3, in contrast, would be wrong-headed, a sure path to bad failure. A tiny slice of education funding, i3 tells us what works and what does not, so we can continually move towards effective strategies and policies. Without i3 and other research and development investments, education policy is just guesswork, and it gets no smarter over time.

No one can honestly argue that American education is as successful as it should be. Our kids, our economy, and our society deserve much better. Policies that seek a mixture of proven success and “good failure” will get us to solid advances in educational practice and policy. Abandoning or cutting programs like i3 is not just wrong. It’s wrong-headed.