Lessons from China

blog_3-22-18_Confucius_344x500Recently I gave a series of speeches in China, organized by the Chinese University of Hong Kong and Nanjing Normal University. I had many wonderful and informative experiences, but one evening stood out.

I was in Nanjing, the ancient capital, and it was celebrating the weeks after the Chinese New Year. The center of the celebration was the Temple of Confucius. In and around it were lighted displays exhorting Chinese youth to excel on their exams. Children stood in front of these displays to have their pictures taken next to characters saying “first in class,” never second. A woman with a microphone recited blessings and hopes that students would do well on exams. After each one, students hit a huge drum with a long stick, as an indication of accepting the blessing. Inside the temple were thousands of small silk messages, bright red, expressing the wishes of parents and students that students will do well on their exams. Chinese friends explained what was going on, and told me how pervasive this spirit was. Children all know a saying to the effect that the path to riches and a beautiful wife was through books. I heard that perhaps 70% of urban Chinese students go to after-school cram schools to ensure their performance on exams.

The reason Chinese parents and students take test scores so seriously is obvious in every aspect of Chines culture. On an earlier trip to China I toured a beautiful house, from hundreds of years ago, in a big city. The only purpose of the house was to provide a place for young men of a large clan to stay while they prepared for their exams, which determined their place in the Confucian hierarchy.

As everyone knows, Chinese students do, in fact, do very well on their exams. I would note that these data come in particular from urban Eastern China, such as Shanghai. I’d heard about but did not fully understand policies that contribute to these outcomes. In all big cities in China, students can only attend schools in their city neighborhoods, where the best schools in the country are, if they were born there or own apartments. In a country where a small apartment in a big city can easily cost a half million dollars (U.S.), this is no small selection factor. If parents work in the city but do not own an apartment, their children may have to remain in the village or small city they came from, living with grandparents and attending non-elite schools. Chinese cities are growing so fast that the majority of their inhabitants come from the rest of China. This matters because admirers of Chinese education often cite the amazing statistics from the rich and growing Eastern Chinese cities, not the whole country. It’s as though the U.S. only reported test scores on international comparisons from suburbs in the Northeastern states from Maryland to New England, the wealthiest and highest-achieving part of our country.

I do not want to detract in any way from the educational achievements of the Chinese, but just to put it in context. First, the Chinese themselves have doubts about test scores as the only important indicators, and admire Western education for its broader focus. But just sticking to test scores, China and other Confucian cultures such as Japan, South Korea, and Singapore have been creating a culture valuing test scores since Confucius, about 2500 years ago. It would be a central focus of Chinese culture even if PISA and TIMSS did not exist to show it off to the world.

My only point is that when American or European observers hold up East Asian achievements as a goal to aspire to, these achievements do not exist in a cultural vacuum. Other countries can potentially achieve what China has achieved, in terms of test scores and other indicators, but they cannot achieve it in the same way. Western culture is just not going to spend the next 2500 years raising its children the way the Chinese do. What we can do, however, is to use our own strengths, in research, development, and dissemination, to progressively enhance educational outcomes. The Chinese can and will do this, too; that’s what I was doing traveling around China speaking about evidence-based reform. We need not be in competition with any nation or society, as expanding educational opportunity and success throughout the world is in the interests of everyone on Earth. But engaging in fantasies about how we can move ahead by emulating parts of Chinese culture that they have been refining since Confucius is not sensible.

Precisely because of their deep respect for scholarship and learning and their eagerness to continue to improve their educational achievements, the Chinese are ideal collaborators in the worldwide movement toward evidence-based reform in education. Colleagues at the Chinese University of Hong Kong and the Nanjing Normal University are launching Chinese-language and Asian-focused versions of our newsletter on evidence in education, Best Evidence in Brief (BEiB). We and our U.K. colleagues have been distributing BEIB for several years. We welcome the opportunity to share ideas and resources with our Chinese colleagues to enrich the evidence base for education for children everywhere.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

Advertisements

On High School Graduation Rates: Want to Buy My Bridge?

FSK Bridge 02 13 18
 

Francis Scott Key Bridge (Baltimore) By Artondra Hall [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons, edited for size

 

I happen to own the Francis Scott Key Bridge in Baltimore, pictured here. It’s lovely in itself, has beautiful views of downtown and the outer harbor, and rakes in more than $11 million in tolls each year. But I’m willing to sell it to you, cheap!

If you believe that I own a bridge in Baltimore, then let me try out an even more fantastic idea on you. Since 1992, the achievement of America’s 12th graders on NAEP reading and math tests has been unchanged. Yet high school graduation rates have been soaring. From 2006 to 2016, U.S. graduation rates have increased from 73% to 84%, an all-time record. Does this sound plausible to you?

Recently, the Washington Post (https://www.washingtonpost.com/local/education/fbi-us-education-department-investigating-ballou-graduation-scandal/2018/02/02/b307e57c-07ab-11e8-b48c-b07fea957bd5_story.html?utm_term=.84c1176bb8ff) reported a scandal about graduation rates at Ballou High School in Washington, DC, a high-poverty school not known (in the past) for its graduation rates. In 2017, 100% of Ballou students graduated, and 100% were accepted into college. An investigation by radio station WAMU, however, found that a large proportion of the graduating seniors had very poor attendance, poor achievement, and other problems. In fact, the Post reported that one third of all graduating seniors in DC did not meet district graduation standards. Ballou’s principal and the DC Director of Secondary Schools resigned, and there are ongoing investigations. The FBI has recently gotten involved.

In response to these stories, teachers across America wrote to express their views. Almost without exception, the teachers said that the situation in their districts is similar to that in DC. They said they are pressured, even threatened, to promote and then graduate every student possible. Students who fail courses are often offered “credit recovery” programs to obtain their needed credits, and these were found in an investigation by the Los Angeles Times  to have extremely low standards (https://robertslavinsblog.wordpress.com/2017/08/17/the-high-school-graduation-miracle/). Failing students may also be allowed to do projects or otherwise show their knowledge in alternative ways, but these are derided as “Mickey Mouse.” And then there are students like some of those at Ballou, who did not even bother to show up for credit recovery or Mickey Mouse, but were graduated anyway.

The point is, it’s not just Ballou. It’s not just DC. In high-poverty districts coast to coast, standards for graduation have declined. My colleague, Bob Balfanz, coined the term “dropout factories” many years ago to describe high schools, almost always serving high-poverty areas, that produced a high proportion of all dropouts nationwide. In response, our education system got right to work on what it does best: Change the numbers to make the problem appear to go away. The FBI might make an example of DC, but if DC is in fact doing what many high-poverty districts are doing throughout the country, is it fair to punish it disproportionately? It’s not up to me to judge the legalities or ethics involved, but clearly, the problem is much, much bigger.

Some people have argued with me on this issue. “Where’s the harm,” they ask, “in letting students graduate? So many of these students encounter serious barriers to educational success. Why not give them a break?”

I will admit to a sympathy for giving high school students who just barely miss standards legitimate opportunities to graduate, such as taking appropriately demanding makeup courses. But what is happening in DC and elsewhere is very far from this reasonable compromise with reality.

I have done some research in inner-city high schools. In just about every class, there are students who are actively engaged in lessons, and others who would become actively engaged if their teachers used proven programs (in my case it was cooperative learning). But even with the best programs, there were kids in the back of the class with headphones on, who were totally disengaged, no matter what the teacher did. And those were the ones who actually showed up at all.

The kids who were engaged, or became engaged because of excellent instruction, should have a path to graduation, one way or another. The rest should have every opportunity, encouragement, and assistance to reach this goal. Some will choose to take advantage, some will not, but that must be their choice, with appropriate consequences.

Making graduation too easy not only undermines the motivations of students (and teachers). It also undermines the motivation of the entire system to introduce and effectively implement effective programs, from preschool to 12th grade. If educators can keep doing what they’ve always done, knowing that numbers will be fiddled with at the end to make everything come out all right, then the whole system can and will lose a major institutional incentive for improvement.

The high dropout rate of inner-city schools is indeed a crisis. It needs to be treated as such-not a crisis of numbers, but a crisis encountered by hundreds of thousands of vulnerable, valuable students. Loosening standards and then declaring success, which every educator knows to be false, corrupts the system, undermining confidence in the numbers even when they are legitimate. It fosters cynicism that nothing can be done.

Is it too much to expect that we can create and implement effective strategies that would enable virtually all students to succeed on appropriate standards in elementary, middle, and high school, so that virtually all can meet rigorous requirements and walk across a stage, head held high, knowing that they truly attained what a high school diploma is supposed to certify?

If you agree that high school graduation standards have gone off the rails, it is not enough to demand tougher standards. You also have to advocate for and work for application of proven approaches to make deserved and meaningful graduation accessible to all.

On the other hand, if you think the graduation rate has legitimately skyrocketed in the absence of any corresponding improvement in reading or math achievement, please contact me at www.buy-my-bridge.com. It really is a lovely bridge.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.

On Motivation

Once upon a time there was a man standing on a city street selling pencils from a tin cup. An old friend came by and recognized him.

“Hank!” said his friend. “What happened to you? Didn’t you have a big job at the Acme Dog Food Company?”

Hank hung his head. “I did,” he said mournfully. “I was its chief scientist. But it closed down, and it was all my fault!”

“What happened?” asked his friend.

“We decided to make the best dog food ever. We got together the top experts in dog nutrition in the whole world to find out what dogs really need. We put in the very best ingredients, no matter what they cost.”

“That sounds wonderful!” exclaimed the friend.

“It sounded great,” sighed Hank, “but the darned dogs wouldn’t eat it!”

In educational development, research, and dissemination, I think we often make the mistake made by the mythical Acme Dog Food Company. We create instructional materials and software completely in accord with everything the experts recommend. Today, for example, someone might make a program that is aligned with the Common Core or other college- and career-readiness standards, that uses personalization and authentic problem solving, and so on. Not that there is anything wrong with these concepts, but are they enough?

The key factor, I’d argue, is motivation. No matter how nutritious our instruction is, it has to appeal to the kids. In a review of secondary reading programs my colleagues and I wrote recently (www.bestevidence.org), most of the programs evaluated were 100% in accord with what the experts suggest. In particular, most of them emphasized the teaching of metacognitive skills, which has long been the touchstone for secondary reading, and many also provided an extra instructional period every day, in accord with the popular emphasis on extra-time strategies.

However, the approaches that made the biggest differences in reading outcomes were not those that provided extra time. They included small-group or individual tutoring approaches, cooperative learning, BARR (a program focusing on building relationships between teachers and students), and a few technology approaches. The successful approaches usually included metacognitive skills, but so did many programs that did not show positive outcomes.

What united the successful strategies is that they all get to the head through the heart.

Tutoring allows total personalization of instruction, but it also lets tutors and students build personal, close relationships. BARR (Building Assets, Reducing Risks) is all about building personal relationships. Cooperative learning focuses on building relationships among students, and adding an element of fun and engagement to daily lessons. Some technology programs are also good at making lessons fun and engaging.

I can’t say for sure that these were the factors that made the difference in learning outcomes, but it seems likely. I’d never say that instructional content and strategies don’t matter. They do. But the very best teaching methods with the very best content are unlikely to enhance learning very much unless they make the kids eager to learn.

The High School Graduation Miracle

High school graduation rates have skyrocketed in recent years. From 2006 to 2013, U.S. graduation rates increased from 73% to 82%. Yet over this same time period, the National Assessment of Educational Progress (NAEP) reports that the reading and math achievement of 12th graders have not budged at all.

How can these two apparently contradictory facts be reconciled? The unavoidable conclusion is that many students who were not graduating before are graduating now, or put another way, high school graduates today have lower skills than did students just a few years ago.

I don’t know exactly why this is happening, but I have a few guesses. One is that the use of what is called “credit recovery” has increased dramatically. Credit recovery means providing students who failed a given course another opportunity to pass. Apparently these courses are much easier to pass than the initial course. For example, a July 2, 2017 article in the LA Times described a credit recovery program in which a student could raise his grade from F to C in one week during the winter break. The report followed one student, who never did any lab work, but was seen copying a food pyramid from the Internet onto a worksheet. Credit Recovery courses are often offered online, in which case students can take them at home. Does this worry you? It does me.

Another possibility is that as graduation has become a focus of school accountability in many states and districts, teachers come under pressure to let marginal students pass. Unlike other accountability measures, graduation is determined by students’ grades, course credits, and other indicators that are subjective. Teachers may reason that passing such students benefits the students, the school, and themselves. So why not?

There is nothing wrong in principle with higher graduation rates, but if they are accomplished by lowering standards, then a high school diploma becomes even less valued than diplomas were in the past. This is unfair to students who work hard and pass their courses fairly, and it may contribute to cynicism throughout the system.

Further, reducing graduation standards undermines the efforts of administrators and teachers who truly want to improve student achievement as a way to improve graduation rates. If it’s a lot easier to provide credit recovery classes or to lower standards, then genuine reformers may be discouraged.

I hope there is some more optimistic explanation for the increase in high school graduation contrasted with the lack of gains in achievement. I’d love to believe that graduation rates are truly going up because of better schools and teachers, harder-working students, or other factors. Graduation is important for students, but for our society and our economy, it matters more what students can actually do. Letting students graduate without adequate skills is something we should not let pass.

This blog is sponsored by the Laura and John Arnold Foundation

Make No Small Plans

In recent years, an interest has developed in very low-cost interventions that produce small but statistically significant effects on achievement. The argument for their importance is that their costs are so low that their impacts are obtained very cost-effectively. For example, there is evidence that a brief self-affirmation exercise can produce a small but significant effect on achievement, and that a brief intervention to reduce “social identity threat” can do the same. A study in England found that a system to send 50 text messages over the course of a school year, announcing upcoming tests and homework assignments, feedback on grades, test results, and attendance, and updates on topics being studied in school, improved math achievement slightly but significantly, at a cost of about $5 a year.

There is nothing wrong with these mini-interventions, and perhaps all schools should use them. Why not? Yet I find myself a bit disturbed by this type of approach.

Step back from the small-cost/small-but-significant outcome and consider the larger picture, the task in which all who read this blog are jointly engaged. We face an educational system that is deeply dysfunctional. Disadvantaged students remain far, far behind middle-class students in educational outcomes, and the gap has not narrowed very much over decades. The U.S. remains well behind peer nations in achievement and is not catching up. Dropout rates in the U.S. are diminishing, but skill levels of American high school graduates from disadvantaged schools are appalling.

For schools with limited budgets to spend on reform, it may be all they can do to adopt a low-cost/low-but-significant outcome intervention on the basis that it’s better than nothing. But again, step back to look at the larger situation. The average American student is educated at a cost of more than $11,000 per year. There are whole-school reform approaches, such as our own Success for All in elementary and middle schools and BARR in secondary schools, that cost around $100 per student per year, and have been found to make substantial differences in student achievement. Contrast this to a low-cost program that costs, say, $5 per student per year.

$100 is less than 1% of the ordinary cost of educating a student, on average. $5 is less than .05%, of course. But in the larger scheme of things, who cares? Using a proven whole-school reform model might perhaps increase the per-student cost from $11,000 to $11,100. Adding the $5 low-cost intervention could increase per-student costs from $11,000 to $11,005. From the perspective of a principal who has a fixed budget, and simply does not have $100 per student to spend, the whole-school approach may be infeasible. But from the system perspective, the difference between $11,000 and $11,100 (or $11,005) is meaningless if it truly increases student achievement. Our goal must be to make meaningful progress in reducing gaps and increasing national achievement, not make a small difference that happens to be very inexpensive.

I once saw a film in England on the vital role of carrier pigeons in the English army in World War II. I’m sure those pigeons played their part in the victory, and they were very cost-effective. But ultimately, it was expensive tanks and planes and ships and other weapons, and courageous men and women, who won the war, not pigeons, and piling up small (even if effective) interventions was just not going to do it.

We should be in a war against inequality, disadvantage, and mediocre outcomes in education. Winning it will require identification and deployment of whole-school, whole-district, and whole-state approaches that can be reliably replicated and intelligently applied to ensure positive, widespread improvements. If we just throw pigeon-sized solutions at huge and tenacious problems, our difficulties are sure to come home to roost.

This blog is sponsored by the Laura and John Arnold Foundation

Good Programs? Bad Programs? Show Me the Data!

Since launching Evidence for ESSA on February 28, I’ve gotten a lot of emails. In general, the responses to the website have been very positive. However, a small minority of emails have been really angry about the entire project.

The writers of these angry emails are upset that positive ESSA evidence levels were assigned to what they considered “bad programs” and less positive ESSA evidence levels were assigned to what they considered “good programs.” Of course, in each case I explain that we are only reviewing existing evidence for demonstrated impact on students’ learning and assigning ESSA evidence levels according to the standards defined by the evidence provisions included in the Every Student Succeeds Act (ESSA), which is now the law of the land. We are not assigning ESSA evidence levels to programs based on their “goodness” or “badness” on any dimension other than impact on achievement.

Evidence for ESSA critics were having none of it. In their minds, “good programs” are ones that adhere to well-established principles, or have been supported by experts, or are aligned with state or national standards. “Bad programs” are ones that, in their opinions, violate these standards or fail to incorporate well-supported principles.

Expert opinions and standards are important, of course, but how about effectiveness? I asked how anyone could tell if a program was good or bad unless they knew if it actually benefitted students. This did no good. “Don’t you understand?” they asked. “Such-and-such experts or so-and-so standards support these programs, so they are good.” End of story.

But adhering to principles of good practice is not at all the same as demonstrated effectiveness. To understand this, imagine a textbook that meets every standard and conforms to all current conceptions of good practice, yet teachers are given only three hours of in-service to use it. An evaluation would probably find no improvement in learning. Now imagine a program built around the very same textbook that provides a week of training, in-school coaching once a month, videos to demonstrate program elements, and so on. This program is much more likely to work. The point is, the content of a curriculum is part of what might make it effective or ineffective. The professional development and other features are also essential. So declaring a program or curriculum “good” or “bad” based on content alone is misleading.

The conversations I am having with Evidence for ESSA critics illustrate the sea change being brought about by the ESSA evidence standards. Way back in . . ., well, 2016, educational programs were largely judged according to alignment with standards, state textbooks and software reviews, correspondence with expert opinion, or most often, perhaps, based on leaders’ preferences, tips from nearby districts, and appeals from sales reps. Actual proven impact on students was hardly ever involved. Today, as the ESSA evidence standards begin to be implemented, evidence of effectiveness is beginning to get some respect. This is a good thing for students, teachers, parents, and our nation, but it is deeply uncomfortable for those who have long relied on curriculum content or opinion to drive selection of educational programs. Those are the people contacting me to complain about the “bad programs” being assigned positive ESSA evidence levels, the ones that, “bad” as they may be according to some peoples’ opinions, actually enhance student achievement.

For many years, in school principals’ and superintendents’ offices all over America, I’ve seen the following statement proudly displayed on the wall:

“In God we trust. All others bring data.”

At long last, this saying is beginning to apply to the critical choices educators make in selecting programs, books, software, and professional development. Good programs? Bad programs? Don’t tell me your opinions. Show me the data!

This blog is sponsored by the Laura and John Arnold Foundation

Gambling With Our Children’s Futures

I recently took a business trip to Reno and Las Vegas. I don’t gamble, but it’s important to realize that casinos don’t gamble either. A casino license is permission to make a massive amount of money, risk free.

Think of a roulette table, for example, as a glitzy random numbers generator. People can put bets on any of 38 numbers, and if that number comes up, you get 36 times your bet. The difference between 38 and 36 is the “house percentage.” So as long as the wheel is spinning and people are betting, the casino is making money, no matter what the result is of a particular spin. This is true because over the course of days, weeks, or months, that small percentage becomes big money. The same principle works in every game in the casino.

In educational research, we use statistics much as the casinos do, though for a very different purpose. We want to know what the effect of a given program is on students’ achievement. Think of each student in an experiment as a separate spin of the roulette wheel. If you have just a few students, or a few spins, the results may seem very good or very bad, on average. But when you have hundreds or thousands of students (or spins), the averages stabilize.

In educational experiments, some students usually get an experimental program and others serve as controls. If there are few students (spins) in each group, the differences are unreliable. But as the numbers get larger, the difference between experimental and control groups gets reliable.

This explains why educational experiments should involve large numbers of students. With small numbers, differences could be due to chance.

Several years ago, I wrote an article on the relationship between sample size and effect size in educational experiments. Small studies (e.g., fewer than 100 students in each group) had much larger experimental-control differences (effect sizes) than big ones. How could this be?

What I think was going on is that in small studies, effect sizes could be very positive or very negative (favoring the control group). When positive results are found, results are published and publicized. When results go the other way? Not so much. The studies may disappear.

To understand this, go back to the casino. Imagine that you bet on 20 spins, and you make big money. You go home and tell your friends you are a genius, or you credit your lucky system or your rabbit’s foot. But if you lose your shirt on 20 spins, you probably slink home and stay quiet about the whole experience.

Now imagine that you bet on 1000 spins. It is statistically virtually certain that you will lose a certain amount of money (about 2/38 of what you bet, to be exact, because of 0 and 00). This outcome is not interesting, but it tells you exactly how the system works.

In big studies in education, we can also produce reliable measures of “how the system works” by comparing hundreds or thousands of experimental and control students.

Critics of quantitative research in education seem to think we are doing some sort of statistical mumbo-jumbo with our computers and baffling reports. But what we are doing is trying to get to the truth, with enough “spins” of the roulette wheel to even out chance factors.

Ironically, what large-scale research in education is intended to do is to diminish the role of chance in educational decisions. We want to help educators avoid gambling with their children’s futures.

This blog is sponsored by the Laura and John Arnold Foundation