Do Different Textbooks Have Different Effects on Student Achievement?

The British comedy group Monty Python used to refer to “privileged glimpses into the perfectly obvious.”

And just last week, there they were. In a front-page article, the March 13 edition of Education Week reported that a six-state study of the achievement outcomes of different textbooks found . . . wait for it. . . near-zero relative effects on achievement measures (Sawchuck, 2019).

Really!

The study was led by Harvard’s Thomas Kane, a major proponent of the Common Core, who was particularly upset to find out that textbooks produced before and after the Common Core influenced textbook content had few if any differential effects on achievement.

I doubt that I am the only person who is profoundly unsurprised by these findings. For the past 12 years, I’ve been doing reviews of research on programs’ effects on achievement in rigorous research. Textbooks (or curricula) are usually one of the categories in my reviews. You can see the reviews at www.bestevidence.org. Here is a summary of the average effect sizes for textbooks or curricula:

Review	No. of Studies	Mean Effect Size
Elementary Reading (Inns et al., 2019)	9	+0.03
Elementary Math Pellegrini et al., 2018)	16	+0.06
Secondary Math (Slavin et al., 2009)	40	+0.03
Secondary Science (Cheung et al., 2016)	8	+0.10
Weighted Average	73	+0.04

None of these outcomes suggest that textbooks make much difference, and the study-weighted average of +0.04 is downright depressing.

Beyond the data, it is easy to see why evaluations of the achievement outcomes of textbooks rarely find significant positive outcomes. Such studies compare one textbook to another textbook that is usually rather similar. The reason is that textbook publishers respond to the demands of the market, not to evidence of effectiveness. New and existing textbooks were shaped by similar market forces. When standards change, as in the case of the Common Core State Standards in recent years, all textbook companies generally are forced to make changes in the same direction. There may be a brief window of time when new textbooks designed to meet new standards have a temporary advantage, but large publishers are extremely sensitive to such changes, and if they are not up to date in terms of standards today, they soon will be. Still, as the Kane et al. study found, changes in standards do not in themselves improve achievement on a substantial scale. Changes in standards do change market demand, which changes the content of textbooks, but fundamentally, the changes are not enough to make a measurable difference in learning.

Kane was quoted by Education Week as drawing the lesson from the study that perhaps it isn’t the textbooks that matter, but rather how the textbooks are used:

“What levels of coaching or more-intensive professional development are required to help teachers use rigorous materials at higher levels of fidelity, and does that produce larger benefits?” (Sawchuk, 2019, p. 17).

This sounds logical, but recent research in elementary mathematics calls this approach into question. Pellegrini et al. (2018) examined a category of programs that provide teachers with extensive professional development focused on math content and pedagogy. The average effect size across 12 studies was only +0.04, or essentially zero. In contrast, what did work very well were one-to-one and one-to-small group tutoring (mean effect size = +0.29) and professional development focused on classroom management and motivation (mean effect size = +0.25). In other words, programs focusing on helping teachers use standards-based materials added little if anything to the learning impact of textbooks. What mattered, beyond tutoring, were approaches that change classroom routines and relationships, such as cooperative learning or classroom management methods.

Changing textbooks matters little, and adding extensive professional development focused on standards adds even less. Instead, strategies that engage, excite, and accommodate individual needs of students are what we find to matter a great deal, across many subjects and grade levels.

This should be a privileged glimpse into the perfectly obvious. Everyone knows that textbooks make little difference. Walk through classrooms in any school, teaching any subject at any grade level. Some classes are exciting, noisy, fully engaged places in which students are eager to learn. Others are well, teaching the textbook. In which type of class did you learn best? In which type do you hope your own children will spend their time in school, or wish they had?

What is obvious from the experience of every teacher and everyone who has ever been a student is that changing textbooks and focusing on standards do not in themselves lead to classrooms that kindle the love of learning. Imagine that you, as an accomplished adult educator, took a class in tennis, or Italian, or underwater basket weaving. Would a teacher using better textbooks and more advanced standards make you love this activity and learn from it? Or would a teacher who expresses enthusiasm for the subject and for the students, who uses methods that engage students in active social activities in every lesson, obtain better outcomes of every kind? I hope this question answers itself.

I once saw a science teacher in Baltimore teaching anatomy by having students take apart steamed crabs (a major delicacy in Baltimore). The kids were working in groups, laughing at this absurd idea, but they were learning like crazy, and learning to love science. I would submit that this experience, these connections among students, this laughter are the standards our schools need to attain. It’s not about textbooks, nor professional development on textbooks.

Another Baltimore teacher I knew taught a terrific unit on ancient Egypt. The students made their own sarcophagi, taking into the afterlife the things most important to them. Then the class went on a field trip to a local museum with a mummy exhibit, and finally, students made sarcophagi representing what Egyptians would value in the afterlife.  That’s what effective teaching is about.

The great 18^th century Swedish botanist Carl Linnaeus took his students on walks into forests, fields, and lakes around Uppsala University. Whatever they found, they brought back held high singing and playing conch shell trumpets in triumph.  That’s what effective teaching is about.

In England, I saw a teacher teaching graph coordinates. She gave each student’s desk a coordinate, from 1, 1 to 5, 5, and put up signs labeled North, South, East, and West on the walls. She then made herself into a robot, and the students gave her directions to get from one coordinate to another. The students were laughing, but learning. That’s what effective teaching is about.

No textbook can compete with these examples of inspired teaching. Try to remember your favorite textbook, or your least favorite. I can’t think of a single one. They were all the same. I love to read and love to learn, and I’m sure anyone reading this blog is the same. But textbooks? Did a textbook ever inspire you to want to learn more or give you enthusiasm for any subject?

This is a privileged glimpse into the perfectly obvious to which we should devote our efforts in innovation and professional development. A textbook or standard never ignited a student’s passion or curiosity. Textbooks and standards may be necessary, but they will not transform our schools. Let’s use what we already know about how learning really happens, and then make certain that every teacher knows how to do the things that make learning engage students’ hearts and emotions, not just their minds.

References

Cheung, A., Slavin, R.E., Kim, E., & Lake, C. (2016). Effective secondary science programs: A best-evidence synthesis. Journal of Research on Science Teaching, 54 (1), 58-81. Doi: 10.1002/tea.21338

Inns, A., Lake, C. Byun, S., Shi, C., & Slavin, R. E. (2019). Effective Tier 1 reading instruction for elementary schools: A systematic review. Paper presented at the annual meeting of the Society for Research on Educational Effectiveness, Washington, D.C.

Pellegrini, M., Inns, A., Lake, C., & Slavin, R. E. (2018). Effective programs in elementary mathematics: A best-evidence synthesis. Manuscript submitted for publication.

Sawchuk, S. (2019, March 13). New texts failed to lift test scores in six-state study. Education Week, 38(25), 1, 17.

Slavin, R.E., Lake, C., & Groff, C. (2009). Effective programs in middle and high school mathematics: A best-evidence synthesis. Review of Educational Research, 79 (2), 839-911.

This blog was developed with support from the Laura and John Arnold Foundation. The views expressed here do not necessarily reflect those of the Foundation.