Whenever I speak about evidence-based reform someone always asks this question: “Won’t all of these randomized evaluations just reinforce teaching to the (very bad word) standardized reading and math tests?” My wife Nancy and I were recently out at our alma mater, Reed College. I gave a speech on evidence-based reform, and of course I got this question.
Just to save time in future talks I’m going to answer this question here once and for all. And the answer is:
No! In fact, evidence-based reform could be America’s escape route from excessive use of accountability and sanctions to guide education policy.
Here’s how this could happen. I’d be the first to admit that today, most studies use standardized tests as their outcome measures. However, this need not be the case, and there are in fact exceptions.
Experiments compare learning gains made by students in a given treatment group to those in a control group. Any assessment can be used as the posttest as long as it meets two key standards:
a) It measures content taught equally in both groups, and
b) It was not made up by the developer or researchers.
What this means is that authentic performance measures, measures aligned with today’s standards, measures that require creativity and non-routine problem solving, or measures of subjects (such as social studies) that are taught in school but not tested for accountability, can all be valid in experiments, as long as experimental and control students had exposure to the content, skills, or performances being assessed.
As a practical example, imagine a year-long evaluation of an innovative science program for seventh graders. The new program emphasizes the use of open-ended group laboratory projects and explanations of processes observed in the lab. The control group uses a traditional science text, demonstrations, and more traditional lab work.
Experimenters might evaluate the innovative approach using a multifaceted measure (made by someone other than the researchers) covering all of seventh grade science, including open-ended as well as multiple choice tests. In addition, students might be asked to set up, carry out, and explain a lab exercise, involving content that was not presented in either program.
If this independent measure showed positive effects on all types of measures, great. If it showed advantages on open-ended and lab assessments but no differences on multiple choice tests, this could be important, too.
The point here is that measures used in program evaluations need not be limited to standardized measures, but can instead help break us free from standardized measures. One reason it’s hard to get away from multiple choice tests is that they are cheap and easy to administer and score, which means they are much easier to use when you have to test a whole state. It would be very difficult to have every student in a state set up a lab experiment for example. But program evaluations need not be bound by this practical restriction, because they only involve a few dozen schools, at most. Even if the state does not test writing, for example, program evaluations can. Even if the state does not test social studies, program evaluations can.
As evidence-based reform becomes more important, it may become more and more possible to justify the teaching of approaches proven to be effective on valid measures, even if the subject is not currently assessed for accountability and even if the tests are not accountability measures. States may never administer history tests, but they may encourage or incentivize use of proven approaches to teaching history. Perhaps proven approaches to music, art, or physical education could be identified and then supported and encouraged.
Managing schools by accountability alone leaves us trying to enforce minimums using limited assessments. This may be necessary to be sure that all schools are meeting minimums and making good progress on basic skills, but we need to go beyond standardized tests to consider all of the needs shared by all children. Evidence that goes beyond what states can test well may be just the ticket out of the standardized test trap.