Making the case for their plan to tie teacher evaluations to test scores, Indiana Gov. Mitch Daniels and state Superintendent Tony Bennett have been saying that a teacher’s influence on student achievement scores is 20 times greater than any other factor, including class size and poverty.
Does that sound like a bit of an exaggeration? It does, and it is.
Bennett floated the claim in a Dec. 8 presentation outlining his school-reform agenda to the Indiana Education Roundtable. Daniels has repeated it in media interviews, including with WANE TV in Fort Wayne and the Louisville Courier-Journal.
But it seems that no one has bothered to check it out. We did, and here’s what we found.
Bennett’s presentation attributes the information to Daniel Fallon, an official with the Carnegie Corp. And Fallon did in fact make the claim in a 2003 speech. He based the idea on a research paper presented in 1998 by University of Texas-Dallas professor John Kain. But the “20 times greater” statement doesn’t show up in the paper.
Unfortunately, John Kain died in 2003. So School Matters sent a message to the Texas Schools Project at UT Dallas, asking whether his data supported the Fallon-Daniels-Bennett claim.
We promptly received a thoughtful and thorough reply from Eric Hanushek of the Hoover Institution, Kain’s long-time collaborator and the person known as “the first researcher to measure teacher effectiveness by the learning gains of the teacher’s students.”
Hanushek said the paper that Fallon referenced was “a very preliminary draft of some ongoing work” that wasn’t continued in the same vein. “Unfortunately, I think it is difficult to get the ‘20 times’ statement from the article — or from any other research for that matter,” he wrote.
There are two basic problems, he said. First, “there are no natural ways to define what ‘20 times’ would mean.” Do you separate out the influences on achievement and give each a numerical value? Or compare the effect you get for each dollar spent on improvement? “None really work very well.”
The bigger problem, he said, is that statistical models of teacher effectiveness already consider past student achievement levels that incorporate socio-economic status, “making it impossible to compare teachers and socio-economic factors … As far as I can tell, John Kain never tried to develop this.”
That’s not to say that Hanushek downplays the importance of teacher effectiveness. He’s probably the leading academic source for the idea that schools should use test-score data to evaluate teachers and incorporate such evaluations into teacher personnel decisions.
He sent a couple of his own research papers that seek to quantify the impact of teachers on learning. One concludes that replacing the least effective 5-8 percent of teachers with just average teachers would move U.S. students to the top of international rankings for math and science.
Hanushek said the difference between a bad teacher and a good teacher is a full year of growth in a student’s learning. “Having 3-5 teachers in a row at the 85th percentile as opposed to the 50th percentile … can make up for the average academic deficits of being from a poor family,” he said.
This finding is disputed by some other researchers and experts, including Diane Ravitch, who refers to it as an “urban myth” in her book The Death and Life of the Great American School System.
But the claim that teachers matter 20 times more than any other factor? It seems that only Daniels and Bennett are making that argument.
Daniels, Bennett, and the Republican leadership of the Indiana legislature are determined to make big changes in the way teachers are evaluated, paid and retained (or not) in their jobs. If they can truly create a fair and honest way to encourage good teachers and make sure the worst teachers get better or get out, students should benefit. But if they get it wrong, the changes could demoralize teachers without helping students at all. If they do it, it’s essential that they get it right.
And getting it right starts with being honest about the facts.