For all the education research and policy types out there, there’s a pretty good food fight taking place between the National Education Policy Center at the University of Colorado and the Los Angeles Times.
It concerns value-added modeling of teacher effectiveness – in particular, the Times project last summer that used test-score data and a value-added formula to give an effectiveness rating to 6,000 elementary-school teachers in the Los Angeles Unified School Districts.
The Times project, which was highly controversial and sent the LA teachers’ union into a frenzy, was based on a value-added analysis by Richard Buddin, a researcher with Rand Corp. The paper put LA teachers into one of five “effectiveness” categories based on their students’ test-score improvement and published a database of the teacher ratings on its website.
UC researchers Derek Briggs and Ben Domingue re-analyzed what they believed to be the same data. First they conducted a “sensitivity analysis,” which produced results that cast doubt on whether Buddin’s approach measured teacher effectiveness, not other factors that influence student achievement.
Next, they ran the data using a slightly different, but arguably valid, value-added model. The result was that about half the LA teachers ended up with a different effectiveness rating, calling into question the validity of the entire exercise.
“This study makes it clear that the LA Times and its research team have done a disservice to the teachers, students, and parents of Los Angeles,” said NEPC director Kevin Welner. “The Times owes its community a better accounting for its decision to publish the names and rankings of individual teachers when it knew or should have known that those rankings were based on a questionable analysis.”
The Times reported on the NEPC study but claimed that it “confirms the broad conclusions of a Times analysis of teacher effectiveness in the Los Angeles Unified School District” because it showed that the effectiveness of teachers varies widely and can be reasonably estimated. The newspaper also pointed out that NEPC receives funding from the Great Lakes Center for Education Research and Practice, which is affiliated with teachers’ unions – suggesting that could bias the research.
Normally these disputes just fade away. But the NEPC hit back. It published a “fact sheet” that pointed to flaws in the Times’ coverage of the report, and it questioned the Times’ decision to assign the story to Jason Felch, one of the writers of the original story that the NEPC report criticized.
But the NEPC didn’t back down either. It published a second fact sheet, alleging that the Times had “repeatedly misrepresented and distorted the facts.”
“While the NEPC researchers explain why they think a stronger model is preferable, that’s not really the point,” the fact sheet says. “Instead, the point is this: when two reasonable models reach such different results, the Times’ decision to publish ratings based on their preferred model is reckless.”
It’s a convention of journalism, when different researchers come to different conclusions, to throw up your hands and report the story as a “he said, she said,” “studies differ” sort of affair. But in this case, the point of the NEPC report is that, well, studies differ. If equally valid approaches produce markedly different results, is it responsible to base a high-impact project on results that, quite simply, may be wrong?