Big, impressive study, questionable policy conclusions

A study of the impact of teachers on student success has been drawing lots of attention, including a big story in the New York Times, praise from columnist Nicholas Kristof and analysis in the blogosphere.

On the one hand, the paper by economists Raj Chetty, John Friedman and Jonah Rockoff offers new evidence that good teaching has long-lasting and far-reaching effects. This suggests that the recruitment, preparation and support of teachers should be a high priority for the nation.

But the economists also use their findings to call for rating teachers on the basis of “value-added” models, which use complex formulas to measure teachers’ impact on student test scores – and for firing teachers who don’t measure up. Annie Lowrey writes in the Times:

The authors argue that school districts should use value-added measures in evaluations, and to remove the lowest performers, despite the disruption and uncertainty involved.

“The message is to fire people sooner rather than later,” Professor Friedman said.

Professor Chetty acknowledged, “Of course there are going to be mistakes — teachers who get fired who do not deserve to get fired.” But he said that using value-added scores would lead to fewer mistakes, not more.”

This is a little surprising, given that, in the study itself, they caution against that sort of policy conclusion.

“Overall, our study shows that great teachers create great value and that test score impacts are helpful in identifying such teachers. However, more work is needed to determine the best way to use VA for policy,” they write in the executive summary.

They add that two important questions must be resolved before value-added models are used to evaluate teachers. One is whether attaching high stakes to test scores will skew results so much that it undermines the accuracy of the models. The other has to do with the economic cost of firing teachers, sometimes in error – the very “mistakes” that Chetty said would be trivial.

Then the New York Times calls and they throw caution to the wind.

The study reportedly breaks new ground by quantifying the long-term impact of teachers who are effective at raising students’ test scores. It finds that students with high-value-added teachers are more likely to attend college and more likely to get into “good” colleges. Their lifetime earnings are higher. Girls who have more effective teachers are less likely to become teen mothers.

Also, the study presents evidence that value-added models can effectively identify effective teachers, overcoming the “bias” that enters the formulas from different ways of grouping students and assigning teachers to classes.

The authors reach these conclusions by analyzing data for 2.5 million students over 20 years, including test scores, teacher assignments, college attendance records, tax returns, etc.

Even scholars who question the conclusions seem impressed. Bruce Baker at School Finance 101 writes that it’s “a really cool academic study” with “a freakin’ amazing data set!” Matt Di Carlo at Shanker Blog calls it “one of the most dense, important and interesting analyses on this topic in a very long time.”

But while the study shows that good teachers matter, Di Carlo adds, “What it does not show is how to measure and improve teacher quality, which are still open questions.”

Other studies have shown that value-added ratings can be highly unstable; a teacher who is rated ineffective one year may be rated highly effective the next. Different value-added models produce different results. Research has found that, even in ideal conditions, most models will mis-classify significant numbers of teachers as above or below average.

The Chetty-Friedman-Rockoff study doesn’t make those problems go away. (Neither does the fact that many states, including Indiana, are already moving to evaluating teachers on the basis of test-score improvement. Indiana will use not value-added but a growth model, a simpler and arguably less sophisticated approach to tying teachers to test-score gains).

Another issue is what Baker calls the use of “super-multiplicative-aggregation” to make findings seem significant. Friedman tells the Times, for example, that replacing a “low-value” teacher with an average one for 10 years would increase the lifetime earnings of all the students in that particular class by $2.5 million.

That’s a big number, but Baker notes that, at the individual level, this isn’t such a big deal: $250 a year for the average student.

Again, as the authors admit in the study, it would be helpful to be able to compare these economic gains with even the purely economic cost of mistakenly firing good teachers because of test scores. Or even the cost of carrying out the evaluations needed to support such high-stakes decisions.

And of course, even this assumes that the purpose of schools is to grow the economy. With that in mind, the last word in this too-long post goes to writer and mom Maria Bustillos, writing at The Awl:

I’m an ordinary parent and I do not give .5% of a tinker’s curse whether my kid earns an extra $250 a year or not. I know many parents who feel as I do about this. What we want for our kids is for them to grow up to be adults who are happy, well-informed, engaged citizens. Love their work and be good at it. Have a lot of pleasure and love in their lives, eyes open, good friends and good companions. Responsible, kind, reliable, open-minded people. Good parents themselves, maybe, someday. If anyone is going to be envisioning education reforms, then let him reckon with these principles first.

1 thought on “Big, impressive study, questionable policy conclusions

  1. Pingback: Live by the test, die by the test – but don’t teach to the test? « School Matters

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s