Math for America president: ‘Value-added’ flawed, but advocates won’t let go

John Ewing wrote a classic 2011 journal article, titled “Mathematical Intimidation,” that lamented the growing use of value-added models to evaluate schools and teachers. Three years later, he sees little evidence that education policy makers understand or care about the flaws in the approach.

Yes, he said, critics of value-added have grown more vocal. But lots of people with power and influence are still wedded to the idea that we can use test scores to identify bad teachers – and either weed them out of the profession or make them improve.

“People just can’t let it go,” Ewing told me this week. “Policy makers bought in, in a big way, and they can’t let go of it.”

John Ewing (Math for America)

John Ewing (Math for America)

Ewing, a mathematician, is president of Math for America, a New York-based organization that promotes mathematics education. He previously spent 14 years as executive director of the American Mathematical Society. Before that, he was an Indiana University math professor for two decades.

“Mathematical Intimidation” was directed at his fellow mathematicians, urging them to stand against policies that make bad use of their discipline. But it’s a concise, easy read. You don’t need to be a mathematician, or even know a lot of math, to follow its clear and persuasive argument.

Ewing wrote that that proponents of value-added use the supposed objectivity of the models – they’re based on mathematics, after all – to close off discussion of what the goals of education should be. But the models rest on a shaky foundation: The idea that standardized tests in math and English provide a valid and complete measure of what schools and teachers should accomplish. Continue reading

Advertisements

State Board of Education member visiting schools

State Board of Education member Gordon Hendry is doing a good thing by traveling around Indiana and meeting with teachers, administrators and school board members. Is he hearing what people say?

Friday he visited Bloomington High School North, from which he graduated in 1988. He met with administrators and a school board member, and talked with social studies teachers. Later, we spoke by phone about three big education issues facing the state.

Teacher evaluations

Hendry said much of the discussion at North involved state-mandated evaluations that require all teachers to be rated highly effective, effective, needs improvement or ineffective. He said the trick will be to craft evaluations that are accurate but don’t discourage teachers from collaborating.

“I don’t think I heard anyone say we don’t believe we should be accountable,” he said. “They want evaluations to be fair and to measure the right things. To me that makes sense.”

The 2011 law that required the evaluations says teachers rated needs improvement or ineffective can’t get a raise. With successive low ratings, they can be fired.

“It’s not only to identify teachers who need help,” Hendry said. “Just as important, it’s to identify teachers who are really doing an outstanding job.” Continue reading

Legislative oversight for the Indiana Department of Education

School Matters’ recap last week of Indiana’s 2012 education legislation missed this interesting and potentially significant item: Lawmakers voted to create a “select commission on education” to evaluate certain operations of the Indiana Department of Education and the State Board of Education.

The measure, added late in the process to House Enrolled Act 1376, a catch-all education and public administration bill, calls for specific focus on two areas: 1) the process and content of creating new metrics for giving schools A-to-F grades; and 2) the implementation of the new teacher evaluation system that the legislature approved last year. It adds that the commission may also take up any other education issue that members and legislative leaders deem necessary.

Why might lawmakers think that the Department of Education and State Board of Education could use some oversight? We can speculate:

// In 2010, Gov. Mitch Daniels and Superintendent of Public Instruction Tony Bennett pushed for a law that said third-graders who don’t pass a state reading test wouldn’t be promoted to fourth grade. After considerable debate, the legislature declined to approve the law. Instead, it passed a compromise measure that called for taking steps to ensure that all third-graders can read at grade level, “including retention as a last resort, after other methods of remediation have been evaluated or used, or both …(emphasis added).” The State Board of Education then adopted a rule that exactly mirrors the failed 2010 legislation: It says third-graders who don’t pass a state reading test won’t be promoted.

// In a classic case of putting the cart before the horse, the State Board of Education in November 2011 voted to let the state take over schools that get an F on state ratings for four consecutive years or a D or F for five straight years. Continue reading

Gates draws the line a ‘public shaming’

Microsoft chief Bill Gates made a strong argument last week that, while teachers should be evaluated in part on the value they add to their students’ test scores, those ratings shouldn’t be made public.

“Value-added ratings are one important piece of a complete personnel system,” Gates wrote in the New York Times. “But student test scores alone aren’t a sensitive enough measure to gauge effective teaching, nor are they diagnostic enough to identify areas of improvement. Teaching is multifaceted, complex work. A reliable evaluation system must incorporate other measures of effectiveness …”

This is interesting because Gates, through the education research and advocacy efforts of the Bill and Melinda Gates Foundation, has arguably done more than just about anyone to promote the belief that teachers should be evaluated on whether they improve student test scores. A majority of states, including Indiana, have latched onto the idea, adopting evaluation schemes that rely on test results.

But Gates, responding to a New York court decision, said it would be “a big mistake” to let just anyone know the results. “Developing a systematic way to help teachers get better is the most powerful idea in education today,” he wrote. “The surest way to weaken it is to twist it into a capricious exercise in public shaming.” Continue reading

Big, impressive study, questionable policy conclusions

A study of the impact of teachers on student success has been drawing lots of attention, including a big story in the New York Times, praise from columnist Nicholas Kristof and analysis in the blogosphere.

On the one hand, the paper by economists Raj Chetty, John Friedman and Jonah Rockoff offers new evidence that good teaching has long-lasting and far-reaching effects. This suggests that the recruitment, preparation and support of teachers should be a high priority for the nation.

But the economists also use their findings to call for rating teachers on the basis of “value-added” models, which use complex formulas to measure teachers’ impact on student test scores – and for firing teachers who don’t measure up. Annie Lowrey writes in the Times:

The authors argue that school districts should use value-added measures in evaluations, and to remove the lowest performers, despite the disruption and uncertainty involved.

“The message is to fire people sooner rather than later,” Professor Friedman said.

Professor Chetty acknowledged, “Of course there are going to be mistakes — teachers who get fired who do not deserve to get fired.” But he said that using value-added scores would lead to fewer mistakes, not more.”

This is a little surprising, given that, in the study itself, they caution against that sort of policy conclusion.

“Overall, our study shows that great teachers create great value and that test score impacts are helpful in identifying such teachers. However, more work is needed to determine the best way to use VA for policy,” they write in the executive summary.

They add that two important questions must be resolved before value-added models are used to evaluate teachers. One is whether attaching high stakes to test scores will skew results so much that it undermines the accuracy of the models. The other has to do with the economic cost of firing teachers, sometimes in error – the very “mistakes” that Chetty said would be trivial.

Then the New York Times calls and they throw caution to the wind.

The study reportedly breaks new ground Continue reading

A ‘scarlet letter’ for bad teachers?

Today Indiana residents can go to the State Department of Education website and learn which schools in their community have earned As, Bs, Cs, Ds and Fs from the state. In the future, will we be able to find out which teachers are rated as highly effective, effective, improvement needed and ineffective?

Probably, even though the state the law that prescribes a new system of teacher evaluation was supposed to prevent that from happening.

The law, Senate Enrolled Act 1, orders schools to adopt an evaluation system that rates all teachers by those four categories of effectiveness. It requires school districts to report aggregate data on their teachers’ ratings to the Department of Education once a year – i.e., how many are rated highly effective, how many are ineffective and so on. But those reports, the law says, may not include names or any other personally identifiable information regarding the teachers.

So we presumably won’t be able to get information about individual teachers from DOE officials, because they won’t have it. But can we get it from local school districts?

Maybe. If a school district has a list of teachers and their effectiveness ratings, it would be a public record. If we make a request under the Indiana Access to Public Records Act to inspect or copy the list, the district would arguably have to make it available. It wouldn’t appear to fall under any of the exceptions in Section 4 of the act, which lists types of records that don’t have to be made public.

School attorneys may dispute this. They could argue that evaluations won’t have to be disclosed because they are part of teachers’ personnel files. But even if they’re right, there are two ways in which some information about evaluations will become public.

First, SEA 1 specifies that teachers who are rated ineffective or improvement necessary may not receive any increase in pay. It’s well established that salaries for employees of public schools are public information. (Some newspapers make such data available online). Once the new law is in effect, we can compare any teacher’s salary for 2012-13 and 2013-14. If the teacher gets a raise, we know he or she was rated effective or highly effective. If not, we may surmise otherwise.

Second, SEA 1 says schools should avoid putting a student in a classroom taught by an ineffective teacher for two straight years. But if the school can’t avoid doing so – for example, the student had an ineffective teacher in third grade and the school has only one fourth-grade teacher, who is also ineffective – then the school must inform the student’s parents. There’s nothing to stop the parents from going public.

Interestingly, it’s not just teachers unions that think this is a bad idea. The National Council on Teacher Quality, in a lengthy report that largely praises Indiana and a few other states for their changes in teacher evaluation, calls this a “scarlet letter” approach and says it is bad policy.

“NCTQ thinks this does a tremendous disservice to the teaching profession,” the report says. “If a district has evidence that a teacher is ineffective, state policy should provide the means for the district to take the necessary steps to remove the individual from the classroom, not humiliate the teacher. Reporting on teacher effectiveness data by the state, district and school level is essential. But when it comes to accountability for ineffective teachers, sending a note home to let families know their child’s teacher is not so good is no solution at all.”

Teacher evaluation: It’s a race, but to where?

New York Times education columnist Michael Winerip visited Tennessee recently to check on that state’s implementation of performance-based evaluations of teachers. What he found wasn’t pretty.

Principals bogged down by non-stop evaluations, paperwork and micro-managing by the state. Teachers judged on student test results in subjects they don’t teach. Lousy morale all around.

“I’ve never seen such nonsense,” Will Shelton, principal of Blackman Middle School in Murfreesboro, told Winerip. “In the five years I’ve been principal here, I’ve never known so little about what’s going on in my own building.”

After winning a half billion dollars in the Obama administration’s Race to the Top competition last year, Tennessee rushed to implement a system that bases 50 percent of teachers’ evaluations on student test scores. It’s apparently easier said than done.

“Because there are no student test scores with which to evaluate over half of Tennessee’s teachers — kindergarten to third-grade teachers; art, music and vocational teachers — the state has created a bewildering set of assessment rules,” Winerip writes. “Math specialists can be evaluated by their school’s English scores, music teachers by the school’s writing scores.”

After recounting several examples that sound flat-out absurd, he adds, “This would all be hilarious, except these evaluations can cost people their jobs.”

Probably Winerip could have found some principals – and maybe even some teachers – who like what Tennessee is doing. A New York Times editorial, also last week, insisted that Tennessee officials “must resist any backsliding” on teacher evaluation. And business leaders have come out strongly against tweaking the law to make it more workable, according to the Nashville Tennessean.

Tennessee may be leading the race to test-based teacher evaluation, but many other states are following close behind – including Indiana with its implementation of Senate Enrolled Act 1. So it makes sense to follow what’s going on in Tennessee, good or bad.

The Indiana State Department of Education’s teacher evaluation model, called RISE, says 35 percent of some teachers’ evaluations will be based on improvement in their students’ scores on standardized tests. Teachers in subject areas that aren’t covered by state tests will have 20 percent of their evaluation based on meeting “student learning outcomes” and 5 percent based on school-wide success.

Six Indiana school districts are piloting RISE and other teacher evaluation models this school year. Other districts will have to start implementing new teacher-evaluation schemes in 2012-13.

RISE may sound reasonable, except that 1) Research casts doubt on whether tests scores can be reliably used to make high-stakes decisions about teachers (The august National Research Council says don’t do it); and 2) it seems questionable that Indiana school principals, or their designees, will have the time and resources necessary to carry out the extensive teacher evaluations called for by the system.

And as teacher-evaluation guru Charlotte Danielson warned at a school-reform conference this week in Indianapolis, teachers who are penalized as the result of unfair evaluations are likely to sue. “And they’ll probably win,” she said, according to the Indianapolis Star.