Math for America president: ‘Value-added’ flawed, but advocates won’t let go

John Ewing wrote a classic 2011 journal article, titled “Mathematical Intimidation,” that lamented the growing use of value-added models to evaluate schools and teachers. Three years later, he sees little evidence that education policy makers understand or care about the flaws in the approach.

Yes, he said, critics of value-added have grown more vocal. But lots of people with power and influence are still wedded to the idea that we can use test scores to identify bad teachers – and either weed them out of the profession or make them improve.

“People just can’t let it go,” Ewing told me this week. “Policy makers bought in, in a big way, and they can’t let go of it.”

John Ewing (Math for America)

John Ewing (Math for America)

Ewing, a mathematician, is president of Math for America, a New York-based organization that promotes mathematics education. He previously spent 14 years as executive director of the American Mathematical Society. Before that, he was an Indiana University math professor for two decades.

“Mathematical Intimidation” was directed at his fellow mathematicians, urging them to stand against policies that make bad use of their discipline. But it’s a concise, easy read. You don’t need to be a mathematician, or even know a lot of math, to follow its clear and persuasive argument.

Ewing wrote that that proponents of value-added use the supposed objectivity of the models – they’re based on mathematics, after all – to close off discussion of what the goals of education should be. But the models rest on a shaky foundation: The idea that standardized tests in math and English provide a valid and complete measure of what schools and teachers should accomplish. Continue reading

State Board of Education member visiting schools

State Board of Education member Gordon Hendry is doing a good thing by traveling around Indiana and meeting with teachers, administrators and school board members. Is he hearing what people say?

Friday he visited Bloomington High School North, from which he graduated in 1988. He met with administrators and a school board member, and talked with social studies teachers. Later, we spoke by phone about three big education issues facing the state.

Teacher evaluations

Hendry said much of the discussion at North involved state-mandated evaluations that require all teachers to be rated highly effective, effective, needs improvement or ineffective. He said the trick will be to craft evaluations that are accurate but don’t discourage teachers from collaborating.

“I don’t think I heard anyone say we don’t believe we should be accountable,” he said. “They want evaluations to be fair and to measure the right things. To me that makes sense.”

The 2011 law that required the evaluations says teachers rated needs improvement or ineffective can’t get a raise. With successive low ratings, they can be fired.

“It’s not only to identify teachers who need help,” Hendry said. “Just as important, it’s to identify teachers who are really doing an outstanding job.” Continue reading

Legislative oversight for the Indiana Department of Education

School Matters’ recap last week of Indiana’s 2012 education legislation missed this interesting and potentially significant item: Lawmakers voted to create a “select commission on education” to evaluate certain operations of the Indiana Department of Education and the State Board of Education.

The measure, added late in the process to House Enrolled Act 1376, a catch-all education and public administration bill, calls for specific focus on two areas: 1) the process and content of creating new metrics for giving schools A-to-F grades; and 2) the implementation of the new teacher evaluation system that the legislature approved last year. It adds that the commission may also take up any other education issue that members and legislative leaders deem necessary.

Why might lawmakers think that the Department of Education and State Board of Education could use some oversight? We can speculate:

// In 2010, Gov. Mitch Daniels and Superintendent of Public Instruction Tony Bennett pushed for a law that said third-graders who don’t pass a state reading test wouldn’t be promoted to fourth grade. After considerable debate, the legislature declined to approve the law. Instead, it passed a compromise measure that called for taking steps to ensure that all third-graders can read at grade level, “including retention as a last resort, after other methods of remediation have been evaluated or used, or both …(emphasis added).” The State Board of Education then adopted a rule that exactly mirrors the failed 2010 legislation: It says third-graders who don’t pass a state reading test won’t be promoted.

// In a classic case of putting the cart before the horse, the State Board of Education in November 2011 voted to let the state take over schools that get an F on state ratings for four consecutive years or a D or F for five straight years. Continue reading

Gates draws the line a ‘public shaming’

Microsoft chief Bill Gates made a strong argument last week that, while teachers should be evaluated in part on the value they add to their students’ test scores, those ratings shouldn’t be made public.

“Value-added ratings are one important piece of a complete personnel system,” Gates wrote in the New York Times. “But student test scores alone aren’t a sensitive enough measure to gauge effective teaching, nor are they diagnostic enough to identify areas of improvement. Teaching is multifaceted, complex work. A reliable evaluation system must incorporate other measures of effectiveness …”

This is interesting because Gates, through the education research and advocacy efforts of the Bill and Melinda Gates Foundation, has arguably done more than just about anyone to promote the belief that teachers should be evaluated on whether they improve student test scores. A majority of states, including Indiana, have latched onto the idea, adopting evaluation schemes that rely on test results.

But Gates, responding to a New York court decision, said it would be “a big mistake” to let just anyone know the results. “Developing a systematic way to help teachers get better is the most powerful idea in education today,” he wrote. “The surest way to weaken it is to twist it into a capricious exercise in public shaming.” Continue reading

Big, impressive study, questionable policy conclusions

A study of the impact of teachers on student success has been drawing lots of attention, including a big story in the New York Times, praise from columnist Nicholas Kristof and analysis in the blogosphere.

On the one hand, the paper by economists Raj Chetty, John Friedman and Jonah Rockoff offers new evidence that good teaching has long-lasting and far-reaching effects. This suggests that the recruitment, preparation and support of teachers should be a high priority for the nation.

But the economists also use their findings to call for rating teachers on the basis of “value-added” models, which use complex formulas to measure teachers’ impact on student test scores – and for firing teachers who don’t measure up. Annie Lowrey writes in the Times:

The authors argue that school districts should use value-added measures in evaluations, and to remove the lowest performers, despite the disruption and uncertainty involved.

“The message is to fire people sooner rather than later,” Professor Friedman said.

Professor Chetty acknowledged, “Of course there are going to be mistakes — teachers who get fired who do not deserve to get fired.” But he said that using value-added scores would lead to fewer mistakes, not more.”

This is a little surprising, given that, in the study itself, they caution against that sort of policy conclusion.

“Overall, our study shows that great teachers create great value and that test score impacts are helpful in identifying such teachers. However, more work is needed to determine the best way to use VA for policy,” they write in the executive summary.

They add that two important questions must be resolved before value-added models are used to evaluate teachers. One is whether attaching high stakes to test scores will skew results so much that it undermines the accuracy of the models. The other has to do with the economic cost of firing teachers, sometimes in error – the very “mistakes” that Chetty said would be trivial.

Then the New York Times calls and they throw caution to the wind.

The study reportedly breaks new ground Continue reading

A ‘scarlet letter’ for bad teachers?

Today Indiana residents can go to the State Department of Education website and learn which schools in their community have earned As, Bs, Cs, Ds and Fs from the state. In the future, will we be able to find out which teachers are rated as highly effective, effective, improvement needed and ineffective?

Probably, even though the state the law that prescribes a new system of teacher evaluation was supposed to prevent that from happening.

The law, Senate Enrolled Act 1, orders schools to adopt an evaluation system that rates all teachers by those four categories of effectiveness. It requires school districts to report aggregate data on their teachers’ ratings to the Department of Education once a year – i.e., how many are rated highly effective, how many are ineffective and so on. But those reports, the law says, may not include names or any other personally identifiable information regarding the teachers.

So we presumably won’t be able to get information about individual teachers from DOE officials, because they won’t have it. But can we get it from local school districts?

Maybe. If a school district has a list of teachers and their effectiveness ratings, it would be a public record. If we make a request under the Indiana Access to Public Records Act to inspect or copy the list, the district would arguably have to make it available. It wouldn’t appear to fall under any of the exceptions in Section 4 of the act, which lists types of records that don’t have to be made public.

School attorneys may dispute this. They could argue that evaluations won’t have to be disclosed because they are part of teachers’ personnel files. But even if they’re right, there are two ways in which some information about evaluations will become public.

First, SEA 1 specifies that teachers who are rated ineffective or improvement necessary may not receive any increase in pay. It’s well established that salaries for employees of public schools are public information. (Some newspapers make such data available online). Once the new law is in effect, we can compare any teacher’s salary for 2012-13 and 2013-14. If the teacher gets a raise, we know he or she was rated effective or highly effective. If not, we may surmise otherwise.

Second, SEA 1 says schools should avoid putting a student in a classroom taught by an ineffective teacher for two straight years. But if the school can’t avoid doing so – for example, the student had an ineffective teacher in third grade and the school has only one fourth-grade teacher, who is also ineffective – then the school must inform the student’s parents. There’s nothing to stop the parents from going public.

Interestingly, it’s not just teachers unions that think this is a bad idea. The National Council on Teacher Quality, in a lengthy report that largely praises Indiana and a few other states for their changes in teacher evaluation, calls this a “scarlet letter” approach and says it is bad policy.

“NCTQ thinks this does a tremendous disservice to the teaching profession,” the report says. “If a district has evidence that a teacher is ineffective, state policy should provide the means for the district to take the necessary steps to remove the individual from the classroom, not humiliate the teacher. Reporting on teacher effectiveness data by the state, district and school level is essential. But when it comes to accountability for ineffective teachers, sending a note home to let families know their child’s teacher is not so good is no solution at all.”

Teacher evaluation: It’s a race, but to where?

New York Times education columnist Michael Winerip visited Tennessee recently to check on that state’s implementation of performance-based evaluations of teachers. What he found wasn’t pretty.

Principals bogged down by non-stop evaluations, paperwork and micro-managing by the state. Teachers judged on student test results in subjects they don’t teach. Lousy morale all around.

“I’ve never seen such nonsense,” Will Shelton, principal of Blackman Middle School in Murfreesboro, told Winerip. “In the five years I’ve been principal here, I’ve never known so little about what’s going on in my own building.”

After winning a half billion dollars in the Obama administration’s Race to the Top competition last year, Tennessee rushed to implement a system that bases 50 percent of teachers’ evaluations on student test scores. It’s apparently easier said than done.

“Because there are no student test scores with which to evaluate over half of Tennessee’s teachers — kindergarten to third-grade teachers; art, music and vocational teachers — the state has created a bewildering set of assessment rules,” Winerip writes. “Math specialists can be evaluated by their school’s English scores, music teachers by the school’s writing scores.”

After recounting several examples that sound flat-out absurd, he adds, “This would all be hilarious, except these evaluations can cost people their jobs.”

Probably Winerip could have found some principals – and maybe even some teachers – who like what Tennessee is doing. A New York Times editorial, also last week, insisted that Tennessee officials “must resist any backsliding” on teacher evaluation. And business leaders have come out strongly against tweaking the law to make it more workable, according to the Nashville Tennessean.

Tennessee may be leading the race to test-based teacher evaluation, but many other states are following close behind – including Indiana with its implementation of Senate Enrolled Act 1. So it makes sense to follow what’s going on in Tennessee, good or bad.

The Indiana State Department of Education’s teacher evaluation model, called RISE, says 35 percent of some teachers’ evaluations will be based on improvement in their students’ scores on standardized tests. Teachers in subject areas that aren’t covered by state tests will have 20 percent of their evaluation based on meeting “student learning outcomes” and 5 percent based on school-wide success.

Six Indiana school districts are piloting RISE and other teacher evaluation models this school year. Other districts will have to start implementing new teacher-evaluation schemes in 2012-13.

RISE may sound reasonable, except that 1) Research casts doubt on whether tests scores can be reliably used to make high-stakes decisions about teachers (The august National Research Council says don’t do it); and 2) it seems questionable that Indiana school principals, or their designees, will have the time and resources necessary to carry out the extensive teacher evaluations called for by the system.

And as teacher-evaluation guru Charlotte Danielson warned at a school-reform conference this week in Indianapolis, teachers who are penalized as the result of unfair evaluations are likely to sue. “And they’ll probably win,” she said, according to the Indianapolis Star.

Indiana’s teacher evaluation law: some promise, lots of peril

A policy brief released last week by the Center on Evaluation & Education Policy at Indiana University spells out a number of reasons to be concerned about Indiana’s new teacher-evaluation law.

The law, Senate Enrolled Act 1, requires annual teacher evaluations that rely significantly on student test results. All teachers will be put in one of four categories: highly effective, effective, improvement needed and ineffective. Compensation and tenure will be tied to the evaluations, and teachers in the two lower categories won’t get raises and may be at risk of losing their jobs.

The law requires schools to start implementing the new evaluation and merit-pay system in 2012-13. But as the authors of the CEEP report, Rodney Whiteman, Dingjing Shi and Jonathan Plucker, point out, this won’t be easy, and there are bound to be unintended consequences.

Here are a few of the issues that they highlight:

— Tying teacher evaluations to student performance may seem like a no-brainer. But in fact, there’s not much evidence that you can accurately attribute student test scores, or even improvement in test scores, to the effect of a single teacher. Standardized tests are designed to measure whether students meet grade-level standards, not how much they improve from year to year. And can schools develop tests that measure teacher effectiveness in subjects like art, music and physical education?

Research has shown that students learn best when teachers collaborate and share expertise. But most schools will have a limited amount of money available for raises, and it must all go to teachers rated effective and highly effective. “This creates an indirect competition for compensation and an incentive to out-perform colleagues,” the brief says. “Teachers may begin viewing their materials and techniques as proprietary intellectual property and elect to not share that property with their competitors.”

— Finally – and this may be the biggest issue – where will schools will find the time, resources and expertise to design and implement annual evaluations that are fair and meaningful? SEA 1 is an “unfunded mandate,” the authors say. At a time when most Indiana school districts are pinched for money, they may have pull some of their best teachers out of the classroom to develop and carry out evaluations; either that or hire outside experts, at considerable cost.

The brief concludes, “If done correctly, with sufficient time, finances, and people, changing teacher evaluation can be a powerful reform in public education.”

That’s a mighty big if.

The debate on tying test scores to evaluations: teachers vs. policy analysts

The New York Times did a “room for debate” feature this week on the growing practice of using student test scores to evaluate teachers. And it included a couple of teachers among the eight people selected to debate the issue – something that seems to almost never happen when education policy is discussed.

Not surprisingly, the teachers weren’t exactly crazy about the idea.

“This testing-students-to-grade-teachers initiative is not coming out of what people who actually work with children in schools know,” writes New York City teacher Francesa Burns. “It is not even research-based … Instead, the plans are based on politics and sound bites, corporate sleight of hand … and high talk. In short: nothing.”

Molly Putnam, a high-school teacher in Brooklyn, says the money going to develop more tests should instead “be spent on methods that have been proven to improve teacher quality and retention rates — like intensive student teaching and training in lesson planning, instruction and classroom management. A culture change would also mean having principals and senior teachers become even more engaged in mentoring and guiding younger teachers.”

A couple of policy analysts, Kevin Carey of Education Sector and Marcus Winters of the Manhattan Institute, argue that it only makes sense to use test scores to gauge how well teachers do their job. Carey approvingly cites a recent National Education Association policy statement that allows for using “valid, reliable, high quality standardized tests” to help evaluate teachers.

But another analyst, Michael Petrilli of the Thomas B. Fordham Institute, says using centralized, bureaucratic tests to evaluate teachers is like “attacking a fly with a sledgehammer.” His alternative: Give principals the power to fire bad teachers and increase pay for good teachers. “If we can’t trust school leaders to identify their best and worst teachers, then the whole project of school reform is sunk,” he says.

Indiana will use test results as a “significant” part of teacher evaluations beginning with the 2012-13 school year. But the state has a long way to go to figure out how this will work. To that end, the Department of Education chose six school districts – Bloomfield, Greensburg, Fort Wayne, Beech Grove, Bremen and Warren Township (Indianapolis) – to test out the new evaluation systems in 2011-12.

“Things are changing in Indiana in education,” Bloomfield superintendent Dan Sichting tells Bethany Nolan of the Bloomington Herald-Times. “Some people would argue it’s changing for the better, others not for the better. But if we sit back and don’t participate, we’re not going to have any kind of input in what the final product will be.”

Note to journalists: It could make an interesting project to follow one of those districts over the next year and track the results, good or bad, of changing how teachers are evaluated.

Indiana education legislation is available online

The Daniels-Bennett education agenda is showing up in bills posted to the Indiana General Assembly’s website, and so are other school-related measures. Here’s a preliminary look, starting with the proposals backed by the governor and state superintendent:

HB 1337, teacher contracts: rewrites the collective bargaining law; creates a new system for evaluating teachers, including performance-based criteria; lessens seniority protections; limits bargaining to wages and wage-related benefits.

HB 1002, charter schools: establishes a state charter schools board; allows private colleges and mayors of second-class cities to sponsor charter schools; makes other changes to encourage charters.

SB 496 and HB 1250, “parent trigger” law: allows a majority of parents in a low-performing school to have it closed, convert it to a charter school, or use its funding to send their children to private schools.

HB 1249, early graduation: diverts school funding to college scholarships for students who finish high school early; directs the State Board of Education to establish procedures for completing school requirements by the end of 11th grade.

Here are other education-related bills that aren’t part of the official agenda:

HB 1238, referendum gag rule: restricts school corporations, school boards and school employees from advocating a tax increase to support schools.

SB 410, fund transfers: extends through 2012 the option that school corporations can transfer half their capital projects tax levy to support operating expenses.

SB 326, school board elections: requires board members to be chosen in partisan elections, running as representatives of political parties.

SB 171 and HB 1195, school start date: delays the start of the school year until after Labor Day (Senate) or Sept. 1 (House).

You can peruse legislation by category at the General Assembly website, where there are about 20 subcategories under “schools” and more bills are being posted every day. Keep in mind that bills can be debated and amended numerous times, in committees, the House and the Senate. Some of these may not even be considered. But the ones that make up the Daniels-Bennett agenda are likely to spark interesting discussions.