PISA testing glitch a lesson for Indiana

Back in 2006, there was an apparently minor glitch in administering the Programme for International Student Assessment, a system of student tests given by the Organisation for Economic Cooperation and Development. Some U.S. reading-test booklets were misprinted and included confusing directions.

The result: The OECD didn’t even report PISA reading scores for the United States. Officials reasoned the scores wouldn’t be valid and they shouldn’t put out bad results.

Leslie Rutkowski, an assistant professor at the Indiana University School of Education and an expert on statistical modeling, brought up the PISA experience in connection with the problems Indiana experienced with its 2013 ISTEP-Plus exam. Testing was plagued by disruptions traced to the computer servers of CTB/McGraw-Hill, Indiana’s test contractor.

“I feel like that is such a telling anecdote,” Rutkowski said. “PISA is low-stakes: The test doesn’t have consequences for students or for schools. And for something as small as a printing error to invalidate the results … that was something that really resonated with me.”

ISTEP, by contrast, is a high-stakes test – all the more reason there needs to be full confidence in its results. “We’re making decisions about whether a child graduates, whether a teacher keeps her job, whether a school stays open” on the basis of test scores, Rutkowski said. “If the data is questionable in any way, I don’t think we can use it.”

Indiana Superintendent of Public Instruction Glenda Ritz appears to agree. The state Department of Education contracted with the New Hampshire-based National Center for the Improvement of Educational Assessment to determine if ISTEP results are valid. State lawmakers, meanwhile, are calling CTB/McGraw-Hill President Ellen Haley to appear before a legislative committee June 21 to explain what went wrong.

But educators won’t have much faith in this year’s ISTEP, regardless of what the DOE study finds. Superintendents in Fort Wayne and Evansville said they consider the results invalid. At Monroe County Community Schools in Bloomington, two-thirds of students experienced disruptions on the first day of testing. Even after the server problems were supposedly fixed, problems continued with delayed computer connections, inaccurate answer choices and other issues.

“We do not believe that the 2013 scores will ever be reliable or valid,” Monroe County Superintendent Judy DeMuth said in a statement on the district’s website. “The entire testing session should be discounted and/or vacated.”

Indiana law builds a great deal of school accountability around ISTEP. It’s the main factor in determining school grades, and it’s supposed to weigh heavily in teacher evaluations. High-school students who don’t pass algebra and English exams need a waiver to graduate. It’s not immediately clear if Ritz has the authority to lessen the consequences.

But if nothing else, the ISTEP debacle suggests Indiana has erected a massive system for judging schools, students and teachers on a pretty shaky foundation.

Standardized tests have a place in education, Rutkowski said. But things get sketchy when we assume test scores can tell us whether students are getting the education they deserve and whether schools and teachers are getting the job done.

“I’m a methodologist by training,” she said, “and I think we need to sit down and look at the assumptions we’re making about testing, to make sure they hold up under critical analysis.”


1 thought on “PISA testing glitch a lesson for Indiana

  1. Thanks Steve for another excellent piece. This is an important issue and the PISA folks got it right in understanding what “standardized” means.

    Phil Harris

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s