Examining data for Indiana’s ‘disaggregated groups’

Here’s a question that arguably deserves more attention from education researchers and policy types: Why are some schools better than others at getting students from low-income families to pass tests?

We hear a lot about high-poverty schools that produce better test scores than you’d expect. We pay a lot of attention to no-excuses charter schools and public schools that focus relentlessly on data. But poor kids are scattered throughout all kinds of schools and school districts, urban, rural and suburban. And judging by test scores, some districts do a better job of helping them learn than others.

The Indiana Department of Education recently posted district-by-district and school-by-school passing rates on the ISTEP+ exam for “disaggregated groups” of students: minorities, students who qualify for free or reduced-price lunches, English language learners and special-needs students.

The data are a carry-over from the No Child Left Behind Act, which required schools to hit targets for the percentage of students in each group who passed standardized tests.

The results vary from school to school – a lot. Looking at students who qualify for free and reduced-price lunches, for example, the proportion who passed both the math and English ISTEP+ exams in 2014 ranged from 85.9 percent to 45.2 percent. The state average was 62.3 percent.

Some of the districts with the lowest passing rates for free-and-reduced lunch students are high-poverty urban districts. But some aren’t. Some of the districts with the highest passing rates are low-poverty schools with relatively few poor students. But some aren’t. It’s a mix, with no obvious pattern.

Here’s a chart that plots districts’ ISTEP+ passing rates against their poverty rates. Each dot represents an Indiana school corporation. The horizontal axis is the percentage of free-and-reduce-price lunch students who pass both the math and English tests. The vertical axis is the percent of free-lunch students in each district. There’s a correlation – a tendency for schools with more poverty to have lower test-passing rates. But it’s not what you’d call a tight fit.

poverty & test scores chart

Passing rates for free-reduced lunch students vs. percent of free-lunch students.

By contrast, ISTEP+ passing rates vary less for students from higher-income families. In nearly four of five school districts, passing rates are 80 percent or higher for students who pay for school lunches.

Another way to examine the data is to look at the gap between passing rates for paid-lunch students and free-and-reduced-price lunch students. In some districts, there’s virtually no gap at all. In others, passing rates are as much as 30 percentage points higher for paid-lunch students. Why would that be?

Some caveats:

  • Researchers have pointed out that free-and-reduced lunch data aren’t a very good measure of educational disadvantage. There’s poverty and then there’s poverty.
  • And reduced-price lunch families … that could be a teacher, journalist or social worker with one kid in school and a spouse who’s staying home for now to care for the preschooler.
  • Test scores are without question a flawed measure of school quality. No doubt some schools and some districts focus on test prep to the exclusion of more important activities.

That said, the data for disaggregated groups of students should give us something to think about. It’s disconcerting that the Monroe County Community School Corp. – the district where I live – has test performance gaps that are among the largest in the state.

For readers who can’t get enough data, below are links to spreadsheets with information for the following (Only districts with 100 or more students in each subgroup are included):

12 thoughts on “Examining data for Indiana’s ‘disaggregated groups’

  1. Consider also the recent recession where many families lost a job, sometimes both — the motivation and values were set in many children before the recession, and even though they may “look” poverty on paper, during the years that really counted about education and family values regarding it, they function in school like their better off peers. It is an interesting variable that has not been studied, but it probably has an effect when “poverty” is considered. There is a big difference between generational, chronic poverty than those that entered poverty because of an event in history and may be living in a family that is still trying to get back on their feet.

  2. Hi Steve, this is very interesting. I think it’s important to make a distinction between “getting kids to pass tests” and “providing a good education.” For instance, we are hearing from concerned parents in Monroe County of at least one class, perhaps one school, that has decided not to do much in the way of a 6th grade social studies/history curriculum because the school is under pressure to raise scores–and that subject is not part of ISTEP testing for that grade. (At least, that’s my understanding.) What if the school successfully raises scores? (Personally, I doubt that omitting subject matter is a way to get students to thrive or score well in the long term, but it’s conceivable that there would be short term gains.) Would that be evidence of a good education?

    • Thanks, Jenny. That’s certainly true, and one of the caveats that I included. I saw a Facebook post about a local school neglecting social studies, and that’s troubling. Seems like it would be a self-defeating approach in terms of facilitating literacy and other qualities that are supposed to be part of the standards.

      • The ISTEP data just does not give enough information to be able to make pronouncements about the quality of a school or district. It does not give essential information about school culture, hands-on learning, energetic literacy, or student engagement. It does not tell us if students are getting p.e., recess, art, music, science, and history. In stories across the country, most schools that had a level of cheating going on, now exposed, were first lauded by their districts and by journalists and city councils as well. Where’s the line between suspicious and laudatory? The shallowness of this measure and its corruptibility make it close to worthless. At the same time, Steve makes a very interesting observation about the distance between student performance. I would argue that to understand what that reflects would take visiting schools, meeting students and teachers, attending school board meetings, and talking with parents. It might mean gathering other forms of data as well and searching for correlations.

  3. Don’t forget to factor in the type of poverty . My guess is that schools that deal with intrenched generational poverty tend to perform at a
    lower level than a system afftected by current economic status that has created “new poverty”. But Indiana does need to be concerned about the new poverty, because we are creating it at an alarming rate.

    • OK, I know very little about statistics, but I ran the Excel formula correctly, the correction is -0.461169393. What’s your take on what that tells us?

      • Hi Steve, thanks for the information. I’d like to send you a short paper I wrote on the same topic (posted on Diane Ravitch’s blog). Actually, it’s possible that I sent it a few months ago. My correlation coefficient was much higher but I relied on larger sample sizes. The smaller the sample, the greater the variability. Unless we can show high performance over a number of years in high poverty districts, I would not trust the results.

  4. Steve – thanks for posting. Your thoughts, as always, are both challenging and nuanced.

    In response to the “test scores don’t tell us much about the school” commentary, I must agree. Of course ISTEP doesn’t tell us the rich story of education within a school. It’s there to test minimum competency on grade-level standards. To understand the full scope of what happens within a school, there is much more research to be done. I understand the concern about judging a school based on these scores.

    However, to dismiss the gaps in ISTEP scores that Steve points out by saying that ISTEP doesn’t give us the full story really misses the opportunity to have the conversation that Steve is trying to spark (I think): Some schools are doing a better job of getting more of their free/reduced lunch population over that minimum competency mark. How are they doing it? In the chart he posted, there are some high-achieving outliers with high free/reduced populations. Can other schools learn something from them? Locally – what can Monroe County do to improve education for free/reduced lunch students? (I think his post about the segregation of the schools might loosely relate.)

    It stands to reason that if the students master the basic skills then there’s much more to build from in the classroom (hands-on learning is more rich, projects more meaningful, cross-content lessons more feasible). The fact that a predictable portion of our population regularly performs below other predictable portions on tests of basic competency should be cause for alarm. This conversation is as important as it is complicated.

    Data are never perfect. Numbers never tell the whole story. But data can be powerful, if only to motivate people to ask questions – questions that they may not have known to ask before. We can always strive to get better data, but let’s work with what we have to improve where we can!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s