More evidence against the myth of charter school superiority

Conventional public schools in Indiana are doing better than charter schools when it comes to helping the state’s poorest children achieve passing grades on state tests. Not a lot better, but enough that it should give pause to those who assume charter schools are superior and we need more of them.

This claim rests on a simple comparison of 2013 ISTEP-Plus passing rates for charter schools and district public schools where more than 80 percent of students qualified for free and reduced-price school lunches. On average, students in the district public schools were more likely to pass the tests.

Now this isn’t sophisticated research, and it certainly doesn’t prove charter schools are inferior. It looks only at the schools’ overall passing rates for math and English/language arts exams, which of course make up just one small measure of school effectiveness. It says nothing about individual schools.

But neither should the result be particularly surprising. It meshes with what University of Illinois professors Christopher Lubienski and Sarah Theule Lubienski have found: that conventional public schools, on the whole, outperform not only charter schools but also private schools.

Other research, such as the 2013 CREDO study often cited by charter-school supporters, show almost no difference between the types of schools. (A 2011 CREDO study of Indiana schools concluded that charter schools, overall, performed better).

I merged Department of Education spreadsheets with data on free and reduced-price lunch counts and ISTEP-Plus passing rates. Then I sorted by free-and-reduced-lunch rates and focused on schools where 80 percent or more students qualified for lunch assistance. Results include:


  • For charter schools: passing rate for E/LA, 62.3 percent; passing rate for math, 62.5 percent.
  • For conventional public schools: passing rate for E/LA, 64.1 percent; passing rate for math, 68.1 percent.

The data set includes only schools that enroll students in grades 3-8, who take ISTEP exams; it excludes high schools and many primary-grade schools. I also tried to screen out nonstandard schools such as juvenile detention centers and dropout recovery schools.

Again, the differences aren’t huge but they appear to be significant. Suppose students in high-poverty traditional public schools passed the tests at the same rate as students in high-poverty charter schools. The result would have been nearly 1,000 fewer students passing in E/LA and 2,000 fewer in math.

Why does conventional wisdom say charter schools – and, for that matter, private schools – are better? As College of Holy Cross professor Jack Schneider explains, some of it is the exclusivity factor and some of it is the value that attaches to “choice”: People feel something must be better if they chose it rather than being assigned to it. But a lot of it is simply that charter schools and private schools have done a better job of “shoring up their brands” and boosting their reputations via marketing.

“Traditional public schools need not build their brands in order to ensure survival,” Schneider writes. “After all, they educate 90 percent of young people. But they may need to do so in order to secure the good faith of the American public — faith that is essential to a healthy and thriving system.”

Note: As always, readers are encouraged to check the calculations for this post and invited to challenge its conclusions and suggest alternate interpretations. A previous version of this post included a potentially misleading comparison of percentages of students who passed both ELA and math exams. 


22 thoughts on “More evidence against the myth of charter school superiority

  1. “Note: As always, readers are encouraged to check the calculations for this post and invited to challenge its conclusions and suggest alternate interpretations.”

    Yes, Steve, you majored in English and spent most of your working career as a journalist, but… You think like a scientist!

  2. In keeping with the act of: “I also tried to screen out nonstandard schools such as juvenile detention centers and dropout recovery schools,” you should remove the turnaround academies from the charter school data. They are not considered charter schools.

    • Maybe, but they operate more like charter schools, with charter-like independence and open enrollment, right? But it’s a reasonable point, especially in that they’re new and wouldn’t necessarily be expected to hit it out of the park in their first year. I think there are only two turnarounds that give ISTEP: Gary Roosevelt and Indy Arlington. If I remove those, charter pass rates are 63.8 percent for ELA and 63.9 percent for math — a little better but not much.

  3. Like you say, it’s hard to draw too many conclusions from this data. It may not definitively show that public schools are better. The small advantage public schools have could be something as simple as the positive effect of having more experienced teachers.

    However, more importantly, what the data does show is that public are not worse than charters. My guess is that if you asked the general public which is better, the majority would identify charters as superior.

    All they here from politicians and the media in general is how bad public schools are and that charters are the simple answer to a complex problem- how to best educate high poverty students.
    Charters have a huge advantage in public perception despite their relatively weak performance.

    • We say on one hand it’s hard to draw conclusions (without seeing more detailed analysis, I agree) and then on the other appear to use the data to against charters, i.e. relatively weak performance. It seems to me the article actually makes the point they both have weak performance.

    • Inteach, I agree, and that’s the primary reason I wrote this. According to the Gallup poll that Jack Schneider cites (linked in the post), 60 percent of respondents think charter schools provide a good or excellent education and only 37 percent of respondents think public schools provide a good or excellent education.

  4. This data really doesn’t tell us anything from a statistics perspective. A 2% difference, but are there outliers? What did the distribution look like? What relevance to the 80% reduce/free population is there? Is the reduced/free average for charter schools 90% but public 80%, the data doesn’t say. What about confidence? It is quite a scientific leap of faith to draw any conclusions from it.

    • Steve, these are good points. We can calculate the free-reduced rates for both types of schools in this sample: They are 90 percent for charters, 87.4 percent for publics, not a huge difference. The low N for charter is reason for caution, I agree. There are outliers, but the results are pretty similar using medians as for means. For medians (by my calculation): charters are 64.6 percent pass for ELA, 63.2 percent for math; publics are 68.6 percent for ELA, 71 percent for math. As for drawing conclusions, you have a good point. But I think it’s fair to see these data don’t support the widespread belief that charter schools are better than district public schools.

  5. Steve – I appreciate your work and I know you understand the limitations of it, so I’m not trying to be difficult. But, let me see if I can get you to do a little more work! Free/Reduced lunch is an important measure. I wonder if location (especially rural vs urban) is also important? Charter schools tend to be clustered in urban areas, while high free/reduced lunch populations can extend into rural areas as well. Might you only include non-charter public schools in urban centers as your comparison group to charters? Do the numbers look the same there also? What if you only included non-charter public schools in urban areas that have charter schools (for example, Terre Haute might be considered urban, but they don’t have any charter schools). There might be some argument that charter schools only pop up where they’re needed, so comparison against only local schools seems more appropriate.

    • Great point, Caitlin. Here’s what I get with a quick comparison with high-poverty schools in IPS and also in Marion County (since Indy charters also draw from the township schools).

      E/LA Math
      Charters 62.3 62.5
      IPS 57.8 65.2
      Marion County 62.2 68.5

      So yes, charters do better than IPS and about the same as Marion County for E/LA; and worse than either in math. I wonder how their respective ELL percentages compare; but that’s a question for further research. Also, these are all high-poverty charters, not just Indy charters. Tell me which charters are Indy and I can run that too when I have time. It would also be interesting (assuming we’re putting stock in these test results) to isolate IPS and charter schools by neighborhood and compare them. Don’t think I have the capability to do that.

      • CREDO did the study in 2012 of Indiana charter students compared to their virtual counterpart (similar in all observables). They also isolated the Indianapolis charter schools vs. the state. Results found here. Summary of their conclusions on page 46:

        Basically, overall Indiana charters outperform local traditional schools. Especially for black students and students experiencing poverty. To comment on the ELL/SPED comments below mine – ELL and SPED students seem to perform about the same at charters and traditional schools. Of course, the numbers can’t show the “couseled out” issue (although I don’t think the second count date was instituted for charters alone – there were many issues with just one count date affecting both traditional public schools and charters). In the study, Indianapolis charter schools perform even better than the state overall. And – you’ll see that Ball State has gotten worse at authorizing high performing charter schools over the years.

        For a current list of Indianapolis charters – I’m pulling from the list found on the IDOE website:

        Here are the Indy charters with ISTEP scores. I did this quickly so no promises that it’s 100% correct. Should be fine, though.

        Irvington Community School
        Fall Creek Academy
        Christel House Academy South
        Flanner House Elementary School
        KIPP Indianapolis College Preparatory
        Andrew J Brown Academy
        Charles A Tindley Accelerated School
        Southeast Neighborhood School of Excellence (SENSE)
        Fountain Square Academy
        Indianapolis Lighthouse Charter School
        The Challenge Foundation Academy
        Herron High School
        Hope Academy
        Monument Lighthouse Charter School
        Indiana Math and Science Academy – Indianapolis
        Hoosier Academy – Indianapolis
        Imagine Life Sciences Academy – West
        Paramount School of Excellence
        Andrew Academy
        Padua Academy
        Indiana Math and Science Academy – North
        Hoosier Academy Virtual School
        Indiana Virtual School
        Indiana Connections Academy
        Damar Charter Academy
        Tindley Preparatory Academy
        Carpe Diem – Meridian Campus
        Enlace Academy
        Indiana Math and Science Academy – South
        George and Veronica Phalen Leadership Academy
        Tindley Collegiate Academy
        Tindley Renaissance Academy

      • Thanks! Why so few ELL students at so many charters, though? (I know there are many at CHA). I hear Macke Raymond, who does the CREDO studies, is doing a seminar this month at IU. Hopefully we’ll hear more about it when folks get back from AERA.

  6. When comparing test scores of public and charter schools, one must also compare their special ed. and English Language Learner (ELL) populations which have a huge impact on average scores.

    Of Indiana’s 77 charter schools state-wide (nearly all of which are in urban centers), 25 have ZERO ELL populations. Another 35 charters have less than a 5% ELL population. Traditional public schools have a state-wide average ELL population of 13.5% with higher percentages in major urban areas.

    It takes 5-7 years for a student to become fluent in academic English. Nevertheless, ELL students must take I-STEP in English after their first year here. If students can barely understand and/or write English, their test scores and those of their schools are skewed downward and dramatically – particularly in schools where nearly 1 in every 5 students is an ELL student
    and particularly in school districts dealing with 50 plus different languages.

    Similarly, the overwhelming majority of charter schools enroll a lower proportion of special education students. Like ELL scores, I-STEP scores of special ed. students are averaged into overall school scores and skew scores downward.

    Charters have increased their proportion of special ed. enrollments in the last few years, but it is possible to further ‘game’ the test score averages by the TYPE of special ed. students a charter decides to serve. A charter can achieve higher I-STEP scores by enrolling either a few severely disabled students who are exempt from I-STEP OR by enrolling only the highest performing special ed. students. Charters became so notorious for enrolling special ed. students (who generated more dollars per pupil) and then ‘counseling out’ these students AFTER the ‘pupil count date’ that the state legislature required a pupil count date twice a year to discourage the practice. However, it’s still possible to keep the special ed. student beyond the 2nd pupil count date and THEN counsel them out of the school BEFORE I-STEP tests are taken.

    Urban public schools have scores of parents of special ed. and ELL students who report that their child was ‘counseled out’ of a charter school with the explanation that the charter school didn’t have a program to serve their child’s needs. Few if any parents would keep their child in such a school. However, if charters are truly ‘public’, they shouldn’t have the option NOT to serve a child’s needs. Traditional public schools must find a way to serve such students regardless.

    Given these practices which lower the AVERAGE test scores of traditional public schools, public schools generally STILL score higher than charter schools. I can only conclude that public schools are doing one heckuva job with and for their students.

    • Thanks, Nancy. I wondered about ELL, given the mismatch between math and E/LA passing rates in high-poverty schools overall. I’ll take a look at numbers when I find time.

      • On average the demographics of charters vs traditional public schools doesn’t seem as skewed (based on the CREDO report found here Page 12). 7% ELL in traditional public schools that are feeder schools to charters vs. 3% in charters. For SPED it’s 15% in feeders to 11% in charters. You’re clearly right that charter schools have proportionately less of both of these populations, and the causes for that should be examined. But there could be other explanations than the ones you put forth. One hypothesis mentioned in the report is that parents of SPED students may believe that their children will be better served through traditional public schools. For ELL, I could imagine that many charter schools aren’t reaching out as well as they could to the parents of ELL students. I’m sure many charters could do more to create awareness among the ELL populations and help them complete the process to get their children enrolled in charter schools.

        Those are just hypotheses, but I think they merit consideration.

        Also of note in the demographics comparison in the report is that the percentage of students in poverty for feeder schools is 51% compared to 68% in charters. We can all agree that educating children experiencing poverty comes with its own challenges. According to these figures, charters are facing these challenges at greater rates than their traditional school counterparts.

      • The “feeder school” approach seemed squirrely to me, at least for comparing FRL percentages. CREDO counts any school that contributes at least one student to a charter as a feeder school; and all feeder schools count equally. In other words, if a charter school had one students from a low-poverty Carmel school and 99 from a high-poverty Washington Township school, the “feeder school” FRL rate would be the combination of those two schools — it’s not pro-rated.

      • You’re the best, Steve. Love that you tracked down Raymond to explain “Feeder” schools. On the surface I agree with you that a pro-rated approach seems like it would be more fair.

        I actually used the feeder school numbers because they did seem closer to an urban setting than the average traditional public school (TPS) in Indiana. For TPSs the ELL rate was only 5%, not 7% as it is in the feeder schools. I’m interested where Nancy got the 13.5% average statewide? The TPSs also had a 45% poverty rate whereas the feeders had 51%.

        For comparison, IPS has almost 85% of their students on free/reduced lunch. 18% SPED and 13% ELL. Not sure what the Indy charter school demographics look like. If only all this data were easy to sort out and we had the time to do so!

  7. Pingback: Indiana charter schools lag on serving ELL students | School Matters

  8. Pingback: On charter schools, the ‘story is in the variation’ | School Matters

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s