CCSA Statement on UC Berkeley Report "Differing Effects From Diverse Charter Schools"

December 22, 2015

  • Print
Researchers at the University of California, Berkeley, led by Dr. Bruce Fuller, released a report examining student demographics and achievement at Los Angeles charter schools. While the report confirms the essential findings that have been repeatedly found by other researchers and CCSA - that students attending LA charter schools are learning more than are their counterparts in district-managed schools - the California Charter Schools Association believes the report's methodologies are deeply flawed. As a result, the report creates a portrait of LA charter schools that is simply inaccurate and, unfortunately, very counter-productive.

First and foremost, the report fails to differentiate between district-dependent charter schools and fully independent "autonomous" charter schools. For those who know LA charter schools well, the fundamental difference between dependent and fully autonomous charter schools is well understood. Indeed the Los Angeles Times, when reporting on charter school matters, does not even include enrollment in district-dependent charter schools when it reports total enrollment in LA charter schools. This is because district-dependent charter schools are broadly recognized to be charter schools in name only. They have little, if any, independent control over their governance, finances, staffing or operations. They have no separate governing boards but are governed, rather, by the Los Angeles Unified Board of Education. For most intents and purposes, they are little different than district-managed schools.

Unfortunately, while most close to the Los Angeles charter school movement understand this difference, Dr. Fuller does not. Indeed, perhaps the biggest focus of this paper which purports to be about charter schools is in fact about a segment of schools that most would say are not even charters. As such, the conclusions that he draws about these schools we believe should not even be attributed to charter schools at all. This is particularly worrying because many conclusions he draws are diametrically opposite to what we know to be happening within fully autonomous startup and conversion charter schools.

For example, district-dependent charters serve a fundamentally different student population than fully autonomous charter schools, including fully autonomous conversion schools. Unfortunately, Dr. Fuller's analysis lumps together district-dependent charter conversions (located primarily in wealthy Westside communities) and fully autonomous charter conversions (located primarily in Pacoima) and declares that on average they have higher income and higher test scores. This is a gross oversimplification that does a disservice to LA's many fully autonomous conversion schools that have fundamentally different demographics from dependent schools and are generating far better student outcomes than district-managed schools serving similar student populations. Below are the conversion charters in Los Angeles as of 2007-08 (baseline year in Fuller's analysis).

CCSA issued a recent report that highlights the performance of autonomous charter schools in Pacoima, many of which are conversions. CCSA CEO Jed Wallace notes that their outcomes are "every bit as impressive as the nationally recognized Harlem Children's Zone, only in our backyard and just not fully understood or appreciated." CCSA's Pacoima report describes the transformation in a low-income immigrant community where the autonomous conversion charters have played a pivotal role in generating fundamentally different outcomes on behalf of Los Angeles public school students.

What makes Dr. Fuller's errors regarding district-dependent schools all the more troubling is that he chose not to address them even after CCSA brought them to his attention. For a researcher as respected as Dr. Fuller to fail to understand the fundamental difference between schools that are managed by the school district and schools that are fully autonomous speaks to the need to continue educating researchers about the fundamental characteristics of the charter school movement.

Another major flaw in the report is that it mostly ignores student achievement that occurs in the early years of education. The report begins with students who have completed 2nd grade, failing to take into account the education that students receive in kindergarten, first grade and second grade. Those three years are critically important in a child's learning. There is no control to determine the impact of the three years that a student spent in a charter school on their learning.

Fuller theorizes that because he observes higher baseline test scores in charters at the end of grade 2, it must mean that the students entered the elementary school higher achieving (with more motivated parents). It does not at all consider the possibility that students started with similar levels of achievement but then made substantial gains in their first 3 years of elementary school (K, 1, and 2), such that by the end of 2nd grade they were outperforming. Fuller certainly finds evidence of accelerating gains in grades 3-5. Why wouldn't this same phenomenon be true in the early grades? Certainly Fuller has documented in his other published research the importance of early education experiences.

Given the report's finding that charter students have 6 percentage point gains in elementary ELA and 3 points in Math, it is logical to assume that those same gains could take place between kindergarten and the end of 2nd grade. In that case, there would be a very little or no evidence to suggest that there is a differential in starting gains. In other words, the report ignores the possibility that charter students are ahead of their peers at the end of 2nd grade precisely because these students were enrolled in charter schools and had already made academic strides beyond their peers.

This same flaw applies in the report's examination of high school students. It ignores the possibility that the stronger high school high starting scores for startup charter schools could be due to the fact that many of these charter high school students attended a charter middle school and had made prior academic gains. Again, the report does not control for the prior effects of charter schools on student growth.

CCSA conducted its own analysis to test these conclusions. Thanks to a statewide partnership with many of our member schools, CCSA has access to a sizable dataset of Northwest Evaluation Association (NWEA) MAP Assessment data, which includes charter schools testing students in grades K-2. While we don't have comparable data for LAUSD traditional school, we can compare charter students' baseline test scores (in fall of kindergarten, first grade, and second grade) to national NWEA norms. This represents an important window into a view of early student performance few researchers have in California. What we see from our analysis is that charter students enter with similar baseline scores in fall of grades K, 1, and 2 (compared to the national norm), but then grow at a slightly faster pace than their similar situated peers nationwide. This may contribute to the higher scores Dr. Fuller sees at the end of second grade.

The small sample size of students used in the report is also worrying. The analysis is based on anywhere from 36 to 2,776 traditional students and anywhere from 174 to 2,773 charter students. The report argues that these results are generalizable to the over 100,000 students (150,000 if one includes dependent charter schools) currently in LAUSD charter schools, which is highly questionable. Also the data is quite old (beginning in 2007-08 and ending in 2010-11). Charters have continued to accelerate their achievement since the time of the data Fuller studied.

Fuller notes he detected no achievement differences between pupils attending charter and traditional high schools. While he may not have detected any in his small sample, other indicators clearly show that there are substantial achievement differences. CREDO's 2014 report on Los Angeles documented that charter high school students were experiencing an additional 2.5 months of learning in reading and math. In a 2014 Los Angeles report, CCSA documented that four times as many charter students as district students graduate high school prepared for college. Given Fuller's dramatically different findings, we question the representativeness of his data set and find it to be outdated.

These are some, but not all, of the methodological mistakes that significantly detract from the report's usefulness and credibility.

Dr. Fuller's research in the past has been far more nuanced and rigorous, so we find it surprising that his current effort would be so misguided. Rather than offering concrete evidence to substantiate its claims, the report seems intent on assuming that charter schools benefit from unseen advantages. It is unfortunate that Dr. Fuller and his team seem unwilling to acknowledge that the credit for charter schools' superior performance goes to the highly talented and driven educators who, with a healthy balance of autonomy and oversight, have innovated and adapted to their unique communities' and students' needs, yielding student gains that should be celebrated, not snubbed.

Note: The chart has been updated slightly from a previously issued version of our statement, due to revision of one school's autonomy classification. This revision does not change the analysis.