CCSA Responds to Ohio Value-Added Issue

November 24, 2015

  • Print
In response to the article authored by Dr. Vladimir Kogan of Ohio State University, "Value Added vs. Similar Students' Measures: Comparing Two Methods for Holding Ohio Schools Accountable," the California Charter Schools Association (CCSA) has identified significant misrepresentations and analysis errors by Dr. Kogan. CCSA has reached out to Dr. Kogan, and although he indicated that his methods are different than those created by CCSA, he continues to erroneously associate them with CCSA.

Dr. Kogan has presented a deeply flawed analysis of CCSA's SSM based on a number of inaccuracies and technical mistakes accompanied with sweeping leaps in logic. In fact, his analysis is not comparing the Ohio results to CCSA's SSM at all, but in fact, he is comparing Ohio results to a poor facsimile of CCSA's SSM. We understand that a researcher may choose to debate the particular statistical methods that are in an academic critique. And in this case, Dr. Kogan may have an opinion on which statistical methods he would choose to use as part of the SSM model, but it is patently false to hold his results as anything that closely resembles an implementation of the actual CCSA SSM.

CCSA's Similar Students Measure (SSM) is a useful lens through which to assess student performance given the unique demographic makeup of each school. In our view, the SSM is just one piece of the accountability puzzle - we combine it with achievement indicators, school growth indicators as well as post-secondary readiness indicators to identify those schools that appear to be underperforming according to publicly available data. We then engage with each school that is showing low performance on all of these indicators in a multiple measures review that examines dozens of state and locally collected data to ensure that any decision we make on closure advocacy is based on a deep and careful assessment of school performance across all grade levels and a wide range of student outcome indicators.

The CCSA SSM is a tool that has been tested and honed over the last five years in consultation with educational researchers and policymakers to help identify the lowest performing charter schools in the state of California. CCSA worked extensively with statisticians and educational researchers when initially developing the SSM and also took the extra step of analyzing the data in real world situations. By conducting an intensive qualitative study of schools that were identified as low performing on the SSM, CCSA conducted site visits of the lowest performing schools to verify the conclusions of the model. This work confirmed that the SSM was accurately classifying low performing schools. Read the study: Assessing the Utility of State Academic Indicators. Not only have these classifications of school performance been verified by researchers, but the data also has been relied upon by authorizers and policy makers throughout the state when charter authorizers are making a decision on whether or not to renew a charter school.

Changes to any underlying methodological assumptions should be made carefully. Based on our conversations and evaluation of Dr. Kogan's work, this was not the case. In his paper, Dr. Kogan purports to have tested the CCSA SSM model. Unfortunately, Dr. Kogan introduced his own set of assumptions into the model thus creating an entirely different model that only vaguely resembles CCSA's SSM and therefore creates extremely different results. These differences have been pointed out to Dr. Kogan, yet he refuses to accurately calculate the CCSA model. It appears that Dr. Kogan has introduced his own bias and opinion into his analysis which is unfortunate and if followed could lead to flawed policy outcomes for the state of Ohio.

For further reading: