Fact-checking the checkers on miraculous CPS test score bump

Stanford researcher Sean Reardon conflated test scores with learning. 

I don't trust meteoric rises in student test scores or graduation rates. I didn't buy George Bush'"Texas Miracle" or Arne Duncan's "Chicago Miracle". I don't want to bust anyone's bubble, but there is no "magic sauce" leading to such miraculous gains. If there was, everyone would be eating it.

Neither do I accept the idea that student learning can be accurately or usefully measured by high-stakes standardized testing. More on that later.

Chicago's Better Government Association (BGA) just fact-checked the claim that “CPS students are learning and growing faster than 96% of students in the United States.” I'm glad somebody checked. But unfortunately the BGA got it only partly right and ended up joining CPS in conflating test score gains with learning.

Thank you,BGA, for pointing out that CPS leaders were cherry-picking the results of a recent Stanford study  reporting miraculous test score gains. Unfortunately, they confined their fact-checking to the study's own limited, narrow use of test data and therefore missed the forest (no Claypool pun intended) for the trees.

BGA fact checkers missed the forest for the trees.

The Stanford study shows CPS students making the fastest academic progress of the 100 largest school districts in the country. But even the researchers aren't quite sure how or why that happened or what to make of it all. For one thing, the gains are uneven across the grades. For another, they are percentage gains, and use a metric that can be interpreted in many ways.

Let me use a baseball analogy to explain. A batter strikes out his first 8 times at bat. Then he gets two hits, thereby raising his pitiful batting average from zero to .200, a 200-point increase in just one game. It's the fastest rise of any of his teammates. But at the end of the day, he's still a weak .200 hitter and will likely soon be sent down to the minors to work on his batting stroke.

In other words, rapid percentage increases often mean that the counting began and ended in a very low place. That's the most credible interpretation of the Stanford/CPS study.

According to the BGA:

CPS’ fast-paced gains were assessed in a report prepared recently by Sean Reardon, a professor of education inequality at Stanford’s Center for Education Policy Analysis.

By comparing Chicago Public Schools students’ scores on standardized tests to those of students nationally, Reardon found that the scores of CPS students in grades three through eight improved more from 2009-14 than did the average scores of all U.S. students during that time.

But, asks the BGA,

Improvement aside, how does CPS’ overall academic performance stack up against the rest of the country? Here, the picture was not as rosy. Third- through eighth-graders in the nation’s third-largest district still perform at roughly one half to one-and-a-half grade levels below the national average, which the report describes as a “significant concern.” 

In short, CPS test scores started low, may have improved rapidly, but remain subpar. District leaders, however, were jubilant about the report, even though earlier in the week, the state released scores from the PARCC test it has administered for the past few years, showing that barely more than one in four CPS elementary students can read, write and do math at grade level. CPS officials have refused many requests to discuss those scores.

But I will.

By looking only at standardized test scores, Reardon's study is limited in the insights it can offer as to  whether or not real progress is taking place at CPS. Dramatic increases or sudden drops in test scores could be the result of many things, completely unrelated to any change in district policies or anything new going on in the classroom. For example, they could be driven by a dangerous overemphasis on test prep or a dramatic loss of student population.

Reardon's team never set foot in a CPS classroom.

CPS' student population has decreased by nearly a hundred thousand as more than a quarter-million mostly-black Chicagoans left the city over the past few decades. Many of these children in the out-migration were likely among CPS' poorest and most academically challenged students. This alone could account for the increase in scores, since standardized test scores have been shown to more closely align with parent incomes that with any district policies or professional development strategies.

Stanford researcher Reardon doesn't believe this is the case since

"...the consistency across race as well as similar growth on a nationally administered no-stakes NAEP test convinced him that CPS’ growth was real and not from a demographic shift in students or from holding lots of kids back a grade."

It may have convinced him, but not me. Test score growth may be real. But what does it really show?

As for consistency across race, CPS remains 93% students of color going to mostly segregated schools. Even if test scores increase across race, the gap across race and class remains intact. Inequality is merely reproduced. Not exactly a recipe for increased learning or for educational equity. As for the similar gains in the more highly-regarded NAEP exam, they could also be connected to changes in demographics.

CPS Chief Education Officer Janice Jackson attributes the gains to staffers now "using data to guide instruction", and principals being "empowered to lead schools." You'd think she would have at least tossed a bone to the district's classroom teachers, especially those in special ed, who have been working overtime, with reduced staff and severe district program cuts.

Stovall & Radinsky

Even if the test-score gains are real, this doesn't mean that authentic student learning has improved. As UIC prof and CReATE researcher David Stovall put in on our Hitting Left show Friday,

"We're conflating test taking with learning and if you remove thousands of the poorest students who are are struggling, why wouldn't you have an increase (in CPS test scores)?

 HL guest Josh Radinsky, another CReATE researcher, called all the focus on test scores a "dangerous discourse." The danger being how you generate higher test scores in Chicago Public Schools by minimizing subject areas that aren't tested. There's also danger in the way these reported test score increases are used to justify bad school reform policy.

Says Radinsky:

We have been trying to get reading and math scores pumped up by artificially stimulating student performance on these bubble tests...Walk into any school in Chicago and ask, what are you doing in social studies right now. Social studies has been eviscerated by the focus on test scores. This is one example among many and we can talk about music and art. Teachers who love their kids and teach their hearts out every day are put into this straitjacket of test prep. 

I'm going to save my last point for a future post. But here it is in short. If in fact, dramatically rising test scores show that Rahm/Claypool/Jackson reform policies are working and that increased student learning has brought CPS to number-one in the race to the top, then why did the mayor support the recently-passed voucher bill to grease the exit of students from public to private schools?


Meanwhile in the burbs...While parents and students in mainly white, wealthy, high-scoring suburban districts decry the debilitating pressure resulting from high-stakes testing, CPS continues to mandate more and more testing along with more time spent on test prep.

The Tribune reports:

Parents are sending their kids to therapeutic day schools at hospitals that treat adolescent mental health issues. Teachers are changing their curriculum to factor in students' anxiety and stress. And kids are facing what they say is a constant, grinding strain throughout their academic careers. 

"There is a double-edged sword. We want kids to challenge themselves, but not at the expense of their mental well-being," said Emily Polacek, a social studies teacher at Hinsdale South High School in west suburban Darien.