Claremont McKenna’s recent SAT scandal sparked much interest, on campus and in national media. The CMC community learned the basic facts from President Gann’s email on Monday, January 30th, but questions still remain about why and how former Dean of Admissions Richard Vos manipulated the reported SAT scores.
Unfortunately, the Port Side cannot take you inside Vos’s brain to find out why he altered our SAT scores. But examining the correct and falsified data reveals patterns and tells us some interesting stories.
Gann’s email focused on manipulations to median SAT Math and Reading scores, which are shown in the graphs below, for classes matriculating in 2004 through 2011. So, the rightmost data point, for 2011′s matriculating class, represents current first-year students. The darker of the two lines is CMC’s actual scores; the lighter of the two are those that were falsely reported.
Vos’s manipulations of data hid periodic ten to twenty point drops in the school’s scores, but the reported numbers were relatively flat.
This was one of two reasons cited by Gann for why the manipulation went undetected for so long. “If you look at how flat the reported data were, it did not raise suspicions,” Gann said. That “a sole person had too much authority over reporting” was Gann’s second reason.
Math scores were modified much less than Critical Reading scores. Reported Critical Reading medians were, on average, 15 points higher than reality, but Math medians were, on average, only ten points higher. The only year that median Critical Reading scores were not inflated was 2010.
Score manipulations were relatively small in 2004, the first year that figures were manipulated. However, the actual median score for the 2005 matriculating class in Critical Reading fell by ten points relative to the 2004 class; the 2006 median Math score fell by 20 points relative to 2005 and 2004.
Gann’s email stated that CMC had been “falsely reporting SAT statistics since 2005″ — this is technically correct. The scores for the 2004 matriculating class were reported in 2005; for the same reason, at the time that the scandal broke, CMC had not yet publicly reported the manipulated scores pertaining to the 2011 matriculating class.
Mean SAT Scores
These next graphs show CMC’s mean SAT Math and Critical reading scores.
Because median SAT scores are necessarily a multiple of ten, much more variation is visible in the mean SAT scores. One silver lining visible here is that CMC’s actual mean SAT scores have been increasing year-over-year for both sections. Mean Math scores have increased by 3.57 points per year (found via linear regression); Critical Reading scores by 1.58 points.
The differences between the falsified and actual mean scores show wide gaps in some years. Mean Math scores for the class matriculating in 2007 — who graduated in 2011 — were falsely increased by 28 points. Mean Critical Reading scores were increased by 22 points for that same 2007 matriculating class and by 23 points for the 2011 matriculating class — the current freshmen.
These large differences were not revealed in Gann’s initial email that broke the news of the scandal. “Although the degree of inaccuracies varied over time, we understand that the reported critical reading and/or math SAT scores were generally inflated by an average of 10-20 points each,” the email said. The email did not specify whether the 10-20 point falsifications were medians or means; the claim is true if applied only to medians.
Mean Math scores were inflated by an average of 10.5 points; mean Critical Reading scores were inflated by an average of 17.375 points.
Combined SAT Scores
The graphs below show CMC’s combined SAT mean and median scores, that is, Critical Reading plus Math scores. Writing scores are not submitted, so the total is out of 1600.
The next graph shows the total number of individual scores that were misreported each year.
Due to an issue with the data as it was reported — that the falsified numbers, as reported, literally do not add up to their stated total — this metric and the one in the next graph are not computable for 2004.
Modifications were lower in recent years than in the past, as this graph shows. In fact, there were only four scores in the data CMC reported in 2010. But it also shows that the scale of the data modification was fairly small in its beginning years and spiked in 2007 and 2009.
CMC reports SAT data by reporting the number of matriculating students with a score in a given fifty point range. So, for 2011, CMC originally reported that 44 students had a Critical Reading score between 750 and 800. The data was modified by moving students from a lower range to a higher range, for instance, by reporting that forty-four students received a Critical Reading score between 750 and 800 when only 33 actually had. So, for that range (and the range those scores were moved from — CMC reported that 11 fewer students had Critical Reading scores of 600-640 than actually did) the graph shows 11 scores were modified.
Manipulated Scores and Class Size
The final graph in this analysis shows the size of each matriculating class and the number of scores that were modified that year.
In some years, the scale of the modifications is large. In 2007, 72 scores were modified; since each student submits two scores, the scores belonging to at least 36, and possibly more, students were changed. For the 2009 matriculating class, 80 total scores were altered, so scores belonging to at least 40 students were modified. In other words, the data is not consistent with a theory explaining that the modification was used to hide a small group of students’ scores.
The data used for this analysis is embedded in a Google Spreadsheet below. If you’d like the data in another format, drop me a note.
Critical Reading, 2004-2011
Tables split into Math/Critical Reading, updated 3:30pm 1/31/12. Google Docs by Jonathan O. Hirsch.