U.S. News and World Report has decided to reclassify GW as “unranked” in light of the University’s disclosure of an error in the way one statistic, percentage of incoming freshmen who graduated in the top 10% of their high school class, had been reported. This unfortunate move on the part of USNWR is explained by its director of data research here.
I’d like to offer a few thoughts about this story to our students and their families, and to our faculty, staff, national council members, current and future employers of our graduates, alumni, and other friends in the community.
First, we can all be proud of the way President Knapp, Provost Lerman, and Vice Provost Maltzman handled the discovery that data had been reported incorrectly: they came right out and said so, voluntarily, quickly, and with complete transparency. We should applaud this decision, which included a full independent audit and restructuring of internal oversight procedures, because it makes clear that for GW ethics precedes expediency. As President Knapp notes in his public statement to the community, “we [disclosed the mistake] without regard to any possible action that U.S. News might take as a result…”
In fact, it is worth noting that GW informed USNWR of the error first, to give the editors time to analyze the data and prepare their response. Sadly, though, when USNWR decided to “unrank” us, they didn’t give us a heads-up before going public. Many observers were puzzled by this apparent belligerence on USNWR’s part. Leaving aside the unfortunate timing of the announcement, though, it is the decision itself that raises new questions.
For example, will other universities now be more or less willing to conduct the kind of internal audit that led GW to its embarrassing findings? How will they behave if they discover anomalies or errors in their data? Imagine for a moment that USNWR had not decided to unrank us. That would have signaled that honesty is indeed the best policy; instead, one can’t help but fear that this turnaround creates incentives for less rather than more transparency. USNWR thrives on accurate reporting by its participating schools, and should therefore reinforce institutional self-disclosure of errors in measurement. Ironically, by so quickly pulling us out of the ranks USNWR may have undercut its main raison d’etre, namely to provide relevant information to “families concerned with finding the best academic value for their money” (see this explanation on how USNWR calculates best college rankings). Whether this goal and the general public good are advanced in this increasingly hostile environment is questionable, to say the least.
Inevitably the GW data mishap also raises methodological questions. Many educators and education researchers who care about data are troubled by the disproportionate public attention paid to a ranking system that may be fundamentally flawed. At the risk of sounding like I’m complaining about sour grapes, I do think it pays to look “under the hood” and check on the validity of the rankings even when all the included numbers are assumed to be accurate. What exactly is the meaning of a ranking if previously included institutions are suddenly extracted from the list? Does the remaining ranking have empirical validity? Other questions abound (see, for example, Washington Post reporter Valerie Strauss’s recent blog; Nick Anderson’s commentary here; and the earlier New York Times op-ed that appeared before the GW snafu came to light).
Without in any way trying to excuse GW’s mistakes, we do believe that basic methodological issues and underlying behavioral assumptions warrant a broader discussion. Perhaps the best outcome of this unfortunate episode will be a commitment on the part of the entire higher education community to invent a more stable, valid, and reliable way to provide useful information to current and future constituents.
A final thought: The Graduate School of Education and Human Development doesn’t have its own dog in this fight, so to speak, because our school ranking isn’t affected by data on undergraduate admissions. But we are very much part of GW, and though we were saddened by the news of errors in reporting, we are immensely proud that our university is making efforts to improve internal operations and striving to maintain the highest standards of integrity in all we do.
November 19, 2012