Lies, Damned Lies, and Statistics

Several years ago, at the funeral for University of Virginia Dean of Admissions Jack Blackburn, I ran into Bill Hartog, the long-time Dean of Admissions at Washington and Lee University.  We shared our fond memories of Jack and his legacy for our profession, and commiserated about becoming elder statesmen, or just older.

I shared my experience of attending a meeting of independent school college counselors, including several who had just moved across the desk from the college side.  At the end of the meeting, one of them looked at me and stated how inspiring it was to see the old-timers present.  Thinking he wanted affirmation, I started to reply, “It sure is,” when it hit me that the old-timer he was referring to was me.

Bill observed that he was now 25 years older than the next oldest member of his staff.  He sensed that people in the office were asking, “Who is that old guy, what does he do here, and why is he always so grumpy?”

No one would begrudge Bill for being grumpy this week.  I returned from Toronto on Sunday night to learn that the Washington Post had published an article about colleges that count incomplete applications when computing application totals and admit rate.  The focal point in the article was W&L.  A day later Washington and Lee President Kenneth Ruscio called for an internal review of the school’s procedures for reporting admissions data.   

The original article correctly points out that W&L’s practice is not out-of-line with the accepted guidelines for reporting data to the federal government or as established by the Common Data Set, but there is also an implication that what W&L is doing is comparable to the misrepresentation of admissions stats that occurred at places like Emory and Claremont McKenna and Bucknell.  Based on what I have read and what I know, that is not the case.

I read the article not as an indictment of Washington and Lee but as a criticism of the current state of college admission.  There is a disconnect between the accepted practice within our profession and public understanding of how the admissions process works.  The implicit criticism of W&L was due to counting as completed applications more than 1100 applications that were incomplete, an inclusion that lowered the acceptance rate from 24% to 19%.  That’s common practice, and I don’t consider it unethical, but many of us are guilty of massaging statistics to make us look better.

I also think we have deliberately kept a veil of mystery over how admission works, at least partly because we don’t want the public to know how subjective and imprecise holistic admission can be.  That veil of mystery is a double-edged sword.  It increases public fascination with college admission, but it also leads to skepticism and distrust about our practices and our motives.  As colleges move to a culture driven by marketing, we also send a message that we are motivated by self-interest rather than the public interest.  It is also the case that every instance of misrepresentation at one institution harms all of us.

The larger issue is that we have allowed admissions metrics to become a proxy for institutional quality.  Because educational outcomes are so difficult to measure, we have turned to things that are easy to measure (and easy to manipulate) and assigned them value that is in no way justified.  The unchallenged assumption is that the more popular the institution—application numbers, admit rate, yield—the better it must be.  There are plenty of culprits for that way of thinking—Presidents, Provosts, and Boards; Bond-rating agencies; and of course, U.S. News and World Report, the poster child for measuring educational quality without considering the educational experience.  The logical conclusion of the “Popularity=Quality” mindset is that the ultimate gourmand experience will be found at McDonald’s.

Either Mark Twain or Benjamin Disraeli said that there are three kinds of lies—lies, damned lies, and statistics.  Colleges and universities know better than to spread lies or damned lies, but I think we can expect more scrutiny over our use of statistics. I suspect we will be challenged both by Gen X parents and by the federal government to find new ways to show the value of a college education and the value added by particular institutions.  The Obama administration’s proposal to rank colleges and allocate federal aid based on access, affordability, and outcomes may be the first salvo.  We need to be proactive and not reactive.