Lies, Damned Lies, and Statistics

Several years ago, at the funeral for University of Virginia Dean of Admissions Jack Blackburn, I ran into Bill Hartog, the long-time Dean of Admissions at Washington and Lee University.  We shared our fond memories of Jack and his legacy for our profession, and commiserated about becoming elder statesmen, or just older.

I shared my experience of attending a meeting of independent school college counselors, including several who had just moved across the desk from the college side.  At the end of the meeting, one of them looked at me and stated how inspiring it was to see the old-timers present.  Thinking he wanted affirmation, I started to reply, “It sure is,” when it hit me that the old-timer he was referring to was me.

Bill observed that he was now 25 years older than the next oldest member of his staff.  He sensed that people in the office were asking, “Who is that old guy, what does he do here, and why is he always so grumpy?”

No one would begrudge Bill for being grumpy this week.  I returned from Toronto on Sunday night to learn that the Washington Post had published an article about colleges that count incomplete applications when computing application totals and admit rate.  The focal point in the article was W&L.  A day later Washington and Lee President Kenneth Ruscio called for an internal review of the school’s procedures for reporting admissions data.   

The original article correctly points out that W&L’s practice is not out-of-line with the accepted guidelines for reporting data to the federal government or as established by the Common Data Set, but there is also an implication that what W&L is doing is comparable to the misrepresentation of admissions stats that occurred at places like Emory and Claremont McKenna and Bucknell.  Based on what I have read and what I know, that is not the case.

I read the article not as an indictment of Washington and Lee but as a criticism of the current state of college admission.  There is a disconnect between the accepted practice within our profession and public understanding of how the admissions process works.  The implicit criticism of W&L was due to counting as completed applications more than 1100 applications that were incomplete, an inclusion that lowered the acceptance rate from 24% to 19%.  That’s common practice, and I don’t consider it unethical, but many of us are guilty of massaging statistics to make us look better.

I also think we have deliberately kept a veil of mystery over how admission works, at least partly because we don’t want the public to know how subjective and imprecise holistic admission can be.  That veil of mystery is a double-edged sword.  It increases public fascination with college admission, but it also leads to skepticism and distrust about our practices and our motives.  As colleges move to a culture driven by marketing, we also send a message that we are motivated by self-interest rather than the public interest.  It is also the case that every instance of misrepresentation at one institution harms all of us.

The larger issue is that we have allowed admissions metrics to become a proxy for institutional quality.  Because educational outcomes are so difficult to measure, we have turned to things that are easy to measure (and easy to manipulate) and assigned them value that is in no way justified.  The unchallenged assumption is that the more popular the institution—application numbers, admit rate, yield—the better it must be.  There are plenty of culprits for that way of thinking—Presidents, Provosts, and Boards; Bond-rating agencies; and of course, U.S. News and World Report, the poster child for measuring educational quality without considering the educational experience.  The logical conclusion of the “Popularity=Quality” mindset is that the ultimate gourmand experience will be found at McDonald’s.

Either Mark Twain or Benjamin Disraeli said that there are three kinds of lies—lies, damned lies, and statistics.  Colleges and universities know better than to spread lies or damned lies, but I think we can expect more scrutiny over our use of statistics. I suspect we will be challenged both by Gen X parents and by the federal government to find new ways to show the value of a college education and the value added by particular institutions.  The Obama administration’s proposal to rank colleges and allocate federal aid based on access, affordability, and outcomes may be the first salvo.  We need to be proactive and not reactive.

Postcard from Toronto

Today happens to be the first anniversary of the blog.  It’s been a good year, and as I have told several people at NACAC in Toronto, one of the most fulfilling things I’ve ever done.  I am particularly grateful to all of you who read and comment both privately and publicly.

I celebrated by doing a session this morning with Lee Coffin from Tufts and Chuck Lovelace from the Morehead-Cain Foundation at UNC-Chapel Hill.  Our topic was one that readers of the blog will find familiar.  Are we measuring the right things in the college admissions process?  We addressed (not the same as answering) that question in three different areas: non-cognitive predictors of success; 21^st century skills; and measures of institutional quality.  It’s a discussion worth having, and Lee and Chuck have both done first-rate thinking in how to merge theory with practice.

Last week Eric Hoover at the Chronicle of Higher Education asked me to write a guest post for the Chronicle’s Head Count blog, and it appeared yesterday.  Here’s a link.

This was the first time in five years that I didn’t have official duties.  Several people asked me if I missed it, and I responded by asking if I looked like I had a sense of loss.  No one answered yes.  I enjoyed the opportunity to serve NACAC, and the experience has certainly benefitted me both personally and professionally, but I finished my term with a sense of satisfaction that I had done my best and that it was time to move on to other things.  Writing about the ethics of college admissions has helped make that transition easy.

This morning the NACAC Assembly amended the Statement of Principles of Good Practice to address the use of international agents.  The Assembly adopted amended language to the motion put forth by the Board of Directors and Admission Practices Committee based on the report of the Commission on International Student Recruitment.  The amendment to the SPGP’s Mandatory Practices allows member institutions to use incentive-based agents when working with international students but requires that the institution ensure “accountability, transparency and integrity.”

I think it was important for the Assembly to validate the thoughtful deliberative process employed by the Commission, but "accountability," "transparency," and "integrity" all leave plenty of room for definition and further discussion. I think no one believes this will resolve the agent issue once and for all.