Unranked

Last week U.S. News and World Report moved George Washington University to the Unranked category in the 2013 “America’s Best Colleges” rankings.  The move came in response in GW’s admission that it had misreported data regarding class rank for entering students, both on its website and to U.S. News (see previous post).

I must admit that my first response to the news was simple and without nuance. Does anyone care?  Should anyone care?  Does anyone think less of GW because it lost its U.S. News ranking (which is not the same thing as asking if anyone thinks less of GW because it misreported information)?

“Upon further review” (to borrow National Football League replay language), I realized that:

1)      “Simple and without nuance” makes for short blog posts;

2)      This incident is an opportunity for introspection not only for George Washington University but also for U.S. News and World Report.

I am hoping that the introspection is taking place at GW and guessing that it’s not at U.S. News.  I would be more than happy to have U.S. News and World Report hire me as a consultant to evaluate the methodology and assumptions underlying the rankings, but as a public service here are some questions and recommendations for consideration and introspection.

 

Question:  Should U.S. News rank colleges utilizing information that is unverified?

            U.S. News relies on information self-reported by colleges in compiling rankings.  The fact that there have been three incidents in 2012 alone involving reputable institutions misreporting data would suggest that the Honor System is not working.  One of the foundations of reputable journalism is fact-checking.

Recommendation:  Spend some of the considerable profit U.S. News makes from the rankings and hire an auditor to verify data.

 

Question:  Is it time to get rid of the peer assessment reputation survey?

            The U.S. News rankings began in 1983 as a magazine article (the rankings have outlived the magazine), and were based exclusively on a survey of college presidents.  I was a college faculty member at the time, and the joke on our campus was that no one was sure the President knew much about our campus, much less any others.  Through the years U.S. News has incorporated other data into the rankings, but the reputational survey remains the biggest component, counting 22.5%.  Provosts, admissions deans, and high school counselors (I choose not to participate) are now surveyed in addition to Presidents.  How reliable is the peer assessment?  The percentage of respondents is relatively low and has been declining, reputations may lag behind realities, and Presidents and other officials receive incentives for improving an institution’s ranking, leading to revelations of several Presidents ranking their own institution higher than Harvard, Yale, and Princeton.

Recommendation:  Get rid of the peer assessment altogether or publish it as a separate ranking, making it clear that it reflects opinion rather than fact.

 

Question:  Do the input measures used by U.S. News tell anything about output, a college’s success in educating students?

            U.S. News doesn’t pretend to measure educational quality, although that fact is hidden in the fine print if mentioned at all.  Output is too hard to measure, and colleges are hesitant to share publicly their results on measures such as the Collegiate Learning Assessment and the National Survey of Student Engagement. Is the assumption that selectivity=quality valid? Focusing on admission stats such as selectivity and SAT scores and other stats like Alumni Giving, all of which can be manipulated, is like ranking “America’s Best Churches” without regard for spiritual growth.

Recommendation:  Start a conversation with college and other educational leaders about metrics that might measure how much education is taking place on campus.

 

Question:  Do the year-to-year changes in rankings reflect actual changes in institutional quality or tweaks to the methodology to produce different rankings?

            I don’t have an answer to that question, just a suspicion.

 

Question:  Does U.S. News want to be in the news business or the entertainment business?

            Years ago I attended a NACAC conference session where Bob Morse, the man behind the numbers for the U.S. News rankings and author of the “Morse Code” blog, described the rankings as a “good product.”  He took umbrage when I asked him if it was good journalism.  That question is just as relevant today.  Is U.S. News reporting the news or making news?

            The plethora of stories each fall about the new rankings would suggest that U.S. News has become a newsmaker, perhaps even a trendsetter, rather than a news outlet.  In fairness to U.S. News, though, that is consistent with the direction that journalism, and particularly television journalism, has taken.  Today journalists are celebrities who socialize with those they are supposed to be covering, and career advancement is more tied to Q rating or ability as an entertainer rather than ability to sniff out news.

            I would argue that U.S. News chose entertainment over news as early as 1983, long before it became clear how closely the U.S. News brand would become tied to college rankings. The original rankings article listed only top ten lists in the National Universities and National Liberal Arts Colleges categories and ignored the real news story.  The #10 school in the National Universities category was Brown.  The fine print showed that Brown was considered one of the top ten schools by only 25% of those responding, meaning that 75% didn’t think Brown belonged in the top ten.  The real news from the survey was the diversity of quality schools in American higher education and how little agreement there is about top schools beyond the first three or four.

Recommendation:  Add a disclaimer to the rankings, either “For Entertainment Purposes Only” or “Your Results May Vary.”

 

Question:  Do the rankings help students and parents make more thoughtful college decisions?

            U.S. News states that “The intangibles that make up the college experience can’t be measured by a series of data points,” then proceeds to rank America’s “best” colleges based on a series of data points.  The U.S. News rankings are part of a balanced college search the way Sugar Smacks or Count Chocula are part of a balanced breakfast.  The balance comes from everything other than the product.

            There is a lot of helpful information in “America’s Best Colleges,” ranging from the topical articles to the use of Carnegie categories to divide schools.  The attempt to rank college negates most of those benefits.  College rankings provide a precision (We’re #6!) that leads students and parents away from thinking about the quality of the college experience.  They also simplify a process that should be both complex and personal.

Recommendation:  Expand the “Unranked” category to include not only George Washington University but all other colleges and universities as well.