US: Alert to policy-makers over university rankings

A warning by the influential Washington think-tank, the Institute for Higher Education Policy, that university rankings should be viewed with caution by decision-makers received little media attention in the US.

Perhaps because the IHEP Issue Brief The Role and Relevance of Rankings in Higher Education Policymaking appears to state the obvious - policy-makers are not likely to be so rash as to act solely on rankings whose limitations are frequently acknowledged even by the people who compile them - only the online InsideHigherEd carried the briefest of stories.

Or perhaps it was because even this 'look both ways before stepping off the kerb' warning obscured the report's more nuanced finding, that "while many educators may look down on rankings, they have the potential to have significant impact on public policy".

The report provides an overview of national post-secondary assessment efforts and notes the similarities and differences these approaches have taken in comparison with college rankings. Its findings are drawn from a review of the literature as well as interviews with individuals from federal and state government agencies, trade associations and other groups.

Michelle Asha Cooper, the institute's president, commented: "Discussions of college rankings often fail to see the influence these popular schemes may hold in the policy-making process."

Ellen Hazelkorn, director of Research and Enterprise at the Dublin Institute of Technology, told last year's International Association of Universities conference in Utrecht that 63% of higher education leaders surveyed in her research had taken strategic organisational, managerial or academic actions in response to rankings. Only 8% indicated that they had taken no action.

The IHEP believes that if decision-makers focus too heavily on rankings they run the risk of missing opportunities to enhance institutional success and improve equity.

The report sets out recommendations to ensure policy-makers are aware of these pitfalls while increasing the policy relevance of rankings. These include ensuring that rankings are used only as part of overall system assessment efforts and not as a stand-alone evaluation, backing initiatives to collect data to craft more policy-relevant college rankings, and guiding public attention to college rankings as a means to shape general notions of quality and advance equity goals.

"Thus, when rankings enter government policy deliberations, attention must be paid to ensure definitions of educational quality represented by rankings align with policy goals; access and equity policy are areas where doing so is particularly important."

The report says: "Most of the indicators used in the construction of college rankings have little to do with policy goals relating to access and equity; they create uniform notions of educational quality and overlook important distinctions in educational preparation, personal experiences, and historical treatment of various student populations in higher education. Policy-makers and the public are ill-served by rankings that rely on data indicators that by their nature are exclusionary.

"Policy-makers should take note of the data inputs used to construct rankings and ensure that incentives for an institution to move up the rankings do not run counter to public policy goals, especially in the areas of access and equity."

The best-known American college and university rankings have been compiled since 1983 by the magazine US News & World Report and are based on data collected through an annual survey sent to each university or from the institution's website. US News also conducts surveys of university faculty and administrators.

Bob Morse, director of data research at US News, emphasises that while the IHEP report and its recommendations are aimed at higher education policy-makers and academics, the US News America's Best Colleges rankings are primarily geared toward consumers - prospective students and their parents both in the US and globally.

"Of course, it is true that the rankings are more and more being used by many colleges for benchmarking and other higher education policy goals. It is not the intent of US News to provide US higher education with such policy tools and US News has cautioned higher education about problems with using rankings for benchmarking and other goals.

"The IHEP report is correct to point out that leaders at colleges and universities need to consider that there are many complicated issues that can develop when they use our rankings to set policy goals. US News backs the goals of the report that institutions need to use rankings responsibly and to encourage them to publish more assessment and learning outcome data."

Some critics say that US News merely lists criteria that mirror the superficial characteristics of élite colleges and universities.

Kevin Carey of the think-tank Education Sector said in 2006: "[The] US News ranking system is deeply flawed. Instead of focusing on the fundamental issues of how well colleges and universities educate their students and how well they prepare them to be successful after college, the magazine's rankings are almost entirely a function of three factors: fame, wealth, and exclusivity." He suggested there were more important characteristics of significance, such as how well students learn and how likely students are to earn a degree.

The Role and Relevance of Rankings in Higher Education Policymaking
www.ihep.org

Free access version of the US News and World Report Best Colleges 2010 rankings at colleges.usnews.rankingsandreviews.com

david.jobbins@uw-news.com