GLOBAL: Reputation surveys important to rankings

One of the most common and, let's be honest, most vociferous complaints about university rankings is their use of reputation surveys. At a recent series of high-level events on the growing influence of university rankings, from Paris to Toronto via Brussels, I witnessed the criticisms boil over: reputational ratings are based on style over substance, the critics cried.

They conform to stereotypes and perpetuate historical hierarchies. As journalist Malcolm Gladwell put it in The New Yorker, reputation measures reflect little more than "prejudices".

There is no doubt that many in higher education remain to be convinced of the merits of reputational surveys.

But Times Higher Education magazine knows of at least 30,942 university staff, from 149 countries, who are persuaded of the merits of such exercises: they are the individuals who have taken part in the annual Academic Reputation Survey that fuels the Times Higher Education rankings, since the new and improved tables were first introduced in 2010.

Carried out by our rankings data provider Thomson Reuters, with the expert polling company Ipsos MediaCT, the reputation survey has taken place just twice. In three months in spring 2010, 13,388 people responded.

This spring, following the launch of a new methodology that has made the Times Higher Education World University Rankings the most frequently quoted in the world, 17,554 entirely different people completed the survey - a 31% increase from the first year.

The survey, even in its infancy, is already clearly established as the largest exercise of its kind undertaken anywhere in the world, producing an exceptionally rich data set.

The unprecedented volume of responses is a resounding testament to the widespread engagement with our world rankings project: each respondent took the time to fill out our detailed and comprehensive questionnaire, answering questions on both research and teaching, and the only inducement they were offered was a few free electronic copies of THE magazine, and the chance to get a detailed report back on the results of the poll.

There were no gimmicks, no prize draws for those who participated: just the satisfaction of contributing to a serious piece of global research of interest to all of world higher education. There was no duplication of invitations from one year to the next, and each respondent was hand-picked and invited to take part because they were statistically representative of both their country, based on UNESCO data, and of their academic discipline.

We have a strict policy: we accept no volunteers, or nominations of potential respondents. And no data from any one year will be merged with data from previous years to artificially bolster the headline sample size.

The result of all this is a high quality sample: 90% of respondents described their role as an academic, researcher or institutional leader and respondents had spent on average 16 years in the academy.

There is a spread of responses across regions and disciplines, with the regional distribution of the survey following United Nations percentage estimates of global academic researchers by sub-continent. Thomson Reuters also does analysis after the results come in, to counter any response bias by region.

Some 36% of respondents in 2011 were from North America, 28% from Europe, 18% from Asia and the Middle East, 7% from Latin America, 7% from Oceania, 4% from Africa and 1% from Central America and the Caribbean. The survey was distributed in nine languages including Chinese, Portuguese and Spanish, and for the first time this year, in Arabic,

The physical sciences and engineering and technology took about 20% of the responses each, with 19% for social sciences, 17% for clinical subjects, 16% for life sciences and 7% for the arts and humanities.

But a large and high quality sample needs to be properly harnessed by a high quality survey methodology.

One of the big criticisms of reputation measures by Malcolm Gladwell, in his now famous February 2011 New Yorker piece, "The Order of Things", was based on a famous domestic US ranking. This asks college presidents to grade every school in their category on a scale of one to five, with some presidents asked to rate up to 261 other institutions.

"It's far from clear how one individual could have insight into that many institutions," Gladwell writes. Worse, there is evidence that, when confronted with such a list, the university presidents simply turn to last year's rankings for guidance.

But even within Gladwell's devastating critique of the US domestic rankings, some hope for reputational surveys is offered.

Michael Bastedo, a sociologist at the University of Michigan, is reported as saying that some reputation surveys can work very well. Gladwell paraphrases Bastedo: "It makes sense, for example, to ask professors within a field to rate others in their field: they read one another's work, attend the same conferences and hire one another's graduate students, so they have real knowledge on which to base an opinion."

This is exactly the approach taken by Times Higher Education.

To obtain meaningful responses, the survey for Times Higher Education asks respondents to judge institutions only in their own fields, based as much as possible on direct, personal experience. It asks action-based questions, asking where a respondent with a highly talented undergraduate student would recommend the best postgraduate teaching.

Crucially, while the survey includes 6,000 universities in its menu, it requires that each respondent lists no more than 15 institutions that they regard to be producing the best research, or responsible for the best teaching, first in the region where they have most experience, then globally. The methodology and survey instrument are published for open scrutiny.

So Times Higher Education is fully satisfied that its reputation survey has serious rigour, and produces serious results.

But despite this, despite the investment by us and the global academic community in a high quality piece of research, we do accept that reputation is subjective, and a reputation for excellence among informed and engaged academic peers is just part of the detailed and complex picture of a world-class, research-driven university.

That is why when the 2011-12 Times Higher Education World University Rankings are published in a few months time, the reputation survey will be used to provide just two of at least 13 separate performance indicators, with objective indicators given the lion's share in making up each university's overall ranking position.

Times Higher Education is committed to producing an annual ranking that is of serious value to our core readers - academics and administrators working in universities around the world - as well as to the many millions of students who consult it. Our own reputation depends on it.

* Phil Baty is Editor of Times Higher Education World University Rankings, and Deputy Editor of Times Higher Education magazine.

Comment:
So a long defence of reputational indicators ends up by admitting that they are subjective and of limited value and that's why the lion's share of the THE ranking is based on "objective indicators".

A good defence of the THE ranking, or a clever attempt to deflect attention away from the equally limited value of these "objective indicators"?

David Crosier