Korean universities unite against QS ranking changes

In an unprecedented move, all of South Korea’s top research universities have come together in a call to boycott global university rankings issued by QS, saying the United Kingdom-based ranking organisation has not heeded their claim that changes made to the rankings this year are not transparent and contain “mathematical flaws”.

The 52 universities in South Korea united in setting up the University Rankings Forum of Korea (URFK) last month in a concerted action against methodology changes in the QS World University Rankings released at the end of June.

All Korean universities except one experienced a drop in the QS ranking this year, as did a number of well-known Japanese universities, such as Tokyo Institute of Technology, universities in Hong Kong and some in Taiwan.

They believe this was due to changes relating, in particular, to QS’ International Research Network (IRN) indicator, newly added this year, which the group says is unreasonable for non-English speaking countries. Australian universities rose this year on this indicator.

But QS says this is not primary cause of the shift in Korean universities’ positions, which it argues is the reduced emphasis on the faculty-student ratio indicator, whose weighting has been halved.

URFK includes the country’s most prestigious institutions including Seoul National University (SNU), Korea University, and Yonsei University (collectively known as SKY), as well as major research and technology universities such as the Korea Advanced Institute of Science and Technology (KAIST) and private institutions such as Pohang University of Science and Technology (POSTECH).

Some Korean universities dropped by 200 to 300 places under the new methodology. However, URFK said that their key concern was the magnitude of the drop for Korean universities as a whole, not just the drops for specific universities.

“The magnitude is unthinkably large,” URFK said in an email just after the QS rankings release on 27 June.

“The competitiveness of Korean universities is steadily increasing in many respects. Thus, there is no reason that Korean university rankings [should] drop this drastically,” URFK said, adding “the change should be step by step, not this radical and abrupt”.

It described the rankings as having “mathematical flaws”, which should be corrected, and said it had requested such “corrections” from QS prior to the release of the June ranking. They now say they will withdraw permanently from QS rankings, withholding their data, unless the methodology is made more transparent and the flaws “corrected”.

Only Sejong University rose by 150 places in the latest ranking but nonetheless is participating in the call to boycott. “Sejong University might have got even better scores with the old methodology,” Kilsun Kim, vice-president for public affairs at Sogang University, which is also part of the URFK group, pointed out.

QS denies miscalculation or mistakes

In response to the Korean action, Ben Sowter senior vice-president, institutional performance, at QS said: “We have gone and double checked our data, we have gone and recalculated indices, and made sure that at least against our intended methodology, there hasn’t been some calculation or mistake that caused this challenge [by Korean universities].”

He pointed to highly weighted indicators for research power, including citations which are 50% of scores overall. “So there’s no shortage of emphasis, that gets credit for a lot of those fantastic things that are done.”

Simona Bizzozero, QS director of communications, said the change to include use of the IRN indicator, which is designed to capture the diversity of cross-border research collaboration, has been well documented and “part of our evaluation for our Asia Rankings since 2018 and for the World University Rankings by Subject for the past two editions. Until now, this inclusion has not raised any concerns.”

Unusual show of unity

It is not unusual for individual universities to try to justify falls in global university rankings from year to year, or even for rankings changes to be met with howls of protest, and even ‘boycotts’ of rankings by small groups of universities such as some Indian Institutes of Technology, and some Chinese universities, who withhold information for the Times Higher Education (THE) rankings while calling for rankings adjustments.

However, a coordinated country-wide boycott has not occurred before.

Seungbum Hong, head of the department of materials science and engineering at KAIST and the institution’s global competitiveness advisor, told University World News that it was highly unusual for more than 50 Korean universities to coordinate in this way over any issue.

“This is a unique moment for us that we united together because every university suffered from this curious, intriguing index.”

Hong said that prior to the release of this year’s rankings they asked QS to explain how the new indices were calculated, “but also to specifically explain why Asian universities and science and technology universities have experienced such a hit”.

IRN now accounts for 5% of the total score, while faculty-student ratios were reduced by 10%. This affected countries with good faculty-student ratios due to demographic decline, such as South Korea, Japan and Taiwan.

“It is not just South Korean universities but other science and technology-oriented universities in the world also experienced the same drop. So, we raised our voice to say that the IRN formula is not fair to assess global engagement, which is the philosophy behind this indicator,” Hong said.

URFK President Euiho Suh, professor emeritus at the department of industrial engineering at POSTECH, told University World News that they wanted more transparency on why Korean universities had collectively done so badly this year.

He said URFK had requested a postponement of this year’s rankings release for the problems associated with the change in QS’ methodology to be corrected. But this did not happen.

“QS continues to say that our ranking is not that bad and that all [Korean universities] are doing OK. But this is not the problem. The issue is a serious mistake [in the calculations] and they [QS] need to accept that it is a problem,” Suh told University World News.

The group also argues that another new indicator on Sustainability was also “arbitrarily” calculated without providing sufficient definition, making it difficult to know the consistency of the scores.

Questions around IRN formula

According to URFK, the current IRN formula requires universities to increase the number of countries they associate with. IRN is extremely disadvantageous to those universities which are exclusively associated with strong research universities in a specific country (such as ones in the US), even if those universities have country-like diversity, it said.

“The assumption that an institution with more partners typically has them distributed across more places is not right. It may be true for some comprehensive universities with more languages and cultural programmes. But, for universities specialised in some specific areas, the number of partners is limited.

“Thus, it is not realistic that we demand professors find new locations when they (co-)publish papers,” it said.

Hong argued that it was illogical that under QS’ current IRN indicator collaborating with many institutions in a single country – such as the United States in the case of Korean universities – “you are seen as not being globally engaged”.

Kim told University World News that “Sogang University’s overall score [would be] better if you apply last year’s methodology. With the three new indices our ranking was about 50 lower. Most Korean schools got lower rankings.

“Sogang University got about 1 point in the IRN index. So, we wanted to know which school got the best score as we want to learn from them, but QS are not telling us that. Among Korean universities most are around 1 point [out of 100] and just Seoul National University got about 30.”

He added: “Universities in Europe have a better environment to do research with an institute in another country. But, for universities in Korea it requires more effort and cost to increase the number of countries.”

There is “an easy way” according to the current methodology, Kim noted. “We can let each professor co-author a number of papers with a different assigned country, and as we do publish quite many co-authored papers we can get a way higher score, but that’s not how international research cooperation should work.”

Jongkyu Kim, professor in the department of materials science and engineering at POSTECH, told University World News that his institution was a small one specialising in science and technology.

“Schools like ours are highly affected by the new index as the number of countries are critical in the IRN index,” he said.

“Universities with just science and technology orientations have a limited option to cooperate, especially in the terms of countries. It is difficult to do meaningful research cooperation with institutes in underdeveloped countries in the field of science or technology. That is why POSTECH and UNIST [Ulsan National Institute of Science and Technology] just got 1.3 and 1.2 [out of 100] in this index.”

KAIST’s score on the IRN was just 10 points out of a possible 100, despite a branch campus in Africa, and other major global collaborations.

First major change in decades

For QS, Ben Sowter noted that since the QS World University Rankings were launched two decades ago, “we’ve been undertaking a programme of continuous improvement, refining those indicators to be more comprehensive and fair over that time. But we haven't made any major changes to the top line weights and measures until this year”.

Three new indicators were introduced – employment outcomes, IRN and sustainability, each weighted at just 5%, so QS had to take away from weightings on other things, he explained, such as the weighting of faculty-student ratios.

Sowter acknowledges a significant number of institutions in Korea have ranked a substantial number of places – in some cases hundreds of places – lower.

“One of the biggest drivers of system wide shifts, a bit like the one we're seeing in Korea, has been not because of the new indicators, but because of the reduction in emphasis on faculty-student ratio,” he said.

“That reduction in emphasis has been something that a lot of commentators in the rankings have been suggesting for a very long time. The faculty-student ratio isn’t the strongest measure of quality and capability to deliver good teaching.”

As for the new metrics, he noted, SNU is eighth in the world in new employment outcomes metric and pointed to two other Korean universities in the top 100 for that metric.

“Korea punches above its weight in terms of the strength of its universities,” Sowter said, arguing, “it is not the case that Korea has done very badly”.

Sowter also noted that the majority of Korean institutions’ collaborations are “substantially dominated” by partnership with US institutions, “which is what we're really trying to measure with this indicator [IRN], … is not trying to not look at the size of their research network, but the number of location nodes”.

Sowter added: “It’s designed to encourage institutions to diversify their international engagement rather than simply to grow it. Our indicator is designed to help identify institutions, even those that aren’t producing a huge amount of research, even those that have only got a small number of partners, to recognise the extra effort that it takes to diversify at that early stage to reach out to new places, and to and to bring the right minds together to solve to solve big problems.”

What happens with a boycott?

QS said that it does not like to see universities leave the rankings, but if Korean universities were to do so, it would use other means to obtain the data. “Ultimately, we do have a methodology that relies very little on self-reported data. There are global, or in some cases national data sources on student numbers, faculty numbers, and much of the information that we need,” said Sowter.

“We prefer to work hand in glove with institutions for data collection efforts, but we can typically find the data we need to publish a complete ranking.”

This appears to have angered URFK even more. URFK’s Suh said, “So, although we do not want to participate in the ranking, which means we do not submit the data, they will still rank us. That’s a real problem.”

According to Suh, some universities wanted to sue if that happens. “If these are data from other sources, we do not accept it, because from other sources the data may be incorrect,” he noted.

Simona Bizzozzero, communications director at QS, said that an online meeting was held on 3 July with the group of Korean universities. “The interaction was challenging but also positive and productive, leading to a greater understanding of our shared objectives.”

She told University World News: “We are pleased with this constructive engagement and are committed to fostering this relationship further.”

Sowter said: “We will continue to look at these ranking measures and work on additional datasets that can fuel these measures and look at how we can continue to calibrate them to capture the rich fabric of international higher education.”