Measuring university impact – A rankings innovation

The University Impact Rankings 2019 published by Times Higher Education is a novel initiative that has captured global attention because it aims to measure the extent to which universities are working towards fulfilling the United Nations Sustainable Development Goals, or SDGs.

This ranking is the first launch product Times Higher Education, or THE, has under the ownership of private equity firm Inflexion. THE is positioning itself away from the newspaper business to one of business and data analytics.

One thing that clearly stands out from this exercise is the willingness of institutions to participate. It required a significant investment of time, staff resources and coordination to ensure institutions’ data, policies and evidence were compiled, verified and submitted to THE by the due date.

Out of the 556 institutions that submitted data on one or more of the SDGs, there were 141 institutions (25%) which submitted data on all of the 11 SDGs that feature in the ranking.

Guidelines were issued for the new ranking by THE in early October 2018. Institutions were asked to submit data and evidence by late January 2019. For a new ranking schema – and with no previous precedent of data gathering against the UN SDGs (other than what institutions may report as part of their environmental and sustainability activity report) – this was a massive effort on the part of institutions to meet THE deadlines.

Institutions that participated in the ranking can assess the evidence they produced and ascertain the effort required for improvement. Also, they can determine their relative progress towards fulfilling the UN goals. In addition, institutions have a roadmap for embedding SDGs in their learning and teaching.

Neither a ranking of excellence nor reputation

The THE University Impact Rankings are not designed to measure an institution’s excellence nor its reputation. Rather, it is an assessment of performance against 11 SDGs that THE considers most salient to higher education.

With this ranking, there is the danger that focus will be placed on the performance of the world’s top 100 as reflected on the World University Rankings. It would be a pity if that happens as it would detract from every institution’s efforts to address inequality and every other SDG goal.

The ranking is not authoritative in measuring the extent and magnitude of an institution’s impact on the economy, society and environment. While offering useful insights into the contributions universities make to the economic and societal development of communities they serve, the THE University Impact Rankings can be viewed as a conversation starter.

The probabilities for non-research-intensive and less well-resourced universities to rank high are limited because, for 10 of the 11 SDGs, research metrics weigh 27% of the score. For SDG 9 (industry, innovation and infrastructure), the entire emphasis is on research output and impact (for instance, publication and income research commercialisation).

There are 111 measurements that THE developed for the ranking exercise. This is a commendable effort, particularly when there has not been a uniform approach or framework that enables an authoritative analysis of universities’ social and community impacts.

Some of the measurements may need further refinement, such as the proportion of a university’s total research output that is authored by women. The gender of authors was estimated by Elsevier and not by institutions.

Visibility of the keyword phrases used by THE to map bibliometric information against every SDG would be helpful to institutions.

Another methodological refinement may be in reducing the weight given to the research metrics. It may be better to give more emphasis to the knowledge footprint of an institution and its impact on the local and regional communities they serve and in which they operate.

By placing greater emphasis on non-research metrics, the THE University Impact Rankings will provide a more level playing field to many institutions across world regions that would otherwise not have a chance to shine.

Because of the nature of the metrics within each SDG, it is not surprising that universities from Australia, Canada, Finland, New Zealand, South Korea, Sweden and the United Kingdom dominated in the top 10 and top 20.

In this ranking – as in most global rankings – we see that institutions from the most advanced educational systems shine. It is also a reminder of the importance of developing robust systems for collecting statistical information across the entire suite of universities’ endeavours.

This ranking may be better suited to be compared on a national or regional basis, within the context of individual SDGs and the university mission.

You are what you measure

The expression ‘you are what you measure’ resonates in this instance. This ranking demonstrates that institutions are what they can measure. It also highlights the propensity for universities to engage with others and be subject to scrutiny and external judgement.

Out of the 11 SDGs for which information was collected – apart from SDG 17 (partnerships for the goals) which was compulsory – there are two that stand out: SDG 3 (good health and well-being) and SDG 4 (quality education), because these were completed by most participating institutions.

By contrast, less than 50% of participating institutions chose to provide data and evidence in relation to SDG 12 (responsible consumption and production) and SDG 13 (climate action). Thus, there is some work to do for universities on these SDGs.

Sustainable relationships

THE can take due credit that its efforts paid off in producing this innovative ranking. Credit is also due to the more than 500 institutions globally that partnered with THE in this endeavour. There will have been many more institutions who pondered on submitting data, but decided not to enter yet another ranking race at this stage.

Future iterations of this ranking are likely to show volatility. Invariably new entrants are likely to rank higher than some universities that have taken part in this first edition.

Methodological refinements are required to ensure universities from middle-income economies have the opportunity to shine rather than continue to entrench institutional inequality on a global scale. The THE University Impact Ranking may be one that is best conducted on a regional basis rather than through a global list.

Angel Calderon is principal advisor, planning and research, at RMIT University in Australia. He is a rankings expert and a Latin American specialist.