Should we rank universities by equity performance?

University ranking systems are ubiquitous, yet most do not take into account the extent to which institutions’ enrolments reflect the diversity of the population from which they draw their students and thus the extent to which they contribute to equitable outcomes in terms of access to higher education.

There are exceptions. The US News Education Rankings includes some measures of ‘equity performance’, such as the graduation rates for recipients of funding under the United States’ Pell Grant scheme. Another, the Social Mobility Index, includes only equity performance measures.

Yet these ranking systems, and university rankings in general, suffer from the same issue, namely that they only draw upon extant data not designed specifically for an equity ranking system. In other words, these ranking systems value only what they can measure, instead of measuring what they should be valuing.

Measuring what we value

In 2018, with support from the Australian government’s Department of Education, Skills and Employment, we investigated whether it would be possible to construct a higher education equity ranking system using indicators generated for the sole purpose of measuring equity performance.

While we found that it was possible to rank institutions in this way, there were some caveats that need to be carefully considered before incorporating such a system into policy and practice.

First, there does not appear to be a universal definition of what ‘equity’ in higher education is. Our research showed that there was broad agreement regarding measures around widening participation, retention and completion.

There was also strong, but not necessarily majority, support for measuring activities that raised higher education aspirations (for instance, community outreach), academic preparation (such as foundation courses), student satisfaction with teaching and, ultimately, graduate outcomes.

This constitutes a wide array of indicators from across the student engagement cycle. However, although it may be impossible to arrive at a universally accepted definition of equity performance, it should be possible to arrive at a pragmatic one. For example, if a government wishes to widen access and participation, then its definition of equity should reflect this.

Weightings

Another problem emerged. In consultation with university administrators and stakeholders we found it is equally difficult to achieve consensus regarding indicator weightings.

For example, all stakeholders might agree that their definition of equity is excellence across the engagement cycle – from access (application and enrolment) to success (academic and employment outcomes). But are they equally important? If not, is access more important? And if so, is it twice, or three times, as important as academic success?

Our study confirmed that in Australia there is an inverse correlation between certain measures of access and success. Universities that enrol, proportionally, above average rates of equity-group students tend to have below average rates of first-year retention, and vice versa – a finding that is very much applicable to most university systems. This means that equity rankings are dependent on the choice of weighting given to indicators.

We also determined that equity performance is highly contextual. For instance, in Australia this plays out along geographical lines, with the majority of universities established in major cities and around 90% of domestic students applying to an institution in their home state or territory. In the United Kingdom, a similar stratification can be noticed by neighbourhood or the type of school a student attended. This plays out in assessing the impact of equity factors.

Consider the case where universities are ranked on how accessible they are for working-class students. In Australia, this is measured by the location of the student, each relating to a specific socio-economic status or SES. A university located in a high-SES region will have to expend significant resources to achieve the same numbers that a university located in a low-SES area would by default.

Consequently, performance can be measured both absolutely (that is, how well the institution is performing compared to all other institutions) and relatively (that is, how well the institution is performing compared to institutions in the same situation). Hence, a ranking will only ever measure relative performance because it can only provide comparisons within the group being ranked, not insights into actual performance.

We believe that while it is possible to construct a higher education equity ranking system, policy-makers need to ensure that it is fit for the purpose intended rather than a generic measure if they are to derive the full benefit of introducing it.

Tim Pitman is a researcher of higher education policy at Curtin University in Australia. He will be hosting a Centre for Global Higher Education webinar on this topic on Thursday 14 May 2020, from 14:00 to 15:00 (BST). You can click here for further details and to register.