There are better ways to capture graduate outcomes data

Early indications are that the United Kingdom Higher Education Statistics Agency’s new Graduate Outcomes survey, due to be published in spring 2020, is in danger of falling short of benchmark response rates and is even undermining the credibility of the Teaching Excellence Framework (TEF).

As importantly, the benefit to students of high-quality data on graduate outcomes would be lost. It’s a toxic mix of outcomes fuelled by outdated data collection methodology and sub-optimal engagement strategies.

The Graduate Outcomes survey will look at graduate employment 15 months after leaving universities and is the successor to the Destinations of Leavers from Higher Education (DLHE) survey, which took those details after six months.

It’s a laudable effort to get a more realistic picture of the immediate benefits of a university education, but it has come with some compromises. Target response rates for home students have been reduced to 60% from the 80% required in DLHE.

The 2016-17 Higher Education Statistics Agency (HESA) DLHE returns were approximately 400,000, making it the largest annual social survey in the UK. To date the returns of the UK Graduate Outcomes survey come up just short of 78,000 (Cohort A, April 2019 55,416; and Cohort B, July 2019 18,472).

The substantial decline in responses from the first to the second cohort suggests that, even with two further cohorts to come, Graduate Outcomes may receive less than half the responses of the 2016-17 DLHE.

Despite the lowering of the target response rates, responses are already lower than planned. The 17 July update shows that there is a long way to go and the text begins to prepare the ground for missing the 60% target. It shows the Cohort C halfway response rate at 35.8%.

Although it says HESA is optimistic that it will “get much closer to the target” by the end of the fieldwork period, it adds that researchers are conscious that the sample becomes “increasingly tired towards the later stages” and that graduates become “harder to reach”.

Problems for ministers, institutions and students

The consequences of failure should be of concern to decision-makers in government at a point when they are trying to build transparency and credibility in the data informing student choices.

Former Universities Minister Jo Johnson, who left the Government last week, was one of the authors of the 2015 Conservative manifesto that contained the line “if there’s a REF [Research Excellence Framework], there should be a TEF”.

It will be interesting to see how his former department responds to learning that an important input into the TEF rankings might be tainted by smaller than anticipated sample sizes.

Poor response rates for HESA’s Graduate Outcomes are sure to be of significant concern for the sector more broadly. A number of higher education leaders are worried about how the conduct of the survey might have an “impact on alumni engagement” and will realise that this level of response will not provide credible data. The impact on TEF rankings will also be on their minds.

With students already seeing shock headlines, however biased, about the value of a degree, rampant grade inflation and lack of transparency, this would be another own goal for the sector. People entering higher education have a right to be able to trust the information they receive about ways of measuring their investment of time and money. The Graduate Outcomes survey is a key resource for them and should be a high priority.

Perhaps most importantly, information on graduate outcomes is increasingly important in helping students make their choices and in informing institutional strategies. As global competitiveness increases, the importance of accurate information on the value of a UK university education is a matter of national importance in supporting Brand UK.

Using all the tools available makes sense now and will become even more critical as the technology improves.

What has gone wrong?

Without being inside the organisation surveying UK graduates it is difficult to know why they are struggling. But information on the reviews for each cohort shows that for each tranche of data collection there are approximately 10 contacts with alumni. It’s mainly emails and SMS messages although it claims to be based on best practice data collection and research methods.

There’s a strong argument that this frequency is already getting close to ‘spamming’ alumni and that the likely shortfall might see even more unsolicited emails being sent.

It should be a concern for the universities that their alumni are either unprepared or unwilling to engage more actively and it would be interesting to see how many are using other alumni channels to help build responses.

The ‘spamming’ communications strategy being used for the Graduate Outcomes survey is not working very well and is likely to impair the way universities build relationships with their alumni. A more coherent and coordinated strategy needs to be considered as a matter of urgency.

As ‘graduate outcomes’ become increasingly important to both domestic and international students when choosing where to study, it is not acceptable that by changing the way data is collected and outsourcing it to the private sector at considerable expense there will be a far inferior data set. This change will do both UK higher education institutions and perspective students deciding on their future a great disservice.

Towards a better approach

In an era of social media, virtual business networks, big data and artificial intelligence, there are better ways of capturing data and analysing it to achieve governmental and individual institutions’ aims of understanding their graduate outcomes.

There are plenty of lessons to be learnt from other industries that have embraced big data:

• • The customer data science company Dunnhumby has revolutionised the retail sector – no longer do supermarkets need to doorstep shoppers and look in their baskets or send endless retail surveys to members of the public.
  
  
• • Many transport systems now function through smart cards – Oyster in London and Touch ’n Go in Malaysia to give just two examples. This means that city planners can map journeys and improve transport systems without asking the general public how long it takes to get from A to B. The data is captured as part of their daily journey to work.
  
  
• • Social media does not have to ask what advertising users want to see. It is captured through social media preferences on a minute by minute basis as they engage with platforms.

Institutions already hold data on their graduates through increasingly successful alumni networks and graduates are actively engaging with social media, business networks, jobs sites and agencies. Well-organised institutions already ensure that they are able to use alumni data for appropriate purposes and there is enormous potential to align this with data that is in the public domain.

Prudent use of these resources allows ethical behaviour that respects confidentiality, General Data Protection Regulation issues and the rights of individuals but opens up the ability to optimise data for the benefit of graduates, universities and employers.

Some agencies and companies are already grasping the nettle in this regard. QS (previously Hobsons) captures data on prospective students through their Enrolment Solutions platforms without the need to survey them. Data HE is actively using HESA data, the majority of which is in the public domain, to analyse student recruitment and course preferences so there is no need to survey students.

The Asia Careers Group uses innovative data techniques to capture employment and career progression information in the public domain for over 50,000 UK graduates returning to Asia.

Surveys have their place, but students and graduates are already subject to a bewildering range of communications and it is little surprise that the UK Graduate Outcomes survey is struggling to get their attention.

It is also arguable that contacting a graduate for the first time more than a year after they leave university and then harassing them with increasing numbers of emails is an approach that is outdated, annoying and likely to result in disappointing results. It’s time to embrace the fourth industrial revolution and relieve their bulging inboxes.

Louise Nicol is director of Asia Careers Group SDN BHD, a company based in Kuala Lumpur, Malaysia, which provides longitudinal graduate outcomes and average salary data to both the UK and ASEAN governments and UK higher education institutions.