A new project to benchmark the performance of higher education systems is on the cards at the Paris-based Organisation for Economic Co-operation and Development, or OECD, to replace the discredited Assessment of Higher Education Learning Outcomes, or AHELO, project, which failed to gain the backing of key OECD member states to go ahead.
The OECD’s Education Policy Committee is launching new work “to better monitor and analyse the changes taking place in the higher education sector, and to strengthen our evidence base, so that we can offer countries more robust policy advice in the future”, OECD Deputy Secretary-General Stefan Kapferer told the "Higher Education Futures" conference, organised jointly by the OECD and the Singapore Ministry of Education, on 14 October.
An annual project “benchmarking higher education systems performance” will provide the backbone for the OECD’s future work on higher education, Kapferer said in Singapore on Wednesday.
“This project will enable us to better assess how higher education systems are performing across a range of different performance dimensions, and to identify strengths and weaknesses within national higher education systems,” he said without providing specific details. He said only that different approaches were being examined.
“The data and indicators for this strand of work will continue to be developed and refined over time, giving us broad and deep understanding of higher education globally,” he added.
But he stressed it was not another ranking or another AHELO, but a means of providing “more transparency” about the real outcome of courses.
The reason many students around the world want to travel to only a few universities globally was because it was difficult to understand which university and which learning outcome are appropriate for a particular student, he said.
Popular universities for foreign students “have a high reputation. They have a very good name. They often have very good learning outcomes. But we do not really know enough about the learning outcomes and I think this is the most relevant topic for the future and we have to establish a system,” Kapferer said.
Frans van Vught, a policy adviser to the European Commission in Brussels and the former rector of the University of Twente in the Netherlands, emphasised a need for more information about outcomes of courses.
Higher education internationally was operating as a hybrid public-private “quasi market”, he told the conference, and it was a crucial market failure that clients-students and their families cannot get the information they need.
“They are unable to make rational informed choices. They pay more but they do not know more,” van Vught said. “Students can only judge the quality while they are actually experiencing it.”
The OECD’s Assessment of Higher Education Learning Outcomes, or AHELO, project has been described as the university-level equivalent of the OECD’s highly influential global Programme for International Student Assessment, or PISA, tests for schools that measures education learning outcomes at secondary level.
AHELO was intended to measure teaching quality rather than institutions’ past reputation.
However, in a letter sent to OECD Secretary-General Angel Gurría in May, the American Council on Education, or ACE, and Universities Canada, which represent the leadership of US and Canadian universities respectively, objected to AHELO’s “one-size-fits-all” approach.
“AHELO, which attempts to standardise outcomes and use them as a way to evaluate the performance of different institutions, is deeply flawed,” states the letter.
An AHELO pilot involving some 17 countries “was deemed by most to be a failure, and it is very difficult to see how a resurrection of the project would yield any better results”, according to Philip Altbach, research professor and founding director of the Center for International Higher Education at Boston College, USA.
“It seems highly unlikely that a common benchmark can be obtained for comparing achievements in a range of quite different countries. Indeed, post-secondary studies start at different ages globally,” Altbach has said.
At the conference in Singapore on Thursday, Simon Marginson, professor of international higher education at the UCL Institute of Education, University College London, said: “It does seem as if AHELO has fallen over. It has clearly been on the rocks for about 18 months or two years, and that will create a whole vacuum in that potential area of rigorous measures [on student achievement and learning outcomes].”
“Clearly we need to be able to measure general cognitive development, discipline-based learning and work-related skills, and these are three distinct domains. We need to measure all of them or we tip the balance too far towards one or the other.”
It was important, he said, because “we need to give learning outcomes equal status with research”, and to provide a better basis for student choice. It was also necessary to have something more solid than subjective “market-oriented student satisfaction surveys” as a basis for measurement.
Marginson said that such measurements were possible where governments had stronger connections with universities than in the US, UK and some European countries, particularly in East Asia, and that could give the region a relative advantage to pioneer something regionally that may not have been possible on a comparative level internationally.
Learning outcomes still on the agenda
Kapferer made it clear the OECD had not abandoned its attempts to compare learning outcomes internationally, as part of the organisation’s efforts to strengthen its work on higher education.
“The backbone of our work on higher education will be to try to organise a situation where results of outcomes of learning at higher education institutions are more comparable,” he told the conference.
He acknowledged that AHELO “was not yet as successful as we have wished for because not enough member countries were interested in starting it”.
“But we still believe – and we will discuss it at the education policy committee in November – that we need an approach... for the learning outcomes.” He indicated an official announcement was likely in the next few weeks.
“A key aim of this [new] project is to facilitate the sharing of higher education policy developments across countries and learning from each other’s experiences about how to tackle some of the common challenges higher education systems are facing,” Kapferer said.
Van Vught believed there was some room for cross-system comparisons of learning outcomes, but “we will never be able to find a best set of learning outcomes that are applicable across all [higher education] systems per field or per discipline in terms of student learning”, he said.
Higher education’s future – Asia and technology
Hubs to take elite universities into world-class club
Asia emerging as ‘third pole’ in higher education
AHELO: the myth of measurement and comparability
The Holy Grail of learning outcomes
Receive UWN's free weekly e-newsletters