A recent study by the European Commission's Joint Research Centre has questioned the substance of statistics on which the Academic Ranking of World Universities conducted by Shanghai Jiao Tong University are based. The study, carried out by researchers at the Centre for Research on Lifelong Learning (CRELL), questions whether the Jiao Tong ranking serves its intended purpose, compares it with the Times Higher Education-QS rankings exercise, and concludes that neither system succeeds in effectively ranking Europe's universities.

Higher Education Rankings: Robustness Issues and Critical Assessment, written by Michaela Saisana and Beatrice D'Hombres, contends that the Jiao Tong system is statistically robust only in the identification of "top performers" either side of the Atlantic and is quite unreliable in its classification of other institutions. Also, Jiao Tong focuses exclusively on universities' research performance. While the THE-QS ranking includes measures of educational quality, it is fragile in its ranking approach which is biased towards British institutions, the authors write, and inconsistent in its treatment of data and variables extracted from surveys.

They find, among other things, that while indicators and league tables are enough to start a discussion on higher education issues in Europe and benchmark it worldwide, "they are not sufficient to conclude it". Secondly, a robustness analysis reveals that the rank of more than half of the institutions is "highly sensitive to the methodological assumptions and the choice of indicators. Thus no conclusive inference regarding the relative performance for the majority of the universities can be drawn from either ranking."

The JRC report points to the need for a ranking system suited to European universities in order to make institutions more accurately comparable, encourage student mobility and facilitate the implementation of a European Research Area. In the meantime, it suggests that an assessment of university performance based on a hybrid set of the 12 indicators used in the THES and SJTU rankings may provide a more reliable average ranking of the institutions.
Full report on the CRELL site