AUSTRALIA: Troubled history of an ERA

In what must be an embarrassing backdown for the Australian Research Council and the Minister for Science and Research, Senator Kim Carr, one element of the controversial Excellence in Research for Australia assessment scheme has been scrapped: the scheme's method of ranking academic journals which had been roundly criticised from the moment anyone started to take notice of it.

Even if the head of the research council and the Minister insisted the journal ranking scheme had wide support throughout the sector, they were either listening to the wrong people or weren't listening at all. As well as the several scholarly papers published on the shortcomings of the scheme, regular media commentary was all negative.

Under the ERA scheme for ranking journals, journals were ranked according to their alleged 'quality' on an ordinal scale from 0% to 100%, with the top 5% of journals being designated as 'A*'. The next 15% of journals were designated 'A', with 'B' and 'C' journals
containing 30% and 50% respectively.

The underlying assumption of the journal ranking scheme was that the higher-rated journals ('A*' and 'A') would contain scholarly papers that were of a higher standard than ranks 'B' and 'C'. Of course, there isn't necessarily any link between the quality of a paper and the journal it is published in.

According to a ministerial statement released last Monday, problems arose because universities deployed the rankings inappropriately, 'in ways that could produce harmful outcomes, based on a poor understanding of the actual role of the rankings'.

This 'poor understanding' led to institutional managers targeting journals only from the top 20% of journals and, in many cases, obstructing their academics from seeking to publish in the other 80%. In effect, at some universities, economic rationalism overtook research impact as a driver of research. Of course the Minister's observation is somewhat disingenuous: what else did he expect universities would do?

Despite the tide of dissent, the research council and the Minister stuck by their guns, even when it became more and more obvious the scheme would have a serious negative impact on Australian policy research, multi-disciplinary research and a range of other areas.

The universities themselves remained fairly silent throughout this debate, apparently deciding to toe the line rather than speak out about the system's illogical and non-transparent nature. Many academics remained silent also for fear of upsetting their heads of department, deans and research managers. Academic employment is precarious for many of them.

Perhaps the best known direct effect of ERA journal ranking was the closure of the Monash University journal People and Place after 18 years. This was an extremely influential journal but it had been demoted by the invisible judges that ranked journals.

Monash withdrew its financial support and a publication that had brought to light many newsworthy social policy issues was to be no more. It was subsequently noted that academics who work in the areas covered by Australian social policy typically publish their research in journals ranked 'B' and 'C'.

Some journals were 'demoted' for no apparent reason between the first and second periods of public consultation of the journal ranking process and were relocated to a different field of research. The research council never explained how this could happen but instead relied on very general information on its website. The invisibility of the judges was a major problem, particularly as to whether the judges were competent to judge the alleged quality of a given journal relative to other journals. Rankings based on possible conflicts of interest were also an issue.

When the process was to be revamped for the next round of the research assessment exercise in 2012, and as part of the consultation process, individual academics could take part in consultations by registering and commenting on various journals of interest to them. This led to journal editors, proprietors and publishers lobbying their various constituencies in an effort to have them register with and comment in glowing terms about this journal or that. This attempt to influence the process by those with a vested interest was one of the several 'unforseen circumstances' that became evident during the journal ranking.

Academics will now be relieved that the Minister and the ARC have back-flipped on journal ranking. The process was a bureaucratic and expensive nuisance for academics and universities alike. It is to be hoped that what has happened will be kept in mind so a repetition of this failed process will be avoided in future.

* Dr Ian R Dobson is editor of the Australian Universities' Review and the Journal of Higher Education Policy and Management. He is also a research director at the University of Helsinki and holds an adjunct post at Monash University.

Comment:
Great. So now instead of having publications judged according to a systematic, public and open rating of journals, they will now be judged in a way that is ad hoc, private and closed.

Gavin Moodie