AUSTRALIA: First research exercise a mixed bag

Depending on who is doing the commenting, the research performance of Australian universities is either world-ranking or pretty damn miserable. The Excellence in Research for Australia report was released on Monday to a distinctly mixed reception.

Professor Margaret Sheil, Chief Executive of the Australian Research Council, which was the main agency for the ERA initiative, noted that the report offered for the first time a comprehensive evaluation of Australia's research achievements against those of its global peers.

Writing a commentary in The Australian newspaper, Sheil said: "The picture is impressive. In total, 65% of units were assessed as performing at world standard, including 21% above and 13% well above the rest of the world."

But on page one, the paper led with a news story that declared: "More than two-thirds of Australia's universities have an overall research performance that doesn't reach international benchmarks, a finding that could have implications for more than A$1.5 billion in tertiary education funding..."

The Australian's higher education writers did their own analysis of the ERA report and, by averaging scores across the different subject areas, found that only 12 universities were performing research at or above international standard, with the top four performing at a rate that could be considered well above that level.

"The remaining 29 out of 41 higher education institutions have an average research performance that does not meet international benchmarks," the analysis found.

The first respondent to comment on the story, 'David of Adelaide', pointed to what he saw as a fundamental flaw in the analysis: "The ARC has just spent years finding a way to judge university research by field of research so we can see where the real discipline-based excellence is located," he wrote: "And The Australian chooses to average the scores over all fields of research. You absolute cretins..."

Disagreement also arose over which universities topped the latest league table.

The University of Melbourne was quick off the mark with a self-congratulatory release declaring it had established itself "as the leading research university in Australia, topping the key indicators in the Excellence in Research for Australia report [which] shows Melbourne had the highest number of research disciplines ranked at the maximum possible - well above world standard," the release gushed.

"Of just over 100 research areas assessed by the ARC over a six-year period, 42 at Melbourne had the highest rating. Another 40 were rated above world standard and 20 at world standard; overall, 88 were above the national average."

The Australian National University, which is funded separately by the federal government and has the biggest research base, had the next highest proportion of research above world standard at 79% followed by the University of Queensland, the University of New South Wales and the University of Sydney.

In its analysis, however, The Australian had the Australian National University at the top "with almost 70% of subjects rated at the highest possible level", followed by Melbourne and then Queensland, New South Wales and Sydney.

Whatever one's view of these differences in who outranks whom, Sheil was right to note that the ERA report drew together rich information about discipline-specific research activity at each institution, "as well as information about each discipline's contribution to the national landscape".

And, as she also said, it was a huge exercise and took into account the work of 55,000 individuals, collecting data on 333,000 publications and research outputs across 157 disciplines. In all, 2,435 areas in 40 institutions were assessed by committees comprised of distinguished Australian and international researchers, "that is, those who know the field interpreted the data".

The committees had access to detailed metrics and a range of other indicators, including results of more detailed peer review of individual works held in online repositories.

Sheil also pointed to the fact that Australia has lagged behind its international counterparts in implementing a research evaluation system.

South Africa has been running a ruler over its researchers for more than 20 years, the British first introduced theirs in 1986 and even New Zealand began one in 2003.

"Because of a long gestation, we have been able to use an Australian Bureau of Statistics classification system designed for Australasia and learn from problems elsewhere, consulting the best available expertise to assist in the design of the initiative, as well as using the latest advances in information tools and technology," she said.

As critics of these sorts of exercises have long pointed out, however, the winners nearly always come from older, high-status universities which deliberately attract top scholars who, in turn, win competitive research grants, especially in medicine and the sciences.

The Group of Eight research-intensive universities seemed to prove the truth of the old saying that "to him who has, shall be given" by claiming the top eight rankings - at least according to The Australian's calculations.

Sheil admits there is a strong correlation between 'excellence' and areas that have won competitive research funding. As she says, the traditional strength of medical science is not surprising given that medical researchers have a separate funding council, a history of strong leadership and many successes, including most of Australia's Nobel laureates.

Debate over the value of such large-scale exercises and who really beat whom in these sorts of status stakes will continue. But the results of this assessment seem certain to have political and funding implications for all universities.

Releasing the report, federal Research Minister Senator Kim Carr said: "The ERA national report reveals for the first time exactly how our country's research efforts compare to the rest of the world. While we celebrate our successes, we must also acknowledge that we have areas where we could do better, and the government will use the report to identify ways to improve."

geoff.maslen@uw-news.com