UK: Consultation begins on research assessment

Britain's funding councils are planning to change the way they decide how much money out of a yearly allocation of £1.76 billion (US$2.86 billion) universities get for their research budgets. The government flagged changes to the Research Assessment Exercise late in 2007, announcing that the 2008 RAE would be the last of its kind. Consultations have now started to formulate the Research Excellence Framework to replace it.

The Higher Education Funding Council for England announced that through the REF it "aims to develop and sustain a dynamic and internationally competitive research sector that makes a major contribution to economic prosperity, national wellbeing and the expansion and dissemination of knowledge".

The new system will be less cumbersome than the old one with fewer expert panels and units of assessment. Professor Eric Thomas, Vice-chancellor of Bristol University who chairs the research policy committee of Universities UK, described the old system as over-burdensome, outdated and outside of the research cycle as funding was based on what was happening years ago.

The council says the new framework will focus on assessing three elements. First, outputs which will include the quality of research publications; second, impact - benefits to the economy, society and culture; and third, the environment - how far the research environment supports continuity of excellence and effective dissemination and application.

These elements will be assessed and rated by expert panels on a five-point scale ranging from 4* 'exceptional' to 'unclassified'. Units of assessment in coherent discipline groups will be reduced from 67 to 30 and the main expert panels will be concentrated into four: medicine and life sciences, physical sciences, social sciences and arts and humanities.

Although more emphasis will be placed on citations, or biometrics - the number of references to research in journals - the council was more cautious about this approach than in the early stages of the changes. Hefce then envisaged that assessments would be made mainly on metrics for the sciences and medicine and a lighter touch peer review for the arts and humanities.

The council explains: "We conducted a substantive pilot exercise to test how to use citation information in the REF. We concluded that citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality, but there is considerable scope for it to inform and enhance the process of expert review.

"We propose that those UOAs for which robust data is available will make use of citation information. Sub-panels will decide this in advance. We expect that medicine, science and engineering panels will do so, but that the arts, humanities and a number of other panels will not."

Dr Wendy Piatt, Director General of the Russell Group of 20 leading research-intensive universities, welcomed the proposals. "We particularly welcome changes which keep peer review at the heart of the assessment, informed by metrics where appropriate," Piatt said.

But the Higher Education Policy Institute highlighted another major change: the assessment of "impact". In a discussion paper, Dr Bahram Bekhradnia said this was now an important separate and explicit element - impact had not been mentioned in the earlier proposals. This entirely new element will have major bearing on the overall assessment of a submission's quality, Bekhradnia said.

"Panels are now required to assess 'impact' separately, and the funding bodies' initial proposal is that 25% of the final score that will be used for determining funding will be based on the impact of research on the economic and social environment. Impact is here defined explicitly to exclude academic impact," he said.

"Nor is there any acknowledgement that the impact criterion may be more appropriate in some disciplines than others. Indeed, there is an explicit statement that it is expected that in all disciplines the impact factor will count for a similar amount."

Bekhradnia said the proposal needed to be handled extremely carefully. "While it is understandable that those in government who provide funding for research wish to see some economic and societal benefit, it seems unduly limited not to be able to value outstanding research that, for example, may change the face of a particular, but perhaps narrow, aspect of an academic discipline as highly as other research whose academic impact may be less, but whose societal impact is greater.

"Despite the fundamental continuity with previous exercises, the extent of the change that the introduction of `impact' introduces is significant, and greater than any changes made previously."

The funding councils plan to complete the first REF in 2013 to inform funding from 2014. They are asking for responses to their proposals by mid-December.

diane.spencer@uw-news.com