Sir Peter Gluckman ONZ FRS
Chair, FRAP International Advisory Panel
IG Government conference | Evaluating REF 2021: Looking Towards The Future Research Assessment Programme
May 24 2022

I am afraid I am going to disappoint you, because I am not going to say very much or anything that is very new; and those of you who read David Sweeney’s blog in Research Professional will see a high level of alignment.

Firstly, when the International Advisory panel was established last year, it was important that we did not in any way be seen to influence the just completed round so our discussions have focused on principles rather than the detail. Yet, inevitably academics and managers want to know the detail.

Secondly, we have not met since the REF 2021 results were released, nor have we had time to digest and reflect on them and the associated reports and the consultations that have been underway.

Thus it would be premature for me to say much more than identify some of the challenges which you all know so well.

As a final disclaimer you are hearing my personal comments, not those of the panel which has not met since the results have been published.

It is always important that form follows function and thus we need to start by thinking about the overall concept before the specific processes, which is where so much of the discussion tends to focus.

The original concept of the REF, or at least its predecessors, was fairly clear to provide an accountable and transparent system of providing QR funds to Universities. That remains its core function, and I would suggest individual academics who criticize the REF reflect on how important QR funding is in the context of sustaining the UK’s university research system. It is critical glue money that provides strategic capacity in the system in multiple ways, and it is perhaps more important than ever given the consequences of Brexit to research funding.

From the early policy work, three purposes were clearly assigned to the REF or its predecessors:

1. Accountability to the taxpayer
2. A means to allocate strategic funds to support research to each higher education institute (HEI)
3. Reputational benchmarking – although that is also achieved through other ranking mechanisms

Nick Stern’s committee effectively identified another three purposes:

4. To provide evidence base on the research profile for prioritization by UKRI, etc
5. To provide performance incentives for universities and researchers, although one might hope institutions and academics have enough other incentives to be self-reflexive and enhance their performance. But the magnitude of the internal effort by institutions suggests otherwise, or at least reflects the growing managerialism that has infected universities worldwide.
6. And to assist internal choices made by a University – this is inherent in how QR funds can be used.

The Stern recommendations added greater emphasis that certainly overlaps with the above concept of accountability – that is, to have the institutions demonstrate the value of their work to the taxpayer via demonstrating impact – a matter I shall return to.

But a further inference identified in subsequent commentary has led to a potential additional and explicit purpose, namely culture change within institutions. A start in that direction was met by the focus on interdisciplinarity in the last round and indeed in the growing emphasis on impact. But critically such an objective extends to the environmental assessment and to matters such as EDI and the nature of the research community, diversity, the role of early career researchers, etc.

And there is one other implied but critical aspect: on one hand the outcome must be judged by the Crown to meet the first 3 objectives and the system as a whole must see the entire exercise to be judged as ‘fair’, yet on the other hand the latter will always be subject to debate across stakeholder groups.

All funding systems have an implied or explicit incentive dimension. In some countries, research excellence and funding have been decoupled, but to completely decouple them takes away the advantage that the QR funding brings. The UK has been a research superpower and in a post-Brexit world, tools such as REF that help sustain that position are of overwhelming economic, social and geopolitical importance.

There is a challenge in that the 7 or 8 implied or explicit purposes of the REF are not necessarily aligned and that ambiguity may not be helpful. How those purposes are explicitly ranked must inform any reshaping of the incentive scheme. Incentivising culture change is not the same as rewarding a highly impactful research unit for its outputs.

And that leads to a big question. The REF as it stands now is largely summative – reflecting past performance. Is that the most desirable or should it be more proactive in incentivising future performance, that is, focused more on inputs than outputs? I shall return to this point later.

I imagine that, in any reconsideration of the REF, this matter will for relative purpose be core to its top-level design.

Caught up in this is the contestation between what the Crown wants from the REF and what the institutions do with it for their own purposes. And here there is, in some commentators’ minds, a quagmire. How much of the cost is due to institutional uses and needs, and how much process is required to have a trusted system to distribute funds? Large within-institution factories have been created around the REF, but for purposes such as internal management that are well outside the Crown’s direct needs. Yet confusion on this matter underlies some critiques of the REF.

Form must follow function, and so discussion of the processes and mode of evaluation require some relative clarity on these matters.

One of the problems is that of language. ‘Excellence’ in research cannot be easily defined and will vary depending on the person defining it. Excellence has many dimensions: the nature of the question, how the work was done, how the team that did the work was composed, how the results were communicated, where they were communicated (which has undue influence), and what the work led to. Yet, metrics and the academic preference to trying to make a subjective  judgment objective leads to a narrow focus. The lower the unit of assessment, the more distorting that narrow focus becomes. In that sense, the move away from selected staff to universal portfolio submission following Stern’s report was critically valuable. What might seem like excellent research to one person may not be at a different time or under a different circumstance, and vice versa. Hence, averaging by integration becomes logical. As David Sweeney points out, even the word ‘excellence may be misleading and biasing.

Impact is an equally problematic but critically important concept. It is too early to know what the shape of the impact portfolio was in the 2021 round, but the past tendency to submit what some called exceptional impact cases at the expense of normal impact cases is a concern. Exceptional impact is what happens when one research finding leads to something happening, like the discovery of a new drug. But much impact occurs over many years in a less dramatic way. Some examples of critical forms of impact might be a HEI assisting local communities in environmental restoration, or providing government confidentially with a progressive understanding of the origins of extremism, or a novelist providing a narrative that changes public understandings. Often these more normal types of impact are the most critical and valuable, yet appear harder for institutions to focus on. This in turn relates to the ongoing need to better define and know how to assess transdisciplinary and stakeholder engaged research. To what extent are these the types of impact that the University as a whole rather than any lower unit of assessment should be accountable for? Is this a form of cultural change the REF should promote?

The unit of assessment is an important matter to consider. The REF is currently very output focused; is that the right balance between input and output, decision maker and performer? To quote from a recent email from a committee member to to me:

The institutions benefit from the REF funding and have the power to make change but it is the performance (excellence & impact) of the researchers that is assessed.

Does this mean that we should be more focused on assessing institutional efforts that will create incentives for institutional change and enhanced research quality and impact by internal decisions that benefit the researchers and their outputs in a desired manner? If so, how?

I cannot answer any of these questions, and you should not read any conclusion into my comments, but these questions are appropriate at this stage of my panel’s contributions in a review that can only now explore the results of the last round and the various consultations underway.

I am pleased that James Wilsdon and colleagues are undertaking a mini-review of whether we can use metrics in any way to further assist. The issue remains that different domains of scholarship have very different kinds of output, and that is a challenge of the REF process.

So, finally, to the REF process itself. No one wants excessive bureaucracy, and yet everyone wants fairness and objectivity. Given what we are dealing with, these aren’t easy to reconcile. As i have said earlier, institutions and academics can be complicit in generating this complexity. As we all know from the (likely apocryphal) quote from Einstein, Not everything that is important can be measured, and not everything that can be measured is important.

In closing, let me acknowledge the other members of the FRAP independent advisory panel; our global membership creates some challenging time zones for our Zoom calls, and I particularly acknowledge the officials at Research England and in the other three jurisdictions. I have worked with many officials in many countries over the years, and their grasp of the issues is absolutely exemplary.

So, the question remains: is it valuable to the UK to have an incentive system to reward impactful and quality research? Does it give confidence to the Crown to sustain funding in a way that can censure the continued importance of UK research of every type? In my view, the short answer is yes, and the long answer is a lot more reflection is needed to know what is the best and most acceptable way to do it.