Table of Contents
Comments
Commenters1
[87.] Assessment will be undertaken by expert panels covering coherent research disciplines. Panels will include leading experts in their fields as well as people with experience in commissioning and using research – in businesses, the public and third sectors. Panel members will be appointed by the UK funding bodies after consulting widely with national associations, learned societies and other research bodies, research funders, and a range of bodies with a research user interest.
[87.] Assessment will be undertaken by expert panels covering coherent research disciplines. Panels will include leading experts in their fields as well as people with experience in commissioning and using research – in businesses, the public and third sectors. Panel members will be appointed by the UK funding bodies after consulting widely with national associations, learned societies and other research bodies, research funders, and a range of bodies with a research user interest. 3
[88.] We propose to retain a two-tiered structure in which groups of sub-panels work together under the guidance of main panels, to ensure consistency both in terms of process and standards of assessment. This will however incorporate significant new elements in the structure and management of the panels and in their working methods.
[88.] We propose to retain a two-tiered structure in which groups of sub-panels work together under the guidance of main panels, to ensure consistency both in terms of process and standards of assessment. This will however incorporate significant new elements in the structure and management of the panels and in their working methods. 4
1
[89.] In previous assessment exercises there were a number of comparatively fluid boundaries between UOAs. This provided institutions with substantial discretion about where to submit and how to divide up their work into discrete submissions. The panel structure also resulted in uneven panel workloads, with the number of FTE staff submitted to each UOA ranging from less than 100 to more than 3,000. For the REF we propose to have substantially fewer UOAs with fewer fluid boundaries between them than in previous assessment exercises; and to reduce the number of main panels – to as few as four – to achieve greater consistency.
1
[89.] In previous assessment exercises there were a number of comparatively fluid boundaries between UOAs. This provided institutions with substantial discretion about where to submit and how to divide up their work into discrete submissions. The panel structure also resulted in uneven panel workloads, with the number of FTE staff submitted to each UOA ranging from less than 100 to more than 3,000. For the REF we propose to have substantially fewer UOAs with fewer fluid boundaries between them than in previous assessment exercises; and to reduce the number of main panels – to as few as four – to achieve greater consistency. 7
- preparing statements of relevant criteria and working methods – but with limited scope to vary these (only varying where justified by specific disciplinary differences)
- preparing statements of relevant criteria and working methods – but with limited scope to vary these (only varying where justified by specific disciplinary differences) 15
[93.] A REF steering group, to be established by the UK funding bodies, will have oversight of the conduct of the exercise; will issue clear guidance to the panels on key elements in the assessment process, with particular reference to elements that we wish to be common to all panels; and will sign off the criteria and working methods of the panels.
[93.] A REF steering group, to be established by the UK funding bodies, will have oversight of the conduct of the exercise; will issue clear guidance to the panels on key elements in the assessment process, with particular reference to elements that we wish to be common to all panels; and will sign off the criteria and working methods of the panels.16
[94.] The proposed panel structure results in some UOAs covering a number of related though discrete fields; and a number of sub-panels that would be expected to receive large volumes of material for assessment. Nevertheless we believe that the proposed breadth and scale of coverage of the proposed UOAs are manageable. UOAs with a similar scale and breadth of material were dealt with by a number of the 2008 RAE sub-panels. Some RAE sub-panels effectively covered diverse fields of research, by ensuring an appropriate mix of expertise and to some extent working in informal ‘sub-groups’. Some sub-panels managed very large volumes of material, and have provided feedback about how this could be done more effectively in future.
[94.] The proposed panel structure results in some UOAs covering a number of related though discrete fields; and a number of sub-panels that would be expected to receive large volumes of material for assessment. Nevertheless we believe that the proposed breadth and scale of coverage of the proposed UOAs are manageable. UOAs with a similar scale and breadth of material were dealt with by a number of the 2008 RAE sub-panels. Some RAE sub-panels effectively covered diverse fields of research, by ensuring an appropriate mix of expertise and to some extent working in informal ‘sub-groups’. Some sub-panels managed very large volumes of material, and have provided feedback about how this could be done more effectively in future.17
[95.] We propose that the sub-panels, particularly those covering relatively large and diverse UOAs, will operate by:
[95.] We propose that the sub-panels, particularly those covering relatively large and diverse UOAs, will operate by:18
- varying or increasing the number of members to reflect the scale and breadth of material to be assessed
- varying or increasing the number of members to reflect the scale and breadth of material to be assessed19
- appointing ‘associate members’ to provide additional expertise in reviewing outputs and to ensure full user involvement in assessing impact
- appointing ‘associate members’ to provide additional expertise in reviewing outputs and to ensure full user involvement in assessing impact 21
- reducing the workload of assessing outputs, through the options discussed at paragraph 50
using specialist advisors to advise on the assessment of particular outputs for which members and associate members do not have sufficient expertise (although we propose more limited use of specialist advisors than in the 2008 RAE).
- reducing the workload of assessing outputs, through the options discussed at paragraph 50using specialist advisors to advise on the assessment of particular outputs for which members and associate members do not have sufficient expertise (although we propose more limited use of specialist advisors than in the 2008 RAE).
22
[96.] By way of illustration, a typical REF sub-panel could receive submissions for around 2,000 staff (a maximum of 6,000-8,000 outputs and 200-400 impact case studies), and could operate as follows:
[96.] By way of illustration, a typical REF sub-panel could receive submissions for around 2,000 staff (a maximum of 6,000-8,000 outputs and 200-400 impact case studies), and could operate as follows: 23
1
[a.] Around 20 panel members would be involved in reviewing all aspects of submissions (this would include any user members who are willing to review outputs).
1
[a.] Around 20 panel members would be involved in reviewing all aspects of submissions (this would include any user members who are willing to review outputs). 24
[b.] These members could be assisted by around 15-20 ‘associate members’: around half of these would be additional practising researchers to assist with reviewing the breadth and volume of outputs; the other half would be additional user members to focus on assessing impact.
[b.] These members could be assisted by around 15-20 ‘associate members’: around half of these would be additional practising researchers to assist with reviewing the breadth and volume of outputs; the other half would be additional user members to focus on assessing impact.25
[c.] Although there would be more academic members on panels overall (this is necessary to cope with the volume and breadth of outputs), the impact submissions would be reviewed mainly by the user members.
[c.] Although there would be more academic members on panels overall (this is necessary to cope with the volume and breadth of outputs), the impact submissions would be reviewed mainly by the user members. 26
[d.] The members could organise themselves into informal sub-groups if appropriate, to assess particular fields or types of research. The panel could provide guidance to institutions about the sub-groups and request that institutions indicate which sub-groups are relevant to their submitted research.
[d.] The members could organise themselves into informal sub-groups if appropriate, to assess particular fields or types of research. The panel could provide guidance to institutions about the sub-groups and request that institutions indicate which sub-groups are relevant to their submitted research. 29
- the assessment of work in discipline fields, especially in particular UOAs where there may be potential alternative options to consider. (A number of these are highlighted in Annex E.)
- the assessment of work in discipline fields, especially in particular UOAs where there may be potential alternative options to consider. (A number of these are highlighted in Annex E.) 30
[98.] In commenting on specific UOAs, respondents should note that we are committed to substantially reducing the number of UOAs (and the number and fluidity of boundaries between these) and to ensuring that all REF panels are of substantial size, even if this means combining relatively discrete fields into a single panel (which could operate with informal sub-groups). We do not wish to consider arguments for retaining comparatively small discrete UOAs.
[98.] In commenting on specific UOAs, respondents should note that we are committed to substantially reducing the number of UOAs (and the number and fluidity of boundaries between these) and to ensuring that all REF panels are of substantial size, even if this means combining relatively discrete fields into a single panel (which could operate with informal sub-groups). We do not wish to consider arguments for retaining comparatively small discrete UOAs.
I appreciate the demands on panels, but the REF must be the only area where people are proposing having less fluid boundaries is the best way forward. This is almost entirely at odds with both the way research is progressing in a digital age, and the way we would wish to develop it.
Properly disinterested academics should be invited to assess submissions, by which I mean academics (European, US, Asian) not involved in any way in any REF submission. This is particularly important for the smaller panels, but of relevance to all panels.