Table of Contents
Comments
Commenters1
[99.] We intend to achieve greater consistency in the standards of assessment between panels than in the RAE, with fewer elements in the assessment process left open to panels’ discretion. We propose to achieve this through:
[99.] We intend to achieve greater consistency in the standards of assessment between panels than in the RAE, with fewer elements in the assessment process left open to panels’ discretion. We propose to achieve this through:2
[a.] Determining key elements of the criteria and assessment processes – and producing more detailed guidance – centrally; and by allowing flexibility for panels to supplement the guidance or vary the process only in specific defined areas and where differences in research in the discipline can be shown to justify this.
[a.] Determining key elements of the criteria and assessment processes – and producing more detailed guidance – centrally; and by allowing flexibility for panels to supplement the guidance or vary the process only in specific defined areas and where differences in research in the discipline can be shown to justify this. 3
[b.] Having fewer broader main panels, with a clear responsibility to ensure consistency in the process and comparability of outcomes among their sub-panels. Feedback from the 2008 RAE indicates that the main panels had a significant influence in achieving consistency across their fields, and we see scope to build further on this.
[b.] Having fewer broader main panels, with a clear responsibility to ensure consistency in the process and comparability of outcomes among their sub-panels. Feedback from the 2008 RAE indicates that the main panels had a significant influence in achieving consistency across their fields, and we see scope to build further on this. 4
[c.] Providing panels with information at an early stage in their work to support the consistent application of internationally benchmarked standards of assessment. This will build on the type of data used by the Government to monitor its public service agreement targets for the international performance of the UK research base11 and any other information of sufficient authority such as discipline reviews by international experts commissioned by the Research Councils. This will include bibliometric indicators where available, comparing the citation performance of disciplines in the UK as a whole with other key competitor countries.
11See ‘International comparative performance of the UK research base’ (July 2008, PDF) available on the Department for Business, Innovation and Skills web-site.
[c.] Providing panels with information at an early stage in their work to support the consistent application of internationally benchmarked standards of assessment. This will build on the type of data used by the Government to monitor its public service agreement targets for the international performance of the UK research base11 and any other information of sufficient authority such as discipline reviews by international experts commissioned by the Research Councils. This will include bibliometric indicators where available, comparing the citation performance of disciplines in the UK as a whole with other key competitor countries.11See ‘International comparative performance of the UK research base’ (July 2008, PDF) available on the Department for Business, Innovation and Skills web-site.
6
[100.] We propose that the following aspects of the assessment be standardised across all UOAs, with generic guidance provided centrally:
[100.] We propose that the following aspects of the assessment be standardised across all UOAs, with generic guidance provided centrally:7
[a.] The assessment of the three elements (outputs, impact and environment) against generic criteria and definitions of four starred quality levels; the same weightings between the elements; and a consistent approach to forming sub-profiles and combining them into an overall profile.
[a.] The assessment of the three elements (outputs, impact and environment) against generic criteria and definitions of four starred quality levels; the same weightings between the elements; and a consistent approach to forming sub-profiles and combining them into an overall profile. 8
[b.] Definitions of staff eligibility and output types; and the criteria for and treatment of individual staff circumstances. This is discussed further at paragraph 112.
[b.] Definitions of staff eligibility and output types; and the criteria for and treatment of individual staff circumstances. This is discussed further at paragraph 112. 9
[c.] The role of sub-panels and main panels, processes for nomination and appointment, the role of user members, and processes for cross-referring material and using special advisors, including for handling interdisciplinary research.
[c.] The role of sub-panels and main panels, processes for nomination and appointment, the role of user members, and processes for cross-referring material and using special advisors, including for handling interdisciplinary research.10
[d.] The format and core data for submissions, including the maximum number of outputs submitted per researcher; generic templates for submitting evidence about impact and environment; a common menu of indicators; and standardised reporting of PGR student and research income data.
[d.] The format and core data for submissions, including the maximum number of outputs submitted per researcher; generic templates for submitting evidence about impact and environment; a common menu of indicators; and standardised reporting of PGR student and research income data. 12
[f.] Provision of citation data in a standard format to all sub-panels that make use of citation data.
[f.] Provision of citation data in a standard format to all sub-panels that make use of citation data. 13
[101.] There will be discretion given to panels only where they have identified significant features of research activity in their disciplinary area that require some flexibility in the assessment. We propose that these are:
[101.] There will be discretion given to panels only where they have identified significant features of research activity in their disciplinary area that require some flexibility in the assessment. We propose that these are:14
[a.] The panel’s working methods, to manage variable workloads (this varies according to both the volume and type of outputs – for example, monographs are more time-consuming to assess than journal articles). Panels could, for example, sample outputs, organise themselves into informal sub-groups, make greater use of ‘associate members’, or decide that certain types of output would be double-weighted, as appropriate to the volume and diversity of material to be assessed.
[a.] The panel’s working methods, to manage variable workloads (this varies according to both the volume and type of outputs – for example, monographs are more time-consuming to assess than journal articles). Panels could, for example, sample outputs, organise themselves into informal sub-groups, make greater use of ‘associate members’, or decide that certain types of output would be double-weighted, as appropriate to the volume and diversity of material to be assessed. 15
[b.] Whether the panel will make use of citation information to inform the assessment of outputs; and if so, how they will make use of such data (although for all panels making use of citation data, the data will be provided in a standard format, and there will be guidelines on their use).
[b.] Whether the panel will make use of citation information to inform the assessment of outputs; and if so, how they will make use of such data (although for all panels making use of citation data, the data will be provided in a standard format, and there will be guidelines on their use).16
[c.] Guidance on what information should be included in the short statements that may be provided with outputs, to demonstrate the output’s significance to users or, for example, explain the individual’s contribution to a co-authored output.
[c.] Guidance on what information should be included in the short statements that may be provided with outputs, to demonstrate the output’s significance to users or, for example, explain the individual’s contribution to a co-authored output.17
[d.] Preferred indicators of impact. Panels could identify particular indicators from the common menu that they would want to include in the impact statements or emphasise within case studies, or provide additional guidance on how some specific indicators should be presented, as may be appropriate to their disciplines.
[d.] Preferred indicators of impact. Panels could identify particular indicators from the common menu that they would want to include in the impact statements or emphasise within case studies, or provide additional guidance on how some specific indicators should be presented, as may be appropriate to their disciplines. 18
[e.] Assessment of environment: although all UOAs would use a standard template for the assessment of environment, panels could determine the extent to which critical mass would be taken into account, place more or less emphasis on particular elements in the template, or provide guidance on specific indicators to be included within it (in addition to the standard reporting of PGR and research income data).
[e.] Assessment of environment: although all UOAs would use a standard template for the assessment of environment, panels could determine the extent to which critical mass would be taken into account, place more or less emphasis on particular elements in the template, or provide guidance on specific indicators to be included within it (in addition to the standard reporting of PGR and research income data). 19
Consultation question 7
Do you agree with the proposed approach to ensuring consistency between panels?
21
[102.] We propose to follow similar processes for appointing panel members as in previous assessment exercises. We will identify a list of nominating bodies, to put forward nominations for chairs, members and associate members. A draft list of nominating bodies (Annex F) is available on the HEFCE web-site alongside this document under Publications. We invite suggestions for additions or amendments to this list – particularly for additional bodies that represent users of research or that could nominate research users as potential panellists.
[102.] We propose to follow similar processes for appointing panel members as in previous assessment exercises. We will identify a list of nominating bodies, to put forward nominations for chairs, members and associate members. A draft list of nominating bodies (Annex F) is available on the HEFCE web-site alongside this document under Publications. We invite suggestions for additions or amendments to this list – particularly for additional bodies that represent users of research or that could nominate research users as potential panellists. 22
[103.] Main panel and sub-panel chairs will be appointed by the funding bodies through a robust and transparent process, with clear criteria for appointment reflecting the demands of the role.
[103.] Main panel and sub-panel chairs will be appointed by the funding bodies through a robust and transparent process, with clear criteria for appointment reflecting the demands of the role. 23
[104.] Following this consultation exercise, we will publish the finalised panel configuration and details of the panel recruitment process, and seek nominations for panellists.
[104.] Following this consultation exercise, we will publish the finalised panel configuration and details of the panel recruitment process, and seek nominations for panellists. 24
Consultation question 8
Do you have any suggested additions or amendments to the list of nominating bodies? (If suggesting additional bodies, please provide their names and addresses and indicate how they are qualified to make nominations.)
26
[105.] In general, members of 2008 RAE panels felt they were able to assess interdisciplinary research without difficulty, given the breadth of panel membership and use of specialist advisors, although some noted practical difficulties with the cross-referral process. We are looking at how to build on the RAE approach to ensure the REF encourages interdisciplinary research.
[105.] In general, members of 2008 RAE panels felt they were able to assess interdisciplinary research without difficulty, given the breadth of panel membership and use of specialist advisors, although some noted practical difficulties with the cross-referral process. We are looking at how to build on the RAE approach to ensure the REF encourages interdisciplinary research. 27
1
[106.] In revising the panel structure, we aim to ensure that whichever panel interdisciplinary research is submitted to, there will be effective mechanisms for ensuring it is reviewed fairly by people with appropriate expertise. The proposed broader sub-panels and main panels will assist with this as will our proposals for expanding the breadth of expertise within panels, by the addition of associate members. We also propose to retain specialist advisors where necessary, although we anticipate more limited use of such advisors than in the previous assessment exercise, given the proposed changes to the panel structure.
1
[106.] In revising the panel structure, we aim to ensure that whichever panel interdisciplinary research is submitted to, there will be effective mechanisms for ensuring it is reviewed fairly by people with appropriate expertise. The proposed broader sub-panels and main panels will assist with this as will our proposals for expanding the breadth of expertise within panels, by the addition of associate members. We also propose to retain specialist advisors where necessary, although we anticipate more limited use of such advisors than in the previous assessment exercise, given the proposed changes to the panel structure. 28
[107.] Within the environment element we will ask submitted units to explain how they support interdisciplinary and collaborative research. Within the impact element they will be given credit for impact achieved where interdisciplinary research contributes to solving real-world problems.
[107.] Within the environment element we will ask submitted units to explain how they support interdisciplinary and collaborative research. Within the impact element they will be given credit for impact achieved where interdisciplinary research contributes to solving real-world problems. 29
[108.] We anticipate that this approach to interdisciplinary research will help to dispel any perceptions that the assessment process acts to discourage activity of this kind, including high-priority areas such as research linked to sustainable development and a low carbon economy.
[108.] We anticipate that this approach to interdisciplinary research will help to dispel any perceptions that the assessment process acts to discourage activity of this kind, including high-priority areas such as research linked to sustainable development and a low carbon economy.30
Consultation question 9
Do you agree that our proposed approach will ensure that interdisciplinary research is assessed on an equal footing with other types of research? Are there further measures we should consider to ensure that this is the case and that our approach is well understood?
32
[109.] Significant benefits can flow from the movement of researchers between HE and other sectors, including business and industry and the public and voluntary sectors. This movement may take several forms, including the secondment of academic researchers to work in other sectors for a significant period and the recruitment in HE of people with significant experience in research while working elsewhere, as well as less formal arrangements for people to spend time working outside their normal research environment. Its benefits can include the achievement of more frequent and more significant outcomes, harnessing research findings to real world problems, and the development of a culture within HE that values and supports interactions of all kinds. Encouraging researcher mobility is a specific policy aim of the REF and we wish to ensure that the framework creates a positive environment for researcher mobility.
[109.] Significant benefits can flow from the movement of researchers between HE and other sectors, including business and industry and the public and voluntary sectors. This movement may take several forms, including the secondment of academic researchers to work in other sectors for a significant period and the recruitment in HE of people with significant experience in research while working elsewhere, as well as less formal arrangements for people to spend time working outside their normal research environment. Its benefits can include the achievement of more frequent and more significant outcomes, harnessing research findings to real world problems, and the development of a culture within HE that values and supports interactions of all kinds. Encouraging researcher mobility is a specific policy aim of the REF and we wish to ensure that the framework creates a positive environment for researcher mobility. 33
[110.] We propose to ensure that the REF encourages and supports researcher mobility, chiefly by securing that:
[110.] We propose to ensure that the REF encourages and supports researcher mobility, chiefly by securing that:34
[a.] Research undertaken outside HE during the assessment period by staff whose outputs are submitted for assessment is considered on an equal footing and against appropriate criteria for excellence in all cases where an assessable output can be identified, including where this may not be published in a form generally adopted by academic researchers in its field.
[a.] Research undertaken outside HE during the assessment period by staff whose outputs are submitted for assessment is considered on an equal footing and against appropriate criteria for excellence in all cases where an assessable output can be identified, including where this may not be published in a form generally adopted by academic researchers in its field.35
[b.] Staff who had spent time in industry or other sectors undertaking research or other work that builds on their research expertise, will be able to submit fewer than the maximum number of outputs. We propose to make this a specific category of ‘individual staff circumstances’ to be taken into account.
[b.] Staff who had spent time in industry or other sectors undertaking research or other work that builds on their research expertise, will be able to submit fewer than the maximum number of outputs. We propose to make this a specific category of ‘individual staff circumstances’ to be taken into account. 36
[c.] In their submissions institutions are encouraged to identify cases where movement of staff between HE and other sectors has contributed to the impact of their research during the assessment period.
[c.] In their submissions institutions are encouraged to identify cases where movement of staff between HE and other sectors has contributed to the impact of their research during the assessment period. 37
[d.] In assessing research environment, credit is given where departments can show that they have a strong record in encouraging and supporting researcher mobility and interactions of all kinds with other sectors.
[d.] In assessing research environment, credit is given where departments can show that they have a strong record in encouraging and supporting researcher mobility and interactions of all kinds with other sectors. 38
Consultation question 10
Do you agree that our proposals for encouraging and supporting researcher mobility will have a positive effect; and are there other measures that should be taken within the REF to this end?
Tags: advisors, Department for Business, Feedback, Innovation and Skills, researcher, special advisors, specialist advisors
My feeling thus far is that interdisciplinary work, which didn’t fare well under the RAE, is being rather legislated against in the REF also, see the comment on less fluid boundaries, the narrow focus on metrics, etc. So it’s good to see this aim, but I don’t think it’s borne out in the rest of the document.