Table of Contents
Comments
Commenters2
[28.] We have considered and discussed widely whether the work of all research staff at an institution should be assessed, or whether the REF should focus on the work of selected research staff. We perceive a broad consensus that the outputs of selected academic staff should be assessed within each submitted unit, and that the selection of staff for submission should be done by the employing HEI in all cases. Considering the work of selected staff reflects our underlying policy to identify research excellence. As noted in paragraph 27g, the exercise is not designed to assess all research activity irrespective of its quality, and it is not necessary (even if it were feasible in practice) to consider the work of staff who have not engaged in significant research activity of high quality. Requiring HEIs to select staff whose work will be assessed recognises the reality that only the institution can identify those with substantive research roles. It would be unworkable for the HE funding bodies to try to identify all active researchers through nationally available data, without an element of choice by the institution.
[28.] We have considered and discussed widely whether the work of all research staff at an institution should be assessed, or whether the REF should focus on the work of selected research staff. We perceive a broad consensus that the outputs of selected academic staff should be assessed within each submitted unit, and that the selection of staff for submission should be done by the employing HEI in all cases. Considering the work of selected staff reflects our underlying policy to identify research excellence. As noted in paragraph 27g, the exercise is not designed to assess all research activity irrespective of its quality, and it is not necessary (even if it were feasible in practice) to consider the work of staff who have not engaged in significant research activity of high quality. Requiring HEIs to select staff whose work will be assessed recognises the reality that only the institution can identify those with substantive research roles. It would be unworkable for the HE funding bodies to try to identify all active researchers through nationally available data, without an element of choice by the institution. 3
[29.] We recognise that comparisons between quality profiles based on the work of selected staff in different HEIs will inevitably be coloured by variations in how individual institutions determine which staff to include. However, we anticipate that institutions will wish to include all of their research staff working at high quality levels, and that the quality profiles will thus identify the actual volume of activity at high quality levels within each HEI. Moreover, both the quality ratings and, more recently, the quality profiles produced by previous research assessment exercises have assessed the work of selected staff; this is something that the funding bodies and other users of the RAE outcomes are accustomed to working with.
[29.] We recognise that comparisons between quality profiles based on the work of selected staff in different HEIs will inevitably be coloured by variations in how individual institutions determine which staff to include. However, we anticipate that institutions will wish to include all of their research staff working at high quality levels, and that the quality profiles will thus identify the actual volume of activity at high quality levels within each HEI. Moreover, both the quality ratings and, more recently, the quality profiles produced by previous research assessment exercises have assessed the work of selected staff; this is something that the funding bodies and other users of the RAE outcomes are accustomed to working with. 4
[30.] We have also considered how far the assessment should have regard to variations in the proportion of total academic staff selected by HEIs and submitted units. It remains our view that the proportion of staff selected should not be a significant factor in assessing quality in the REF. A strong research culture can be maintained within a department or unit even if a significant number of its academic staff are not actively engaged in high-quality research. In the event that an assessment panel may feel that the activity of a submitted unit lacks desirable critical mass, this can be taken into account through the assessment of the research environment. Moreover, collection of auditable statistics at the level of submitted units would pose considerable practical problems.
[30.] We have also considered how far the assessment should have regard to variations in the proportion of total academic staff selected by HEIs and submitted units. It remains our view that the proportion of staff selected should not be a significant factor in assessing quality in the REF. A strong research culture can be maintained within a department or unit even if a significant number of its academic staff are not actively engaged in high-quality research. In the event that an assessment panel may feel that the activity of a submitted unit lacks desirable critical mass, this can be taken into account through the assessment of the research environment. Moreover, collection of auditable statistics at the level of submitted units would pose considerable practical problems. 5
[31.] Nonetheless we recognise that some HEIs feel strongly that they should be able to demonstrate their research intensivity by reference to statistics for the proportion of academic staff selected across the institution. We plan for the Higher Education Statistics Agency (HESA) to collect data about all eligible staff for the purpose of equal opportunities monitoring, and that these data will also provide statistics about the proportion of staff selected for assessment. While such data are not expected to inform the assessment process, we expect HESA to publish such statistics after the completion of the REF assessment. We will clearly define eligible staff in order to avoid the problems experienced in this regard following the 2008 RAE.
[31.] Nonetheless we recognise that some HEIs feel strongly that they should be able to demonstrate their research intensivity by reference to statistics for the proportion of academic staff selected across the institution. We plan for the Higher Education Statistics Agency (HESA) to collect data about all eligible staff for the purpose of equal opportunities monitoring, and that these data will also provide statistics about the proportion of staff selected for assessment. While such data are not expected to inform the assessment process, we expect HESA to publish such statistics after the completion of the REF assessment. We will clearly define eligible staff in order to avoid the problems experienced in this regard following the 2008 RAE.6
[32.] We propose to simplify the categories of staff that are eligible for selection by institutions. In the 2008 RAE, institutions could submit the work of a range of individuals whose research was focused in the submitting unit even though they were not employed by the HEI (Category C staff). The work of such staff was taken into account in constructing the quality profiles, though not in the volume measure for funding purposes. Concerns have been expressed about the difficulty of determining whether the submitted unit was indeed the focus for the research activity of such staff, as required by the rules of the RAE, and how far they contributed to its achievements.
[32.] We propose to simplify the categories of staff that are eligible for selection by institutions. In the 2008 RAE, institutions could submit the work of a range of individuals whose research was focused in the submitting unit even though they were not employed by the HEI (Category C staff). The work of such staff was taken into account in constructing the quality profiles, though not in the volume measure for funding purposes. Concerns have been expressed about the difficulty of determining whether the submitted unit was indeed the focus for the research activity of such staff, as required by the rules of the RAE, and how far they contributed to its achievements.7
[33.] We propose that in order to reduce complexity in the REF, institutions can select and include the work of the following categories of staff:
[33.] We propose that in order to reduce complexity in the REF, institutions can select and include the work of the following categories of staff:8
[a.] Academic staff employed by the institution on the census date: these are academic staff employed under a contract of employment with the submitting HEI on the census date, whose primary employment function is research, teaching or both. (These staff would also comprise the volume measure for funding purposes.)
[a.] Academic staff employed by the institution on the census date: these are academic staff employed under a contract of employment with the submitting HEI on the census date, whose primary employment function is research, teaching or both. (These staff would also comprise the volume measure for funding purposes.) 9
[b.] Staff employed by another organisation on the census date, who undertake research that is demonstrably focused in the submitting unit, and they are employed to do so. This will permit the continuing inclusion of outputs by some NHS staff and others working in units embedded within the HEI and funded by third party bodies in partnership with the HEI. The research of these staff would contribute to the quality profiles, though not the volume measure.
[b.] Staff employed by another organisation on the census date, who undertake research that is demonstrably focused in the submitting unit, and they are employed to do so. This will permit the continuing inclusion of outputs by some NHS staff and others working in units embedded within the HEI and funded by third party bodies in partnership with the HEI. The research of these staff would contribute to the quality profiles, though not the volume measure.10
[34.] We are aware of some concern about the inclusion of staff on fractional contracts, whose professional research activity may take place mainly outside the HEI, in relation to the contribution that their submitted outputs may be judged to have made to the achievement of the submitted unit overall. We are considering whether there should be some means to establish a connection between the work of such staff and the submitted unit. Respondents to the consultation may wish to comment on this.
[34.] We are aware of some concern about the inclusion of staff on fractional contracts, whose professional research activity may take place mainly outside the HEI, in relation to the contribution that their submitted outputs may be judged to have made to the achievement of the submitted unit overall. We are considering whether there should be some means to establish a connection between the work of such staff and the submitted unit. Respondents to the consultation may wish to comment on this. 11
[35.] The assessment process will be designed to encourage institutions to submit any eligible staff who have produced research of high quality during the assessment period but who may not have published outputs spanning the entire period. Staff with particular circumstances that constrained their capacity to undertake research during the assessment period will be able to submit fewer than the maximum number of outputs, without reducing their contribution to the volume measure for funding purposes. These will include:
[35.] The assessment process will be designed to encourage institutions to submit any eligible staff who have produced research of high quality during the assessment period but who may not have published outputs spanning the entire period. Staff with particular circumstances that constrained their capacity to undertake research during the assessment period will be able to submit fewer than the maximum number of outputs, without reducing their contribution to the volume measure for funding purposes. These will include: 12
- ‘early career’ researchers who may only recently have established themselves as principal investigators undertaking work of high quality
- ‘early career’ researchers who may only recently have established themselves as principal investigators undertaking work of high quality 14
- researchers who had spent time away from their academic base for reasons including working in or with businesses or other sectors, or personal circumstances such as maternity leave or ill-health, and might not have published during that period
- researchers who had spent time away from their academic base for reasons including working in or with businesses or other sectors, or personal circumstances such as maternity leave or ill-health, and might not have published during that period15
- established researchers recruited into HE from businesses or other sectors where traditional academic outputs are not normally produced; this is to support greater movement of staff between research user organisations and academia.
- established researchers recruited into HE from businesses or other sectors where traditional academic outputs are not normally produced; this is to support greater movement of staff between research user organisations and academia.17
[36.] Institutions will select outputs to be submitted for assessment. We propose there be a maximum of either three or four outputs submitted for each researcher. We consider this number to be sufficient to enable panels to form a robust view as to the achievements of individuals and of submitted units while discouraging the submission of individual staff who do not have a sufficiently strong personal research record (subject to the safeguards discussed above). A reduction in the overall maximum from four to three could make a material contribution to reducing the burden on expert panels, especially in fields where citation indices are not well developed, and would in our view be justified if the assessment period were to be set at five years (the anticipated period between the 2008 RAE and a REF exercise in 2013).
[36.] Institutions will select outputs to be submitted for assessment. We propose there be a maximum of either three or four outputs submitted for each researcher. We consider this number to be sufficient to enable panels to form a robust view as to the achievements of individuals and of submitted units while discouraging the submission of individual staff who do not have a sufficiently strong personal research record (subject to the safeguards discussed above). A reduction in the overall maximum from four to three could make a material contribution to reducing the burden on expert panels, especially in fields where citation indices are not well developed, and would in our view be justified if the assessment period were to be set at five years (the anticipated period between the 2008 RAE and a REF exercise in 2013). 18
[37.] Panels may also wish to ‘double-weight’ certain types of output, as discussed at paragraph 50b.
[37.] Panels may also wish to ‘double-weight’ certain types of output, as discussed at paragraph 50b.19
1
[38.] All types of outputs from research that meets the Frascati principles (involving original investigation leading to new insights) will be eligible for submission. This includes ‘grey literature’ and outputs that are not in conventional published form, such as confidential reports to government or business, software, designs, performances and artefacts. Given that we see research as a process of investigation that has led to new insights effectively shared, we would expect all submitted work to include evidence of the research process, as well as presenting the insights in a form meeting the needs of its potential audience both within and beyond the academic community.
1
[38.] All types of outputs from research that meets the Frascati principles (involving original investigation leading to new insights) will be eligible for submission. This includes ‘grey literature’ and outputs that are not in conventional published form, such as confidential reports to government or business, software, designs, performances and artefacts. Given that we see research as a process of investigation that has led to new insights effectively shared, we would expect all submitted work to include evidence of the research process, as well as presenting the insights in a form meeting the needs of its potential audience both within and beyond the academic community. 21
[39.] Outputs will be assessed against criteria of ‘rigour, originality and significance’. By ‘significance’, we mean the extent to which research outputs display the capacity to make a difference either through intellectual influence within the academic sphere, or through actual or potential use beyond the academic sphere, or both. These three generic criteria will be applied to all types of research, and clarifying the meaning of ‘significance’ in this way should help to encourage institutions to submit high-quality applied and translational research in particular. Panels will also be guided to recognise that while research of the highest quality must meet all three criteria, individual outputs may do so to varying degrees.
[39.] Outputs will be assessed against criteria of ‘rigour, originality and significance’. By ‘significance’, we mean the extent to which research outputs display the capacity to make a difference either through intellectual influence within the academic sphere, or through actual or potential use beyond the academic sphere, or both. These three generic criteria will be applied to all types of research, and clarifying the meaning of ‘significance’ in this way should help to encourage institutions to submit high-quality applied and translational research in particular. Panels will also be guided to recognise that while research of the highest quality must meet all three criteria, individual outputs may do so to varying degrees. 22
[40.] To assist in assessing user significance (beyond the academic sphere), institutions will be invited to include a short statement with any output for which they believe that such significance may convincingly be asserted.
[40.] To assist in assessing user significance (beyond the academic sphere), institutions will be invited to include a short statement with any output for which they believe that such significance may convincingly be asserted. 23
[41.] The assessment of outputs will produce a graded sub-profile for each submission, to show the proportion of submitted work meeting defined standards of excellence. We propose to revise the descriptor for four star quality used in the 2008 RAE, to achieve the highest degree of discrimination at the top end of the scale, as set out below. It is our intention that research activity should be assessed at the four star level only where the assessment panel is confident in its view that the activity meets the very highest standards. We envisage that this will result in greater discrimination between research meeting the very highest standards and research which may be excellent but still falls short of these. (We recognise that, in combination with other changes to the framework, this will result in outcomes that are not fully comparable to those from the 2008 RAE.)
[41.] The assessment of outputs will produce a graded sub-profile for each submission, to show the proportion of submitted work meeting defined standards of excellence. We propose to revise the descriptor for four star quality used in the 2008 RAE, to achieve the highest degree of discrimination at the top end of the scale, as set out below. It is our intention that research activity should be assessed at the four star level only where the assessment panel is confident in its view that the activity meets the very highest standards. We envisage that this will result in greater discrimination between research meeting the very highest standards and research which may be excellent but still falls short of these. (We recognise that, in combination with other changes to the framework, this will result in outcomes that are not fully comparable to those from the 2008 RAE.)25
1
1
| Four star | Exceptional: Quality that is world-leading and meets the highest standards of excellence in terms of originality, significance and rigour |
| Three star | Excellent: Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence |
| Two star | Very good: Quality that is recognised internationally in terms of originality, significance and rigour |
| One star | Good: Quality that is recognised nationally in terms of originality, significance and rigour |
| Unclassified | Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of the assessment |
27
[42.] The expert panels will construct the outputs sub-profiles based on their review of the submitted outputs, informed and supplemented by citation information where appropriate. Panels will need to include a sufficient breadth of expertise for this purpose, and we anticipate that all panels, including those using citation information, will review a substantial proportion of the submitted outputs.
[42.] The expert panels will construct the outputs sub-profiles based on their review of the submitted outputs, informed and supplemented by citation information where appropriate. Panels will need to include a sufficient breadth of expertise for this purpose, and we anticipate that all panels, including those using citation information, will review a substantial proportion of the submitted outputs.28
[43.] In the RAE, many panels reviewed large volumes of outputs and the workload was considerable. Through the bibliometrics pilot exercise (see Annex B) we tested ways in which citation analysis could supplement or replace expert review of outputs. We have also considered other options for reducing panels’ workloads, especially where bibliometrics would not be available. These are discussed below.
[43.] In the RAE, many panels reviewed large volumes of outputs and the workload was considerable. Through the bibliometrics pilot exercise (see Annex B) we tested ways in which citation analysis could supplement or replace expert review of outputs. We have also considered other options for reducing panels’ workloads, especially where bibliometrics would not be available. These are discussed below.29
Use of citation information
[44.] Based on the bibliometrics pilot exercise and the advice of the EAGs and wider discussions, we have concluded that:
Use of citation information[44.] Based on the bibliometrics pilot exercise and the advice of the EAGs and wider discussions, we have concluded that:
30
[a.] Bibliometrics are not sufficiently mature to be used formulaically as a sole indicator or to replace expert review, but there is considerable scope for citation indicators to inform and supplement expert review of outputs in the REF, in certain UOAs.
[a.] Bibliometrics are not sufficiently mature to be used formulaically as a sole indicator or to replace expert review, but there is considerable scope for citation indicators to inform and supplement expert review of outputs in the REF, in certain UOAs. 31
[b.] There are several ways in which bibliometrics can be used to inform and supplement expert review to enhance the reliability of the process and to reduce panel workloads. This varies between UOAs, depending on the availability of citation data and the robustness of citation indicators.
[b.] There are several ways in which bibliometrics can be used to inform and supplement expert review to enhance the reliability of the process and to reduce panel workloads. This varies between UOAs, depending on the availability of citation data and the robustness of citation indicators. 32
1
[c.] For the purposes of REF, citation information will be most useful where it focuses on the selected outputs. There would be little additional value within the framework – and considerable additional work – in trying to produce citation indicators for all of an institution’s or submitted unit’s outputs.
1
[c.] For the purposes of REF, citation information will be most useful where it focuses on the selected outputs. There would be little additional value within the framework – and considerable additional work – in trying to produce citation indicators for all of an institution’s or submitted unit’s outputs. 34
[a.] Citation data relating to submitted outputs will be provided to panels to inform expert review in UOAs covering the medical, health, biological and physical sciences, psychology, engineering and computer science. For other UOAs, panels should decide whether or not they would use citation information, after consulting their communities. We do not expect that the arts, humanities or many social sciences would opt to use citation information, given the limitations of such data in these subjects4.
4 To inform these decisions, we will publish data for each discipline, about the coverage of the available citation databases and patterns of citation (in particular, the average number of times that outputs are cited over time).
[a.] Citation data relating to submitted outputs will be provided to panels to inform expert review in UOAs covering the medical, health, biological and physical sciences, psychology, engineering and computer science. For other UOAs, panels should decide whether or not they would use citation information, after consulting their communities. We do not expect that the arts, humanities or many social sciences would opt to use citation information, given the limitations of such data in these subjects4.4 To inform these decisions, we will publish data for each discipline, about the coverage of the available citation databases and patterns of citation (in particular, the average number of times that outputs are cited over time).
35
2
[b.] For those panels that use citation data, the data will be provided for all submitted outputs that can be matched to the relevant citation database(s). In the pilot exercise we used two databases, the Web of Science and Scopus. For the REF, we will procure one or more databases through a rigorous procurement process with clear criteria to ensure appropriate coverage, data quality, suitability for international benchmarking, transparency, responsiveness to our requirements and value for money.
2
[b.] For those panels that use citation data, the data will be provided for all submitted outputs that can be matched to the relevant citation database(s). In the pilot exercise we used two databases, the Web of Science and Scopus. For the REF, we will procure one or more databases through a rigorous procurement process with clear criteria to ensure appropriate coverage, data quality, suitability for international benchmarking, transparency, responsiveness to our requirements and value for money.36
[c.] Citation data will be provided to the relevant panels in a standard format, including clear international benchmarks and sufficient information to enable the data to be interpreted. Examples of the kind of information to be provided to panels are at Annex C.
[c.] Citation data will be provided to the relevant panels in a standard format, including clear international benchmarks and sufficient information to enable the data to be interpreted. Examples of the kind of information to be provided to panels are at Annex C. 37
[d.] The panels will have flexibility to decide how to make use of the data as appropriate to their discipline, depending on the robustness and coverage of the data. For example, they could use the information to inform judgements about the quality of individual outputs; or they could use citation information relating to whole submissions as a cross-check against sub-profiles arrived at through reviewing the outputs. Further discussion of the possible uses of the data is at Annex C.
[d.] The panels will have flexibility to decide how to make use of the data as appropriate to their discipline, depending on the robustness and coverage of the data. For example, they could use the information to inform judgements about the quality of individual outputs; or they could use citation information relating to whole submissions as a cross-check against sub-profiles arrived at through reviewing the outputs. Further discussion of the possible uses of the data is at Annex C. 38
[e.] We will provide guidance to the panels on how to ensure that whatever approach is taken is robust and free from bias. This will include guidance on:
[e.] We will provide guidance to the panels on how to ensure that whatever approach is taken is robust and free from bias. This will include guidance on:39
[i.] Ensuring that judgements about the quality of individual outputs are not made solely on the basis of citation information; scoring of individual outputs must always reflect expert judgement.
[i.] Ensuring that judgements about the quality of individual outputs are not made solely on the basis of citation information; scoring of individual outputs must always reflect expert judgement.40
[ii.] Where panels decide to review samples of the outputs, robust sampling techniques should be used and sufficient samples of outputs should be reviewed for each submission.
[ii.] Where panels decide to review samples of the outputs, robust sampling techniques should be used and sufficient samples of outputs should be reviewed for each submission.41
[iii.] Particular caution should be exercised when using citation data for more recently published outputs – and we do not plan to provide any citation information for outputs published in the final year of the REF period (2012).
[iii.] Particular caution should be exercised when using citation data for more recently published outputs – and we do not plan to provide any citation information for outputs published in the final year of the REF period (2012).42
[iv.] Outputs that are not matched to citation databases or are published too recently for citation data to be available must be treated equally, and should be at least as likely to be reviewed in detail, as those outputs for which there is citation data available.
[iv.] Outputs that are not matched to citation databases or are published too recently for citation data to be available must be treated equally, and should be at least as likely to be reviewed in detail, as those outputs for which there is citation data available.43
[v.] The value of citations as a quality indicator for research with a more applied focus is limited, and the short statements about the ‘user significance’ of such outputs, where provided, should be taken into account.
[v.] The value of citations as a quality indicator for research with a more applied focus is limited, and the short statements about the ‘user significance’ of such outputs, where provided, should be taken into account.44
[f.] We intend that panels will specify in their criteria whether they will use citation data; and if so, how they intend to make use of such data (including any criteria for sampling), within the guidelines provided.
[f.] We intend that panels will specify in their criteria whether they will use citation data; and if so, how they intend to make use of such data (including any criteria for sampling), within the guidelines provided.45
[46.] This approach to the assessment of outputs retains scope for the assessment of grey literature and work published in non-standard forms (for which citation data are unlikely to be available), and to assess the quality of applied and other types of research for which citation indicators may have limited value. This will be important in enabling panels to give full and equal consideration to all outputs of high quality, including from work undertaken directly for or in collaboration with research users and research to inform public policy.
[46.] This approach to the assessment of outputs retains scope for the assessment of grey literature and work published in non-standard forms (for which citation data are unlikely to be available), and to assess the quality of applied and other types of research for which citation indicators may have limited value. This will be important in enabling panels to give full and equal consideration to all outputs of high quality, including from work undertaken directly for or in collaboration with research users and research to inform public policy. 46
[47.] In keeping with this approach we will issue clear guidance to institutions advising them to select for submission those research outputs of their staff that best represent the quality of their research, regardless of whether citation information is available for these outputs.
[47.] In keeping with this approach we will issue clear guidance to institutions advising them to select for submission those research outputs of their staff that best represent the quality of their research, regardless of whether citation information is available for these outputs. 47
[48.] Within those UOAs using citation information, institutions will be expected to verify links made between their submitted outputs and the relevant citation database(s). The REF data collection system will provide a means for doing this. The citation statistics associated with the matched outputs will be generated centrally through a standardised process, and added by the REF Team to institutions’ submissions. These data will be made available to institutions prior to the submission deadline and before they are provided to the expert panels. We do not anticipate that institutions would verify the citation statistics; however we will require that the selected database supplier(s) provide facilities to correct errors regarding citation links in the underlying data that institutions may find.
[48.] Within those UOAs using citation information, institutions will be expected to verify links made between their submitted outputs and the relevant citation database(s). The REF data collection system will provide a means for doing this. The citation statistics associated with the matched outputs will be generated centrally through a standardised process, and added by the REF Team to institutions’ submissions. These data will be made available to institutions prior to the submission deadline and before they are provided to the expert panels. We do not anticipate that institutions would verify the citation statistics; however we will require that the selected database supplier(s) provide facilities to correct errors regarding citation links in the underlying data that institutions may find.48
[49.] Given that a number of expert panels will make use of citation data to inform their judgements, we anticipate that institutions may wish to make use of such data to inform their selection of outputs. In all cases, however, this should only inform decisions made alongside other considerations of quality, and bearing in mind the limitations of citation information (especially regarding recently published outputs, and limitations in the coverage of the citation databases). We will work with JISC Collections and institutions to ensure that citation information is accessible to those institutions that may want it for this purpose, without undue additional cost and in a form that is consistent with the information to be provided to panels. Ideally such data will be available through the REF data collection system, subject to future developments and procurement decisions.
[49.] Given that a number of expert panels will make use of citation data to inform their judgements, we anticipate that institutions may wish to make use of such data to inform their selection of outputs. In all cases, however, this should only inform decisions made alongside other considerations of quality, and bearing in mind the limitations of citation information (especially regarding recently published outputs, and limitations in the coverage of the citation databases). We will work with JISC Collections and institutions to ensure that citation information is accessible to those institutions that may want it for this purpose, without undue additional cost and in a form that is consistent with the information to be provided to panels. Ideally such data will be available through the REF data collection system, subject to future developments and procurement decisions.50
[50.] We have discussed with the EAGs the wider question of how the burden on panels of reviewing large volumes of outputs could be reduced. One option is to reduce the number of outputs submitted per person from four to three (particularly if the assessment period is five rather than seven years). Further options include:
[50.] We have discussed with the EAGs the wider question of how the burden on panels of reviewing large volumes of outputs could be reduced. One option is to reduce the number of outputs submitted per person from four to three (particularly if the assessment period is five rather than seven years). Further options include:51
[a.] Sampling the outputs to be reviewed in detail, particularly where citation data are available. We propose that panels should specify in their criteria statements the proportion of outputs they expect to review in detail.
[a.] Sampling the outputs to be reviewed in detail, particularly where citation data are available. We propose that panels should specify in their criteria statements the proportion of outputs they expect to review in detail. 52
[b.] Explicitly double-weighting more substantive outputs such as monographs that encapsulate several years’ work. We propose that panels should consider whether certain types of output should be double-weighted (counting as two submitted outputs) and where applicable to develop criteria for this. The criteria should be sufficiently clear so that when making submissions, the institution can be confident about which outputs will be double-weighted.
[b.] Explicitly double-weighting more substantive outputs such as monographs that encapsulate several years’ work. We propose that panels should consider whether certain types of output should be double-weighted (counting as two submitted outputs) and where applicable to develop criteria for this. The criteria should be sufficiently clear so that when making submissions, the institution can be confident about which outputs will be double-weighted. 53
Consultation question 2
What comments do you have on the proposed approach to assessing outputs? If you disagree with any of these proposals please explain why.Comments are especially welcomed on the following proposals:
- that institutions should select research staff and outputs to be assessed
- for the categories of staff eligible for selection, and how they are defined
- for encouraging institutions to submit – and for assessing – all types of high-quality research outputs including applied and translational research
- for the use of citation information to inform the review of outputs in appropriate UOAs (including the range of appropriate UOAs, the type of citation information that should be provided to panels as outlined in Annex C, and the flexibility panels should have in using the information)
and on the following options:
- whether there should be a maximum of three or four outputs submitted per researcher
- whether certain types of output should be ‘double-weighted’ and if so, how these could be defined.
Tags: HEI, Higher Education Statistics Agency, NHS, researcher
This is encouraging – I’d like to see new forms of output more explicitly recognised, eg blogs, videos, etc but at least it is a move away from the standard article.
These are not really definitions, more successive subjective scales. The danger that excellence as determined by an existing panel may not reward innovation is still present.
While this is true for standard outputs the metrics might be a useful measure of overall impact of new types of scholarly activity. For example, it may be useful to look at the overall statistics relating to a blog, rather than one specific article or post.
This is a _very_ limited approach to the use of metrics, and effectively limits it to standard articles, thus undermining the principle above that any outputs will be recognised if they meet the definition of research. If we wanted to encourage new types of scholarly activity then using metrics that measured the influence of these would be a significant step forward.
An example of the types of metrics that could measure the impact of non-article type work can be found at this Public Library Of Science initiative: http://article-level-metrics.plos.org/