Quick Analysis of RAE Panel Statements Concerning the Treatment of Refereed Items

Panel A
UOA 1 Cardiovascular Medicine
UOA 2 Cancer Studies
UOA 3 Infection and Immunology
UOA 4 Other Hospital Based Clinical Subjects
UOA 5 Other Laboratory Based Clinical Subjects

The main panel expects the majority of research output in its field to consist of papers describing original research, published in peer reviewed scientific journals. The main and subpanels will give equal weight to individual and collaborative research, and to work of direct relevance to the needs of commerce and industry, potential applicability to human health, including to the NHS or other health care agencies, and to other parts of the public and voluntary sectors. Other forms of output – including monographs and exceptionally books, patents and other applied research output such as software packages, images and devices – will also be accepted and judged on their individual research merits. The sub-panel will assess research output based on the definition of research provided in Annex 3

UOA 17, Earth Systems and Environmental Sciences

13. Research outputs will be weighted at 65% of the overall quality profile. The sub-panel will consider any form of output. Irrespective of the form and type of output, the sub-panel will seek, above all, to consider the intrinsic research quality of items submitted. The sub-panel will neither rank nor regard any particular form of output as inherently of greater or lesser quality than another. Outputs related to basic research and those relating to practical applications, including intellectual property such as patents, and work published in non-print media, will be equally acceptable. Although refereeing and editorial standards may be used to indicate high quality, the sub-panel will not use a ranked list of journals in its assessment process. The context of the output and its contribution to the research field, particularly the impact on the wider field of science, will be considered in relation to the significance of a particular output.

Panel C
10 Dentistry
11 Nursing and Midwifery
12 Allied Health Professions and Studies
13 Pharmacy

16. The normal expectation is that staff should submit four outputs for assessment. Four is also the maximum. In the event that fewer than four outputs are cited for individuals, departments should draw the relevant sub-panel’s attention in RA5b to any individual staff circumstances they deem relevant, including career and equal opportunities issues.

17. Co-authored work will not be treated any differently from single-authored work. It is expected that the author who cites co-authored work will have made a substantial contribution to it. Because it is the quality of outputs which is being assessed, no inference will be placed on the order of attribution of authors.

18. The fullest and most favourable impression of a department’s research will normally be gained when each output is only cited once. If an output is cited by two or more co-authors within a single submission, this must be identified in RA2 ‘Other relevant details’ – for each occurrence of the output – with a justification for the multiple citing in up to 50 words (eg, the scale of the output, or a clear and fundamental difference in the co-authors’ contributions to the work). If a sub-panel is not persuaded by the reason for citing the output more than once, all occurrences of the output but one may be accorded an Unclassified quality level.

19. Evidence that research outputs have already been reviewed or refereed by experts and judged to embody research of high quality may be used as one measure of quality. However the absence of such review will not, in itself, be taken to imply lower quality.

20. Indicators used to judge research outputs will include originality, significance and rigour, as demonstrated by the extent to which knowledge or theory in the field has been increased or practice or policy has been, or is likely to be, improved.

Panel B
6 Epidemiology and Public Health
7 Health Services Research
8 Primary Care and Other Community Based Clinical Subjects
Psychiatry, Neuroscience and Clinical Psychology

14. These will normally comprise papers in scientific journals, monographs and books; patents and other outputs of applied research and other forms of output will also be accepted and judged on their individual merits. The normal expectation is that staff should submit four outputs for assessment; four is also the maximum. The main and sub-panels will not undertake analysis of bibliometric data when making judgements on research outputs

UOA 51, Russian, Slavonic and East European Languages

UOA 52, French

UOA 53, German, Dutch and Scandinavian Languages

UOA 54, Italian

UOA 55, Iberian and Latin American Languages

UOA 56, Celtic Studies

UOA 57, English Language and Literature

UOA 58, Linguistics

14. The sub-panel will assess all forms of output equally according to the published criteria, and give full recognition to achievements irrespective of form or mode of delivery. However, the subpanel appreciates that the nature of the discipline is such that from time to time there may be projects of significant scale and scope requiring an investment of time and personal effort considerably greater than the expected norm. It will therefore use its discretion to recognise special achievement in such works and credit their contribution to the discipline appropriately through additional weighting in the quality profile. It anticipates that such judgements will be exceptional. It will not require HEIs to nominate such outputs in submissions, and will disregard any such claims.

15. The principal forms of output that the subpanel expects to assess are listed below:

a. Academic journal articles.

b. Bibliographies* (see paragraph 17 below).

c. Books [... etc]

16. All categories include printed and electronically published items and those produced in other media. No ranking or weighting should be inferred from the order in which the categories are listed. While acknowledging the value of the refereeing process, the sub-panel recognises that some research is published in journals or other outlets which do not use refereeing procedures. Each item will be assessed on its individual merits, according to the sub-panel’s stated criteria, described in paragraphs 47-60 below, and place of publication will not influence the sub-panel’s independent assessment of the quality of an output.

UOA 59, Classics, Ancient History, Byzantine and Modern Greek Studies

11. A research output is the outcome of a research process that is presented in the public domain (unless it is a confidential output. As well as judging the quality of submitted outputs, the sub-panel will attach additional weight in the quality profile to those outputs which it considers to be of significant scale and scope (see the main panel statement paragraph 21 and paragraphs 24-27 below).

12. In judging outputs the sub-panel will be guided solely by its view of their research quality. All cited outputs will be judged on academic merit regardless of the medium (for example, in paper or electronic form) or location of publication. Web-based publications, including for instance those published on publicly available departmental web-sites, will be judged by the same standards and criteria as other outputs.

13. The sub-panel will not treat any category of output as intrinsically superior or inferior to any other. Outputs not already subject to peer review or refereeing will not automatically be regarded as of lesser quality.

UOA 6, Epidemiology and Public Health

10. All forms of research output will be evaluated by the sub-panel in the same way using criteria including the originality, scientific rigour, contribution to knowledge and conceptual framework of the field, as well as the challenge and logistical difficulty posed by the work. The sub-panel will also assess the potential and actual implementation of the research, and its potential impact on health and healthcare and on public health and public policy.

11. The sub-panel expects to receive outputs in the forms described in paragraphs 14-15 of the main panel statement. It is recognised that original research may include systematic reviews (eg, the Cochrane type); meta-analyses; the analysis and interpretation of secondary data and sample collections; qualitative research and the presentation of important new hypotheses; work of direct relevance to the needs of commerce and industry, the NHS and other parts of the public and voluntary sectors, including patents and other applied research, innovative statistical routines and software. Departments are also encouraged to submit applied research and practice-based research, which will be considered equally with other forms of research and will be judged according to the way in which it has advanced a field and/or changed its practice

UOA 7, Health Services Research

8. All forms of research output will be evaluated by the sub-panel in the same way using criteria including the originality, scientific rigour, contribution to knowledge and conceptual framework of the field as well as the challenge and logistical difficulty posed by the work. According to these criteria, meta-analyses, systematic reviews, qualitative and quantitative research, and new hypotheses will be considered equally. The subpanel will also assess the potential and actual implementation of the research and its potential impact on health and healthcare, and on public health, public policy and society more broadly. The sub-panel expects to receive outputs in the forms described in paragraphs 14-15 of the main panel statement. These will most commonly consist of journal articles but can be published in a wide range of media. [...]

UOA 8, Primary Care and Other Community Based Clinical Subjects

10. The sub-panel expects that much of the research submitted to it will have been published in peer-reviewed journals. The sub-panel recognises that, especially in some social science disciplines, significant advances may also be reported in books, and sometimes in book chapters.

UOA 30, Architecture and the Built Environment

14. The sub-panel wishes to encourage the submission of the diverse range of types of research output defined in Annex B of RAE 03/2005 ‘Guidance on submissions’ – including papers, books, materials, images, devices, patents, artefacts, designs, design codes, buildings, prototypes and installations, products and processes, time-based media, exhibits, software, and work published in non-print media. It will assess these against the common criteria of originality, significance and rigour, while attaching no greater weight to any particular form of output.

17. All types of research and all forms of research output will be assessed on a fair and equal basis. The sub-panel will take into account the context in which an output was produced, including any prior peer review process or other selection procedure.

UOA 23, Computer Science and Informatics

18. In assessing the quality of outputs the subpanel will look for originality, rigour and significance to the discipline and wider research community and, where appropriate, to users. The assessment will be based on the content of the output and additional evidence provided in RA2 (see paragraph 22). Evidence that outputs have already been reviewed or refereed by experts and judged to embody work of high quality may be used as one measure of quality. However the absence of such review will not, in itself, be taken to imply lower quality.

UOA 9, Psychiatry, Neuroscience and Clinical Psychology

11. The sub-panel expects to receive outputs in the forms described in paragraphs 14-15 of the main panel statement. These will most commonly consist of primary, data-based research papers but meta-analyses and systematic reviews will also be considered, as will outputs relating to the development of psychometric tests. Books or review articles that contain a substantial innovative or original contribution, eg, presenting a fresh interpretation of data or putting forward a novel hypothesis or theory, will also be judged as research. Outputs such as standard review articles or textbook chapters that simply survey previously published work without a significant, novel, intellectual contribution will not be considered as research and will be rated as Unclassified.

UOA 60, Philosophy

13. Types of output will not be ranked against each other, and outputs not already subject to a peer review or refereeing process will not for that reason be regarded as of lesser quality. No form of output will be regarded as intrinsically inferior to any other.

UOA 14, Biological Sciences

UOA 15, Pre-clinical and Human Biological Sciences

UOA 16, Agriculture, Veterinary and Food Science

19. The sub-panel expects to receive outputs mainly in the form of scientific papers, but other outputs (eg, books, databases, decision-support software, informatics resources, published accounts of new techniques or new therapies, patents, plant breeders’ rights, policy papers, reviews) will be considered. Where items other than scientific papers are submitted, departments are encouraged to use the ‘Other relevant details’ field in RA2 to identify how the output embodies original research.

UOA 33, Archaeology

14. The sub-panel expects to receive submissions in a wide range of formats, including journal papers, books, book chapters, edited works, primary data reports, conference proceedings, reviews, exhibitions, software, CD-ROM, and web resources. For the purposes of assessment, all forms of research output will be treated equally against the standard criteria of originality, significance and rigour. All outputs, including teaching materials, must comply with the RAE definition of research and be accessible to the subpanel in a format suitable for assessment.

20. [...] All work will be assessed in the context of comparable work in that subject and the scope of its impact will be considered. The sub-panel will not employ a ranking system for different forms of output or publishers, but editorial and refereeing practices will be taken into account.

UOA 24, Electrical and Electronic Engineering

UOA 25, General Engineering and Mineral & Mining

Engineering

UOA 26, Chemical Engineering

UOA 27, Civil Engineering

UOA 28, Mechanical, Aeronautical and Manufacturing

Engineering

UOA 29, Metallurgy and Materials

17. When considering journal articles, conference papers and other outputs the sub-panel may consider the editorial and refereeing standards as part of the indication of quality, but absence of these standards will not be taken to mean an automatic absence of quality.

Panel D
14 Biological Sciences
15 Pre-clinical and Human Biological Sciences
16 Agriculture, Veterinary and Food Science

32. Up to four outputs per person returned may be submitted; four is also the normal expectation.

33. All forms of assessable research output will be welcome, reflecting the range of activity (curiosity-driven, applied, practice-based) undertaken.

34. Sub-panels will assess each output submitted. They will read each output to the level of detail required to arrive at a judgement of its quality.

35. Assessment will be based on the originality, significance and rigour of the items submitted.

36. Sub-panels will not make assumptions about quality based on output type or place of publication.

37. Departments will be invited to use the ‘Other relevant details’ field in RA2 to:

• indicate an individual’s discrete and substantial contribution to a multi-author output

• illustrate how outputs other than scientific papers embody original research.

38. There is an overall limit on the ‘Other relevant details’ field in RA2 of 50 words per output.

UOA 61, Theology, Divinity and Religious Studies

15. Types of output will not be ranked against each other, and outputs not already subject to a peer review or refereeing process will not for that reason be regarded as of lesser quality. No form of output will be regarded as intrinsically inferior to any other.

UOA 20, Pure Mathematics

20. The sub-panel expects that most outputs listed will have been through a rigorous refereeing process and will take account of this in assessing quality.

UOA 19, Physics

15. The sub-panel will examine all outputs; and no less than 50% of the research outputs will be examined in detail. Sub-panel members will use their professional judgement and their initial examination of the outputs to identify which will be examined in detail. Books, edited works and review articles will carry weight if they contain new and original material and relate to work to which the author has made major contributions. Publication in academic journals with rigorous editorial and refereeing standards may be taken as an indicator of quality. However, the guiding principle will be that the sub-panel will base its deliberations on the quality and significance of the scientific work in the outputs submitted, regardless of the medium of publication.

UOA 18, Chemistry

16. The sub-panel will neither rank nor regard any particular form of output as inherently of greater or lesser quality than another. In addition to printed academic outputs, it will consider (but not be limited to) outputs in the following form:

• new materials, devices, images and products

• intellectual property, whether in patents or other forms

• work published in non-print media.

UOA 62, History

16. Types of output will not be ranked against each other, and outputs not already subject to a peer review or refereeing process will not for that reason be regarded as of lesser quality. No form of output will be regarded as intrinsically inferior to any other.

UOA 21, Applied Mathematics

13. All forms of research output will be treated equally. The sub-panel expects that the majority of research outputs submitted will take the form of original research papers in refereed journals or proceedings (including electronic journals); it expects also to see major review articles, research monographs and other forms of research output. If other forms of research output (eg, software, patents) are submitted, departments should provide an explanation of the research content in up to 300 words in the ‘Other relevant details’ field in RA2. The sub-panel does not expect the

field to be used for any other purpose except in exceptional circumstances.

OA 43, Development Studies

10. All forms of research output will be treated equally. The panel expects to receive outputs across a wide range of formats, including monographs, journal papers/articles, chapters in edited volumes, working/discussion papers, electronic publications, research reports, publications of development donors and multilateral and international agencies.

11. Development studies is not an area in which there is an established hierarchy of journals, and the sub-panel recognises that work of the highest quality can be found in a range of media and forms. It will however use (where it exists) evidence of the robustness of peer review procedures to inform its own professional judgement of the quality of relevant outputs.

Panel G
24 Electrical and Electronic Engineering
25 General Engineering and Mineral & Mining Engineering
26 Chemical Engineering
27 Civil Engineering
28 Mechanical, Aeronautical and Manufacturing Engineering
29 Metallurgy and Materials

11. These outputs might take the form of research monographs, in whole or part; authored articles in professional journals; conference contributions; conference reports; descriptions of new devices and instrumentation; descriptions of new processes and materials; patents awarded; published papers in peer-reviewed journals; software; and technical reports.

UOA 22, Statistics and Operational Research

14. All forms of research output will be treated equally. The sub-panel expects to receive a majority of research outputs in the form of original research papers in refereed journals (including electronic journals); it also expects to see major review articles, research books and monographs, substantial software packages and other forms of research output.

UOA 44, Psychology

12. All forms of research output will be judged according to the same criteria. The sub-panel expects to receive outputs in various forms including: journal articles, books, chapters in books, published conference papers (including abstracts), data sets or software, electronic publications, government reports, technical or other reports, test materials and patents. [...]

17. Already applied standards of peer review will be used to inform quality judgements. The subpanel’s reading will be focused on refining quality judgements at category boundaries and on gaining evidence in less familiar areas and forms of output. The sub-panel expects collectively by the end of the exercise to have examined in detail virtually all outputs. In the case of journal articles, the sub-panel will not collectively rank publication outlets nor will it draw up a list of journals which are assumed to be automatic indicators of quality. It will, instead, be guided by its individual and collective judgements of reviewing, refereeing and editorial standards. The sub-panel emphasises that work need not be published in outlets based overseas in order to be judged as meeting levels of international quality. Conversely, outputs published in outlets based overseas will not automatically be assumed to be of international quality. The sub-panel will not assume that outputs in refereed or non-refereed outlets with which it is not familiar are of lesser quality, but will judge the individual quality of such outputs. The sub-panel recognises that it is appropriate for some types of research to be published in less prominent mainstream outlets, and that high quality psychological research may

also be published in outlets associated with other disciplines. The sub-panel may request advice from specialist advisers or other sub-panels as appropriate. It will take an interest in the impact of an output on practice as well as in the academic environment.

UOA 31, Town and Country Planning

15. The sub-panel expects to assess a wide range of research outputs. All forms of research output will be given full consideration, including refereed journal articles, research-based books and book chapters, refereed conference papers and research outputs from projects commissioned by all levels of government, industry and other research funding bodies. Research outputs in electronic formats, embracing developments in appraisal, forecasting, data processing and analysis, web-based case studies and associated software may also be included. Pedagogical research in the field will be welcomed.

UOA 32, Geography and Environmental Studies

20. The sub-panel expects to receive outputs in a wide range of forms, providing evidence of research that may be considered as basic/strategic, applied, or practice-based (see paragraphs 46-47). Irrespective of the form and type of output, the sub-panel will seek, above all, to consider the intrinsic research quality of items submitted. Outputs must satisfy the RAE definition of research and be submitted in a form suitable for assessment. Submitted outputs could include: papers in refereed journals; authored books; chapters in books; monographs; edited books; refereed conference papers; software; electronic and web-based publications; published maps; other research-based contributions to debates on major scientific and policy issues. The sub-panel will work with the presumption that all forms of output are capable of achieving the highest standards of excellence.

Panel I
34 Economics and Econometrics
35 Accounting and Finance
36 Business and Management Studies
37 Library and Information Management

18. The main panel expects its sub-panels to consider all the components of submissions when recommending a quality profile. As noted in paragraph 11, the main panel regards cited outputs as the most important indicator of research quality. It is expected that most of the work submitted will be articles in refereed journals, books (including chapters in books) and research monographs, but the panel will not regard any form of output as necessarily being of higher or lower quality than others.

UOA 38, Law

15. All forms of research output will be treated equally. The sub-panel recognises that scholarly work of significance, originality and rigour may be found across all forms of output (including non-print media).

UOA 39, Politics and International Studies

16. All forms of research output will be treated equally. The sub-panel will not rank nor regard any particular form of output as of greater or lesser quality than another per se.

UOA 40, Social Work and Social Policy & Administration

10. All forms of research output will be treated equally – the sub-panel will not rank nor regard any particular form of output as of greater or lesser quality than another per se. The sub-panel expects to receive a range of research outputs, including articles in refereed journals, professional and practitioner journals and published conference proceedings, books, chapters in edited books, edited books, research reports, and training materials which embody the results of original research undertaken by the authors. [...]

UOA 41, Sociology

13. All forms of research output will be treated equally. The sub-panel expects to receive outputs in the form of (but not limited to) journal articles, books, chapters in edited books, edited books, research reports, reports to statutory, official, and private-sector bodies, and other materials that embody the results of original research undertaken by the authors. Sociology as a discipline is not characterised by a hierarchy of journals and the sub-panel recognises that work of the highest quality can be found in a range of media.

UOA 42, Anthropology

12. All forms of research output will be treated equally – the sub-panel will not rank nor regard any particular form of output as of greater or lesser quality than another per se. While published books and articles are likely to comprise the majority of research outputs submitted for review, the sub-panel expects to receive a broad range of other research outputs, given the diverse nature of anthropology. The sub-panel will consider finished material which is in the public domain including, but not limited to, articles, books, film, exhibitions, electronic media (including date-stamped copies of web-sites). It will also consider policy reports and other works of consultancy that may be confidential.

Panel K
44 Psychology
45 Education
46 Sports-Related Studies

16. The panel agrees that research excellence can be found within a wide range and variety of forms. All types of output will be judged on the basis of the criteria of rigour, significance and originality as defined within the sub-panels’ assessment criteria. Since the main panel does not feel that there are easily identified categories of research, these criteria are not separately defined for different types of research.

UOA 45, Education

12. The sub-panel welcomes the submission of all forms of research output and will treat them equally. The sub-panel expects to receive outputs in the following forms: articles in journals and e-journals; books; chapters in books; published conference papers; research reviews, electronic and online publications; government reports; technical or other reports. Institutions may also wish to submit other forms of output where these meet the definition of research set out in Annex 3. [...]

17. Because of the range of work and publication media within the discipline, the sub-panel will not collectively rank publications, nor will it draw up a list of journals which will be assumed to be automatic indicators of quality. [..]

UOA 46, Sports-Related Studies

11. Departments may submit all forms of output where these meet the definition of research set out in Annex 3. All forms of research output will be

judged according to the same criteria. The subpanel expects to receive a variety of forms of research outputs for both theoretical and practice based or applied research, including: journal articles, books, academic papers, materials, chapters in books, government, technical and other reports, published conference papers, patents, devices, and electronic and online publications. The sub-panel will, in all cases, have regard to the quality and not to the type of output.

UOA 48, Middle Eastern and African Studies

16. All forms of research output will be treated equally. The sub-panel expects that the principal categories of research output that will be cited are

as follows:

a. Academic journal articles.

b. Bibliographies (to the extent that they embody research).

c. Books [...etc]

17. No ranking or weighting should be inferred from the order in which these items are listed. The sub-panel will assess all forms of research output in the same fashion, regardless of its medium or source of publication, or publisher.

UOA 47, American Studies and Anglophone Area Studies

15. The sub-panel is prepared to assess any publicly accessible academic work, single- or co-authored, including academic journal articles, advanced textbooks, bibliographies, books, edited books or special issues of journals with substantial research input on the part of the editor, editions of texts, chapters in books, contributions to conference proceedings, creative writing/production, short works including dictionary and encyclopaedia entries, pamphlets, published lectures, consultancy-based reports, review articles, translations, review articles, teaching materials; equivalent electronically available work in the public domain; practice based and performance outputs and exhibitions; and other media where there is a demonstrable research content.

UOA 49, Asian Studies

14. The sub-panel will not draw up a predetermined ranking of forms of output or publication outlets, or rely on citation indices. All contributions to research will be reviewed for their originality, imaginative range, significance, impact and rigour in the context of the quality levels set out in Annex 1 and interpreted by Main Panel L as described in paragraph 12 of the main panel statement.

15. The sub-panel is prepared to assess any publicly accessible academic work single- or co-authored, including:

• academic journal articles

• advanced textbooks, descriptive grammars and teaching materials

• bibliographies [...etc]

UOA 50, European Studies

13. All forms of research output will be treated equally.

14. The sub-panel is particularly conscious that the use of the quality levels must be relevant to the range of outputs across the sub-fields of activity covered by the UOA. The criteria developed by the sub-panel take account of the diversity of output, rather than imposing a narrow and prescriptive characterisation of research in the field of activities covered. The sub-panel will not operate a hierarchy of outlets for different forms of output.

16. The sub-panel’s definition of excellence in terms of research outputs is not concerned with the publisher or journal or with the nature of the topic, but with the achievement of the highest standards attainable. The sub-panel’s professional judgement of excellence is based upon the quality, ambition and originality of the research design and methodology; on the depth and imaginative range of scholarship displayed; on the rigour and clarity of the work; and on the impact or potential impact of findings on a particular field.

Panel O
63 Art and Design
64 History of Art, Architecture and Design
65 Drama, Dance and Performing Arts
66 Communication, Cultural and Media Studies
67 Music

4. Research will be assessed where it: has been published, exhibited, performed, recorded, screened or broadcast during the publication period; meets the definition of research for the RAE; has entered the public domain during the publication period; and can be judged against the assessment criteria and methods described in this statement alongside those for the sub-panels. With these conditions in mind, the sub-panels will assess the intrinsic quality of research wherever and however it is undertaken, and whatever its form of output.

UOA 63, Art and Design

UOA 64, History of Art, Architecture and Design

UOA 65, Drama, Dance and Performing Arts

UOA 67, Music

8. The sub-panel will neither advantage nor disadvantage any type of research or form of output, whether it be physical or virtual, textual or non-textual, visual or sonic, static or dynamic, digital or analogue. Outputs may include, but are not limited to (in no particular order): books (authored or edited); chapters in books; journal articles; conference contributions; curatorship and conservation; digital and broadcast media; performances and other types of live presentation; artefacts, designs and exhibitions; films, videos and other types of media presentation; advisory reports; and the creation of archival or specialist collections to support the research infrastructure. In all cases the research outputs will be assessed against the indicators of excellence and degrees of quality described in Table 2 and paragraphs 15-23 of the main panel statement.

UOA 66, Communication, Cultural and Media Studies

8. The sub-panel will neither advantage nor disadvantage any type of research or form of output, whether it be physical or virtual, textual or non-textual, visual or sonic, static or dynamic, digital or analogue. In all cases the research outputs will be assessed against the indicators of excellence and degrees of quality described in Table 2 and paragraphs 15-23 of the main panel statement.