The Stern Review of the REF (Research Excellence Framework) is the latest British government review of research assessment in the UK, following on from the Metric Tide assessment. I have just made a submission to the enquiry. My main comment in response to the first question (1. What changes to existing processes could more efficiently or more accurately assess the outputs, impacts and contexts of research in order to allocate QR? Should the definition of impact be broadened or refined? Is there scope for more or different use of metrics in any areas?) follows:
"I think that there is substantial scope for using bibliometrics in the conduct of the REF. In Australia the Australian Research Council uses metrics to assess natural science disciplines and psychology. Research that I have conducted with my coauthor, Stephan Bruns, shows that this approach could be extended to economics and probably political science and perhaps other social sciences. We have written a working paper presenting our results that is currently under review by Scientometrics.
The paper shows that university rankings in economics based on long-run citation counts can be easily predicted using early citations. The rank correlation between universities' cumulative citations received over ten years for economics articles published in 2003 and 2004 and citations received in 2003 to 2004 alone is 0.91 in the UK and 0.82 in Australia. We compare these citation-based university rankings with the rankings of the 2008 Research Assessment Exercise in the UK and the 2010 Excellence in Research assessment in Australia. Rank correlations are quite strong but there are differences between rankings based on this type of peer review and rankings based on citation counts. However, if assessors are willing to consider citation analysis to assess some disciplines as is the case for the natural sciences and psychology in Australia there seems no reason to not include economics in this set.
Previously, I published a paper, published in PLoS One showing that the predictability of citations at the article level is similar in economics and political science. This supports the view that metrics based research assessment can cover both economics and political science in addition to the natural sciences and economics.
I believe the REF review should seriously consider these findings in producing recommendations for a lighter touch future REF."
I also made briefer responses to some of their other questions. In particular:
5. How might the REF be further refined or used by Government to incentivise constructive and creative behaviours such as promoting interdisciplinary research, collaboration between universities, and/or collaboration between universities and other public or private sector
bodies?
"A major issue with the REF and the ERA in Australia is the pigeon-holing of research into disciplines, which might not match well the nature of the research conducted. This clearly will discourage publication in interdisciplinary venues that may not be as respected by mainstream reviewers. The situation is less acute in Australia where a single output can be allocated across different assessment disciplines, but I still think that assessment by pure disciplinary panels discourages interdisciplinary work in Australia. So, I imagine this is exacerbated in the UK.
7. In your view how does the REF process influence the development of academic disciplines or impact upon other areas of scholarly activity relative to other factors? What changes would create or sustain positive influences in the future?
Johnston et al. (2014) show that the total number of economics students has increased in UK more rapidly than the total number of all students, but the number of departments offering economics degrees has declined, particularly in post-1992 universities. Also, the number of universities submitting to the REF under economics has declined sharply with only 3 post-1992 universities submitting in the latest round. This suggests that the REF has driven a concentration of economics research in the more elite universities in the UK.
Johnston, J., Reeves, A. and Talbot, S. (2014). ‘Has economics become an elite subject for elite UK universities?’ Oxford Review of Education, vol. 40(5), pp. 590-609.
"I think that there is substantial scope for using bibliometrics in the conduct of the REF. In Australia the Australian Research Council uses metrics to assess natural science disciplines and psychology. Research that I have conducted with my coauthor, Stephan Bruns, shows that this approach could be extended to economics and probably political science and perhaps other social sciences. We have written a working paper presenting our results that is currently under review by Scientometrics.
The paper shows that university rankings in economics based on long-run citation counts can be easily predicted using early citations. The rank correlation between universities' cumulative citations received over ten years for economics articles published in 2003 and 2004 and citations received in 2003 to 2004 alone is 0.91 in the UK and 0.82 in Australia. We compare these citation-based university rankings with the rankings of the 2008 Research Assessment Exercise in the UK and the 2010 Excellence in Research assessment in Australia. Rank correlations are quite strong but there are differences between rankings based on this type of peer review and rankings based on citation counts. However, if assessors are willing to consider citation analysis to assess some disciplines as is the case for the natural sciences and psychology in Australia there seems no reason to not include economics in this set.
Previously, I published a paper, published in PLoS One showing that the predictability of citations at the article level is similar in economics and political science. This supports the view that metrics based research assessment can cover both economics and political science in addition to the natural sciences and economics.
I believe the REF review should seriously consider these findings in producing recommendations for a lighter touch future REF."
I also made briefer responses to some of their other questions. In particular:
5. How might the REF be further refined or used by Government to incentivise constructive and creative behaviours such as promoting interdisciplinary research, collaboration between universities, and/or collaboration between universities and other public or private sector
bodies?
"A major issue with the REF and the ERA in Australia is the pigeon-holing of research into disciplines, which might not match well the nature of the research conducted. This clearly will discourage publication in interdisciplinary venues that may not be as respected by mainstream reviewers. The situation is less acute in Australia where a single output can be allocated across different assessment disciplines, but I still think that assessment by pure disciplinary panels discourages interdisciplinary work in Australia. So, I imagine this is exacerbated in the UK.
7. In your view how does the REF process influence the development of academic disciplines or impact upon other areas of scholarly activity relative to other factors? What changes would create or sustain positive influences in the future?
Johnston et al. (2014) show that the total number of economics students has increased in UK more rapidly than the total number of all students, but the number of departments offering economics degrees has declined, particularly in post-1992 universities. Also, the number of universities submitting to the REF under economics has declined sharply with only 3 post-1992 universities submitting in the latest round. This suggests that the REF has driven a concentration of economics research in the more elite universities in the UK.
Johnston, J., Reeves, A. and Talbot, S. (2014). ‘Has economics become an elite subject for elite UK universities?’ Oxford Review of Education, vol. 40(5), pp. 590-609.
No comments:
Post a Comment