Replicating Stern (1993)

Last year, Energy Economics announced a call for papers for a special issue on replication in energy economics. Together with Stephan Bruns and Johannes König we decided to do a replication of my 1993 paper in Energy Economics on Granger causality between energy use and GDP. That paper was the first chapter in my PhD dissertation. It is my fourth most cited paper and given the number of citations could be considered "classic" enough to do an updated robustness analysis on it. In fact, another replication of my paper has already been published as part of the special issue. The main results of my 1993 paper were that in order to find Granger causality from energy use to GDP we need to  use both a quality adjusted measure of energy and control for capital and labor inputs.

It is a bit unusual to include the original author as an author on a replication study, and my role was a bit unusual. Before the research commenced, I discussed with Stephan the issues in doing a replication of this paper, giving feedback on the proposed design of the replication and robustness analysis. The research plan was published on a website dedicated to pre-analysis plans. Publishing a research plan is similar to registering a clinical trial and is supposed to help reduce the prevalence of p-hacking. Then, after Stephan and Johannes carried out the analysis, I gave feedback and helped edit the final paper.

Unfortunately, I had lost the original dataset and the various time series I used have been updated by the US government agencies that produce them. The only way to reconstruct the original data would have been to find hard copies of all the original data sources. Instead we used the data from my 2000 paper in Energy Economics, which is quite similar to the original data. Using this close to original data, Stephan and Johannes could reproduce all my original results in terms of the direction of Granger Causality and the same qualitative significance levels. In this sense, the replication was a success.

But the test I did in 1993 on the log levels of the variables is inappropriate if the variables have stochastic trends (unit roots). The more appropriate test is the Toda-Yamamoto test. So, the next step was to redo the 1993 analysis using the Toda-Yamamoto test. Surprisingly, these results are also very similar to those in Stern (1993). But, when Stephan and Johannes used the data for 1949-1990 that are currently available on US government websites, the Granger causality test of the effect of energy on GDP was no longer statistically significant at the 10% level. Revisions to past GDP have been very extensive, as we show in the paper:

Results were similar when they extended the data to 2015. However, when they allowed for structural breaks in the intercept to account for oil price shocks and the 2008-9 financial crisis, the results were again quite similar to Stern (1993) both for 1949-1990 and for 1949-2015.

They then carried out an extensive robustness check using different control variables and variable specifications and a meta-analysis of those tests to see which factors had the greatest influence on the results.

They conclude that p-values tend to be substantially smaller (test statistics are more significant) if energy use is quality adjusted rather than measured by total joules and if capital is included. Including labor has mixed results. These findings largely support Stern’s (1993) two main conclusions and emphasize the importance of accounting for changes in the energy mix in time series modeling of the energy-GDP relationship and controlling for other factors of production.

I am pretty happy with the outcome of this analysis! Usually it is hard to publish replication studies that confirm the results of previous research. We have just resubmitted the paper to Energy Economics and I am hoping that this mostly confirmatory replication will be published. In this case, the referees added a lot of value to the paper, as they suggested to do the analysis with structural breaks.

What Do Crawford School Economists Do?

I'm doing quite a bit of background work for our School Review, a review of the Future of Asia-Pacific Economics etc. The following table is based on the self-identified "Fields of Research" of core Crawford economics faculty. Most people chose more than one field. If, for example, someone chose three fields, then I attributed 1/3 of an FTE to each for that person. The result looks like this:

Our research foci are economic development and growth, environmental and resource economics, and international economics and finance. The (non-geographical) fields that we rank best in globally in RePEc are: Environment 7, Energy 7, Resources 11, Agriculture 23, Growth 29, International Trade 32, Development 39. So, this focus also is where we perform well.

Most Crawford economists have countries that they focus on. Using a similar approach I put together this table:

Naturally, Australia is number one, then follow China, Japan, Indonesia, and Vietnam. In RePEc, we rank 4th in the SE Asia ranking, 18th in Central/Western Asia (which actually includes South Asia), and 39th in the China subject ranking. This reflects more of our historical focus, while the current faculty is more focused on NE Asia. We don't have any current faculty with a professed interest in Thailand, for example! Of course, there is also less competition in research on SE Asia than on China and so that will also affect our ranking.

Impact Factors for Public Policy Schools

As part of our self-evaluation for the upcoming review of the Crawford School, I have been doing some bibliometric analysis. One thing I have come up with is calculating an impact factor for the School and some comparator institutions. This is easy to do in Scopus. It's the same idea as computing one for an individual or a journal, of course. I am using a 2016, 5 year impact factor. Just get total citations in 2016 to all articles and reviews published in 2011-2015. Divide by the number of articles. Here are the results with 95% confidence intervals:

The main difficulty I had was retrieving articles for some institutions such as the School of Public Affairs at Sciences Po. Very few articles came back for various variants of the name that I tried. I suspect that faculty are using departmental affiliations. I had a similar problem with IPA at LSE. So, I report the whole of LSE in the graph. It is easy to understand this metric in comparison to journal impact factors. As an individual metric the confidence interval will usually be large, though my 2016 impact factor was 5.9 with a 4.2 to 7.5 confidence interval. That's more precise than the estimate for SIPA.

My Submission to Stern Review of the REF

The Stern Review of the REF (Research Excellence Framework) is the latest British government review of research assessment in the UK, following on from the Metric Tide assessment. I have just made a submission to the enquiry. My main comment in response to the first question (1. What changes to existing processes could more efficiently or more accurately assess the outputs, impacts and contexts of research in order to allocate QR? Should the definition of impact be broadened or refined? Is there scope for more or different use of metrics in any areas?) follows:

"I think that there is substantial scope for using bibliometrics in the conduct of the REF. In Australia the Australian Research Council uses metrics to assess natural science disciplines and psychology. Research that I have conducted with my coauthor, Stephan Bruns, shows that this approach could be extended to economics and probably political science and perhaps other social sciences. We have written a working paper presenting our results that is currently under review by Scientometrics.

The paper shows that university rankings in economics based on long-run citation counts can be easily predicted using early citations. The rank correlation between universities' cumulative citations received over ten years for economics articles published in 2003 and 2004 and citations received in 2003 to 2004 alone is 0.91 in the UK and 0.82 in Australia. We compare these citation-based university rankings with the rankings of the 2008 Research Assessment Exercise in the UK and the 2010 Excellence in Research assessment in Australia. Rank correlations are quite strong but there are differences between rankings based on this type of peer review and rankings based on citation counts. However, if assessors are willing to consider citation analysis to assess some disciplines as is the case for the natural sciences and psychology in Australia there seems no reason to not include economics in this set.

Previously, I published a paper, published in PLoS One showing that the predictability of citations at the article level is similar in economics and political science. This supports the view that metrics based research assessment can cover both economics and political science in addition to the natural sciences and economics.

I believe the REF review should seriously consider these findings in producing recommendations for a lighter touch future REF."

I also made briefer responses to some of their other questions. In particular:

5. How might the REF be further refined or used by Government to incentivise constructive and creative behaviours such as promoting interdisciplinary research, collaboration between universities, and/or collaboration between universities and other public or private sector
bodies?

"A major issue with the REF and the ERA in Australia is the pigeon-holing of research into disciplines, which might not match well the nature of the research conducted. This clearly will discourage publication in interdisciplinary venues that may not be as respected by mainstream reviewers. The situation is less acute in Australia where a single output can be allocated across different assessment disciplines, but I still think that assessment by pure disciplinary panels discourages interdisciplinary work in Australia. So, I imagine this is exacerbated in the UK.

7. In your view how does the REF process influence the development of academic disciplines or impact upon other areas of scholarly activity relative to other factors? What changes would create or sustain positive influences in the future?

Johnston et al. (2014) show that the total number of economics students has increased in UK more rapidly than the total number of all students, but the number of departments offering economics degrees has declined, particularly in post-1992 universities. Also, the number of universities submitting to the REF under economics has declined sharply with only 3 post-1992 universities submitting in the latest round. This suggests that the REF has driven a concentration of economics research in the more elite universities in the UK.

Johnston, J., Reeves, A. and Talbot, S. (2014). ‘Has economics become an elite subject for elite UK universities?’ Oxford Review of Education, vol. 40(5), pp. 590-609.

Research Assessment Using Early Citation Information

A new paper with Stephan Bruns on carrying out research assessment like the UK REF and the Australian ERA using citations data rather than peer review. We did a lot of the work of processing the data (doing fancy things with R and manually checking names of universities in Excel) when I visited Stephan in Kassel in November.

The problem with research assessment as carried out in Britain and in the social sciences in Australia is that publications that have already passed through a peer review process are again peer reviewed by the assessment panels. This involves a significant workload for many academics who are supposed to read these papers as well as the effort a each university put into selecting the publications that will be reviewed. However, this second peer review though is inferior to the first. If instead citation based metrics were used the whole process could be done much faster and cheaper. In Australia the natural sciences and psychology are assessed using citation analysis. I think this can be extended to at least some other social sciences including economics.

UK REF panels can also put some weight on citations data in some disciplines including most natural sciences and economics, but only as a positive indicator of academic significance and in very much a secondary role to peer review. This represents a change from the previous RAE, which prohibited the use of citations data by panels. This paper provides additional evidence on the potential effectiveness of citation analysis as a method of research assessment. We hope our results can inform the future development of assessment exercises such as the REF and ERA.

One reason why citations analysis is less accepted in the social sciences than in the natural sciences is the belief that citations accumulate too slowly in most social sciences such as economics to be useful for short-term research assessmen.

My 2014 paper in PLoS ONE shows that long-run citations to articles in economics and political science are fairly predictable from the first few years of citations to those articles. However, research assessment evaluates universities rather than single articles. In this new paper, we show that rank correlations are greatly increased when we aggregate over the economics publications of a university and also when we aggregate publications over time. The rank correlation for UK universities for citations received till the end of 2004 (2005) by economics articles published in 2003 and 2004 with total citations to those articles received through 2014 is 0.91 (0.97). These are high correlations. Correlations for Australia are a bit lower.

Our results here show that at the department or university level citations definitely accumulate fast enough in economics in order to be able to predict longer run citation outcomes of recent publications. It's not true that citations accumulate too slowly in the social sciences to be used in research assessment.

On the other hand, the rank correlation between our early citations indicators and the outcome of research assessment exercises in the UK and Australia ranges from 0.67-0.76. These results suggest that citation analysis is useful for research assessment in economics if the assessor is willing to use cumulative citations as a measure of research strength, though there do appear to be some systematic differences between peer-review based research assessment and our citation analysis, especially in the UK. Part of the difference will emerge due to the differences between the sample of publications we selected to assess and the publications actually selected in the 2010 ERA and 2008 RAE.

How Has Research Assessment Changed the Structure of Academia?

Does measuring something change it?  In quantum mechanics measurement disturbs what is being measured, which is referred to as the observer effect. The same is often true in social systems, especially of course when measurement is attached to rewards. The UK and Australia have been conducting periodical research assessment exercises - the REF and ERA. In the case of the UK, research assessment started almost three decades ago. In Australia, the first research assessment was only conducted in 2010 but the founding of the ARC in 1988 and its independence in 2001 are both milestones in the road to increased emphasis on competition in research in Australia.

Johnston et al. (2014) show that the total number of economics students has increased in UK more rapidly than the total number of all students, but the number of departments offering economics degrees has declined, particularly in post-1992 universities. Also, the number of universities submitting to the REF under economics has declined sharply with only 3 post-1992 universities submitting in the latest round. This suggests that the REF has driven a concentration of economics research in the more elite universities in the UK. BTW the picture above is of the Hotel Russell, which the Russell Group of British universities is named after.

Neri and Rodgers (2014) investigate whether the increased emphasis on research in Australia has had the desired effect in the field of economics. They investigate the output of top economics research by Australian academics from 2001 to 2010. By constructing a unique database of 26,219 publications in 45 top journals, they compare Australia’s output internationally, determine whether Australia’s output increased, and rank Australian universities based on their output. They find that Australia’s output, in absolute and relative terms, and controlling for differences in page size and journal quality, increased and, on a per capita basis, is converging to the levels of the most research-intensive countries. Finally, they find that the historical dominance of the top four universities is diminishing. The correlation between the number of top 45 journal articles published in 2005-2010 and the ERA 2012 ranking is 0.83 (0.78 for 2003-8 and ERA 2010).

References

Johnston, J., Reeves, A. and Talbot, S. (2014). ‘Has economics become an elite subject for elite UK universities?’ Oxford Review of Education, vol. 40(5), pp. 590-609.

Neri, F. and Rodgers, J. (2014). ‘The contribution of Australian academia to the world’s best economics research: 2001 to 2010’, Economic Record.

Peer Review vs. Citation Analysis in Research Assessment Exercises

Existing research finds strong correlations between the rankings produced by UK research assessment exercises (RAE) and bibliometric analyses for several specific humanities and social science disciplines (e.g. Colman et al., 1995; Oppenheim, 1996; Norris and Oppenheim, 2003) including economics (Süssmuth et al., 2006). Clerides et al. (2011) compare the 1996 and 2001 RAE ratings of economics departments with independent rankings from the academic literature. They find RAE ratings to be largely in agreement with the profession’s view of research quality as documented by independent rankings, although the latter appear to be more focused on research quality at the top end of academic achievement. This is because most rankings of departments in the economics literature are based on publications in top journals only, which lower ranked departments have very few of.

Mryglod et al. (2013) analyse the correlations between the values of Thomson Reuters Normalised Citation Impact (NCI) indicator and RAE 2008 peer-review scores in several academic disciplines, from the natural to social sciences and humanities. The NCI computes the normalized impact factor across a unit of assessment (an academic discipline at a given university) in the RAE based on only the publications actually submitted to the RAE. Mryglod et al. (2013) compute both average (or quality) and total (or strength) values (average multiplied by number of staff submitted to the RAE) of these two indicators for each institution. They find very high correlations for the strength indicators for some disciplines and poor correlations at for the quality indicators for all disciplines. This means that, although the citation-based scores could help to describe institution level strength (which is quality times size), in particular for the so-called hard sciences, they should not be used as a proxy for ranking or comparison of research groups. Moreover, the correlation between peer-evaluated and citation-based scores is weaker for the “soft” sciences. Spearman rank correlation coefficients for their quality indicators range from 0.18 (mechanical engineering) to 0.62 (chemistry). However for strength the correlations range from 0.88 (history and sociology) to 0.97 (biology). This is because quality is correlated with size and so the two factors reinforce each other.

Mryglod et al. (2014) attempt to predict the 2008 RAE retrospectively and the 2014 Research Excellence Framework (REF) before its results were released. They examined biology, chemistry, physics, and sociology. Of the indicators they trialled, they found that the departmental h-index had the best fit to the 2008 results. Departmental h-index is based on all publications published by a department in the time window assessed by the relevant assessment exercise. The rank correlation ranged from 0.83 in chemistry to 0.58 in sociology. They find that the correlation with the RAE for the immediate h-index is as good as the correlation in later years with the h-index of the same set of publications.

Bornmann and Leydesdorff (2014) argue that one of the downsides of bibliometrics as a research assessment instrument is that citations take time to accumulate while research assessment exercises are designed to assess recent performance:

“This disadvantage of bibliometrics is chiefly a problem with the evaluation of institutions where the research performance of recent years is generally assessed, about which bibliometrics—the measurement of impact based on citations—can say little…. the standard practice of using a citation window of only 3 years nevertheless seems to be too small.” (1230)

They argue further that bibliometrics:

“can be applied well in the natural sciences, but its application to TSH (technical and social sciences and humanities) is limited.” (1231)

But rather than assuming that peer review is the preferred approach to research assessment and citation analysis should only be used to reduce cost, we can ask whether the review conducted by research assessments such as the REF and the Australian ERA meets the normal academic standards for peer review. Research does show that peer review at journals has predictive validity for the citations that will be received by accepted papers compared to those received by rejected papers. However, evidence for the predictive validity of peer review of grant and fellowship applications is more mixed (Bornmann, 2011). Therefore, further research is warranted on the use of citation analysis to rank academic departments or universities in research assessment exercises. Sayer (2014) argues that the peer review undertaken in research assessment exercises does not meet normal standards for peer review. He compares university and national-level REF processes against actual practices of scholarly review as found in academic journals, university presses, and North American tenure procedures. He finds that the peer review process used by the REF falls far short of the level of scrutiny or accuracy of these more familiar peer review processes. The number of items each reviewer has to assess alone means that the review cannot be of the same quality as reviews for publication. And reviewers will have to assess much material outside their area of specific expertise. Sayer argues that though metrics may have problems, a process that gives such extraordinary gatekeeping power to individual panel members is far worse.

Given the large number of items that panels need to review they are likely to focus on the venue of publication and at least in business and economics handy mappings of journals to REF grades exist (Hudson, 2013). Regibeau and Rockett (2014) build imaginary economics departments entirely composed of Nobel Prize winners and evaluate them using standard journal rankings geared to the UK RAE. Performing the same evaluation on existing departments, they find that the rating of the Nobel Prize departments does not stand out from other good departments. Compared to recent research evaluations, the Nobel Prize departments’ rankings are less stable. This suggests a significant effect of score “targeting” induced by the rankings exercise. They find some evidence that modifying the assessment criteria to increase the total number of publications considered can help distinguish the top. But if departments composed entirely of Nobel Prize winners perform worse than current departments then it is hard to know what such assessment means.

Sgroi and Oswald (2013) examine how research assessment panels could most effectively use citation data to replace peer review. They suggest a Bayesian approach that uses prior information on where a item was published combined with observations on citations to derive a posterior distribution for the quality of the paper. We could then estimate, for example, what is the probability that a paper belongs in the 4* category given where it was published and the early citations it has received. Stern (2014) and Levitt and Thelwall (2011) show that the journal impact factor has strong explanatory power in the year of publication but that this declines very quickly as citations accumulate. So, this approach would be most useful for papers published in the last year or two before the assessment, but for earlier research outputs the added value over simply counting citations would be minimal.

References

Bornmann, L. (2011) ‘Scientific peer review’, Annual Review of Information Science and Technology, vol. 45, pp. 199‐245.

Bornmann, L. and Leydesdorff, L. (2014). ‘Scientometrics in a changing research landscape’, EMBO Reports, vol. 15(12), pp. 1228–32.

Clerides, S., Pashardes, P. and Polycarpou, A. (2011) ‘Peer review vs metric-based assessment: testing for bias in the RAE ratings of UK economics departments’, Economica, vol. 78(311), pp. 565–83.

Colman, A. M., Dhillon, D. and Coulthard, B. (1995) ‘A bibliometric evaluation of the research performance of British university politics departments: Publications in leading journals’, Scientometrics vol. 32(1), pp. 49-66.

Hudson, J. (2013). ‘Ranking journals’, Economic Journal, vol. 123, pp. F202-22.

Levitt, J.M. and Thelwall, M. (2011). ‘A combined bibliometric indicator to predict article impact’, Information Processing and Management, vol. 47, pp. 300–8.

Mryglod, O., Kenna, R., Holovatch, Y. and Berche, B. (2013). ‘Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence’, Scientometrics, vol.97, pp. 767–77.

Mryglod, O., Kenna, R., Holovatch, Y. and Berche, B. (2014). Predicting Results of the Research Excellence Framework Using Departmental H-Index, arXiv:1411.1996v1.

Norris, M. and Oppenheim, C. (2003) ‘Citation counts and the research assessment exercise V: Archaeology and the 2001 RAE’, Journal of Documentation, vol. 59(6): pp. 709-30.

Oppenheim, C. (1996) ‘Do citations count? Citation indexing and the research assessment exercise’, Serials, vol. 9, pp. 155–61.

Regibeau, P. and Rockett, K.E. (2014). ‘A tale of two metrics: Research assessment vs. recognized excellence’, University of Essex, Department of Economics, Discussion Paper Series 757.

Sayer, D. (2014). Rank Hypocrisies: The Insult of the REF. Sage.

Sgroi, D. and Oswald, A.J. (2013). ‘How should peer-review panels behave?’ Economic Journal, vol. 123, pp. F255–78.

Stern, D.I. (2014). ‘High-ranked social science journal articles can be identified from early citation information’, PLoS ONE, vol. 9(11), art. e112520. 

Süssmuth, B., Steininger, M. and Ghio, S. (2006) 'Towards a European economics of economics: Monitoring a decade of top research and providing some explanation', Scientometrics, vol. 66(3), pp. 579-612.

Workshop on Research Metrics

I submitted one of the 152 responses to HEFCE's call for evidence on research metrics. BTW, my paper was just accepted by PLoS ONE :) Anyway, HEFCE and the University of Sussex are holding a one day workshop at SPRU on the potential for metrics in research assessment. Register here and see the program here. As I am in Australia, unfortunately I can't attend - I'm actually going to be in England in late October and early November - but maybe some of you can.

High-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information

I have posted a new bibliometric working paper , which investigates how well we can predict future cumulative citations from the first citations received by a paper in the disciplines of economics and political science.

It is usually assumed that citations accumulate too slowly in social sciences apart from psychology to be useful for short-term research assessment. For this reason, the Australian Government’s Excellence in Research for Australia (ERA) exercise, which attempts to assess the research quality of universities in the previous 5 years, uses peer review in social science disciplines apart from psychology for this reason but uses citation analysis for psychology and all natural sciences. This peer review process seems to me to be a wasteful duplication of effort to review research outputs that have already passed through a peer review process once.

I show that, surprisingly, citations received by journal articles in the social sciences in the first one to two years after publication are strongly predictive for citations received in future years. By contrast, I show that journal impact factors are mostly useful in the year of publication and their contribution to predicting citations declines rapidly thereafter.

If it is actually possible to predict citations fairly reliably in social science disciplines, then it should also be easy to predict them in the natural sciences. This means that it should be possible to expand bibliometric analysis in research evaluation exercises to all disciplines apart from the humanities and arts. It also means that we should pay attention to the early citations received by papers when we evaluate individual academics for hiring and promotion. Impact factors are reflective of journal selectivity, which we frequently do not have easily available data on. But they only explain about 16-17% of the variation in rankings of papers six years later conditional on the citations already received in the year of publication. The latter explain 13-14% of the variation. But at the end of the year following publication, accumulated citations explain 52-53% of the variation in cumulative citations after 6 years and 73% at the end of the second year after publication.

These models could be improved by adding information on the characteristics of the articles themselves and their authors, but that was much too time consuming to do for the almost 12,000 articles in my sample.

I have submitted a copy of my paper to the HEFCE inquiry on the use of metrics in research assessment.

Draft ERA 2015 Submission Guidelines Released

This document provides a helpful list of changes from ERA 2012 to ERA 2015. The changes are minor. A good one, is a clampdown on abuse of affiliated researchers - some universities submitted a lot of publications by foreign-based researchers that they claimed as affiliates that did not include any mention of the university submitting the publication to ERA. So it looks like that for the moment the aim of measuring the broader impact of research as the UK REF is attempting to do has been shelved. At least as part of the ERA exercise.

Evolution of the UK REF

Interesting article on how the UK's Research Excellence Framework evolved. An interesting comment is: "Per pound distributed, the RAE and REF are vastly cheaper than distributing the same sums via grant applications to the research councils" Very significant funding is tied to the results of the REF. By contrast, the Australian system is primarily driven by winning grants and then getting overheads based on them sent to institutions by the government a couple of years later with a smoothing process over time. PhD completions also feature strongly. So far, the ERA is only linked to a very small allocation of funding.

Australian Research Assessment Not Heading in UK Direction

That's the message I get from this interview with the head of the ARC, Aidan Byrne (formerly a science dean at ANU) in the Australian. The assumption of many in the sector, myself included is that the Australian research assessment exercise, ERA, and the funding attached to it would evolve in a way that generally followed UK practice with something of a time lag. In the UK, much more money is tied to the REF, the funding ratio associated with the three highest rankings of departments is 9:3:1, and case studies are being used very heavily to assess broader impact. Prof. Byrne argues that case studies should be used sparingly if at all to measure impact, not much money should be tied to ERA outcomes, and the funding ratio should be flatter. On Wednesday, I saw a presentation by Tim Cahill of the ARC on the ERA 2012 process and outcomes. One key finding was that for the citation based disciplines (most STEM disciplines (but not math or computer science) and psychology) there is a weak correlation between the ERA ranks assigned to universities and their citation performance relative to the benchmarks. A lot of subjectivity still seems to come into the ranking by the ERA committees. As they only count the number of citations per paper and not where they were cited, I guess that makes sense. So should Australian universities pay as much attention to ERA as they have been doing? For example, ANU has tied indicators in it strategic plan to the number of disciplines that achieve given ERA rankings by 2020. If I was the minister and looking for budget cuts would I want to continue with ERA on this basis?

ANU is Number One in the World...

On mean kilometres between collaborators on papers! On average our collaborators are 4,522km from us:

This is just one the rankings you can get from the Leiden Ranking of universities. They aim to be more transparent than other well-known rankings like the Shanghai Jiaotong or QS rankings. Another very interesting feature are confidence intervals obtained by boot-strapping.

ANU does not rank very well by more conventional measures of impact. The authors preferred ranking is by the PP top 10% indicator. This is the percentage of a university's publications that are in the top 10% most cited publications. It's not source normalized, however, so universities with strong biomedical research efforts will rank higher. ANU is ranked only 114th with 12.9%. But note that even the top school - MIT - only scores 25.2%. So there is not much variation across most of the 500 research universities.

Another ranking is by source normalized impact factor. This should take into account the differences between disciplines. ANU ranks 118th with an impact of 1.21. MIT has an impact of 2.17. So there is again little variation. Much less variation than among journals which have less variation than individual researchers, which have less variation than articles. I'm actually surprised how little variation there is across universities. Of course, all these are research oriented universities, but still.

Something missing from these ranking is the average number of articles per researcher. It's not possible to work this out from the Web Science in any accurate way. Based on the RePEc rankings Australian economists publish above average numbers of papers given their RePEc rank, but these are cited less. I suspect because of the distance of Australia from elsewhere, despite our high levels of international collaboration.

The Inside Story on the 2010 ERA Economics Journals Rankings

Two Monash economists have written a working paper on bias in the selection of the economics journal ranks used in the ERA research assessment exercise in Australia. The selectors were all the full professors in economics in Australia identified by the Economic Society of Australia (ESA). They tended to vote for journals they had published in etc... In fact the rankings of economics journals (since abolished) were not that bad. There were a few glaring anomalies but these weren't the responsibility of the ESA. We are still using these rankings in practice. It's a meme that has stuck at least in economics.

How Does the Australian Government Use the ERA Results?

Excellence in Research for Australia (ERA) is the Australian research assessment exercise that is very similar to the British REF. We are currently working on our submission to the 2012 ERA assessment, which is the second assessment following ERA 2010. There has been a lot of discussion of the assessment methods and results but few people seem to know about how these results are being used.

So far the ERA results will be used as part of the method of allocating money in the SRE scheme. Currently SRE is around $150 million of the $1.6 billion of research block grants paid to Australian universities. This number will rise to more than $300 million over the next year. The program is supposed to provide universities with overhead or indirect costs of funded research. Unlike in the US, the Australian Research Council (ARC) and NHMRC doesn't pay universities any overhead payments on grants. These come later through these block grant schemes.

The government released a consultation paper on the method to be used in 2012. Basically, money would be allocated according to the amount of competitive research grants each university earned weighted by some ERA based indicator. They haven't yet released the exact algorithm that will be used.

So by next year about 20% of research block grants will be weighted by ERA derived measures. I expect though that ERA will be used to allocate more of the money in the future.

ERA Consultation Part II

I previously submitted comments on the ARC's ranked journal list for the 2012 ERA research assessment exercise. You can also submit more general comments on the ERA process. I only submitted one comment, suggesting that economics should be evaluated using citation analysis. Psychology was the only social science evaluated using citations in the 2010 ERA. All natural science, medical, and mathematical fields were evaluated using citation analysis. Citation analysis is widely used in economics, not least by RePEc, and I think the ARC should definitely think again about applying it to economics. One advantage is that it will save a lot of effort on the part of external reviewers who had to provide peer assessment for the 2010 ERA.

ERA Ranked Journal List is Out

The ARC (Australian Research Council) has finally released the ERA (Excellence in Research for Australia) ranked journal list that will be used in this year's research assessment exercise. As I've commented that the assessment of economics will not count citations and so the ranking of journals is all that counts in measuring research outputs. All journals are ranked as A*, A, B, and C. A* is best. There are some anomalies - journals ranked surprisingly low and some journals ranked surprisingly highly. For energy and environmental economists, key issues are that Environment and Development Economics and Review of Environmental Economics and Policy are both ranked at only B. Among interdisciplinary journals, Energy Policy, Annals of the New York Academy of Sciences (who publish Ecological Economics Reviews), Journal of Environmental Management, and Ambio are only ranked B. It's not surprising that Journal of Environment and Development is ranked B as it isn't in the ISI database.

Most of my articles have been published in A journals with four A* publications in Nature, Journal of Geophysical Research, and Trends in Parasitology and B publications in JED, EDE, Energy Policy, Policy Studies Journal, Applied Economics Letters, Professional Geographer (this is another anomaly), and Geoforum.

And I just noticed that Journal of Economic Issues doesn't even appear to be on the ERA list! If you are looking for The Energy Journal, it is classified under Resources Engineering and Extractive Metallurgy. This is despite it being published by the International Association for Energy Economics.

This is already influencing our decision about where to send the paper that I presented here in Adelaide.

P.S.
I've found the Journal of Economic Issues. It's listed as JEI and ranked C. So I have published one paper in a C ranked journal. As it is an ISI journal I would think B is more appropriate.

ERA 2010

Detailed information is now available for the ERA (Excellence in Research for Australia) 2010 exercise to be carried out by the ARC (Australian Research Council). This is similar to the research quality assessment exercises carried out in the UK. The only social science where they will use citation analysis is psychology. I can see no reason not to use citation analysis in economics except for the field of economic history where books are important and citations in journals are low. The main tools, therefore, will be peer review of nominated work (20% of your outputs) and assessment of the quality of journals in which researchers have published. The list of ranked journals for economics is not yet available it seems.