Monday, September 28, 2015

Influential Publications in Ecological Economics Revisited

Back in 2004 I published with Bob Costanza, Chunbo Ma, Brendan Fisher, and Lining He a paper that looked at the most influential publications on the field of ecological economics. Then, about this time last year Gaël Plumecocq contacted me about participating in a session he was organizing at the European Society of Ecological Economics meeting in Leeds on "ecological economics understood as an epistemic community". I had the idea of revisiting our 2004 paper a decade later and seeing how the field had changed in the meantime. Eventually, Gaël also came on board our author team, contributing a textual analysis of the key themes in the influential papers. Gaël gave a presentation on the paper at ESEE and now we finally have a working paper version of our new paper on the web. The full author team includes: Bob Costanza, Rich Howarth, Ida Kubiszewski, Shuang Liu, Chunbo Ma, Gaël, and myself.

We downloaded from the Web of Science (WoS) information on all the papers published in Ecological Economics from 2004 to 2014 including the number of citations each received and the full reference list from all 2960 articles. We define outwardly influential papers as the 10% of articles published in the journal in each year from 2004 to 2014 that received the most citations in the Web of Science. The inwardly influential publications are all publications that received more than 15 citations in the journal in the period 2004-2014. For each of these publications we collected the total number of citations in the Web of Science, Google Scholar, and Ecological Economics. The inward influence data needed a lot of cleaning up, which was mainly done by Chunbo and Ida with my help.

Shuang produced these graphs of inward and outward influence using Tableau:

For inward influence, there are many publications at the bottom right that have been relatively much more influential across science as a whole than they have been in ecological economics. Publications towards the left have been mostly influential in ecological economics alone. By contrast, there is a stronger correlation between citations received in the journal and more broadly for the outwardly influential articles. One outlier is the Pimentel et al. paper on the costs of invasive species that is the most cited article (by WoS cites) ever published in ecological economics.

The theme analysis found, as we expected, that the most influential topic was ecosystem services and payments for ecosystem services, which received 25% of the citations of the influential publications. By contrast, sustainable development and foundations of ecological economics were the most influential topics prior to 2004.

We also followed up on my 2006 paper with Chunbo Ma by looking at the journals, which cite Ecological Economics the most and which are cited by Ecological Economics the most:

There have been quite dramatic changes in these lists with more than half the journals being new entrants. In general there has been an increase in citation links to interdisciplinary environmental science and environmental studies journals and a reduction in links to mainstream economics journals including environmental economics journals. No general interest economics journals are now on the top 20 inward and outward lists.

We think that these trends reflect a maturation of ecological economics as a transdisciplinary field.

Wednesday, September 23, 2015

Nature Energy

I have joined the editorial advisory board for Nature Energy, a new journal from the Nature Group that will begin publishing at the beginning of 2016. The position involves giving feedback on submitted manuscripts, but I can also make suggestions for commentary and news and views articles. The latter are a special feature of Nature journals that provide less technical overviews of articles published both in Nature journals and other academic journals. I would appreciate any suggestions you might have for upcoming articles that the journal might cover in news and views, or topics/authors for commentaries.

Thursday, August 27, 2015

Global Energy Use: Decoupling or Convergence Accepted to be Published in Energy Economics

My paper with Zsuzsanna Csereklyei on trends in global energy use and whether they can be better explained by decoupling or convergence in energy intensity or some mixture has been accepted to be published in Energy Economics. I wrote a blogpost about the paper in December last year when we first completed the paper.

This was not a bad review experience though Zsuzsanna who is in a short-term post-doc position would have liked it to be faster! We first sent the paper to World Development who desk-rejected it. Then we sent it to Energy Economics. The first review took seven months due to one of the referees dropping out due to family health problems and then the journal needed to get another referee. We turned around the paper in 12 days and then got a conditional acceptance a month later. We turned that around in one day and got the final acceptance a day later. This paper applies the method developed in Anjum et al. (2014) to energy and also adds treatment of spatial autocorrelation. The whole process of writing and publishing this paper took place while Anjum et al. has been in review...

Monday, August 24, 2015

Having Small Government Should Never Be a Reason for Making It Bigger

It seems that the Grattan Institute argues in its submission to the National Reform Summit that because Australia's government is relatively small by the standards of other developed countries we can or should increase its size. This argument makes no sense to me, though it seems to be often made. Perhaps other countries spend inefficiently or perhaps Australians are not interested in spending on some of the things that those countries spend on. We should look first at the outcomes we have as a country and, if there are poor outcomes that Australians would like improved, we then need to ask whether government can make a difference. Only then does it make sense to ask whether spending or taxes should be higher. I don't think this is really a political statement, as I leave open the final choice on whether to increase the size of government or not. But it makes no sense to talk about increasing the size of government simply to be nearer OECD means.

As for whether we should change the taxation arrangements for superannuation the simplest, fairest, and possibly relatively efficient  approach is to use the same approach as US 401k and 403b funds and tax payouts at normal income tax rates but not tax contributions or earnings in the accumulation stage. It's simple to show that under reasonable assumptions that this approach generates higher retirement incomes than the current Australian approach. It also has the huge advantage of reducing bureaucracy. Self managed superannuation funds are costly to run in Australia because of the need for accounting and auditing to make sure that they are paying the correct taxes. In the US, an IRA is just like another brokerage account except for the rules on contributions and withdrawals, which can be managed by the broker.

Sunday, August 9, 2015

The Extent and Consequences of P-Hacking in Science

Interesting paper from my biology colleagues at ANU on the effects of "p-hacking" - searching for more significant results by looking at various statistical models or samples and picking the more significant ones to report - on reported science. They conclude that when there is a strong real effect it can be detected despite p-hacking by looking at the "p-curve". The p-curve is the distribution of p-values across all the studies collected in a meta-analysis. If the curve is skewed right - there is a peak at very high significance levels (numbers a lot smaller than 5%) then there is a real effect. However, p-hacking can inflate the estimated size of the effect if we use a simple average of effect sizes in the literature. The main novelty of their paper I think is that they collected a large number of p-values from various fields of science using text-mining to test these ideas in the empirical literature.

In meta-analysis in economics, a popular approach is to test the effect of degrees of freedom or precision (inverse of the standard error) on the values of the reported test statistics using regression analysis. This effect is called the power-trace. The idea is that if there is a true effect, then, due to increasing statistical power, reported test statistics will be more significant the higher the degrees of freedom in the underlying study.* Some of these methods can also be used to estimate the true effect size adjusted for publication bias.

In our meta-analysis of energy-GDP Granger causality tests we also present graphs of the distribution of the test-statistics. These seemed to be roughly normal with a mean of about 1, which means there is excess significance in this literature but that the mean test statistic is not statistically significant (the solid histogram in the background is the standard normal distribution):

To help interpret these graphs, note that a normal test statistic (-probit(p)) of zero means that the original Granger causality test p-value was 0.5. A test statistic of 1.65 implies that the original p-value was 0.05 and a test statistic of -1.65 implies that the p-value was 0.95. The econometric analysis in the paper showed that there was no statistically significant relationship between these test statistics and degrees of freedom, also suggesting that there was no genuine effect. We showed in the paper that there did seem to be a robust effect from GDP to energy when underlying studies controlled for energy prices.

We didn't report the actual p-values though, and so I am curious what the p-curves look like. First I made a couple of histograms with bins for each 1% increment of p-values:

Uh-oh! The mode is for 0-1%! According to Head et al.'s methodology that means there is a true effect in each direction of causality. When I broke down the range from p=0 to p=0.1 into 100 bins, again the mode was for the smallest value. So, what does it mean when the overwhelming majority of studies find results that are less significant than the 1% or 0.1% level and yet the mode is for 0-1% or 0-0.1%? And when these results are for not particularly large sample sizes? Either the p-curve or the meta-regression/power trace method is wrong here. One hypothesis is that non-stationarity in macro-economic time series and the over-fitting problem discussed in our paper result in many spuriously significant test statistics in relatively small samples that wouldn't arise with more classically behaved data.

* Though this method can detect a "genuine effect" there is no guarantee that this is a "causal effect". If no studies control for the relevant variables or effects to identify a causal effect then the meta-analyst won't be able to detect a causal effect either. Similarly, if the meta-analyst doesn't control for all the relevant variables included in the underlying studies they may also fail to identify a causal effect when some papers do identify one. All the meta-analyst can find is a robust partial correlation in the underlying studies if one exists.

Saturday, August 8, 2015

Donglan Zha

Donglan Zha is visiting Crawford School for the next year. She is an associate professor at Nanjing University of Aeronautics & Astronautics and works on energy economics including research on substitution possibilities and the rebound effect. Her office is in Constable's Cottage across the road from the main Crawford Building. Her ANU e-mail address is Please welcome Donglan to the Crawford School, I'm sure she will be happy to meet with you.

Monday, August 3, 2015

International Energy and Poverty: The emerging contours

New book that will be released this month: International Energy and Poverty: The emerging contours. I am the authors of the introductory chapter on energy and growth. There will be a book launch at University of Denver on 11 September at 9:30am as part of the "Access to Energy for All" event. Contact Lakshman Guruswamy for details if you want to attend.

Thursday, July 30, 2015

Scopus Adds More Article Level Metrics

Scopus has added a new set of article level metrics. I think the most interesting one is "Field-Weighted Citation Impact" which tells you how cited your article is relative to other similar articles. I think this metric has a big potential in tenure and promotion cases. Here is Scopus' explanation:

Field-weighted Citation Impact (FWCI) 

Field-Weighted Citation Impact is sourced directly from SciVal.

As defined in Snowball Metrics, Recipe Book/Field-Weighted Citation Impact Field-Weighted Citation Impact is the ratio of the total citations actually received by the denominator’s output, and the total citations that would be expected based on the average of the subject field. A Field-Weighted Citation Impact of:
  • *Exactly 1* means that the output performs just as expected for the global average.
  • More *than 1* means that the output is more cited than expected according to the global average. For example, 1.48 means 48% more cited than expected.
  • Less than 1 means that the output is cited less than expected according to the global average.
Field-Weighted Citation Impact takes into account the differences in research behaviour across disciplines. It is particularly useful for a denominator that combines a number of different fields, although it can be applied to any denominator.
  • Researchers working in fields such as medicine and biochemistry typically produce more output with more co-authors and longer reference lists than researchers working in fields such as mathematics and education; this is a reflection of research culture, and not performance.
  • In a denominator comprising multiple disciplines, the effects of outputs in medicine and biochemistry dominate the effects of those in mathematics and education.
  • This means that using non-weighted metrics, an institution that is focused on medicine will appear to perform better than an institution that specialises in social sciences.
  • The methodology of Field-Weighted Citation Impact accounts for these disciplinary differences.

Sunday, July 12, 2015

Increasing Requirements for Publication

From "Accelerating Scientific Publication in Biology":

Somewhat tongue – in - cheek, let’s imagine a contemporary editorial decision on the 1953 Watson and Crick papers (assuming that they were submitted together):

“Dear Jim and Francis: Your two papers have now been seen by three referees. Based upon these reviews, I regret to say that we cannot offer publication at this time. While your model is very appealing, referee 3 finds that it is somewhat speculative and premature for publication. Indeed, your model proposing a semi-conservative replication of DNA raises many obvious questions. As two of the referees point out, it should be possible to determine experimentally if the two strands can separate and serve as templates. This would address referee 3’s concern that strand separation is not feasible thermodynamically. I regret to say that without such experimental evidence, we will not be able to publish your work in Nature and suggest publication in a more specialized journal. Should you be able to furnish more direct experimental evidence, we would be willing to reconsider such a revised paper. Naturally we would need to consult our referees once again. Furthermore, since space in our journal is at a premium, if you do decide to resubmit, then we recommend that you combine your two submitted papers into a single and more cohesive Article, potentially including the X-ray studies of your colleagues at Cambridge. Thank you again for submitting your papers to Nature. I am sure that this revision will delay your Nobel Prize and the discovery of the genetic code by only one or two years."

Saturday, July 11, 2015

Papers from Google Scholar

One way that I keep up to date is to track the papers that cite me using Google Scholar alerts. This time I thought some of the papers were more interesting than usual, particularly the economic history papers. Well it's one way to produce a quick blogpost :)

Y Ren, D Parker, G Ren, R Dunn - Climate Dynamics, 2015
Abstract The spatial and temporal pattern of sub-daily temperature change in mainland
China was analysed for the period from 1973 to 2011 using a 3-hourly dataset based on 408
stations. The increase in surface air temperature was more significant by night between ...

H Nielsen
Abstract This paper examines the role of foreign trade in the consumption of primary energy
in the Czech Republic and to what extent adjustment for energy embodied in trade effects
the country's energy intensity curve. As opposed to previous studies, this article takes a ...

G Esenduran, E Kemahlıoglu-Ziya, JM Swaminathan
ABSTRACT In the last two decades, many countries have enacted product take-back
legislation that holds manufacturers responsible for the collection and environmentally
sound treatment of end-of-use products. In an industry regulated by such legislation, we ...

R Hölsgens, B Gales, JP Smits, F Notten
In this paper we analyze recent estimates of annual CO2 (carbon dioxide) emissions from
energy consumption in the Netherlands since 1800 alongside another emission to air
resulting from energy consumption: SO2 (sulfur dioxide). The new time series on CO2 can ...

E Ömer, M BAYRAK - Anemon Muş Alparslan Üniversitesi Sosyal Bilimler …, 2015
Özet Enerji; kullanım şekli, miktarı, bileşimi, yapısı ve mahiyetiyle ekonomik ve sosyal
gelişmişliğin temel ölçütlerinden biridir. Bir ülkede mevcut enerji arzının enerji talebini
karşılayamadığı durum olarak tanımlanan enerji açığı; büyüme ve kalkınma sürecinde, ...

R Hölsgens, C Ducoing, M Rubio, B Gales
Abstract The relationship between energy and capital is one of the most important
relationships of modern economic growth. Machines need energy to produce all the goods
we enjoy; energy without machinery is useless. However, the great majority of the ...

Z Guevaraa, JFD Rodriguesc, T Domingosb
Abstract Conventional energy input-output models were developed about 40 years ago and
have not been significantly improved since. These conventional models offer a limited
description of energy flows in the economy. This paper introduces a novel energy input- ...

M Amoah, O Marfo, M Ohene - Forests, Trees and Livelihoods, 2015
Firewood is the dominant fuel type used by rural households in Ghana. However, the
scarcity of firewood species has raised concerns about the sustainable use of this fuel type.
This study investigated the firewood consumption pattern, firewood species used by rural ...

B Deng, Y Li
Abstract: Efficiency Power Plant (EPP) promotes the use of energy-efficiency power plant
technology and energy efficient equipment, coupled with its low-input, zero pollution, zero
emissions and other advantages, has an important role in the control of energy ...

JD Urrutia, MLT Olfindo, R Tampis
Abstract: The researchers aim to formulate a mathematical model to forecast Exchange Rate of the Philippines from the 1st Quarter of 2015 up to the 4th Quarter of 2020 using
Autoregressive integrated Moving Average (ARIMA). The researchers used the data ...

Monday, July 6, 2015

Energy Leapfrogging (or Not)

Arthur van Benthem has a recent paper in the Journal of the Association of Environmental and Resource Economists titled "Energy Leapfrogging". The main thesis of the paper is that despite presumed improvements in the energy efficiency of individual technologies such as cars and refrigerators, energy intensity in developing countries today is similar to what it was in today's developed countries when they were at a similar income level. There is no "energy leapfrogging". This is also an implication of our paper "Energy and Economic Growth: the Stylized Facts". If there has been an almost constant log-linear relationship between energy use and GDP per capita then there is no energy leapfrogging.

van Benthem suggests that a major contributor to this is that the consumption bundle in developing countries today is much richer in energy services like personal transport than was the consumption bundle at a similar level of development in today's developed countries. Consumers have substituted towards these now cheaper energy services (what are they consuming less of though?).

On the face of it, this suggests that there would be a very large rebound effect due to substitution towards energy services. This is on top of any indirect rebound effect due to increased energy productivity boosting income and thus energy demand as originally proposed by Harry Saunders.

On the other hand, there must be some shift away from energy services as income increases so that energy intensity is lower in richer countries. Anyway, this is pretty speculative but worth thinking about, I think.

Thursday, June 25, 2015

Changes at ANU

Yesterday we heard the surprising news that Brian Schmidt would be the next vice-chancellor (president) of the ANU. Here at Crawford there is change too with the recent announcement that Tom Kompas would step down as School director after five years. We will be searching for a new school director and in the interim Bob Breunig will be acting director. That means that director of the International and Development Economics Program became vacant. I have agreed to be acting director of the program while Bob is directing the School (till 15 July 2016). There is more to the program that just the Masters of International and Development Economics. We also have a Masters of Environmental and Resource Economics. Effectively, though, it is the department of economics located at the Crawford Building. Crawford also has another department of economics - the Arndt-Corden Department of Economics - located in the Coombs Building. I was based there in 2009-2010. Other changes at Crawford is that Frank Jotzo is becoming deputy director of the School and John McCarthy is replacing him as READ director. I am leaving the READ group.

Monday, June 22, 2015

Population, Economic Growth and Regional Environmental Inefficiency: Evidence from U.S. States

I have a new paper in the Journal of Cleaner Production coauthored with George Halkos and Nikalaos Tzeremes. George was a lecturer at University of York when I was a post-doc there. We haven't previously put out a working paper version of this paper.

In this paper, we apply a conditional directional distance function allowing multiple exogenous factors to measure environmental performance to evaluating the air pollution performance levels of U.S. states for the years 1998 and 2008. The overall results reveal that there is much variation in environmental inefficiencies among the U.S. states. A second stage nonparametric analysis indicates a nonlinear relationship between states’ population size, GDP per capita levels and states’ environmental inefficiency levels.

Our results indicate that environmental inefficiency on the whole decreases with increased population and income per capita but there are limits to this improvement and at high income and population levels the tendency may reverse. In particular, small poor states tend to be environmentally inefficient, whereas large states tend to be more efficient regardless of their level of income. The results show that there is not so much of a trade off between environmental quality and economic development in small and poor US states in the South and Mid-West. As these states grow in income and population they can improve their environmental efficiency. However, large and richer states face more environmental challenges from growth. This may explain the differences in policy across states. For example, California which is already an environmentally efficient state is also a state which has lead in environmental regulation. There are fewer local environmental policies in states across the South and parts of the mid-West. Politicians and populations in these states may see less trade off between environmental quality and development and hence be reluctant to adopt specific environmental policies. These patterns also match recent trends in voting for the Republican and Democratic parties the so-called Blue and Red States. However, there are exceptions to a simplistic analysis along these lines as Texas for example is an environmentally efficient state in our analysis as would be expected from its large population size.

Wednesday, June 17, 2015

Meta-Granger Causality Testing

I have a new working paper with Stephan Bruns on the meta-analysis of Granger causality test statistics. This is a methodological paper that follows up on our meta-analysis of the energy-GDP Granger causality literature that was published in the Energy Journal last year. There are several biases in the published literature on the energy-output relationship, which we document in the Energy Journal paper:

1. Publication bias due to sampling variability - the common tendency for statistically significant results to be preferentially published. This is either because journals reject papers that don't find anything significant or more likely because authors don't bother submitting papers without significant results. So they either scrap studies that don't find anything significant or data-mine until they do. This means that the published literature may over-represent the frequency of statistically significant tests. This is likely to be a problem in many areas of economics, but especially in a field where results are all about test statistics and not about effect sizes.

2. Omitted variables bias - Granger causality tests are very susceptible to omitted variables bias. For example, energy use might seem to cause output in a bivariate Granger causality test because it is highly correlated with capital. This is a very serious problem in the actual empirical Granger causality literature, which I noted in my PhD dissertation.

3. Over-fitting/over-rejection bias - In small samples, there is a tendency for vector autoregression model fitting procedures to select more lags of the variables than the true underlying data generation process has. There is also a tendency to over-reject the null hypothesis of no causality in these over-fitted models. This means that a lot of Granger causality results from small sample studies are spurious. We realized in our Energy Journal paper that this was also a serious problem in the empirical Granger causality literature. The following graph illustrates this using studies from the energy-output causality literature:

Each graph shows normalized test statistics for causality in one of the two directions. Rather than fit models with more lags in larger samples, researchers tend to deplete the degrees of freedom by adding more lags. Therefore, there tend to be fewer degrees of freedom for studies with three lags than with two, and fewer for those with two than with one. Also, we see that the average significance level increases as the lags increase and degrees of freedom reduces.

Of course, the second two types of biases give researchers additional opportunities to select statistically significant results for publication and so, more generally, "publication bias" includes selection of statistically significant results from those provided by sampling variability and by various biases.

The standard meta-regression model used in economics deals with the first of the three biases by exploiting the idea that if there is a genuine effect then studies with larger samples should have more statistically significant test statistics than smaller studies. If there is no real effect then there will be either no relation or even a negative relation between significance and sample size. Meta-analysis can test for the effects of omitted variables bias by including dummy variables and interaction terms for the different variables included in primary studies. Finally, in our Energy Journal paper we controlled for the over-fitting/over-rejection bias by including the number of degrees of freedom lost in model fitting in our meta-regression.

The new paper focuses on the latter issue and examines both the potential prevalence of over-fitting and over-rejection and the effectiveness of controlling for over-fitting. The approach used in this paper is a little different to the Energy Journal paper - here we include the number of lags selected as the control variable. We show by means of Monte Carlo simulations that, even if the primary literature is dominated by false-positive findings of Granger causality, the meta-regression model correctly identifies the absence of genuine Granger causality. The following graphs show the key results:

Power is the probability of reject the null hypothesis of no causality when it is incorrect - so here we have set up a simulated VAR where there is causality from energy to GDP. Mu is the mean sample size of the primary studies in our simulation and var is the variance. So, the lefthand graph is a simulation of mostly small sample studies. The middle one has a mixture of small and large studies, and the right hand graph has mostly large studies (but a few small ones too). The meta sample size is the number of studies that are brought together in the meta-analysis. DGP2a is a data generating process with a small effect size - DGP2b has a larger effect size.

So, what do these graphs show? When the samples in primary studies are small and we only have a meta sample of 10 or 20 studies, it is hard to detect a genuine effect, whatever we do. When the effect size is small it is still hard to detect an effect even when we have 80 primary studies using the traditional economics meta-regression model ("basic model"). Our "extended model" which controls for the number of lags really helps a lot in this situation. With large primary study sizes it is quite easy to detect a true effect with only 20 studies in the meta-analysis and our method adds little value. However, the energy-GDP causality literature has mostly small similar sized samples and is trying to detect what is quite a small effect in the energy causes GDP direction (elasticity of 0.05 or 0.1). Our approach has much to offer in this context.

Wednesday, June 10, 2015

Two Papers Accepted for Publication

Two of our papers have just been accepted for publication. One is my paper with Yingying Lu on sensitivity analysis of climate policy computable general equilibrium models. It has been accepted for publication in Environmental and Resource Economics. The other is a paper with George Halkos and Nikalaos Tzeremes. The paper is on environmental efficiency across the U.S. States and has been accepted by the Journal of Cleaner Production. We haven't put out a working paper version of this one. I'll do a blogpost on it when it is available online at the journal. George was a lecturer at University of York when I was a post-doc there.

In the case of the first paper, we only sent it to one journal (JEEM) before the one it was finally published in. We sent the second paper to quite a few journals but managed to get it into one with a pretty high impact factor after significant revision.

Monday, June 8, 2015


Haven't blogged for over a month. Partly this is because I am doing more tweeting and also because I have been busy with the end of semester and traveling to Turkey (IAEE conference), Israel, and Abu Dhabi (International Energy Workshop). This was my first time attending the IEW. I think there is good feedback in the parallel sessions - better than at the IAEE meeting. On the other hand, the IAEE plenaries are more consistent, I think. IEW is strongly tied to the ETSAP modeling forum, which precedes it - TIMES/MARKAL models - and is attended by "modelers" rather than the mix of business, government, and academic communities at IAEE.

In other news, our paper on the behavior of carbon dioxide emissions in the short-run has been published in Global Environmental Change. The article is open access until 26 July. Also our survey paper in the Review of Economics is also open access.

Wednesday, May 6, 2015

Research Assessment Using Early Citation Information

A new paper with Stephan Bruns on carrying out research assessment like the UK REF and the Australian ERA using citations data rather than peer review. We did a lot of the work of processing the data (doing fancy things with R and manually checking names of universities in Excel) when I visited Stephan in Kassel in November.

The problem with research assessment as carried out in Britain and in the social sciences in Australia is that publications that have already passed through a peer review process are again peer reviewed by the assessment panels. This involves a significant workload for many academics who are supposed to read these papers as well as the effort a each university put into selecting the publications that will be reviewed. However, this second peer review though is inferior to the first. If instead citation based metrics were used the whole process could be done much faster and cheaper. In Australia the natural sciences and psychology are assessed using citation analysis. I think this can be extended to at least some other social sciences including economics.

UK REF panels can also put some weight on citations data in some disciplines including most natural sciences and economics, but only as a positive indicator of academic significance and in very much a secondary role to peer review. This represents a change from the previous RAE, which prohibited the use of citations data by panels. This paper provides additional evidence on the potential effectiveness of citation analysis as a method of research assessment. We hope our results can inform the future development of assessment exercises such as the REF and ERA.

One reason why citations analysis is less accepted in the social sciences than in the natural sciences is the belief that citations accumulate too slowly in most social sciences such as economics to be useful for short-term research assessmen.

My 2014 paper in PLoS ONE shows that long-run citations to articles in economics and political science are fairly predictable from the first few years of citations to those articles. However, research assessment evaluates universities rather than single articles. In this new paper, we show that rank correlations are greatly increased when we aggregate over the economics publications of a university and also when we aggregate publications over time. The rank correlation for UK universities for citations received till the end of 2004 (2005) by economics articles published in 2003 and 2004 with total citations to those articles received through 2014 is 0.91 (0.97). These are high correlations. Correlations for Australia are a bit lower.

Our results here show that at the department or university level citations definitely accumulate fast enough in economics in order to be able to predict longer run citation outcomes of recent publications. It's not true that citations accumulate too slowly in the social sciences to be used in research assessment.

On the other hand, the rank correlation between our early citations indicators and the outcome of research assessment exercises in the UK and Australia ranges from 0.67-0.76. These results suggest that citation analysis is useful for research assessment in economics if the assessor is willing to use cumulative citations as a measure of research strength, though there do appear to be some systematic differences between peer-review based research assessment and our citation analysis, especially in the UK. Part of the difference will emerge due to the differences between the sample of publications we selected to assess and the publications actually selected in the 2010 ERA and 2008 RAE.

Friday, April 24, 2015

Carbon dioxide emissions in the short run: The rate and sources of economic growth matter

Another paper in our ""trends and drivers" series that emerged from my IPCC work. We already released a paper on total greenhouse gas emissions in the long run that I coauthored with Luis Sanchez and a more methodological paper with Reyer Gerlagh, Paul Burke, and Zeba Anjum on fossil fuel CO2 and SO2 emissions in the long run.

The new paper coauthored with Paul Burke and Md. Shahiduzzaman focuses on what happens to fossil fuel carbon dioxide emissions over the business cycle time frame. This topic has received a bit of attention recently. We looked at the issue of what happened after the 2008-9 "Great Recession" in a short 2012 paper in Nature Climate Change that followed up on a paper by Glen Peters and others. Emissions grew very strongly in the recovery in 2010. Our new research shows that 2010 was an unusual year and usually emissions do not rise strongly in the recovery from a recession. In another 2012 paper in Nature Climate Change,, Richard York reported that the response of emissions to expansions and recessions is asymmetric. When the economy is growing the elasticity of emissions with respect to GDP is greater than when it is declining. Our paper tests York’s results and finds that asymmetry is only statistically significant when expansions and recessions of several years in length are considered.

That Gross World Product and CO2 emissions growth rates are tightly linked can be easily seen in the following graph. But there are many details to the story. For example, using country-level data we find that lagged effects are important: around 40% of the effect of economic growth on emissions isn’t realized until a subsequent year.

Some of the paper’s results are summarized in the graph below. We find that the average same-year emissions-income elasticity is about 0.5, but that this elasticity varies depending on the source of economic growth. Agricultural growth has relatively small emissions effects, whereas industrial growth is relatively emissions intensive. External shocks from export markets have quite large domestic emissions implications, presumably because they mostly affect industrial output.

Full abstract: This paper investigates the short-run effects of economic growth on carbon dioxide emissions from the combustion of fossil fuels and the manufacture of cement for 189 countries over the period 1961–2010. Contrary to what has previously been reported, we conclude that there is no strong evidence that the emissions-income elasticity is larger during individual years of economic expansion as compared to recession. Significant evidence of asymmetry emerges when effects over longer periods are considered. We find that economic growth tends to increase emissions not only in the same year, but also in subsequent years. Delayed effects – especially noticeable in the road transport sector – mean that emissions tend to grow more quickly after booms and more slowly after recessions. Emissions are more sensitive to fluctuations in industrial value-added than agricultural value-added, with services being an intermediate case. On the expenditure side, growth in consumption and in investment have similar implications for national emissions. External shocks have a relatively large emissions impact, and the short-run emissions-income elasticity does not appear to decline as incomes increase. Economic growth and emissions have been more tightly linked in fossil-fuel rich countries.

P.S. 4 May 2015
The paper was accepted for publication in Global Environmental Change. Yeah, we posted the working paper when we resubmitted the paper to the journal. Still, overall that was a very fast publication experience with the first journal we submitted to (on 18 December 2014) accepting the paper. This isn't always the case :)

Saturday, March 28, 2015

Drivers of Industrial and Non-Industrial Greenhouse Gas Emissions

Another new working paper this time coauthored with my masters student Luis Sanchez. We use the new approach to modeling the income-emissions relationship pioneered by Anjum et al but using total greenhouse gas emission rather than just CO2 emissions from fossil fuel combustion and cement production. This is closer to the discussion I wrote in Chapter 5 of the Working Group III IPCC Report. Anjum et al. used the more limited emissions variable because the IPCC wouldn't allow us to use the data assembled for the report in other research and it took a lot of effort on Luis' part to put the data together from the raw Edgar data. Also, economists are more familiar with the narrow industrial CO2 emissions variable and so we thought we'd do an analysis of that first.

There has been extensive analysis of the drivers of carbon dioxide emissions from fossil fuel combustion and cement production, but these only constituted  55% of global greenhouse gas (GHG) emissions (weighted by global warming potential) in 1970 and 65% in 2010. There has been much less analysis of the drivers of greenhouse gases in general and especially of emissions of greenhouse gases from agriculture, forestry, and other land uses, which we call non-industrial emissions in the paper, that constituted 24% of total emissions in 2010.

The graphs show that non-industrial emissions have a different relationship to income than do industrial emissions. However, there is still a positive relationship between the growth rates of the two variables, especially when we give more weight to larger countries as we do in the paper. Increases in the economic growth rate have about half the effect on non-industrial emissions than they have on industrial emissions.

In both of these graphs China is the large circle on the right. The country with highest non-industrial emissions is Indonesia, which is the largish circle above and to the right of China in the second graph.

We econometrically analyze the relationship between both industrial and non-industrial greenhouse gas emissions and economic growth and other potential drivers for 129 countries over the period from 1971 to 2010. As in Anjum et al., our method combines the three main approaches in the literature to investigating the evolution of emissions and income. We find that economic growth is a driver of both industrial and non-industrial emissions, though growth has twice the effect on industrial emissions. Both sources of emissions decline over time though this effect is larger for non-industrial emissions. There is also convergence in emissions intensity for both types of emissions but given these other effects there is again no evidence for an environmental Kuznets curve.

Monday, March 23, 2015

Telstra Internet

Back in 2010 I reported on the speed of the internet at home in Canberra and from my office on the ANU campus.

I just moved house and because service with iiNet was so bad and getting worse we switched to Telstra internet service. The speed is much, much higher:

The download speed is more than 8 times higher and the upload speed almost 4 times higher when accessing a server in Canberra. Accessing a server in San Jose, California:

downloading is more than 5 times faster and uploading 4 times faster.

Saturday, March 14, 2015

Seminar @ Arndt-Corden 17 March

I am giving a seminar at Arndt-Corden on Tuesday 17th March at 2pm (Seminar Room B, Coombs Building, ANU) titled: "Directed Technical Change and the British Industrial Revolution". The abstract isn't entirely accurate any more - well specifically you won't see me talk about the last two sentences as we don't use a Monte Carlo analysis and we left the low elasticity of substitution for further research. We (myself, Jack Pezzey, and Yingying Lu) are close to having a paper that we are ready to put out as a working paper and submit to a journal. So, am looking forward to getting some useful comments to help us get there.

Wednesday, March 11, 2015

Kander et al. Paper on National Greenhouse-Gas Accounting in Nature Climate Change

Astrid Kander and coauthors at Lund and the University of New South Wales have a paper in Nature Climate Change that proposes a new way to account for embodied carbon in trade that improves on existing measures of consumption based emissions. The collaboration with UNSW was sparked when Astrid gave a presentation at Crawford School in 2012 on the topic, which was attended by Tommy Wiedmann who was then at CSIRO but moved soon after to UNSW. Astrid was visiting ANU to work on our ARC project.

The most common way to compute carbon emissions is based simply on where the emissions are produced. These are called production based emissions (PBA). It is often argued though that this approach overly penalizes countries that export emissions intensive goods and makes countries that import these goods look like their emissions are low when they benefit from emissions intensive production elsewhere. Consumption based emissions (CBA) count all the emissions produced by a country's consumption wherever in the world the goods consumed were produced. Usually, developed countries look more carbon intensive and developing countries less carbon intensive on this basis than when using production based emissions. The following Figure from Kander et al. shows that in the European Union and the USA consumption based emissions exceed production based emissions and vice-versa in China:

But if developed countries tried to produce all their imported goods at home, it is likely that their production techniques would be less emissions intensive than those in the countries that they are importing from. So, consumption based emissions accounting gives a biased view of how much developed countries have managed to reduce emissions by offshoring production. Also, if consumption based emissions were used to apportion world responsibility for reducing emissions the only strategy an importer would have to reduce emissions accounted this way is to stop importing and produce domestically which might not be economically efficient, while the exporter has no incentive to cut these emissions.

However, accounting for emissions embodied in imports based on how much carbon would be emitted if they were produced in the importing country will underestimate total global emissions and so if we want a system of apportioning emissions fairly and usefully for global climate policy purposes it is not so useful.

Kander et al.'s approach deals with the incentive issue. They measure embodied emissions in imports in the same way as conventional CBA. However, they account for exports using the world average emissions intensity for the given good to deduct emissions from exporters instead of deducting the actual emissions produced. This reduces the emissions total for exporters who produce in a low emissions intensive way and increase the emissions of emissions intensive exporters compared to CBA. These technology adjusted consumption based (TCBA) emissions do sum to world total emissions. All exporters now have an incentive to reduce their exports emissions intensity if they were held responsible for their TCBA emissions. The resulting TCBA per capita emissions are shown in the map below and the graphs above.

On this basis emissions per capita in Europe are even less than production based emissions while in the USA they are similar to consumption based emissions. Australia also doesn't look too good on the map. On the other hand, in China TCBA emissions are intermediate between CBA and PBA emissions. The strong performance of Europe is because they have lower than average emissions intensity for the products they export. The latter means that world average emissions for those products is deducted from Europe's balance but their actual emissions for producing those products is lower than that.

The biggest "winners" are Austria, Ireland, and Belgium, which look much more emissions intensive under CBA than under PBA but much less emissions intensive under TCBA.

Astrid discusses the rationale for their approach further in this news article.

Sunday, March 8, 2015

Energy Prices, Growth, and the Channels in Between: Theory and Evidence

Lucas Bretschger has an interesting new paper in Resource and Energy Economics titled: "Energy prices, growth, and the channels in between: Theory and evidence". The paper argues that countries with higher energy use per capita grow slower in the long-run though reductions in energy use lower output in the short-run. The long-run effect is due to an induced increase in capital accumulation and because the model is similar to AK endogenous growth models, innovation. The paper is motivated by the stylized fact that in a sample of 37 countries, countries with higher per capita energy use grow slower. The sample includes mostly developed countries but also China and India.

This negative correlation is, however, easy to explain in terms of catch-up growth dynamics. As we show in our stylized facts paper, there is a strong positive correlation between the level of GDP per capita and the level of energy use per capita. Per capita income in countries such as China and India has risen faster than in the developed countries due to the fact that they are poorer and undergoing catch-up growth. This results in a spurious negative relationship between energy use per capita and the rate of economic growth.

This is not to say at all that Bretschger's theoretical model is wrong, but the motivation can easily be explained in another way. In fact, I'm very sympathetic to the idea that plentiful energy resources could slow the rate of economic growth as I discussed in my presentation at the AARES conference in Rotorua and my upcoming Arndt-Corden seminar on 17th March.

Bretschger also estimates an econometric model that is loosely related to his theoretical model (for example, using energy quantity rather than price due to a lack of internationally comparable data - a big problem for energy economics) that regresses the investment/output ratio on energy intensity and GDP (there are additional equations). It is not surprising that reduced energy intensity could encourage increased investment if it represents increased energy efficiency - this is one of the factors in macro-level rebound as in Harry Saunders (1992) model.

The bottom line is that the energy-output relationship is quite complicated and is probably not at all well captured by reduced form time series models. Bretschger is also making this point with his paper.

Sunday, February 22, 2015

How Has Research Assessment Changed the Structure of Academia?

Does measuring something change it?  In quantum mechanics measurement disturbs what is being measured, which is referred to as the observer effect. The same is often true in social systems, especially of course when measurement is attached to rewards. The UK and Australia have been conducting periodical research assessment exercises - the REF and ERA. In the case of the UK, research assessment started almost three decades ago. In Australia, the first research assessment was only conducted in 2010 but the founding of the ARC in 1988 and its independence in 2001 are both milestones in the road to increased emphasis on competition in research in Australia.

Johnston et al. (2014) show that the total number of economics students has increased in UK more rapidly than the total number of all students, but the number of departments offering economics degrees has declined, particularly in post-1992 universities. Also, the number of universities submitting to the REF under economics has declined sharply with only 3 post-1992 universities submitting in the latest round. This suggests that the REF has driven a concentration of economics research in the more elite universities in the UK. BTW the picture above is of the Hotel Russell, which the Russell Group of British universities is named after.

Neri and Rodgers (2014) investigate whether the increased emphasis on research in Australia has had the desired effect in the field of economics. They investigate the output of top economics research by Australian academics from 2001 to 2010. By constructing a unique database of 26,219 publications in 45 top journals, they compare Australia’s output internationally, determine whether Australia’s output increased, and rank Australian universities based on their output. They find that Australia’s output, in absolute and relative terms, and controlling for differences in page size and journal quality, increased and, on a per capita basis, is converging to the levels of the most research-intensive countries. Finally, they find that the historical dominance of the top four universities is diminishing. The correlation between the number of top 45 journal articles published in 2005-2010 and the ERA 2012 ranking is 0.83 (0.78 for 2003-8 and ERA 2010).


Johnston, J., Reeves, A. and Talbot, S. (2014). ‘Has economics become an elite subject for elite UK universities?’ Oxford Review of Education, vol. 40(5), pp. 590-609.

Neri, F. and Rodgers, J. (2014). ‘The contribution of Australian academia to the world’s best economics research: 2001 to 2010’, Economic Record.

Peer Review vs. Citation Analysis in Research Assessment Exercises

Existing research finds strong correlations between the rankings produced by UK research assessment exercises (RAE) and bibliometric analyses for several specific humanities and social science disciplines (e.g. Colman et al., 1995; Oppenheim, 1996; Norris and Oppenheim, 2003) including economics (Süssmuth et al., 2006). Clerides et al. (2011) compare the 1996 and 2001 RAE ratings of economics departments with independent rankings from the academic literature. They find RAE ratings to be largely in agreement with the profession’s view of research quality as documented by independent rankings, although the latter appear to be more focused on research quality at the top end of academic achievement. This is because most rankings of departments in the economics literature are based on publications in top journals only, which lower ranked departments have very few of.

Mryglod et al. (2013) analyse the correlations between the values of Thomson Reuters Normalised Citation Impact (NCI) indicator and RAE 2008 peer-review scores in several academic disciplines, from the natural to social sciences and humanities. The NCI computes the normalized impact factor across a unit of assessment (an academic discipline at a given university) in the RAE based on only the publications actually submitted to the RAE. Mryglod et al. (2013) compute both average (or quality) and total (or strength) values (average multiplied by number of staff submitted to the RAE) of these two indicators for each institution. They find very high correlations for the strength indicators for some disciplines and poor correlations at for the quality indicators for all disciplines. This means that, although the citation-based scores could help to describe institution level strength (which is quality times size), in particular for the so-called hard sciences, they should not be used as a proxy for ranking or comparison of research groups. Moreover, the correlation between peer-evaluated and citation-based scores is weaker for the “soft” sciences. Spearman rank correlation coefficients for their quality indicators range from 0.18 (mechanical engineering) to 0.62 (chemistry). However for strength the correlations range from 0.88 (history and sociology) to 0.97 (biology). This is because quality is correlated with size and so the two factors reinforce each other.

Mryglod et al. (2014) attempt to predict the 2008 RAE retrospectively and the 2014 Research Excellence Framework (REF) before its results were released. They examined biology, chemistry, physics, and sociology. Of the indicators they trialled, they found that the departmental h-index had the best fit to the 2008 results. Departmental h-index is based on all publications published by a department in the time window assessed by the relevant assessment exercise. The rank correlation ranged from 0.83 in chemistry to 0.58 in sociology. They find that the correlation with the RAE for the immediate h-index is as good as the correlation in later years with the h-index of the same set of publications.

Bornmann and Leydesdorff (2014) argue that one of the downsides of bibliometrics as a research assessment instrument is that citations take time to accumulate while research assessment exercises are designed to assess recent performance:

“This disadvantage of bibliometrics is chiefly a problem with the evaluation of institutions where the research performance of recent years is generally assessed, about which bibliometrics—the measurement of impact based on citations—can say little…. the standard practice of using a citation window of only 3 years nevertheless seems to be too small.” (1230)

They argue further that bibliometrics:

“can be applied well in the natural sciences, but its application to TSH (technical and social sciences and humanities) is limited.” (1231)

But rather than assuming that peer review is the preferred approach to research assessment and citation analysis should only be used to reduce cost, we can ask whether the review conducted by research assessments such as the REF and the Australian ERA meets the normal academic standards for peer review. Research does show that peer review at journals has predictive validity for the citations that will be received by accepted papers compared to those received by rejected papers. However, evidence for the predictive validity of peer review of grant and fellowship applications is more mixed (Bornmann, 2011). Therefore, further research is warranted on the use of citation analysis to rank academic departments or universities in research assessment exercises. Sayer (2014) argues that the peer review undertaken in research assessment exercises does not meet normal standards for peer review. He compares university and national-level REF processes against actual practices of scholarly review as found in academic journals, university presses, and North American tenure procedures. He finds that the peer review process used by the REF falls far short of the level of scrutiny or accuracy of these more familiar peer review processes. The number of items each reviewer has to assess alone means that the review cannot be of the same quality as reviews for publication. And reviewers will have to assess much material outside their area of specific expertise. Sayer argues that though metrics may have problems, a process that gives such extraordinary gatekeeping power to individual panel members is far worse.

Given the large number of items that panels need to review they are likely to focus on the venue of publication and at least in business and economics handy mappings of journals to REF grades exist (Hudson, 2013). Regibeau and Rockett (2014) build imaginary economics departments entirely composed of Nobel Prize winners and evaluate them using standard journal rankings geared to the UK RAE. Performing the same evaluation on existing departments, they find that the rating of the Nobel Prize departments does not stand out from other good departments. Compared to recent research evaluations, the Nobel Prize departments’ rankings are less stable. This suggests a significant effect of score “targeting” induced by the rankings exercise. They find some evidence that modifying the assessment criteria to increase the total number of publications considered can help distinguish the top. But if departments composed entirely of Nobel Prize winners perform worse than current departments then it is hard to know what such assessment means.

Sgroi and Oswald (2013) examine how research assessment panels could most effectively use citation data to replace peer review. They suggest a Bayesian approach that uses prior information on where a item was published combined with observations on citations to derive a posterior distribution for the quality of the paper. We could then estimate, for example, what is the probability that a paper belongs in the 4* category given where it was published and the early citations it has received. Stern (2014) and Levitt and Thelwall (2011) show that the journal impact factor has strong explanatory power in the year of publication but that this declines very quickly as citations accumulate. So, this approach would be most useful for papers published in the last year or two before the assessment, but for earlier research outputs the added value over simply counting citations would be minimal.


Bornmann, L. (2011) ‘Scientific peer review’, Annual Review of Information Science and Technology, vol. 45, pp. 199‐245.

Bornmann, L. and Leydesdorff, L. (2014). ‘Scientometrics in a changing research landscape’, EMBO Reports, vol. 15(12), pp. 1228–32.

Clerides, S., Pashardes, P. and Polycarpou, A. (2011) ‘Peer review vs metric-based assessment: testing for bias in the RAE ratings of UK economics departments’, Economica, vol. 78(311), pp. 565–83.

Colman, A. M., Dhillon, D. and Coulthard, B. (1995) ‘A bibliometric evaluation of the research performance of British university politics departments: Publications in leading journals’, Scientometrics vol. 32(1), pp. 49-66.

Hudson, J. (2013). ‘Ranking journals’, Economic Journal, vol. 123, pp. F202-22.

Levitt, J.M. and Thelwall, M. (2011). ‘A combined bibliometric indicator to predict article impact’, Information Processing and Management, vol. 47, pp. 300–8.

Mryglod, O., Kenna, R., Holovatch, Y. and Berche, B. (2013). ‘Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence’, Scientometrics, vol.97, pp. 767–77.

Mryglod, O., Kenna, R., Holovatch, Y. and Berche, B. (2014). Predicting Results of the Research Excellence Framework Using Departmental H-Index, arXiv:1411.1996v1.

Norris, M. and Oppenheim, C. (2003) ‘Citation counts and the research assessment exercise V: Archaeology and the 2001 RAE’, Journal of Documentation, vol. 59(6): pp. 709-30.

Oppenheim, C. (1996) ‘Do citations count? Citation indexing and the research assessment exercise’, Serials, vol. 9, pp. 155–61.

Regibeau, P. and Rockett, K.E. (2014). ‘A tale of two metrics: Research assessment vs. recognized excellence’, University of Essex, Department of Economics, Discussion Paper Series 757.

Sayer, D. (2014). Rank Hypocrisies: The Insult of the REF. Sage.

Sgroi, D. and Oswald, A.J. (2013). ‘How should peer-review panels behave?’ Economic Journal, vol. 123, pp. F255–78.

Stern, D.I. (2014). ‘High-ranked social science journal articles can be identified from early citation information’, PLoS ONE, vol. 9(11), art. e112520. 

Süssmuth, B., Steininger, M. and Ghio, S. (2006) 'Towards a European economics of economics: Monitoring a decade of top research and providing some explanation', Scientometrics, vol. 66(3), pp. 579-612.