Standard Errors of Impact Factors

A few times recently I've seen it claimed on the web that there is no correlation between journals ISI impact factors and the number of citations that papers published in them get. On the face of it, this sounds a priori silly. The 2 year impact factor for 2010 is the average number of citations that articles published in a journal in 2008 and 2009 got in 2010 in the Web of Science. The relationship could be weak but it's unlikely to be zero. I've checked this for my own journal articles and the correlation between Article Influence Score and citations per year is 0.20.

A different way to look at this is to ask what is the standard error of the estimated impact factor? I decided to compute this for the Journal of Economic Growth. This journal has a reasonably high impact factor - 2.458 in 2010 - and only publishes twelve articles a year, which makes it easy to compute. Here are the number of citations each paper published in the previous two years got in 2010:



As you can see, half the papers got 1 or 0 citations. This doesn't necessarily mean they are duds, maybe they were published near the end of 2009 or are slow to gain popularity. The top three papers got 13, 10, and 8 citations respectively and the fourth only 4. The standard deviation of the mean is 0.69. That means that a 95% confidence interval stretches from 1.07 to 3.84. Possibly one can be more confident about impact factors for journals with larger numbers of papers or for 5 year impact factors, we'll have to see.