How to Calculate the Practical Significance of Citation Impact Differences?

An Empirical Example from Evaluative institutional bibliometrics using adjusted predictions and marginal effects

Lutz Bornmann and Richard Williams

Article

Bornmann, L., & Williams, R. How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects. Journal of Informetrics (2013), http://dx.doi.org/10.1016/j.joi.2013.02.005

 

Abstract

Evaluative bibliometrics is concerned with comparing research units by using statistical procedures. According to Williams (2012) an empirical study should be concerned with the substantive and practical significance of the findings as well as the sign and statistical significance of effects. In this study we will explain what adjusted predictions and marginal effects are and how useful they are for institutional evaluative bibliometrics. As an illustration, we will calculate a regression model using publications (and citation data) produced by four universities in German-speaking countries from 1980 to 2010. We will show how these predictions and effects can be estimated and plotted, and how this makes it far easier to get a practical feel for the substantive meaning of results in evaluative bibliometric studies. An added benefit of this approach is that it makes it far easier to explain results obtained via sophisticated statistical techniques to a broader and sometimes nontechnical audience. We will focus particularly on Average Adjusted Predictions (AAPs), Average Marginal Effects (AMEs), Adjusted Predictions at Representative Values (APRVs) and Marginal Effects at Representative Values (MERVs).

Files

Here are the files needed to replicate the Bornmann/ Williams analysis. Stata 12.1 or higher is required

bornmann.do - Stata program to construct variables and run the analysis

bornmann.dta - Data set used in the analylsis