Reproducibility in Econometrics Research



This page is an attempt to encourage a dialogue about reproducibility in econometric research. Like the weather, reproducibility of research is a topic of frequent discussion, but little action. Experience suggests that mandates by journals and the NSF notwithstanding, research in both econometric theory and applications remains difficult to reproduce. This is particularly true for students or new Ph.ds in the field who, quite properly, are reluctant to approach senior colleagues about "filling in the gory details" of what they may have done a decade ago.

Central archives for data and programs have not met with widespread acceptance in econometrics. There is no archive in econometrics which plays the important role that statlib plays in statistics. But no central archive can serve the full function required of providing complete details of published work in a transparent form, easily accessible by a worldwide audience. If this is to be realized, it seems it must happen in a decentralized manner. Individual researchers must be convinced that it is in their own interest to provide details as part of the effort to encourage the dissemination of their ideas.

There are many impediments, not the least of which is the "Tower of Babel" of econometric software. But this should not prevent us from making a start. In this spirit we suggest adopting the following general principles taken from recent work by David Donoho:

All the code underlying figures and tables is made available
Together with the underlying software environment necessary to execute that code
Together with documentation of both the tools and environment
Using standard internet methods (ftp, www) for anonymous access.

Some ancient notes describing some tools for facilitating reproduciblity using Splus and latex are described in the paper Reproducible Econometric Research which is available in postscript format. Some more recent notes along similar lines are available in Protocol for Simulations in R. A file used as an example in this document can be downloaded from this page as plink.R. A new review essay on the subject written with Achim Zeileis is available from here.

I would very much like to have comments on all of this, I would particularly like to encourage others to suggest other www links which provide other examples of this sort. We would also welcome comments on further elucidation of the principles proposed above and ways to make them more operational.

Last Revised on August 2011 by Roger Koenker

roger@ysidro.econ.uiuc.edu