Density Estimation on TV


Density Estimation by Total Variation Regularization
Roger Koenker and Ivan Mizera

L_1 penalties have proven to be an attractive regularization device for nonparametric regression, image reconstruction, and model selection. For function estimation, L_1 penalties, interpreted as roughness of the candidate function measured by their total variation, are known to be capable of capturing sharp changes in the target function while still maintaining a general smoothing objective. We explore the use of penalties based on total variation of the estimated density, its square root, and its logarithm -- and their derivatives -- in the context of univariate and bivariate density estimation, and compare the results to some other density estimation methods including L_2 penalized likelihood methods. Our objective is to develop a unified approach to total variation penalized density estimation offering methods that are: capable of identifying qualitative features like sharp peaks, extendible to higher dimensions, and computationally tractable. Modern interior point methods for solving convex optimization problems play a critical role in achieving the final objective, as do piecewise linear finite element methods that facilitate the use of sparse linear algebra.



Software is under development in Matlab and R. The paper is available in pdf.


The alter egos of the regularized maximum likelihood density estimators: deregularized maximum-entropy, Shannon, Renyi, Simpson, Gini, and stretched strings
Roger Koenker and Ivan Mizera

Various properties of maximum likelihood density estimators penalizing the total variation of some derivative of the logarithm of the estimated density are discussed, in particular the properties of their dual formulations and connections to stretched (taut) string methodology.

The paper is available in pdf.

Primal and dual formulations relevant for the numerical estimation of a probability density via regularization
Roger Koenker and Ivan Mizera

We investigate general schemes relevant for the estimation of a probability density via regularization---their primal and dual versions in the discretized setting. In particular, conditions for the dual solution to be a probability density are given, and a strong duality theorem is proved.

The paper is available in pdf.

Voronograms
A somewhat abortive attempt to consider density and and regression estimators in R^2 that were regularized by total variation penalization of the functions themselves, resulting in piecewise constant solutions on Voronoi tessilations never got far enough along to produce a paper, but there is an R package that produces some amusing pictures. It is available here.



What Do Kernel Density Estimators Optimize?
Roger Koenker Ivan Mizera and Jungmo Yoon

In economics it is commonly believed that we understand a thing if and only if we can formulate an optimization problem that produces the thing. In this respect the appeal and apparent success of kernel density estimation methods in econometrics is something of an anomaly. By partially answering the question posed in the title we hope to help remedy this neglect. We make no great claims for the novelty of our account, indeed many aspects would be familiar to those conversant with the regularization literature, particularly the influential papers of Silverman (1982, 1984), but they may be less so within the econometrics community.

The paper is available in pdf.



Comments are, of course, always welcome.