Parametric Density Estimation Using Polynomials and Fourier Series

Imagine you obtain samples of two types, such as the red and green points in the graph shown here, and you would like to generalize by classifying some other points into one of these two classes. This is a basic problem in machine learning called supervised classification, with the class of each sampled point known. The standard approach is based on linear classifiers: using (hyper) planes for separation into two classes, analogous to its use in support vector machines or neural networks.

This Demonstration lets you play with direct nonlinear (two-class) classifiers in a 2D square: fitting a polynomial or Fourier series to the sample, using weights of opposite signs to distinguish the classes, followed by classification based on the sign of the difference of densities.

You can add points from the class of a chosen color by clicking its position. You can also generate some chosen number of random points. The "degree" controller lets you choose the degree of the polynomial. The number of coefficients used is shown below it. The "method" controller lets you switch between fitting a polynomial or a Fourier series. The "plot" controller lets you choose between RegionPlot and DensityPlot.

The method used for fitting density is described in [1]. The formula was derived by smoothing the sample through convolution with a kernel, then fitting a polynomial or Fourier series to the smoothed sample. It turns out that for mean-square fitting, we can perform the limit to a zero-width kernel, giving asymptotically optimal coefficients with a very simple formula.

Specifically, assuming orthonormal bases of functions (Legendre polynomials or sines and cosines here), the linear coefficient for a given function turns out to be just the average of the function over the sample:

,

where is the average of over the sample.

For the classification problem in this Demonstration, this density for green points was subtracted from the density for red points. Either the region of positive values (RegionPlot) or its density (DensityPlot) is then drawn.