Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Cross Validation for Kernel Density Estimation

DZone's Guide to

Cross Validation for Kernel Density Estimation

· Big Data Zone
Free Resource

Learn how you can maximize big data in the cloud with Apache Hadoop. Download this eBook now. Brought to you in partnership with Hortonworks.

In a post published in July, I mentioned the so-called Goldilocks principle, in the context of kernel density estimation, and bandwidth selection. The bandwidth should not be too small (the variance would be too large) and it should not be too large (the bias would be too large). Another standard method to select the bandwidth, as mentioned this afternoon in class is the cross-validation technique (described in Chiu (1991)). Here, we would like to minimize

http://latex.codecogs.com/gif.latex?\mathbb{E}\left[\int%20[\widehat{f}_h(x)-f(x)>^2dx\right]

The integral can be written

http://latex.codecogs.com/gif.latex?\int%20\widehat{f}_h(x)^2dx-2\int%20\widehat{f}_h(x)f(x)dx+\int%20f(x)^2dx

Since the third component is constant, we have to minimize the expected value of the sum of the first two.

The idea is to approximate it as

http://latex.codecogs.com/gif.latex?J(h)=\int%20\widehat{f}_h(x)^2dx-\frac{2}{n}\sum_{i=1}^n%20\widehat{f}_{(-i)}(X_i)

which can easily be computed. Consider here some sample, with 50 observations, from a Gaussian distribution,

> set.seed(1)
> X=rnorm(50)

From Silverman’s rule of thumb (which should be appropriate here since the sample has a Gaussian sample) the optimal bandwidth is

> 1.06*sd(X)*length(X)^(-1/5)
[1] 0.4030127

Using the cross-validation technique mentioned above, compute

> J=function(h){
+ fhat=Vectorize(function(x) density(X,from=x,to=x,n=1,bw=h)$y)
+ fhati=Vectorize(function(i) density(X[-i],from=X[i],to=X[i],n=1,bw=h)$y)
+ F=fhati(1:length(X))
+ return(integrate(function(x) fhat(x)^2,-Inf,Inf)$value-2*mean(F))
+ }
> vx=seq(.1,1,by=.01)
> vy=Vectorize(J)(vx)
> df=data.frame(vx,vy)
> library(ggplot2)
> qplot(vx,vy,geom="line",data=df)

The function has the following shape

and the optimal value is

> optimize(J,interval=c(.1,1))
$minimum
[1] 0.4687553

$objective
[1] -0.3355477

Note that, indeed, it is close to Siverman’s optimal bandwidth.

Hortonworks DataFlow is an integrated platform that makes data ingestion fast, easy, and secure. Download the white paper now.  Brought to you in partnership with Hortonworks

Topics:

Published at DZone with permission of Arthur Charpentier, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}