Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Some Intuition About the Theory of Statistical Learning

DZone's Guide to

Some Intuition About the Theory of Statistical Learning

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

While I was working on the Theory of Statistical Learning, and the concept of consistency, I found the following popular graph (e.g. from  thoses slides, here in French)

The curve below is the error on the training sample, as a function of the size of the training sample. Above, it is the error on a validation sample. Our learning process is consistent if the two converge.

I was wondering if it was possible to generate such a graph, with some data, and some statistical model. And indeed, it is rather simple, and it gives nice intuition about possible interpretations. Consider some (simple) classification problem. Here, we consider a logistic regression. We generate a sample of size , we fit our model, we compute the misclassification rate, then we generate another sample of size , we use our previous model to make some prediction, and we compute the misclassifiation rate. And we play with .

missclassification <- function(n){
  U=data.frame(X1=runif(n),X2=runif(n))
  p=(U[,1]+U[,2])/2
  U$Y=rbinom(n,size=1,prob=p)
  reg=glm(Y~X1+X2,data=U,family=binomial)
  pd=function(x1,x2) predict(reg,newdata=data.frame(X1=x1,X2=x2),type="response")>.5
  x=seq(0,1,length=101)
  z=outer(x,x,pd)
  cl2=c(rgb(1,0,0,.4),rgb(0,0,1,.4))
  cl1=c("red","blue")
  image(x,x,z,col=cl2,xlab="",ylab="",main="Training Sample")
  points(U$X1,U$X2,pch=19,col=cl1[1+U$Y])

  V=data.frame(X1=runif(n),X2=runif(n))
  p=(V[,1]+V[,2])/2
  V$Y=rbinom(n,size=1,prob=p)
  screen(4)
  image(x,x,z,col=cl2,xlab="",ylab="",main="Validation Sample")
  points(V$X1,V$X2,pch=19,col=cl1[1+V$Y])

  MissClassU=mean(abs(pd(U$X1,U$X2)-U$Y))
  MissClassV=mean(abs(pd(V$X1,V$X2)-V$Y))
return(c(MissClassU,MissClassV))
}


If we plot it, we get (in purple, it is the training sample, and in black, the validation sample)

The graph is not exactly the same as above, but it is probably due to the randomness of our samples. If we generate hundreds of samples, it should be just fine.

MCU=rep(NA,500)
MCV=rep(NA,500)
n=250
  for(i in 1:500){
    U=data.frame(X1=runif(n),X2=runif(n))
    p=(U[,1]+U[,2])/2
    U$Y=rbinom(n,size=1,prob=p)
    reg=glm(Y~X1+X2,data=U,family=binomial)
    pd=function(x1,x2) predict(reg,newdata=data.frame(X1=x1,X2=x2),type="response")>.5
    MCU[i]=mean(abs(pd(U$X1,U$X2)-U$Y))

    V=data.frame(X1=runif(n),X2=runif(n))
    p=(V[,1]+V[,2])/2
    V$Y=rbinom(n,size=1,prob=p)
    MCV[i]=mean(abs(pd(V$X1,V$X2)-V$Y))
  }
  MissClassV=mean(MCU)
  MissClassU=mean(MCV)


Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.

Topics:
bigdata ,big data ,statistical learning

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}