Face Detection with getUserMedia

there are quite a few interesting apis evolving in the "modern web", but not all of them are going to be things you would use in most projects. i've been very public about my feelings concerning canvas for example. great for games and charting - but not much else. that doesn't make it a bad feature. it just makes it one i won't use terribly often. whenever i read about some new cool feature being developed, my mind starts trying to figure out what they could be used for in a practical sense. obviously what's practical to you may not be practical to me, but figuring out how i would actually use a feature is part of how i learn it.

one such feature is getusermedia ( w3c spec ). this is a javascript api that gives you access to (with permission) the user's web cam and microphone. getusermedia is currently supported in opera and chrome (i believe it is in version 18 now, but you may need to grab canary. you also need to enable it. instructions on that here .) once you get past actually enabling it, the api is rather simple. here's a quick request for access:

//a video tag
var video = document.getelementbyid('monitor');

//request it
navigator.webkitgetusermedia('video', gotstream, nostream);

function gotstream(stream) {

	video.src = webkiturl.createobjecturl(stream);
	video.onerror = function () {
		stream.stop();
		streamerror();
	};
}

function nostream() {
	document.getelementbyid('errormessage').textcontent = 'no camera available.';
}


function streamerror() {
	document.getelementbyid('errormessage').textcontent = 'camera error.';
}

the first argument to getusermedia is the type. according to the spec, this is supposed to be an object where you enable audio, video, or both, like so: {audio:true, video:true}. however in my testing, passing a string, "video", worked fine. the demo you will be seeing is based on another demo so that line possibly came from an earlier build that still works with chrome. the second and third arguments are your success and failure callbacks respectively.

you can see in the gist where the success handler assigns the video stream to an html5 video tag. what's cool then is that once you have that running you can use the canvas api to take pictures. for a demo of this, check out greg miernicki's demo:

http://miernicki.com/cam.html

if this demo doesn't work for you - then stop - and try following the instructions again to enable support. (although i plan on sharing a few screen shots so if you just want to keep reading, that's fine too.)

based on greg's demo, it occurred to me that there is something cool we can do with pictures of our web cams. (cue the dirty jokes.) i remembered that face.com had a very cool api for parsing pictures for faces. (i blogged a coldfusion example back in november.) i wondered then if we could combine greg's demo with the face.com api to do some basic facial recognition.

turns out there are a few significant issues with this. first - while face.com has a nice rest api, how would we use it from a javascript application? secondly - face.com requires you to either upload a picture or give it a url. i know i could send a canvas picture to a server and have my backend upload it to face.com, but is there a way to bypass the server and send the picture right to the api?

the first issue actually turned out to be a non-issue. face.com implements cors (cross-origin resource sharing). cors basically allows a server to expose itself to ajax calls from documents on other domains. it's a great feature and i hope more services enable it.

the more complex issue then was taking the canvas data and sending it to face.com. how can i fake a file upload? turns out there's another cool new trick - formdata. fellow coldfusion blogger sagar ganatra has an excellent blog entry on the topic. here's how i used it:

function snapshot() {
	$("#result").html("<p><i>working hard for the money...</i></p>");

	canvas.width = video.videowidth;
	canvas.height = video.videoheight;
	canvas.getcontext('2d').drawimage(video, 0, 0);

	var data = canvas.todataurl('image/jpeg', 1.0);
	newblob = datauritoblob(data);

	var formdata = new formdata();
	formdata.append("api_key", facekey);
	formdata.append("api_secret", facesecret);
	formdata.append("filename","temp.jpg");
   
	formdata.append("file",newblob); 

	$.ajax({
	   url: 'http://api.face.com/faces/detect.json?attributes=age_est,gender,mood,smiling,glasses',
   	   data: formdata,
	   cache: false,
	   contenttype: false,
	   processdata: false,
	   datatype:"json",
	   type: 'post',
	   success: function (data) {
	       	handleresult(data.photos[0]);
	   }

	});    
}

let's look at this line by line. first off - i need to get the binary data from the canvas object. there's a few ways of doing this, but i wanted a binary blob specifically. notice the datauritoblob method. this comes from a stackoverflow post i found a few weeks back.

i create a new formdata object and then simply begin setting my values. you can see i pass in a few api requirements but the crucial parts are the filename and file object itself.

below that you can see the simple jquery ajax call. face.com has a variety of options, but i basically just asked it to return an estimated age, gender, mood, and whether or not the person was smiling and wearing glasses. that's it. i get a nice json packet back and format it.

now obviously no api is perfect. i've had different levels of results from using the api. sometimes it's pretty damn accurate and sometimes it isn't. overall though it's pretty cool. here are some scary pictures of yours truly testing it out.

ok, ready to test it yourself? just click the demo button below. for the entire source, just view source! this is 100% client-side code.

for another look at getusermedia, check out these examples:

• it's curtains for marital strife thanks to getusermedia
• testing webrtc on chrome
• bleeding edge html5, webrtc & device access
• capturing audio & video in html5