Three Awesome TensorFlow.js Models for Visual Recognition

last week, i open sourced a web application called blue cloud mirror , which is a game where players need to show five specific emotions and do five specific poses in two levels. the faster, the better.

you can play the game online ; it only takes a minute. all you need is a webcam and a chrome browser.

the game uses three different tensorflow.js models for visual recognition. the code is executed in browsers to run predictions, which are pretty fast.

faces

face-api.js can detect faces, find face similarities and detect face landmarks. try the online demo .

emotions

face_classification predicts emotions and can identify genders. the output of face-api.js is passed into this model. try the online demo .

poses

posenet can detect 17 key points of bodies and works for single persons and multiple persons. try the online demo .

using the models in vue.js

the usage of the models in vue.js is pretty straight forward. here is the code to load and run the emotions model.

<script>
import * as tf from "@tensorflow/tfjs";

export default {
  name: "emotionmodel",
  data() {
    return {
      model: {}
    };
  },
  mounted() {
    let url = "models/emotion/model.json";
    tf.loadmodel(url).then(result => {
      this.model = result;
...
   var r = this.model.predict(input);
   var result = r.datasync();
   var tresult = tf.tensor(result);
   var label_index = tf.argmax(tresult).datasync()[0];
   var label_percent = result[label_index].tofixed(4) * 100;
...
</script>

if you want to run everything from your local machine, run these commands:

$ git clone https://github.com/ibm/blue-cloud-mirror.git
$ cd blue-cloud-mirror/game
$ yarn install
$ yarn run serve

this is a screenshot of level one of the game where five different emotions need to be shown.

thanks, and let me know what you think in the comments section!