Misty’s got a new job: security guard.

Misty loves to be helpful, and this job combines a few of her different capabilities into a single working skill. When the skill runs, Misty activates face recognition and scans the environment for strangers. If Misty sees an unknown face, she calls out to third-party APIs and prints a picture of the intruder. Face recognition, external HTTP calls, and image capture are the core capabilities that Misty uses for this job.

In this series, we’ll walk through these capabilities one at a time so you can build a security guard skill yourself. We’ll start by understanding how face recognition works on Misty and in your code.

But first — what’s a skill?

When we talk about skills, we mean your JavaScript code, running locally on Misty. Local skills can be as simple as two files — a code file with the JavaScript Misty executes when the skill runs and a .json file with meta information about the skill. In local skills, events and callback functions handle most of the data from Misty’s sensors and processes. Misty’s JavaScript API provides methods for subscribing to event data, and you define the callbacks for handling this data in your skill code.

With that out of the way, let’s get to the fun stuff.

Understanding Face Recognition

Face recognition has changed how we interact with computers and people alike. It enables applications that range from absurd — like animating rainbow vomit in chat apps — to actually very useful — like searching your phone for pictures of a specific friend. As the tech has improved, face recognition has taken on critical roles in more parts of everyday life. Think about how many times you’ve unlocked your phone today by looking at the camera, or how face recognition has emerged (for better or worse) in video surveillance networks around the world.

Face recognition doesn’t just feel magical, it comes in handy for all kinds of robot jobs. Whether you’re coding Misty to deliver a package to the right recipient or to express joy when your spouse gets home, face recognition hugely multiplies her usefulness as a human companion.

Misty uses a module based on the Snapdragon Neural Processing Engine to detect and recognize faces captured by the camera in her visor. This process runs locally on each Misty robot, so she can use face recognition even without a connection to the internet. Check out this video for a much deeper dive into Misty’s computer vision capabilities.

Registering for Face Recognition Events

Before Misty can recognize people, she must be “trained” on their faces. To do this, you can use either her API or Misty’s API Explorer. The latter allows you to get started quickly if you don’t want to handle this in your own application. Once Misty knows your face, starting face recognition in a skill is as easy as:

misty.StartFaceRecognition();

This command activates Misty’s face recognition process so that Misty starts evaluating whether she recognizes the faces she sees. For your skill, however, that’s not quite enough. To use face recognition data in your code, you must register for the FaceRecognition event type. This is the event Misty sends whenever she sees a face, whether she recognizes it or not.

In the snippet below, the user-defined function registerFaceRec() does this registration. It starts with a call to misty.AddPropertyTest() to ensure FaceRecognition events only trigger the callback function when a face is actually in view, and not when registration or other types of messages are sent. This means there should be a valid string value for PersonName, be it the name for a trained face or “unknown person”.

We then call misty.RegisterEvent() to register to receive those specific FaceRecognition events.

function registerFaceRec(){
misty.AddPropertyTest("FaceRec", "PersonName", "exists", "", "string");
misty.RegisterEvent("FaceRec", "FaceRecognition", 1000, true);
}

registerFaceRec();

Creating a Face Recognition Callback

The next step is to define the callback for handling the face recognition event data. By default, callback functions have the same name you specified for the event, prefixed with an underscore. So, in this case, the callback for handling FaceRec data would be called _FaceRec().

When Misty detects but doesn’t recognize a face, the value of PersonName in the FaceRec data object is “unknown person”. Good security guards get suspicious when they see strangers where they don’t belong. For Misty to react appropriately, you might write a callback that looks like this:

function _FaceRec(data) {
// Check if the FaceRec event was triggered by a stranger
if (data.PropertyTestResults[0].PropertyValue == "unknown person"){
// Misty doesn't recognize this person. Sound the alarm!
} 
else {
// Misty knows this person. Do something else.
 }
}

If the face recognition data indicates an unknown person, we can have Misty perform security-related responses in the callback’s if block. If Misty does know the person, we can use the else block to have her behave warmly to her known companion.

The job at hand determines the complexity of your skill’s architecture. In the code for the security guard skill, the _FaceRec() callback triggers some fairly nuanced behavior; catching an intruder kicks off the execution of the other abilities we’ll discuss in this series. When you look at the full code file, you’ll also see helper functions, logic to handle edge cases (i.e. Misty won’t react to faces she detects unless they’re within a certain distance), and a loop that keeps the skill running indefinitely.

What’s Next?

In the next post, we’ll look at how Misty sends data to third-party cloud services. In the meantime, you can check out the full demo here. Head over to GitHub to see the skill code for yourself, or check out what else Misty developers are working in the Misty Community forum.