Taking Pictures: Misty as a Security Guard: Part 3

Welcome back!

In Part 1 of Misty as a security guard, we learned how Misty uses facial recognition to detect unknown people in your space. Part 2 focused on Misty’s use of web services to communicate with IoT devices. So what’s left?

In Part 3, we learn how security guard Misty photographs intruders and transforms their portraits with a Raspberry Pi.

The “camera” category of robot sensors includes a diverse collection of hardware, and these sensors enable an assortment of robot capabilities. Learning to recognize objects? Mapping a new environment? Analyzing the chemical makeup of Martian soil? Whatever your task, there’s a camera for you.

Next to all the innovative use cases for cameras in robotics, taking pictures might seem tame. But photography can be a shared language between us humans and our high-tech companions. Our human ability to interpret visual images comes standard issue — no additional computation required. Pair this notion with a robot’s ability to go where you can’t and you unlock a bit of magic.

Misty’s cameras include a wide-angle grayscale camera, a depth camera, and a 13-megapixel color camera. She can’t use this hardware to vaporize rocks on Mars (yet), but she’s up for printing pictures of intruders she catches in your space. Ready to see how she does it?

Taking the Photo

Misty’s API provides raw access to data from each of Misty’s cameras. The security guard job uses the 13-megapixel color camera. In Misty’s skills, you take a picture with this camera by calling the misty.TakePicture() command. This command accepts arguments that tell Misty how to handle the captured photo, including:

• whether to return the image data as a Base64 string
• what name to save the file with
• whether to resize the captured photo
• whether to display the photo on her screen
• whether to overwrite an existing image with the same filename as the new photo

In the security guard skill, misty.TakePicture() is called in the block that executes when Misty doesn’t recognize a face. Misty saves the photo with the name "Intruder" and sets the size of the image to 1200 x 1600 pixels. The photo doesn’t appear on Misty’s display, and if an image called Intruderalready exists, Misty overwrites it.

With these arguments in place, the command looks like this:

misty.TakePicture(false, "Intruder", 1200, 1600, false, true);

When this command executes, Misty takes a portrait of your intruder, who probably looks a little bewildered by your home robot security system. The next step is for Misty to do something interesting with the photo. For us, that means sending the image to a Raspberry Pi and printing a mugshot.

Printing the Photo

Before the Raspberry Pi can print the photo, Misty needs to communicate that she’s captured a new portrait. She does this by calling the misty.SendExternalRequest() command to send an update to Dweet.io. Dweet is a web service that internet-connected devices can use to post data objects (called “dweets”) online. Misty sends this request to Dweet after executing misty.TakePicture(). 

misty.SendExternalRequest("POST", "https://dweet.io/dweet/for/misty",null,null,null,"{\"status\":\"Intruder_Alarm_On\"}");

When the security guard skill runs, you can see Misty’s latest dweet at https://dweet.io/get/latest/dweet/for/misty. It looks something like this:

{
  "this": "succeeded",
  "by": "getting",
  "the": "dweets",
  "with": [
    {
      "thing": "misty",
      "created": "2019-03-06T06:20:49.288Z",
      "content": {
        "status": "Intruder_Alarm_On"
      }
    }
  ]
}

Meanwhile, a Raspberry Pi on the same WiFi network as Misty runs a Python script with the logic for handling the mugshots Misty captures. This script uses libraries like Dweepy and OpenCV to check for dweets from Misty and manipulate intruder portraits in interesting ways.

When this script runs, it calls dweepy.get_latest_dweet_for_('misty') on a loop to check for updates from your robot. Misty only dweets when she catches an intruder, so the Raspberry Pi looks at the value of the "created" field each time it checks a dweet. When it discovers a new value in this field, it executes the code to download and print the image:

while True:
    try:
        time.sleep(1.5)
        response = dweepy.get_latest_dweet_for('misty')
        new_status = str(response[0]["created"]) 
        print("NOW:  ",new_status)
        if new_status != old_status:
            old_status = new_status
            print("Trigger Alarm")
            print("Preparing Image")
            prepare_image()
            print("Connecting to printer network")
            x = switch_to_printer_network()
            print("Printing Image")
            print_image()
            print("Connecting to robots network")
            x = switch_to_robot_network()

    except KeyboardInterrupt:
        print ("Program Stopped")
        break

    except:
        pass

The bulk of this action happens in the prepare_image() function (defined on line 35 here). This function downloads the photo of the captured intruder via Misty’s REST endpoint for the GetImage command. It uses the scikit-image library to save the image to a variable, like this:

url_path = "http://"+robot_ip+"/api/images?FileName=Intruder.jpg&Base64=false"
image = io.imread(url_path)

After the image downloads, the prepare_image() function overlays the Misty Robotics logo onto the portrait without covering up the intruder’s face. It performs a similar calculation to position the word “Busted,” “Gotcha,” or “Wanted” on the mugshot. The details of these calculations are beyond the scope of this post, but if you’re interested in seeing how it works, check out the code for this script.

With the image prepared, the Raspberry Pi connects to a printer and calls a function to print the new mugshot. It then reconnects to the robot’s network and jumps back into the loop to check for new dweets. Watch the full video to see it in action:

Misty’s security guard job builds on her core capabilities of face recognition, external HTTP calls, and image capture. In this series, we’ve learned about the technology behind each of these capabilities and explored how you can use Misty’s API to build a security guard skill yourself.

Be sure to check out the full repository for the security guard skill on GitHub, and browse the Misty Sample Skills organization for even more Misty skills. Have an idea for another job for Misty? Don’t keep it to yourself — share it with the Misty Community!

Read this article (and others) on the Misty Robotics blog.