Raspberry Pi, OpenCV, Deep Neural Networks, and — Of Course— a Bit of Clojure

I had to write a simple IoT prototype recently that counted the number of people in a queue in real-time. Of course, I could have hired someone to do that and just keep counting people, or ... I could write a program in Clojure using a Raspberry Pi to detect the number of heads via a video stream.

You may also like: IoT OpenCV Scripting With Clojure on a Raspberry Pi

We learned recently that when using inlein, you can easily write scripts in Clojure with dependencies and run them just about anywhere, at a quite decent speed.

Now, the other good news is that Origami just released a version of its DNN sibling, origami-dnn. This new version comes with you need to get started using deep neural networks in Clojure.

Clojure-based Origami DNN, which is a wrapper around OpenCV's DNN features, does three things rather nicely:

• It makes a set of known trained networks immediately available for use
• It integrates wonderfully with Origami, so you can immediately use returned results from the network with your image, videos read from the file, or video streams from a webcam.
• It also focuses on simply extending the Clojure core threading macro — with usage from Origami — and intuitively plug itself in.

Of course, since it's its sibling, there's no need to install OpenCV or compile anything; this works on Raspberry, OSX, Windows, Linux, etc... the binaries are pre-compiled and bundled, ready to be used transparently.

To set expectations, the goal of this article will not talk about training a network yet, only how to use a pre-trained network on a Raspberry Pi.

Getting Started

We mostly need the JDK, and again, here, I like the ARM-ready JDKs packaged by Bellsoft. There is virtually no noticeable speed difference between most recent JDKs — whether you are talking about JDK 8 or JDK13, they are mostly running at the same performance levels, IMHO.

The other thing we need is something that runs the Clojure scripts — the daemon that makes us ready to run Clojure scripts, and this is where inlein comes in.

Installing inlein (Again)

Just to be sure — and because apart from the Java install, this is the only thing you need — download the inlein scripts and put them somewhere on a path ready for usage. For example, something on Raspberry /usr/local/bin is cool.

wget https://github.com/hypirion/inlein/releases/download/0.2.0/inlein
chmod +x inlein
mv inlein /usr/local/bin

We're mostly there. Now, let's get some Clojure scripts from GitHub.

Getting the Scripting Samples (Again)

Some Origami and Origami DNN scripting samples are located on GitHub, and you can clone them with:

git clone https://github.com/hellonico/origami-scripting.git

Notably, the first example we will be looking at is this one.

Running a DNN Network on an Image

Yolo v3 may not be the fastest network to perform object detection, but it's still one of my favorites. Our first goal is to run a Yolo pre-trained network, the one provided if you do a local yolo install, to recognize and classify a cat.

Sounds like your usual Neural Network exercise, and yes, we just want to make sure things are kept simple.

The core of the example is in the following few lines below, a simple threading usage on an input image. Let's drop this here, analyze what it does, and see how it does it.

(-> input
        (imread)
        (yolo/find-objects net)
        (blue-boxes! labels)
        (imwrite output))

input is a path to an image that you then feed to imread or  u/mat-from-url of the Origami library.

I think eventually those two functions will be merged into a single one at some stage, so it does not matter whether the path to the image is local or not.

We then thread the OpenCV's mat object through  yolo/find-objects. That function from origami-dnn internally converts the origami/opencv mat to a blob image in a format based on the number of channels, the order of the channels, sie of the picture, etc., as expected by the Yolo network.

The network is always trained on pre-formatted and prepared images. Here, when using Yolo, the images are 416x416, and this is, indeed, the size of the input matrix used to feed the network.

(d/blue-boxes! labels) or (blue-boxes! labels) gets the results from the find-objects call, which is:

• The image itself, so it can be piped through a threading macro for further usage
• A sequence of maps with keys:  {:label :box :confidence}, one map per detected object.

The labels themselves are retrieved when loading the network. Oh yes, that's right, we haven't seen that part yet. The network is loaded from a remote repository using  read-net-from-repo. This downloads the network files and puts them on the local device/machine for usage or re-usage. You only need to download the network files once. Later, the runs will re-use the locally cached version.

(let [[net opts labels] (read-net-from-repo "networks.yolo:yolov2-tiny:1.0.0")]
      ; use the network on an image here 
)

When using  read-net-from-repo, you retrieve three variables:

•  net— this is an OpenCV object ready to be acted upon
•  opt— there are read from a custom edn file and can be used to tweak runs of the network
• labels — I still can't believe how difficult it can be to find labels used when training a network, so sometimes, the output just doesn't even make sense.

Ok, so we loaded the network. Then, we threaded the result of the network run onto blue-boxes! You can write your own blue-boxes. The function is simply looping (doseqin Clojure) over the results and drawing a blue rectangle along by adding the label and confidence, as is usually done. You can, of course, set a color of your choice, depending on the label, quite easily with a map or something similar.

(defn blue-boxes! [result labels]
  (let [img (first result) detected (second result)]
  (doseq [{confidence :confidence label :label box :box} detected]
    (put-text! img (str (nth labels label) "[" confidence " %]") 
       (new-point (double (.-x box)) (double (.-y box))) 
       FONT_HERSHEY_PLAIN 2 
       (new-scalar 255 0 0))
    (rectangle img box (new-scalar 255 0 0) 2))
  img))

Running a DNN Network on a Webcam

If you've previously used Origami to conduct stream processing, you realize it is quite easy to plug-in the object detection on an image, as was done above, directly on the Mat read from the video stream.

Cutting out the imports, the snippet below is enough to do object detections on a video stream:

(let [[net _ labels] (origami-dnn/read-net-from-repo "networks.yolo:yolov3-tiny:1.0.0")]
    (u/simple-cam-window
    {:frame {:fps true} :video {:device 0}}
     (fn [buffer]
       (-> buffer 
             (yolo/find-objects net) 
             (d/blue-boxes! labels) ))))

Note that we just plugged in the u/simple-cam-window to retrieve the stream and have access to a Mat object— here named buffer. We then apply the samefind-objects and blue-boxes! functions.

Other Packaged Networks

Some of the other ready-to-use networks are listed below. You can just cut and replace the ones used in the examples above...

For places365, convnet's gender, and convnet's age, you will need slightly different functions to use the return values of the network, so be sure to have a closer look. Additionally, more demos are available if you want to try and plug in your own.

Share with us in the comments below what you detected using origami-dnn.

Further Reading

IoT OpenCV Scripting With Clojure on a Raspberry Pi

OpenCV + Apache MiniFi for IoT

How to Deploy OpenCV on Raspberry Pi and Enable Machine Vision