Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Age and Gender Recognition With JavaCV and Neural Networks

DZone's Guide to

Age and Gender Recognition With JavaCV and Neural Networks

Learn about all that JavaCV and convolutional neural networks are capable of, including performing age and gender recognition on webcam images in real-time.

· AI Zone
Free Resource

Find out how AI-Fueled APIs from Neura can make interesting products more exciting and engaging. 

We all have heard about OpenCV, the famous C++ library for computer vision-related applications. Since it is written in C++, many wrappers/bindings (such as OpenCV Python) have been created to provide the functionalities provided by the OpenCV library to other languages. There's also emgucv, the .NET wrapper for OpenCV. OpenCV has a Java binding, too, but that is not JavaCV. JavaCV is more than a Java binding.

JavaCV is a collection of wrappers for many famous libraries used for image processing and computer vision-related applications. JavaCV includes wrappers for OpenCV, FFmpeg, OpenKinect, and much more. JavaCV has everything required for computer vision and image processing in one place. Currently, JavaCV has released its latest version (1.3) of wrappers which support the OpenCV 3.1 through JNI. Even though JavaCV comes with a lot of features (OpenCV and much more), it is quite less famous among developers. Therefore, I've decided to put some light on JavaCV and expose what it's capable of doing. Let’s get started!

Here, we will be doing some face recognition/detection stuff and furthermore and use convolutional neural networks (CNNs) for age and gender predictions. Since most of you have seen how to do face detection using Haar cascades and how to do face recognition using fisherfaces and so on, the interesting part will be the usage of CNNs for age and gender predictions. First, let’s create a Maven project and add a JavaCV dependency as follows. I will be using JavaCV version 1.2. The latest version available is JavaCV 1.3, which has several improvements over the version that I’m using here.

    org.bytedeco
    javacv
    1.2

Capturing Video From a Camera

To capture the video, I'm using the FFmpegFrameGrabber shipped with JavaCV. As I have read in several places, this is favored over the conventional OpenCVFrameGrabber when you are using a Linux machine.First, we have to create and configure the frame grabber.

frameGrabber = new FFmpegFrameGrabber("/dev/video0");
frameGrabber.setFormat("video4linux2");
frameGrabber.setImageWidth(1280);
frameGrabber.setImageHeight(720);

Here, the parameter taken in by the constructor of FFmpegFrameGrabber is the camera device (“/dev/video0” is our default laptop webcam; if you connect another camera, it will be identified by the Linux system as “/dev/video01” and so on). Since I’m a Linux fan, I have done this example on an Ubuntu machine. The camera device and format (“video4linux2”) may differ in Windows environments. However, the steps to be followed are still the same.

Next, the captured frame’s width and height are specified. If you have a 720p HD camera, this resolution is optimal. Once set up, we can start the frame grabber.

frameGrabber.start();

Once started, we have to repeatedly grab frames from the frame grabber as follows.

boolean running = true;
while (running) {
    try {
        // Here we grab frames from our camera
        final Frame frame = frameGrabber.grab();

    } catch (FrameGrabber.Exception e) {
        logger.error("Error when grabbing the frame", e);
    } 
}

In the above code, we are capturing frames from the webcam. These frames are not shown in a UI, so let’s do that part along with adding the gender and age recognition functionality to our code.

Face Detection With Haar Cascades

This is a part most of us at least have heard of. OpenCV/JavaCV provide direct methods to import Haar cascades and use them to detect faces. I will not be explaining this part in deep detail. You can refer this class in GitHub and this blog post to learn more on using Haar cascades to detect faces. I have written the HaarFaceDetector class to detect faces.

The following code snippet initializes the Haar cascade classifier:

File haarCascade = new File(this.getClass().getResource("/detection/haarcascade_frontalface_alt.xml").toURI());

haarClassifierCascade = new opencv_objdetect.CvHaarClassifierCascade(cvLoad(haarCascade.getAbsolutePath()));

This method detects faces in a grabbed frame and crops the faces. I have put the cropped faces along with the coordinates of the faces (as CvRect objects) into a Map.

public Map<CvRect, Mat> detect(Frame frame) {
    Map<CvRect, Mat> detectedFaces = new HashMap<>();

    IplImage iplImage = iplImageConverter.convert(frame);

    /*
     * return a CV Sequence (kind of a list) with coordinates of rectangle face area.
     * (returns coordinates of left top corner & right bottom corner)
     */
    CvSeq detectObjects = cvHaarDetectObjects(iplImage, haarClassifierCascade, storage, 1.5, 3, CV_HAAR_DO_CANNY_PRUNING);
    Mat matImage = toMatConverter.convert(frame);

    int numberOfPeople = detectObjects.total();
    for (int i = 0; i < numberOfPeople; i++) {
        CvRect rect = new CvRect(cvGetSeqElem(detectObjects, i));
        Mat croppedMat = matImage.apply(new Rect(rect.x(), rect.y(), rect.width(), rect.height()));
        detectedFaces.put(rect, croppedMat);
    }

    return detectedFaces;
}

That’s it for face detection. Let’s see how these detected faces are processed to detect gender and age.

Gender Recognition With CNN

Gender recognition using OpenCV's fisherfaces implementation is quite popular. But, in this example, I will be using a different approach to recognize gender. This method was introduced by two Israel researchers, Gil Levi and Tal Hassner, in 2015. I have used the CNN models trained by them in this example. We are going to use the OpenCV’s DNN (Deep Neural Networks) package.

In the DNN package, OpenCV/JavaCV has provided a class called Net that can be used to populate a neural network. Furthermore, this packages supports importing neural network models from well-known deep learning frameworks like Caffe, TensorFlow, and Torch. Levi and Hassner published their CNN models as Caffe models. Therefore, we will be using CaffeImporter to import that model into our application.

A Caffe model has two associated files:

  1. .prototxt: The definition of the CNN goes in here. This file defines the layers in the neural network and each layer’s inputs, outputs, and functionality.
  2. .caffemodel: This contains the information of the trained neural network (trained model).

The following code segment is responsible for loading the CNN definitions and the trained models (CNNGenderDetector.java):

genderNet = new Net();
File protobuf = new File(getClass().getResource("/caffe/deploy_gendernet.prototxt").toURI());
File caffeModel = new File(getClass().getResource("/caffe/gender_net.caffemodel").toURI());
Importer importer = createCaffeImporter(protobuf.getAbsolutePath(), caffeModel.getAbsolutePath());
importer.populateNet(genderNet);
importer.close();

“deploy_gendernet.prototxt” contains the definition of the CNN responsible for gender recognition and “gender_net.caffemodel” contains the trained Caffe model. Note the following lines...

createCaffeImporter(protobuf.getAbsolutePath(),caffeModel.getAbsolutePath())
importer.populateNet(genderNet)

...which do the importing and population of the CNN.

I wrote detectGender(Mat face, Frame frame) in the CNNGenderDetector class to detect the gender of faces. The first parameter is the cropped face of which we want to detect the gender and the second parameter is the captured frame (used for color and depth information inside the method) in which we detected the corresponding face.

Following is the processing part:

Mat croppedMat = new Mat();
resize(face, croppedMat, new Size(256, 256));
normalize(croppedMat, croppedMat, 0, Math.pow(2, frame.imageDepth), NORM_MINMAX, -1, null);

Blob inputBlob = new Blob(croppedMat);
genderNet.setBlob(".data", inputBlob);
genderNet.forward();
Blob prob = genderNet.getBlob("prob");

Indexer indexer = prob.matRefConst().createIndexer();
logger.debug("CNN results {},{}", indexer.getDouble(0, 0), indexer.getDouble(0, 1));
if (indexer.getDouble(0, 0) > indexer.getDouble(0, 1)) {
    logger.debug("Male detected");
    return Gender.MALE;
} else {
    logger.debug("Female detected");
    return Gender.FEMALE;
}

First, I resize the face (of type Mat passed as a method parameter) and fill the resized Mat to a new Mat. This is because the trained model has been trained to take inputs of size 256x256. Therefore, I resize the face Mat to that size. I have normalized the Mat.

Then, I create a Blob using the face Mat  and set that Blob as the input of the first layer of the CNN. .data layer. genderNet.forward() will then move the input through the layers of the neural network. Finally, we can take the output of the last layer of the CNN, which is the probability layer (prob). In JavaCV, we have to use indexers to iterate matrices and blobs since JavaCV is just using the JNI to invoke OpenCV methods. This gender recognition CNN outputs two values which are indexed as (0,0) and (0,1) in a one-dimensional matrix. The value at (0,0) corresponds to the probability of the face being a male and (0,1) to the probability of it being female. Since these values sum up to 1, the greater one will be the predicted gender.

The complete class is available here.

Age Recognition With CNN

This is almost similar to the gender detection part, except that the corresponding prototxt file and the Caffe model file are “deploy_agenet.prototxt” and “age_net.caffemodel”. Furthermore, the CNN’s output layer (probability layer) in this CNN consists of eight values for eight age classes (“0–2”, “4–6”, “8–13”, “15–20”, “25–32”, “38–43”, “48–53”, and “60-"). The CNNAgeDetector class is written to process detected faces and predict their age classes.

Mat resizedMat = new Mat();
resize(face, resizedMat, new Size(256, 256));
normalize(resizedMat, resizedMat, 0, Math.pow(2, frame.imageDepth), NORM_MINMAX, -1, null);

Blob inputBlob = new Blob(resizedMat);
ageNet.setBlob(".data", inputBlob);
ageNet.forward();
Blob prob = ageNet.getBlob("prob");

DoublePointer pointer = new DoublePointer(new double[1]);
Point max = new Point();
minMaxLoc(prob.matRefConst(), null, pointer, null, max, null);
return AGES[max.x()];

Similarly to the gender recognition scenario, first I resize the face (of type Mat passed as a method parameter) and fill the resized Mat to a new Mat because the model has been trained to take inputs of size 256x256. Now, I have normalized the Mat.

Then, I create a Blob using our face Mat and set that Blob as the input of the first layer of the CNN. .data layer. ageNet.forward() will then move the input through the layers of the neural network. Finally, we can take the output of the last layer of the CNN which is the probability layer (prob). minMaxLoc(prob.matRefConst(), null, pointer, null, max, null) gets the predicted age class. The neural network returns an array of eight values consisting of probabilities of each age class being the predicted age of the user. The complete class can be found here.

Putting Things Together

I have created the main class JavaCVExample to put together the things we have done so far and come up with a small UI application. You can refer that class and understand what I have done there. I have also put some text around detected faces stating the predicted gender and age.

Try It Out

Now we are ready to run our application. Let’s try this out. The main method is located in the JavaCVExample class. When you run this class, your webcam will be switched on and the feed captured through that camera will be processed by our neural networks.

I would love to put a screen shot of myself being detected by my application, but why would I ruin your nice day?! You will see a red box around your face and the gender and age label will be shown in real-time. Gender prediction is considerably accurate while the age prediction is not that accurate. However, age prediction can also predict the nearest class that matches your age.

Hopefully, you can follow the entire article and get the same results. The complete source code is available here. If you have any questions, feel free to comment. I will answer them as best as I can. Thank you for reading!

To find out how AI-Fueled APIs can increase engagement and retention, download Six Ways to Boost Engagement for Your IoT Device or App with AI today.

Topics:
opencv ,javacv ,convolutional neural networks ,ai ,tutorial ,facial recognition

Published at DZone with permission of Imesha Sudasingha. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}