Identifying People in Photos Using Azure Cognitive Services

DZone 's Guide to

Identifying People in Photos Using Azure Cognitive Services

Learn about not just detecting faces in photos but actually identifying people in photos by using an Azure Cognitive Services sample application.

· AI Zone ·
Free Resource

My previous post about Azure Cognitive Services described how to detect faces on photos. In this post, I will focus on identifying people in photos. This post is based on my Azure Cognitive Services sample application that has most of the Face API support implemented. The goal is to describe identifying process in brief and also show some code I have written.

How Azure Cognitive Services Identify People

Face API has its own mechanism to identify people of photos. In short, it needs a set of analyzed photos about people to identify them.

Face API: Identifying people

Before identifying people, we need to introduce their faces to cognitive services. With our brand new cognitive services account, we start with creating a Person group. We may have groups like Family, Friends, Colleagues, etc. After creating a group, we add people to the group and introduce up to 10 photos per person to Cognitive Services. After, this group must be “trained.” Then, we'll be ready to identify people on photos.

Training means that the cloud service analyzes face characteristics detected before and creates some mystery to better identify these people in photos. After adding or removing photos, a person group must be trained again. Here is the example of points that define a face for Face API. It’s possible to draw polyline through sets of points to draw out parts of the face like lips, nose, and eyes.

Face API: Face landmarks

Getting Started With Face API

Before using Cognitive Services, we need access to Microsoft Azure and we need a Cognitive Services Faces API account. Faces API has a free subscription that offers more API calls than it is really needed to get started and build the first application that uses face detection.

Azure Portal: Add Face API subscription

Code alert! I am working on sample application called CognitiveServicesDemo for my coming speaking engagements. Although it is still pretty raw, it can be used to explore and use the Faces API of Cognitive Services.

After creating a Face API account, there are API keys available. These keys with a Face API service endpoint URL are needed to communicate with Face API in a sample application.

Person Groups and People

Here are some screenshots from my sample application. The first one shows a list of Person groups and the other one, people in the Family group.

Cognitive Services: List of person groups

Cognitive Services: People in family group

The Family group has some photos and it is trained already. Training is easy to do in code. Here is the controller action that takes the Person Group ID and lets FaceServiceClient train the given person group.

public async Task<ActionResult> Train(string id)
    await FaceClient.TrainPersonGroupAsync(id);

    return RedirectToAction("Details", new { id = id });

Most of the calls to Face API are simple ones and don’t get very ugly. Of course, there are few exceptions, like always.

Identifying People

To identify people, I use one photo that was taken in in Chisinau, Moldova when my daughter was very small.

Gunnar Peipman @ Chisinau with family

To identify who is in the photo, three service calls are needed:

  1. Detect faces from the photo and save face rectangles.

  2. Identify faces from given person group based on detected face rectangles.

  3. Find names of people from person group.

Identifying is a little bit crucial as we don’t always get exact matches for people but also so called candidates. It means that identifying algorithm cannot make accurate decision which person of two or three possible candidates is shown in face rectangle.

Here is the controller action that does the identification. If it is a GET request, then the form with an image upload box and person groups selection is shown. In the case of POST, it is expected that there is an image to analyze. I did some base controller magic to have a copy of the uploaded image available for requests automagically. RunOperationOnImage is the base controller method that creates a new image stream and operates on it because Face API methods dispose given image streams automatically.

public async Task<ActionResult> Identify()
    var personGroupId = Request["PersonGroupId"];
    var model = new IdentifyFacesModel();

    var groups = await FaceClient.ListPersonGroupsAsync();
    model.PersonGroups = groups.Select(g => new SelectListItem
                                            Value = g.PersonGroupId,
                                            Text = g.Name

    if (Request.HttpMethod == "GET")
        return View(model);

    Face[] faces = new Face[] { };
    Guid[] faceIds = new Guid[] { };
    IdentifyResult[] results = new IdentifyResult[] { };

    await RunOperationOnImage(async stream =>
        faces = await FaceClient.DetectAsync(stream);
        faceIds = faces.Select(f => f.FaceId).ToArray();

        if (faceIds.Count() > 0)
            results = await FaceClient.IdentifyAsync(personGroupId, faceIds);

    if (faceIds.Length == 0)
        model.Error = "No faces detected";
        return View(model);

    foreach (var result in results)
        var identifiedFace = new IdentifiedFace();
        identifiedFace.Face = faces.FirstOrDefault(f => f.FaceId == result.FaceId);

        foreach(var candidate in result.Candidates)
            var person = await FaceClient.GetPersonAsync(personGroupId, candidate.PersonId);
            identifiedFace.PersonCandidates.Add(person.PersonId, person.Name);

        identifiedFace.Color = Settings.ImageSquareColors[model.IdentifiedFaces.Count];

    model.ImageDump = GetInlineImageWithFaces(model.IdentifiedFaces.Select(f => f.Face));
    return View(model);

Here is the result of identifying people by my sample application. Me, my girlfriend, and our minion were all identified successfully.

People detected by Faces API

My sample application also introduces how to draw rectangles of different colors around detected faces. So, take a look at it.

Wrapping Up

Using Azure Cognitive Services to detect faces on photos and to identify people is actually simple. Although there are REST services available, Microsoft also provides us with well-designed API packages to use Face API services. One interesting thing to notice: an image to be analyzed is in no way the matter on the API client — it handles the image only as a stream of bytes, meaning that there are no dependencies to graphics and image processing libraries. I have a working sample application called CognitiveServicesDemo available for those who want to get more familiar with Azure Cognitive Services, and it is a good place to start.

ai ,image recognition ,azure cognitive services ,tutorial ,face api

Published at DZone with permission of Gunnar Peipman , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}