How to Integrate HUAWEI ML Kit's Hand Keypoint Detection Capability

A tutorial on how to use machine learning (ML) software to write code that can detect different parts of your hand and how they're moving in real-time.

Guo Tiecheng

Dec. 29, 20 · AI Zone · Tutorial

Like (1)

Save

2.22K Views

Introduction

In the previous post, we looked at how to use HUAWEI ML Kit's skeleton detection capability to detect points such as the head, neck, shoulders, knees, and ankles. But as well as skeleton detection, ML Kit also provides a hand keypoint detection capability, which can locate 21 hand keypoints, such as fingertips, joints, and wrists.

Application Scenarios

Hand keypoint detection is useful in a huge range of situations. For example, short video apps can generate some cute and funny special effects based on hand keypoints, to add more fun to short videos.

Or, if smart home devices are integrated with hand keypoint detection, users could control them from a remote distance using customized gestures, so they could do things like activate a robot vacuum cleaner while they’re out.

Hand Keypoint Detection Development

Now, we’re going to see how to quickly integrate ML Kit's hand keypoint detection feature. Let’s take video stream detection as an example.

1. Preparations

You can find detailed information about the preparations you need to make on the HUAWEI Developers-Development Process.

Here, we'll just look at the most important procedures.

1.1. Configure the Maven Repository Address in the Project-Level build.gradle File

    Java
   
xxxxxxxxxx

buildscript   {
     repositories {
      ...
         maven {url   'https://developer.huawei.com/repo/'}
     }
 }
  dependencies {
    ...
         classpath   'com.huawei.agconnect:agcp:1.3.1.300'
     }
 allprojects   {
     repositories {
      ...
         maven {url   'https://developer.huawei.com/repo/'}
     }
 }

1.2. Add SDK Dependencies to the App-Level build.gradle File

    Java
   
xxxxxxxxxx

dependencies{
 //   Import the base SDK.
     implementation   'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.2.300'
 //   Import the hand keypoint detection model package.
     implementation   'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.2.300'
 }

1.3. Add Configurations to the File Header

    Java
   
xxxxxxxxxx

apply plugin: 'com.android.application'
apply plugin: 'com.huawei.agconnect'

1.4. Add these Statements to the AndroidManifest.xml File so the Machine Learning Model can Automatically Update

    Java
   
xxxxxxxxxx

<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value= "handkeypoint"/>

1.5. Apply for Camera Permission and Local File Reading Permission

    Java
   
xxxxxxxxxx

<!--Camera permission-->
<uses-permission android:name="android.permission.CAMERA" />
<!--Read permission-->
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

2. Code Development

2.1 Create a Hand Keypoint Analyzer

    Java
   
xxxxxxxxxx

MLHandKeypointAnalyzerSetting setting = new MLHandKeypointAnalyzerSetting.Factory()
// MLHandKeypointAnalyzerSetting.TYPE_ALL   indicates that all results are returned.
// MLHandKeypointAnalyzerSetting.TYPE_KEYPOINT_ONLY   indicates that only hand keypoint information is returned.
// MLHandKeypointAnalyzerSetting.TYPE_RECT_ONLY   indicates that only palm information is returned.
setSceneType(MLHandKeypointAnalyzerSetting.TYPE_ALL)
// Set the maximum number of hand regions   that can be detected within an image. A maximum of 10 hand regions can be   detected by default.
setMaxHandResults(1)
create();
MLHandKeypointAnalyzer analyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer(setting);
create();

2.2 Create the HandKeypointTransactor Class for Processing Detection Results

This class implements the MLAnalyzer.MLTransactor<T> API and uses the transactResult method in this class to obtain the detection results and implement specific services. In addition to coordinate information for each hand keypoint, the detection results include a confidence value for the palm and each of the keypoints. Palm and hand keypoints which are incorrectly detected can be filtered out based on the confidence values. You can set a threshold based on misrecognition tolerance.

    Java
   
          x
         
public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
    @Override
    public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results){
        SparseArray<List<MLHandKeypoints>> analyseList = result.getAnalyseList();
// Determine detection result processing   as required. Note that only the detection results are processed.
// Other detection-related APIs provided   by ML Kit cannot be called.
    }
    @Override
    public void destroy() {
    // Callback method used to release   resources when the detection ends.
    }
}

2.3 Set the Detection Result Processor to Bind the Analyzer to the Result Processor

    Java
   
xxxxxxxxxx

analyzer.setTransactor(new HandKeypointTransactor());

2.4 Create an Instance of the LensEngine Class

The LensEngine class is provided by the HMS Core ML SDK to capture dynamic camera streams and pass these streams to the analyzer. The camera display size should be set to a value between 320x320 px and 1920x1920 px.

    Java
   
xxxxxxxxxx

LensEngine lensEngine = new LensEngine.Creator(getApplicationContext(), analyzer)
setLensType(LensEngine.BACK_LENS)
applyDisplayDimension(1280, 720)
applyFps(20.0f)
enableAutomaticFocus(true)
create();

2.5 Call the run Method to Start the Camera and Read Camera Streams for Detection

    Java
   
xxxxxxxxxx

// Implement other logic of the SurfaceView control by yourself.
SurfaceView mSurfaceView = findViewById(R.id.surface_view);
try {
    lensEngine.run(mSurfaceView.getHolder());
    } catch (IOException e) {
    // Exception handling logic.
}

2.6 Stop the Analyzer to Release Detection Resources Once the Detection is Complete

    Java
   
xxxxxxxxxx

if (analyzer != null) {
    analyzer.stop();
}
if (lensEngine != null) {
    lensEngine.release();
}

Demo Effect

And that's it! We can now see hand keypoints appear when making different gestures. Remember that you can expand this capability if you need to.

GitHub source code

To find out more, take a look at our official website: HUAWEI ML Kit.

For detailed information, please refer to HUAWEI Developers.

For more detailed instructions, please refer to the Documents.

You can join the HMS Core developer discussion by going to Reddit.

You can download the demo and sample code from GitHub.

If you experience any problems during the integration, you can find solutions on Stack Overflow.

Topics:

ai, machine learning, ml kit

Opinions expressed by DZone contributors are their own.

Comments