How to Integrate HUAWEI ML Kit's Hand Keypoint Detection Capability
A tutorial on how to use machine learning (ML) software to write code that can detect different parts of your hand and how they're moving in real-time.
Join the DZone community and get the full member experience.Join For Free
In the previous post, we looked at how to use HUAWEI ML Kit's skeleton detection capability to detect points such as the head, neck, shoulders, knees, and ankles. But as well as skeleton detection, ML Kit also provides a hand keypoint detection capability, which can locate 21 hand keypoints, such as fingertips, joints, and wrists.
Hand keypoint detection is useful in a huge range of situations. For example, short video apps can generate some cute and funny special effects based on hand keypoints, to add more fun to short videos.
Or, if smart home devices are integrated with hand keypoint detection, users could control them from a remote distance using customized gestures, so they could do things like activate a robot vacuum cleaner while they’re out.
Hand Keypoint Detection Development
Now, we’re going to see how to quickly integrate ML Kit's hand keypoint detection feature. Let’s take video stream detection as an example.
You can find detailed information about the preparations you need to make on the HUAWEI Developers-Development Process.
Here, we'll just look at the most important procedures.
1.1. Configure the Maven Repository Address in the Project-Level build.gradle File
1.2. Add SDK Dependencies to the App-Level build.gradle File
1.3. Add Configurations to the File Header
1.4. Add these Statements to the AndroidManifest.xml File so the Machine Learning Model can Automatically Update
1.5. Apply for Camera Permission and Local File Reading Permission
2. Code Development
2.1 Create a Hand Keypoint Analyzer
2.2 Create the HandKeypointTransactor Class for Processing Detection Results
This class implements the MLAnalyzer.MLTransactor<T> API and uses the
transactResult method in this class to obtain the detection results and implement specific services. In addition to coordinate information for each hand keypoint, the detection results include a confidence value for the palm and each of the keypoints. Palm and hand keypoints which are incorrectly detected can be filtered out based on the confidence values. You can set a threshold based on misrecognition tolerance.
2.3 Set the Detection Result Processor to Bind the Analyzer to the Result Processor
2.4 Create an Instance of the LensEngine Class
LensEngine class is provided by the HMS Core ML SDK to capture dynamic camera streams and pass these streams to the analyzer. The camera display size should be set to a value between 320x320 px and 1920x1920 px.
2.5 Call the run Method to Start the Camera and Read Camera Streams for Detection
2.6 Stop the Analyzer to Release Detection Resources Once the Detection is Complete
And that's it! We can now see hand keypoints appear when making different gestures. Remember that you can expand this capability if you need to.
To find out more, take a look at our official website: HUAWEI ML Kit.
For detailed information, please refer to HUAWEI Developers.
For more detailed instructions, please refer to the Documents.
You can join the HMS Core developer discussion by going to Reddit.
You can download the demo and sample code from GitHub.
If you experience any problems during the integration, you can find solutions on Stack Overflow.
Opinions expressed by DZone contributors are their own.