Building a Hand Gesture Game

Introduction

Have you already piled the products you want to buy into your online shopping cart? What if you're feeling a little more frugal and want to save your hard-earned cash instead? Fortunately, there is Crazy Shopping Cart, a fast-paced and addictive virtual shopping game that integrates HUAWEI ML Kit's hand keypoint detection service to allow you to control a virtual shopping cart using hand gestures. In this post, I'll show you how the game's hand gesture control feature was developed using ML Kit.

How to Play

Players perform hand gestures to control the left-right movement of a shopping cart and collect as many items as possible. The cart speeds up every 15 seconds.

Looks fun right? Let's take a look at how the game's hand gesture control system is developed.

Development Process

1. Configure the Maven repository address.

Open the build.gradle file in the root directory of your Android Studio project.

xxxxxxxxxx

buildscript {

    repositories {

        google()

        jcenter()

        maven {url 'https://developer.huawei.com/repo/'}

    }

    dependencies {

        ...

        classpath 'com.huawei.agconnect:agcp:1.4.1.300'

    }

}

allprojects {

    repositories {

        google()

        jcenter()

        maven {url 'https://developer.huawei.com/repo/'}

    }

}

2. Perform integration in full SDK mode. 

xxxxxxxxxx

dependencies{

    // Import the base SDK.

    implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'

    // Import the hand keypoint detection model package.

    implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'

}

After integrating the SDK, add the following configuration to the file header:

Add apply plugin: 'com.huawei.agconnect' after apply plugin: 'com.android.application'.

3. Create a hand keypoint analyzer. 

xxxxxxxxxx

MLHandKeypointAnalyzer analyzer =MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer();

4. Create the detection result processing class HandKeypointTransactor. 

xxxxxxxxxx

public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {

    @Override

    public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {

        SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();

        // Process the detection result as required.

        // Other detection-related APIs provided by ML Kit cannot be called.

    }

    @Override

    public void destroy() {

        // Callback method used to release resources when detection completes.

    }

}

5. Set the detection result processor, and bind it to the analyzer. 

xxxxxxxxxx

analyzer.setTransactor(new HandKeypointTransactor());

6. Create a LensEngine instance. 

xxxxxxxxxx

LensEngine lensEngine = new LensEngine.Creator(getApplicationContext(), analyzer)

    .setLensType(LensEngine.BACK_LENS)

    .applyDisplayDimension(1280, 720)

    .applyFps(20.0f)

    .enableAutomaticFocus(true)

    .create();

7. Call the run method to start the camera and read video streams for detection. 

xxxxxxxxxx

// Implement other logic of the SurfaceView control by yourself.

SurfaceView mSurfaceView = findViewById(R.id.surface_view);

try {

    lensEngine.run(mSurfaceView.getHolder());

} catch (IOException e) {

    // Handle exceptions.

}

8. After the detection is complete, stop the analyzer, and release detection resources. 

xxxxxxxxxx

if (analyzer != null) {

    analyzer.stop();

}

if (lensEngine != null) {

    lensEngine.release();

}

Summary

As you can see, the development process is really simple and fast. In addition to adding hand gesture control to your mobile games, ML Kit's hand keypoint detection service also has many other useful applications. For example, users can add cute or funny special effects to their videos when using short-video apps with this service integrated. Also, users of smart home apps with this service integrated can customize the hand gestures they use to remotely control appliances.

Try the service out for yourself now by integrating HUAWEI ML Kit with your own apps.

GitHub Demo

References

Official website of Huawei Developers

Development Guide

HMS Core official community on Reddit

Demo and sample code

Discussions on Stack Overflow