How to Develop an AR-Based Health Check App

Now that spring has arrived, it's time to get out and stretch your legs! As programmers, many of us are used to being seated for hours and hours at a time, which can lead to back pain and aches. We're all aware that building a workout plan, and keeping track of health indicators around-the-clock, can have enormous benefits for body, mind, and soul. 

Fortunately, AR Engine makes that remarkably easy. It comes with face tracking capabilities, and will soon support body tracking as well. Thanks to core AR algorithms, AR Engine is able to monitor heart rate, respiratory rate, facial health status, and heart rate waveform signals in real-time during your workouts. You can also use it to build an app, for example, to track the real-time workout status, perform real-time health checks for patients, or monitor real-time health indicators of vulnerable users, like the elderly or the disabled. With AR Engine, you can make your health or fitness app more engaging and visually immersive than you might have believed possible. 

Advantages and Device Model Restrictions

1. Monitors core health indicators like heart rate, respiratory rate, facial health status, and heart rate waveform signals in real-time. 
2. Enables devices to better understand their users. Thanks to technologies like Simultaneous Localization and Mapping (SLAM) and 3D reconstruction, AR Engine renders images to build 3D human faces on mobile phones, resulting in seamless virtual-physical cohesion. 
3. Supports all of the device models listed in Software and Hardware Requirements of AR Engine Features. 

Demo Introduction

A simple demo is available to give you a grasp of how to integrate AR Engine, and use its human body and face tracking capabilities. 

• ENABLE_HEALTH_DEVICE: indicates whether to enable health checks. 
• HealthParameter: health check parameter, including heart rate, respiratory rate, age and gender probability based on facial features, and heart rate waveform signals. 
• FaceDetectMode: face detection mode, including health rate checking, respiratory rate checking, real-time health checking, and all of the three above. 

Effect

The following details how you can run the demo using the source code.

Key Steps

1. Add the Huawei Maven repository to the project-level build.gradle file.

buildscript {
    repositories {
        maven { url 'http://developer.huawei.com/repo/'}
    }
dependencies {
        ...
        // Add the AppGallery Connect plugin configuration.
        classpath 'com.huawei.agconnect:agcp:1.4.2.300'
    }
}allprojects {
    repositories {
        maven { url 'http://developer.huawei.com/repo/'}
    }
}

2. Add dependencies on the SDK to the app-level build.gradle file. 

implementation 'com.huawei.hms:arenginesdk:3.7.0.3'

3. Declare system permissions in the AndroidManifest.xml file. 

<uses-permission android:name="android.permission.CAMERA" />

4. Check whether AR Engine has been installed on the current device. If yes, the app can run properly. If not, the app automatically redirects the user to AppGallery to install AR Engine. 

boolean isInstallArEngineApk = AREnginesApk.isAREngineApkReady(this);
if (!isInstallArEngineApk && isRemindInstall) {
    Toast.makeText(this, "Please agree to install.", Toast.LENGTH_LONG).show();
    finish();
}
if (!isInstallArEngineApk) {
    startActivity(new Intent(this, ConnectAppMarketActivity.class));
    isRemindInstall = true;
}
return AREnginesApk.isAREngineApkReady(this);

Key Code

1. Call ARFaceTrackingConfig and create an ARSession object. Then, set the human face detection mode, configure AR parameters for motion tracking, and enable motion tracking.

mArSession = new ARSession(this);
mArFaceTrackingConfig = new ARFaceTrackingConfig(mArSession);
mArFaceTrackingConfig.setEnableItem(ARConfigBase.ENABLE_HEALTH_DEVICE);
mArFaceTrackingConfig
    .setFaceDetectMode(ARConfigBase.FaceDetectMode.HEALTH_ENABLE_DEFAULT.getEnumValue());

2. Call FaceHealthServiceListener to add your app and pass the health check status and progress. Call handleProcessProgressEvent() to obtain the health check progress. 

mArSession.addServiceListener(new FaceHealthServiceListener() {
    @Override
    public void handleEvent(EventObject eventObject) {
        if (!(eventObject instanceof FaceHealthCheckStateEvent)) {
            return;
        }
        final FaceHealthCheckState faceHealthCheckState =
            ((FaceHealthCheckStateEvent) eventObject).getFaceHealthCheckState();
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                mHealthCheckStatusTextView.setText(faceHealthCheckState.toString());
            }
        });
    }
    @Override
    public void handleProcessProgressEvent(final int progress) {
        mHealthRenderManager.setHealthCheckProgress(progress);
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                setProgressTips(progress);
            }
        });
    }
}); 
private void setProgressTips(int progress) {
    String progressTips = "processing";
    if (progress >= MAX_PROGRESS) {
        progressTips = "finish";
    }
    mProgressTips.setText(progressTips);
    mHealthProgressBar.setProgress(progress);
}

Update data in real-time and display the health check result. 

mActivity.runOnUiThread(new Runnable() {
    @Override
    public void run() {
        mHealthParamTable.removeAllViews();
        TableRow heatRateTableRow = initTableRow(ARFace.HealthParameter.PARAMETER_HEART_RATE.toString(),
            healthParams.getOrDefault(ARFace.HealthParameter.PARAMETER_HEART_RATE, 0.0f).toString());
        mHealthParamTable.addView(heatRateTableRow);
        TableRow breathRateTableRow = initTableRow(ARFace.HealthParameter.PARAMETER_BREATH_RATE.toString(),
            healthParams.getOrDefault(ARFace.HealthParameter.PARAMETER_BREATH_RATE, 0.0f).toString());
        mHealthParamTable.addView(breathRateTableRow);
    }
});