How To Implement Image Segmentation

Changing an image/video background has always been a hassle, whereby the most tricky part is extracting the element other than the background.

Traditionally, it requires us to use a PC image-editing program that allows us to select the element, add a mask, replace the canvas, and more. If the element has an extremely uneven border, then the whole process can be very time-consuming.

Luckily, ML Kit from HMS Core offers a solution that streamlines the process: the image segmentation service, which supports both images and videos. This service draws upon a deep learning framework, as well as detection and recognition technology. The service can automatically recognize — within seconds — the elements and scenario of an image or a video, delivering a pixel-level recognition accuracy. By using a novel framework of semantic segmentation, image segmentation labels each and every pixel in an image and supports 11 element categories, including humans, the sky, plants, food, buildings, and mountains.

This service is a great choice for entertaining apps. For example, an image-editing app can use the service to realize swift background replacement. A photo-taking app can count on this service for the optimization of different elements (for example, the green plant) to make them appear more attractive.

Below is an example showing how the service works in an app.

Development Procedure

Before app development, there are some necessary preparations in AppGallery Connect. In addition, the Maven repository address should be configured for the SDK, and the SDK should be integrated into the app project.

The image segmentation service offers three capabilities: human body segmentation, multiclass segmentation, and hair segmentation.

1. Human body segmentation: supports videos and images. The capability segments the human body from its background and is ideal for those who only need to segment the human body and background. The return value of this capability contains the coordinate array of the human body, the human body image with a transparent background, and the gray-scale image with a white human body and black background. Based on the return value, your app can further process an image to, for example, change the video background or cut out the human body.
2. Multiclass segmentation: offers the return value of the coordinate array of each element. For example, when the image processing by the capability contains four elements (human body, sky, plant, and cat & dog), the return value is the coordinate array of the four elements. Your app can further process these elements, such as replacing the sky.
3. Hair segmentation: segments hair from the background, with only images supported. The return value is a coordinate array of the hair element. For example, when the image processing by the capability is a selfie, the return value is the coordinate array of the hair element. Your app can then further process the element by, for example, changing the hair color.

Static Image Segmentation

1. Create an image segmentation analyzer.

• Integrate the human body segmentation model package.

// Method 1: Use default parameter settings to configure the image segmentation analyzer.
// The default mode is human body segmentation in fine mode. All segmentation results of human body segmentation are returned (pixel-level label information, human body image with a transparent background, gray-scale image with a white human body and black background, and an original image for segmentation).
MLImageSegmentationAnalyzer analyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(); 
// Method 2: Use MLImageSegmentationSetting to customize the image segmentation analyzer.
MLImageSegmentationSetting setting = new MLImageSegmentationSetting.Factory() 
    // Set whether to use fine segmentation. true indicates yes, and false indicates no (fast segmentation).
    .setExact(false) 
    // Set the segmentation mode to human body segmentation.
    .setAnalyzerType(MLImageSegmentationSetting.BODY_SEG) 
    // Set the returned result types.
    // MLImageSegmentationScene.ALL: All segmentation results are returned (pixel-level label information, human body image with a transparent background, gray-scale image with a white human body and black background, and an original image for segmentation).
    // MLImageSegmentationScene.MASK_ONLY: Only pixel-level label information and an original image for segmentation are returned.
    // MLImageSegmentationScene.FOREGROUND_ONLY: A human body image with a transparent background and an original image for segmentation are returned.
    // MLImageSegmentationScene.GRAYSCALE_ONLY: A gray-scale image with a white human body and black background and an original image for segmentation are returned.
    .setScene(MLImageSegmentationScene.FOREGROUND_ONLY) 
    .create(); 
MLImageSegmentationAnalyzer analyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(setting);

• Integrate the multiclass segmentation model package.

When the multiclass segmentation model package is used for processing an image, an image segmentation analyzer can be created only by using MLImageSegmentationSetting.

MLImageSegmentationSetting setting = new MLImageSegmentationSetting 
    .Factory()
    // Set whether to use fine segmentation. true indicates yes, and false indicates no (fast segmentation).
    .setExact(true) 
    // Set the segmentation mode to image segmentation.
    .setAnalyzerType(MLImageSegmentationSetting.IMAGE_SEG)
    .create(); 
MLImageSegmentationAnalyzer analyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(setting);

• Integrate the hair segmentation model package.

When the hair segmentation model package is used for processing an image, a hair segmentation analyzer can be created only by using MLImageSegmentationSetting.

MLImageSegmentationSetting setting = new MLImageSegmentationSetting 
    .Factory()
    // Set the segmentation mode to hair segmentation.
    .setAnalyzerType(MLImageSegmentationSetting.HAIR_SEG)
    .create(); 
MLImageSegmentationAnalyzer analyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(setting);

2. Create an MLFrame object by using android.graphics.Bitmap for the analyzer to detect images. JPG, JPEG, and PNG images are supported. It is recommended that the image size range from 224 x 224 px to 1280 x 1280 px.

// Create an MLFrame object using the bitmap, which is the image data in bitmap format.
MLFrame frame = MLFrame.fromBitmap(bitmap);

3. Call asyncAnalyseFrame for image segmentation.

// Create a task to process the result returned by the analyzer.
Task<MLImageSegmentation> task = analyzer.asyncAnalyseFrame(frame); 
// Asynchronously process the result returned by the analyzer.
task.addOnSuccessListener(new 
OnSuccessListener<MLImageSegmentation>() {
    @Override
    public void onSuccess(MLImageSegmentation segmentation) {
        // Callback when recognition is successful.
    }})
    .addOnFailureListener(new OnFailureListener() {
    @Override 
    public void onFailure(Exception e) {
        // Callback when recognition failed.
}});

4. Stop the analyzer and release the recognition resources when recognition ends.

if (analyzer != null) {
    try {
        analyzer.stop(); 
    } catch (IOException e) {
        // Exception handling.
    }
}

The asynchronous call mode is used in the preceding example. Image segmentation also supports the synchronous call of the analyseFrame function to obtain the detection result:

SparseArray<MLImageSegmentation> segmentations = analyzer.analyseFrame(frame);