DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workkloads.

Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Recurrent Workflows With Cloud Native Dapr Jobs
  • Thermometer Continuation in Scala
  • Thumbnail Generator Microservice for PDF in Spring Boot
  • From Zero to Meme Hero: How I Built an AI-Powered Meme Generator in React

Trending

  • GDPR Compliance With .NET: Securing Data the Right Way
  • Cookies Revisited: A Networking Solution for Third-Party Cookies
  • Emerging Data Architectures: The Future of Data Management
  • Failure Handling Mechanisms in Microservices and Their Importance
  1. DZone
  2. Data Engineering
  3. Data
  4. How a Background Remover Is Born

How a Background Remover Is Born

A walkthrough to making an app remove backgrounds.

By 
Jackson Jiang user avatar
Jackson Jiang
DZone Core CORE ·
Sep. 21, 22 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
4.3K Views

Join the DZone community and get the full member experience.

Join For Free

Why Do I Need a Background Remover?

A background removal tool is not really a new feature, but rather its importance has grown as the world shifted over to online working and learning over the last few years. And I did not find how important this tool could be until just two weeks ago. On a warm, sunny morning with a coffee on hand, I joined an online conference. During this conference, one of my colleagues pointed out to me that they could see my untidy desk and an overflowing bin in the background. Naturally, this left me feeling embarrassed. I just wish I could travel back in time to use a background remover.

Now, I cannot travel in time, but I can certainly create a background removal tool. So, with this new-found motive, I looked online for some solutions and came across the body or head segmentation capability from HMS Core Video Editor Kit, and developed a demo app with it.

This service can divide the body or head from an input image or video and then generate a video, an image, or a sticker of the divided part. In this way, the body or head segmentation service helps realize the background removal effect.

Now, let's go deeper into the technical details about the service.

How the Background Remover Is Implemented

The algorithm of the service performs a series of operations on the input video, including extracting frames, using an AI model to process the video, and encoding. Among all these, the core is the AI model. How a service performs is affected by factors like device computing power and power consumption. Considering these, the development team of the service manages to equip it with a light-weight AI model that does a good job in feature extraction, by taking measures like compression, quantization, and pruning. In this way, the processing duration of the AI model is decreased to a relatively low level, without compromising the segmentation accuracy.

The mentioned algorithm supports both images and videos. An image takes the algorithm a single inference for the segmentation result. A video is actually a collection of images. If a model features poor segmentation capability, the segmentation accuracy for each image will be low. As a result, the segmentation results of consecutive images will be different from each other, and the segmentation result of the whole video will appear shaking. To resolve this, the team adopts technologies like inter-frame stabilization and the objective function for inter-frame consistency. Such measures do not compromise the model inference speed yet fully utilize the time sequence information of a video. Consequently, the algorithm sees its inter-frame stabilization improved, which contributes to an ideal segmentation effect.

By the way, the service requires that the input image or video contains up to 5 people whose contours should be visible. Besides, the service supports common motions of the people in the input image or video, like standing, lying, walking, sitting, and more.

The technical basics of the service conclude here, and let's see how it can be integrated with an app.

How to Equip an App with the Background Remover Functionality

Preparations

1. Go to AppGallery Connect and configure the app's details. In this step, we need to register a developer account, create an app, generate a signing certificate fingerprint, and activate the required services.

2. Integrate the HMS Core SDK.

3. Configure the obfuscation scripts.

4. Declare necessary permissions.

Setting Up a Video Editing Project

Prerequisites

1. Set the app authentication information either by:

  • Using an access token: Call the setAccessToken method to set an access token when the app is started. The access token needs to be set only once.
 
MediaApplication.getInstance().setAccessToken("your access token");
  • Using an API key: Call the setApiKey method to set an API key when the app is started. The API key needs to be set only once.

2. Set a License ID. The ID is used to manage the usage quotas of the kit, so make sure the ID is unique.

 
MediaApplication.getInstance().setLicenseId("License ID");

Initializing the Runtime Environment for the Entry Class

A HuaweiVideoEditor object serves as the entry class of a whole video editing project. The lifecycle of this object and the project should be the same. Therefore, when creating a video editing project, create a HuaweiVideoEditor object first and then initialize its runtime environment. Remember to release this object when exiting the project.

1. Create a HuaweiVideoEditor object.

 
HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());

2. Determine the preview area position.

This area renders video images, a process that is implemented by creating SurfaceView within the SDK. Make sure that the position of this area is specified before the area is created.

 
<LinearLayout    
    android:id="@+id/video_content_layout"    
    android:layout_width="0dp"    
    android:layout_height="0dp"    
    android:background="@color/video_edit_main_bg_color"    
    android:gravity="center"    
    android:orientation="vertical" />
// Specify the preview area position.
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);

// Specify the layout of the preview area.
editor.setDisplay(mSdkPreviewContainer);

3. Initialize the runtime environment of HuaweiVideoEditor. LicenseException will be reported when the license verification fails.

The HuaweiVideoEditor object, after being created, has not occupied any system resources. We need to manually set the time for initializing its runtime environment, and then the necessary threads and timers will be created in the SDK.

 
try {
        editor.initEnvironment();
   } catch (LicenseException error) { 
        SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());    
        finish();
        return;
   }

Integrating the Segmentation Capability

 
// Initialize the segmentation engine. segPart indicates the segmentation type, whose value is an integer. Value 1 indicates body segmentation, and a value other than 1 indicates head segmentation.
visibleAsset.initBodySegEngine(segPart, new HVEAIInitialCallback() {
    @Override
    public void onProgress(int progress) {
        // Callback when the initialization progress is received.
    }

    @Override
    public void onSuccess() {
        // Callback when the initialization is successful.
    }

    @Override
    public void onError(int errorCode, String errorMessage) {
        // Callback when the initialization failed.
    }
});

// After the initialization is successful, apply the segmentation effect.
visibleAsset.addBodySegEffect(new HVEAIProcessCallback() {
    @Override
    public void onProgress(int progress) {
        // Callback when the application progress is received.
    }

    @Override
    public void onSuccess() {
        // Callback when the effect is successfully applied.
    }

    @Override
    public void onError(int errorCode, String errorMsg) {
        // Callback when the effect failed to be applied.
    }
});

// Stop applying the segmentation effect.
visibleAsset.interruptBodySegEffect();

// Remove the segmentation effect.
visibleAsset.removeBodySegEffect();

// Release the segmentation engine.
visibleAsset.releaseBodySegEngine();

And now the app is capable of removing the image or video background.

This function is ideal for e-conferencing apps, where the background is not important. For learning apps, it allows teachers to change the background to the theme of the lesson, for better immersion. Not only that, but when it's used in a short video app, users can put themselves in unusual backgrounds, such as space and the sea, to create fun and fantasy-themed videos.

Have you got any better ideas of how to use the background remover? Let us know in the comments section below.

Wrap Up

Background removal tools are trending among apps in different fields, given that such a tool helps images and videos look better by removing unnecessary or messy backgrounds, as well as protecting user privacy.

The body or head segmentation service from Video Editor Kit is one such solution for removing a background. It supports both images and videos, and outputs a video, an image, or a sticker of the segmented part for further editing. Its streamlined integration makes it a perfect choice for enhancing videos and images.

Data Types

Published at DZone with permission of Jackson Jiang. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Recurrent Workflows With Cloud Native Dapr Jobs
  • Thermometer Continuation in Scala
  • Thumbnail Generator Microservice for PDF in Spring Boot
  • From Zero to Meme Hero: How I Built an AI-Powered Meme Generator in React

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!