DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Securing Your Software Supply Chain with JFrog and Azure
Register Today

Trending

  • DZone's Article Submission Guidelines
  • Is Podman a Drop-In Replacement for Docker?
  • Effective Java Collection Framework: Best Practices and Tips
  • Microservices With Apache Camel and Quarkus

Trending

  • DZone's Article Submission Guidelines
  • Is Podman a Drop-In Replacement for Docker?
  • Effective Java Collection Framework: Best Practices and Tips
  • Microservices With Apache Camel and Quarkus
  1. DZone
  2. Data Engineering
  3. Data
  4. Parallel Data Processing Strategies: (Almost) Always Use java.util.concurrent!

Parallel Data Processing Strategies: (Almost) Always Use java.util.concurrent!

Ricardo Zuasti user avatar by
Ricardo Zuasti
·
Apr. 02, 12 · Interview
Like (0)
Save
Tweet
Share
9.38K Views

Join the DZone community and get the full member experience.

Join For Free

Version 5 of the Java platform introduced a high level concurrency API, located in the java.util.concurrent package.

It allows for a much elegant and intuitive multi-threaded programming. I know this is old news for some, but I have found that most programmers still rely on the Thread class and Runnable interface to solve most concurrent problems in Java, when almost all of them can be implemented in a much cleaner way using the new API.

In this post series I’ll provide several examples on the usage of java.util.concurrent classes to solve common challenges. Let’s start with a simple parallel solution for data processing.

Imagine you have a set of data elements and you need to perform some kind of processing over each one of them, you want to maximize the speed at which this processing is done, but, on the other hand you don’t want to hog every system resource available if the system is being used.

A good strategy would be to have a thread pool with a pre-defined number of maximum active threads that will process the data one item at a time as soon as the threads become available.

This strategy can be quickly implemented using a fixed thread pool executor service.

We can model our data processing task as a runnable:

package com.ricardozuasti;
 
public class DataProcessor implements Runnable {
    public DataProcessor(int data){
        this.data = data;
    }
 
    @Override
    public void run() {
        System.out.println("Processing data: " + data);
        // Data processing goes here
    }
 
    private int data;
}

And then use the utility Executors class to create a new thread pool service (with a given maximum active thread count) with the newFixedThreadPool(n) method. The returned service (which implements the ExecutorService interface) can be used by submitting new tasks to it using the execute() and submit() methods.

package com.ricardozuasti;
 
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
 
public class Concurrency1 {
 
    public static void main(String[] args) {
        ExecutorService executor = Executors.newFixedThreadPool(5);
 
        for (int i = 0; i<100; i++){
            executor.execute(new DataProcessor(i));
        }
 
        System.out.println("Starting shutdown...");
        executor.shutdown();
 
        try {
            executor.awaitTermination(100, TimeUnit.SECONDS);
        } catch (InterruptedException ex) {
            System.out.println("Interrupted...");
        }
 
        System.out.println("All done!");
 
    }
}

After submitting all our work units to the ExecutorService we can instruct it to shutdown, this will not block our current working thread, nor avoid any previously submitted tasks from running, but only prevent new tasks from being passed onto the ExecutorService.

To actually wait for all the tasks to be done, we can use the awaitTermination() method.

And that’s it… it looks nice and clean, doesn’t it? :)

Check out the Executors API to see other kinds of executor services you can build right out of the box (cached and scheduled for example).

 

 

Data processing

Published at DZone with permission of Ricardo Zuasti, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • DZone's Article Submission Guidelines
  • Is Podman a Drop-In Replacement for Docker?
  • Effective Java Collection Framework: Best Practices and Tips
  • Microservices With Apache Camel and Quarkus

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: