DZone
Java Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Java Zone > Compute Grids - What to Expect?

Compute Grids - What to Expect?

Dmitriy Setrakyan user avatar by
Dmitriy Setrakyan
·
Feb. 17, 11 · Java Zone · Interview
Like (0)
Save
Tweet
4.62K Views

Join the DZone community and get the full member experience.

Join For Free
Recently I have been seeing products popping up calling themselves compute grids while in reality they have nothing, or very little, to do with compute grids. While data grids are provided by quite a few vendors and people generally know what to expect from them, compute grids are not as common, and people may end up thinking that a compute grid is a simple RPC-like function execution on a remote node. As you may have guessed, I will show that to be untrue.

Let's take Cloudy Akka for example. While the API seems pretty simple from reading this blog, calling it a compute grid is rather overstating the reality. I would probably not go any further than naming it for what it is - a Convenient Scala API for RPC.

Having worked on GridGain compute grid myself for over 5 years already (and then on data grid), and having studied quite a few others, here are some features that are minimally required in any product in my opinion, before it can claim itself as a compute grid.
  • Auto Discovery - all nodes in the grid should auto-discover each other, i. e. user should never have to manually add nodes to a topology.
  • MapReduce - support for splitting execution into multiple sub-jobs and then aggregating the results is just a MUST. Otherwise you are offloading most of the dirty work onto your users, which is not fair.
  • Auto Failure Detection - compute grid must be smart enough to automatically detect node crashes and proportionally distribute all the load among remaining nodes.
  • Fault Tolerance - all failed grid jobs must be automatically failed-over to other nodes, which are better suited for executing these jobs.
  • Load Balancing - compute grid should automatically distribute load equally among nodes, usually utilizing many different policies for load balancing. GridGain even has support for work-stealing, where less-loaded nodes can steal jobs from overloaded nodes.
  • Job Collision Resolution - this gives users control over how many jobs can run in parallel, while other jobs should wait in waiting queues, ordered by multiple available collision resolution strategies.
  • Auto Deployment - compute grid users should never be forced to manually deploy their libraries on all available grid nodes, this is just way too inconvenient and error-prone. The approach I like the best (available in GridGain) is auto-deployment, where code just automatically penetrates throughout the grid without any explicit action from users.
  • Nested Jobs And Continuations - compute grid jobs should be able to invoke other compute grid jobs when executing remotely. This is a very powerful feature, especially when grid jobs are recursive. Continuations should allow to suspend a job and release its resources while it's waiting for a result of another job within the grid.

I could probably continue, but I will stop here. I think it is clear that compute grid products are way more advanced than RPC frameworks. Just as a comparison, here are the GridGain examples implemented in GridGain Scalar, which cover examples provided by Cloudy Akka in a simpler fashion, and also do all the other cool stuff compute grids do in the background:

scalar {
// 1. Execute a simple job on some remote node.
grid !!< (() => println("> GridGain ROCKS <"))

// 2. Execute a simple job on all remote nodes.
grid !!! (() => println("> GridGain ROCKS <"))

// 3. Use MapReduce to split a phrase into multiple words and
// print each word on remote nodes.
grid !!~ (for (w <- "GridGain ROCKS".split(" ")) yield () => println(w))

// 4. Use MapReduce to count number characters by spreading
// workload to the grid and reducing on local node.
val cnt = grid !*~ (for (w <- "GridGain REALLY ROCKS!".split(" "))
yield () => w.length, // Map step.
(s: Seq[Int]) => s.sum) // Reduce step.
}

From http://gridgain.blogspot.com/2011/02/compute-grids-what-to-expect.html

career

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • What Is Lean Software Development?
  • How the TypeScript ReturnType Works
  • Testing Schema Registry: Spring Boot and Apache Kafka With JSON Schema
  • Creating Event-Based Architecture on Top of Existing API Ecosystem

Comments

Java Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo