DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Developing a Graph in Spark and Scala

Developing a Graph in Spark and Scala

A discussion of some of the basics of graph theory and how to apply this theory in code using Scala and the Spark framework.

Ederson Corbari user avatar by
Ederson Corbari
·
Jan. 04, 19 · Tutorial
Like (4)
Save
Tweet
Share
9.15K Views

Join the DZone community and get the full member experience.

Join For Free

I read an article a while back from here on DZone about processing hierarchical data using Apache GraphX. I would like to explore this solution further.

Overview

Recently in my work, I needed to create a solution using a graph to create a hierarchy of relationships between employees of the company. This is a simple problem, but I had never done it in the big data area using Apache GraphX.

In this case, we were using Cloudera’s cluster environment (commercially licensed), but the solution works the same way for the open source versions.

For anyone interested in Cloudera solutions, you can download the virtual machine with a cluster configured and ready to use.

1. The Problem

We need to ingest the employee data that was in the relational Oracle database for the cluster, do some data processing, and save it in Hive. But we did not have the relationships to create a hierarchy between the employee levels, so we used a table with auxiliary data and created a top-down graph to display the relationships.

Detail: Relational databases such as Oracle have the CONNECT BY function that works with the hierarchy.

Below I did a hypothetical design of the employee hierarchy that has 9 levels:

Screen Employee Hierarchy

GraphX (pregel) has two types of graphs: top-down and bottom-up.

Screen Graph Top Down and Button

1.1 Start

After several transformations of the data, we arrive at a DataFrame similar to this one:

Level Role Name Connect By
1 PRESIDENT Tim Cox null
2 VP Robert Watson 1
3 VP Matthew Powers 1
4 VP Alex Pess 1
5 DIRECTOR João Bastos 2
6 DIRECTOR Martin Jay 3
7 SUPERVISOR Ana Becker 5
8 SUPERVISOR Marcos Silverio 6
9 MANAGER Carlos Klaus 8
10 ANALYST Jacob Oliver 9
11 ANALYST Charlie Noah 9
12 ANALYST Claudio Stwart 9
13 SENIOR DEVELOPER Jack Connor 10
14 SENIOR DEVELOPER Daniel Mason 10
15 JUNIOR DEVELOPER George Reece 13

But this is only an example for our test, using the diagram above the hierarchy. The data containing null is considered the root of the relationship, meaning that, in this case, it would be the President or CEO of the company.

1.2 Analyzing the Result

Creating a DataFrame with the above data for tests:

val empDF = sparkSession.sparkContext.parallelize(Array(
  ("1", "Tim Cox", "PRESIDENT", null.asInstanceOf[String]),
  ("2", "Robert Watson", "VP", "1"),
  ("3", "Matthew Powers", "VP", "1"),
  ("4", "Alex Pess", "VP", "1"),
  ("5", "Joao Bastos", "DIRECTOR", "2"),
  ("6", "Martin Jay", "DIRECTOR", "3"),
  ("7", "Anna Becker", "SUPERVISOR", "5"),
  ("8", "Marcos Silverio", "SUPERVISOR", "6"),
  ("9", "Carlos Klaus", "MANAGER", "8"),
  ("10", "Jacob Oliver", "ANALYST", "9"),
  ("11", "Charlie Noah", "ANALYST", "9"),
  ("12", "Claudio Stwart", "ANALYST", "9"),
  ("13", "Jack Connor", "SENIOR DEVELOPER", "10"),
  ("14", "Daniel Mason", "SENIOR DEVELOPER", "10"),
  ("15", "George Reece", "JUNIOR DEVELOPER", "13")))
  .toDF("id", "name", "role", "id_connect_by")

After running that DataFrame in the graph, you will get the following result:

+---+---------------+----------------+-------------+-----+-------+-----------------------------------------------------------------------------------------------------+--------+------+
|id |name           |role            |id_connect_by|level|root   |path                                                                                                 |iscyclic|isleaf|
+---+---------------+----------------+-------------+-----+-------+-----------------------------------------------------------------------------------------------------+--------+------+
|1  |Tim Cox        |PRESIDENT       |null         |1    |Tim Cox|/Tim Cox                                                                                             |0       |0     |
|4  |Alex Pess      |VP              |1            |2    |Tim Cox|/Tim Cox/Alex Pess                                                                                   |0       |1     |
|2  |Robert Watson  |VP              |1            |2    |Tim Cox|/Tim Cox/Robert Watson                                                                               |0       |0     |
|3  |Matthew Powers |VP              |1            |2    |Tim Cox|/Tim Cox/Matthew Powers                                                                              |0       |0     |
|5  |Joao Bastos    |DIRECTOR        |2            |3    |Tim Cox|/Tim Cox/Robert Watson/Joao Bastos                                                                   |0       |0     |
|6  |Martin Jay     |DIRECTOR        |3            |3    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay                                                                   |0       |0     |
|7  |Anna Becker    |SUPERVISOR      |5            |4    |Tim Cox|/Tim Cox/Robert Watson/Joao Bastos/Anna Becker                                                       |0       |1     |
|8  |Marcos Silverio|SUPERVISOR      |6            |4    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio                                                   |0       |0     |
|9  |Carlos Klaus   |MANAGER         |8            |5    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio/Carlos Klaus                                      |0       |0     |
|11 |Charlie Noah   |ANALYST         |9            |6    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio/Carlos Klaus/Charlie Noah                         |0       |1     |
|12 |Claudio Stwart |ANALYST         |9            |6    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio/Carlos Klaus/Claudio Stwart                       |0       |1     |
|10 |Jacob Oliver   |ANALYST         |9            |6    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio/Carlos Klaus/Jacob Oliver                         |0       |0     |
|14 |Daniel Mason   |SENIOR DEVELOPER|10           |7    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio/Carlos Klaus/Jacob Oliver/Daniel Mason            |0       |1     |
|13 |Jack Connor    |SENIOR DEVELOPER|10           |7    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio/Carlos Klaus/Jacob Oliver/Jack Connor             |0       |0     |
|15 |George Reece   |JUNIOR DEVELOPER|13           |8    |Tim Cox|/Tim Cox/Matthew Powers/Martin Jay/Marcos Silverio/Carlos Klaus/Jacob Oliver/Jack Connor/George Reece|0       |1     |
+---+---------------+----------------+-------------+-----+-------+-----------------------------------------------------------------------------------------------------+--------+------+

Let’s understand the results:

Field Description
id The number of the identifier or the enrollment of the employee. This number must be unique.
name The employee’s name.
role The position or job by the employee in the company.
id_connect_by Describes how an employee is connected to another person.
level The level number that the employee is in the company, each possition must have a level.
root The beginning of the hierarchy.
path The path traveled until reaching the highest position (top dog).
iscyclic The number of vertices in Cn equals the number of edges, and every vertex has degree 2.
isleaf Determines whether a vertex is a leaf.

1.3 Code

To do more tests, please download the code in the repository:

git clone https://github.com/edersoncorbari/scala-lab.git

In the readme file of the project you can find tips on how to run and test.

1.4 References

Below are some references to remember and study:

  • MIT Computer Science Course Lecture on Graph Theory
  • https://spark.apache.org/graphx/
  • https://dzone.com/articles/processing-hierarchical-data-using-spark-graphx-pr
  • Spark Graphx in Action
  • https://edersoncorbari.github.io/tutorials/scala-spark-graph/

Thanks!

Graph (Unix) Big data Scala (programming language)

Published at DZone with permission of Ederson Corbari. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • The 12 Biggest Android App Development Trends in 2023
  • Asynchronous HTTP Requests With RxJava
  • How Observability Is Redefining Developer Roles
  • Remote Debugging Dangers and Pitfalls

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: