DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > How Microservices is Imperative to Bedrock Data Lake Management

How Microservices is Imperative to Bedrock Data Lake Management

Zaloni’s Bedrock Data Lake Management platform is built modularly, using Microservices, to enable, orchestrate, and govern a Hadoop infrastructure.

Aashish Majethia user avatar by
Aashish Majethia
·
Nov. 16, 16 · Big Data Zone · Opinion
Like (6)
Save
Tweet
6.01K Views

Join the DZone community and get the full member experience.

Join For Free

Microservices are a key enabler of large, scalable implementations making optimum use of hardware. Companies such as Netflix have pioneered their use and have shown the promise of what they can achieve. Netflix has used Microservices to support over 35% of all internet download traffic in North America.

Leveraging Microservices requires developing software in an architecture that allows deployment of small, modular programs or services that communicate in a well-defined way. By deploying the services on separate hardware, the software can quickly scale to handle higher loads. Additionally, by developing well defined communication channels, high availability can be achieved. Basically, in case a process dies, there is fault tolerance to assure that the process starts again.

Implementing Hadoop holds the promise of working with ingesting, analyzing, and managing large datasets. Data can be pulled from existing infrastructure (Mainframes, Data Warehouses (DWs), flat files, etc.) or from new sources (SaaS products like Salesforce, Feeds from services like Twitter, etc.) As data needs increase, this requires a software infrastructure that provides scalability across the entire data management process and Microservices are an ideal way of achieving that goal. Although, legacy programs can implement some modularity, it is a major architectural change that can require prohibitively significant investment to achieve much benefit. The ideal approach is to build a product from the ground up with a modular architecture.

Zaloni’s Bedrock Data Lake Management platform is built modularly, using Microservices, to enable, orchestrate, and govern a Hadoop infrastructure. It is built to manage any velocity, variety, and volume of data as long as there is hardware to support it. Bedrock leverages two key Microservices – the Bedrock Data Collection Agent (BDCA) and the Workflow Executor (WFE), among others. The BDCA helps to manage the movement in and out of the Data Lake as well as within the Lake itself. The WFE manages jobs that are run in Hadoop and communicates with Bedrock on execution status. These two Microservices can be deployed on the same Edge node that Bedrock resides. As business grows and needs change, the BDCA and WFE can be run on as many edge nodes as necessary. This allows more ‘pipes’ into the heart of the data processing engine and ultimately increased performance.

microservice Data processing Data lake

Published at DZone with permission of Aashish Majethia. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Is NoOps the End of DevOps?
  • Build a Data Pipeline on AWS With Kafka, Kafka Connect, and DynamoDB
  • Caching Across Layers in Software Architecture
  • Troubleshooting HTTP 502 Bad Gateway in AWS EBS

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo