DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
The Latest "Software Integration: The Intersection of APIs, Microservices, and Cloud-Based Systems" Trend Report
Get the report
  1. DZone
  2. Data Engineering
  3. Data
  4. How Microservices is Imperative to Bedrock Data Lake Management

How Microservices is Imperative to Bedrock Data Lake Management

Zaloni’s Bedrock Data Lake Management platform is built modularly, using Microservices, to enable, orchestrate, and govern a Hadoop infrastructure.

Aashish Majethia user avatar by
Aashish Majethia
·
Nov. 16, 16 · Opinion
Like (6)
Save
Tweet
Share
6.18K Views

Join the DZone community and get the full member experience.

Join For Free

Microservices are a key enabler of large, scalable implementations making optimum use of hardware. Companies such as Netflix have pioneered their use and have shown the promise of what they can achieve. Netflix has used Microservices to support over 35% of all internet download traffic in North America.

Leveraging Microservices requires developing software in an architecture that allows deployment of small, modular programs or services that communicate in a well-defined way. By deploying the services on separate hardware, the software can quickly scale to handle higher loads. Additionally, by developing well defined communication channels, high availability can be achieved. Basically, in case a process dies, there is fault tolerance to assure that the process starts again.

Implementing Hadoop holds the promise of working with ingesting, analyzing, and managing large datasets. Data can be pulled from existing infrastructure (Mainframes, Data Warehouses (DWs), flat files, etc.) or from new sources (SaaS products like Salesforce, Feeds from services like Twitter, etc.) As data needs increase, this requires a software infrastructure that provides scalability across the entire data management process and Microservices are an ideal way of achieving that goal. Although, legacy programs can implement some modularity, it is a major architectural change that can require prohibitively significant investment to achieve much benefit. The ideal approach is to build a product from the ground up with a modular architecture.

Zaloni’s Bedrock Data Lake Management platform is built modularly, using Microservices, to enable, orchestrate, and govern a Hadoop infrastructure. It is built to manage any velocity, variety, and volume of data as long as there is hardware to support it. Bedrock leverages two key Microservices – the Bedrock Data Collection Agent (BDCA) and the Workflow Executor (WFE), among others. The BDCA helps to manage the movement in and out of the Data Lake as well as within the Lake itself. The WFE manages jobs that are run in Hadoop and communicates with Bedrock on execution status. These two Microservices can be deployed on the same Edge node that Bedrock resides. As business grows and needs change, the BDCA and WFE can be run on as many edge nodes as necessary. This allows more ‘pipes’ into the heart of the data processing engine and ultimately increased performance.

microservice Data processing Data lake

Published at DZone with permission of Aashish Majethia. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • 4 Best dApp Frameworks for First-Time Ethereum Developers
  • What “The Rings of Power” Taught Me About a Career in Tech
  • Utilizing Database Hooks Like a Pro in Node.js
  • Browser Engines: The Crux of Cross-Browser Compatibility

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: