Data Lake ROI and What You Need to Know About Bedrock
Data Lake ROI and What You Need to Know About Bedrock
Today we often go fishing for our information in our private data lakes. But just maintaining the lake can be expensive at worst and distracting at best. If there were only a platform for that. Here's one you might find interesting.
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
IT analyst, research and strategy firm Enterprise Strategy Group (ESG) recently did an economic value validation report of our Bedrock data lake management platform (disclosure: we commissioned it) and we wanted to share the results. I recommend you read the report itself, but here are the highlights.
To determine the value of using Bedrock, ESG compared two ways of approaching the same use case:
- Using Zaloni’s Bedrock data lake management platform
- Doing it “DIY” or what ESG calls “PMO/present mode of operation” – using a blend of legacy and homegrown tools and internally developing custom and commercial ETL (extract, transform, load) and data warehousing solutions and other point products to sit on top of the Hadoop cluster
It’s Not Even Close: 671% Roi With Zaloni Bedrock Compared to DIY Data Lake
Data governance is essential. I think one of the most compelling conclusions in the report – in addition to the financial benefits – is this: “…there is no comfort in having ‘sort of’ clean data, ‘somewhat’ locked down access, or ‘kind of’ secure data lakes… The business ramifications of building a data lake without these common-sense controls are frightening.” In their research, ESG found that providing these controls with traditional or most existing tools is very difficult.
Bedrock makes good financial sense. Overall, ESG determined that using Bedrock to manage a 50-node Hadoop cluster housing 50 TB of data over three years (with seven analysts growing to a team of 21):
- Resulted in a 671% return on investment vs. 20% for the PMO
- Lowered TCO (total cost of ownership) by about $1.6 million
- Added $4.5 million in incremental IT efficiencies
- Added $3.5 million in user productivity benefits
- Added $1.3 million in delay-avoidance benefits
- Enabled breakeven in about six months, compared to more than two and a half years
ESG concluded that Bedrock was able to perform so much better because it is designed to lower costs while also adding significant incremental business value compared with what was expected in the PMO scenario. This was achieved through:
- Eliminating the need for employing a large development team to build and maintain the tools needed to support the data lake
- Enabling faster time to insight and more valuable insights
- Democratizing analytics by allowing less-technical analysts and subject matter experts to query and extract data without IT assistance and development changes
Bedrock Significantly Reduced Data Lake Maintenance and Staffing Costs
In addition to the initial purchase, traditional software tools require significant annual maintenance costs. Bedrock’s subscription model eliminated these costs. Overall, with Hadoop software maintenance included, maintenance and support costs for the PMO scenario were estimated to be more than $620,000. In the ESG analysis, Bedrock was estimated to reduce an organization’s maintenance cost over three years by 80%.
In regards to staffing, in the PMO scenario, an enterprise would need to hire data scientists, Hadoop experts, data warehouse architects, administrators and others. With Bedrock, the development and maintenance burden was shifted to Zaloni and reduced the need for hiring a large team. In total, over three years, Zaloni’s staffing costs were estimated as $378,000, while the PMO’s staffing costs were estimated at $1.3 million. Zaloni reduced staff costs by an estimated 71% over three years.
Bedrock Enabled Faster Time to Insight
ESG defined the key drivers of analytics time to value as the speed with which a data lake can be built and the avoidance of development delays when responding to analysts’ requests. In ESG’s model, the deployment of the data lake was much faster with Zaloni than with DIY development. Also, in the PMO scenario, when an analyst submits a development request, it involves a multi-step process. When there are multiple requests, some requests will be delayed. Bedrock eliminated these delays because all the data in the data lake is automatically tagged, sourced, protected and available to be queried from the moment it is added to the cluster. The report provides a detailed breakout of the model’s assumptions, but the result is that the value of avoiding delays for the analyst community was approximately $242,000 over three years.
Bedrock Enabled the Enterprise to Derive More Value From Its Data
Metadata management enables data governance. Data governance enables data democratization. ESG found that self-service analytics for end users (including less-technical business users and data scientists) through the use of Bedrock’s data catalog and Query Builder features was critical to the ROI calculation. ESG’s model takes into account the increase in analyst productivity and found that the incremental value delivery to an enterprise through an improvement in analyst reporting and analysis value was nearly $2.5 million over three years.
Getting Serious About Big Data
ESG stated that if enterprises want to get serious about using big data in a valuable way, they have to consider how to apply enterprise-grade governance to the data lake. Of course, we agree! We design our solutions to make it as efficient and cost-effective as possible for companies to deploy a data lake and leverage big data – responsibly and securely – to its fullest potential. For more details on ESG’s evaluation, please read the report.
Published at DZone with permission of Scott Gidley , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.