Over a million developers have joined DZone.

4 Reasons ETL is Giving Way to New Data Integration Technologies

· Big Data Zone

Hortonworks DataFlow is an integrated platform that makes data ingestion fast, easy, and secure. Download the white paper now.  Brought to you in partnership with Hortonworks

Extract-transform-load (ETL) technology served well as enterprises started dealing with bigger and bigger piles of data. ETL technology could read data from a database, transform it into a different form using rules or lookup tables, and then load it into another database. If you had a data migration project in the past few years, ETL was almost certainly involved. 

But now that big data is no longer limited to huge enterprises with massive data warehouses and staffs full of PhD data scientists, ETL's limitations are becoming more apparent. Today's business uses about twice as many cloud applications as it did just a few years ago, and companies that provide software as a service have to be able to perform data integrations with alacrity and speed, at a reasonable cost. 

Not many organizations can pull their engineering team every time an API changes or whenever they get a new SaaS solution. Increasingly, ETL is giving way to new data integration technologies. Here are 4 reasons why:

1. The Rise of Social-Mobile-Analytics-Cloud (SMAC)

Today's data more frequently comes from social platforms and mobile devices. Analytics and the cloud handle this data and provide companies with insights on what it all means and what their next steps might be. The traditional ETL paradigm isn't agile enough and doesn't provide the self-service that today's companies want. SMAC represents a generational shift, and data integration and migration services are popping up to assist smaller, more nimble companies with data integration when they don't want to shoehorn themselves into the traditional ETL template.

2. Companies Want More Flexibility and Agility

Tools from companies like Informatica, IBM, and Oracle that provide data migration into analytical data warehouses won't go away yet. When they reach the end of their useful lives, however, companies will want the ability to move semi-structured or unstructured data from any number of sources into a Hadoop framework or cloud repository. Companies want to deploy data how they want and they don't want to use multiple integration platforms to do so. Data migration and integration providers are eliminating the need for businesses to use multiple integration platforms, and making it possible to quickly move lots of smaller data containers into the data lake, as well as slowly moving fewer, larger containers of data.

3. Data Integration Is Increasingly Complex 

marketing-innovationETL and data warehousing have coped with data integration for years, but these technologies simply can't handle thetsunamis of data businesses deal with today. Data integration and migration are more complex due to the increased use of cloud-based resources, the remaining legacy database technology still in use, and the huge proliferation of devices that both produce and consume huge amounts of data. New demands are changing how data is managed and used. As the future unfolds, data services are expected to be the most common means for accessing data, and that means data integration technology will have to enable access to big data - structured or unstructured. The cloud will be the vehicle for achieving the scale that businesses need.

4. Data Migrations Are Needed More Often

Data and what businesses do with it are more dynamic than ever. A recent study found that over 90% of companies will engage in some type of data migration project. These are needed more than ever due to things like mergers, spin-offs, system upgrades, and regulatory changes. But if data migrations are overly complicated or time-consuming, projects can bog down and competitors can get ahead. Companies that make data integration and data migration straightforward without requiring major IT initiatives are the ones that businesses in every industry will turn to when they want to get their arms around big data and squeeze insights from it.


Data integration technologies like those offered by Bedrock Data answer the integration and migration needs of today's businesses. With a continuous integration platform that updates data in near real-time, Bedrock frees up companies from worrying about manually moving data so they can focus more on getting insights from it. Use the form below to contact us at any time. We would love to show you how much faster and straightforward data integration and migration can be.

Hortonworks Sandbox is a personal, portable Apache Hadoop® environment that comes with dozens of interactive Hadoop and it's ecosystem tutorials and the most exciting developments from the latest HDP distribution, brought to you in partnership with Hortonworks.

etl,big data,data integration,bigdata

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}