Emerging Trends in Data Warehousing and Analytics in Cloud

Over the past few decades, cloud technologies evolved from traditional server offerings to higher level services. Big Data Analytics and Data Warehousing are emerging areas in cloud today. The adoption of cloud towards these areas is ever increasing due to the on-demand availability of storage, computing resources, and higher level services.

Although Data Warehousing in cloud is getting popular, Data Warehousing has a long history beyond cloud. It was born as an architectural concept to move data from operational systems to decision support systems.

First of all, it is important to understand Data Warehousing is a combination of processes and tools to prepare data by cleaning, integration, & data consolidations, having Data Warehouse at its core.

This article focuses on several emerging trends and technologies in cloud, that helps to design efficient and cost-effective solutions for Data Warehousing and analytics in cloud.

Using Managed Services

Managed services are the type of higher-level services, where most of the challenging concerns for a specific use case are automatically handled by cloud. Most of the challenges for Data Warehousing are related with scalability, reliability, security, performance, and efficiency which are mostly managed by the cloud provider when using these managed services.

When it comes to Data Warehouse architecture, you can use Fully-Managed ETL Services (e.g; Amazon Glue, Azure Data Factory), Managed Data Warehouse Services (e.g; Amazon RedShift, Azure SQL Data Warehouse.) and many more. See this article by Panoply for more details on DW architecture. When using these services it is also possible to find inter-connectable services in cloud to reduce the implementation effort even further. It is also possible to find cloud infrastructure and services provisioning templates, for that makes it even simpler to setup Data Warehousing solutions

In addition, using these services also creates possibilities to reduce costs since most of them are billed on-demand by cloud providers, and you don't need to pay for most of these services unless you use them.

Data Marts for Production Lines

In a large centralized data repository like a Data Warehouse, it is also important to analyze data for different production lines. Data Marts provides a solution by containing the summarized data from a specific business unit. Data Marts can be used as an intermediate source to the Data Warehouse as well as for each business unit to analyze their own data in isolation.

Data Lake Inspired

There are fundamental differences between Data Lakes and Data Warehousing. However, we have seen that the Data Lake is getting popular in data analytics and reporting world. One of the main differences between Data Lake and Data Warehousing is that Data Lake defines the schema of data on reading while Data Warehousing defines the schema on write. Although there are both pros and cons of using Data Lake, we can take inspiration from its core strengths for Data Warehousing.

One of the popular technologies for a Data Lake is to utilize distributed storage and processing using tools like Hadoop File System. This can also be beneficial for Data Warehousing, where it allows to pre- or post-process the data in an efficient and parallel process reducing time and cost.

Using Columnar Storage

When it comes to data warehousing it is important to store data from various sources in a data warehouse such that it is efficient to query for analytical purposes. For this purpose, using Columnar storage can improve disk performance in comparison to row-based storage when retrieving complex analytical queries. There are data warehouse services in cloud that provides these capabilities (e.g. Amazon RedShift) both for storage and querying at a lower cost. Using these services not only reduces the complexity of setting up a data warehouse, but also provides tight integrations for access control, integrating various data sources, and more.

In-Memory Analytical Engines

When performing analytics and reporting, it is more efficient to use in-memory processing engines, that can not only import massive amounts of data, but also to process them in parallel for fast responses and visualizations. Cloud services such as Microsoft Azure Power BI Embedded, and Amazon QuickSight are readily available for in-memory analytics and visualizations.