A Quick Look at Big Data Layers, Landscape, and Principles

Extracting valuable, meaningful information (insights) from enormous volumes of data to improve organizational decisions may involve many challenges such as data regulations, interactions with customers, and dealing with legacy systems, disparate data sources, and so on.

Many thanks to many big data scientists and researchers, as they have designed and come up with a unified architectural approach comprised of different layers at different levels so that we can address all those big data challenges faster and more effectively.

The following article mostly is inspired by the book Architectural Patterns and intends to give the readers a quick look at data layers, unified architecture, and data design principles.

Big Data Layers

The following pyramid depicts the most common (yet significant) attributes of big data layers and the problem that is addressed in each layer. As you may already know, big data is not a single technology or a framework to solve any set of use cases; it is a set of tools, process, technology, and system infrastructure that helps business to do much smarter analyses and make more intelligent decisions from the massive volume of data traces.

Big data layers

As you see in the preceding diagram, big data architecture or unified architecture is comprised of several layers and provides a way to organize various components representing unique functions to address distinct problems.

Big Data Landscape

This section will serve as a comprehensive overview of big data concepts and the realization of values in each big data layer that we just discussed.

The following image depicts different levels and layers of the big data landscape:

Big data landscape

Let’s get a brief idea on each layer from the following points:

• Data sources: Data coming from several channels such as handheld devices, software applications, sensors, legacy databases, and so on.
• Data messaging and store: Acquire data from the data sources and consider data compliance and storage formatting.
• Data analysis: Data model management, analytics engines, and access to data message store.
• Data consumption: Dashboards, presenting insights, reporting, and so on.

Big Data Architecture Principles

As stated earlier, before we conclude this article, we will list out the following big data architecture principles:

• Decoupled data bus
  • Right tool usage for the job
  • Data structure, latency, throughput, and access patterns
• Lambda architecture
  • Immutable logs
  • Batch processing
• Cloud-based infrastructure
  • System maintenance with low or no admin
  • Cost-effective

I conclude this article with the hope you have an introductory understanding of different data layers, big data unified architecture, and a few big data design principles.