Multi-Cloud Data Layer Essentials

For decades, the separation of business logic from the persistent data it manipulates has been well established in software application architectures. This is because the same data can be used in many different ways to serve many different use cases, so it cannot be embedded in any one application. With the emergence of microservices, the need has come for this data to become a service.  

But this is not monolithic data. Many microservices using a single database results in very tight coupling. This makes it impossible to deploy new or evolving services individually if they need to make database changes that will affect other services. Breaking complex service interdependencies requires a data layer of multiple databases, each serving a single microservice or possibly a few, tightly related microservices.  

Delivering Data Services for Any Use Case 

Cloud computing offers microservices the perfect operating environment: resilient, scalable, and distributable. But the database lagged the embrace of the virtual world of the cloud. Traditional RDBMS, the workhorse of data services since before the turn of the millennium, is monolithic. You scale a traditional RDBMS by running it on more powerful hardware. But in the cloud, this has limitations. And resilience comes only from database replication and complex disaster recovery processes.  

The response to this was a proliferation of NoSQL databases and other data management tools. But monolithic RDBMS hung on because it can deliver ACID transactional consistency, which is vital for systems of record and systems of engagement. And lately, SQL databases are fighting back with cloud-native transactional capability.  

But in reality, it is not a fight, a war of competing database models. It is a co-existence. ACID, SQL databases are still the best at transactional consistency. But NoSQL, graph, document, analytic, and streaming databases and technologies excel elsewhere in the spectrum of data services’ needs. And mature technologies are available to address all these workloads.  

By implementing more applications in the cloud as microservices, the data services needed to support these applications will be delivered by different technologies and databases — each optimized by the needs of the microservices it is supporting. 

The result is not an attempt to shoehorn all workloads into a single winner-takes-all database. The optimal result is a multi-cloud data layer with the capability to deliver the appropriate data services for any use case. 

Multi-Cloud Data Layer Core Capabilities 

A multi-cloud data layer delivers three core capabilities for both existing applications to help bridge digital transformation initiatives as well as new cloud-native applications and microservices. These core capabilities include:  

1. Freedom from tradeoffs: With a multi-cloud data layer, developers can achieve low latency, ACID transactions, familiar interfaces, relational capabilities, horizontal scale, high performance, geographic distribution, and ease of deployment, all in one database system. 
2. Simplified operations: A multi-cloud data layer should be easy to deploy and scale to help minimize disruption going from a traditional RDBMS to a cloud-native database for cloud-native applications. It also provides the flexibility to allow organizations to start running in minutes through a fully managed offering, or be able to deploy anywhere, including containers across public, private, and hybrid cloud environments. For all deployments, organizations should be able to start small and scale horizontally without impacting performance, uptime, or data integrity (i.e., scale-out vs. scale-up).  
3. Built-in security and resiliency: Uncompromising security and availability are expected, with these core features designed into a multi-cloud data layer from the start to make it easy and seamless to enable.  

It is tempting to define workloads by the classes of data management tools available (e.g., SQL and NoSQL). That’s a shortcut that will suffice much of the time, but it is a better practice to consider the needs of individual use cases. Independently of the business value that the use case implements, it will have a number of orthogonal data requirements.   

For example: Is it interactive? Does it need to ingest high-arrival rate events? Will it involve an ad-hoc query of large volumes of data? There are a number of such questions, and how they are answered for a specific use case indicates the appropriate data services it will need.  

There are several key factors that define multi-cloud data layer workloads. Let’s walk through them to better understand what they are and how they function. 

Latency 

How much time can elapse before the service is regarded as a failure? This might be a time-out in a time-critical API call, or it might be a loss of customers over time on a high-volume, poor-performing website. Regardless of the latency in processing the request, significant network latency must not be injected.   

• This also means local processing. If there is shared data involved, this will mean distributed data — or data being available close to where it is needed.  
• In addition, local processing means data not just distributed in cloud regions; it may also indicate edge computing capability. 

Transactional Consistency 

Transactional consistency is the need for two different entities to be updated and it being mandatory that both fail or succeed. Monetary transactions are the most obvious, but there are others, such as reserving stock for an order against Inventory.  

• This means ACID compliance, usually a SQL database.   

Arrival Velocity 

How fast do requests arrive? Internet of Things (IoT) applications often generate millions of events per second. The first requirement is to ingest them at sustained peak levels. But processing them may be a different operation.  

A business use case defined to be a monitoring, alert, or dashboard application based on the event stream may be delivered as (at least) two services: ingest, analytic derivation of complex KPIs, and dashboard delivery of indicators and trends.  

• The rapid ingest will normally predicate stream processing capabilities enabled on a persistent store on the back-end that is appropriate for the processing requirements.  
• The appropriate back-end capability might be a NoSQL database, a transactional SQL database, or a data lake. The determining factor will probably relate to the further processing needs of the arrival data.  

Data Access Volume 

How much data needs to be accessed to service the request? Ad-hoc analytic queries will often consume huge volumes of data.  

• If the queries are ad-hoc on structured data and data aggregations (such as OLAP cubes) cannot be pre-built to serve them, the most appropriate technology might be a massively parallel data warehouse or data lakehouse.  
• But if the queries are ad-hoc on unstructured data, the most appropriate technology might well be enterprise search software.  
• If “families” of queries can be defined once and reused many times, then OLAP processing in many kinds of stores may meet the need.  

Data Complexity 

Is the data to be accessed a complex structure? 

• In some cases, apparent complexity is a matter of looking at too granular a level. For example, this content is made up of sections, containing paragraphs, sentences, words, and letters. Each letter has fonts and many other properties. But take a step back and it’s a simple hierarchical structure, easily managed in a document database or a key-value pair structure. 
• Other cases may involve genuinely complex data structures. Relational databases have often been used for such structures. But nowadays, other technologies, such as graph databases, can offer significant performance improvements for particular use cases. 

There are potentially other categories and ontologies. Even within this categorization, important refinements have been omitted for clarity and brevity (e.g., the differences for read-heavy versus write-heavy workloads). However, this ontology can be regarded as a good rule-of-thumb. 

The critical point is that it is important to analyze the use case backlog by what is needed, not by what has been used in the past.  

Mapping Use Cases Onto Workloads 

A use case may have contradictory requirements. In the streaming data dashboard example mentioned above, low latency and huge data volume access are genuinely not achievable, but a reappraisal of the core requirement can usually define a series of services that will achieve the objective. 

Selecting Data Service Technologies to Support Workloads 

To the man with only a hammer, everything looks like a nail. The art of the data layer architect is to have the right tool at hand or to recognize the right tool, and add it to their toolbox. 

If we adopt the broad workload categorization above, we can map them onto the capabilities of the family of data management tools that we might expect to see in a data layer. Table 1, below, shows this graphically: 

There are a few other important considerations to keep in mind when defining multi-cloud data layer workloads. 

Services Within the Data Layer 

Looking at Table 1 above, it becomes obvious that the analysis of workload requirements for a specific use case will often leave choices that must be taken on a case-by-case basis. What is equally apparent is that for any typical set of enterprise use cases, no single date engine will suffice.   

The need for a multi-cloud data layer is apparent. But it is not the case that all the component databases and other data engines in the data layer operate as siloes, each with their own set of use cases. They must combine, for example, to serve high-arrival velocity use cases; and they also need to share data.  

One widely used pattern for sharing enterprise reference data in real time is a service request from one database instance, serving a particular subject domain (say Orders), to another database, serving another domain like Customers. By treating data layer servers as cloud-native components and wrapping them in APIs, intra-service layer service requests appear and behave exactly as application service requests. This makes for a consistent and manageable architecture.  

Figure 1 summarizes this configuration: 

But if the topology spreads over multiple cloud regions, the latency introduced might be unacceptable in some cases. Even in a modest portfolio of use cases, there will be overlap in the domain schemas needed to support them. User, Customer, Product, Location, and others are entities that will be needed over and over again.  

When separating the subject domains and using service requests to “join” them is too slow (and it will be in some cases), other methods must be considered. The right solution for a given workload depends on multiple factors, but the question to be asked of every use of every entity is: Does the transaction demand the latest version?  

For example, to create an order for a customer, we only need the ID of the Customer, which is immutable, and possibly the Name for search and verification. Our service doesn’t need the guaranteed latest version. Asynchronous replication of a distributed Customer database with an availability zone close to us will meet the need.  

Or perhaps change-data-capture (CDC) integration with the core Customer schema might suffice. But this integration will not happen if we’re scheduling deliveries of Orders. We need to ensure the address is the latest point in which we convey to an external delivery service. In this use case, we probably want a data-layer-to-data-layer service invocation of the Customer Location entity (probably served by a dedicated Customer service). We can afford the latency involved because we don't have a web user waiting; this is a back-office function.  

So, a topology that distributes the reference data (the most commonly reused entities), using asynchronous replication to each region, providing a local copy without imposing high latency, is a useful pattern. And because reference data (systems of record) generally change relatively infrequently, asynchronously rippling change across the network will almost certainly be sufficient.  

Figure 2, below, shows this as a database configuration, representing in this case, three domain databases each replicated across three regions: 

In this example, each database would need to be replicated in at least one other availability zone in each region. But the simplified Figure 2 illustrates the concept of distributed replicated databases. 

Agility 

We have been looking at an architectural view of the data layer. And it is important to have that vision. Building a multi-cloud data layer is not a matter for emergent design. The vision is fairly consistent, including the choices, upgrades, and replacement of specific tools. But we don’t need all its capability on Day One. Implementation of data layer capability can be driven by what the business use cases demand of it.  

Develop, communicate, and agree on the vision and strategy for the data layer, but implement as needed.   

Organizational Variability

The material that we have explored here suggests an architectural framework within which application developers must deliver. A well-trodden path makes it easier and quicker to travel from A to B. But modern application developers expect, with good reason, to pick the tools — and path — that best meet their particular needs, for their particular objectives. The days of a central architects’ group imposing a solution space for every problem space have not generally survived in agile cloud-native development.  

However, ignoring the benefits of the well-trodden path, and failing to learn from the experience of others, is an equally undesirable anti-pattern. Data layer architects and developers, following the approach discussed here, will have to find their own balance. This balance will depend on many human and organizational variables beyond this discussion. 

Microservices and cloud-native architectures deliver an effective engine for application modernization, but the modernization of the data layer lags. What is more, the variety of different application types demands multiple workload types, each with their unique needs and capabilities. No single database can meet all these needs and deliver all these capabilities without unacceptable tradeoffs. As a result, we are entering the age of the multi-cloud data layer. 

With a properly architected multi-cloud data layer, you can build your data foundation to handle all cloud workloads with ease across any location or region. It’s a powerful solution for workloads that require resiliency from failure, low latency, and horizontal scaling to meet unbounded and potentially peaky demand.