Designing and Operating Single-Tenant Architectures at Scale

Single-tenant architecture plays a very important role when driven by regulatory/compliance needs, workload isolation to improve security posture and improved performance with dedicated resources. At the same time, these architectures are highly complex and hard to manage, specifically at scale.

In this post, we’ll explore common deployment patterns for single-tenant resources, walk through architecture examples, and share automation and observability best practices to run such environments at scale. This post is platform-agnostic and applicable across off-premise or on-premise setups.

What is a Single-Tenant Architecture?

In a single-tenant architecture, each customer (or "tenant") is provisioned with their own isolated set of resources — from compute and storage to databases and network boundaries. These environments do not share back-end components with other customers and are isolated from other tenants. Outside of regulatory requirements around data sovereignty, the key benefit of a single-tenant architecture is security isolation - reduce blast radius of security vulnerabilities or misconfiguration (access controls) from impacting all tenants.

Common Deployment Strategies

When it comes to your single-tenant deployment strategy, there’s no one-size fits all approach and really depends on your use case and challenges that you are trying to solve with such architecture. Couple of common patterns that organizations tend to implement are: 

1. Dedicated Stack per Tenant: In this pattern, all of the resources from compute/storage/database/networking/monitoring/backup are dedicated per tenant providing complete isolation w.r.t security posture, operational availability and compliance requirements. However, this pattern also brings in complexity in management at scale and is more expensive with redundant dedicated databases for strong tenant metadata, logging and monitoring infrastructure.
2. Hybrid Model: In this pattern, while the underlying compute and the application is isolated per tenant, the metadata store, monitoring, logging is centralized or multi-tenant in nature to improve scalability. This pattern, while certain components are not isolated still ensures improved security and availability through isolation of specific components such as the software application, data store for the software application, networking components, storage and compute instance.

In this post, we will further explore the Hybrid Model and best practices.

Hybrid Model Architecture

This high-level architecture provides per-tenant isolation for core application resources with shared observability and automation to reduce the management overhead that comes with dedicated stack pattern.

1. API Layer: The API Layer is multi-tenant, enabling customers with a single interface to provision their single-tenant stacks with customized configurations for their software application (storage, compute size, networking, application configuration). This simplifies the overall management as compared to having these APIs as single-tenant. One of the key advantages of this approach is to simplify IAM (Identity and Access Management) for customers as they can setup their access control once instead of per single-tenant API and then building additional tooling to keep them all in sync.
2. Metadata Store: A multi-tenant metadata store simplifies governance while ensuring each tenant is mapped to a unique stack along with their preferred configurations. The multi-tenant metadata store simplifies operational management of the the single-tenant stacks, governance and overall life-cycle management of the stacks.
3. Provisioning Pipeline: A CI/CD provisioning pipeline ensures the single-tenant stacks are created per available stack templates.
4. Per Tenant Stack: This is the core single-tenant resource or group of resources that are isolated and dedicated per customer ensuring the application data/secrets are specific to the tenant.
5. Central Monitoring: The central monitoring component is a multi-tenant system which relies on single-tenant health metrics to manage the overall availability of the single-tenant resources. Building a centralized monitoring system enables the system to scale and have a separate deployment process to add new monitors, tweak thresholds without modifying the underlying single-tenant stack.
6. Central Logging: A central logging system enables logs from each single-tenant stack to be logically separated (unique tenant id) but at the same time enables central security monitoring of logs and retention per compliance needs.
7. Central Automation: This system is responsible for automated remediation of operational/health issues of single-tenant stacks based on the monitoring metrics that the monitoring system reports back to the metadata store. For example, this central automation system can rebuild stacks that are unhealthy or scale out stacks horizontally or vertically based on performance needs.

Automating Tenant Provisioning With CI/CD

Scaling single-tenant stacks without automation is a nightmare! The provisioning of these stacks should be through CI/CD pipelines for all life-cycle management operations such as provision, update, scale up/down, decommission. For example, trigger the provisioning pipeline based on events such as a new request from the API layer or from the central automation system for remediation actions such as performance based scale up/down, rebuild from backup.

Operational Best Practices

Centralize Observability: Route all logs and health metrics to the central monitoring and logging system to allow centralized rules and thresholds to detect anomalies and trigger the Central automation to perform automated remediation of the single-tenant stacks. The key here is to ensure the unique tenant id is your primary dimension. In addition to the central observability, build tenant-id specific dashboards through automation to have deeper insights.

Deployment Safety: While a CI/CD deployment pipeline removes the management bottleneck in the Hybrid approach, it also comes with operational risks of impacting multiple single-tenant stacks with one bad deployment. Ensuring test coverage is important but bugs have special powers to circumvent the best of test coverages so it is critical to have deployment guard rails such as staggered deployments/ bake times/ automated rollback to reduce blast radius of impact and early detection.

Backup and Restore: With a single-tenant stack, your customers have control over the software application and it is essential to have regular backups and ability for your tenants/customers to restore their stack from a point in time. The central automation system can be leveraged to ensure scheduled backups are performed.

Conclusion

Single-tenant architectures don’t have to mean manual toil or scaling limitations. With the right mix of isolation at the layers that is important, shared observability and automation, you can deliver secure, reliable and scalable system for each tenant.