Cloud Architecture Resources

DZone's Featured Cloud Architecture Resources

Building a Practical Cloud-Native Golden Path: A Guide to Kubernetes-Based Service Delivery, Self-Service, and Developer-Friendly Defaults

By Naga Santhosh Reddy Vootukuri

CORE

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale. Every engineering organization that I have worked with eventually faces the same issue, which is that each team ships services differently. One team used Helm, another wrote raw manifests, and a third would have built a custom Bash script. As these different approaches accumulate, the supporting deployment steps often end up scattered across multiple Wiki pages that quickly go stale. New engineers then spend their first two weeks copying configuration values from an old repository and hoping they still work. A golden path fixes this without turning the platform team into a gatekeeper. It provides users with a standardized workflow for the shortest and most obvious route from a fresh repo to a production workload. This guide walks you through designing a minimum viable golden path, where guardrails belong, and how to keep it useful after v1. Choose the First Golden Path Start with one workflow to standardize first; the strongest candidate is usually the workflow your teams ship most often, or one that teams experience the most friction with. In many organizations, that workflow is a stateless HTTP service exposing a REST or gRPC API endpoint, deployed to Kubernetes and owned by one application team. For this walkthrough, we will use orders-api, a stateless HTTP service on Kubernetes, as our reference throughout this article. The intended users are application developers, not platform engineers — those who create the golden path itself. The path starts with a create-service command in a CLI or a form in an internal developer portal. It should end when the service is running in production with logs, metrics, ownership, and on-call rotation attached. Keep the first version deliberately narrow. A workload that needs GPU nodes, a queue-driven scaling model, or a stateful sidecar can wait. Trying to capture every exception at the beginning turns a practical delivery path into a long platform program. A golden path’s success criteria are qualitative, not quantitative. Analyze the first release by user adoption and experience. Are teams using standardized workflows instead of copying an old repository? Can a new engineer understand the end-to-end deployment process without asking around? Are on-call handoffs easier because services have the same operational shape? The answers to these questions matter more than looking at any adoption numbers displayed on a dashboard in the first few months. Define What the Path Standardizes A golden path is a curated set of decisions that are made once and reused consistently across services: The workload template should provide a Dockerfile, fully maintained base image, Kubernetes manifests, probes, resource requests and limits, a Pod Disruption Budget (PDB), autoscaling defaults, and consistent labels.The delivery pipeline should build, test, scan, sign, and publish the image.The platform defaults should include namespace rules, quotas, network policies, ingress, TLS, logging, metrics, tracing, and basic alerts. The path should not own product decisions; teams will still choose their language, framework, business logic, schema, feature flags, test strategy, and service-specific objectives. This boundary is very important. If we over-standardize, developers will work around the platform, and if we under-standardize, every instance will start with a different set of commands and dashboards. Also make sure the path is easy to find. One internal documentation page, one command, and one entry in the developer portal are enough. If a developer has to ask which template to use, the path has already failed and created friction. The table below shows the differences between shared standards the path owns and decisions each service team owns. Shared Standards vs. Team-Owned Decisions shared standard team decision Dockerfile, base image, patching cadence Language and framework choiceDeployment manifests, probes, resource requests/limits, PDB, Horizontal Pod Autoscaler Business logic, schema, feature flags Build, test, scan, sign, and publish pipeline Test suites specific to the service Namespaces, quotas, network policies, ingress, and TLS defaults Non-standard scaling (queue-driven consumers, GPU jobs) Logging, metrics, tracing, and alerting defaults Business-specific dashboards and SLOs Turn Common Requests Into Self-Service Actions Once the path is created and available to users, review the top 10 tickets your platform team receives. Look for repeated requests such as creating namespaces, adding a database, registering a DNS name, rotating a secret, or creating another environment. These are all good candidates because the desired outcome is already understood, and the steps are mostly predictable. For the Orders API golden path, the platform team can provide the following self-service actions and apply guardrails based on the risk from each change: Fully automated. These actions are reversible and have a limited blast radius. Creating a development namespace for orders-api, spinning up a preview environment on a PR, or rotating a non-production secret happens on demand without a human involved to review.Light review. Actions that change cost, security exposure, or shared infrastructure should require a light review. Provisioning production Postgres for orders-api opens a pre-filled change request that needs one approval. A new public DNS record on a shared domain is reviewed through a one-click approval on a pre-filled PR.Approval mechanism. Every self-service action generates a PR against a config repo, pre-fills the values, tags the reviewer, and merges on approval. The change flows through the same pipeline as code, and every action leaves an audit trail because it’s a git commit. The self-service interface should offer supported choices instead of exposing raw cloud APIs. For example, allowing every team to choose any PostgreSQL version, instance class, or backup schedule can leave the platform team operating 30 different database configurations. A better approach is to provide a small, opinionated set of options such as small, medium, and large. This gives developers enough flexibility while keeping the operational model understandable. For our Orders API, the developer-facing configuration can stay small: YAML # svc.yaml name: orders-api owner: team-orders tier: standard # small | standard | high runtime: http dependencies: - kind: postgres size: small # opinionated preset, not raw config on_call: orders-oncall The configuration captures the developer’s intent, while the golden path translates each request into an approved action with the right guardrail and a clear record of what happened. The table below shows how this works for the Orders API. Orders API Self-Service Actions, Guardrails, and Evidence Step Self-Service Action Guardrail Evidence Create service Run svc new via CLI or submit a portal form Template pinned to current version; namespace quotas applied Repository created with owner metadata; entry in service catalog Add dependency Pick from opinionated list (small/medium/large DB) One-click PR review for prod-tier resources Merged PR against config repo with reviewer name Deploy to prod Merge to main triggers promotion Progressive rollout with auto-rollback on error/latency signals Deployment record with canary metrics and rollback status Rotate secret Run svc rotate-secret New version issued; old version revoked after grace window Audit log entry linked to requester Create a Consistent Path From Code to Deployment Every service on the golden path should move through the same basic stages: pull request → merge to main → staging → production. The exact tooling can vary, but the meaning of each stage should not. At the PR stage, CI runs unit tests, linting, the container build, and security checks. Produce an immutable image tagged with the commit identifier, but do not deploy it to production.On merge to main, the same image is promoted to staging automatically. Rebuilding at each stage creates uncertainty because the artifact tested is no longer guaranteed to be the artifact released. Run integration and smoke tests in this stage.Promoting the image to production reveals the delivery guardrails. Start with a small percentage of traffic (5-10%), monitor health signals, and continue increasing traffic to 25%, then 100%. Roll back automatically when error rate, latency, or probe failures cross agreed thresholds. A developer should not have to recreate this logic in every repository — it should be baked into the deployment tooling. A failed orders-api canary would look like this end to end: The pipeline promotes the new image to 5% of production pods.The error rate for the /orders endpoint rises sharply during the observation window.The deployment controller restores the previous image and drains the new pods based on the rollback threshold.The pipeline posts a message in the orders-oncall service channel with a link to the failing dashboard and offending commit identifier (SHA).An incident record is created automatically only when rollback fails, or the service remains unhealthy. Teams may skip a stage for a documented case (e.g., configuration-only change), but the exception should be an explicit setting with an owner, not an informal workaround. Plain Text # pipeline stages (pseudo) on_pr: [test, lint, build, scan, sign] on_merge: [promote_to_staging, integration-tests] on_green: [canary-5, wait-signals, canary-25, wait-signals, full-rollout] On_regress: [auto-rollback, notify-oncall, record-failure, open-incident] Observability and Day-1 Operational Defaults Even if its pods are running, a service is not ready until the owning team can determine whether it is healthy and knows what action to take when it is not. The golden path should therefore create the minimum operational surface at the same time as the service. The template includes the following list on day one: Structured logs to the central log store, with request ID and trace identifiersRequest rate, error rate, latency percentiles, and saturation metricsDistributed traces with a platform-managed sampling defaultA standard dashboard created from the service nameAlerts for high errors, high latency, restart loops, and resource pressureLiveness and readiness checks connected to a health endpoint Ownership should also be captured during service creation. Ask for the team, on-call rotation, and support channel, then reuse those values in alert routing, the service catalog, and the runbook. Generate a simple runbook with sections dedicated to common failures such as stalled deployments, elevated errors, and pod eviction. A partially completed runbook with a familiar structure is far more useful than a blank page, and consistency here pays off during an incident. Keep the Golden Path Useful Over Time Exceptions are inevitable, so record the failure reason, owner, and expiry date rather than letting the exception become a permanent member. At review time, either the service returns to the path or the platform team decides the pattern is common enough to support. Treat templates and defaults like product code: review changes, version them, and provide a propagation method. When a base image or manifest default changes, open a change against each service instead of relying on teams to notice a document update. Silent drift is one of the fastest ways to lose developer trust in the path. Track a small set of signals such as the time from service creation to first production deployment, template version distribution, open exceptions, and the percentage of new services created through the path. Pair those numbers with developer feedback. A slow step that teams repeatedly bypass tells you where the next path improvement belongs. A new template version without a propagation plan becomes a fork. Extend the path when a pattern is used by three or more teams, but keep it narrow while it is still one team’s edge case. Plain Text # template bump propagation (pseudo) on template_release(new_version): for svc in services_on_path(): open_pr(svc, bump_template = new_version, auto_merge = svc.opts.auto_bump, reviewer = svc.owner) Making the Golden Path Useful in Practice A golden path succeeds when it is easier to follow than to work around. Start with one common workflow, standardize what is shared, and leave product choices with the service team. Make routine actions self-service, place checks in the delivery flow, and include observability from the first deployment. Usage signals can then inform future improvements to the path. A small path that ships, earns trust, and changes steadily will have a greater impact on engineering speed than a broad platform program that remains unfinished. Resources: CNCF TAG App DeliveryOpenTelemetry General Semantic ConventionsKubernetes Pod Security StandardsBackstage Software Templates“Building a CI/CD Pipeline With Kubernetes” by Naga Santhosh Reddy VootukuriKubernetes Security Essentials, DZone Refcard by Yitaek HwangPlatform Engineering Essentials, DZone Refcard by Apostolos Giannakidis This is an excerpt from DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale.Read the Free Report More

Cloud Complexity Is an Operating Model Problem: Why Infrastructure Maturity Alone Can’t Solve Scale, Reliability, and Team Friction

By Igboanugo David Ugochukwu

CORE

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale. After a few years of operating a shared Kubernetes environment, the shift in the center of gravity becomes clear. Cluster provisioning, container scheduling, and upgrades become routine, yet releases still stall over ownership, access, telemetry, and cost allocation. Consider a hypothetical product team adding a stateful order-processing service to a shared platform. The service has an API, a worker, database migrations, and a data store backed by cluster-managed persistent storage, and it must run in staging and production. We will follow that service through its delivery path to examine where mature infrastructure stops helping, how local workflow differences compound, and which operating model decisions restore consistency without stripping teams of useful autonomy. When Infrastructure Maturity Stops Solving the Hard Part At first glance, onboarding the service should be routine. The cluster exists, the CI system can build an image, and infrastructure as code can create the namespace. Then the service reaches production and encounters a different StorageClass, quota profile, network policy, or service account configuration from staging. Each difference may be valid, but the delivery workflow didn’t surface the environment contract early enough. This is the practical limit of infrastructure maturity. Reliable clusters provide capable building blocks, while reliable delivery also requires a shared agreement about how teams use those blocks, what evidence a release produces, where exceptions go, and who owns the outcome. How Cloud Complexity Starts to Compound Follow the service through one release and the pattern becomes clearer: The same components use inconsistent service and environment identifiers across logs and traces.Ownership labels exist in one cluster but not the other.The team copies a pipeline because the shared template cannot sequence migrations.Production access and policy exceptions move through separate ticket queues. The operational cost shows up in the manual coordination required before each deployment. An engineer has to reconstruct which rules apply every time. During an incident, responders can’t move cleanly from an alert to the owning team, deployment record, runbook, and cost center. Finance sees shared-cluster spend that cannot be attributed reliably, while the security team receives evidence in different formats. OpenTelemetry semantic conventions and FinOps allocation practices rely on consistent service, environment, and allocation metadata, so local naming schemes undercut the value of the underlying tools. As the same pattern spreads across clusters and cloud accounts, small differences become a persistent operating burden. Operating Models Set the Terms of Scale The operating model decides who turns those building blocks into a usable delivery system. For our example, the product team owns the order domain, data model, migration safety, scaling behavior, SLOs, and on-call response. The platform team owns the interface through which the service receives a namespace, workload identity, baseline policy, deployment workflow, and telemetry defaults. Security, SRE, and FinOps teams contribute requirements and review the evidence that the workflow produces. This split keeps service-specific decisions close to the people who understand them while centralizing cross-cutting capabilities that every team would otherwise rebuild. CNCF’s platform guidance makes an important distinction here: A platform team is responsible for the interfaces and experience around shared capabilities, even when another team or provider operates the backing service. In practice, a mature platform offers versioned workflows, clear support boundaries, self-service for common requests, and feedback loops based on real usage. In this way, the platform team is an enabler of consistency rather than the operator of every component. Standardization, Autonomy, and Shared Operating Logic The defensible baseline is narrower than a universal application architecture. For this service, shared standards should cover: Service and environment identityWorkload identity and minimum network policyResource requests, quota expectations, and cost-allocation metadataRelease evidence, rollback behavior, and minimum telemetry These rules belong in the shared workflow because inconsistency affects other teams and complicates incident response, security, and cost allocation. The product team still chooses its schema, partitioning strategy, cache design, scaling thresholds, and release timing, and defines SLOs around the behavior users experience. The stateful workload then tests that boundary. A default pipeline built for stateless HTTP services may need a supported hook for migrations and worker rollout. An overly broad standard becomes an approval layer or bottleneck, while an overly narrow one leaves every team maintaining its own release and recovery logic. A bounded extension with an owner, tests, constraints, and review date preserves autonomy without creating an unsupported parallel system. Why Team Friction Turns Into a Scaling Tax Weaknesses in the operating model become most visible in the friction between teams. If a developer must request a namespace, ask another team for credentials, copy a pipeline, and find a production approver, the architecture may be automated while delivery remains ticket-driven. Each handoff adds queue time and loses context. Adding a portal without changing that path gives the developer one more place to check. Effective self-service completes the request, applies policy, records the change, and returns a clear support path. To see whether self-service is reducing friction, track metrics like request-to-environment time, time to first production deployment, exception rate, support demand, and failed-deployment recovery time. CNCF recommends tracking fulfillment and new-service delivery latency; DORA advises applying delivery metrics in the context of a specific service. Together, these measures show whether the workflow reduced coordination overhead or moved it to another queue. When Control Models Backfire The order-processing service example exposes two ways the control model can fail: Overly rigid standardization. A workflow designed only for stateless services forces the team to create a separate migration path, fragmenting release evidence.Unbounded local variation. Unrestricted cluster access allows identity, policy, and resource controls to drift between teams. The scalable approach pairs a narrow baseline, enforced through mechanisms like admission policies, with a documented extension path for legitimate workload-specific behavior, keeping the standard credible without turning each exception into a permanent fork. Operating Assumptions That Fail at Scale Old Assumption Why It Breaks Operating Model Replacement Healthy clusters make a workload portable Storage, identity, policy, and quota profiles differ by environment Versioned environment contract with a shared baseline One shared pipeline can serve every workload Stateful rollout and migration steps don’t fit the default sequence Core workflow with bounded, tested hooks A portal provides self-service Tickets and manual approvals remain behind the interface Workflow that provisions, enforces policy, and records evidence Local conventions remain harmless when teams own their services Metadata and controls drift across services Small enforced baseline with governed exceptions Making Cloud Complexity More Manageable To begin, you don’t need to redesign your entire platform. You can trace one representative delivery workflow and find where coordination breaks. For the order-processing service example, map the path from repository creation to production, including owners, queues, controls, evidence, and exceptions. Improvements should then be tested through adoption and outcomes such as lead time, failed-deployment recovery time, support demand, exception volume, and cost-attribution coverage. This sequence shows whether the platform is reducing operational variation for real workloads before the model expands to more teams and environments. References: Platforms for Cloud-Native Computing, CNCFResource Quotas, KubernetesStorage Classes, KubernetesAdmission Control in Kubernetes, KubernetesResource Semantic Conventions, OpenTelemetryAllocation FinOps Framework Capability, FinOps FoundationService Level Objectives, Google SRESoftware Delivery Performance Metrics, DORA This is an excerpt from DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale.Read the Free Report More

Kubernetes Operations Playbook: The Essentials for Keeping Scale, Complexity, and Drift Under Control

By Abhishek Gupta

CORE

Your Terraform Monolith Isn't Too Big. It's Tightly Coupled.

By Naveen Kalapala

Edge AI: Why Inference Is Moving Away From the Cloud

By Uthej Mopathi

CORE

Understand the Sidecar Pattern by Deploying n8n to AWS Fargate

A sidecar is a container that runs alongside another container as part of the same deployment unit. Just because two containers are in the same cluster or deployed around the same time doesn't make one a sidecar. There are two things that make a sidecar. First is that they share a network namespace, so they can reach each other over localhost rather than a network address. Second, they share a lifecycle. This means that they are created together, scaled together, and by default torn down together. Neither container has an existence independent of the other. The problem it solves is giving a specific concern its own boundary. For example, it can have its own filesystem, its own memory space, and often its own permissions or dependency set, without giving up the simplicity of deploying and operating one unit. You get isolation without paying for the operational overhead of running and coordinating a fully separate service. The test that defines the pattern across all of these is this: does it live and die with its partner container as one unit of deployment? If yes, it's a sidecar. If you have to reach it by hostname, through service discovery, or via a queue, it isn't one anymore. That is a separate service that happens to sit next to the first. That test matters because two adjacent patterns get called "sidecar" when they aren't: Decoupled worker/microservice. A separately deployed container, reached over the network, scaled on its own. A web application offloading work to Celery workers via Redis is a common instance of this: the app enqueues a job (send this signup email), a pool of workers pulls jobs off the queue independently, and neither side shares a network namespace or a lifecycle with the other. The workers scale on queue depth, not on how many web replicas are running, and a web app restart doesn't take queued or in-flight jobs down with it. n8n has its own version of the same shape: "queue mode," where a main node accepts webhooks and separate worker nodes pull jobs off a Redis queue. It's tempting to call either of these a sidecar relationship since the worker and the web app do feel paired, but neither qualifies: they don't share a deployment unit, and killing one doesn't touch the other.Ambassador/adapter. A container that proxies or translates traffic on its parent's behalf, like the Envoy example above, is actually this, more precisely. Structurally it's still a sidecar; it just gets a more specific name for what it does. Using n8n to Understand It What n8n Is n8n is a workflow automation platform like Zapier, but self-hostable and node-based rather than form-based. A handful of components make up a running instance: The editor/UI, where workflows are built visually as a graph of nodes.The main process, which serves that UI, listens for webhooks, and orchestrates workflow execution. The workflow execution decides what runs next, passing data between nodes and recording results.Nodes, the individual units of a workflow: trigger nodes (a webhook arrives, a schedule fires), action nodes (call an API, write to a database, send an email), and the Code node. The code node lets you drop in arbitrary JavaScript or Python to transform data however the built-in nodes can't. The code node is relevant in this article. The database, where workflow definitions, credentials, and execution history persist. In this article, Postgres is used. For most of what n8n does, the main process is the only thing doing work: routing a webhook, calling an API, writing a database row. The exception is the Code node, and that exception is the whole reason task runners exist. The Task Runner Feature and Its Use Case By default, a Code node's JavaScript or Python executes inside n8n's main process. This main process holds the database connection, the encryption key, and every credential stored in every workflow you've built. That's fine for trusted, well-understood scripts. It becomes a real problem the moment the code in that node is untrusted, third-party, or arbitrary enough that you can't fully audit it before it runs. By the way, that is how most Code nodes are used in practice. Task runners exist to solve exactly that use case: run Code node logic somewhere the main process's credentials and connections aren't reachable from it, without turning "write some JavaScript to reshape this JSON" into a separately deployed microservice every time. Going Deep on the Task Runner Feature n8n ships two modes for this: Internal mode (the default) runs Code nodes inline, in-process. No isolation. This is the fastest to set up, but the weakest boundary.External mode moves execution into a separate runner process entirely. That process connects back to the main n8n instance over a broker (an authenticated connection the main process listens on) and receives individual tasks to execute rather than having any standing access to n8n's internals. The runner never touches the database connection, the encryption key, or stored credentials directly; it only ever sees the specific input data for the task it's been handed. External mode goes further than just "a different process," too. The runner's own configuration (the n8n-task-runners.json file built in Phase 4) sets explicit allowlists — which environment variables the runner process can see at all, and which JavaScript built-ins or Python modules it's permitted to import, standard library and third-party tracked separately. So the boundary isn't just "different memory space," it's "different memory space, plus a declared, auditable list of exactly what this process is allowed to touch." That's a specific concern (arbitrary code execution) given its own boundary, without turning it into a fully independent service you have to deploy, discover, and monitor separately. It's the sidecar problem, stated exactly: external mode gives you the isolation; running the external runner as its own container in the same task definition is what makes that isolation a sidecar rather than just a separate process sharing a machine. Why This Needs to Scale Independently and Why "In the Same Container" Isn't Enough Most n8n deployment guides run n8n with task runners in internal mode, or with the external runner living inside the same container as the main process. For example, you will see guides about deploying n8n on a single EC2 instance, Render, DigitalOcean, or any platform's basic tier. That gets you the process isolation, which solves the security half of the problem. It doesn't solve the other half, which is that a runner sharing a container with the app can't be scaled, resourced, or restarted independently of it. That stops mattering the moment Code-node execution becomes the actual bottleneck rather than webhook handling or UI traffic. Imagine workflows doing heavy data transformation in Python, running numpy/pandas operations across large payloads, or executing many Code nodes concurrently. If the runner is bundled into the main container, giving it more CPU means giving the entire n8n instance more CPU, whether the UI and webhook layer need it or not. There's no way to say "the runner needs 2 more vCPUs, n8n itself is fine". Why AWS Fargate's Task Definition Is the Right Fit A Fargate task definition lets each container in the task carry its own CPU and memory reservation, its own health check, and its own essential flag governing what happens if it fails while still keeping every container in the task on one shared network interface. That's the sidecar promise made literal: isolation and independent resourcing for the runner, without losing the operational simplicity of one task, one deploy, one thing to scale as a unit when you do want to scale both together. The rest of this guide deploys exactly that: one Fargate task, two containers, wired together the way the definition above requires. Each infrastructure decision below gets tied back to a specific part of what's laid out here, so that by the end, the concept isn't something read once at the top, but it's something built. Prerequisites AWS account with billing enabledA domain you control, with DNS accessDocker installed locally, with docker buildx availableAWS CLI configured (aws configure) with permissions for ECR, ECS, RDS, ACM, and IAMThe runner image source (Dockerfile + n8n-task-runners.json) — built in Phase 4 Architecture Markdown User's Browser (HTTPS) | [Application Load Balancer] <- Certificate Manager (SSL Cert) | (Port 5678, HTTP internal) [ECS Fargate Task] |-- Container: n8n (main) <-- shared network namespace --> Container: n8n-runner (sidecar) | (Port 5432, PostgreSQL) [RDS PostgreSQL Database] The load balancer and RDS layers are ordinary AWS plumbing. The box in the middle is where the sidecar relationship actually lives. There is one task and two containers, each with its own resourcing. Phase 1: RDS PostgreSQL RDS Console → Create database → Standard create → Engine: PostgreSQLDB instance identifier: n8n-db. Master username: postgres. Generate and save a strong master password.Instance size: db.t4g.microStorage: 20 GB gp3, autoscaling on, max 100 GBConnectivity: the VPC you'll use throughout. Public access: No. New security group: n8n-db-sg, left empty for now.Additional configuration → Initial database name: n8n. Skip this and n8n fails on first connect with "database does not exist" — the DB instance identifier names the server, this field names the database inside it.Create, wait for "Available," copy the endpoint from Connectivity & security. Phase 2: ACM Certificate n8n requires HTTPS for webhooks to function Certificate Manager, in the same region you'll deploy the Load Balancer in → Request a public certificateDomain name: n8n.yourdomain.comValidation method: DNS validationCreate the CNAME record ACM provides at your registrar. If your registrar auto-appends your domain to the Host field, paste only the portion before your domain — the full string duplicates it and validation never completes.Wait for status: Issued Phase 3: Security Groups Two connections need rules: Security groupInbound rulePurposen8n-alb-sg443 from 0.0.0.0/0Public HTTPSn8n-ecs-sg5678 from n8n-alb-sgALB → n8n containern8n-db-sg (edit existing)5432 from n8n-ecs-sgn8n container → RDS Phase 4: Build and Push the Runner Image Dockerfile: Dockerfile FROM n8nio/runners:1.121.0 USER root RUN cd /opt/runners/task-runner-javascript && pnpm add moment uuid adm-zip RUN cd /opt/runners/task-runner-python && uv pip install numpy pandas pydantic requests boto3 certifi COPY n8n-task-runners.json /etc/n8n-task-runners.json ENV N8N_RUNNERS_CONFIG_FILE=/etc/n8n-task-runners.json USER runner It starts from n8n's own n8nio/runners base (containing the launcher and both runner processes), adds only the dependencies workflows actually need, and drops back to a non-root user once the root-only install steps finish. n8n-task-runners.json is where the isolation described above stops being architectural and becomes enforced: JSON { "task-runners": [ { "runner-type": "javascript", "health-check-server-port": "5681", "allowed-env": ["PATH", "GENERIC_TIMEZONE", "NODE_OPTIONS"], "env-overrides": { "NODE_FUNCTION_ALLOW_BUILTIN": "crypto,zlib", "NODE_FUNCTION_ALLOW_EXTERNAL": "moment,uuid,adm-zip" } }, { "runner-type": "python", "health-check-server-port": "5682", "env-overrides": { "N8N_RUNNERS_STDLIB_ALLOW": "json,zipfile,io,base64,datetime,re,math,random,statistics", "N8N_RUNNERS_EXTERNAL_ALLOW": "numpy,pandas,pydantic,requests,boto3,certifi" } } ] } allowed-env restricts which environment variables the runner process can see; N8N_RUNNERS_STDLIB_ALLOW / EXTERNAL_ALLOW restrict which Python modules it can import, stdlib and third-party separately. One container, two runner processes — the launcher inside n8nio/runners spawns both. Build and push: Shell docker buildx build -t n8nio/runners:custom . aws ecr create-repository --repository-name n8n-runners --region us-east-1 aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com docker tag n8nio/runners:custom <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/n8n-runners:custom --username AWS is a fixed literal, not your actual username — ECR auth always uses it. The password piped via --password-stdin is a short-lived token generated by the CLI, not your account password. Phase 5: The Task Definition This is where the two containers become an actual sidecar pair, and where the independent-resourcing argument from the introduction becomes a real field rather than a claim. JSON { "family": "n8n-task", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "1024", "memory": "2048", "executionRoleArn": "arn:aws:iam::<account-id>:role/n8n-task-execution-role", "containerDefinitions": [ { "name": "n8n", "image": "n8nio/n8n:1.121.0", "essential": true, "entryPoint": ["sh", "-c"], "command": [ "mkdir -p /home/node/certs && wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem -O /home/node/certs/rds-ca.pem && /docker-entrypoint.sh" ], "portMappings": [{ "containerPort": 5678, "protocol": "tcp" }], "environment": [ { "name": "DB_TYPE", "value": "postgresdb" }, { "name": "DB_POSTGRESDB_HOST", "value": "<rds-endpoint>" }, { "name": "DB_POSTGRESDB_PORT", "value": "5432" }, { "name": "DB_POSTGRESDB_DATABASE", "value": "n8n" }, { "name": "DB_POSTGRESDB_USER", "value": "postgres" }, { "name": "DB_POSTGRESDB_SSL_CA", "value": "/home/node/certs/rds-ca.pem" }, { "name": "DB_POSTGRESDB_SSL_REJECT_UNAUTHORIZED", "value": "false" }, { "name": "WEBHOOK_URL", "value": "https://n8n.yourdomain.com/" }, { "name": "GENERIC_TIMEZONE", "value": "Africa/Lagos" }, { "name": "N8N_RUNNERS_ENABLED", "value": "true" }, { "name": "N8N_RUNNERS_MODE", "value": "external" }, { "name": "N8N_RUNNERS_BROKER_LISTEN_ADDRESS", "value": "0.0.0.0" }, { "name": "N8N_RUNNERS_BROKER_PORT", "value": "5679" } ], "secrets": [ { "name": "DB_POSTGRESDB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/db-password" }, { "name": "N8N_ENCRYPTION_KEY", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/encryption-key" }, { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n" } } }, { "name": "n8n-runner", "image": "<account-id>.dkr.ecr.<region>.amazonaws.com/n8n-runners:custom", "cpu": 512, "memory": 1024, "essential": false, "dependsOn": [{ "containerName": "n8n", "condition": "START" }], "environment": [ { "name": "N8N_RUNNERS_TASK_BROKER_URI", "value": "http://localhost:5679" } ], "secrets": [ { "name": "N8N_RUNNERS_AUTH_TOKEN", "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:n8n/runners-auth-token" } ], "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:5680/healthz || exit 1"], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 20 }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/n8n-task", "awslogs-region": "<region>", "awslogs-stream-prefix": "n8n-runner" } } } ] } Five fields here map directly back to the introduction: Per-container cpu/memory on n8n-runner. This is the independent-resourcing argument made literal. The runner gets its own 512 CPU units and 1024 MB, carved out of the task total, separate from whatever n8n is allotted. If Code-node execution turns out to be the bottleneck, this is the number you raise without touching the main container's allocation at all. That's the exact thing a same-container runner can't offer you. networkMode: awsvpc is the mechanical basis of "shared network namespace." Every container in the task gets one elastic network interface between them. This is the setting that makes Phase 3's missing security group rule make sense. There's one network surface, not two. N8N_RUNNERS_TASK_BROKER_URI: http://localhost:5679 only works because of the line above. The runner reaches n8n over localhost because they are the same task. If this pointed anywhere else, you would have built the decoupled-worker pattern from the introduction instead, no matter what you called the container. A shared N8N_RUNNERS_AUTH_TOKEN, pulled from Secrets Manager by both containers. Sharing a network namespace means the runner is reachable by anything else in the task. The isolation the whole pattern exists for still needs a trust boundary at the process level, not just the network level. A plaintext token here would defeat that, since task definitions are readable by anyone with ecs:DescribeTaskDefinition. essential: false on the runner. This governs how tightly the two containers' lifecycles are actually coupled. essential: true would mean a runner crash tears down the whole task, main container included. false means the runner can crash and recover independently: Code-node executions fail until it's back, but the UI and webhooks keep serving. The pattern doesn't mandate one answer; it just means this has to be a decision, not a default you inherited. The health check on port 5680 hits the launcher's own endpoint, separate from the per-runner-type ports (5681 JS, 5682 Python) set in Phase 4's config file. ECS is checking the supervisor, not each runner process individually. Register it: aws ecs register-task-definition --cli-input-json file://n8n-task-def.json Phase 6: Cluster, Service, and Load Balancer ECS → Create cluster → n8n-cluster → Infrastructure: AWS FargateCreate a service inside it: Task definition: n8n-task, latest revisionDesired tasks: 1Networking: your VPC, at least two subnets across AZs, security group n8n-ecs-sg, public IP onLoad balancing: Application Load Balancer, listener on 443 using the Phase 2 certificateTarget group: HTTP, port 5678, health check path /healthzCreate, wait for steady state. Notice the target group and health check only ever reference the n8n container. It did not mention n8n-runner at all. The n8n-runner container doesn't get a port that maps to the load balancer, doesn't get its own listener, doesn't get its own DNS entry. Everything that makes it reachable from outside the task goes through n8n . Phase 7: DNS At your registrar, add a CNAME: Host n8n, Value = your Load Balancer's DNS name. Confirm with nslookup n8n.yourdomain.com once it propagates. Verifying the Sidecar Relationship Visiting https://n8n.yourdomain.com and completing owner setup confirms the main container and database are working. To confirm the runner specifically: Create a workflow with a Code node (JavaScript or Python), and run it.Pull CloudWatch logs for both streams (/ecs/n8n-task, prefixes n8n and n8n-runner). The n8n-runner stream should show the launcher starting both runner processes and reporting a broker connection. The n8n stream should show the Code node's execution dispatched out rather than run inline. If the workflow completes but nothing appears in n8n-runner's logs, check N8N_RUNNERS_MODE=external on the main container first. That's the setting that actually hands execution off instead of running it in-process regardless of what else is configured.

By Iyanuoluwa Ajao

Porting GPU Drivers to Rust on ARM64: The Hardest Trial for Kernel-Level Computing

Rust Has Entered the Kernel. Now Comes the Dangerous Part. A kernel driver does not fail politely. It does not throw a friendly exception, generate a neat stack trace, and ask whether you would like to restart. It corrupts memory, wedges hardware, leaks secrets, freezes the compositor, and leaves engineers spelunking through logs at 2 a.m. with the emotional range of a haunted printer. Nowhere is this more obvious than in GPU drivers. GPU drivers are some of the most complex pieces of kernel-level software in modern systems. They sit between userspace graphics APIs, memory managers, firmware, hardware queues, display engines, DMA buffers, synchronization fences, interrupts, power states, and error recovery paths. They are not “just drivers.” They are operating systems within the operating system. That is why Rust in GPU drivers matters. Rust support exists in the Linux kernel documentation today, but the kernel documentation is careful about its scope: Rust support is still primarily aimed at kernel developers and maintainers building abstractions, drivers, infrastructure, and tooling. It is not a blanket promise that every Rust kernel module is production-ready everywhere. That caution is healthy. Kernel engineering is allergic to magic, and rightly so. Rust does not sprinkle safety dust on MMIO registers. It does not fix firmware bugs. It does not turn a bad architecture into a good one. But Rust does attack one of the oldest and most expensive problems in systems software: memory unsafety. And if Rust can survive in GPU drivers on ARM64, it can survive almost anywhere. Why GPU Drivers Are the Real Test Most Rust-in-kernel discussions start too gently. They talk about simple drivers, toy modules, or safe wrappers around existing C APIs. Useful, yes. Convincing, not enough. The real test is graphics. A modern GPU driver must handle: - Device probing and initialization - Firmware loading and communication - Command submission queues - Shared memory between userspace, kernel, and device - DMA buffer ownership - Synchronization fences - Interrupt handling - Runtime power management - GPU reset and recovery - Userspace ABI compatibility - Performance under real graphical workloads This is where C’s sharp edges show up in full costume. A buffer may outlive the object that owns it, a command queue may still point to freed memory, or firmware may enter a state it should never reach. Reset and teardown paths add more chances for things to go wrong, especially if userspace still holds handles, an interrupt arrives at the wrong moment, or a fence is never signaled. These are not rare problems. They are normal driver engineering problems. The security motivation is also real. Google reported that memory-safety vulnerabilities accounted for 76% of Android vulnerabilities in 2019 and 24% in 2024 after shifting new development toward memory-safe languages. Google later reported major reductions in memory-safety vulnerability density for Rust code compared with Android’s C and C++ code, along with lower rollback rates and less code-review time for Rust changes. Do not overread that. Android is not the Linux DRM subsystem. A phone platform is not a GPU kernel driver. But the broader lesson is hard to ignore: when memory-safety bugs dominate the risk profile, changing the language of new code can change the shape of future vulnerability data. GPU drivers are exactly the kind of high-risk subsystem where that bet deserves serious attention. Why ARM64 Makes the Story More Important ARM64 is no longer just “the phone architecture.” It is in laptops, cloud servers, edge systems, automotive platforms, developer boards, AI devices, and embedded systems. On many ARM64 systems, the GPU is not a discrete PCIe card sitting at a safe distance. It is part of a tightly integrated SoC, sharing memory, power constraints, thermal limits, and firmware relationships with the rest of the system. That changes the stakes. A GPU driver bug on an ARM64 SoC can affect: - System memory safety - Display stability - Battery life - Thermal behavior - Compositor responsiveness - Application latency - AI and graphics workloads sharing the same memory fabric Rust support for AArch64 entered the Linux kernel development story as part of the broader Rust-for-Linux effort, and Linux kernel documentation now includes Rust materials for kernel developers working on Rust abstractions and drivers. That makes ARM64 GPU work more than a curiosity. It is a practical proving ground for the next decade of heterogeneous computing. The future machine is not CPU-only. It is CPU plus GPU plus NPU plus DSP plus video accelerator plus firmware-controlled subsystems. The kernel increasingly becomes an orchestration layer for compute fabrics. That means more shared memory, more queues, more firmware protocols, and more places for C lifetime bugs to hide like raccoons in ductwork. The Serious Case Study: Tyr for Arm Mali The strongest ARM64 GPU example today is Tyr, a Rust-based DRM driver for CSF-based Arm Mali GPUs. Tyr is especially interesting because it is not a random greenfield fantasy. It is a Rust port of Panthor, the C driver for the same class of hardware. Tyr is being developed as a joint effort involving Collabora, Arm, and Google engineers, and it aims to implement the same userspace API as Panthor for compatibility, so it can eventually be used as a drop-in replacement by PanVK, the Vulkan driver. That one design choice is the difference between serious engineering and conference glitter. Tyr is not trying to rewrite the entire graphics stack at once. It is trying to preserve the userspace contract while changing the kernel implementation language. That is exactly how infrastructure migration should be done. Change one major variable. Measure the result. Then decide. The target is also meaningful. CSF-based Arm Mali GPUs use a command-stream frontend where the driver must coordinate with firmware and hardware scheduling mechanisms. That naturally creates state-machine-heavy code, shared buffers, queues, and lifetime-sensitive resource handling. In other words: the kind of code where Rust’s ownership model is not academic. It is directly relevant. The Other Serious Case Study: Nova for NVIDIA GSP GPUs The second important example is Nova, a Rust-based driver for NVIDIA GPUs that use the GPU System Processor, or GSP. Nova is intended to become the successor to Nouveau for GSP-based NVIDIA GPUs in Linux and targets NVIDIA GPUs beginning with the GeForce RTX 20-series Turing family and newer. Nova is not primarily an ARM64 story, but it matters because it shows Rust entering serious DRM and GPU-driver territory, not just small demo modules. Together, Tyr and Nova point toward the same pattern: Rust is being explored where GPU drivers interact with firmware protocols, memory objects, queues, and kernel graphics APIs. This is the important architectural shift. As GPU firmware takes on more low-level responsibilities, host drivers often become protocol coordinators. They manage firmware boot, message queues, device objects, error states, recovery paths, memory handles, and userspace interfaces. That is a very Rust-shaped problem. Not because Rust is trendy. Trendy is how JavaScript frameworks reproduce. Rust is relevant because protocol state, resource ownership, and invalid transitions can often be modeled explicitly in the type system. The Core Engineering Idea: Make Illegal States Hard to Represent Here is the difference between a shallow Rust port and a serious one. A shallow port translates C into Rust line by line and celebrates because the file extension changed. A serious port rethinks dangerous state transitions. GPU drivers are full of implicit states: Buffer: Allocated -> Mapped -> Submitted -> Retired -> Freed Queue: Created -> Active -> Hung -> Recovering -> Destroyed Firmware: Absent -> Loaded -> Booting -> Running -> Failed Device: Probed -> Initialized -> Suspended -> Resuming -> Resetting -> Removed In C, these states are often spread across flags, pointers, locks, comments, and prayers. In Rust, they can be modeled more directly: enum BufferState { Allocated, Mapped, Submitted, Retired, } struct GpuBuffer<S> { handle: BufferHandle, size: usize, state: S, } struct Allocated; struct Mapped; struct Submitted; struct Retired; impl GpuBuffer<Allocated> { fn map(self) -> Result<GpuBuffer<Mapped>, DriverError> { // Map buffer into GPU-visible address space. Ok(GpuBuffer { handle: self.handle, size: self.size, state: Mapped, }) } } impl GpuBuffer<Mapped> { fn submit(self, queue: &mut CommandQueue) -> Result<GpuBuffer<Submitted>, DriverError> { queue.push(self.handle)?; Ok(GpuBuffer { handle: self.handle, size: self.size, state: Submitted, }) } } This is simplified, but the principle is powerful: make the dangerous lifecycle visible in the type system. In C, the rule might live in a comment: /* Do not free this buffer after submission until the fence signals. */ That comment is useful until someone edits a cleanup path six months later and accidentally turns it into historical fiction. Rust lets engineers encode more of that rule into APIs. It does not remove the need for review. It makes review more focused. Unsafe Rust Is Not a Loophole. It Is the Blast Radius. Kernel Rust still needs unsafe. Anyone claiming otherwise should be escorted away from the whiteboard. Drivers touch hardware. They read and write MMIO registers. They interact with C APIs. They manage DMA. They cross boundaries where the compiler cannot verify everything. The right goal is not “no unsafe code.” The right goal is small, explicit, audited unsafe code. Example: struct RegisterBlock { base: *mut u32, } impl RegisterBlock { unsafe fn read_raw(&self, offset: usize) -> u32 { core::ptr::read_volatile(self.base.add(offset)) } fn read_status(&self) -> DeviceStatus { let raw = unsafe { self.read_raw(STATUS_REGISTER_OFFSET) }; DeviceStatus::from_bits(raw) } } The outer driver should not scatter volatile pointer arithmetic everywhere. It should interact with typed operations such as: read_status() submit_queue() reset_engine() map_buffer() signal_fence() This is the real win: concentrate unsafety behind abstractions whose invariants can be documented, reviewed, and tested. Diffuse unsafety is archaeology. Concentrated unsafety is engineering. Research around Rust safety continues to focus on the fact that unsafe Rust and linked unsafe libraries can still compromise memory safety if not isolated or analyzed properly. That is directly relevant to kernel work. Rust is not a force field. It is a tool for shrinking the zone where humans must be perfect. Humans are bad at being perfect. That is why we invented compilers. What a Real Porting Plan Looks Like A credible GPU subsystem port should not start with “rewrite the driver.” That sentence is how you summon budget demons. A better plan looks like this: Phase 1: Choose a narrow subsystem Start where Rust gives clear leverage: - Buffer lifetime tracking - Command submission validation - Firmware message queues - Fence ownership - Reset and recovery state machines Do not begin with the entire DRM subsystem. That is not bravery. That is poor impulse control. Phase 2: Preserve the userspace API Tyr’s compatibility goal with Panthor’s userspace API is exactly the right instinct. If the userspace API remains stable, the migration can focus on kernel-internal safety and maintainability rather than forcing the whole graphics stack to change at once. Stable outside. Safer inside. That is the migration pattern. Phase 3: Wrap unsafe boundaries Every unsafe block should answer three questions: 1. What invariant must be true before this code runs? 2. Who guarantees that invariant? 3. How do we test that the invariant remains true? If the answer is “trust me,” the code is not ready. Trust is not a test strategy. Phase 4: Measure performance with real workloads A Rust GPU driver that is safer but introduces unacceptable frame-time spikes will not survive. Kernel developers care about safety, but they also care about latency, throughput, and not humiliating themselves in front of perf. A real benchmark plan should include: - Frame-time mean, p95, and p99 - Command submission latency - CPU cycles during graphics workloads - Context switches - Interrupt rate - GPU reset recovery time - Firmware boot time - Memory bandwidth - Power draw under sustained load - Thermal throttling behavior A basic harness might look like this: #!/usr/bin/env bash set -euo pipefail DRIVER="${1:?usage: ./bench.sh <driver-name>}" FRAMES="${2:-600}" OUT="results-${DRIVER}-$(date +%Y%m%d-%H%M%S)" mkdir -p "${OUT}" echo "Driver: ${DRIVER}" | tee "${OUT}/metadata.txt" uname -a | tee -a "${OUT}/metadata.txt" lscpu | tee "${OUT}/cpu.txt" sudo dmesg -C perf stat -d \ -o "${OUT}/perf.txt" \ -- ./gpu_workload_runner \ --driver "${DRIVER}" \ --frames "${FRAMES}" \ --json "${OUT}/frames.json" dmesg > "${OUT}/dmesg.log" echo "Benchmark complete: ${OUT}" That is not enough for a final paper-quality result, but it is the start of an honest engineering conversation. One run is a screenshot. Ten controlled runs are evidence. A bar chart with no methodology is decorative nonsense. The ARM64 Benchmark Matrix For an ARM64-focused evaluation, use a matrix like this: Hardware: - Rockchip RK3588 board, such as Rock 5B - Stable power supply - Active cooling - Fixed CPU governor - Fixed GPU governor, where available Software: - Same kernel baseline for C and Rust driver tests - Same Mesa version - Same compositor setting - Same Vulkan or OpenGL workload - Same thermal constraints Workloads: - Synthetic command submission stress test - Vulkan sample workload - Mesa or IGT graphics tests - Real application trace - Forced GPU reset and recovery test Metrics: - Mean frame time - p95 frame time - p99 frame time - CPU cycles - Context switches - Interrupts - GPU resets - Kernel warnings - Power draw This is where many articles fail. They show a chart but hide the setup. That is amateur hour. For DZone, the article can be powerful even without original benchmark results if it clearly presents the benchmark plan. But to become exceptional, it needs real numbers from a reproducible setup. No fake numbers. No “up to 5x faster” nonsense unless measured. The internet already has enough performance astrology. What Rust Will Not Fix Rust can reduce some classes of bugs, but it does not solve the hard parts of driver development by itself. It cannot compensate for poor hardware documentation, opaque firmware behavior, bad scheduling decisions, or flawed abstractions, and it does not make DRM any less complex. Low-level drivers will still require unsafe code, and deadlocks, design mistakes, and logic errors remain very much on the table. The Linux kernel documentation itself remains cautious: Rust support is still aimed at developers and maintainers working on abstractions, drivers, infrastructure, and tools, and it notes that Rust support is still under development, especially for certain configurations. That caution should be repeated, not buried. The correct argument is not: Rust makes kernel drivers safe. The correct argument is: Rust can reduce specific classes of memory and lifetime bugs in new kernel driver code, especially when unsafe hardware access is isolated behind reviewed abstractions. That is less flashy. It is also true. True wins. Why This Matters Beyond GPUs GPU drivers are a proxy for where kernel-level computing is headed. Modern systems are becoming accelerator orchestras. The CPU no longer owns the whole performance story. Work moves across GPUs, NPUs, DSPs, video encoders, SmartNICs, security processors, and firmware-managed islands. That means kernel software must manage: - Shared memory across devices - Complex queue lifetimes - Cross-device synchronization - Firmware protocols - Userspace-visible handles - Device reset semantics - Security boundaries around accelerators If Rust helps in GPU drivers, it can help in other accelerator drivers too. That includes: - AI accelerator drivers - Media encode and decode engines - Camera pipelines - SmartNIC offload paths - Storage acceleration - Embedded display controllers The real innovation is not “Rust replaces C.” That is a bumper sticker. The real innovation is selective memory-safe kernel development for high-risk hardware boundaries. That is a more mature thesis, and it is the one engineering leaders should care about. The Takeaway For application developers, this matters because kernel reliability eventually becomes application reliability. A browser tab, a video call, a game engine, a dashboard, a vision model, or an edge AI pipeline can all be ruined by a GPU stack that mishandles memory or fails recovery. For systems developers, this matters because Rust offers a practical way to encode ownership and state transitions that C leaves to discipline and code review. For engineering leaders, this matters because memory-safety work is not just a security initiative. It is a maintenance initiative. Safer new code can reduce future review burden, rollback risk, and vulnerability exposure, as Google’s Android reporting suggests. For kernel maintainers, this matters because the only acceptable Rust adoption path is incremental, measurable, and compatible with existing kernel development culture. The path is not revolution. It is disciplined infiltration. A Practical Checklist for Teams Before starting a Rust GPU driver effort, answer these questions: 1. What exact bug class are we trying to reduce? 2. Which subsystem has the worst lifetime complexity? 3. Can we preserve the userspace API? 4. Where must unsafe code exist? 5. Can unsafe code be isolated behind reviewed abstractions? 6. What real workload will prove performance? 7. What metric would make us abandon or redesign the port? 8. Who will maintain the Rust abstractions after the prototype hype fades? That last question is brutal and necessary. A prototype is easy. Maintenance is the boss fight. Conclusion: Rust Must Earn Its Place Where the Bugs Are Worst Rust in the Linux kernel should not be judged by toy modules. It should be judged where kernel bugs are expensive: drivers, firmware interfaces, DMA, shared memory, synchronization, and recovery paths. That is why GPU drivers on ARM64 are such an important proving ground. They combine modern hardware complexity with exactly the memory and lifecycle hazards Rust was designed to reduce. Tyr shows the most direct ARM Mali path: a Rust DRM driver for CSF-based Arm Mali GPUs, designed as a port of the C Panthor driver while aiming for userspace API compatibility. Nova shows that Rust GPU driver work is also moving into NVIDIA GSP territory, with ambitions to succeed Nouveau for supported modern NVIDIA GPUs. The lesson is not that Rust is perfect. The lesson is that the next generation of kernel-level computing is becoming too heterogeneous, too concurrent, and too security-sensitive to keep writing every new dangerous subsystem in C by default. Rust will not replace engineering discipline. It will punish the lack of it earlier. And in kernel development, earlier is everything. The Buffer That Came Back From the Dead The board had been running for nine hours. Same workload. Same scene. Same cursed GPU path that used to crash whenever the moon was wrong and the scheduler sneezed. In the old driver, the bug never arrived on command. It preferred drama. Sometimes frame 417. Sometimes frame 12,004. Sometimes only when the engineer walked away, because bugs respect neither science nor lunch. The Rust port failed too, at first. But it failed differently. Not with a corrupted pointer three layers below the crime scene. Not with a dead queue holding a ghost reference to a buffer that should have been buried. It failed at compile time, loudly, rudely, and before anyone had to read 900 lines of logs with the dead-eyed stare of a person reconsidering their career. The compiler pointed at the illegal transition. The engineer stared at it. The GPU kept rendering. For once, the monster had left footprints.

By Aakash Chaudhary

The Startup Time Trick Hiding Inside Your Docker Build

Every Java developer who runs services on Kubernetes has watched this scene play out. Traffic spikes, the autoscaler adds a pod, and then everyone waits. The container is running in two seconds. The application is not ready for another twelve seconds. During those ten seconds, your existing pods absorb the extra load, latency climbs, and if things are bad enough, the autoscaler panics and adds even more pods that are also not ready. I spent years treating Spring Boot startup time as a fact of life, the way you treat weather. Then I found out the JVM has had a fix for a big chunk of it since Java 12; it works beautifully inside Docker, and almost nobody bakes it into their images. It is called Class Data Sharing, CDS for short, and this article shows you how to make your Docker build do the work Where Those Twelve Seconds Actually Go When a Spring Boot application starts, the JVM is not mostly running your code. It is loading classes. A plain REST service with Spring Web, Spring Data, and a driver or two loads somewhere between ten and twenty thousand classes before it serves its first request. For every single one of those classes, the JVM does the same ritual. Find the class file inside a jar, read the bytes, parse them, verify the bytecode is legal, and build the internal metadata structures it needs at runtime. Thousands of times. Every startup. In every pod. Here is the part that should bother you. Your container image never changes after you build it. The same jar, the same classes, the same parsing work, repeated identically in every pod that ever starts from that image. The JVM is solving the same puzzle again and again and throwing away the answer each time. CDS is the JVM saying: let me solve it once, write the answer to a file, and just memory map that file next time. What a CDS Archive Is A CDS archive is a file, usually ending in .jsa, that contains classes already parsed and verified, stored in the exact internal format the JVM uses in memory. On startup, the JVM maps this file straight into memory. No finding, no parsing, no verifying. The work was done ahead of time. You have been using CDS without knowing it. Modern JDKs ship with a default archive covering the core JDK classes, which is why java -version is fast. The step almost everyone skips is creating an archive for your application classes, all fifteen thousand of them. That is where the real win lives. The mechanism has one rule that matters for us. The archive must be created with the same JVM and the same classpath that will use it. That rule sounds annoying until you realize a Docker image is the one place in your entire infrastructure where JVM and classpath are frozen forever. Docker is not just compatible with CDS. It is the perfect home for it. The Training Run Creating the archive takes two steps. First you do a training run, where the JVM starts your application, watches which classes get loaded, and writes the list down. Then you exit, and the JVM turns that list into the archive. Since Java 13, this is pleasantly simple: Shell java -XX:ArchiveClassesAtExit=app.jsa -jar app.jar Run the app, let it come up, stop it, and app.jsa appears. From then on you start the app like this: Shell java -XX:SharedArchiveFile=app.jsa -jar app.jar There is an obvious question here. The training run wants to actually start the application, and inside docker build there is no database, no message broker, nothing to connect to. A Spring Boot app that cannot reach Postgres will crash during training. Spring Boot 3.3 solved this neatly. Setting one property makes the application run through its entire startup sequence, create all bean definitions, and then exit just before touching the outside world: Shell java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app.jsa -jar app.jar The application loads nearly everything it will ever load, writes the archive, and exits cleanly with no infrastructure needed. This is exactly what a Docker build stage can do. The Dockerfile Here is the complete picture: a multi-stage build where the image trains itself: Shell FROM eclipse-temurin:21-jdk-alpine AS build WORKDIR /build COPY . . RUN ./mvnw -B package -DskipTests # Explode the jar so the classpath is stable RUN java -Djarmode=tools -jar target/app.jar extract --destination /app FROM eclipse-temurin:21-jre-alpine AS runtime WORKDIR /app COPY --from=build /app /app # Training run: start the context, record classes, exit RUN java -Dspring.context.exit=onRefresh \ -XX:ArchiveClassesAtExit=/app/app.jsa \ -jar /app/app.jar ENV JAVA_TOOL_OPTIONS="-XX:SharedArchiveFile=/app/app.jsa" ENTRYPOINT ["java", "-jar", "/app/app.jar"] Two details in there deserve a closer look. The extract step unpacks the fat jar into a folder with the dependencies laid out as plain files. CDS is picky about the classpath being identical between training and real runs, and a fat jar with nested jars inside it makes that fragile. The exploded layout keeps the classpath boring and stable, which is exactly what CDS wants. On Spring Boot 3.2 and older, the same idea works through the layertools jarmode instead. The training run happens as a RUN instruction, which means it executes once at build time on your CI server. Every container that ever starts from this image inherits the archive for free. You did the class loading homework once, in the build, and ten thousand pod starts copy the answer. What You Get Numbers vary with how heavy your application is, but the pattern is consistent. A typical Spring Boot 3 web service that started in 10 to 12 seconds lands somewhere between 5 and 7. The JVM portion of startup shrinks dramatically, and as a bonus, the archive is memory-mapped and shared, so if you run several JVMs on one node, they share those pages and total memory drops too. You can verify the archive is actually being used, which I recommend, because CDS fails silently by design. If something mismatches, it just quietly falls back to normal class loading: Shell docker run --rm my-service -Xlog:class+load=info | head -5 Classes loaded from the archive say source: shared objects file. If you see jar paths instead, the archive is being ignored, and the log will usually tell you why. The usual culprit is a classpath that differs from training, even by one entry. One honest caveat. The training run exercises startup, not your traffic. Classes that only load when a specific endpoint gets hit for the first time are not in the archive, so those first requests still do normal loading. The archive covers the framework and wiring, which is most of the cost, but it is not a magic warm-up for everything. Why This Beats the Alternatives You Have Heard Of Whenever container startup time comes up, someone mentions GraalVM native images, and native images are impressive. Millisecond startup is real. But they come with a price list: long build times, a closed-world assumption that fights with reflection, some libraries that simply do not work, and a different runtime profile you have to learn to debug. CDS costs you five lines of Dockerfile. Your application is still a completely normal JVM application. Same debugging, same profilers, same libraries, same behavior, just faster out of the gate. For most teams, that trade-off is not even close. It also stacks with what is coming. Project Leyden's AOT cache in Java 24 and beyond is essentially this same idea grown up, caching not just parsed classes but resolved linkage and compiled code. The Dockerfile pattern you build today, a training run at build time producing a cache file shipped in the image, is exactly the shape Leyden uses. Learning it now means the future is a flag change. The Takeaway Your Docker image is immutable. Your JVM does expensive, perfectly repeatable work on every startup. Those two facts fit together like puzzle pieces, and a training run inside docker build is where they connect. One extra build step, and every pod your autoscaler ever creates comes up in half the time. The next time you watch a rollout crawl because pods take forever to go ready, remember that the answer was hiding inside the build all along.

By Garima Agarwal

Making Running Optional: Scaling AI Agents on Kubernetes With Agent Substrate

What if you could multiplex roughly 250 stateful agent sessions across eight Kubernetes worker Pods, then reactivate any one without losing its in-memory or filesystem state? The repository's demo reports 30x+ actor-to-worker oversubscription for that sample workload, with sub-second activation. It is a demonstration, not a production capacity guarantee. Agent Substrate is interesting not because it makes Kubernetes faster, but because it challenges a common deployment pattern around Kubernetes: coupling a workload's logical lifecycle to the compute allocated to run it. For AI agents that spend most of their time idle while retaining valuable state, separating those lifecycles could become an important building block for operating agents at significantly higher density. Kubernetes gave us an exceptionally durable abstraction for running workloads: the Pod. But emerging agent workloads expose places where that abstraction may become inefficient. They can be stateful, sandboxed, and overwhelmingly idle. A one-agent-per-Pod deployment model, while simple, couples each session's logical lifecycle to a Pod's runtime lifecycle. When sessions spend much of their time idle, that coupling can leave compute capacity allocated to workloads that are not actively executing. This is not a claim that Kubernetes is obsolete. It is an exploration of where Kubernetes remains the right substrate--provisioning capacity, managing worker Pods, and enforcing infrastructure policy--and where an agent-specific control plane may need a faster path for high-frequency lifecycle operations. The Thesis: Make Running Optional The central idea is surprisingly simple: an agent does not need to occupy compute merely because it exists. Agent Substrate calls an instance of a managed workload an actor. The deliberately broader term matters: an actor does not have to be an AI agent; it can be any OCI workload that benefits from being bursty, checkpointable, and independently suspendable. The system provides an agent-oriented workload runtime and control plane; it is not an agent framework or SDK. An actor can be suspended into a snapshot containing process memory, filesystem state, or both. The worker Pod is then freed. When another request arrives, the system restores the actor to a ready worker and routes the request to it. In this model, the actor becomes the logical workload, while the Pod becomes temporary compute capacity. That gives the architecture three defining properties: Warm capacity replaces per-session capacity. A smaller pool of ready workers serves a much larger population of actors over time.State survives worker reassignment. The next activation need not use the worker that ran the actor previously.Requests can initiate activation. The router can hold a request while the control plane brings a suspended actor back. The project's architecture document defines north-star targets including 100 ms p95 activation, one billion active and idle actors per cluster, and 1,000 wakeup events per second. These are architectural targets, not production benchmarks or guarantees. That distinction is important because the repository itself is candid that large parts of the architecture are still evolving. Architecture at a Glance The control flow becomes easier to follow once the logical workload is separated from the physical capacity: Why the Pod Becomes an Awkward Unit for Agents A conventional one-Pod-per-session deployment model can become inefficient when sessions spend most of their lifetime idle but retain valuable state. Agent-like workloads have a different shape: They wait much more than they compute.They may execute untrusted code, so multi-tenancy often means one sandbox per session.They keep useful state in memory and local filesystem changes.Their active periods can be short enough that creating and initializing a dedicated Pod for each session becomes noticeable user-facing latency. In a one-Pod-per-session design, the logical unit a user cares about — a coding session, sandboxed tool, or stateful agent — does not align cleanly with the physical unit Kubernetes schedules: a Pod. One straightforward approach is to keep each session's Pod alive. Agent Substrate asks whether that physical allocation can be temporary instead. Its answer is a pool of pre-provisioned worker Pods plus a separate actor record that tracks identity, lifecycle state, placement, and snapshots. This is an important architectural departure from Kubernetes' conventional control-plane model. Kubernetes intentionally optimizes for declarative desired state and asynchronous reconciliation. Agent Substrate moves high-churn actor state out of the Kubernetes API machinery so that wakeup, placement, and snapshot transitions can occur without making each actor a Kubernetes object. The point is not that Valkey or Redis is universally "faster than etcd." The interesting boundary is low-frequency desired state versus high-frequency runtime state: the two have different frequency and latency profiles, so the system gives them different control paths. Three Planes of State The resource model separates declarative configuration from dynamic runtime records. Operationally, snapshot contents form a useful third plane: STATE TYPEWHERE IT LIVESWHYActorTemplate, WorkerPool, and SandboxConfigKubernetes CRDsLow-frequency infrastructure configuration benefits from Kubernetes RBAC, auditability, and reconciliation.Actors, workers, assignments, lifecycle state, and snapshot referencesControl-plane store (Redis/Valkey by default; experimental PostgreSQL support is also available)These records change on lifecycle transitions and need low-latency reads and writes.Snapshot contentsNode-local storage for Pause; object storage for snapshots committed during SuspendSnapshot scopes trade off locality, durability, and transfer cost. An ActorTemplate defines an actor class: its container image, snapshot behavior, and compatible worker selection. A WorkerPool declares warm Pods. An Actor is a specific instance that moves between workers through its lifetime. Here is a trimmed ActorTemplate from the repository's multi-template demo: YAML apiVersion: ate.dev/v1alpha1 kind: ActorTemplate metadata: name: counter spec: containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter workerSelector: matchLabels: workload: multi-template-shared The important omission is a dedicated Pod. The template describes the workload and selects compatible reusable capacity; a separate WorkerPool provides the warm Pods, while the actor's identity and lifecycle remain independent of whichever worker hosts it. The deeper architectural pattern is a separation of three related lifecycles: infrastructure, workload, and execution. Kubernetes manages infrastructure capacity; the actor control plane manages logical workload identity, placement, and lifecycle; snapshot and sandbox machinery preserve and reconstitute execution state. Once those lifecycles are separated, a worker Pod becomes a reusable execution slot rather than the identity of the workload itself. An actor is addressed by (atespace, name), not by name alone. That is more than a naming detail: the glossary defines an atespace as a logical actor isolation boundary, not a replacement for Kubernetes namespaces or a sandbox security boundary. The same actor name can exist in different atespaces. The atespace also appears in the actor's routable DNS name: Plain Text <actor-name>.<atespace>.actors.resources.substrate.ate.dev This is the first place the project begins to look less like a set of Kubernetes objects and more like a runtime: stable logical identity remains while the physical worker assignment changes. The Request Path: Routing Becomes Placement The most consequential design choice is that ingress is part of activation. The networking architecture provides the actor DNS model and an Envoy-based router. The router's ext_proc handler reads the actor reference from the request authority, calls the control plane to ensure that actor is running, and then selects the assigned worker as the upstream. Plain Text Error: Parse error on line 22: ...atunnel Ateom->>Actor: Forward over ----------------------^ Expecting '+', '-', '()', 'ACTOR', got 'participant_actor' The ingress detail matters. The router does not forward directly to the actor's application endpoint. It opens an mTLS connection to the worker's atunnel listener. atunnel validates the router and forwards only to the actor currently assigned to that worker. That makes routing part of the security boundary, not merely service discovery. There is also a practical admission-control insight here. A saturated worker pool should not turn a burst of requests into an unbounded queue. The router can park a bounded number of requests while it retries transient capacity and control-plane conditions; once the parking limit is reached, it sheds new work. Activation latency is therefore not just a restore-time problem. It is also a backpressure problem. What Happens During Suspend and Resume The control plane coordinates a distributed workflow rather than pretending this is a single atomic operation. For a resume, it locks the actor, reads its state and template, selects an eligible idle worker, asks the node-level supervisor to restore a snapshot or cold boot, and marks the actor running only after the worker is ready. For a suspend, it checkpoints state, persists the requested snapshot scope, clears the worker assignment, and returns the worker to the pool. The details reveal deliberate distributed-systems trade-offs. In the default Redis backend, actor and worker records are separate keys that may occupy different cluster slots, so they cannot be updated in one cross-slot action. The implementation uses per-record version checks, actor locking, ordering, retries, and idempotent workflow steps to coordinate these transitions. The architectural implication is that lifecycle operations are treated as recoverable workflows rather than atomic infrastructure mutations. A repeated lifecycle call can discover completed steps and move forward instead of blindly redoing them. The Worker Is a Reusable Sandbox, Not the Actor Below the control plane, atelet runs as a DaemonSet and manages the node-side work: image preparation, OCI bundle assembly, snapshot transfer, and communication with the worker. ateom runs inside the worker Pod and drives the sandbox runtime. The repository currently defines gVisor and microVM sandbox classes. In the gVisor path, ateom drives runsc checkpoint and restore. Depending on the configured scope, snapshots can preserve process and filesystem state, allowing an actor to resume later on another worker. This is why the demo can show an in-memory counter continuing after a suspend/resume cycle: the application was restored, rather than restarted from scratch. The security model should be described with care. Sandboxing and mTLS are real implementation elements, and the project has a detailed threat model. But that document explicitly says security hardening remains early. A fair reading is that Agent Substrate is making the right boundaries visible--sandbox, worker reuse, actor identity, snapshot access, and router-to-worker authentication--rather than claiming those boundaries are already production complete. What the Demos Prove--and What They Do Not The most accessible proof is the counter demo. A tiny HTTP service increments an in-memory counter. Create an actor in an atespace, send requests through atenet-router, suspend it, and resume it. The counter continues. The demo makes the abstract claim concrete: memory and filesystem state can outlive a worker assignment. The README's published density demonstration goes further: about 250 stateful actors multiplexed across eight physical worker Pods. The repository also includes examples for Claude Code multiplexing, request parking, autoscaled worker pools, and different templates sharing a worker pool. These examples validate the model and its developer experience. They do not prove the project's one-billion-actor target, production reliability, or a universal cost model. Treating that distinction honestly makes the architecture more interesting, not less: the open questions are precisely where the difficult engineering begins. From Traffic Locality to Compute Locality My previous DZone article, "Zone-Aware Routing in Kubernetes", examined a related infrastructure question: how should a platform place traffic so requests stay local when that improves latency, resilience, or cost? That work led me to a broader question: if locality matters for packets, what happens when locality also matters for stateful compute? Zone-aware routing asks where traffic should go. Agent Substrate raises a harder question: where should the compute state itself live when a workload can disappear from one worker and reappear on another? This turns locality from a networking concern into a workload-lifecycle concern. That change has consequences: Scheduling cannot be evaluated only by where free CPU exists; snapshot location and resume cost matter too.Routing cannot be evaluated only by endpoint availability; it can trigger a state transition.Security cannot stop at the Pod boundary; worker reuse and snapshot access become first-class concerns.Autoscaling cannot only count replica demand; it must account for how long actors remain active, parked, or suspended.Storage cannot be treated as an afterthought; snapshot placement, transfer time, durability, and locality become part of the activation path. This suggests a broader infrastructure question for agent workloads. The challenge is not simply running more containers; it is hosting large populations of mostly-idle, stateful, potentially untrusted processes without allocating dedicated compute to each one. Where the Hard Work Remains The project is unusually direct about its unfinished work: control-plane performance and reliability, worker autoscaling, identity and policy, actor network isolation, storage design, observability, and support for different sandbox runtimes all remain active areas of development. Those concerns are not peripheral; they determine whether the architecture can operate reliably at the scale it targets. A system that makes activation fast must still decide how to shard state, restore safely, apply policy before execution, isolate one actor from the state left by another, and reason about locality without turning every wakeup into a storage bottleneck. Four questions are especially important: Snapshot locality. If an actor's state is remote, resume latency becomes partly a storage and network-transfer problem.Snapshot correctness. Checkpointing a live process is not equivalent to serializing application state. Open connections, timers, external leases, credentials, and dependencies can make a restored process semantically different from a freshly initialized one.Activation bursts. Multiplexing improves average utilization, but a correlated wake-up event can turn many inexpensive idle actors into a sudden demand spike. The system therefore needs admission control and worker autoscaling that respond to activation pressure, not only steady-state utilization.Fairness. A small number of highly active actors can monopolize workers unless scheduling and admission control account for competing demand. Agent Substrate is therefore more compelling as an emerging architectural pattern than as a product claim. Its value is in making these trade-offs explicit and providing a runnable implementation that exposes where the abstractions are strong and where they remain unfinished. Conclusion The Pod is unlikely to disappear. But it may stop being the only unit we think about when we build infrastructure for agents. Kubernetes remains a powerful system for provisioning and operating compute. Agent Substrate is exploring what happens when the logical lifecycle of an agent is separated from the lifecycle of the Pod that temporarily runs it. If agents become ubiquitous--long-lived, intermittently active, stateful, and capable of executing untrusted code--the infrastructure challenge will not simply be running more Pods. It will be deciding where an agent should exist when it is inactive, how quickly it can become active, and how efficiently thousands or millions of them can share the same underlying compute. That is the problem Agent Substrate is attempting to solve. At a billion actors, the central question is no longer how to run more agents. It is how to make running optional. Further Reading Agent Substrate repositoryArchitectureThreat modelRequest parkingCounter demo Agent Substrate is Apache-2.0 licensed, explicitly not an officially supported Google product, and in active early development. The architectural analysis and opinions in this article are my own.

By Mayowa Fajobi

Inside terraform-provider-archive: A Memory Pattern From 2016 That Scales With Your Lambdas

A CI Runner That Shouldn't Have Died If you deploy AWS Lambdas through Terraform, you almost certainly use archive_file. With enough lambdas, a single terraform apply can kill the CI runner with OOM. The trickiest part is that you will not see any errors in Terraform output and have no clue what just happened. I noticed this when my lambdas started failing — every first terraform apply after a routine change. SIGKILL from the kernel OOM killer and nothing in Terraform logs. The strange part is that reapply sometimes worked — not always on the first try, but eventually it went through. I've named the ticket "Flaky CI," and two weeks of investigation was focused on the CI itself: runner memory, parallel jobs, Docker leaks. terraform apply was the last suspect — from my perspective, there was no way or reason for it to consume so much memory. If you've never wondered how Terraform providers work, it's actually pretty simple. Most of them are just API wrappers. They send HTTP requests, parse responses, and update state. archive_file is one of the exceptions — it works with real files on disk. This means that its memory usage is actually determined not by the number of defined resources, but by the total size of the data it should process. That's why the pattern went unnoticed for years — without knowing about the provider's insides, the issue looks like some CI flakiness. When I finally reached the source code, the answer was found in a few lines in zip_archiver.go file. What archive_file Actually Does archive_file data source creates a zip or tar archive from a directory or file. This is a standard pattern for lambdas: you point source_dir at the function code and pass the resulting archive to aws_lambda_function. YAML data "archive_file" "lambda" { type = "zip" source_dir = "${path.module}/src" output_path = "${path.module}/lambda.zip" } Nothing suspicious at first glance, but behind these lines is a call chain, which is worth a deeper look. When Terraform processes this data source, the provider calls archiveFile — it creates a ZipArchiver and iterates over files in source_dir. For each file, it calls the ArchiveFile method, which does the following: Go content, err := os.ReadFile(fname) // ... f, err := a.writer.Create(name) // ... _, err = f.Write(content) os.ReadFile reads the entire file into a []byte — one contiguous buffer in memory. Then that buffer is passed to the zip writer via Write. After the write, the buffer becomes garbage. This was a design choice from 2016, and at the time, it was reasonable. Terraform configurations archived small files — configs, scripts, and templates. A typical source_dir weighed something like kilobytes, so there was nothing to optimize at this point. That's why the simplest way to read a file was chosen — os.ReadFile. The code looks like a textbook example. But the context changed. Lambda zips today are 50-250 MB uncompressed. ML models, large dependencies (numpy, pandas, puppeteer), bundled assets. And teams deploy not one lambda but five, ten, or twenty through a single Terraform workspace. The code from 2016 didn't change. The scale of the data did. Why Can't the Garbage Collector Help The natural and reasonable question: doesn't Go's garbage collector reclaim memory between files? GC runs indeed — it just has nothing to reclaim. All ten archive_file data sources are independent — they have different source directories and no shared references (if you do not specify them directly). Terraform's graph walker places them at the same level and evaluates them concurrently. This is usually a good thing timewise, but not in this case, as all 10 buffers are alive at the same time. Each goroutine holds its 50 MB until zip write completes. The garbage collector scans the heap and identifies every buffer as still in use, so it reclaims nothing. Meanwhile, peak heap hits 10 x 50 MB = 500 MB (measured: 508 MB). If the model is right, peak memory should scale linearly with parallelism. Your CI runner's memory limit doesn't. Measuring the Pattern I've chosen two ways of measurement: a standard Go benchmark for precision (isolating the archiver) and a Terraform integration test for realism (a real provider during terraform plan). The headline: for 10x50 MB concurrent archives, peak heap drops from 508 MB to 8 MB -- a 98% reduction. Full results, heap growth during archiving, buffered versus streaming: 1x50MB: 50.8 → 0.8 MB (98% reduction)10x10MB: 108 → 8.1 MB (92% reduction)10x50MB: 508 → 8.1 MB (98% reduction) Real Terraform under terraform plan with parallelism matrix, peak RSS in MB: Implp=1p=2p=5p=10Buffered1232765791034Streaming173275384533 Buffered RSS scales linearly with parallelism. Streaming flattens the curve. One anomaly you could've noticed: at p=1, streaming shows a higher RSS than buffered. I'm fairly sure it's just noise. Single-archive runs finish fast, and sampling RSS every 100ms is too coarse to catch what's really happening in that window. The number that matters is p>=2, and that's where the pattern holds. On speed: Go benchmark wall time stays within about 3% across every scenario. So the streaming fix isn't quietly buying memory savings with a performance hit. You get the memory back for free. All measurements are reproducible: https://github.com/olegmmv/terraform-archive-memory-research. Putting these measurements together gives a three-stage picture of the memory cost: StagePeak Heap (10x50MB, p=10)StatusBaseline (current provider)1034 MBMeasuredWith input-side streaming533 MBMeasuredWith full pipeline streaming~320 KBArithmetic projection The third row isn't measured, but is arithmetic. I'll describe later why, but for now, just keep in mind that it shows what we'd see if a second os.ReadFile in the output path is also streamed. The Tar Archiver Already Streams The fix isn't speculative; just open a neighboring file in the same provider. In tar_archiver.go, addFile opens the file, defers close, and copies via io.Copy into tarWriter. No buffering — streaming by default. Go file, err := os.Open(filePath) // ... defer file.Close() // ... _, err = io.Copy(a.tarWriter, file) The zip_archiver.go path, though, chose the buffered approach: Go content, err := os.ReadFile(infilename) // ... _, err = f.Write(content) Same codebase and job to be done, but two different choices. archive/zip.Writer.Create returns an io.Writer that streams, with CRC-32 computed during the write via crc32.NewIEEE. There was never a technical barrier. The only thing needed for the fix now is applying the same pattern. The Streaming Fix Here is the diff: replace os.ReadFile with os.Open and Write with io.Copy: diff - content, err := os.ReadFile(infilename) + file, err := os.Open(infilename) if err != nil { return err } + defer file.Close() if err := a.open(); err != nil { ... - _, err = f.Write(content) + _, err = io.Copy(f, file) Everything else stays the same; the only thing that's different is the read-write pattern. This is the actual implementation behind the streaming numbers in the previous section. The streaming version does still allocate memory, of course — you can't get to zero. But it's way down: my benchmark put it at around 0.8 MB. This is due to archive/zip internal buffering: the io.Copy buffer, the deflate compressor state, and small zip metadata structures. One caveat worth flagging: this is the input side only. On the output path, the provider uses its own ReadFile function to compute checksums on the completed zip archive. The Second ReadFile: Output Checksums The Go benchmark showed a 98% reduction, but terraform plan with parallelism=10 only drops from 1034 MB to 533 MB -- about 50%. Where's the missing 48%? Once the zip lands on disk, the provider turns around and reads it straight back. That's what genFileChecksums does: it opens the output file and computes four hashes -- md5, sha1, sha256, sha512 -- for Terraform state. And each one of those hashes wants the full file content. So the provider pulls the entire output zip into memory, using the same os.ReadFile we've been dealing with all along. In my benchmark, the output zip comes out roughly the size of the input. The test data is random bytes, and Deflate can't do much with those. Real Lambda packages compress a lot better, but the pattern remains: the provider reads whatever the output size is back into memory. Run ten of these in parallel at 50 MB a pop, and you're already 500 MB deep, purely on checksums. The PR goes after the input side. It removes the os.ReadFile allocation during archive creation, and the effect is big. In straight Go benchmarks, heap usage drops by 98%, from 508 MB to 8 MB. Real Terraform runs are tamer, about half: peak RSS falls from 1034 MB to 533 MB. So where's that remaining 533 MB coming from? It's the second os.ReadFile, the one inside genFileChecksums, still reading the finished zip back into memory so it can hash it for Terraform state. Technically, you can stream the checksums too. hash.Hash already satisfies io.Writer, so nothing stops you from wrapping all four hashes in an io.MultiWriter and feeding them while the zip is being written. One pass, no second read. The catch is that it's a very different patch from the input-side one. genFileChecksums is structured around post-hoc reading. Making it streaming means restructuring how the provider integrates checksum computation with archive creation. That's state-management territory, not plain I/O. If both sides streamed, the only thing left to allocate would be io.Copy's default buffer. Ten goroutines, 32 KB each, and you land at 320 KB total. Throw in a sliver of zip writer state per goroutine, and that's basically it. The theoretical floor. What the PR actually does is the first half: input streaming, leaving that 533 MB residual behind. The output half, streaming through MultiWriter, is written down as future work. So one PR cuts the problem in half. Closing it out takes two. What It Costs in Practice At the Lambda deployment limit of 250 MB, ten concurrent archives push peak heap to roughly 5 GB -- well past most CI runner allocations. There are workarounds, each with a price tag. Dial parallelism down, and you trade throughput for memory. Spin up beefier CI runners, and you trade dollars for memory. Both get you unstuck, but neither addresses the root cause. The PR is up at https://github.com/hashicorp/terraform-provider-archive/pull/501. The fix is under ten lines of Go, so the investigation took much longer than the implementation. Some design choices age well, but some scale with your infrastructure.

By Oleg Mamiev

Deliberate Decoupling: 6 Architectural Patterns From a Regulated WAS-to-AWS Migration

Key Takeaways In regulated industries, cloud migration success is determined less by technology selection and more by how deliberately you decouple risk vectors — compliance risk, organizational hesitation, user adoption gaps, and integration changes — so no single failure can derail the whole program.You can successfully migrate an application to AWS while keeping data on-premises by routing through a REST API abstraction (e.g., IBM’s DB2 REST API layer) paired with dedicated AWS security groups controlling cloud-to-on-prem traffic, allowing the data migration to proceed on its own compliance and trust-building timeline.The most dangerous compliance gap in regulated applications isn’t declared sensitive fields — it’s free-form text fields where users may inadvertently type SSNs, credit cards, or other regulated identifiers; proactive tokenization in the application’s write path closes this gap before any audit finds it.Long-tenured business users carry a decade of UX muscle memory that QA testing cannot replicate; allocating real production validation time (such as a 15-day dark deployment cohort) is essential when migrating systems users have relied on daily for 10+ years.Before starting a regulated cloud migration, ask which risk vector each architectural decision is decoupling and whether your team is aligned on why — this single question reframes "cloud migration" from a technology project into a coordinated risk-management exercise. Introduction Most published writing on legacy-to-cloud migration treats it as a technical exercise: pick the stack, plan the cutover, flip the switch. In regulated industries, that framing fails — and the failure mode isn’t a missed deployment window. It’s a stalled program, a failed compliance audit, or a client who pulls back from the cloud strategy entirely. A cloud migration in healthcare insurance is as much about regulatory risk management, organizational trust-building, and user adoption as it is about microservices and Fargate. Get the technology right and miss the risk choreography, and the project doesn’t ship. I led the first WebSphere-to-AWS migration in the health division of a Fortune 50 insurer — a multi-year program touching PHI data, long-tenured business partners, and downstream services concurrently migrating to the cloud. Over that program, six architectural patterns emerged as decisive. Not for the technology they enabled, but for the risks they made manageable. None are individually novel. What’s distinctive is how they work together — as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. Pattern 1: Strangler Fig With Dark Deployment When migrating critical production systems to the cloud, the temptation is a hard cutover — flip the switch at 2 AM on a Sunday and hope for the best. We chose a different path: a 15-day dark deployment on AWS production, accessible only to a designated cohort of business partners. Three factors drove this decision. 1. First-mover risk in the department. This was the first WAS-to-AWS migration in this Fortune 50 insurer’s health division. There was no internal precedent to draw from — no playbook, no lessons learned from a prior AWS rollout. A "big bang" cutover would have exposed our full user base to whatever unknowns we hadn’t anticipated. Dark deployment let us pioneer the path with limited blast radius. 2. Regulatory exposure on PHI data. The application processes Protected Health Information. Any data integrity issue — a missed field, a misformatted record, a sync gap — could have triggered regulatory scrutiny. By exposing the new AWS environment to a small group of business partners first, we could validate end-to-end data flow in real production conditions without putting the full user base or compliance posture at risk. 3. UX learning curve. We had explicitly rejected a lift-and-shift approach. The new application wasn’t just re-hosted — the UI had been redesigned, the APIs restructured, and user workflows updated. Even excellent technical execution couldn’t eliminate the learning curve our users would face. Dark deployment gave us 15 days of real-world UX observation: where do users hesitate, what do they misunderstand, which workflows feel awkward? By the time we cut over publicly, we had already addressed the rough edges. The result: When we replaced the WAS production URL with the AWS production URL, end users perceived the change as a routine UI update, not a foundational technology migration. Pattern 2: Decouple Application Migration From Data Migration The default assumption in cloud migration is that application and data should move together. We made the opposite choice: migrate the application to AWS while keeping the underlying DB2 data on-premises. Three factors made this the right call. 1. PHI/HIPAA compliance complexity. The application processes Protected Health Information governed by HIPAA. Moving regulated healthcare data to a new environment raises a long list of compliance questions — encryption-at-rest configurations, audit logging, access control policies, business associate agreements with the cloud provider, breach notification readiness. None of these are insurmountable, but they take months of compliance review. Treating data migration as a separate workstream with its own compliance approval cycle was significantly less risky than bundling it into the application cutover. 2. Client comfort and trust-building. Cloud migration is as much a psychological transition for the client as a technical one. Moving an application to AWS is one decision; moving sensitive data off the client’s own infrastructure is a much larger one — it changes their security perimeter, their incident response posture, and in some cases their regulatory filings. Insisting on moving both at once would have either delayed the program waiting for full executive comfort, or risked a "no" on the entire initiative. Application-first let us demonstrate the new architecture working successfully before the data migration conversation began. 3. Parallel team enablement. Decoupling created room for a separate analytics team to independently assess which data could move to the cloud, on what timeline, and under what compliance framework. The application architecture was designed from day one to support a hybrid future — partial data on AWS, other data on-prem — so the analytics team’s work didn’t block application progress. How the technical decoupling works. The natural temptation when keeping data on-prem is to expose a direct database connection from the AWS application back to the on-prem DB2 instance. We rejected that — opening database ports across the cloud-to-on-prem boundary is a security liability, a latency problem, and a fragile dependency. Instead, we used IBM’s DB2 REST API layer to expose data access through authenticated HTTPS-based service calls. The AWS application talks to data through an API, not a database connection. This abstraction also positions the application to seamlessly switch to AWS-resident data later, without any application code change — only the API endpoint moves. Network-layer security follows the same decoupling principle. We provisioned dedicated AWS security groups on the Fargate side specifically for the IMS and DB2 connections back to the on-premises environment — only requests from those approved security groups can traverse the firewall to the on-prem data tier. Combined with the REST API abstraction, this gives us both application-layer (authenticated HTTPS) and network-layer (security-group-controlled) protection across the cloud-to-on-prem boundary. The result: A successful cloud migration with regulatory exposure isolated to a single workstream, and a forward path that doesn’t force the client into uncomfortable decisions before they’re ready. Pattern 3: EJB Monolith → Containerized Microservices on Fargate The original application was a Java EJB monolith running on WebSphere. The "lift-and-shift" temptation would have been to containerize the existing EJB code as-is into AWS Fargate — preserving the architecture, just moving the deployment substrate. We rejected that and instead decomposed the monolith into bounded REST microservices. Three reasons drove this decision. 1. Downstream services were also migrating. The application integrated with 5–7 SOAP-based services owned by adjacent teams — agreement service, customer service, sensitive data masking, and others. Those teams were simultaneously migrating their own services from WAS to AWS, which meant interface contracts, protocols, and endpoints would inevitably change. Inside an EJB monolith, every downstream integration change forces a recompile-redeploy-retest cycle of the entire application. Inside microservices, only the integration adapter for the affected service needs to change. With multiple active migration interfaces, the flexibility difference compounds quickly. 2. EJB development velocity is structurally slow. Even routine changes to EJB code require a full WAR/EAR build, redeployment to the WAS instance, and a heavy test cycle. The technology wasn’t designed for the iteration speed we needed to support a multi-year migration alongside actively changing downstream dependencies. Microservices on Fargate gave us a development model — fast container builds, independent deployments, isolated test environments — that matched the pace of the work. 3. Future data migration optionality. As noted in Pattern 2, the underlying data was kept on-premises for now, but a phased data migration to AWS was planned. By isolating database calls and IMS calls into dedicated microservices, the change required when the data eventually moves is localized — swap one service’s data access logic rather than reworking the monolith. The architecture is positioned for the data move whenever the client is ready. How we sized the decomposition. The boundaries followed natural integration points: each external SOAP integration became its own bounded microservice with a thin REST API. Data access calls (DB2 via REST, IMS) were isolated into dedicated services. The frontend talks to a coordination layer that orchestrates calls across these services. The result was a clean set of containerized microservices on AWS Fargate — each independently deployable, scalable, and testable. The result: A modernization that didn’t just relocate the code, but restructured it to absorb the inevitable changes coming from adjacent migrations across the organization — without recompile-redeploy-retest pain. Pattern 4: Frontend Decoupling via S3 + CloudFront The original WAS application followed the classic tightly-coupled pattern: JSP pages rendered server-side, deployed alongside the backend, scaling and updating as one unit. We made an architectural break in the migration — the frontend became a fully independent single-page React application hosted on Amazon S3 and served via CloudFront. Three factors made this the right call. 1. Independent deployment cadence. Frontend and backend evolve at different speeds. UI tweaks — copy changes, validation logic, visual updates — are frequent and low-risk. Backend API changes are slower and require careful coordination with downstream service migrations. Decoupling them means UI changes can be deployed instantly through a separate UI pipeline (different Git repository, different infrastructure, different release cadence) without touching the backend microservices. A small label change no longer requires a full backend deployment. 2. Adopting an accessibility-first enterprise UI library. Alongside our migration, an internal innovation track was building a shared component library to unify UX patterns across the organization’s applications — consistent typography, controls, brand elements, and critically, accessibility as a first-class concern: full screen reader support, keyboard navigation, sufficient color contrast, and ARIA-compliant semantics. JSP-based legacy pages couldn’t meaningfully integrate this kind of library. By rebuilding the frontend as a React single-page application, we adopted the library fully — and incorporated rigorous accessibility testing into every release cycle. Users who rely on assistive technologies (screen readers, alternative input devices, magnification) get full application access. For an application processing PHI in a regulated industry, this proactive accessibility-first approach is itself a substantial improvement over the legacy app. 3. Global performance through edge caching. S3 alone would have served the static assets, but we layered CloudFront on top to push content to edge locations closer to users. Business partners access the application from different geographic regions; CloudFront cuts load times by serving cached assets from the nearest edge, not the S3 origin in a single AWS region. This is a substantial UX improvement that simply wasn’t possible with WAS-hosted JSPs. How the architecture flows. User requests hit CloudFront, which serves cached React bundles, HTML shells, and static assets from the nearest edge. The React application then makes authenticated REST API calls back to the backend microservices on AWS Fargate. The frontend has no awareness of which microservice serves any particular request — it talks to a coordination API layer that handles orchestration. The result: A UI architecture that’s faster (edge-cached), cheaper (no application servers for the frontend), easier to update (independent pipeline), more inclusive (accessibility-first), and aligned with the broader enterprise UX modernization effort. Pattern 5: Business Partner Real-Production Validation Cohort Pattern 1 described the deployment mechanism — a 15-day dark deployment exposing AWS production to a limited cohort. Pattern 5 is about who was in that cohort and why we deliberately chose real business partners over our QA team for production validation. Two factors shaped this decision. 1. Decades of muscle memory in the existing UX. Our business partners — long-tenured users of the application — had been using the legacy UI for 10–15 years. They knew every workflow, every shortcut, every quirk. The new React application introduced not just a new visual style but new patterns from the organization’s modern component library. Even with rigorous accessibility and usability testing in QA, a brand-new UI in front of users with a decade of habits guaranteed friction. The 15-day validation cycle gave those users time to acclimate to the new patterns and surface UX issues that only show up at the speed of real daily work — keyboard shortcuts they used unconsciously, screens they navigated to multiple times an hour, validation logic that affected their flow. QA testers, by definition, don’t have that muscle memory. 2. First-of-its-kind migration with concurrent change. This was the first WAS-to-AWS migration in the health division, and we’d simultaneously re-architected the UI, the API layer, and incorporated changes from downstream services that were also mid-migration. With that many concurrent changes, even thorough QA can’t realistically simulate the full combinatorial space of real production usage — real customer data, real edge cases, real integration timing, real load patterns. Putting real business partners on the actual AWS production environment for 15 days was our safety net: anything QA missed, the cohort would surface, and we could fix it before broad cutover. Beyond the cohort: maturing the delivery pipeline. A secondary benefit of running an extended validation window was that it gave the engineering team time to mature the CI/CD pipeline alongside the application. By the second application in the migration program, we’d evolved the cohort approach into a full blue/green deployment model on AWS — building organizational learning alongside the application portfolio. The validation pattern isn’t static; it strengthens with each subsequent migration. The result: a validation approach that combined deep domain familiarity (real business partners) with controlled exposure (limited cohort, real production) — catching the issues QA can’t, well before public cutover. Pattern 6: Defensive Tokenization for Sensitive Data in Free-Form Fields In regulated industries, the obvious sensitive data — SSN fields, credit card fields, account number fields — gets protected automatically. The dangerous category is the unstructured data: a free-form text field where a user can type anything. In our application, users entered "health notes" — narrative text describing customer interactions. The risk: nothing in the application schema prevents a user from typing an SSN, a credit card number, a driver’s license, or other regulated identifiers directly into that note. Once stored, that PHI/PII data is sitting in a free-text column with no encryption-at-rest tailored to it, no masking on display, no controlled access — and our compliance posture changes accordingly. We addressed this proactively by integrating an internal sensitive-data-masking service into the application’s write path. Before any free-form text reaches the data layer, the masking service scans the input, identifies regulated identifiers (SSN-pattern strings, credit card numbers via Luhn check, driver’s license formats), and applies tokenization — replacing the identifier with a non-reversible token or masked representation. The original value never lands in the database in plaintext. Three things made this a deliberate architectural pattern, not an afterthought: 1. It was incorporated before the formal risk assessment, not in response to it. Risk assessment was a new exercise for the team — none of us had been through one for AWS-hosted PHI before. Rather than wait for the assessment to flag the free-form field as a finding, we performed our own data classification first, identified the free-form notes as a regulated-data risk vector, and integrated the masking service pre-emptively. When the formal risk assessment ran, this control was already in place. 2. We reused an existing internal service, not built a new one. The masking service already existed in another WAS-hosted application within the broader life/health portfolio. Instead of re-implementing tokenization logic, we adopted the existing service — saving development time and inheriting the existing security review and operational maturity of that service. Migrations are a good moment to identify reusable internal capabilities rather than reinvent them. 3. It addresses a class of risk most compliance reviews don’t anticipate. Compliance checklists focus on declared sensitive fields ("the SSN field," "the account number field"). They rarely interrogate free-form text fields, because those fields aren’t supposed to hold sensitive data. But in practice, users type whatever they need to type — and what they type is what your application stores. Proactive defensive tokenization closes that gap. The result: free-form notes that look normal to users, but whose backend storage is sanitized of any regulated identifiers the user may inadvertently include. The application’s compliance posture is robust to user behavior, not just to user intent. Conclusion: The Through-Line Is Decoupling Looking back across the six patterns, the through-line isn’t any specific technology — it’s a posture: deliberate decoupling of risk vectors so that no single failure, regulatory finding, organizational hesitation, or user adoption gap can derail the whole migration. Pattern 1 (Strangler Fig with Dark Deployment) decouples cutover risk from broader rollout.Pattern 2 (Decouple App from Data) decouples application migration from the data-and-compliance timeline.Pattern 3 (EJB → Microservices) decouples downstream integration changes from our own deployment cadence.Pattern 4 (Frontend on S3/CloudFront) decouples UI release cadence from backend release cadence.Pattern 5 (Business Partner Validation Cohort) decouples real-world UX surprises from public rollout.Pattern 6 (Defensive Tokenization) decouples user behavior risk from data-layer compliance posture. None of these patterns are individually novel. What’s distinctive is choosing them together, as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. The result was a migration that didn’t surprise our compliance team, didn’t surprise our users, and didn’t surprise our auditors — which, in a regulated industry, is the kind of unsexy outcome that defines success. If you’re starting a similar program, the question isn’t which of these patterns to adopt. It’s: which risk vector are you decoupling, and is your team aligned on why?

By Alka Nimje

Containerizing Spark and Lakehouse Development with Docker

Most Docker content targets web developers shipping stateless services. However, data engineers, who represent a huge and growing population of Dockers users, are mostly left to figure things out alone, and it shows. The get pipelines that pass locally, but explode on clusters. They pit notebook-only development against expensive cloud workspaces, and more. This article applies six years of production data platform experience in financial services and healthcare to a question nobody answers well: How to you make a laptop behave like a lakehouse? A Familiar Routine If you build data pipelines for a living, you've lived this story. Your PySpark job runs perfectly in a cloud notebook. You productionize it, push it through CI, deploy it to the cluster, and it fails. A dependency mismatch. A different Spark minor version. A Delta Lake protocol feature your local wheel doesn't know about. A timezone default nobody set. Web developers solved "works on my machine" a decade ago with containers. Data engineers, somehow, are still developing against shared cloud workspaces, paying per-minute cluster costs to debug a GROUP BY, and discovering environment drift in production. This article is the workflow I wish someone had handed me years ago: a fully containerized lakehouse development environment — Spark, Delta Lake, object storage, a catalog, and orchestration — that runs on a laptop, mirrors production closely enough to trust, and plugs into CI without mocks. The Real Problem: Data Pipelines Have Four Environments, Not One A typical stateless web service has one environment to reproduce: the app runtime. A data pipeline has at least four, and they drift independently: The compute runtime — Spark version, Scala version, JVM, Python, native libs (Arrow, Parquet, libhdfs).The table format layer — Delta Lake / Iceberg versions and protocol versions, which are not the same thing.The storage layer — S3/ADLS semantics: multipart uploads, eventual consistency quirks, path-style vs virtual-hosted access.The orchestration layer — the scheduler's Python environment, which is famously not your job's environment. Mocking any one of these in tests means you aren't testing the thing that breaks. The goal of containerizing a lakehouse is to pin all four layers in code and version them together. Step 1: A Reproducible Spark Image You Actually Control Don't develop against latest. Build a base image that pins every layer of the compute runtime and treat it like an artifact: Dockerfile # syntax=docker/dockerfile:1.7 FROM eclipse-temurin:17-jre-jammy AS base ARG SPARK_VERSION=3.5.4 ARG DELTA_VERSION=3.3.0 ARG HADOOP_AWS_VERSION=3.3.6 RUN apt-get update && apt-get install -y --no-install-recommends \ python3.11 python3-pip tini && \ rm -rf /var/lib/apt/lists/* # Pin Spark itself, not just PySpark RUN curl -fsSL https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz \ | tar -xz -C /opt && mv /opt/spark-${SPARK_VERSION}-bin-hadoop3 /opt/spark ENV SPARK_HOME=/opt/spark PATH=$PATH:/opt/spark/bin PYTHONHASHSEED=0 TZ=UTC # Delta + S3 connectors resolved at build time, never at job submit time RUN /opt/spark/bin/spark-shell --packages \ io.delta:delta-spark_2.12:${DELTA_VERSION},org.apache.hadoop:hadoop-aws:${HADOOP_AWS_VERSION} \ -e "println(\"deps cached\")" && \ cp /root/.ivy2/jars/*.jar /opt/spark/jars/ COPY requirements.lock /tmp/ RUN pip install --no-cache-dir -r /tmp/requirements.lock # Never run Spark as root RUN useradd -m -u 1001 spark USER 1001 ENTRYPOINT ["/usr/bin/tini", "--"] Three details that matter more than they look: --packages at build time, not submit time. Resolving connector JARs at spark-submit is the #1 source of "it worked yesterday" failures — Maven Central is a runtime dependency you didn't mean to have.PYTHONHASHSEED=0 and TZ=UTC kill two classes of "non-deterministic only in prod" bugs.A lockfile, not requirements.txt. Compile with pip-compile or uv pip compile so transitive dependencies (looking at you, pandas/pyarrow) can't drift. Step 2: The Lakehouse-In-A-Box With Docker Compose Here's the part most teams never build: the rest of the lakehouse, locally. MinIO stands in for S3 (it speaks the same API), and a real Spark master/worker pair stands in for the cluster, because local[*] mode hides every serialization and shuffle bug you'll meet in production. Dockerfile # compose.yaml services: spark-master: build: . command: /opt/spark/sbin/start-master.sh environment: [SPARK_NO_DAEMONIZE=true] ports: ["7077:7077", "8080:8080"] spark-worker: build: . command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077 environment: - SPARK_NO_DAEMONIZE=true - SPARK_WORKER_MEMORY=4g - SPARK_WORKER_CORES=2 depends_on: [spark-master] deploy: replicas: 2 # >1 worker = real shuffles, real serialization minio: image: minio/minio:RELEASE.2025-09-07T16-13-09Z command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: localdev MINIO_ROOT_PASSWORD: localdev-secret ports: ["9000:9000", "9001:9001"] volumes: [lake-data:/data] healthcheck: test: ["CMD", "mc", "ready", "local"] interval: 5s mc-init: # create the bronze/silver/gold buckets on boot image: minio/mc:latest depends_on: { minio: { condition: service_healthy } } entrypoint: > /bin/sh -c "mc alias set local http://minio:9000 localdev localdev-secret && mc mb -p local/lakehouse/bronze local/lakehouse/silver local/lakehouse/gold" volumes: lake-data: Point Spark at MinIO with three config lines and your medallion pipeline reads and writes s3a://lakehouse/... paths exactly like production: Python spark = (SparkSession.builder .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000") .config("spark.hadoop.fs.s3a.path.style.access", "true") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate()) docker compose up and you have bronze → silver → gold on your laptop. Total cloud cost of a debugging session: $0. Step 3: Integration Tests That Run Real Spark — Testcontainers The payoff of all this is CI you can trust. With Testcontainers, your pipeline tests spin up the same images your developers use: Python import pytest from testcontainers.minio import MinioContainer from pyspark.sql import SparkSession @pytest.fixture(scope="session") def lake(request): with MinioContainer("minio/minio:RELEASE.2025-09-07T16-13-09Z") as minio: yield minio def test_silver_dedup_keeps_latest_record(lake, spark): # write duplicate customer events to bronze bronze_path = f"s3a://test/bronze/customers" write_fixture_events(spark, bronze_path, duplicates=True) run_silver_dedup(spark, bronze_path, "s3a://test/silver/customers") result = spark.read.format("delta").load("s3a://test/silver/customers") assert result.count() == EXPECTED_UNIQUE assert latest_record_wins(result) No mocked DataFrames. No unittest.mock.patch("boto3..."). The test exercises Delta's actual transaction log against actual object storage. When this suite is green, deployments stop being scary. A pattern I use in regulated environments: keep a fixtures/ directory of small, synthetic Parquet files that mirror production schemas (never production data), and version them with the code. Schema drift then fails a unit test instead of a 2 a.m. pipeline run. Step 4: One Image From Laptop → CI → Production The final principle: the image you test is the artifact you ship. Multi-stage builds let one Dockerfile serve dev (with Jupyter, debuggers) and prod (minimal, non-root): Dockerfile FROM base AS dev USER root RUN pip install --no-cache-dir jupyterlab pytest debugpy USER 1001 FROM base AS prod COPY --chown=1001:1001 src/ /app/src/ COPY --chown=1001:1001 jobs/ /app/jobs/ # nothing else — no notebooks, no test deps, no shell tools you don't need In CI: build once, tag with the git SHA, run the Testcontainers suite against prod, scan it (Docker Scout, or your registry's scanner), sign it, and promote that exact digest through staging to the scheduler. Whether the scheduler is Airflow's DockerOperator/KubernetesPodExecutor or a managed Spark platform pulling custom containers, the principle holds: environments are immutable, versioned, and identical by construction. Lessons Learned From Production Run ≥2 workers locally.local[*] mode never serializes between JVMs. The day you switch to a real cluster, every closure-capture and UDF-pickling bug appears at once. Two 2-core workers in Compose surfaces them on day one.Pin the table format protocol, not just the library. Delta and Iceberg both evolve table protocol versions. A newer writer can produce tables an older reader can't open. Encode the protocol version in your image build args and test reads with the oldest reader you support.MinIO is a stand-in, not a clone. It won't reproduce S3 request throttling or cross-region latency. Keep a small smoke-test suite that runs against real object storage nightly; do everything else locally.Resource-limit your local Spark. Without SPARK_WORKER_MEMORY caps, a skewed join will cheerfully eat your laptop. Limits also force you to think about partitioning early — which is the point.Treat the orchestrator's image as layer four. Airflow DAG-parse environments drift too. Containerize the scheduler with the same lockfile discipline as the jobs. Production Considerations Before you take this pattern to a real platform team, three things to plan for: secrets (local Compose uses throwaway creds; production should inject via your cloud's secret manager or Docker secrets — never baked into images), image provenance (sign images and generate SBOMs in CI; regulated industries will ask, and in 2026 the tooling is mature enough that "we didn't get to it" no longer flies), and base image hygiene (start from minimal, hardened bases and rebuild on a schedule, not just on code change — CVEs don't wait for your sprint). Conclusion Containers gave application developers reproducibility ten years ago. Data engineering is finally having the same moment — and the teams that containerize their lakehouse development loop ship faster, test honestly, and stop paying cloud bills to find typos. Try it: clone the Compose stack above, point your gnarliest pipeline at it, and see what breaks locally that used to break in prod. Then tell me about it — I'd genuinely like to hear which layer drifted on you. If this was useful, follow me here and on LinkedIn. Next up in this series: load-testing Delta merge performance locally, and contract testing between pipeline stages.

By Aniket Abhishek Soni

Multi-Account AWS Architecture: Isolating PHI Workloads Without Slowing Down Engineering Teams

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings.

By Garik H

From Bottlenecks to Reliability: A Practical Guide to Scaling Temporal in Production

Temporal is designed to preserve Workflow state through process crashes and infrastructure failures, but durable state does not remove ordinary capacity limits. In production, the control plane can remain healthy while throughput collapses because Worker slots are saturated, Task Queues mix incompatible workloads, or a failover activates a region without enough Worker capacity. Temporal Workers run outside the Temporal Service and execute Workflow and Activity code, so production scalability depends as much on Worker and routing design as on the service itself. The Worker Fleet Is Usually the First Capacity Boundary Schedule-to-Start latency is best treated as queueing delay rather than application execution time. It measures the interval between a Task being enqueued and a Worker starting it. Rising Schedule-to-Start latency, growing approximate backlog, and exhausted Worker task slots indicate that Tasks are arriving faster than the fleet can consume them. Temporal Cloud exposes temporal_cloud_v1_approximate_backlog_count, while SDK metrics expose Workflow and Activity Schedule-to-Start latency and available task slots. Temporal guidance recommends watching these signals together because backlog depth alone does not identify whether the limit is Worker count, Worker configuration, or polling behavior. Worker scaling has two layers. Horizontal scaling adds Worker processes, while concurrency tuning changes how many Tasks each process can execute simultaneously. For well-benchmarked workloads, fixed slot limits place a predictable ceiling on local resource consumption. The Java SDK exposes separate concurrency controls for Workflow Tasks and Activities, and a server-side Activity rate limit can cap dispatch across all Workers polling the same Task Queue. Java WorkerOptions options = WorkerOptions.newBuilder() .setMaxConcurrentWorkflowTaskExecutionSize(120) .setMaxConcurrentActivityExecutionSize(80) .setMaxTaskQueueActivitiesPerSecond(250) .build(); The values in this example are capacity-test outputs, not universal defaults. A CPU-heavy Activity fleet may need a lower Activity slot count than an I/O-heavy fleet. Newer Worker tuners can allocate slots dynamically from CPU and memory signals, while fixed-size suppliers remain more predictable when task resource cost is well understood. Temporal also recommends poller autoscaling for most workloads because too few pollers constrain ingestion and too many waste connections and reduce efficiency. Task Queue Topology Determines Isolation and Backpressure Adding replicas cannot repair a Task Queue topology that couples unrelated bottlenecks. A shared Task Queue is reasonable when Workflows and Activities have similar latency and resource characteristics, but it becomes risky when fast orchestration work shares capacity with slow database calls, GPU jobs, tenant bursts, or Activities constrained by a downstream API. Temporal supports specialized routing through separate Task Queues, and Activity-level server-side throttling applies to the entire queue. A throttled Activity therefore should not share a queue with work that must remain unrestricted. A Workflow can route a costly Activity to a dedicated fleet without changing the Workflow’s own Task Queue. The separation creates an independent scaling and backpressure boundary. Java ActivityOptions options = ActivityOptions.newBuilder() .setTaskQueue("payments-io") .setStartToCloseTimeout(Duration.ofSeconds(20)) .build(); PaymentActivities payments = Workflow.newActivityStub(PaymentActivities.class, options); With payments-io isolated, replicas, concurrency, credentials, network placement, and queue-wide rate limits can be tuned for payment traffic without changing the Worker pool that advances Workflow Tasks. The same principle applies to multi-tenant systems. Temporal documents per-tenant Task Queues as a strong isolation pattern and also supports fairness keys when many tenants share one queue. Priority and fairness operate within Task Queue partitions, so they manage contention inside a queue rather than replacing isolation when resource requirements differ fundamentally. Task Queue partitioning should also be distinguished from application-level queue proliferation. Temporal Task Queues are lightweight and scale internally through partitions; current documentation states that Task Queues use four partitions by default. Multiple partitions increase throughput but relax strict FIFO behavior because Tasks are distributed among partitions. Separate named queues should therefore be created for routing, isolation, or rate-control reasons, not merely to manufacture throughput that Temporal’s matching layer can already scale internally. Autoscaling Should Follow Queue Pressure, Not CPU Alone CPU-based autoscaling is insufficient for many Temporal workloads. An I/O-bound Activity can leave CPU utilization low while all Activity slots are occupied and backlog grows. Conversely, high CPU with near-zero Schedule-to-Start latency may mean that the fleet is efficiently utilized. A stronger autoscaling policy combines queue delay, backlog trend, slot availability, and host resource saturation. Temporal’s Worker health guidance treats Schedule-to-Start latency as a primary symptom of insufficient processing capacity and recommends correlating it with sync-match behavior and available slots before changing fleet size. On Kubernetes, Temporal’s Worker Controller can attach HPA or KEDA resources to versioned Worker deployments and scale from CPU, memory, Task Queue backlog, slot utilization, or custom metrics. Current guidance recommends HPA with a Prometheus adapter as the general default, while KEDA is positioned for scale-to-zero, long idle periods, or faster event-driven reactions. This matters because old and new Worker versions can coexist during safe rollout, so autoscaling should follow each active Worker Deployment Version rather than treating the fleet as a single anonymous pool. Scale-down deserves the same attention as scale-up. Backlog can reach zero while Activities are still running, and terminating aggressively can create retries or latency spikes. Worker shutdown should therefore be graceful, minimum replica counts should reflect availability requirements, and cooldowns should account for Activity duration and startup time. Pre-production tests should include Worker termination, burst recovery, and partial failure because Temporal durability preserves state but does not guarantee that an undersized replacement fleet will meet latency objectives. Regional Failover Has to Include Workers and Dependencies Regional failover is often mis-scoped as a Temporal Service feature. Temporal Cloud High Availability replicates a Namespace to a secondary region and can automatically promote the replica during an outage, but application Workers remain separately operated compute. Temporal documents a 20-minute RTO and sub-one-minute RPO for its HA service, yet application recovery can still be slower when the secondary region lacks ready Worker capacity, network access to the active Namespace, or available downstream systems. For latency-sensitive systems, Active/Hot-Passive is the most deterministic failover model: a full Worker fleet runs in both regions, the secondary fleet stays warm, and only the fleet local to the active replica processes Tasks. On failover, the warm fleet begins processing without a Worker cold start. Active/Passive costs less but requires starting or scaling Workers after failover, while Active/Active runs Workers in multiple regions even though the HA Namespace still has one active replica underneath. Connectivity must be tested as part of the failover path. For HA Namespaces, the Namespace Endpoint follows the active region through DNS; Temporal documents a 15-second TTL and roughly 30 seconds for clients to converge when resolvers honor that TTL. Private connectivity requires routes and DNS design that allow Workers to reach the promoted region. A test that switches only the Namespace but omits Worker connectivity, database promotion, queue access, secrets, codec servers, or proxies validates only part of the production path. Self-hosted multi-cluster deployments require explicit planning as well. Temporal’s Global Namespace model uses asynchronous cross-cluster replication and eventual conflict resolution, and successful failover requires Worker Processes to poll the Namespace in clusters that may become active. Replication versions determine which cluster can mutate Workflow history after failover, but they do not provision Worker compute or external dependencies. Conclusion Temporal becomes a production bottleneck when durable orchestration is treated as a substitute for capacity engineering. Stable performance comes from measuring queue delay and slot saturation, scaling Worker fleets from demand signals rather than CPU alone, separating Task Queues where workloads need independent isolation or rate control, and designing regional failover around ready Workers and reachable dependencies. With those boundaries in place, Temporal remains the durable coordination layer rather than the slowest component in the execution path.

By Akhil Madineni

CORE

AWS Bedrock vs Vertex AI vs Azure Foundry: Stop Comparing Benchmarks, Start Asking This Instead

Every few weeks, someone on my team, or in a client meeting, asks me the same question: "Which cloud should we use for our AI workloads?" I have been building enterprise integrations for over fourteen years now, and lately most of my time goes into RAG pipelines, vector databases, and agentic orchestration on top of these platforms. So I get this question a lot, and honestly, there is no single right answer. The right cloud depends on where your data already lives, what your compliance team will accept, and which models your architecture actually needs. In this article, I want to walk through the three big players, AWS Bedrock, Google Vertex AI, and Microsoft Azure AI Foundry, and share what I have learned working with these platforms in real enterprise settings, not just from reading marketing pages. AWS Bedrock Bedrock started as a model marketplace back in 2023, and it has grown into a full platform with Guardrails for content filtering, Knowledge Bases for RAG, and AgentCore for building agentic workflows. What I like most about Bedrock is the sheer breadth of models available behind a single API. You get Claude from Anthropic, Llama from Meta, Mistral, Cohere's Command models, and Amazon's own Nova family, all through one consistent interface. If your architecture needs to swap models without rewriting your integration layer, Bedrock makes that easier than the other two. Pros: Broadest model catalog of the three, so you are not locked into one vendor's models.Strong identity and governance story if you are already running on AWS, since it plugs directly into IAM, CloudTrail, and Macie.Bedrock is one of the few places where you get Claude with enterprise indemnification, which matters a lot when legal teams get involved.Provisioned throughput options give you predictable latency for production workloads that cannot tolerate spikes. Cons: If your organization is not already AWS-native, the onboarding curve is steeper than it looks.Cross-cloud portability is basically nonexistent. A model you fine-tune on Bedrock does not export cleanly to Vertex AI or Foundry. That is a real switching cost you should plan for on day one, not something to figure out later.Some of the newer agentic tooling is still maturing, so documentation gaps show up more than I would like. Google Vertex AI Vertex AI feels different from the other two because Google's DNA here is research first. If your team cares about multimodal capability, or you want access to Gemini models the moment they ship, Vertex AI tends to be ahead. It is also the strongest option if your data already lives in BigQuery, because the integration between Vertex and BigQuery for feature engineering and MLOps pipelines is genuinely smooth. Pros: Best fit for teams doing custom model training, not just calling a hosted API. AutoML and the broader MLOps tooling cut training time noticeably compared to the other two.Tight coupling with BigQuery is a huge advantage if your organization already runs its analytics there. You avoid a lot of data movement overhead.Gemini-first multimodal workflows, plus Google Search grounding for agents, which is something neither Bedrock nor Foundry offers natively.TPU support gives real throughput advantages for heavy batch processing. Cons: If your organization is not GCP-centric already, the value proposition weakens fast. You end up paying a data-gravity tax to move information into Google's ecosystem.Governance and compliance tooling, while solid, is not as battle-tested across regulated industries as AWS's certifications.The agent ecosystem, while improving, still trails Bedrock's AgentCore and Foundry's Azure AI Agents in terms of enterprise adoption stories I have personally seen. Azure AI Foundry Foundry, formerly Azure AI Services, is Microsoft's rebranded and expanded platform, and it is the one I have written about before because it is what my own recent client work has centered on. If your enterprise already lives inside Microsoft 365, Entra ID, and Azure infrastructure, Foundry removes almost all of the identity and governance friction you would otherwise deal with. That matters more than people expect once you are past the proof of concept stage and into actual production rollout with security review. Pros: Deep Microsoft 365 and Entra ID integration means your existing enterprise approvals and identity workflows extend naturally into your AI layer.Strong OpenAI-led model access, since Microsoft's partnership with OpenAI gives Foundry early and deep access to GPT-family models.Hybrid deployment options are genuinely better here than on the other two platforms, which matters if you have on-prem systems you are not ready to fully cloud-migrate.Roughly three-quarters of Fortune 500 companies already run on Microsoft's stack, so for a lot of enterprises Foundry is simply the path of least resistance. Cons: Model breadth is narrower than Bedrock's catalog, so if you need a specific non-OpenAI model family, you may find yourself stitching together a secondary platform anyway.Because it is tied so closely to Azure compute pricing, cost predictability requires more upfront modeling than teams expect.Some newer agentic and orchestration features are still catching up to what AWS has shipped with AgentCore. So Which One Should You Actually Pick? Here is the honest answer I give in client meetings: do not choose based on a benchmark screenshot or a features table. Choose based on where your data already lives and where your governance and compliance story already works. If you are AWS-first and want maximum model flexibility, go with Bedrock. If you are Microsoft-heavy and need your AI layer to inherit existing Entra ID and 365 approvals without a fight, Foundry is the path of least resistance. If your analytics already lives in BigQuery and multimodal Gemini capability is core to your roadmap, Vertex AI earns its place. What I am increasingly seeing among the teams I work with is a hybrid pattern. A primary cloud handles the bulk of regulated workloads, and a secondary cloud gets called in only when a specific model family is not well supported on the primary platform. It is not the cleanest architecture on paper, but it reflects how fast this space is still moving. None of these three platforms is standing still, and the leader on any given feature this quarter is not guaranteed to hold that spot by next year. My suggestion, whichever cloud you land on: build your RAG and orchestration layer with enough abstraction that swapping the underlying model provider is a configuration change, not a rewrite. That single decision will save you more pain than picking the "right" cloud ever will.

By Balaji Venkatasubramaniyar

CORE

Cloud Architecture

DZone's Featured Cloud Architecture Resources

Monthly Top Cloud Architecture Experts

The Latest Cloud Architecture Topics