Tools

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings.

In this article, we will build a simple understanding of the following: What a model isWhy a model needs toolsWhat tools areHow an agent uses tools Model vs. ChatGPT Before understanding agents, let's clarify the difference between a model and ChatGPT. Whatever question we type into ChatGPT is sent to a model behind the scenes, which generates the response. You can think of ChatGPT as a web or mobile application — an interface through which we interact with the underlying Model/LLM. A model is a component that processes our query and generates a response. Models are trained on large amounts of data from many different sources, such as books, articles, publicly available websites, and other information. Because models learn from large, diverse datasets, they can develop broad knowledge and generate meaningful responses to many types of queries. However, models have limitations. A model or LLM can only work with the information it is trained on. If it doesn't have access to information, it cannot retrieve that information by itself. This is where tools and agents become important. Let's understand this with an example. Why Do We Need Tools? Suppose a user asks, "What is the value of my 0.5 BTC in INR right now?" To answer the user's question accurately, the model or LLM need the current Bitcoin price. A model may know about Bitcoin from its training data, but that doesn't mean it has access to the current Bitcoin price. It might respond with something like: "I don't have access to live market data, but Bitcoin is generally valued in several million INR." This isn't sufficient because the user specifically asked for the value at this time. We need to extend the model's capabilities. This is where tools come in. What Is a Tool? A tool can be thought of as a piece of code that performs a specific task. In a Python application, for example, a tool can be implemented as a Python function that: Calls an external APIRetrieves informationPerforms calculationSearches databaseInteracts with another application For our Bitcoin example, let's assume we have two tools: Tool 1: get_crypto_price This tool retrieves the current Bitcoin price in INR from an external source, such as an API. Tool 2: calculate_investment_value This tool calculates the total value of the user's Bitcoin investment. The calculation is straightforward: Investment Value = Current Price X quantity So, if the user owns 0.5 BTC, we can multiply the current Bitcoin price by 0.5 to determine the current value. Now, we have given the model additional capabilities through tools, but these tools are not executed directly by the model. So, how does the model actually use these tools? How Does the Model Use Tools? Let's simplify the process. The user provides a query and makes the available tools known to the model. For example: get_crypto_price - gets the latest crypto pricecalculate_investment_value: calculates the investment value The model can then determine whether one of these tools is required to answer the user's query. For our example, the model needs the current Bitcoin price first. So, it generates a request to call get_crypto_price. The user can execute the tool and send the result to the model. The model then examines the result and determines what needs to happen next. Since the user wants to know the value of their 0.5 BTC, the model determines that the calculate_investment_value tool needs to be executed. User executes the tool and returns the result to the model. Finally, the model has enough information to generate the answer for the user. The whole process can be visualized as: This example demonstrates the important concept: the model can determine which tool is needed and in what sequence, but someone or something needs to execute these tools. The above example involves a lot of manual intervention. The user shouldn't have to remain involved every time the model needs to perform the action. We could create an application that communicates with the model and executes these tools on the user's behalf. And this brings us to the agents. What Is an Agent? An agent is a piece of code that can work with a model and a set of tools to accomplish a goal. The agents act as an orchestration layer between the model and the tools. Instead of the user manually executing every tool, the agent can execute the appropriate tool based on the model's output, collect the result, and send it back to the model. Let's look at the process step by step: Step 1: User Provides a Goal The user asks, "What is the value of my 0.5 BTC in INR right now?" Step 2: Agent Sends a Query to the Model The agent sends the user's query to the Model along with the information available about the available tools. The model can now determine what needs to be done to answer the user's query. Step 3: Model Determines the Required Tool The model determines that it needs the current Bitcoin price. It generates a tool execution request for: get_crypto_price. Step 4: Agent Executes the Tool The agent receives the model's tool execution request and executes the corresponding tool immediately. The tool retrieves the current Bitcoin price. Step 5: Agent Sends the Result Back to the Model The agent sends the tool's result back to the model. The model now has the current Bitcoin price and can determine the next action. Step 6: Model Determines the Next Tool The model determines that it needs the value of the user's 0.5 BTC. It generates a tool execution request for: calculate_investment_value. Step 7: Agent Executes the Second Tool The agent executes the tool and obtains the calculated investment value. The result is again returned to the model. Step 8: Model Generates the Final Answer Once the model has the required information, it generates the final response for the user. The user doesn't have to manually execute either tool. The agent has handled the tool execution on the user's behalf. Model, Agent, and Tool: How Are They Different? At this point, it helps to separate the responsibilities of the three components: Model The model provides the reasoning and determines what should happen next based on the available information and tools. Tool A tool performs a specific task, such as retrieving current data, calling an API, performing calculations, or interacting with another system. Agent The agent orchestrates the interaction between the model and the tools. It receives the model's instructions, executes the appropriate tools, collects their results, and provides those results back to the model. A simplified view is: User has goal -> Model determines the next action/Tool -> Agent executes the tool -> Tool produces a result -> Model evaluates the result This cycle continues until the model determines that it has enough information to provide the final answer. Do Agents Make Decisions? It is important to understand the distinction here. The agent is responsible for executing actions and tools, while the model provides the reasoning that determines which tool or action should be taken next. So, rather than thinking of the agent as an independent intelligence, it is useful to think of it as the code that takes actions towards a goal based on the model's guidance. Where Do Frameworks Come In? Frameworks such as LangChain, Google ADK, etc. provide abstractions that make it easier for developers to build applications that work with models, tools, and agents. Instead of implementing all the logic from scratch, developers can use framework components to connect models with tools and build agentic applications. Video For a visual explanation of Agents and Tools, watch the YouTube video below. This video is one of the lessons from my Udemy course, LangChain: Agentic AI and RAG Made Clear. Conclusion Models are powerful, but they don't automatically have access to real-time information or external capabilities. Tools provide additional capabilities, and the agent executes these tools based on the model's guidance. This model-tool-agent relationship is one of the fundamental building blocks for understanding Agentic AI.

Every few weeks, someone on my team, or in a client meeting, asks me the same question: "Which cloud should we use for our AI workloads?" I have been building enterprise integrations for over fourteen years now, and lately most of my time goes into RAG pipelines, vector databases, and agentic orchestration on top of these platforms. So I get this question a lot, and honestly, there is no single right answer. The right cloud depends on where your data already lives, what your compliance team will accept, and which models your architecture actually needs. In this article, I want to walk through the three big players, AWS Bedrock, Google Vertex AI, and Microsoft Azure AI Foundry, and share what I have learned working with these platforms in real enterprise settings, not just from reading marketing pages. AWS Bedrock Bedrock started as a model marketplace back in 2023, and it has grown into a full platform with Guardrails for content filtering, Knowledge Bases for RAG, and AgentCore for building agentic workflows. What I like most about Bedrock is the sheer breadth of models available behind a single API. You get Claude from Anthropic, Llama from Meta, Mistral, Cohere's Command models, and Amazon's own Nova family, all through one consistent interface. If your architecture needs to swap models without rewriting your integration layer, Bedrock makes that easier than the other two. Pros: Broadest model catalog of the three, so you are not locked into one vendor's models.Strong identity and governance story if you are already running on AWS, since it plugs directly into IAM, CloudTrail, and Macie.Bedrock is one of the few places where you get Claude with enterprise indemnification, which matters a lot when legal teams get involved.Provisioned throughput options give you predictable latency for production workloads that cannot tolerate spikes. Cons: If your organization is not already AWS-native, the onboarding curve is steeper than it looks.Cross-cloud portability is basically nonexistent. A model you fine-tune on Bedrock does not export cleanly to Vertex AI or Foundry. That is a real switching cost you should plan for on day one, not something to figure out later.Some of the newer agentic tooling is still maturing, so documentation gaps show up more than I would like. Google Vertex AI Vertex AI feels different from the other two because Google's DNA here is research first. If your team cares about multimodal capability, or you want access to Gemini models the moment they ship, Vertex AI tends to be ahead. It is also the strongest option if your data already lives in BigQuery, because the integration between Vertex and BigQuery for feature engineering and MLOps pipelines is genuinely smooth. Pros: Best fit for teams doing custom model training, not just calling a hosted API. AutoML and the broader MLOps tooling cut training time noticeably compared to the other two.Tight coupling with BigQuery is a huge advantage if your organization already runs its analytics there. You avoid a lot of data movement overhead.Gemini-first multimodal workflows, plus Google Search grounding for agents, which is something neither Bedrock nor Foundry offers natively.TPU support gives real throughput advantages for heavy batch processing. Cons: If your organization is not GCP-centric already, the value proposition weakens fast. You end up paying a data-gravity tax to move information into Google's ecosystem.Governance and compliance tooling, while solid, is not as battle-tested across regulated industries as AWS's certifications.The agent ecosystem, while improving, still trails Bedrock's AgentCore and Foundry's Azure AI Agents in terms of enterprise adoption stories I have personally seen. Azure AI Foundry Foundry, formerly Azure AI Services, is Microsoft's rebranded and expanded platform, and it is the one I have written about before because it is what my own recent client work has centered on. If your enterprise already lives inside Microsoft 365, Entra ID, and Azure infrastructure, Foundry removes almost all of the identity and governance friction you would otherwise deal with. That matters more than people expect once you are past the proof of concept stage and into actual production rollout with security review. Pros: Deep Microsoft 365 and Entra ID integration means your existing enterprise approvals and identity workflows extend naturally into your AI layer.Strong OpenAI-led model access, since Microsoft's partnership with OpenAI gives Foundry early and deep access to GPT-family models.Hybrid deployment options are genuinely better here than on the other two platforms, which matters if you have on-prem systems you are not ready to fully cloud-migrate.Roughly three-quarters of Fortune 500 companies already run on Microsoft's stack, so for a lot of enterprises Foundry is simply the path of least resistance. Cons: Model breadth is narrower than Bedrock's catalog, so if you need a specific non-OpenAI model family, you may find yourself stitching together a secondary platform anyway.Because it is tied so closely to Azure compute pricing, cost predictability requires more upfront modeling than teams expect.Some newer agentic and orchestration features are still catching up to what AWS has shipped with AgentCore. So Which One Should You Actually Pick? Here is the honest answer I give in client meetings: do not choose based on a benchmark screenshot or a features table. Choose based on where your data already lives and where your governance and compliance story already works. If you are AWS-first and want maximum model flexibility, go with Bedrock. If you are Microsoft-heavy and need your AI layer to inherit existing Entra ID and 365 approvals without a fight, Foundry is the path of least resistance. If your analytics already lives in BigQuery and multimodal Gemini capability is core to your roadmap, Vertex AI earns its place. What I am increasingly seeing among the teams I work with is a hybrid pattern. A primary cloud handles the bulk of regulated workloads, and a secondary cloud gets called in only when a specific model family is not well supported on the primary platform. It is not the cleanest architecture on paper, but it reflects how fast this space is still moving. None of these three platforms is standing still, and the leader on any given feature this quarter is not guaranteed to hold that spot by next year. My suggestion, whichever cloud you land on: build your RAG and orchestration layer with enough abstraction that swapping the underlying model provider is a configuration change, not a rewrite. That single decision will save you more pain than picking the "right" cloud ever will.

The first time I containerized a fine-tuned Llama model for a client's internal search tool, the build finished at 38 gigabytes. I remember staring at the terminal thinking there was no way that was right. It was right. The image included a CUDA base, PyTorch with every backend compiled in, model weights baked directly into the layer, and a pip cache that had not been cleaned. Pushing that to our registry took eleven minutes on a good connection. Pulling it onto a fresh node during an autoscale event took even longer, and by the time the pod was ready, the traffic spike it was supposed to handle had already passed. That's the moment I stopped treating LLM containers like regular application containers, because they are not the same animal at all. Why This Problem Actually Matters Most Docker advice out there is written for stateless web services, small images, fast cold starts, and horizontal scaling on demand. LLM workloads break almost every assumption baked into that advice. The artifact is huge, the runtime is GPU-bound, startup involves loading gigabytes into VRAM, and half your "application code" is actually a C++/CUDA binary blob you didn't write and can't easily trim. If you treat an inference container like a Flask app with a bigger base image, you end up with slow deploys, wasted GPU spend, and autoscaling that technically works but arrives too late to matter. The First Wrong Turn: One Image to Rule Them All Our early approach was a single monolithic image model with weights, tokenizer, inference server, and dependencies all baked together, rebuilt on every model version bump. It felt simple. It wasn't. Every retrain meant rebuilding a 30+ GB image even when the code hadn't changed a single line. Registry storage costs gradually increased until someone in finance questioned why our container registry bill resembled that of a second AWS account. Worse, rollbacks were painful because reverting to a previous model meant pulling an entire previous image rather than swapping a much smaller artifact. The solution that actually worked was separating the model weights from the serving image entirely. The image contains the runtime, the inference server (we used vLLM for most of our transformer workloads), and pinned dependencies. Weights live in object storage and are pulled at container start via an init container or a lazy loading entry point. The approach felt counterintuitive at first. Are we effectively transitioning the slower process to startup instead of build time? — but it turned out to be the right trade. Startup pulls are parallelizable, cacheable on the node, and don't bloat the registry. Build time dropped from twenty-plus minutes to under four. A Smaller Base Image Than You'd Expect This is where the challenges began. Everyone defaults to using nvidia/cuda:*-devel images because the framework documentation recommends them, but these devel images include the entire CUDA toolkit, which contains compilers that you will never use at runtime. Switching to the runtime variant and only installing the exact CUDA and cuDNN versions your framework's wheel actually needs cuts roughly 4GB off the base alone. A minimal multi-stage build looks something like this: Dockerfile FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 AS builder RUN pip install --no-cache-dir vllm==0.4.2 FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 COPY --from=builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 COPY --from=builder /usr/local/bin/python3.10 /usr/local/bin/ ENV MODEL_PATH=/mnt/models ENTRYPOINT ["python3", "-m", "vllm.entrypoints.api_server"] The build stage compiles anything that needs the full toolkit; the runtime stage only carries what's needed to execute. It's a basic Docker pattern, but I've seen it skipped constantly on ML teams because the assumption is always, "the model is the heavy part; the image doesn't matter." The model is heavy, sure, but a bloated base image adds real minutes to every autoscale event, and in production that's the difference between absorbing a traffic spike and dropping requests. The OOM Kill: Nobody Explained Well This is the war story I bring up most often. We had a container that ran fine locally and in staging, then got silently killed in production under load — no crash log, no stack trace, just a pod restart and a confused on-call engineer at 2 AM. It turned out to be the kernel OOM killer, not an application-level exception, because our memory limit accounted for the model weights in VRAM but excluded the growing KV cache for long-context requests plus the CPU-side tokenizer buffers. GPU memory and container memory limits are two completely separate accounting systems, and Kubernetes will happily kill your pod over host RAM even if your GPU has headroom to spare. The fix was unglamorous: we set explicit memory requests and limits with a real margin above peak KV cache usage, moved batch size and max sequence length into environment-configurable values instead of hardcoding them, and added a lightweight health assessment that reported GPU memory utilization alongside the standard liveness probe. None of that is exotic. All of it was missing because we'd copy-pasted a manifest template built for a stateless API and never revisited the resource math for a model that holds state in memory for the duration of a request. Where I'd Push Back on Common Advice A lot of guidance recommends one model per container for isolation, and for many teams that's right. But if you're serving several small fine-tunes of the same base model, that pattern wastes GPU memory by duplicating base weights across containers. We transitioned to a multi-adapter setup, where one base model is loaded once, and LoRA adapters are swapped for each request; this approach is more complex operationally but reduces the GPU footprint by nearly half. I wouldn't consider it a default; it represents a level of complexity that is justified only after demonstrating that plain per-model containers are indeed the bottleneck. I'd also push back on containerizing every workload the same way. Batch inference and real-time serving have almost opposite goals: one wants throughput and tolerates slow cold starts; the other needs rapid readiness and predictable latency. We split these into separate images with separate resource profiles, even though it meant more Dockerfiles. Fewer surprises beat fewer files. Key Takeaways Separate model weights from the serving image; bake them in the runtime and pull weights at startup from object storage.Use CUDA runtime images, not devel images, unless you genuinely compile something at container start.Account for GPU memory and host memory as two separate budgets; KV cache growth is the usual silent killer.Split batch and real-time serving into different images; their optimization goals are conflicting.Don't reach for multi-adapter serving or other density tricks until you've measured that plain per-model containers are actually the bottleneck. Closing Thought None of this required exotic tooling, no custom orchestrator, and no proprietary platform. It required treating the container as part of the model's runtime behavior rather than a packaging afterthought bolted on after the research work was done. The teams that struggle most with this approach usually aren't lacking Docker knowledge; they're applying web-service intuition to a workload that behaves nothing like a web service. If you're mid-migration on something similar, I'd genuinely ask: are you optimizing your image for build convenience or for what actually happens the moment traffic hits a cold node? Those answers are rarely the same, and figuring out which one you've been solving for is usually the first real fix.

It stopped being just a packaging tool the day our onboarding doc got shorter instead of longer. Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. That's the environment I inherited, and it's the reason I ended up rebuilding our entire local AI dev loop around Docker Compose instead of the notebook-and-prayer setup we'd been running. Why This Isn't Just a Packaging Problem The instinct on most teams is to treat Docker as something you reach for at deploy time. You write the model, get it working in a notebook, and only think about containers once it's time to ship. That instinct falls apart with AI workloads specifically because the dev-time dependencies are just as fragile as the prod ones. A GPU-backed embedding model, a local vector store, a retrieval service, and an orchestration layer all need to talk to each other during development, not just in production. If your local loop doesn't mirror that, you spend your debugging time chasing environment drift instead of chasing actual bugs. That was our exact situation, and it cost us roughly a day of onboarding per person plus a steady trickle of 'works for me' bug reports that turned out to be dependency version mismatches. The Setup We Rejected First Our initial response was to improve the Conda environment file and create a more detailed README. In hindsight, that was doomed from the start. Conda solved the Python dependency problem reasonably well but said nothing about the GPU driver version, the vector database binary, or the fact that two people were running Ollama locally with completely different default models pulled. We also floated the idea of just giving everyone a cloud dev environment with GPU access baked in. It solved the consistency problem, but the latency for interactive debugging was miserable, and the monthly bill for keeping GPU instances warm for a six-person team was not something I wanted to defend in a budget review. Neither approach addressed the real issue: we needed one definition of the environment that was runnable identically on a Mac laptop and a Linux workstation. What We Actually Built We moved the whole local AI stack into a single Compose file: an inference service running a small local model, a vector store, and the application layer, all networked together the same way they'd be networked in staging. Here's a trimmed version of what that looked like: YAML services: llm: image: ollama/ollama:latest volumes: ["ollama-data:/root/.ollama"] deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] vectordb: image: pgvector/pgvector:pg16 environment: POSTGRES_PASSWORD: devpass volumes: ["pgdata:/var/lib/postgresql/data"] app: build: ./app depends_on: [llm, vectordb] environment: OLLAMA_HOST: http://llm:11434 That file, plus a one-line 'docker compose up,' replaced two days of onboarding pain with about fifteen minutes. New hires no longer needed tribal knowledge about which conda channel had the right cuDNN build. It also resolved unforeseen bugs by ensuring everyone used the same version of the embedding model, eliminating reports of differing search results caused by dependency drift. The GPU Passthrough Headache Here's where things got tricky. GPU passthrough on Linux with the NVIDIA Container Toolkit is straightforward once it's configured, but it's not portable to Apple Silicon, and half our team was on M-series MacBooks. We ended up maintaining two Compose override files: one that requests GPU reservations for Linux workstations and one for Mac that falls back to CPU inference with a smaller quantized model, accepting slower generation for the sake of a working local loop. It's not elegant, and I still dislike maintaining two code paths for something as basic as "run the model," but the alternative was blocking half the team from working locally at all, which is worse. Where I'd Push Back on the Hype There's a growing narrative that Docker is quietly turning into a full AI platform with model registries, one-command local model pulls, and built-in GPU scheduling for dev. Some of that is genuinely useful, and I would rather not undersell it. But I'd push back on treating Docker as a replacement for a real experiment-tracking or model-serving platform in production. What it's good at is collapsing the dev-time chaos into something reproducible; it is not a substitute for proper GPU orchestration at scale, and teams that try to run Compose-style setups in production tend to relearn the lessons Kubernetes already solved, just slower and with worse observability. The platform shift is real at the development layer. I'm far more skeptical that it fully extends to production serving without a lot of additional tooling wrapped around it. Key Takeaways Treat local AI dev environments with the same seriousness as production ones. Dependency drift in embedding models and vector stores causes real, challenging-to-trace bugs.Conda and README discipline don't solve GPU driver and binary-level mismatches; a single Compose definition does.Plan for hardware heterogeneity early: GPU passthrough doesn't travel to Apple Silicon, so budget for a CPU fallback path.Don't overextend this pattern into production serving; Compose is a dev-loop win, not a Kubernetes replacement. Conclusion What changed for our team wasn't really about Docker getting new AI-specific features, though some of that helped. Realizing that the development environment for an AI application is as complex and failure-prone as production and treating it as an afterthought cost us real engineering hours each week. Whether Docker keeps expanding into model management and becomes a genuine AI platform, or whether that space gets carved up by more specialized tools, I think the underlying lesson holds either way: if your local AI loop isn't reproducible, nothing built on top of it will be either. I'm curious how far other teams have pushed this before Compose starts creaking. Is there a scale at which this pattern breaks down, or a project where you gave up and rebuilt around something heavier?

If different Docker Engine versions are running simultaneously in a Docker Swarm cluster, this may lead not to an obvious service outage but to a more subtle scenario: partial traffic degradation on individual nodes. In this case, the issue appeared on one of the manager nodes, Traefik started reporting an unavailable status (health=0) for the router-app service, and the cause, according to the working hypothesis, was related to differences in iptables rules and overlay networking between Docker 28.1.1 and 28.2.2. On June 22, 2025, this exact scenario occurred in the production cluster of the backend infrastructure for a socially significant public transportation mobile application. The system serves about 2 million users, several tens of thousands of daily active users, and a total load of around 1000–1600 RPS, so even partial degradation at a single entry point affected a high-load segment of traffic and could have had a noticeable impact on SLA metrics if it had not been localized in time. Context At the time of the incident, the Docker Swarm cluster consisted of 5 manager nodes, several dozen worker nodes, and approximately 40–50 services. External HTTP traffic passed through Traefik, deployed on each of the five manager nodes, and was then routed by Traefik to the backend application containers. One of the key services was router-app, responsible for building public transportation routes on the frontend. It was one of the critical entry-point services with the highest SLA, and any disruption to its availability could have led to severe penalties from the customer, so any deviation in its availability required an immediate response. Grafana dashboards showed application availability through Traefik health-check statuses. Those statuses were generated based on HTTP health-check endpoints implemented by the developers for most services, primarily the most critical ones. This was enough to quickly localize the problem at the ingress traffic level. The cluster had one important characteristic. Some nodes were running Ubuntu 20.04 (focal), while others were running Ubuntu 22.04 (jammy), and different APT repositories were pulling different Docker Engine versions. As a result, after another scheduled Docker update on the nodes, the production environment ended up with a mix of nodes running 28.1.1 and 28.2.2 at the same time. The docker node ls screenshot additionally confirmed that mixed versions were present not only on worker nodes but also on manager nodes, including swarm4 and swarm5. How the Incident Manifested The incident was detected not through user complaints and not through a general service outage, but through Traefik monitoring. The triggered alerts showed partial unavailability of one of the router-app containers, after which the health-check dashboard confirmed that the issue affected not the entire service but one of the manager nodes. This is important because the red blocks on the dashboard did not indicate a complete outage of router-app. It meant that Traefik on one of the manager nodes started receiving health=0 when checking that service’s backend endpoint, while the other manager nodes continued to see the backend as healthy. In practice, it looked like this: traffic through one of the manager nodes stopped reaching the router-app containers correctly, but Traefik, running on all five manager nodes, automatically excluded requests to the unhealthy entry point. As a result, from the outside the incident appeared as partial degradation rather than full unavailability. That is exactly what made the situation tricky. Fault tolerance limited the impact of the incident, but the underlying cause remained inside the cluster and continued to affect one of the entry points. What Docker Showed After localizing the issue to one of the manager nodes, it became clear that the cause should be sought not in router-app itself but in the network path between Traefik and the backend containers. At the same time, docker service ps did not show a widespread service failure, and the containers still appeared as running. The next useful signal came from the dockerd logs on swarm5. Repeated messages appeared there, including Peer delete operation failed, neighbor entry not found, and errors related to deleting FDB and neighbor entries for the VXLAN interface vx-001001-5lk08. For example: Plain Text Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.366807202Z" level=warning msg="Peer delete operation failed" error="could not delete fdb entry for nid:5lk08r7jjvtq5idqggzeygmlv eid:4e7d63d00fffaa6be7ce6362f47acd7912f7c11e5ac6e018393722decc16c210 into the sandbox:neighbor entry not found for IP 10.170.0.37, mac 02:42:0a:1b:14:2c, link vx-001001-5lk08" Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.765092537Z" level=warning msg="error deleting neighbor entry" error="no such file or directory" ifc=vx-001001-5lk08 ip=10.170.0.138 mac="02:42:0a:1b:14:65" Such messages were highly consistent with problems in Docker Swarm’s overlay network. In essence, Docker was trying to delete network records that were no longer present in the tables, which usually points to desynchronization of network state at the VXLAN, FDB, or neighbor-table level. By themselves, these messages still did not provide a complete explanation, but they pushed the investigation in the right direction. It became clear that the problem was not in the application’s business logic but in the network layer on one of the nodes. Additionally, docker node inspect self --pretty on swarm5 showed that from Swarm’s point of view the node looked normal: State: Ready, Availability: Active, Raft Status: Reachable, Leader: No, while Engine Version was already 28.2.2. This was an important point: the control plane still considered the node healthy, even though at the traffic-flow and network-state level it was already behaving differently. Diagnostics The investigation was carried out at the node level. The tools used included docker node ls, docker version, apt-cache policy docker-ce, as well as ip link show, bridge fdb show, ip neigh show, and comparisons of iptables chains across different nodes. The key fact became visible after docker node ls. The cluster was not homogeneous in terms of Docker Engine version: some manager nodes and some worker nodes were already running 28.2.2, while the others remained on 28.1.1. This led to the assumption that the issue might be at the iptables rules level. After that, iptables had to be compared separately on healthy and problematic nodes. On swarm5, a full rules dump was collected using the combination of iptables -S, iptables -t nat -S, and iptables -t mangle -S. Those rules showed the DOCKER, DOCKER-FORWARD, DOCKER-INGRESS, and DOCKER-USER chains, as well as ACCEPT, DROP, and DNAT rules for traffic through docker_gwbridge, published ports, and ingress routing.... To test the hypothesis, not only the problematic swarm5 but also the first manager node, swarm1, was compared, where a stable stack with Docker 28.1.1 had long been running. On swarm1, the output of iptables -S and iptables -t nat -S showed the expected picture: the DOCKER and DOCKER-INGRESS chains contained a full set of ACCEPT and DNAT rules for all published ports (80, 8080–8082, and dozens of internal service ports) with symmetric dport/sport pairs, while DOCKER-USER effectively boiled down to a clean RETURN. Taken together with the dump from swarm5, this reinforced the conclusion that on nodes running 28.1.1, the iptables configuration for ingress and routing was consistent, and the differences seen on 28.2.2 were related not to manual changes but to the behavior of Docker Engine itself. After that, iptables had to be compared separately on other nodes running different Docker versions. On nodes with 28.1.1, the DOCKER and DOCKER-USER chains and the associated rules were in the expected state, whereas on nodes with 28.2.2 some of the required rules were missing or the chains were reduced to a minimal RETURN. This explained the observed behavior well. The services remained running, Swarm did not appear broken, but external traffic and part of the overlay routing through a specific manager node were working incorrectly, causing Traefik on that node to report health=0 for router-app. It is worth noting separately that journalctl -u docker and docker service ps did not provide a simple direct cause for the incident. They did not show a picture of a general failure, so the conclusion had to be assembled from several sources: Traefik monitoring, dockerd logs, Docker versions, and the state of iptables on different nodes. Fix Once the main hypothesis had narrowed down to mismatched Docker Engine versions, the solution was fairly straightforward: return the cluster to a homogeneous configuration by rolling back to version 28.1.1 as the fastest solution. The rollback was performed for swarm4, swarm5, and all worker nodes where version 28.2.2 had already been installed. To do this, a specific package version was pinned via apt, then Docker was restarted, and the installed version was verified. One version of the commands looked like this: Shell apt-cache madison docker-ce | grep 28.1.1 apt-get install docker-ce=5:28.1.1-1~ubuntu.22.04~jammy \ docker-ce-cli=5:28.1.1-1~ubuntu.22.04~jammy \ containerd.io && systemctl restart docker && docker --version Additionally, it made sense to check the package sources and, if necessary, remove conflicting APT entries so that the nodes would no longer receive an unsuitable Docker version from another repository. In practice, it looked like this: Shell sudo rm /etc/apt/sources.list.d/download_docker_com_linux_ubuntu.list sudo apt update After the rollback, docker version was checked again, as well as the DOCKER and DOCKER-USER chains. After Docker Engine had been unified to 28.1.1 on both manager and worker nodes, the issue disappeared. From the perspective of external behavior, this was confirmed immediately. Health checks in Traefik returned to the green zone, and the partial unavailability of router-app on one of the manager nodes could no longer be reproduced. Root Cause Based on the available data, the most well-founded working version is this: in this environment, Docker Engine 28.2.2 formed or applied iptables rules related to DOCKER, DOCKER-USER, FORWARD, ingress, and overlay networking differently. In a mixed cluster, this led to one of the manager nodes no longer forwarding traffic correctly to the router-app backend containers, even though from the perspective of the control plane and service state this did not look like a direct failure. It is important here not to overstate what the data allows. This case does not prove a universal upstream bug in Docker 28.2.2 for all Swarm installations, but it does show that even closely related Docker Engine versions can affect the cluster’s network plane differently, especially when different Ubuntu distributions and different package sources are present in production at the same time. What Follows From This The first conclusion is simple: Docker Swarm is sensitive to Docker Engine version mismatches. If some manager or worker nodes have been updated while others have not, this can lead not only to version drift as an organizational problem, but also to practical issues with traffic, published ports, and overlay routing. The second conclusion is that after updating Docker, it is necessary to check not only docker version but also the node’s network behavior. The minimum set includes iptables -L DOCKER -v -n, iptables -L DOCKER-USER -v -n, checking published ports, ingress/overlay state, and health checks from the edge proxy. The third conclusion is that it is useful to maintain a single baseline stack across all nodes. One Ubuntu LTS distribution, unified repositories, and the same update order reduce the chance that cluster state will remain formally healthy while part of the network traffic is already being handled incorrectly. The fourth conclusion concerns update order. In our case, that was exactly how it happened, but it is worth noting separately. When Docker is updated in a Swarm cluster, it is better to update worker nodes first, then manager nodes, and the leader last, while after each stage separately checking the node’s behavior under real traffic conditions (service availability, correct routing, and published ports). In our case, enhanced monitoring was in place, so no additional manual checks of node behavior in traffic were required: if any part of the infrastructure became unavailable, we would promptly receive an alert. The final conclusion relates to monitoring. In this case, Traefik not only helped limit the impact of the incident by routing around the unhealthy node, but also provided the first precise signal that the problem was localized to a specific entry point rather than existing at the level of the entire service or the entire cluster.

In a previous article, we built a static supply chain graph in Neo4j using Apache Spark, with suppliers, warehouses, distribution centers, and retailers connected by shipping routes. That gave us a snapshot of the network at a point in time. In this article, we'll add the streaming layer: shipment events flow through Confluent Cloud Kafka in real time, land in Neo4j as enriched graph properties, and a live dashboard shows network health updating as events arrive. The full source code is available on GitHub. The Stack Each tool in the stack does what it does best: ToolRoleConfluent Cloud (free tier)Managed Kafka cluster and topicPython producer (Jupyter)Generates and publishes synthetic shipment eventsPython consumer (Jupyter)Consumes events and writes them into Neo4jNeo4j AuraDBGraph database storing the supply chain and shipment eventsPlotlyLive dashboard visualization One deliberate omission is that we aren't using the Neo4j Kafka Sink Connector, which is available as a managed connector on Confluent Cloud. That connector handles the consumer side automatically but carries a per-task hourly charge. For this article, we'll keep everything free by writing a Python consumer that does the same job. This also has a practical benefit: all the pipeline logic is visible in Python rather than hidden inside a managed connector configuration, which makes it easier to understand and adapt. The managed connector is a natural next step for production workloads. Setting Up Confluent Cloud Sign up at confluent.io and create a free cluster.Once the cluster is running, create a topic named shipment-events with 1 partition and default settings.Create an API key and secret under API Keys.Note the bootstrap server address from the cluster settings. Export these as environment variables in your shell: Shell export CONFLUENT_BOOTSTRAP_SERVERS=your_cluster.confluent.cloud:9092 export CONFLUENT_API_KEY=your_api_key export CONFLUENT_API_SECRET=your_api_secret Setting Up Neo4j AuraDB AuraDB is Neo4j's fully managed cloud database. A free tier is available with no credit card required. Sign up at console.neo4j.io/graphacademy.Create a new AuraDB Free instance.When the instance is created, download or note the credentials — the connection URI, username, and password. Neo4j only shows the password once, so save it somewhere safe.Once the instance is running, open the built-in Query tab and verify connectivity: MATCH (n) RETURN count(n). This should return 0. We are ready to load data. Before starting Jupyter, export the connection details as environment variables in your shell: Shell export NEO4J_URI=neo4j+s://xxxx.databases.neo4j.io export NEO4J_USERNAME=your_username_here export NEO4J_PASSWORD=your_password_here export NEO4J_DATABASE=your_database_name_here The Data Model Each shipment event represents a single status update for a shipment at a point in time. A shipment does not generate a sequence of events as it progresses — each event is an independent snapshot, which keeps the producer simple and the consumer stateless. The event structure is: JSON { "shipment_id": "c60eb761-f153-4840-8427-17fa9e34c56c", "supplier_id": "S013", "warehouse_id": "W005", "dist_center_id": "DC004", "retailer_id": "R025", "status": "delayed", "timestamp": "2026-08-04T12:57:15Z", "delay_minutes": 34 } Status follows one of four values — departed, in_transit, delayed or delivered, with a configurable delay probability. We use 15% delayed to make the dashboard interesting without overwhelming it. When the consumer writes an event into Neo4j, it creates a Shipment node and links it to the existing supply chain nodes via four relationship types: Cypher MERGE (sh:Shipment {shipment_id: $shipment_id}) SET sh.status = $status, sh.timestamp = $timestamp, sh.delay_minutes = $delay_minutes WITH sh MATCH (s:Supplier {id: $supplier_id}) MATCH (w:Warehouse {id: $warehouse_id}) MATCH (dc:DistributionCenter {id: $dist_center_id}) MATCH (r:Retailer {id: $retailer_id}) MERGE (s)-[:HAS_SHIPMENT]->(sh) MERGE (sh)-[:VIA_WAREHOUSE]->(w) MERGE (sh)-[:VIA_DIST_CENTER]->(dc) MERGE (sh)-[:DESTINED_FOR]->(r) MERGE on shipment_id means re-running the consumer never creates duplicate nodes. The Producer The producer notebook uses a fixed random seed to generate reproducible shipment events using IDs drawn from the existing supply chain and publishes them to Confluent Cloud via the confluent-kafka library: Python producer = Producer({ "bootstrap.servers": BOOTSTRAP_SERVERS, "security.protocol": "SASL_SSL", "sasl.mechanisms": "PLAIN", "sasl.username": API_KEY, "sasl.password": API_SECRET, "log_level": 0, }) Setting "log_level": 0 suppresses the librdkafka telemetry messages that appear otherwise. The producer supports both batch and continuous modes. For example: Python produce_events(num_events = -1) # stream continuously produce_events(num_events = 100) # publish exactly 100 events The display refreshes every PRINT_EVERY events using clear_output, showing the latest event and a running status breakdown — so the cell output stays manageable even when streaming thousands of events. The Consumer and Live Dashboard Rather than two separate notebooks, we combine the consumer and dashboard into a single pipeline. On each cycle, the loop: Polls Kafka for up to POLL_BATCH events and writes them to Neo4jQueries Neo4j for the current graph stateRebuilds and redraws the dashboardSleeps for REFRESH_INTERVAL seconds before repeating Rebuilding the full dashboard on every cycle is straightforward and works well at demo event rates. At higher throughput, a more efficient approach would be to update only the changed data rather than redrawing all eight panels on each refresh. The consumer uses its own Kafka group ID (supply-chain-dashboard) so it reads the topic independently, catching up on all existing events first before staying live: Python consumer = Consumer({ "bootstrap.servers": BOOTSTRAP_SERVERS, "security.protocol": "SASL_SSL", "sasl.mechanisms": "PLAIN", "sasl.username": API_KEY, "sasl.password": API_SECRET, "group.id": "supply-chain-dashboard", "auto.offset.reset": "earliest", "log_level": 0, }) The Live Dashboard The dashboard uses Plotly's make_subplots in a 4x2 grid, rebuilt on every refresh cycle using clear_output. Eight panels give a complete picture of network health: Row 1 – Overall Health Network status table: Total shipments, delayed count, delay rate, Kafka events consumed, refresh count, and any disabled nodesShipment status distribution: Donut chart showing the split between departed, in transit, delayed, and delivered, as shown in Figure 1 Figure 1. Shipment Status Distribution Row 2 – Warehouse View Delayed shipments by warehouse: Which warehouses are handling the most delayed shipments right nowWarehouse health score: A heatmap scoring each warehouse from 0.0 (everything delayed) to 1.0 (fully healthy), colored red through orange to green, as shown in Figure 2 Figure 2. Warehouse Health Score Row 3 – Origin and Destination Supplier performance: Which suppliers are generating the most delayed shipmentsRetailer impact: Which retailers are receiving the most delayed shipments — the downstream effect of any disruption Row 4 – Mid-Network and Flow Average delay by distribution center: Where in the middle layer delays are accumulatingShipment flow: A Sankey diagram (Figure 3) showing which suppliers are routing through which warehouses Figure 3. Shipment Flow - Suppliers to Warehouses The warehouse health score is the most immediately readable panel. The Cypher behind it computes the score directly in the graph: Cypher MATCH (sh:Shipment)-[:VIA_WAREHOUSE]->(w:Warehouse) WHERE w.active IS NULL OR w.active <> false WITH w.id AS warehouse, count(sh) AS total, count(CASE WHEN sh.status = 'delayed' THEN 1 END) AS delayed RETURN warehouse, round(1.0 - toFloat(delayed) / total, 3) AS health_score ORDER BY warehouse Simulating a Network Disruption One of the more compelling features of the graph model is how easy it is to simulate and visualize a disruption. Setting active = false on any node excludes it from the dashboard queries and the dashboard immediately reflects the simulated disruption on the next refresh cycle. We can do this before the dashboard starts: Python REMOVE_NODE = "W007" # mark this warehouse as inactive Or live, while the dashboard is running, using the Neo4j AuraDB Query tab: Cypher // Disable a node MATCH (n {id: "W007"}) SET n.active = false // Re-enable a node MATCH (n {id: "W007"}) REMOVE n.active // Check what is currently disabled MATCH (n) WHERE n.active = false RETURN labels(n)[0] AS label, n.id AS id Within 5 seconds, the dashboard reflects the change. The warehouse health heatmap shows the gap, the delayed shipments bar shifts to other warehouses as traffic reroutes, and the network status table shows the node as disabled. Re-enabling it and watching the metrics recover completes the disruption and recovery story. Standalone Operation At startup, the consumer notebook creates the supply chain nodes using MERGE. This operation is idempotent, so any existing nodes from the previous article are left unchanged. Note that this step creates nodes only — the relationships between supply chain nodes (supplier -> warehouse -> distribution center -> retailer) are assumed to exist from the previous article, or can be added separately if running this notebook in isolation. Python with driver.session(database = NEO4J_DATABASE) as session: for i in range(20): session.run("MERGE (:Supplier {id: $id})", id = f"S{i:03d}") for i in range(12): session.run("MERGE (:Warehouse {id: $id})", id = f"W{i:03d}") for i in range(10): session.run("MERGE (:DistributionCenter {id: $id})", id = f"DC{i:03d}") for i in range(30): session.run("MERGE (:Retailer {id: $id})", id = f"R{i:03d}") Gotchas and Lessons Learned Suppress librdkafka Logging Without "log_level": 0 in the producer and consumer config, Confluent's underlying librdkafka library prints telemetry messages to the cell output every time a connection is established. The messages are harmless. Suppress Neo4j Property Warnings Querying a property that does not yet exist on any node produces a GqlStatusObject warning from Neo4j for every query that references it. The active property falls into this category when no node has been disabled. The fix is one line to set notifications to "OFF" on the driver, as follows: Python driver = GraphDatabase.driver( NEO4J_URI, auth = (NEO4J_USERNAME, NEO4J_PASSWORD), notifications_min_severity = "OFF", ) Consumer Group Isolation Kafka distributes partitions across consumers in the same group, so each consumer processes only its assigned partitions. If we run multiple consumers using the same group ID against the same topic, each will only process a subset of the events. The dashboard uses supply-chain-dashboard as its group ID, and the tip is to run only one instance of this notebook at a time against the same topic and cluster. auto.offset.reset = earliest Without this setting, a consumer that starts after events have been published will miss everything that arrived before it connected. Setting earliest means the consumer always catches up on the full history of the topic before going live, which is essential if we stop and restart the dashboard mid-session. Clear Shipment Nodes Between Runs Each run of the consumer creates new Shipment nodes. Since the producer generates synthetic demo data, it's safe to clear these between runs; otherwise, successive runs would accumulate all historical shipments, and the dashboard counts would grow unbounded. The notebook clears all Shipment nodes at startup: Cypher MATCH (sh:Shipment) CALL (sh) { DETACH DELETE sh } IN TRANSACTIONS OF 10000 ROWS Summary We've built a real-time supply chain event streaming pipeline using Confluent Cloud Kafka and Neo4j. The producer generates synthetic shipment events continuously, the consumer writes them into the graph, and a live dashboard shows network health updating in near real-time. The disruption simulation — marking a node inactive mid-run and watching the dashboard respond — demonstrates one of the most compelling aspects of the graph model: the ability to ask structural questions about a network as it evolves. The same architecture adapts naturally to real logistics, IoT, or manufacturing event streams where understanding network structure matters as much as raw throughput. The full source code is available on GitHub.

By Akmal Chaudhri CORE

For years, Arm64 was the platform people talked about as a future bet. It was useful in embedded systems, interesting in research, and easy to dismiss as “not the main thing.” That era is over. In a conversation between Dave Neary, Director of Developer Relations at Ampere Computing, and Greg Kroah-Hartman, Linux stable kernel maintainer and long-time kernel developer, the message is clear: Arm64 has become mainstream. It is no longer a special-case architecture. It is a first-class platform in Linux development, deployment, and maintenance. Arm64 Has Become a First-Class Platform in Linux Development Kroah-Hartman’s history with Linux goes back to the late 1990s, when his work in embedded systems led him into kernel development. He started by solving practical device problems, such as getting USB hardware working across many systems. That hands-on work turned into a career built around making Linux more reliable, more portable, and more useful across different hardware. One of the biggest changes he describes is how the Linux community matured. Early on, Linux developers often borrowed ideas from Unix, BSD, and Windows. The goal was to make things function. Over time, Linux moved from catching up to leading. Once that happened, the work became harder. Developers were no longer copying proven models; they were building new infrastructure, new interfaces, and new processes that had to work at scale. That shift also explains why the stable kernel process matters so much. In 2005, Linux moved toward time-based releases and created a stable kernel series focused only on bug fixes. That decision made it possible to keep improving Linux without breaking user space or workloads. For developers, that means a reliable update path. For users, it means confidence that the system will continue to work. Arm64’s growth has made that stability even more important. Today, Arm64 is everywhere: phones, laptops, embedded systems, cloud servers, appliances, and high-performance computing. Linux now runs across all of it. That breadth has changed the ecosystem. When Arm64 breaks, the impact is no longer small. It affects real products and real users across the industry. Upstream Development Improves Arm64 Linux Reliability and Maintainability Kroah-Hartman also highlighted the role of upstream development. The Linux community has long encouraged vendors to work directly on the mainline kernel rather than maintain private patches. That approach saves time, reduces long-term cost, and improves quality. Some vendors learned this the hard way. Others embraced it early and benefited from tighter collaboration with the community. Native Arm64 Testing Gives Kernel Developers Faster Feedback A major practical change for Kroah-Hartman came from using a native Arm64 build server from Ampere. Before that, he mostly tested on x86 and only discovered Arm64 issues later. Now he can build and test Arm64 kernels locally before sending patches out for review. That means fewer mistakes, faster feedback, and less wasted time for everyone involved. The value of that setup is simple: it matches the reality of modern development. Arm64 is no longer a side project. It is part of the core infrastructure of Linux. Native Arm64 tools help developers build better software for the platforms where Linux actually runs. For the Arm64 community, the lesson is direct. Mainstream status brings responsibility. It also brings leverage. The more Arm64 developers work upstream, test locally, and focus on reliability, the stronger the ecosystem becomes. View the full video here: To learn more about Ampere’s developer efforts and find best practices, visit Ampere’s Developer Center and join the conversation in the Ampere Developer Community. Check out the full Ampere article collection here.

Senior data engineers are trained to be skeptical of proprietary platforms. When I entered a Palantir Foundry training bootcamp, I expected to find a slow, expensive alternative to the mature tools I know on AWS and Azure. What I found instead was a platform built for a radically different user, one who cannot write SQL but needs answers now. I want to write about what I actually observed honestly, including where I think the hype is justified and where I think it is not, because most Foundry content I have seen is either from Palantir's own marketing or from practitioners so embedded in the platform they have forgotten what it was like to come to it fresh. I am writing this while that perspective is still clear. The Speed Thing Is Real The surprise that hit me hardest was not a feature. It was pace. During the bootcamp we worked across a range of tasks: connecting data sources, building transformation pipelines, setting up workflows that business users could interact with directly. To make this concrete: building a pipeline that ingested data from multiple sources, applied transformations, and exposed the output to business users took only hours in Foundry. On a standard AWS or Snowflake stack with dbt and an orchestration layer, a comparable setup typically runs to a full sprint for a small team, not because of any single hard step, but because of the coordination overhead between tools. I want to be careful about what I am and am not claiming here. This was a structured training environment with guided examples, not production infrastructure with real enterprise complexity and legacy constraints. The comparison is not controlled. But the direction of the difference was clear enough that I took notice. Foundry's Pipeline Builder abstracts away a lot of the coordination work that consumes time in a more assembled stack. Whether that advantage holds at full scale is a question I cannot answer from a single bootcamp, but it is worth asking seriously. The honest counter-argument: speed in a training environment does not always translate to speed in production. A well-resourced engineering team that already knows Snowflake deeply can move fast too, without the overhead of learning a new paradigm. If your team is highly capable on your current stack, the productivity gain from switching may not justify the learning curve cost. "Tasks I would have planned for a full day on my normal stack were done in a couple of hours." Who Actually Benefits Most However, raw speed is not the platform's most disruptive feature. The more I used it, the more I realized that the real value of that speed is not for engineers. It is for the people who are usually waiting on us. The more I worked with Foundry during the training, the clearer it became that the people getting the most out of it in the room were not the engineers. They were the non-technical participants, the analysts, the operations people, the business users who in a traditional stack would be waiting for an engineer to build them something before they could interact with data at all. Foundry's ontology model, the way it creates a shared semantic layer that different types of users can navigate without writing code, is differentiated from what I work with on AWS, Azure, and Snowflake. On those platforms, self-service data access for non-engineers is possible, but it takes deliberate, often significant engineering effort to expose data in a way that non-technical people can actually use. In Foundry, it felt closer to the default. If I were advising an organization on whether to consider Foundry, the first question I would ask is: what percentage of the people who need to interact with your data can actually write SQL? In organizations where more than half of business analysts and operational users cannot write code, the engineering burden of building self-service access on a traditional stack becomes a recurring, compounding cost. That is the environment where Foundry's default self-service capabilities start to justify serious evaluation. The counter-argument here is worth stating directly: a strong, well-resourced data engineering team could build a better, more tailored self-service layer on Snowflake in the same time it takes to master Foundry's ontology. If your organization has that team and the patience to build the right abstractions, the open platform may serve you better in the long run. Foundry's self-service advantage is most compelling when you do not have that engineering capacity, or when the number of non-technical users is large enough that a custom-built solution would require constant maintenance. The Cost Reality Palantir does not publish list pricing for Foundry. Everything is negotiated. The platform uses a core-based licensing model, meaning you pay based on the computational capacity (server cores) allocated to the platform rather than by the number of users. Based on publicly available government procurement records, core-based licenses start at roughly 66,000 pounds per server core per year, with no additional per-user fees on top. Solution-based use case licenses, which bundle implementation and support, start at 250,000 pounds at entry level and scale significantly from there depending on data complexity, user base, and operational scope. What this means practically is that Foundry's cost is not a fixed number you can evaluate on a spreadsheet. It is a negotiation. According to procurement advisory analysis of Palantir Foundry negotiations conducted between 2024 and 2025, annual platform fees for comparable mid-size deployments varied by a factor of two to three depending purely on negotiation posture (Redress Compliance, 2025). The leverage comes primarily from having a credible, costed alternative, which for most organizations means Databricks or Snowflake with named engineering owners and a realistic build timeline. Organizations that enter Palantir conversations without that alternative built tend to pay significantly more for the same deployment than organizations that do. "The leverage in the Foundry cost negotiation comes primarily from having a credible, costed alternative built before you walk in." My honest assessment after the bootcamp is that the cost is hard to justify for smaller organizations or simpler use cases. If a well-designed Snowflake environment can meet your data engineering needs with dbt and a standard BI layer on top, Foundry is probably not the right answer, and the delta in platform cost will buy you a lot of engineering time on the stack you already know. The calculus changes for large enterprises with complex, multi-team data environments and a significant population of non-technical users who need meaningful data access. What I Would Tell a Data Engineering Leader A few things I would want another senior data engineer or engineering leader to know before evaluating Foundry: Do not evaluate Foundry on pipeline performance alone. That is not its primary differentiator. Compare it to Snowflake or Databricks on what it does for the non-engineer users in your organization, not on compute efficiency.Build your alternative cost model first. Whatever your current stack is, cost out what it would take to build the data product capabilities Foundry promises on that stack, with your own team. That number is your negotiating anchor.Take the learning curve seriously. Foundry has a broad ecosystem: the ontology model, Pipeline Builder, Code Repositories, AI integrations, and coming to it fresh from a traditional data engineering background takes real adjustment. The training helped, but it is not a platform you pick up in a day.Be specific about who your users are. Foundry earns its cost fastest in environments where non-technical users need to do more with data than your current stack allows. If your users are primarily technical, the value proposition narrows considerably.Negotiate the second contract inside the first. Procurement analysis consistently shows that organizations that lock in phase two pricing before signing the initial contract pay significantly less per added use case than those who do not. Treat the pilot as the deal. The Honest Summary I came to Palantir Foundry expecting to be underwhelmed. I was not. But understanding its value requires a paradigm shift for any engineer raised on AWS or Snowflake. Evaluate Foundry not as a faster pipeline tool, but as a platform for organizational data literacy. For enterprises drowning in data but starved of accessible insights, it is a compelling, if expensive, contender. For everyone else, the tools you already have remain the better investment. The challenge is being honest enough with yourself to know which bucket your organization falls into.

Introduction The meeting that changed the platform team's direction was not a technical one. It was a conversation with a product engineer who had been at the company for eight months and had never successfully deployed to production without help from someone on the platform team. Not because she lacked skill. She was smart, experienced, and had successfully launched production systems at two previous jobs, but getting a working service into production meant dealing with fifteen different configuration files across four repositories, figuring out how Helm values files and Kustomize overlays worked together, and knowing which of the three CI pipeline templates to use based on whether the service needed a sidecar, a job scheduler, or neither. She had read the documentation. It was accurate. It just didn't tell her what to do when the documented path didn't match the state of her specific service in her environment. The platform team had built powerful infrastructure. They had not built a usable platform. That distinction between infrastructure and platform is where most Kubernetes-based internal developer platform efforts go wrong, and it's worth being precise about what it means. Infrastructure vs Platform: A Practical Distinction Infrastructure is the machinery: the Kubernetes clusters, the networking layer, the CI pipelines, the secrets management system, and the monitoring stack. A platform is the interface that makes that machinery accessible to developers who aren't Kubernetes experts without requiring them to become ones. The confusion between the two produces a situation that's extremely common in engineering organizations: a technically sophisticated infrastructure that's effectively only usable by the people who built it. The test for whether you have a platform or just infrastructure is simple: can a developer who joined three months ago deploy a new service to production without asking anyone for help? Not by following a tutorial someone wrote last year that may or may not still be accurate, but through tooling that guides them through a current, correct process. If the answer is no, you have infrastructure. The platform is the missing layer. This statement is not an argument against complexity in the underlying system. Kubernetes is complex, and that complexity exists for beneficial reasons: flexibility, programmability, and a rich ecosystem. The platform layer should absorb the complexity, rather than exposing it to every developer who needs to ship a service. What the First Attempt Got Wrong The infrastructure team built the first version of the internal platform in their spare time, juggling it with other priorities. It consisted of a set of Helm chart templates, a GitHub Actions workflow library, and a wiki with deployment instructions. This approach is how most internal platforms start, and it has a predictable failure mode: the templates encode the assumptions of the people who wrote them, the wiki goes stale within weeks, and the gap between the documented process and the actual state of the infrastructure grows invisibly until it becomes a significant tax on every developer who hits it. The fundamental mistake was treating platform work as documentation work rather than product work. A wiki is not a platform. A set of templates that require understanding to use correctly is not a platform. A platform is software that makes the correct path the easy path, that validates inputs before they cause problems downstream, and that fails loudly and helpfully rather than silently and mysteriously. The second attempt started from a different premise: the platform is a product, developers are its users, and the measure of success is whether they can do their jobs without needing the platform team. The Abstraction Layer: Custom Resources and Admission Webhooks The technical decision that made the most difference was introducing a custom resource definition (CRD) that represented a service in the platform's domain model, not a Kubernetes Deployment or Service, but a higher-level construct that encoded the platform's opinionated defaults and generated the underlying Kubernetes objects from a much simpler specification. YAML # Platform-level CRD: what developers actually write apiVersion: platform.company.com/v1 kind: AppService metadata: name: payment-api namespace: production spec: image: payment-api:v1.4.2 tier: backend # drives resource limits, network policy replicas: 3 port: 8080 healthCheck: /healthz env: DATABASE_URL: secretRef: payment-db-credentials This twelve-line manifest replaced the hundred-plus lines of Kubernetes YAML that developers had previously been required to write and maintain. The controller running in the cluster, a standard Kubernetes operator built with controller-runtime, read the AppService resource and generated the Deployment, Service, HorizontalPodAutoscaler, PodDisruptionBudget, and NetworkPolicy that the platform's standards required, with defaults applied consistently across every service. The key design decision was what to expose in the CRD and what to hide. The tier field is a prime example: rather than exposing resource requests and limits directly, which requires understanding what values are appropriate for the cluster, the CRD accepts a tier label (frontend, backend, worker, batch) that maps to a predefined resource profile. A backend tier service receives a specific CPU and memory allocation appropriate for the cluster's node types. A batch tier service receives a different profile with different eviction priorities. The developer specifies intent; the platform enforces the appropriate configuration. Go # Controller logic: tier maps to resource profile (Go pseudocode) func resourceProfileForTier(tier string) corev1.ResourceRequirements { profiles := map[string]corev1.ResourceRequirements{ "frontend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("100m"), corev1.ResourceMemory: resource.MustParse("128Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("500m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, }, "backend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("250m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("1000m"), corev1.ResourceMemory: resource.MustParse("512Mi"), }, }, // batch, worker profiles follow same pattern } return profiles[tier] } Admission webhooks complemented the CRD by catching misconfiguration before it reached the cluster. A validating webhook checked every AppService manifest against a set of rules: the image tag must not be 'latest,' the health verification path must respond within the cluster, and secret references must exist in the target namespace and return a clear error message describing exactly what was wrong and how to resolve it. This shifted error detection from 'runtime, after deployment' to 'submission time, before anything breaks,' which dramatically reduced the debugging load on both developers and the platform team. The Golden Path and Its Limits The CRD and controller approach works well when services fit the platform's model. Here's where things became challenging: not all services fit the platform's model. A service that needed custom init containers, a service that required a specific affinity rule because of a hardware dependency, and a batch job with a complex retry policy that didn't map cleanly to the tier abstraction. Each of these required either extending the CRD or breaking the abstraction and falling back to raw Kubernetes YAML. The temptation is to keep extending the CRD to cover every case. Resist it. A CRD that tries to expose every Kubernetes feature is just a more complicated way to write Kubernetes YAML, and it loses the simplicity that made the abstraction valuable. The better model is a golden path, the CRD for the 80% of services that fit the standard model, and a documented escape hatch for the 20% that don't. The escape hatch is raw Kubernetes resources, maintained by the teams that need them, with the platform team providing support rather than ownership. The key is being honest with developers about which path they're on. A service using the AppService CRD gets platform-managed defaults, automatic updates when the platform evolves, and first-class support. A service using raw Kubernetes resources owns its own configuration and gets best-effort support. That distinction in the support model is what makes the trade-off legible rather than arbitrary. What I'd Do Differently In hindsight, the most important investment was the admission webhook, and I'd build it earlier. The CRD and controller took significant time to design and implement. The webhook could have been built in a few days and would have immediately improved the developer experience by catching misconfiguration at submission time rather than deployment time. Validation before generation is higher-leverage than generation that might produce something invalid. I'd also measure platform adoption from day one. Which teams are using the AppService CRD? Which teams are on raw Kubernetes? What's the conversion rate of new services to the platform abstraction? Without that data, the platform team relies on intuition instead of evidence to guide their investment. The teams that adopt slowly are often the ones with the most valuable feedback about where the abstraction doesn't fit, and they're also the teams most likely to be quietly maintaining fragile custom configurations that will become incidents later. When should you not build a CRD-based platform abstraction? If you have fewer than fifteen to twenty engineers deploying services, the overhead of designing, building, and maintaining a CRD-based platform abstraction almost certainly exceeds the value. Helm charts and excellent templates get you most of the way there with a fraction of the complexity. The operator pattern earns its cost when you have enough services that inconsistency becomes a real operational problem when the differences between how services are configured start causing incidents and nobody can tell you why a particular service is configured the way it is. Key Takeaways Infrastructure and platform are different things. Infrastructure is the machinery; a platform is the interface that makes machinery accessible without requiring expertise in its internals. Most Kubernetes-based IDPs stop at infrastructure. Custom resource definitions let you define a domain model that encodes your platform's opinions. Developers specify intent (tier, replicas, port); the controller generates the correct Kubernetes objects with consistent defaults applied. Admission webhooks shift error detection from runtime to submission time. A clear error message at kubectl apply is worth more than a mysterious pod crash two minutes later. Maintain a golden path for the majority of services and a documented escape hatch for the rest. A CRD that tries to cover every Kubernetes feature loses the simplicity that justified building it. Conclusion The platform engineer's job is to make complexity disappear, not by eliminating it, but by absorbing it into tooling so that the people building products don't have to carry it. That's a harder problem than building the infrastructure itself, and it requires a fundamentally different mindset: less systems engineering, more product thinking. Who are the users? What tasks do they need to accomplish? Where does the current experience fail them? The teams building internal developer platforms who get the process right tend to look, from the outside, like they have unusually productive engineering organizations. Individual contributions ship faster, incidents caused by misconfiguration drop, and the platform team spends less time on support and more time on improvements. The causal chain runs directly from platform quality to engineering output, even though it's usually measured differently. The open question is whether the CRD-based abstraction model scales to genuinely heterogeneous service fleets, the kinds of organizations where services span multiple languages, multiple deployment patterns, and multiple infrastructure dependencies. The golden path works when most services look similar enough that a shared abstraction is useful. What occurs to the platform model when 40% of services utilize the escape hatch? At that point, is the abstraction still earning its cost, or is it adding complexity without delivering the simplicity it promised?