Software Design and Architecture

Over the past few years, vision-language models (VLMs) have moved from experimental research projects to early production use. These systems take in both images and text, so they can read documents, interpret photographs, answer questions about visual content, and reason across modalities in ways that were not practical a few years ago. Better architectures and training methods explain part of this progress. Better ways of deploying models explain at least as much. In 2026, the defining question for enterprises is no longer whether VLMs work, but whether they can run reliably, safely and cost-effectively in production. The answer depends as much on cloud infrastructure, observability and governance as on the models themselves. This article traces how VLMs evolved, why cloud computing became the turning point for adoption, where reliability still falls short, and which cloud-native practices organizations need in order to deploy them responsibly at scale. From Research Breakthroughs to Foundation Models The modern VLM era began with CLIP (Contrastive Language–Image Pretraining). It showed that training on hundreds of millions of image-text pairs could align visual and textual representations in a shared embedding space. That alignment enabled zero-shot image classification and made large-scale multimodal search possible. DeepMind's Flamingo built on this foundation. It introduced cross-attention layers that connect a frozen vision encoder to a large language model, so a single model could learn new tasks from only a few examples. BLIP-2 then showed that a lightweight querying module could bridge frozen image encoders and frozen language models, which sharply reduced training cost. Instruction tuning was the next step. LLaVA (Large Language and Vision Assistant) showed that fine-tuning on visual instruction data produced conversational assistants that could follow complex multimodal prompts. OpenAI's GPT-4V brought these capabilities to a mass audience through a commercial product. Open-weight models such as Qwen2-VL later extended high-resolution perception to the open-source community. Taken together, these advances in cross-attention mechanisms, instruction fine-tuning, and large-scale multimodal pretraining allowed VLMs to combine perception and reasoning in a single system. The Cloud as the Deployment Turning Point The research milestones were essential, but the real turning point for adoption has been cloud deployment. VLMs are computationally demanding: they need high-memory GPUs, efficient batching, and low-latency serving. Few organizations can justify building that infrastructure in-house for workloads that are still evolving. Cloud platforms solve this problem. They offer GPU-backed compute on demand, managed model endpoints, and API-based access that fits directly into existing applications. As a result, VLMs now support practical enterprise workflows: Document intelligence: extracting data from invoices, contracts, and formsVisual inspection: spotting defects in manufacturingMultimodal search: retrieving content by combining images with natural-language queries Elastic scaling also lets organizations match capacity to demand, so they don't pay for idle hardware during quiet periods. Benchmarks Reveal Uneven Reliability Despite these advances, VLM reliability remains inconsistent. MMBench evaluates models across a wide range of perception and reasoning skills, and its results show large differences in performance from task to task. The VERIFY benchmark focuses on visual reasoning fidelity, testing whether models reason correctly about what they see rather than simply producing plausible answers. It exposes significant gaps even in leading systems. Industry reviews of recent computer vision research, including Voxel51's analysis of ICCV 2025 papers, point to the same ongoing weaknesses in fine-grained perception and consistency. A model that does well on one benchmark can still struggle on a closely related task. The practical lesson is clear: choose models based on the specific use case and workload, not on general leaderboard rankings. Choosing Between Open-Source and Managed Models Organizations face a strategic choice between open-source and managed VLMs. Open-source models such as LLaVA and Qwen2-VL can run on Kubernetes clusters or in a private cloud. They give full control over data residency, customization, and cost, which suits regulated industries and sensitive data. The trade-off is that they require in-house expertise in GPU orchestration, model serving, and ongoing maintenance. Managed services offer elastic scaling, built-in security, and much less operational complexity, at the cost of less control and possible vendor lock-in. Many enterprises use a hybrid approach. They route sensitive workloads to self-hosted models and send general-purpose tasks to managed endpoints. Hidden Failure Modes at Scale A major concern in production is hidden failure modes. The most widely discussed is hallucination, where a model describes objects or details that are not in the image, and research on object hallucination shows it remains a persistent problem. Other risks include: OCR errors on low-quality scansSmall objects being missed in dense scenesSpatial reasoning failures, such as confusing left and right or misjudging relative positions Inside automated pipelines, these errors are especially dangerous. One hallucinated invoice total or missed defect can repeat across thousands of transactions before anyone notices. At scale, small error rates add up to significant business and compliance risk. Cloud-Native Guardrails for Production To manage these risks, organizations need guardrails built into their cloud-native architecture: Confidence thresholds send uncertain outputs for further review instead of accepting them automatically.Human-in-the-loop review gives a person oversight of high-stakes decisions.Retrieval-based grounding ties model outputs to verified source data, which reduces hallucination.Audit logging records inputs, outputs, and decisions for traceability and compliance.API gateway policies enforce rate limits, access controls, and content rules consistently across applications. Frameworks such as the NIST AI Risk Management Framework give a useful structure for applying these controls. They help organizations map, measure, and manage AI risk across the lifecycle. Observability tools that track drift, latency, and error patterns complete the picture, so teams can detect problems before they spread. These controls are what move VLMs from impressive demonstrations to dependable production systems. Conclusion The VLM story is ultimately about systems as much as models. Breakthroughs from CLIP to Qwen2-VL made multimodal understanding possible, but cloud computing made it practical. Combining multimodal AI with cloud-native infrastructure has let organizations build AI workflows that are scalable, observable, and governed. VLMs are not yet perfect. Benchmarks show uneven reliability, and hallucination remains a real risk. Even so, they have matured enough for teams to deploy them at scale, provided those teams take careful model selection, rigorous evaluation, and strong guardrails seriously. The organizations that succeed in 2026 will treat VLMs as one part of a well-architected system with governance built in from the start.

By Srinivas Chippagiri CORE

A pod stuck in Pending is one of the most misleading states in Kubernetes. Nothing looks wrong. There's no error banner, no red text screaming at you. The scheduler just quietly gave up finding a home for it and is waiting for the world to change. I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about. This is capacity spillover: demand outgrows what the current set of nodes can absorb, and the cluster needs somewhere for the overflow to land. Handle that well and it's a blip nobody notices. Handle it badly, and it's an outage that shows up in a customer complaint before it shows up in your dashboard. I'll be honest, a few years ago I would have called this an edge case worth a paragraph in the docs and moved on. Not anymore. Ask around, and you'll find most people running workloads in the cloud right now have a story about a specific instance type just not being there when they needed it, GPU-backed machines most visibly, but I've seen it happen with fairly ordinary compute SKUs too, usually right in the middle of a demand spike nobody scheduled around. That's really the whole case for spillover, once you strip away the architecture diagrams. Without it, what you get during a capacity crunch is close to random: whichever pod happens to hit the scheduler first grabs whatever's left, and everyone else waits and hopes. With it, the randomness mostly goes away. Workloads land where you decided they should, in the order you decided matters, instead of wherever the scheduler happened to find room first. New pod needs scheduling -> primary node pool checked -> if room is available, schedule normally; if the pool is full, spill to the secondary pool (overflow capacity). Why Scheduling Failures Happen in the First Place The scheduler's job sounds almost too simple: find a node with enough free CPU and memory for the pod's requests. In practice, a pod can fail to schedule for reasons that have nothing to do with the cluster actually running out of room. Here's the part that trips people up the first time they see it. A cluster can have plenty of free CPU in aggregate and still fail to place a memory-heavy pod, because no single node has enough memory available on its own. Resource requests get evaluated per node, not pooled across the whole cluster. I've had a coworker insist this was a bug the first time they saw it happen. It isn't. It's just how bin-packing works when nobody told you that's what you were looking at. Taints, tolerations, and node affinity narrow the field further. A pod that needs a GPU node, or needs to steer clear of a specific zone, only ever sees the slice of the cluster that matches. Spare capacity can be sitting right there, and the pod simply isn't allowed near it. And then there's the timing problem, which is the one that actually pages people. Even when the cluster genuinely needs more nodes and the autoscaler agrees, provisioning a node isn't instant. There's a real window between "we need capacity" and "capacity exists," and whatever's waiting in that window experiences it as a failure, not as a system doing exactly what it was designed to do. What Spillover Actually Looks Like "Spillover" gets used loosely, but it covers a few genuinely different situations, and I think that's where a lot of confusion comes from. People nod along in a meeting assuming everyone means the same thing, and they don't. The first one, and honestly the one most teams reach for first, is spilling to a different node pool entirely. You run a primary pool sized for steady-state load, and a secondary pool, usually spot or preemptible, that only spins up once the primary's full. Interruption-tolerant stuff gets pushed there when things get tight, which keeps the expensive, stable pool free for whatever actually can't afford to get evicted. It's an easy sell to whoever owns the cloud bill, which is probably why it's the version everyone sets up first. Then there's spilling across zones, or in the worst cases, regions. A single zone running out of a specific instance type during a demand spike isn't some rare event you plan a tabletop exercise around; it's just a Tuesday at this point. So you spread the workload wider. Fine in theory. I've watched a "let's just spread it across zones" fix get shipped on a Friday and turn into a cross-region latency ticket by the following Wednesday, because nobody had checked whether the workload actually tolerated that. And there's a third flavor that's less about running out of room and more about not wanting to spend the money. Technically, there's room on the expensive reserved instances. The better default is letting things spill onto cheap infrastructure first, and only reaching for the premium stuff when nothing cheaper is left over. Build it the other way, and you won't find out until the next billing cycle, which is a genuinely bad time to find out. Designing for It Instead of Reacting to It The cluster autoscaler, or something like Karpenter, handles the mechanics of adding nodes once pods can't be placed anywhere. That's useful, but it's also reactive by definition, closer to a fire alarm than a smoke detector. By the time it fires, everyone already knows something's wrong. A few things actually close the gap between "capacity ran out" and "someone got paged," in my experience. Pod priority and preemption are the ones people underestimate. Not every workload deserves an equal claim on scarce capacity, and treating them as if they do just means everything fails to schedule at once, in whatever order the scheduler happened to try them, which helps nobody. Give the stuff that actually matters a higher PriorityClass and let it evict lower-priority pods when things get tight. Over-provisioning on purpose sounds wasteful until you sit with it for a second. A handful of low-priority placeholder pods, doing nothing except holding a spot, get evicted the instant real demand shows up, and the real workload schedules immediately instead of waiting on a node to boot. Small constant cost, much faster response when it actually counts. Provisioning speed is the one that surprised me. Tools that can add nodes directly, without needing to fit some pre-defined pool shape first, shrink that window between "pod can't schedule" and "a node exists to put it on" a lot more than I expected going in. That window is basically where the pager lives. And testing failure on purpose, which is the step almost everyone skips. It's easy to assume spillover works because it's configured correctly on paper. Deliberately draining a node pool in a test environment and watching whether pods actually land where they're supposed to, in the order they're supposed to, finds a category of misconfiguration you will never catch by reading YAML, however carefully you read it. The Part That's Easy to Skip Past Spillover design tends to get treated as a purely technical exercise. Pick the autoscaler, define the pools, set the priorities, done. The harder question usually isn't technical at all: who decides which workloads are allowed to spill onto cheaper or less reliable capacity, and who's on the hook when a decision made for cost reasons turns into someone else's latency problem three sprints later. Getting the mechanics right is necessary. It's not enough on its own, because nobody's actually agreed on the policy those mechanics are supposed to enforce, and that conversation almost always happens after the first bad incident instead of before it. It's happened to every team I've worked with in some form. I'd rather have that conversation early, even if it's a little awkward and nobody wants to own the answer yet. It beats the alternative meeting, the one scheduled at 9 am the morning after something broke.

Every system that stores sensitive data needs keys that never leave a hardened boundary, an audit trail of every use, and rotation that doesn't mean re-encrypting petabytes. On Google Cloud, that's Cloud KMS; on AWS, it's AWS KMS. Both store symmetric and asymmetric keys, both use HSMs, and both are built for envelope encryption. But AWS KMS vs. Google Cloud KMS comes down to the choices that show up in your code: how keys are named, what rotation means, where authorization lives, and whether the service hands you data keys. This article walks through Cloud KMS in Java and then maps it onto AWS KMS. The Cloud KMS Resource Model Cloud KMS organizes keys in a four-level hierarchy under a project: Location. Where the key material lives: a region such as us-east1, a multi-region such as us, or global. Key material stays in the location you pick for its whole lifecycle.Key ring. A named group of keys in one location. It exists for organization and IAM inheritance.CryptoKey. The key you reference in code. It has a purpose (ENCRYPT_DECRYPT, ASYMMETRIC_SIGN, ASYMMETRIC_DECRYPT, MAC, and a few more) and a rotation schedule.CryptoKeyVersion. The actual key material. A key has one or more versions, and for symmetric keys one of them is the primary, the version that new encryptions use. A symmetric key with three versions: new data uses version 3, data written under version 2 still decrypts, and version 1 is on its way out. A key is identified only by its full resource name, projects/my-project/locations/us-east1/keyRings/payments/cryptoKeys/card-data, optionally followed by /cryptoKeyVersions/2. There are no aliases. Nothing can move after it's created: the location is fixed when you create the key ring, and key rings themselves can't be deleted. Each key version also has a protection level: SOFTWARE: keys are protected by Google's software crypto (BoringCrypto, FIPS 140-3 Level 1).HSM: keys live in Cloud HSM, which is FIPS 140-2 Level 3 validated. HSM_SINGLE_TENANT gives you a dedicated HSM.EXTERNAL and EXTERNAL_VPC: Cloud EKM keeps the key material in a supported external key manager that you operate, reached over the internet or a VPC. Google never holds the key. Creating a Key That Rotates Itself The Java client is KeyManagementServiceClient. It uses Application Default Credentials, so on GKE, Cloud Run, or GCE, the workload's service account authenticates with no keys in the code. This creates a key ring and an HSM-backed symmetric key that rotates every 90 days: Java try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { KeyRing ring = kms.createKeyRing( LocationName.of("my-project", "us-east1"), "payments", KeyRing.getDefaultInstance()); CryptoKey key = CryptoKey.newBuilder() .setPurpose(CryptoKeyPurpose.ENCRYPT_DECRYPT) .setVersionTemplate(CryptoKeyVersionTemplate.newBuilder() .setAlgorithm(CryptoKeyVersionAlgorithm.GOOGLE_SYMMETRIC_ENCRYPTION) .setProtectionLevel(ProtectionLevel.HSM)) .setRotationPeriod(Duration.newBuilder().setSeconds(90L * 24 * 3600)) // every 90 days .setNextRotationTime(Timestamp.newBuilder() .setSeconds(Instant.now().plusSeconds(90L * 24 * 3600).getEpochSecond())) .build(); kms.createCryptoKey(KeyRingName.parse(ring.getName()), "card-data", key); } The rotation period can be anywhere from one day to 100 years. When it fires, Cloud KMS creates a new version and makes it primary. Earlier versions stay enabled, so old ciphertext still decrypts, and nothing is re-encrypted. To rotate by hand, create a version with CreateCryptoKeyVersion and promote it with UpdateCryptoKeyPrimaryVersion. Automatic rotation applies only to symmetric keys; asymmetric keys are rotated manually. The AWS KMS key rotation process differs in how key material is managed, as described below. Envelope Encryption KMS keys aren't meant to encrypt your data directly. A single Encrypt call accepts at most 64 KiB of plaintext (8 KiB of plaintext plus AAD for HSM keys), and every call is a network round trip. The standard pattern is envelope encryption, also used with AWS KMS envelope encryption: encrypt the data locally with a data encryption key (DEK), then have KMS encrypt (wrap) the DEK with a key encryption key (KEK) that never leaves the service. You store the wrapped DEK next to the ciphertext. Envelope encryption: The payload never goes to KMS; only the 32-byte DEK does. Cloud KMS has no data-key API, so you generate the DEK yourself. Google's guidance is to generate a new DEK every time you write data, never share one across users, and prefer Tink, which packages this whole flow. Doing it by hand shows what's happening: Java CryptoKeyName kek = CryptoKeyName.of("my-project", "us-east1", "payments", "card-data"); ByteString aad = ByteString.copyFromUtf8("tenant=acme"); // must be supplied again to decrypt try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { // 1. Generate a fresh DEK locally. Cloud KMS has no GenerateDataKey call. KeyGenerator gen = KeyGenerator.getInstance("AES"); gen.init(256); SecretKey dek = gen.generateKey(); // ... encrypt the payload locally with dek (AES-GCM), then discard dek ... // 2. Wrap the DEK with the KEK, with CRC32C checks against corruption in transit. ByteString dekBytes = ByteString.copyFrom(dek.getEncoded()); EncryptResponse wrapped = kms.encrypt(EncryptRequest.newBuilder() .setName(kek.toString()) .setPlaintext(dekBytes) .setPlaintextCrc32C(crc32c(dekBytes)) .setAdditionalAuthenticatedData(aad) .setAdditionalAuthenticatedDataCrc32C(crc32c(aad)) .build()); if (!wrapped.getVerifiedPlaintextCrc32C() || wrapped.getCiphertextCrc32C().getValue() != crc32c(wrapped.getCiphertext()).getValue()) { throw new IllegalStateException("integrity check failed"); } ByteString wrappedDek = wrapped.getCiphertext(); // store next to the encrypted payload String usedVersion = wrapped.getName(); // the primary version that wrapped it // 3. Later: unwrap. Pass the key, not the version; Cloud KMS picks the version. ByteString unwrapped = kms.decrypt(DecryptRequest.newBuilder() .setName(kek.toString()) .setCiphertext(wrappedDek) .setAdditionalAuthenticatedData(aad) .build()) .getPlaintext(); SecretKey restored = new SecretKeySpec(unwrapped.toByteArray(), "AES"); } static Int64Value crc32c(ByteString data) { return Int64Value.of(Hashing.crc32c().hashBytes(data.toByteArray()).padToLong()); } Encrypt uses the key's current primary version. Decrypt also takes the key, not a version: the ciphertext records which version produced it, so rotation is invisible to readers. The optional CRC32C fields let the server confirm it received your plaintext intact, and let you confirm the ciphertext you got back. The AAD (tenant=acme) is bound to the ciphertext, so one tenant's wrapped DEK can't be swapped into another tenant's record. Disabling, destroying, and deleting Lifecycle operations act on versions, not keys. You disable a version (ENABLED to DISABLED) to stop it from being used, and re-enable it later. There's no switch that disables a whole key; you disable each version. Destroying a version is deliberately slow. DestroyCryptoKeyVersion moves it to DESTROY_SCHEDULED for 30 days by default. The duration is set when the key is created, and an organization policy can enforce a minimum. During that window, RestoreCryptoKeyVersion brings it back in the DISABLED state. After the window, the material is gone and data encrypted by that version can't be recovered. Destroyed versions are no longer billed. Cloud KMS has also added permanent deletion: you can delete key versions and then the key itself, though a deleted key's name can't be reused. Access control is just IAM Cloud KMS has no policy language of its own. Access is ordinary IAM, granted at the organization, folder, project, key ring, or key level and inherited downward. The predefined roles split duties cleanly: roles/cloudkms.admin manages keys but can't encrypt or decrypt.roles/cloudkms.cryptoKeyEncrypterDecrypter, cryptoKeyEncrypter, and cryptoKeyDecrypter can use keys but not manage them.roles/cloudkms.signerVerifier and publicKeyViewer cover asymmetric keys. The operator who rotates keys never sees plaintext, and the service that decrypts can't destroy its key. Grant usage roles on the specific key, not the project. Two more controls round this out. Cloud Audit Logs record admin operations (create, rotate, destroy) automatically, but Encrypt and Decrypt are Data Access logs, which are off by default. If you want a record of every key use, enable them for Cloud KMS. Key Access Justifications, used with Cloud EKM keys, attaches a reason code to every Google request for your key, and your external key manager can refuse it. Bringing your own key To import key material, you create an import job with a wrapping method, such as RSA_OAEP_3072_SHA256_AES_256 or one of the newer post-quantum HPKE methods. The job gives you a public key, and it expires after three days. You wrap your key material with that public key and import it as a new version of a key. For a key that must only ever hold imported material, set importOnly, which means rotation is manual: import new material and make it primary. If you destroy an imported version, you can re-import the same material later. AWS KMS for comparison AWS KMS covers the same ground with a flatter, version-less model: Keys, not versions. A KMS key has a key ID (a UUID), an ARN, and optional aliases such as alias/card-data. There's no hierarchy above it beyond the account and Region.Rotation is hidden. Automatic rotation is off by default for customer managed keys. When enabled, it runs every 365 days by default, configurable from 90 to 2,560 days, and RotateKeyOnDemand allows up to 25 extra rotations per key. The key ID never changes; AWS keeps old material internally and picks it on decrypt.All keys in HSMs. Keys in the default key store live in HSMs validated at FIPS 140-3 Level 3. There's no software tier. CloudHSM key stores and external key stores (XKS) cover the dedicated and hold-your-own-key cases.A data-key API. GenerateDataKey returns a fresh DEK in plaintext along with a copy wrapped under your key, so step 1 and step 2 above are one call.Encryption context. The AWS equivalent of AAD is a map of key-value pairs. Unlike Cloud KMS AAD, it's recorded in CloudTrail and can be used in policy conditions such as kms:EncryptionContext:tenant.Three authorization layers. Every key has exactly one key policy, a resource policy that supports Deny and conditions. IAM policies work only if the key policy enables them, and grants add temporary, programmatic permissions on top.Whole-key lifecycle. DisableKey and ScheduleKeyDeletion act on the whole key. The deletion window is 7 to 30 days, and CancelKeyDeletion backs it out.Replication. Multi-Region keys share the same key ID and material across Regions, and ReplicateKey copies one to a new Region. You choose this at creation; a single-Region key can't be converted. Here's the same envelope flow on AWS: Java Map<String, String> context = Map.of("tenant", "acme"); // logged in CloudTrail, usable in policy try (KmsClient kms = KmsClient.create()) { GenerateDataKeyResponse dataKey = kms.generateDataKey(GenerateDataKeyRequest.builder() .keyId("alias/card-data") .keySpec(DataKeySpec.AES_256) .encryptionContext(context) .build()); byte[] dek = dataKey.plaintext().asByteArray(); // encrypt locally, then discard SdkBytes wrappedDek = dataKey.ciphertextBlob(); // store next to the payload byte[] unwrapped = kms.decrypt(DecryptRequest.builder() .keyId("alias/card-data") // optional for symmetric keys .ciphertextBlob(wrappedDek) .encryptionContext(context) .build()) .plaintext().asByteArray(); } KeyId is optional on Decrypt for symmetric keys because the ciphertext blob identifies the key, but AWS recommends passing it so you don't decrypt with a key you didn't expect. Cloud KMS requires the key name on every Decrypt. Side by side ConceptGoogle Cloud KMSAWS KMSKey identifierResource name projects/…/cryptoKeys/k, no aliasesKey ID, ARN, aliasesVersionsFirst-class CryptoKeyVersion with its own stateHidden behind the key IDProtectionSOFTWARE, HSM (FIPS 140-2 L3), single-tenant HSM, EXTERNAL, EXTERNAL_VPCAll HSM-backed (FIPS 140-3 L3), plus CloudHSM and external key storesMax plaintext per call64 KiB (8 KiB with AAD for HSM keys)4,096 bytesData keysGenerate locally (or with Tink)GenerateDataKeyAuthenticated dataAAD bytes, not loggedEncryption context map, logged and usable in policiesAuto-rotation1 day to 100 years90 to 2,560 days, off by default, plus 25 on-demandDisablePer versionWhole keyDestroyPer version, 30 days scheduled by default, restorableWhole key, 7–30 days, cancellableAccess controlIAM roles on the hierarchy, admin and usage splitKey policy (with Deny) + IAM + grantsUsage loggingData Access audit logs, off by defaultCloudTrail management eventsMulti-regionChoose a multi-region location for the key ringMulti-Region keys, ReplicateKeyBYOKImport job, expires in 3 daysGetParametersForImport, token valid 24 hoursThroughput quota (default)Token-based per region; about 60,000 software operations a minute10,000 to 100,000 symmetric operations a second, by Region What the AWS KMS vs. Google Cloud KMS encryption comparison means in practice: Cloud KMS is simpler to reason about for access. It's one IAM model with roles that separate managing keys from using them. AWS is more expressive: a key policy can Deny, and conditions on encryption context can tie a key's use to a tenant. That power comes with the risk of a key policy that locks out its own administrators.Versions are explicit on Google Cloud. You can disable or destroy just the key material that was compromised and leave the rest working. On AWS you disable or delete the whole key, or rotate and keep the old material.AWS logs every use by default. On Google Cloud, enable Data Access logs for Cloud KMS on day one, or you won't have a record of who decrypted what.Plan for quotas. AWS gives each account and Region a large shared pool of cryptographic operations. Cloud KMS's default is far smaller per project and region, so cache unwrapped DEKs for hot reads where your threat model allows, and request a quota increase before launch. Both quotas are adjustable.Both want envelope encryption. Neither service is built to encrypt bulk data directly. On Google Cloud, Tink makes the local half trivial. On AWS, the AWS Encryption SDK plays the same role. Conclusion Google Cloud KMS treats keys as resources in your project hierarchy: versions are visible, protection level is chosen per key, and IAM is the whole authorization story. That's easy to audit and gives fine-grained control over key material, as long as you turn on Data Access logs and size your quotas. AWS KMS hides versions and puts authorization in a per-key policy language that's more expressive, especially with encryption context, and its data-key API saves a step. These differences are worth considering when reviewing AWS KMS best practices alongside Google Cloud KMS key management. On both clouds, the envelope pattern is the common ground: generate a DEK, wrap it, store it next to the data, and let KMS guard only the key that guards everything else.

RabbitMQ is a message broker that applications use to communicate asynchronously. A publisher sends a message to an exchange with a routing key. The exchange compares the routing key with the binding keys and routes the message to all matching queues, which deliver it to consumers. A successful publish means RabbitMQ accepted the message at the exchange. It does not always mean that a queue received it. If no binding matches, the message is unroutable and may be dropped or returned to the publisher, depending on the publisher settings. RabbitMQ Exchange Types An exchange decides where a message should go. Its type determines how it makes that decision: Direct: Routes when the message routing key exactly matches the binding key.Topic: Uses dot-separated keys and wildcards. * matches one word; # matches zero or more words.Headers: Routes using message headers rather than the routing key.Fanout: Sends a copy to every bound queue and ignores the routing key. The default exchange is a special direct exchange. RabbitMQ automatically binds each queue to it using the queue name as the routing key. Issue We deployed RabbitMQ in a three-node Kubernetes environment. Several applications use it to broadcast messages to multiple consumer applications. Application teams create topic exchanges and bind queues with binding keys. For example, we created a topic exchange as shown below: Publishing m.event.provisioning.started to the exchange should route a copy to every queue whose binding pattern matches it. In some environments, messages published to topic exchanges were not routed to queues even when the expected routing key was used. We inspected the topology using the management console and rabbitmqctl commands. These commands inspect the configured exchanges, queues, and bindings in the selected virtual host: Shell rabbitmqctl list_exchanges -p <vhost> name type durable arguments This command will list all exchanges in rabbitmq rabbitmqctl list_queues -p <vhost> name durable type messages messages_ready consumers This command will list all queues declared in rabbitmq rabbitmqctl list_bindings -p <vhost> source_name source_kind destination_name destination_kind routing_key arguments This command will list exchange, queue and bindings in rabbitmq The configured topology looked correct. However, in one environment, when we tried to publish a message using the RabbitMQ management UI, we received the message: Message published, but not routed. Impact In a field deployment, this will impact multiple customer workflows. Customers can experience the failure even though the publisher reports a successful publish. A provisioning or configuration workflow can remain in a pending state because the service waiting for the event never receives it.Different services can hold different views of the same customer operation. For example, one service may record that a change started while another service never receives the event needed to update inventory, status, or audit information.Customer onboarding, device activation, policy changes, or software upgrades can be delayed or fail when an expected event is lost between services.Operators may retry the operation or manually intervene. That can increase recovery time and, if the original event is later delivered, create duplicate processing or confusing status changes.The failure can look intermittent because only some bindings or queues are affected. The management UI may still show a healthy declarative topology, which makes the issue harder to identify as a broker-routing problem. Root Cause RabbitMQ persists cluster metadata, including topology such as exchanges, queues, and bindings. During startup, the broker loads that metadata into runtime structures used for message routing. We observed this in a Mnesia-based deployment. Mnesia is a distributed internal database used by RabbitMQ to store cluster metadata. And RabbitMQ loads topic exchange configuration into these internal tables: rabbit_topic_trie_edge and rabbit_topic_trie_binding. rabbit_topic_trie_edge This internal runtime table represents topic-binding segments as a trie. For a binding such as m.event.provisioning.#, the dot-separated segments form edges, and each edge refers to its parent. The root edge has no parent. The command to list the contents of the rabbit_topic_trie_edge table is: rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_edge)'. Below is sample output: Shell rabbitm@rabbitmq-0:/$ rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).' [{topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>, {resource,<<"/">>,queue,<<"provisioning-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>, {resource,<<"/">>,queue,<<"audit-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>, {resource,<<"/">>,queue,<<"exact-event">>}, []}, const}] root@my-rabbit:/# rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_edge).' [{topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, root,"m"}, <<217,70,178,13,233,108,102,215,235,100,37,39,215,68,241,133>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>, "#"}, <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>, "completed"}, <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<217,70,178,13,233,108,102,215,235,100,37,39,215,68,241,133>>, "event"}, <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>, "#"}, <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>, "provisioning"}, <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>}] rabbit_topic_trie_binding This internal table associates the topic exchange and destination with the final edge in the topic trie. The command to list the contents of the rabbit_topic_trie_binding table is: rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).' Below is sample output: Shell rabbit@rabbitmq-0:/$ rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).' [{topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>, {resource,<<"/">>,queue,<<"provisioning-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>, {resource,<<"/">>,queue,<<"audit-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>, {resource,<<"/">>,queue,<<"exact-event">>}, []}, const}] We developed a diagnostic utility that reconstructed binding paths from these internal runtime tables and compared them with the topology returned by rabbitmqctl. The comparison showed that some configured bindings were not correctly represented in the runtime routing state. How We Investigated the Problem We first treated the symptom as a normal binding problem. We checked that the publisher and the consumer were using the same virtual host, confirmed that the exchange was a topic exchange, and compared the binding keys character by character. We also checked the queue consumer count and published a test message directly from the management UI. The message was accepted, but the UI reported Message published, but not routed. Next, we used rabbitmqctl list_exchanges, rabbitmqctl list_queues, and rabbitmqctl list_bindings to capture the topology. Those commands showed the exchange, queues, and bindings that we expected. We repeated the same checks after connecting to different RabbitMQ pods and did not find a spelling, virtual-host, or wildcard mismatch. At that point, we stopped changing application routing keys and compared the configured topology with the broker's runtime topic-trie state. The configured binding was present in the management output, but the corresponding path was incomplete in the runtime tables on the affected deployment. This explained why recreating the binding changed the behavior without changing the application code. The publisher confirmed, and the mandatory publish option helped us narrow the problem further. A publisher confirm told us that RabbitMQ accepted the publish operation. The mandatory option made an unroutable message visible to the publisher, but it did not tell us which intended consumer was missing. We still had to verify delivery at each expected queue. What the Publish Result Actually Tells Us RabbitMQ accepts a publish at the exchange before it evaluates all of the topic bindings. For a topic exchange, the routing key is split into dot-separated words and matched against each binding pattern. A single-word wildcard (*) matches exactly one segment, while the multi-word wildcard (#) can match zero or more segments. A small difference such as m.event.provisioning.started versus m.events.provisioning.started is enough to produce no match. This was the important distinction in our investigation: a successful publish was not proof of delivery. When the publish is not mandatory, a message that matches no queue can be discarded after the exchange accepts it. In our case, the binding appeared in the declarative topology but was not fully available to the runtime routing path, so the result looked like an ordinary unroutable message even though the management view appeared correct. Changes We Made After the Incident We kept the exchange, queue, and binding declarations idempotent, but added an explicit verification step after broker restarts, pod rescheduling, and upgrades. We also added a utility that reads the rabbit_topic_trie_edge and rabbit_topic_trie_binding tables, reconstructs the binding paths, and compares them with the exchange, queue, and binding information returned by rabbitmqctl. This gives us a direct way to identify a binding that exists in the configured topology but is missing or incomplete in the runtime topic trie. In addition, we enabled detailed metrics at the exchange level so that we can see the unrouted message count at each exchange level. We correlate those metrics with the virtual host, routing key, queue, node, and timestamp captured by the utility. The combination of runtime-table comparison and exchange-level metrics makes it easier to determine whether the problem is a routing mismatch, a missing runtime binding, or a consumer that received the message but did not process it. We now use the application logs, exchange-level metrics, and the utility output as the first evidence after restarts, pod rescheduling, and upgrades. We correlate an actual failed event with the expected exchange and binding path instead of injecting another message into a customer environment. The expected topology, comparison utility, and diagnostic commands are kept with the deployment so that an operator can compare the intended state with the live state during an incident. Observed Workaround For each affected binding, we captured the virtual host, exchange, queue, binding key, and arguments, then deleted and recreated the binding through the RabbitMQ API. We compared the runtime topic-trie state with the topology reported by rabbitmqctl, retried the affected workflows, and confirmed delivery at every expected queue. This was the workaround for our deployment, not a universal fix. Do not edit Mnesia or ETS files directly; internal tables are version-specific and direct changes can leave the cluster inconsistent. Lessons Learned Management and CLI topology output are necessary first checks, but they do not replace an end-to-end routing test. After restarts, pod rescheduling, or upgrades, publish representative routing keys and verify that every expected queue receives the message. Publisher confirms, and mandatory publishing can also make unroutable messages visible to applications. We observed this issue only with topic exchanges.

Architecture Review Boards (ARBs) have traditionally evaluated application architectures against key enterprise criteria such as security, scalability, integration, resilience, supportability, and compliance. With the rapid adoption of artificial intelligence (AI), machine learning (ML), and generative AI technologies, organizations are increasingly deploying intelligent solutions that introduce new architectural, operational, ethical, and regulatory challenges. As a result, ARBs must expand beyond traditional review dimensions to assess AI-specific considerations, including data governance, model lifecycle management, explainability, bias mitigation, AI security, operational monitoring, and responsible AI practices. The evolution of ARBs is essential to ensure AI solutions deliver business value while maintaining trust, compliance, accountability, and enterprise risk controls. The objectives of an ARB for AI-based solutions are to: Enable responsible AI innovationManage business, operational, and regulatory risksProtect enterprise and customer dataEstablish governance and accountabilityImprove the success rate of AI initiativesEnsure compliance with organizational and regulatory requirements The table below highlights the key differences between a traditional Architecture Review Board and an AI-focused Architecture Review Board. areatraditional arbAI Solutions ARB Focus Area Application architecture quality, stability, and alignment with enterprise standards. Governance of AI solutions while ensuring business value and risk management. Review Architecture Components Applications, integrations, databases, infrastructure, APIs, and middleware. AI models, LLMs, prompts, vector databases, AI agents, training data, MLOps / LLMOps platforms, and AI services. Security Network security, application security, IAM, encryption, and vulnerability management. Traditional security + prompt injection protection, model security, adversarial attacks, AI supply chain security, and data leakage prevention. Data Governance Data quality, ownership, retention, and regulatory compliance. Data lineage, training data quality, AI data usage rights, consent management, data sovereignty, and model input/output governance. Compliance Industry standards, regulatory requirements, and enterprise policies. AI regulations, ethical AI principles, transparency requirements, and responsible AI frameworks. Risk Operational, technical, and cybersecurity risks. Hallucinations, bias, misinformation, model drift, ethical risks, and reputational risks. Explainability Not a major review criterion. Critical requirement to ensure AI decisions and outputs can be understood, justified, and audited. Bias and Fairness Outside review scope. Bias detection, fairness controls, and mitigation mechanisms. Testing & Validation Functional, performance, integration, security, and UAT testing. Prompt testing, adversarial testing, hallucination assessment, and AI output validation. Operations & Monitoring Application monitoring, logging, incident management, and SLAs. Performance, drift detection, output quality, token consumption, and AI safety monitoring. Lifecycle Management Software development lifecycle (SDLC), release management, and application support. Model lifecycle management, retraining strategy, model versioning, LLM updates, and MLOps/LLMOps governance. Accountability Application owner and business sponsor accountability. Accountability for AI outcomes, model ownership, risk ownership, and human-in-the-loop oversight. Decision Criteria Security, scalability, resilience, integration, supportability, and compliance. Explainability, trustworthiness, fairness, governance, responsible AI, and business risk controls. Success Measures System reliability, performance, availability, and compliance. Business value realization, responsible AI adoption, model accuracy, user trust, regulatory adherence, and risk reduction. Prompt Governance Not Applicable Prompt design, validation, security, and lifecycle management Model Selection Not Applicable Evaluation of model suitability, cost, accuracy, and risk Human Oversight Limited Human-in-the-loop review and approval processes AI Ethics Rarely Reviewed Fairness, transparency, accountability, and societal impact Vendor Lock-In Infrastructure focused AI provider dependency and model portability Cost Governance Infrastructure and licensing Token consumption, inference cost, and model hosting costs Responsible AI Limited Mandatory governance and compliance requirements The shift in ARB assessment perspectives can be summarized through the following key questions: Below is a six-pillar AI ARB framework for solution reviews and assessments. Business value assessment: Organizations deploy AI because it is technologically attractive rather than business-driven. The ARB team evaluates revenue growth, cost optimization, productivity improvement, customer experience enhancement, and decision-making effectiveness. Data governance and privacy: Data is the most critical asset in any GenAI solution. Poor data governance can expose sensitive information and create significant compliance risks. The ARB team evaluates data classification, data protection, and regulatory compliance.Security architecture review: GenAI systems introduce new attack vectors not commonly found in traditional enterprise applications. The ARB team evaluation areas include identity management integration, role-based access control, API gateway protection, encryption at rest and in transit, network isolation, key management, and security monitoring.Responsible AI governance: Responsible AI should be a mandatory review category within every GenAI ARB process. The objective is to ensure AI systems are fair, transparent, accountable, and trustworthy. The ARB team evaluates fairness, transparency, accountability, and explainability. Architecture and integration assessment: The ARB team evaluation areas on solution architecture, integration strategy, RAG architecture, and multi-model strategy.Operations, monitoring, and FinOps: GenAI projects succeed technically but struggle operationally after deployment. The ARB team evaluates business, technical, and financial metrics. Below are the best practices for enterprise ARBs reviewing GenAI solutions: Establish a dedicated AI review track within the ARB.Mandate Responsible AI assessments.Require data classification and privacy reviews.Standardize AI architecture reference patterns.Define AI monitoring and observability standards.Include Legal, Security, Data Governance, and EA stakeholders.Adopt a FinOps approach for AI cost governance.Establish a continuous AI governance process rather than a one-time approval activity. Conclusion Future ARBs are no longer responsible solely for evaluating system architecture. The review board must become a multidisciplinary governance body capable of balancing innovation, risk, compliance, and ethics. Organizations that evolve their ARB practices to address generative AI-specific concerns will be better positioned to scale AI adoption while maintaining trust, transparency, and regulatory compliance.

By Ravi Kiran Mallidi CORE

Traditionally, developers use different query optimization techniques, such as complex conditional looping and procedure branching, and query hints like OPTION (RECOMPILE). The query hint option sometimes leads to high CPU utilization. SQL Server 2022 (Compatibility Level 160) introduced a new feature called Parameter Sensitive Plan (PSP) Optimization as part of the core capabilities of the Intelligent Query Processing (IQP) feature family. PSP optimization addresses parameter sniffing at the engine level by allowing a single parameterized query to generate and maintain multiple active execution plans in the plan cache simultaneously, tailored to different data volume boundaries. How It Works: Mechanics, Dispatchers, and Variants Instead of forcing a single execution plan into cache for all parameter invocations, the SQL Server 2022 Query Optimizer uses data distribution statistics to recognize parameter sensitivity automatically during initial compilation. How to Detect Parameter Sensitivity During compilation, the optimizer evaluates predicates referencing parameter variables. It checks the associated column statistics histogram. If the statistics indicate significant data skew across histogram steps, the query is flagged as PSP-eligible. When a query qualifies, the engine creates a Dispatcher Plan. The dispatcher plan contains the logic needed to evaluate incoming runtime parameter values against pre-calculated boundary ranges. It acts as an internal router rather than a direct execution plan. The optimizer divides the parameter data distribution spectrum into distinct boundary buckets (typically low, medium, and high cardinality ranges) based on the histogram: Boundary calculation: The engine calculates threshold values using histogram step densitiesVariant generation: As different parameter values are executed, the dispatcher maps the runtime value to its corresponding boundary range and compiles or reuses a specific Variant Plan associated with that range. When reviewing execution plans in SQL Server Management Studio (SSMS), PSP-optimized queries display distinct Query Plan XML attributes under the Dispatcher and Statement Plan nodes, referencing variant identifiers (e.g., PlanVariantID="1" vs. PlanVariantID="2") Demonstrating PSP Optimization in Action The following example uses customers from the WideWorldImporters database. In the sample data, CustomerID = 90 has 150 orders, while CustomerID = 1060 has only 4. To make the execution-plan difference easier to observe, we create a small demonstration table that preserves this skew at a larger scale. SQL USE WideWorldImporters; GO DROP TABLE IF EXISTS dbo.PSP_OrderDemo; GO CREATE TABLE dbo.PSP_OrderDemo ( OrderID BIGINT IDENTITY PRIMARY KEY, CustomerID INT NOT NULL, OrderDate DATE NOT NULL, OrderAmount MONEY NOT NULL, Padding CHAR(200) NOT NULL ); GO CREATE INDEX IX_PSP_OrderDemo_CustomerID ON dbo.PSP_OrderDemo(CustomerID); GO Populate one customer with approximately 500,000 rows and the rare customer with only a few rows: -- Common customer INSERT dbo.PSP_OrderDemo (CustomerID, OrderDate, OrderAmount, Padding) SELECT TOP (500000) 90, DATEADD(DAY, -(ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 365), GETDATE()), 100.00, REPLICATE('X',200) FROM sys.all_objects a CROSS JOIN sys.all_objects b; GO -- Rare customer INSERT dbo.PSP_OrderDemo (CustomerID, OrderDate, OrderAmount, Padding) SELECT TOP (4) 1060, GETDATE(), 100.00, REPLICATE('X',200) FROM sys.all_objects; GO UPDATE STATISTICS dbo.PSP_OrderDemo WITH FULLSCAN; GO Create a parameterized stored procedure: CREATE OR ALTER PROCEDURE dbo.GetOrdersByCustomer @CustomerID INT AS BEGIN SELECT OrderID, CustomerID, OrderDate, OrderAmount, Padding FROM dbo.PSP_OrderDemo WHERE CustomerID = @CustomerID; END; GO Enable Actual Execution Plan in SSMS with Ctrl+M, then first test with PSP disabled. Because the rare value is compiled first, SQL Server may cache a plan optimized for four rows and reuse it when the same procedure returns approximately 500,000 rows. This illustrates the traditional parameter-sniffing problem. SQL ALTER DATABASE WideWorldImporters SET COMPATIBILITY_LEVEL = 160; GO ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = OFF; GO ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE; GO SET STATISTICS IO, TIME ON; EXEC dbo.GetOrdersByCustomer @CustomerID = 1060; -- Rare EXEC dbo.GetOrdersByCustomer @CustomerID = 90; -- Common GO Because the rare value is compiled first, SQL Server may cache a plan optimized for four rows and reuse it when the same procedure returns approximately 500,000 rows. This illustrates the traditional parameter-sniffing problem. Now enable PSP and repeat the workload: SQL ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = ON; GO ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE; GO EXEC dbo.GetOrdersByCustomer @CustomerID = 1060; EXEC dbo.GetOrdersByCustomer @CustomerID = 90; GO In the PSP-enabled plans, opening the execution plan XML should also reveal QueryVariantID and PLAN PER VALUE, confirming that SQL Server is maintaining separate plans for different cardinality ranges. The Parameter Sniffing Decision Tree When diagnosing and resolving parameter sniffing issues in production, follow this step-by-step decision framework: Summary of Best Practices Avoid blanket global fixes: Resist the urge to disable parameter sniffing database-wide via ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SNIFFING = OFF. Disabling it globally starves steady-state OLTP queries of optimal compilation behavior. Prioritize targeted query hints: Apply OPTION (RECOMPILE) for complex reporting queries run infrequently, and OPTION (OPTIMIZE FOR ...) for high-throughput transactional procedures with well-understood parameters.Establish baseline guardrails in Query Store: Leverage Query Store runtime statistics to track standard deviation spikes in avg_cpu_time and avg_logical_io_reads. Use plan forcing as an immediate temporary fix during active site incidents.Upgrade to compatibility level 160 where feasible: Capitalize on IQP features in SQL Server 2022 to handle non-uniform data distributions natively without littering codebases with query hints. Conclusion Parameter sniffing is not an engine defect. SQL Server’s default mechanism for maximizing plan reuse and minimizing compilation overhead. While skewed data distributions meet volatile execution requirements, captured plans can severely compromise application performance. To conclude, SQL Server 2022’s Parameter Sensitive Plan (PSP) optimization introduces welcome automation; enterprise database systems, at the same time, still require a deliberate, structured approach to query tuning.

Why Running It Locally The usual reason given is privacy, and it’s a real one. A prompt sent to a local model never leaves the machine, which changes the conversation with your security people from a negotiation into a non-event. But three other reasons matter more day to day. Cost goes to zero at the margin. Iterating on a prompt means running it fifty times. Against a hosted API, it's a line item and a rate limit. Against a local model, it’s free.Tests stop depending on the Internet. A build that reaches a hosted endpoint is a build that fails when someone modifies a key or a region has a bad afternoon.You learn where the abstraction leaks. Spring AI presents one ChatClient over every provider. Running a small local model is the fastest way to find out which parts of that promise hold and which parts quietly assume you are talking to a frontier model. The honest cost is quality and speed. An 8-billion-parameter model on a laptop isn’t a frontier model, and the numbers later in this article are not flattering. That trade is fine for development, prompt iteration, and tests, but it’s a decision to deliberately make for production. The Starting Point The initial set-up is the following: Java 25 (Temurin)Spring Boot 4.1.1Spring AI 2.0.1Local LLM runtime and model (details in the next section) Additionally, a freshly generated Spring Boot project, the state you get from start.spring.io before anything interesting has happened. Web MVC, Actuator, DevTools, their test companions and the Spring AI BOM: XML <properties> <java.version>25</java.version> <spring-ai.version>2.0.1</spring-ai.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-devtools</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc-test</artifactId> <scope>test</scope> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> One note on the parent: Spring AI 2.0 is built against Spring Boot 4.x and Spring Framework 7. If you are coming from Spring AI 1.x on Boot 3, this is not a version bump you can do in isolation; package names moved on both sides. Getting a Model onto the Machine In this article, Ollama is the choice: it downloads models, manages their lifecycle, and serves them over HTTP on port 11434. Let’s take it step-by-step. Install Ollama The Windows installer is a one-liner in PowerShell: PowerShell irm https://ollama.com/install.ps1 | iex # >>> Downloading Ollama for Windows... # ######################################## 100.0% # >>> Installing Ollama... # >>> Install complete. Run 'ollama' from the command line. Run Ollama the First Time ollama run pulls the model if it is missing, loads it, and takes you into an interactive prompt. The daemon keeps serving on port 11434 after you leave it: PowerShell ollama run llama3.1 # pulling manifest # pulling 667b0c1932bc: 100% ▕████████████████████████████████████████▏ 4.9 GB # pulling 0ba8f0e314b4: 100% ▕████████████████████████████████████████▏ 12 KB # pulling 455f34728c9b: 100% ▕████████████████████████████████████████▏ 487 B # verifying sha256 digest # writing manifest # success The one-line Windows command below confirms the model is present: PowerShell ollama list # NAME ID SIZE MODIFIED # llama3.1:latest 46e0c10c039e 4.9 GB 12 days ago The detail that matters for Spring AI is that Ollama serves two APIs at once. Its native API lives under /api and an OpenAI-compatible one lives under /v1. Prove the second is alive before you write any Java, as it saves you from debugging a Spring context when the problem is a daemon: PowerShell curl http://localhost:11434/v1/models # {"object":"list","data":[{"id":"llama3.1:latest","object":"model","created":1788341287,"owned_by":"library"}]} Two Ways to Wire It Up Spring AI ships both a dedicated Ollama starter and an OpenAI one that will successfully talk to anything speaking the OpenAI protocol. Both work against the same daemon. Putting both on the classpath and choosing between them with a property is a good way to feel the difference: XML <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-openai</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-ollama</artifactId> </dependency> Two starters on the classpath would normally mean two competing ChatModel beans. Spring AI resolves that with spring.ai.model.chat property, which names the provider that is currently chosen. The other starter’s auto-configuration backs off. Properties files # Specifies the used starter spring.ai.model.chat=ollama # Native Ollama starter (active when spring.ai.model.chat=ollama) spring.ai.ollama.base-url=http://localhost:11434 spring.ai.ollama.chat.model=llama3.1 spring.ai.ollama.chat.temperature=0.7 # OpenAI-compatible route (active when spring.ai.model.chat=openai) # Ollama does not check the API key, but the starter requires one to be set. spring.ai.openai.api-key=ollama # The /v1 must be part of the base-url: the underlying openai-java client # appends only /chat/completions, so omitting it yields a 404. spring.ai.openai.base-url=http://localhost:11434/v1 spring.ai.openai.chat.model=llama3.1 spring.ai.openai.chat.temperature=0.7 A note is worth making on spring.ai.openai.api-key=ollama. Ollama ignores the value entirely, but the starter refuses to configure itself without one, so it needs a placeholder. Any non-empty string does the job. Another aspect worth noting is the base URL needs its own /v1. The natural thing to write is spring.ai.openai.base-url=http://localhost:11434, on the assumption that the client appends the full /v1/chat/completions path. It does not. Spring AI 2.0 delegates to the official openai-java SDK (4.49.0 here), which appends only /chat/completions. Get this wrong and every call comes back as: Plain Text com.openai.errors.NotFoundException: 404: 404 No hostname, no path, no hint, just the status code twice. The fix is to make /v1 part of the base URL, as above. The same applies to any other OpenAI-compatible server you point this at. Flipping to the OpenAI-compatible route needs no code change at all: PowerShell ./mvnw spring-boot:run -Dspring-boot.run.arguments=--spring.ai.model.chat=openai Both spring.ai.ollama.chat.temperature and spring.ai.ollama.chat.options.temperature are valid, although the latter is deprecated. The starter’s configuration metadata registers both spellings, so your IDE will not flag either. Pick one and be consistent, because a typo in this family of properties fails silently rather than at startup. One Client, Built Once With a provider configured, Spring AI auto-configures a ChatClient.Builder. The temptation is to build a client inside the controller. Resist it; the client carries defaults that belong to the application, not to one endpoint and ChatClient is thread-safe and meant to be built once and shared. So it becomes a bean: Java @Configuration public class ChatConfig { @Bean public ChatMemory chatMemory() { return MessageWindowChatMemory.builder() .maxMessages(10) .build(); } @Bean public ChatClient chatClient(ChatClient.Builder builder, @Value("classpath:/prompts/system-prompt.st") Resource systemPrompt, ChatMemory chatMemory) { return builder .defaultSystem(systemPrompt) .defaultAdvisors(new SimpleLoggerAdvisor(Ordered.LOWEST_PRECEDENCE), MessageChatMemoryAdvisor.builder(chatMemory).build()) .build(); } } Three decisions are packed into those twenty lines. Prompts Live in Files defaultSystem() accepts a Resource, so the system prompt is a StringTemplate file under src/main/resources/prompts rather than a string literal in Java. It reads better, diffs better, and can be edited without recompiling: Plain Text You are a generalist. Answer as briefly and precisely as you can both general and specific questions. A system prompt is worth setting even for a toy. Small local models are chatty by default, and a brief system message is the difference between a sentence and five paragraphs of preamble. Memory Is an Advisor Every ChatClient call is stateless unless something makes it otherwise. MessageChatMemoryAdvisor is that something. Before each call, it loads the history for a conversation and appends it to the prompt, and after each call it stores the new user and assistant messages. The store behind it here is MessageWindowChatMemory, an in-memory map keyed by conversation id that keeps the last ten messages and drops the oldest when the window fills. Which conversation a request belongs to is decided per call, in the next chapter. Logging Sits Closest to the Model SimpleLoggerAdvisor logs the request and the response at DEBUG. Its position in the chain is set by the order argument and Ordered.LOWEST_PRECEDENCE puts it last, after the memory advisor has done its work, so what it logs is the prompt the model actually receives, history and format instructions included. It stays quiet until you switch its logger on: Properties files logging.level.org.springframework.ai.chat.client.advisor=DEBUG That one line is the most useful debugging aid in this article. When a model misbehaves, the first question is always “what did it actually see?” and this answers it. The Endpoint The controller is thin. It resolves a conversation ID from an optional header, delegates to a service, and echoes the ID back so the caller can continue the same conversation on the next request: Java @RestController @RequestMapping("/api") class ChatController { private final ChatService chatService; public ChatController(ChatService chatService) { this.chatService = chatService; } @GetMapping("/v{version}/chat") public ResponseEntity<Response> chat(@RequestParam String message, @RequestHeader(value = "X-Conversation-Id", required = false) String conversationId) { String id = conversationId == null ? UUID.randomUUID().toString() : conversationId; var response = chatService.respond(message, id); return ResponseEntity.ok() .header("X-Conversation-Id", id) .body(response); } } The v{version} segment is not a hand-rolled convention. Spring Framework 7 has first-class API versioning and Boot 4 configures it from properties: Properties files spring.mvc.apiversion.use.path-segment=1 spring.mvc.apiversion.supported=1.0 spring.mvc.apiversion.default=1.0 The first line tells MVC that the version is the path segment at index 1, in /api/v1/chat that is v1, and the default parser understands the leading v. The second lists what the application supports, so a request for /api/v2/chat is rejected before any controller runs. The third is the version assumed when a request carries none. The service is where the ChatClient fluent API earns its keep. One chained call covers the prompt template, the conversation, the tools and the output type: Java @Service public class ChatService { private final ChatClient chatClient; private final Resource userPrompt; private final TenantService tenantService; public ChatService(ChatClient chatClient, @Value("classpath:/prompts/user-prompt.st") Resource userPrompt, TenantService tenantService) { this.chatClient = chatClient; this.userPrompt = userPrompt; this.tenantService = tenantService; } public Response respond(String message, String conversationId) { return chatClient.prompt() .user(userSpec -> userSpec.text(userPrompt) .param("message", message)) .advisors(advisorSpec -> advisorSpec.param(ChatMemory.CONVERSATION_ID, conversationId)) .tools(tenantService) .call() .entity(Response.class); } } Line by line, the user message is a template with a {message} placeholder, filled with .param(), which keeps user input out of the template’s structure and the template itself readable: Plain Text Provide a short, to-the-point answer to the below message: {message} The advisors() block hands the conversation id to the previously mentioned memory advisor, under the well-known key ChatMemory.CONVERSATION_ID. Two requests with the same X-Conversation-Id header share history. A request without the header starts a fresh conversation and is told its id in the response. tools() and entity() get their own chapters. Plain Text GET /api/v1/chat?message=In one sentence, what is quantization of an LLM? LLAMA3.1: Quantization of a large language model (LLM) refers to the process of approximating its weights and activations with lower-precision data types (such as integers or fixed-point numbers) to reduce memory usage and computational overhead, while minimizing the impact on model performance. If you are running with DevTools and edit application.properties or a prompt file under src/main/resources, nothing happens until the file is copied into target/classes. DevTools watches the build output, not the source tree. A rebuild in your IDE triggers it; editing the file in a plain text editor doesn’t, and you’ll spend a while wondering why your fixed base URL is still returning 404. Answers as Java Records Text is informative, but typed objects are the reason to put a model behind a Spring service. entity() takes a class, derives a JSON schema from it, appends format instructions to the prompt, and parses what comes back. The endpoint returns this: Java public record Response( @JsonPropertyDescription(""" The category of the request message. \ SPECIFIC when the message refers to a particular tenant, identified by its unique identifier, \ so that answering requires the tenant information tool. \ GENERAL for any other message, such as general knowledge or technical questions \ that can be answered without tenant data.""") ResponseCategory category, @JsonPropertyDescription("The actual response message text") String text) { } public enum ResponseCategory { GENERAL, SPECIFIC } The @JsonPropertyDescription annotations are not decoration. They flow into the generated schema and the schema is what the model reads, so a description is the only place a field’s meaning gets explained. The enum becomes an enum constraint in the schema, which is why the model reliably answers with one of the two constants rather than a synonym. The category description is deliberately a paragraph, not a label. Giving the Model a Tool A model knows nothing about your data specifics (tenants’ information as part of this experiment). Tool calling is how it asks. You describe a method; the model decides whether to call it and with what arguments, Spring AI executes it and feeds the result back, and the model writes its answer from that. The description is the interface; it’s all the model sees: Java @Service public class TenantService { private static final Logger logger = LoggerFactory.getLogger(TenantService.class); private final Map<String, String> tenantInfo = Map.of( "TNG", "Tenant TNG is a well known TEM services provider.", "FED", "Tenant FED is Roger Federer's foundation.", "DJO", "Tenant DJO is Novak Djokovic's foundation." ); @Tool(name = "get-tenant-info-by-id", description = "Retrieve additional information by tenant id") public String infoByTenantId(@ToolParam(description = "The unique identifier of a tenant") String tenantId) { logger.info("Retrieving additional info for tenant with id {}", tenantId); return tenantId == null ? "N/A" : tenantInfo.getOrDefault(tenantId, "N/A"); } } Passing the bean to .tools(tenantService) in the service is all the registration there is. Spring AI scans it for @Tool methods and builds the JSON schema for their parameters from the @ToolParam descriptions. The log line is deliberate. Whether the model decided to call the tool and what it passed, is otherwise invisible. The null guard matters too. A model that decides to call the tool but fails to extract an argument sends nothing for tenantId and a Map.getOrDefault(null, …) on an immutable map throws. Returning "N/A" instead lets the model explain the gap in its own words, which is exactly what happens in the first test case below. Testing Without a Model A local model makes tests possible offline, but it does not make them good tests. The output varies run to run, and every assertion becomes a guess about phrasing. For anything that checks your code rather than the model’s, replace the ChatModel bean with a mock. Spring AI‘s auto-configuration builds the ChatClient.Builder from whatever ChatModel is in the context, so a @MockitoBean is enough to take the daemon out of the picture entirely: Java @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.MOCK) @AutoConfigureMockMvc class ChatControllerTests { @MockitoBean private ChatModel chatModel; @Autowired private MockMvcTester mvc; @BeforeEach void init() { // ChatClient copies the model's defaults into every request, so a bare mock does not work when(chatModel.getOptions()) .thenReturn(ChatOptions.builder().build()); } @Test void chat() { var message = "Local models keep your prompts on your own machine."; var modelOutput = """ {"category": "GENERAL", "text": "%s"} """.formatted(message); var response = new ChatResponse(List.of(new Generation(new AssistantMessage(modelOutput)))); when(chatModel.call(any(Prompt.class))) .thenReturn(response); assertThat(mvc.get() .uri("/api/v1/chat") .param("message", "Why run locally?")) .hasStatusOk() .bodyJson() .convertTo(Response.class) .isEqualTo(new Response(ResponseCategory.GENERAL, message)); } } The mock returns the JSON a model would return, and the test asserts that your controller, service, template and record mapping serve it back as a Response. The model’s ability to produce that JSON is a separate question and not one a unit test should be asking. Two things are quietly bypassed here, and it is worth knowing which. Tool execution happens inside the concrete ChatModel implementation, so a mocked model never calls TenantService. The tool is registered but inert. The memory advisor, on the other hand, is real; it runs, stores the exchange in the in-memory window under a freshly generated conversation ID, and does no harm. A note on the chatModel mock is worth making. If only mocking ChatModel and stubbing just call(), the first request fails with: Plain Text NullPointerException: Cannot invoke "ChatOptions.mutate()" because the return value of "ChatModel.getOptions()" is null ChatClient reads the model’s default options and merges them into every request. Mockito returns null for the unstubbed getter and the merge breaks. One line in a @BeforeEach fixes it, but the stack trace points into Spring AI internals rather than at your test, so it costs more time than it should. Testing the Integration With the application started, four requests on the same conversation exercise the whole chain: template, memory, tool, structured output. The X-Conversation-Id header ties them together. What each one did is known from the log line in TenantService and from the advisor log previously switched on. Use Case 1: Ask by “id” PowerShell curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981" --data-urlencode "message=Provide info for tenant with id FED" # {"category":"GENERAL","text":"Tenant information not available"} The tool was invoked, but with no tenantId. The model recognized that the question needed the tool and then failed to map "FED" onto the parameter, so the method returned N/A and the model paraphrased it. Use Case 2: Same Question, Phrased With the Parameter’s Own Description PowerShell curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981" --data-urlencode "message=Provide info for tenant with unique identifier FED" # {"category":"GENERAL","text":"Tenant FED is Roger Federer's foundation."} Tool invoked, tenantId mapped to FED, correct answer. The only change was “id” becoming “unique identifier,” the exact words in the @ToolParam description. Use Case 3. The Winning Phrasing, Different Tenant PowerShell curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981" --data-urlencode "message=Provide info for tenant with unique identifier DJO" # {"category":"GENERAL","text":"Tenant DJO is an abbreviation for DJ Okaya."} The tool was not invoked. The model answered from nothing, fluently and wrongly. Use Case 4: The Identical Request, Sent Again PowerShell curl -G http://localhost:8080/api/v1/chat -H "X-Conversation-Id: 123-1bc-981" --data-urlencode "message=Provide info for tenant with unique identifier DJO" # {"category":"GENERAL","text":"Tenant DJO is Novak Djokovic's foundation."} This time the tool was invoked, tenantId mapped to DJO, and the answer is right. Nothing changed between use cases 3 and 4 except the conversation history being one exchange longer. Four requests, one tool, three different behaviors: called without arguments, called correctly, and not called at all, the last two for the same input. The temperature of 0.7 explains some of the variance, but the deeper issue is that an 8-billion-parameter model is weak at the two judgment calls tool use requires: should I call this, and what goes in the parameter. Two mitigations are visible in the results. Wording the user’s request in the same vocabulary as the @ToolParam description made the difference between use cases 1 and 2, so descriptions should use the words users will use. And the null guard in TenantService turned a would-be exception into a polite non-answer. Neither makes use case 3 go away, that probably needs a lower temperature, a system prompt that instructs the model to always use the tool for tenant questions, or a bigger model. What It Actually Feels Like Measured on a laptop, Intel Core Ultra 7 155H, 32 GB RAM, RTX 500 Ada, running llama3.1 8B through the endpoint above: 14.2s – first request after startup, three-bullet answer11.3s – same request, model already warm Double-digit seconds for a paragraph and a tool call adds a second round trip to the model on top. That’s workable for development and unacceptable for a request a user is waiting on, which is the honest headline: local models are a development and testing tool at this size. The quality gap is sharper than the speed gap, and it’s worth seeing rather than being told about. Asked a question about the very tool serving it: PowerShell GET /api/v1/chat?message=In one sentence: what is Ollama? LLAMA3.1: Ollama is a brand of vegan, plant-based egg substitute made from a combination of pea protein, potato starch and other ingredients. Confident, fluent, well-formed and entirely invented. The model’s training data predates the tool it’s running inside. Put that next to "DJ Okaya" from before and the pattern is clear. This is the most useful thing about developing against a small local model: the failure modes you will eventually have to design around are loud here instead of rare. Whatever guardrails, validation, and grounding your application needs in production, you will notice you need them on day one. Resources [1] – POC code is here [2] – The picture was taken in Zakynthos, Greece

By Horatiu Dan CORE

I wanted a Nest Hub that I could actually program. Something on the kitchen wall that shows the time, the weather, my teams, what's cooking and what's playing, and that answers when I talk to it. That is Dash: an open-source ambient display for any tablet or screen in kiosk mode, one config.json, six languages, free data sources, MIT. Most of Dash needs no account at all. The part that does is the one I want to talk about here: the "Hey Dash" voice assistant, which runs on Google's real-time voice model, Gemini 3.8 Live, through the Gemini Live API. If you have ever wanted to build a voice assistant without stitching speech-to-text, an LLM, and text-to-speech together yourself, this is the article. You will see how the Live API works at the protocol level, how Dash keeps the API key off the device, how tool calls turn speech into shopping list items and timers, and what a question costs. Everything below is the real code from the repository, as of October 2026, against gemini-3.8-live. What Dash Is Dash is an Express server plus a Vite and vanilla TypeScript front end, built to be left alone on a wall for weeks. Panels rotate every minute: Home with a "Today" strip (weather warnings, holidays, your teams playing today, reminders, bin day, the cheapest electricity hour in Spain, the moon), Agenda from any iCal link, news, markets with sparklines, Spotify with synced lyrics, a kitchen that suggests six recipes a day and walks you through them one step at a time, and live sports boards that take over the screen when one of your teams plays, with a celebration when they score. Everything on screen also has an HTTP endpoint, so an iOS Shortcut or Home Assistant can add to the shopping list or start a timer. And then there is the voice layer, which is where Gemini comes in. Why the Live API and Not a Pipeline The classic way to build a voice assistant is a pipeline: a wake word engine, a speech-to-text service, an LLM call, a text-to-speech service, and a lot of glue to make the whole thing feel responsive. You can build Dash that way. You will spend most of your time on the glue, and it will still feel like a walkie-talkie. The Gemini Live API collapses that pipeline into one stateful WebSocket. You stream raw microphone audio in, and the model streams spoken audio back, with transcripts of both sides, tool calls in the middle, and Google Search when it needs a current fact. This is what that buys you: ConcernPipeline (STT + LLM + TTS)Gemini Live APILatencyThree sequential network calls per turnOne bidirectional stream, audio starts before the sentence is finishedInterruptionsHand-rolled: detect speech, cancel TTS, re-promptBuilt in: barge-in, interrupted event, playback stopsLanguageConfigure STT and TTS per language, detect yourselfReply in whatever language the user spoke, 70+ languagesTranscriptsTwo separate services, two formatsinputTranscription and outputTranscription on the same socketToolsParse LLM output, dispatch, re-prompt, re-synthesizetoolCall and toolResponse messages inside the sessionCurrent factsYour own search integrationgoogleSearch tool with groundingVoicePick a TTS vendor and voicevoiceName in the session config (Dash ships with Kore) The trade-off is control: you do not get to inspect the text between STT and TTS, because there is no text in between. For a kitchen assistant, that is the right deal. How "Hey Dash" Works Here is the whole shape of it. Three parties: the browser running the display, the Dash server on the same machine, and the Gemini Live API. The two rules that drive the design: The API key never leaves the server. The browser never sees it. For each conversation, the server mints a single-use ephemeral token with the entire session setup locked in (model, voice, system instructions, tools), and the browser connects to Gemini with that token.Audio never touches the server. The browser talks to Gemini directly over WebSocket: 16 kHz PCM up, 24 kHz PCM down. The server's only job during a conversation is to run the tools Gemini asks for. Dash is on the kitchen wall, so a third rule follows from the first two: Gemini is only on the line while you are talking to it. Waiting for the wake word is free and runs in the browser. What You Need Node.js 22+ (Dash uses node --test, native fetch, and the WebSocket global)A Gemini API key from Google AI Studio, in a .env file next to config.jsonA browser with the Web Speech API for the wake word (Edge or Chrome; Firefox has no speech recognition, so you tap the microphone button)A tablet or screen in kiosk mode, with microphone access allowed for http://localhost:3000 Shell git clone https://github.com/xavidop/dash && cd dash npm install cp config.example.json config.json echo "GEMINI_API_KEY=your-key" > .env npm run build && npm start # http://localhost:3000 Then switch on Settings → General → Voice assistant (or set features.assistant to true in config.json). The rest of this article walks through what happens from the moment you say "Dash". Step 1: Wait for the Wake Word Without Paying for It A Live session is billed per minute of audio, so you do not want it open all day. Dash uses the browser's own speech recognition to wait for the wake word, and only opens a Gemini session once it hears it. This is the relevant part of web/src/assistant.ts: TypeScript function startWake() { if (!cfg() || wakeBlocked || phase !== 'off' || wakeOn) return; const SR = (window as any).SpeechRecognition || (window as any).webkitSpeechRecognition; if (!SR) return; startMic(); // keeps the pre-roll buffer filling rec = new SR() as Recognition; rec.lang = getState().config?.locale || defaultLocale(); rec.continuous = true; rec.interimResults = true; const re = wakeRegex(); rec.onresult = (e: any) => { for (let i = e.resultIndex; i < e.results.length; i++) { // Cooking: short commands ("next", "siguiente paso") act at once, without Gemini. if (e.results[i].isFinal && cookingCommand(norm(e.results[i][0].transcript))) return; if (re.test(` ${norm(e.results[i][0].transcript)} `)) { restartDelay = 300; open(true); return; } } }; // ... } Two details matter here. The wake regex is forgiving. A recognizer set to Spanish or German hears "Dash" as "das", "dach" or "dasch". Exact wake words match anywhere; sound-alikes only count after "hey", "oye", "ok" and friends, because "das" is just a word in German: TypeScript function wakeRegex() { const words = (cfg()?.wakeWords || ['dash']).map(norm).map((w) => w.trim()).filter(Boolean); const alike = words.flatMap((w) => (w === 'dash' ? ['dasch', 'dach', 'dache', 'dass', 'dahs', ...(getState().config?.language === 'de' ? [] : ['das'])] : [])); const exact = words.map(esc).join('|'); const loose = alike.map(esc).join('|'); return new RegExp(`(?:^|\\s)(?:${exact})(?:\\s|$)${loose ? `|(?:^|\\s)(?:hey|hei|ei|ey|oye|ok|okay|hola|hallo|ehi|eh)\\s+(?:${loose})(?:\\s|$)` : ''}`); } The microphone keeps a 2.5-second ring buffer in memory. People say "Hey Dash, what's the weather?" in one breath. By the time the recognizer fires, the question is already half spoken. An AudioWorklet converts the mic to 16-bit PCM at 16 kHz in 100 ms chunks, and the last 25 chunks are kept so the whole sentence reaches Gemini when the session opens: TypeScript const IN_RATE = 16000; const CHUNK = 1600; // 100 ms const PREROLL_CHUNKS = 25; // 2.5 s of audio before the wake word fired node.port.onmessage = (e) => { ring.push(e.data); if (ring.length > PREROLL_CHUNKS) ring.shift(); onChunk?.(e.data); }; Nothing in that buffer leaves the device before the wake word. What the browser's own speech service hears while waiting is a different matter, and Dash is explicit about it in docs/voice-assistant.md: in Edge and Chrome, recognition runs on the browser vendor's online service, not on Gemini, and Dash stores none of it. Cooking mode adds a shortcut: while a recipe is open, "next", "back", "repeat", "start the timer," and "stop cooking" (in each of the six languages) are matched locally and act instantly, without opening a Gemini session at all. Step 2: Mint an Ephemeral Token on the Server This is the part of the Live API that most examples skip, and the one that makes a client-side connection safe. Instead of handing the browser your API key, your backend calls the token endpoint and gets back a short-lived credential that is locked to a full session setup. The browser can connect with it, but it cannot change the model, the voice, the instructions, or the tools. server/services/assistant.js: JavaScript const API = 'https://generativelanguage.googleapis.com/v1beta'; export const LIVE_WS = 'wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContentConstrained'; /** Mints a single-use token valid for 30 min (a new session must start within 1 min). */ async session() { if (!this.enabled) throw Object.assign(new Error('assistant disabled'), { status: 404 }); if (Date.now() - this.lastSession < 2000) throw Object.assign(new Error('too many sessions'), { status: 429 }); this.lastSession = Date.now(); const iso = (min) => new Date(Date.now() + min * 60e3).toISOString(); const res = await fetch(`${API}/auth_tokens`, { method: 'POST', headers: { 'x-goog-api-key': this.key, 'content-type': 'application/json' }, body: JSON.stringify({ uses: 1, expireTime: iso(30), newSessionExpireTime: iso(1), bidiGenerateContentSetup: this.setup() }), signal: AbortSignal.timeout(10000), }); const d = await res.json().catch(() => ({})); if (!res.ok || !d.name) throw Object.assign(new Error(d.error?.message || `token request failed (HTTP ${res.status})`), { status: 502 }); return { url: `${LIVE_WS}?access_token=${encodeURIComponent(d.name)}`, idleSeconds: this.config.assistant.idleSeconds }; } The fields, one by one: uses: 1: the token starts exactly one session.expireTime: how long messages can flow over a connection opened with it (30 minutes here; the API's default is also 30).newSessionExpireTime: how long you have to open that session (1 minute; again the default). The browser requests a token and connects immediately, so a minute is plenty.bidiGenerateContentSetup: the whole setup message the client would normally send on the WebSocket, baked into the token. In the SDKs this is called liveConnectConstraints; on the REST endpoint it is bidiGenerateContentSetup. The response is tiny, and it is all the browser ever gets from the server: JSON { "name": "auth_tokens/<redacted>" } Two things follow from this. First, the browser connects to the constrained WebSocket endpoint, BidiGenerateContentConstrained, with the token as access_token. That endpoint ignores any setup the client sends, so the browser cannot repurpose the token for something else. Second, the token endpoint is the only place that sees the API key, and Dash puts a guard in front of it: only the display itself (requests from localhost) may ask for a token unless assistant.allowRemote is set, so other devices on your Wi-Fi cannot spend your quota. The setup locked into the token is where the assistant gets its personality: JavaScript setup() { const a = this.config.assistant; return { model: `models/${a.model}`, // gemini-3.8-live generationConfig: { responseModalities: ['AUDIO'], speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: a.voice } } }, // Kore }, systemInstruction: { parts: [{ text: [ /* persona, rules, and this.context() */ ].join('\n') }] }, tools: this.spotify?.enabled && this.spotify.connected ? [TOOLS[0], { functionDeclarations: [...TOOLS[1].functionDeclarations, ...SPOTIFY_TOOLS] }] : TOOLS, inputAudioTranscription: {}, outputAudioTranscription: {}, }; } responseModalities: ['AUDIO'] is the only modality Gemini 3.8 Live speaks on this path; the two empty transcription objects ask for text transcripts of both sides, which is what the on-screen chat is built from. Note how the Spotify tools are only declared when Spotify is actually connected: fewer tools means fewer wrong tool picks. Step 3: Open the WebSocket and Stream Audio Back in the browser, the session is one WebSocket and a handful of message types. This is the core of open() in web/src/assistant.ts: TypeScript const res = await fetch('/api/assistant/session', { method: 'POST' }); const d = await res.json(); ws = new WebSocket(d.url); ws.onopen = () => ws?.send(JSON.stringify({ setup: {} })); The setup message is empty on purpose: everything is already in the token. Gemini answers with setupComplete, and from then on the browser pushes audio: TypeScript const sendAudio = (pcm: ArrayBuffer) => ws?.readyState === WebSocket.OPEN && ws.send(JSON.stringify({ realtimeInput: { audio: { data: b64(pcm), mimeType: `audio/pcm;rate=${IN_RATE}` } } })); if (m.setupComplete) { setPhase('listening'); if (prompt) ws?.send(JSON.stringify({ realtimeInput: { text: prompt } })); // "[briefing]" for (const c of pending) sendAudio(c); // the 2.5 s pre-roll onChunk = (pcm) => { if (!isSpeaking()) sendAudio(pcm); }; maxTimer = window.setTimeout(close, MAX_SESSION_MS); armIdle(); return; } Three things worth noticing: The pre-roll goes first. Those 25 buffered chunks are flushed before live audio starts, so the question you began before the wake word fired arrives intact.A text prompt can open the conversation. The scheduled morning briefing sends [briefing] as realtimeInput.text, and the system instruction tells the model what a briefing is. Same session, same voice, no audio needed to start it.Half-duplex. onChunk drops microphone audio while Dash is speaking. A wall tablet plays through speakers, and without reliable echo cancellation, the model would hear itself. You can still interrupt: tap the card and playback stops. The server's side of the conversation is a few message shapes, all handled in one onmessage: TypeScript if (m.toolCall) return runTools(m.toolCall.functionCalls || []); if (m.goAway) return close(); const sc = m.serverContent; if (!sc) return; if (sc.inputTranscription?.text) say('user', sc.inputTranscription.text); // what you said for (const p of sc.modelTurn?.parts || []) { if (p.inlineData?.data) play(p.inlineData.data); // 24 kHz PCM, base64 } if (sc.outputTranscription?.text) say('dash', sc.outputTranscription.text); // what Dash said if (sc.interrupted) stopPlayback(); if (sc.turnComplete) { /* arm the idle timer, or close if the model said goodbye */ } MessageMeaningsetupCompleteSession is ready; start sending realtimeInputserverContent.inputTranscriptionStreaming transcript of the user's speechserverContent.modelTurn.parts[].inlineDataChunks of 16-bit PCM audio at 24 kHz, base64serverContent.outputTranscriptionStreaming transcript of what the model is sayingserverContent.interruptedThe user barged in; stop playing what you have queuedserverContent.turnCompleteThe model finished its turntoolCall.functionCalls[]The model wants you to run functions and reply with toolResponsegoAwayThe server is about to close the connection Playback is plain Web Audio: decode the base64 into Int16, scale to floats, schedule each buffer right after the previous one on a 24 kHz AudioContext. The playHead bookkeeping is what lets Dash know how long it will still be speaking, which drives both the half-duplex gate and the idle timer. TypeScript function play(b64: string) { outCtx ??= new AudioContext({ sampleRate: OUT_RATE }); const bin = atob(b64); const pcm = new Int16Array(bin.length / 2); for (let i = 0; i < pcm.length; i++) pcm[i] = (bin.charCodeAt(i * 2) | (bin.charCodeAt(i * 2 + 1) << 8)) << 16 >> 16; const buf = outCtx.createBuffer(1, pcm.length, OUT_RATE); const ch = buf.getChannelData(0); for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 0x8000; const src = outCtx.createBufferSource(); src.buffer = buf; src.connect(outCtx.destination); playHead = Math.max(playHead, outCtx.currentTime + 0.05); src.start(playHead); playHead += buf.duration; speakingUntil = Date.now() + (playHead - outCtx.currentTime) * 1000; } Step 4: Let Gemini Act Through Tools A voice assistant that can only talk is a radio. The useful part is that Gemini can change what is on the screen. Dash declares its tools once, in the setup locked into the token, and runs them on the server. The declarations are plain function schemas, plus Google Search as a built-in tool: JavaScript const S = (description) => ({ type: 'STRING', description }); export const TOOLS = [ { googleSearch: {} }, { functionDeclarations: [ { name: 'add_to_shopping_list', description: 'Add one or more items to the household shopping list shown on the display.', parameters: { type: 'OBJECT', properties: { items: { type: 'ARRAY', items: S('One item, short, e.g. "milk" or "2 kg of oranges"') } }, required: ['items'] }, }, { name: 'start_timer', description: 'Start a kitchen timer on the display: for a duration ("10 minutes") or until a clock time ("at 18:30", an alarm). Up to 6 at once.', parameters: { type: 'OBJECT', properties: { minutes: { type: 'NUMBER', description: 'Duration in minutes (may be fractional). Omit when `at` is given.' }, at: S('Local clock time "HH:MM" to ring at (today, or tomorrow if already past). Omit when `minutes` is given.'), label: S('Optional label, e.g. "pasta"'), }, }, }, { name: 'remember', description: 'Save a fact the household wants Dash to remember (people, phone numbers, codes, preferences, where things are). Store it as one self-contained sentence.', parameters: { type: 'OBJECT', properties: { fact: S('E.g. "The plumber is Juan, phone 600 123 456"') }, required: ['fact'] }, }, { name: 'start_recipe', description: 'Open cooking mode on the display with a recipe: big step-by-step instructions the user follows while cooking. ... Every step that takes time must say how long in minutes, with a number ("cook the spaghetti for 9 minutes"): the display turns those into one-tap timers. Never write "until al dente" without a time.', parameters: { type: 'OBJECT', properties: { title: S('Dish name'), servings: S('E.g. "4 servings"'), ingredients: { type: 'ARRAY', items: S('One ingredient with quantity') }, steps: { type: 'ARRAY', items: S('One short step, with its time in minutes when it takes time') }, }, required: ['title', 'ingredients', 'steps'], }, }, { name: 'end_conversation', description: 'Close the assistant when the user is done (e.g. "thanks", "that is all"). Say a short goodbye first.' }, // ... remove_from_shopping_list, get_shopping_list, add_note, list_timers, add_time_to_timer, // cancel_timer, forget, open_saved_recipe, recipe_step, start_step_timer, stop_cooking ], }, ]; The full list is eighteen functions, plus three Spotify ones when it is connected. The start_recipe description is my favorite: it is a prompt disguised as a schema. Telling the model that every timed step needs a number of minutes is what makes the "Timer 8 min" button in cooking mode appear by itself. When the model decides to act, the browser receives toolCall, forwards each call to the server, and sends the results back as toolResponse: TypeScript async function runTools(calls: { id: string; name: string; args?: Record<string, unknown> }[]) { setPhase('thinking'); const functionResponses = []; for (const c of calls) { let response: unknown; try { response = await (await fetch('/api/assistant/tool', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ name: c.name, args: c.args || {} }), })).json(); } catch { response = { error: 'tool failed' }; } if (c.name === 'end_conversation') ending = true; const done = actionText(c.name, c.args || {}, response as Record<string, unknown>); if (done) log.push({ role: 'action', text: done }); // the green receipt on screen functionResponses.push({ id: c.id, name: c.name, response }); } ws?.send(JSON.stringify({ toolResponse: { functionResponses } })); } On the server, tool() is a switch over the names, working against the local JSON database and the timers service. The responses are deliberately rich: add_to_shopping_list returns both what was added and the full list afterward, so the model can say "milk is already on it" instead of pretending it added a duplicate: JavaScript case 'add_to_shopping_list': { const added = this.db.addShopping(args.items); this.onChange('shopping'); return { added: added.map((x) => x.text), list: this.db.shopping().filter((x) => !x.done).map((x) => x.text) }; } case 'start_timer': { let seconds = Math.round(Number(args.minutes) * 60); const at = String(args.at || '').match(/^(\d{1,2}):(\d{2})$/); if (at) { const today = new Date().toLocaleDateString('en-CA', { timeZone: this.config.timezone }); let when = localToEpoch(`${today}T${at[1].padStart(2, '0')}:${at[2]}`, this.config.timezone); if (when != null && when <= Date.now()) when += 864e5; seconds = when == null ? NaN : Math.ceil((when - Date.now()) / 1000); // ring on the minute, never early } if (!(seconds >= 5 && seconds <= 24 * 3600)) return { error: 'give minutes (up to 1440) or a clock time "HH:MM"' }; const tm = this.timers.add(seconds, String(args.label || '').slice(0, 40)); return { started: true, minutes: Math.round(seconds / 6) / 10, endsAt: tm.endsAt, label: tm.label || null }; } Errors go back to the model as data, not as exceptions. "Which timer? say its label" is a tool response, and the model turns it into a question. One Gemini 3.8 Live detail you should know: function calls are asynchronous by default (NON_BLOCKING). The model can keep talking while a tool runs, and schedule the result for when it is idle or interrupt itself with it. For Dash's tools, which take milliseconds, the difference is invisible; for a slow tool you would set BLOCKING on the declaration. Step 5: Give Gemini the Household Context The system instruction is rebuilt for every session, and the second half of it is a snapshot of what Dash already knows. That is why "when does Arsenal play next?" is answered from the display's own data instead of a web search, and why "add it to the list" works when there is only one list. JavaScript /** What Dash knows right now, so "when do Valencia play?" doesn't need a web search. */ context() { const tz = this.config.timezone; const lines = [`Now: ${now.toLocaleString('en-GB', { timeZone: tz, dateStyle: 'full', timeStyle: 'short' })} (${tz}).`]; const w = this.store.get('weather')?.data; if (w) lines.push(`Weather now: ${w.current.temp}°C, ${w.current.desc}; today ${w.today.min}–${w.today.max}°C, rain chance ${w.today.precip}%.`); for (const c of teams.slice(0, 10)) { /* last result and next game per followed team */ } if (events.length) lines.push(`Calendar: ${events.map((e) => `${e.title || 'event'} (${e.allDay ? e.start : fmt(e.start)})`).join('; ')}.`); if (alerts.length) lines.push(`Weather warnings: ...`); if (hol?.today) lines.push(`Today is a public holiday: ${hol.today.name}.`); if (reminders.length) lines.push(`Reminders today: ...`); if (bins.length) lines.push(`Bins: ${bins.join('; ')}.`); if (news.length) lines.push(`Top headlines: ${news.map((n) => n.title).join(' | ')}.`); const cook = this.cooking?.current(); if (cook) { lines.push(`COOKING MODE is open: "${cook.title}" (${cook.servings || 'servings not given'}), on step ${cook.step + 1} of ${cook.steps.length}: "${cook.steps[cook.step].text}".`); lines.push(`Ingredients: ${cook.ingredients.join('; ')}.`); } if (music?.item) lines.push(`Spotify: ${music.playing ? 'playing' : 'paused'} "${music.item.name}" by ${music.item.by}...`); const memories = this.db.memories(); if (memories.length) lines.push(`Things the household asked you to remember:\n${memories.map((m) => `- ${m.text}`).join('\n')}`); lines.push(shopping.length ? `Shopping list: ${shopping.map((x) => x.text).join(', ')}.` : 'The shopping list is empty.'); if (timers.length) lines.push(`Running timers: ...`); return lines.join('\n'); } The memories deserve a word. "Remember that the plumber is Juan, 600 123 456" calls the remember tool, which stores one sentence locally. Every later session gets those sentences in the context, and the instruction tells the model to quote numbers and codes exactly. They live in Dash's own data/db.json, and Settings lists them so you can delete any. It is a simple mechanism, and it is enough: a kitchen does not need a vector database. The persona rules are short and shaped by the fact that the output is spoken: no lists, no markdown, no URLs, one or two sentences, answer first and then maybe a follow-up, never claim a tool ran without calling it, and reply in the language the user is speaking. Step 6: End the Conversation on Purpose A per-minute API needs a clear end. Dash closes a session in four ways: Silence. After the model finishes a turn, an idle timer waits for Dash to finish speaking plus idleSeconds (8 by default). If nothing was heard since, the session closes."Thanks." The end_conversation tool exists so the model can hang up gracefully: say goodbye, call the tool, and the browser closes once the goodbye has played.Five minutes. A hard cap, MAX_SESSION_MS, so a kitchen full of chatter cannot keep the meter running.goAway. When Gemini announces it is about to drop the connection, Dash closes cleanly instead of waiting for the error. TypeScript function armIdle() { clearTimeout(idleTimer); // Wait for Dash to finish talking, then give the user a few seconds to follow up. const wait = Math.max(0, speakingUntil - Date.now()) + (cfg()?.idleSeconds ?? 8) * 1000; idleTimer = window.setTimeout(() => (ending || !heardSinceTurn ? close() : armIdle()), wait); } Closing plays a short down-chime (two oscillators, no audio file), stops playback, restarts the wake-word recognizer a second and a half later, and tells the server to restore the Spotify volume. While the assistant is open, Dash lowers the music ("ducking") so the model does not have to shout over it. What Gemini 3.8 Live Changes Dash ships pinned to gemini-3.8-live, the stable Live model Google updated in September 2026. If you are coming from gemini-3.1-flash-live-preview, these are the differences that affect a voice assistant: Gemini 3.8 LiveInputs / outputsText, image, audio, video in; text and audio outContext131,072 input tokens, 65,536 output tokensFunction callingSupported, asynchronous by default (NON_BLOCKING); BLOCKING, SILENT, WHEN_IDLE, INTERRUPTED scheduling availableGoogle Search groundingSupported, 5,000 free requests a month shared across Gemini 3+ modelsThinkingInterleaved reasoning built in; thinking_level is not a setup option any moreProactive audioAlways on (the model decides when a sound is addressed to it); setting it to false is an errorAffective dialogRemoved from the APITranscriptionInput and output transcription, opt in per sessionVariantgemini-3.8-live-extended-thinking for multi-step reasoning alongside speech Two of those matter for Dash's design. Proactive audio being always on is why the pre-roll buffer and the half-duplex gate are enough: the model is already good at ignoring background noise and its own echo. And asynchronous function calling is why the "thinking" phase on screen is so short: the model does not stall while a tool runs. What It Costs and Where Your Audio Goes Gemini is on the line only during a conversation, and this is what each state costs: StateWhat runsGemini costWaiting for "Dash"The browser's speech recognitionnoneTalkingA live Gemini sessionper minuteClosedAfter 8 s of silence, on "thanks" or ✕, or after 5 minutes at mostnone On the paid tier, gemini-3.8-live audio is priced at $0.005 per minute in and $0.018 per minute out, with the first 5,000 Google searches a month included. A typical question is a few seconds each way, around half a cent. Twenty questions a day comes to roughly $3 a month. The free tier costs nothing, with rate limits, and Google may use free-tier data to improve its products, which is the main reason to pay. On privacy, the three facts you need are the three design rules from the architecture section: the key stays on the server, audio goes from the browser straight to Gemini only during a session, and the wake word runs on the browser's speech service. Memories are stored locally and sent to Gemini as context in each conversation, so do not store anything you would not say to it. Gemini Beyond the Voice: Recipes and Translation The Live API is the headline, but Dash uses plain generateContent on gemini-3.8-flash in two quieter places, both in the kitchen. Any recipe link, turned into steps. Most recipe sites embed a schema.org Recipe card, and that is what Dash parses first. For pages without one, Gemini reads the page text and returns the same shape, with structured output enforced by a response schema: JavaScript async function recipeWithGemini(text, url, { key, model, language }) { const res = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/${model}:generateContent`, { method: 'POST', headers: { 'x-goog-api-key': key, 'content-type': 'application/json' }, signal: AbortSignal.timeout(40000), body: JSON.stringify({ contents: [{ parts: [{ text: `Extract the recipe from this web page. Keep the original language (${language} if unclear). Steps are short and in order. If there is no recipe, return an empty title.\n\nURL: ${url}\n\n${text.slice(0, 30000)}` }] }], generationConfig: { responseMimeType: 'application/json', responseSchema: { type: 'OBJECT', properties: { title: { type: 'STRING' }, servings: { type: 'STRING' }, ingredients: { type: 'ARRAY', items: { type: 'STRING' } }, steps: { type: 'ARRAY', items: { type: 'STRING' } }, }, required: ['title', 'ingredients', 'steps'], }, }, }), }); const d = await res.json().catch(() => ({})); if (!res.ok) throw new Error(d.error?.message || `Gemini HTTP ${res.status}`); return normalizeRecipe(JSON.parse(d.candidates?.[0]?.content?.parts?.[0]?.text || '{}'), url); } "What to cook" in your language. The daily ideas come from TheMealDB, which is English only. With a key set and the display in another language, titles and full recipes are translated in one structured call, with the same responseSchema trick. No SDK in either case. Dash's whole Gemini integration is fetch against the REST API and one WebSocket, which is a good reminder of how small the surface actually is. Turning It On The assistant is a few lines of configuration on top of a running Dash: JSON "features": { "assistant": true }, "assistant": { "model": "gemini-3.8-live", "voice": "Kore", "wakeWords": ["dash"], "idleSeconds": 8, "allowRemote": false }, "briefing": { "time": "07:30", "weekdays": [1, 2, 3, 4, 5] } Any Gemini prebuilt voice works in voice (Kore, Puck, Charon, and the rest). wakeWords can be anything the browser's recognizer can hear reliably. briefing.time opens a session by itself at that time and asks for the morning summary: weather, calendar, your teams, bins, one headline. On a Windows kiosk, windows/edge-kiosk-policy.reg allows the microphone for http://localhost:3000 without a prompt. Then say "Dash", or "Hey Dash, what's the weather?" in one breath, or tap the microphone button at the bottom left. A blue dot on that button means the wake word is listening. Conclusion The Gemini Live API turns a voice assistant from a systems-integration project into an application project. The protocol is small: one token request on your server, one WebSocket in the client, PCM in, PCM out, and toolCall and toolResponse for everything that changes the world. Gemini 3.8 Live adds the pieces that used to be the hard part, interruptions, language detection, asynchronous tools and search grounding, and leaves you with the decisions that are actually yours: when to open a session, what the model is allowed to do, and what it knows about your house. Dash makes those decisions one way: free wake word in the browser, ephemeral tokens with the setup locked in, tools on the server, memories on disk, and a session that ends when you say thanks. The code is at github.com/xavidop/dash. Put it on a tablet, add your teams, and talk to your kitchen. Further Reading Dash on GitHub, and the voice assistant doc with everything it can doGemini Live API overviewEphemeral tokens for the Live APIGemini 3.8 Live model page, including the migration notes from the 3.1 previewGemini API pricingDash architecture and HTTP API Last updated: October 2026. Model versions referenced: gemini-3.8-live for the assistant, gemini-3.8-flash for recipes. Dash at commit 60e909f.

By Xavier Portilla Edo CORE

An AI agent is an "autonomous software program that uses artificial intelligence to make a plan, execute tools, and complete an action." Autonomy (through AI) plus tools is what makes an agent powerful, and these tools can range from a simple calculator to querying a sophisticated database. While autonomy makes things easier for humans, it only opens a big security hole. Because an agent can do almost anything, without proper guardrails, it can cause production outages. For example, in June 2025, a Replit agent deleted an entire database belonging to a SaaStr founder even though it was explicitly told to request permissions. This is why it is important to put proper guardrails around your agent. Guardrails can take the form of rules that prevent destructive tool calls or allow calls only to a specific list of domains. Another name for such a thing is "sandbox." A sandbox is a "security mechanism for separating running programs, usually in an effort to mitigate system failures and/or software vulnerabilities from spreading." There are multiple agent sandbox offerings. For example: Docker SandboxLangsmith SandboxOpen Sandbox Docker Sandbox Today, we will discuss Docker Sandbox. It has gained a lot of traction recently and plays an important role in Docker's AI strategy. Simply put, Docker Sandbox helps you run agents in an isolated environment. These sandboxes can run on your laptop or be hosted in the cloud. To understand how these sandboxes restrict agents, let's look at the architecture. First, there is a host machine. As we discussed earlier, this can be either your host or cloud infrastructure. When you launch a Docker Sandbox, it is primarily made up of two components. An AI agentDocker engine private to its own Sandbox. Each sandbox has its own Docker engine. There are multiple ways to implement a Sandbox architecture, but the industry is pivoting towards microVMs. This is primarily because of: Low resource overheadFaster bootup timesStrong isolationsReduced attack surfaceHyper-scaling A popular microVM tech stack involves Firecracker, Cloud hypervisors, and Kata containers. Now, coming to the communications, let's take a simple example. Imagine there is a hotel booking agent that searches and reserves hotels for customers. When a request comes in: Step 1: The agent makes a tool call to search for hotels. Step 2: The tool call leaves the micro VM to be verified against the Network policy. PowerShell sbx policy allow network api.search.foo.com sbx policy allow network api.search.bar.com Step 3: Above is an example of how you can define network policies for a Sandbox. In this case, we are allowing the agent to search under the foo domain and restrict its access to bar. Step 4: If the request comes for the foo domain, the network policy allows it to be forwarded to the particular website that serves the request. Let's look at how to create these Sandboxes on your laptop. Step 1: Installation PowerShell brew trust docker/tap brew install docker/tap/sbx Step 2: Sign in to your Docker account. PowerShell sbx login If you don't have one, you can create a free Docker account. Step 3: Run the sbx command. PowerShell sbx The first time you run it, you will see the following text: PowerShell Waiting for authentication... Signed in as sirivarma. Daemon started (PID: 58801, socket: /Users/svegiraju/Library/Application Support/com.docker.sandboxes/sandboxes/sandboxd/sandboxd.sock) Logs: /Users/svegiraju/Library/Application Support/com.docker.sandboxes/sandboxes/sandboxd/daemon.log Select a default network policy for your sandboxes: 1. Open — All network traffic allowed, no restrictions. ❯ 2. Balanced — Default deny, with common dev sites allowed. 3. Locked Down — All network traffic blocked unless you allow it. Use ↑/↓ or 1–3 to navigate, Enter to confirm, Esc to cancel. Network policy set to "Balanced". Default deny, with common dev sites allowed. I picked balanced so things like GitHub are on the allow list and everything else is denied by default. When you run sbx from a project folder, the sandbox has read/write permissions to everything in that folder. Step 4: Running sbx ls in a different shell will give the following output. PowerShell sbx ls PowerShell SANDBOX AGENT STATUS PORTS WORKSPACE docker-agent-svegiraju docker-agent running /Users/svegiraju Step 5: Running sbx policy ls will show the current list of allow and deny rules with the "Balanced" path we have chosen. PowerShell sbx policy ls PowerShell NAME TYPE DECISION RESOURCES default-ai-services network allow **.chatgpt.com:443 **.oaistatic.com:443 **.oaiusercontent.com:443 **.openai.com:443 api.anthropic.com:443 api.perplexity.ai:443 cdn.openaimerge.com:443 chatgpt.com:443 gemini.google.com:443 generativelanguage.googleapis.com:443 models.dev:443 nanoclaw.dev:443 platform.claude.com:443 play.googleapis.com:443 statsig.anthropic.com:443 default-package-managers network allow **.bun.sh:443 **.gradle.org:443 **.packagist.org:443 **.yarnpkg.com:443 apache.org:443 Step 6: In order to stop and remove the sandbox, issue the following commands PowerShell sbx stop my-sandbox sbx rm my-sandbox Step 7: Of course, you may want to diagnose issues within the sandbox or while running it. This is when we will use the "diagnose" command. PowerShell sbx diagnose Conclusion What we discussed so far shows why sandboxes matter and how to create one with Docker Sandbox. It is a free, open-source offering from Docker for individual use. In the upcoming blogs, we will discuss Cloud Sandboxes (running sandboxes in the cloud infrastructure instead of locally) and more complex configurations like Credentials, GPU passthrough, and environment files. All in all, Sandbox is an interesting project from Docker. Go try it out.

By Siri Varma Vegiraju CORE

Data integration used to be treated as back-office plumbing. Difficult, yes, and always operationally critical, but rarely seen as strategic. That has changed. The Data Integration Landscape 2026 maps this shift across three communication paradigms and every major vendor. The largest software companies in the world have paid a fortune to own the integration layer: IBM acquired Confluent for $11 billion, Salesforce acquired Informatica for $8 billion, Qlik absorbed Talend, TIBCO folded into Cloud Software Group, and Fivetran merged with dbt Labs. When integration vendors command prices like these, the integration layer has become a strategic asset, not a piece of infrastructure to take for granted. The pressure behind this has been building for a long time. Data volumes keep exploding, digitalization keeps pushing more of the business onto real-time rails, and the demand for fresh data has been rising for years. AI is the latest and most acute driver. Models and autonomous agents need data that is current, governed, and continuously flowing. A nightly batch job and an API that returns yesterday's state cannot support a system making decisions in the moment. An agent working from outdated data does not fail loudly. It confidently does the wrong thing. That raised the stakes on a problem enterprises have been wrestling with for a decade. What Are the Three Data Integration Paradigms? The three data integration paradigms are request-response, batch, and event streaming. The landscape is best understood not by vendor but by how data actually moves. Choosing which paradigm sits at the center is among the most consequential architectural decisions an enterprise makes. Request-response is the most familiar. A system asks another for data and waits. REST APIs, GraphQL, and API management and gateway products work this way. It suits transactional interactions like fetching a record or submitting an order, but it couples the caller to the callee: you must know where the other system is and wait for it to answer. Batch moves data in bulk on a schedule. It is reliable and well understood for historical loads, analytics, regulatory reporting, and AI model training. Its weakness is freshness. The data is always minutes or hours old, which breaks down the moment a system needs to act on what is happening right now. Event streaming moves data continuously as events occur, and Apache Kafka has become its de facto standard. When a transaction completes or a record changes, that event is published to a stream, and any system that needs it consumes independently, without asking the source for anything. It is decoupled, persistent, and real-time, which is why it has become the foundation for both modern integration and agentic AI. Apache Flink has emerged as the de facto standard for stateful stream processing on top of Kafka, covering real-time transformation, aggregation, and AI-enriched data pipelines. The Data Integration Landscape below maps every major vendor across these three paradigms. Filled bubbles are platform leaders; outlined bubbles are specialists and emerging players. Grouped by paradigm, not by ranking. The full analysis is available as a free PDF without registration. Why Should Event-Driven Architecture Be the Center of Data Integration? The central argument of this landscape is simple. Event-driven architecture belongs at the center, with request-response and batch as consumer interfaces on top of it. The reason is decoupling. In an event-driven model, the system that produces data does not need to know who consumes it, when, or how. Producers and consumers are independent, which means a slow or unavailable system does not block everything connected to it. This is most commonly implemented with event streaming platforms like Apache Kafka, but message brokers such as IBM MQ and Solace serve event-driven patterns too, particularly where guaranteed delivery and fine-grained routing matter more than high-throughput streaming. Most enterprises use all three paradigms at once, and that is correct. The mistake is treating the choice as either/or. A business unit running iPaaS workflows, an analytics team pulling batch exports, and an AI agent querying live state can all be served from the same event-driven backbone without coupling to each other or to the underlying systems. Each connects in the way that suits it: some consume the stream directly, others reach it through an API gateway that exposes the stream as an endpoint or pushes events to a webhook or function. No large enterprise runs a single integration platform, so the real question is never which one product to buy. It is which paradigm sits at the center, and how the rest connect to it without recreating the tangle of point-to-point links it was meant to replace. The full landscape works through the market vendor by vendor: Data Integration Landscape. Is Batch Integration Dead? No. Batch is still the right tool for historical loads, analytical refresh cycles, regulatory reporting, and AI model training. What has changed is where the new thinking happens. The innovation has moved to the streaming layer, where data is processed continuously and acted on at the right time and context, often in real time. The batch column of the landscape has no emerging row because new entrants are building for streaming and change data capture, not for scheduled pipelines. Batch endures. It is just no longer where the architecture is evolving. Which Architecture Decisions Matter More Than Vendor Selection? The consolidation wave points to three decisions that outweigh which vendor you pick. The three-paradigm suite is the new baseline. The major platform vendors now each cover all three paradigms in one ecosystem. Choosing one of them increasingly means committing to that ecosystem across API management, data integration, and event streaming at once. That lock-in is far more expensive to unwind than any single product swap. If you are making a five-year platform decision, map which paradigms each candidate actually covers before you compare feature lists. The market is moving toward event-driven architecture. Databricks added Kafka-compatible APIs to its Zerobus Ingest service, allowing existing Kafka producers to stream data directly into Databricks with a configuration change rather than code changes. Snowflake announced Datastream, its own Kafka-compatible service, now heading into private preview. Batch ETL vendors are adding change data capture. iPaaS vendors are adding event-driven capabilities. The direction is consistent because agents and operational systems cannot wait for batch cycles. Legacy integration shapes the architecture more than vendor selection does. Same-vendor shortcuts like Zero-ETL work inside one ecosystem, but they do nothing for the SAP iDoc pipeline, the mainframe, the EDIFACT exchange, or the hospital's HL7 feed. Real enterprise architecture absorbs legacy formats at the edge for years. A platform that connects cleanly to the future but needs custom work to reach the past is not a complete platform for most enterprises. How Does Data Integration Connect to Process Intelligence and Agentic AI? Data integration does not stand alone. It is the foundation beneath process intelligence and trusted agentic AI, the three layers I think of as the Trinity of modern data architecture. The data layer moves and governs the data. Process intelligence understands how the organization actually runs. Trusted agentic AI acts on both. Each layer depends on the one beneath it. The full vendor-by-vendor analysis covers the platform leaders, the specialists, and the emerging players across all three paradigms, plus the legacy integration challenge, industrial IoT, and the trends shaping the next two years. If you are making an integration decision this year, the Data Integration Landscape 2026 is the starting point. Begin with the paradigms, not the vendor list. Map which ones you actually need at the center and which belong at the edges. The vendor choice gets much easier once the architecture is clear. About this landscape: This is an independent analyst perspective. No vendor paid for inclusion or placement. No vendor reviewed its section before publication. I run an advisory practice through Kai Waehner GmbH and have held roles at Talend, TIBCO, and Confluent. I also serve as Global Field CTO at Kestra, a workflow orchestration vendor outside this landscape's scope. The complete methodology note is in the full landscape.

By Kai Wähner CORE