DZone Spotlight

If you've sat through more than one architecture review, you've probably sat through a protracted Kotlin-versus-Java debate that goes nowhere. The team spends weeks comparing Kotlin's concise syntax to Java's maturity, runs a poll on developer preference, and produces a decision matrix nobody fully trusts. Then the meeting ends, and nothing changes, because nobody agreed on what problem they were actually trying to solve. Choosing between infrastructure instruments, such as Kotlin and Java, is an investment decision, and like any investment decision, it needs a thesis. Four business pressures should drive the discussion: time-to-market, operational risk, talent economics, and cross-platform consistency. The Problem With Asking Kotlin or Java When the question of Kotlin or Java arises, syntax and feature comparisons dominate the discussion, because they're concrete and measurable, and a natural place to start. You can count lines of code, benchmark compile times, and run surveys on developer satisfaction, but none of that will tell you if those metrics connect to any business outcome you actually care about. A language can be a good technical fit and still be a poor investment for a particular system. A stable backend handling 50,000 transactions a day with a three-person maintenance team faces a different set of risks than a payments service changing weekly under regulatory pressure, and both look different again from a monorepo running the same business logic across Android, backend, and edge. The JVM language strategy question comes down to where complexity actually lives in the system. Ask where Kotlin meaningfully supports your modernization strategy, and where staying with Java makes better sense right now. Modern Java May Be the Right Answer For a stable, low-change system, modernizing the existing Java stack can address the underlying problem without introducing the costs of another language, toolchain, or learning curve. If the system's main constraints don't align with Kotlin's strengths, staying with Java may offer a better return. If you're considering Kotlin, first identify where its capabilities address a problem worth solving. Start With the Business Problem Drawing on Yuri Geronimus's experience as Senior Staff Engineer and Chief Architect for EMEA Enterprise Payments Platforms at Verifone, the Kotlin Enterprise Playbook frames Kotlin adoption as an enterprise investment decision rather than a language preference. It organizes the business case around four key levers: delivery throughput and time-to-market, defect-cost avoidance, legacy modernization, and talent economics. Time-to-Market If your delivery lead time affects revenue, product launches, or certification cycles, ask whether the team can deliver faster. Kotlin's concise syntax means less boilerplate per feature, and fewer null pointer exceptions reaching production translates to less time spent on hotfixes and incident response. Whether that translates to faster delivery depends on where your bottlenecks sit. If the constraint is product clarity or test coverage rather than boilerplate or type errors, language choice won't fix it. Operational Risk Where are your expensive defects concentrated? Kotlin's type system encodes nullability explicitly, allowing some null-related errors to be caught at compile time rather than in production. For systems where a single production bug triggers a compliance review or a customer-facing outage, that compile-time safety has measurable financial value. For a read-heavy reporting service where reliability is already strong, the business case is thinner. Talent Economics The cost of a JVM talent problem can show up in longer time-to-hire, higher onboarding overhead, attrition, or difficulty moving engineers between teams. Java developers can build on their existing JVM experience when learning Kotlin, making internal mobility a potential part of the business case. The strategic partnership between JetBrains and Spring, announced in May 2025, also means that 27% of Spring developers who've already used Kotlin are a potential internal resource pool you may not have fully mapped. Cross-Platform Consistency If your organization maintains duplicated business logic across an Android app, a backend service, and an edge deployment, that duplication is an operational risk disguised as a technical decision. Kotlin Multiplatform (KMP) can let teams share domain logic, models, and validation across platforms, potentially reducing duplicated implementation and the risk of logic drift. That also reduces the compliance and maintenance burden of finding and fixing the same bug in multiple places. Keep in mind that you should evaluate the case for KMP separately, with the scope of shared code kept deliberately narrow. Kotlin doesn't automatically improve time-to-market any more than a new CI tool does. The question is whether you've identified the constraint it's positioned to address. Measure the Outcome You Actually Care About To make a solid business case for Kotlin, you need a measurable hypothesis. Each business lever maps to an observable metric. Time-to-market connects to delivery lead time and deployment frequency (both DORA metrics your platform team likely already tracks). Operational risk connects to change failure rate, mean time to recovery, and incident cost. Talent economics connects to time-to-hire, time to first commit after onboarding, and attrition rate. Cross-platform consistency connects to duplicated-logic defect rate and the volume of bugs that required fixes in more than one codebase. Those metrics support the business case, but you still have to translate them into financial terms. For an enterprise investment decision, connect the operational improvement to a financial outcome. Faster delivery can reduce the cost of delay, fewer incidents can reduce remediation and customer-impact costs, and shorter onboarding can reduce hiring and training costs. Bear in mind that productivity gains don't translate directly into headcount savings. Faster delivery can instead create capacity for new product work or reduce the cost of delay on a revenue-generating feature. Conflating the two overstates the financial case and creates a credibility problem when someone on the finance team models it out. Also, don't count productivity gains as both lower operating costs and increased revenue in the same proposal. A hypothesis that says "we expect Kotlin's null safety to reduce null-related production incidents by X% over 12 months, based on our current incident log showing Y% of incidents trace to NPEs" is defensible. A hypothesis that says "Kotlin will improve developer productivity" gives you nothing to measure against. Interoperability Makes This a Scoped Decision The Kotlin-or-Java framing misleads because it implies a binary. Kotlin compiles to JVM bytecode and is fully interoperable with Java, so Kotlin and Java files can coexist in the same project without a rewrite. Your existing Spring Boot or Hibernate configurations stay in place. Your Java libraries continue working. A single new service, a shared utilities module, or even the test layer of an existing Java codebase are all valid starting points for introducing Kotlin. IntelliJ IDEA, JetBrains' own IDE, includes a Java-to-Kotlin converter and first-class mixed-project support, so the tooling overhead of running both is lower than teams typically assume. That shifts the question from "should we adopt Kotlin?" to "where should we introduce Kotlin first?" A new, high-change service with a measurable business outcome may be a reasonable candidate. A stable order-management system with a small team and clean Java code can probably wait on Kotlin until a clearer business case emerges. Account for the Iceberg and the J-Curve An honest transition plan accounts for costs that typically don't appear in the optimistic spreadsheet. The J-curve is real. Before a team reaches full Kotlin proficiency, velocity drops. Training takes time, mixed-language code reviews add overhead (reviewers need to understand both languages to catch idiomatic issues), and CI pipelines may need reconfiguration for multi-language builds. Compliance documentation may need updating if your audit trail references language versions or toolchain components. The eventual decommissioning of legacy Java code, if that's the direction, carries its own cost that's easy to leave out of a five-year model. Scope the initial experiment carefully, and build these costs into the transition plan upfront. A proposal that acknowledges the J-curve and plans for it presents a more credible case than one that doesn't. Three Questions for Your Next Architecture Review If your team is heading into an architecture review where Kotlin is on the agenda, these three questions cut through the syntax debate and turn it into a scoped, testable proposal. Where are the expensive defects or operational problems concentrated? Pull the last 12 months of production incidents and categorize the ones with the highest cost (in engineer time, customer impact, or compliance exposure). If a significant share traces to null-related errors or type-safety failures, Kotlin's type system is directly relevant. If the distribution looks different, the investment case needs a different justification. What does your JVM talent pipeline actually look like? Map your current team's Kotlin exposure, your open Java roles, and the average time-to-hire over the past year. If you're competing for senior Java engineers in a tight market and losing candidates to organizations with more modern stacks, that's a talent economics argument worth quantifying. If your team is stable and hiring is closed, the argument doesn't apply. Which service, module, or domain is the lowest-risk place to test whether Kotlin improves the target outcome? Look for something actively changing (so you'll have delivery-cycle data to measure), with a well-defined scope, and without deep compliance dependencies that would complicate a mid-stream language shift. New functionality is often the best candidate. Adding Kotlin tests to an existing Java service can also provide a relatively low-production-risk way to gain experience with the language and toolchain. These questions convert an abstract language debate into a scoped experiment with a falsifiable hypothesis. Run the experiment, measure the result, and expand where it works. Make the Strategic Decision Your CFO and CTO won't ever see a line item for Java versus Kotlin, but they will see the costs of delivery lead time, incidents, onboarding overhead, and duplicated-logic defects. The strategic decision is where to invest to improve those outcomes, domain by domain, with measurement in place. Evaluate Kotlin where you have a number to test against, start with the lowest-risk entry point, and leave stable systems alone when the case is weak. The goal is a better outcome where the investment is justified. If you want to see how other enterprise teams have structured this transition, JetBrains has a practical case for it at kotlinlang.org/lp/enterprise-playbook. How has your team navigated this decision? Share your experience in the comments.

Imagine an employee asks an AI assistant: Can I claim this hotel expense? The assistant cannot answer the question itself, but somewhere in the enterprise there is an Expense Agent that can. The challenge is not simply finding its endpoint. The assistant needs to determine: Does this agent handle expense-policy validation? How do I interact with it? What authentication does it require? In traditional software, APIs and integrations often define these relationships. In a multi-agent ecosystem, agents may need to discover and evaluate one another dynamically. And that is where Agent Cards come in. They provide a machine-readable description of an agent — its identity, capabilities, skills, interaction mechanisms, security needs, and other agents’ capabilities — so that others can get the information they need to learn about, understand, and interact with it. An endpoint tells an agent where to connect. An Agent Card helps it decide whether — and how — to connect. Agents Need More Than an Endpoint Traditional APIs are built on a contract: the client knows the endpoint, knows the interface, and can invoke a predefined operation. In an agentic ecosystem, that is only part of the problem. An agent may encounter another agent dynamically and must decide: Who are you? – identity, ownership, and versionWhat can you do? – capabilities and supported tasksHow can I work with you? – protocols, inputs, outputs, and interaction modesCan I trust you? – authentication, authorization, and other trust signalsAre you suitable for this task? – constraints and operational context An endpoint answers, “Where do I send the request?” It does not answer, “Should I interact with this agent, and how?” This is the gap that Agent Cards address: they provide a machine-readable description of an agent so that other agents can discover its capabilities and understand how to interact with it. What Is an Agent Card? An Agent Card is a machine-readable metadata document that describes an AI agent and the capabilities it exposes to other agents. In the Agent2Agent (A2A) protocol, it is a standardized way for an agent to advertise its identity, capabilities, skills, supported interaction modes, and security requirements. Think of an Agent Card as the agent’s self-description. It allows other agents, orchestrators, or platforms to know the things that the agent can do without having to know what it is or setting up integrations manually. A typical Agent Card captures: Identity – name, description, and basic information about the agentCapabilities – protocol-level features the agent supportsSkills – specific tasks or functions the agent can performInteraction modes – supported input and output formatsSecurity – authentication schemes and security requirementsConnection information – details required to communicate with the agent For example, an enterprise Expense Agent might advertise skills such as validate expense, check policy compliance, and submit reimbursement. Another agent can inspect this metadata and determine whether the Expense Agent has the capability required for a particular task. The Agent Card therefore acts as a standardized capability profile, providing the metadata needed for agents to discover, understand, and interact with one another programmatically. Anatomy of an Agent Card An Agent Card brings together an agent's metadata and its interaction with another agent. The structure depends on the A2A specification version, but the key parts can be grouped into a few areas: Identity – Describes the agent with information such as its name and description, helping other systems understand its purpose.Connection information – Specifies how the agent can be reached and the interfaces or protocols available for communication.Capabilities – Declares protocol-level features supported by the agent, such as streaming or other interaction capabilities.Skills – Defines the tasks the agent provides. A skill typically describes its purpose and may include supported input/output modes, examples, and tags.Security – Specifies the authentication mechanisms and security requirements that have to be followed when communicating with the agent.Metadata for discovery – Descriptions, tags, and skill information provide useful signals for registries, orchestrators, and other agents searching for an appropriate agent. The skills section is particularly important. It moves discovery beyond simply finding an agent to understanding what that agent can do. For example, an Expense Agent could expose skills such as validate-expense and submit-reimbursement, allowing an orchestrator to select it based on the task being performed. Together, these elements turn the Agent Card into a structured capability profile that can be interpreted programmatically across an A2A ecosystem. Agent Skills: Describing What an Agent Can Actually Do An Agent Card describes the agent, but skills describe the specific tasks the agent can perform. They provide a structured way for another agent or orchestrator to determine whether an agent is suitable for a particular request. A skill typically communicates: ID and name – a unique identifier and human-readable nameDescription – what the skill is designed to accomplishTags – keywords that help with discovery and classificationExamples – sample requests that illustrate when the skill should be usedInput/output modes – the types of content the skill can consume or produce Consider an Expense Agent that provides two skills: JSON "skills": [ { "id": "validate-expense", "name": "Validate Expense", "description": "Validates an expense against corporate policy", "tags": ["expense", "validation", "compliance"], "examples": [ "Check whether this hotel expense complies with policy" ] }, { "id": "submit-reimbursement", "name": "Submit Reimbursement", "description": "Submits an approved expense for reimbursement", "tags": ["expense", "reimbursement"] } ] These descriptions allow another agent to reason about which capability is relevant to the user's intent. For example: User: “Can I claim this ₹8,000 hotel expense?” Orchestrator: identifies the intent as expense policy validation → discovers the Expense Agent → finds the validate-expense skill → delegates the task Skills therefore provide the semantic layer for capability discovery. Instead of merely discovering that an Expense Agent exists, other agents can understand the individual tasks it offers and select the capability that best matches the work to be performed. How Agent Cards Enable Discovery In an agent ecosystem, discovery is not simply finding another agent, but rather finding the right agent for a particular task. Agent Cards do this by providing structured and machine-readable information about an agent and its skills. A typical discovery flow looks like this: User Intent → Discover Agents → Retrieve Agent Cards → Match Skills → Select Agent → Initiate A2A Interaction Consider a user asking: “Check whether my hotel expense complies with company policy.” An orchestrating agent can use available Agent Cards to identify agents whose advertised skills match the request. An Expense Agent exposing a validate-expense skill with tags such as expense, validation, and compliance becomes a strong candidate. Agent Cards support discovery at multiple levels: Agent-level discovery – identify agents relevant to a domain or purpose.Skill-level discovery – determine which agent provides the capability required for the task.Compatibility discovery – verify supported interaction modes and protocol capabilities.Security-aware discovery – understand authentication requirements before initiating communication. Importantly, the Agent Card provides the metadata for discovery; the location of Agent Cards may be different if we have to track them. Cards may be obtained from well-known locations, registries, catalogs, configuration, or other discovery infrastructure. This separation is powerful: discovery mechanisms can evolve independently, while the Agent Card provides a common description of what each discovered agent offers. Agent Discovery Is More Than Service Discovery Traditional service discovery is concerned with locating a service instance. Once a service name is assigned to it, Kubernetes DNS or service registries can guide a client to find services. Agent discovery has a broader objective. An agent does not necessarily know in advance which other agent should do a task. It may need to find candidates and reason about which one is the best fit. For example, given the request: “Check whether this travel expense complies with company policy.” An orchestrator might find several agents related to finance, travel, expenses, or compliance. Selecting the right one is more than just locating them. It involves knowing their advertised skills, descriptions, supported interaction modes, security requirements, and other capability metadata. The distinction can be summarized as: Service Discovery Service Name → Location → Connect Agent Discovery Intent → Candidate Agents → Capability/Skill Matching → Compatibility Check → Agent Selection → Interact This makes agent discovery much more semantic and task-oriented. The goal is not simply “Where is the service?” but in order to discover: “Which available agent is capable of performing this task, and can I interact with it appropriately?” Agent Cards provide the structured metadata needed to support this decision. The discovery infrastructure can locate candidate agents, while their Agent Cards allow an orchestrator to evaluate those candidates and select the most appropriate agent and skill. Public vs. Authenticated Agent Cards Not every agent should expose the same information to every caller. A2A supports the idea of making basic agent information available through a public Agent Card, while additional information can be made available to authenticated clients through an authenticated extended Agent Card. Public Agent Card The public Agent Card provides all the info as part of the initial discovery and interactions. It will advertise things like the agent's name, capabilities, skills, supported interfaces, and security requirements. For example, an enterprise Expense Agent might advertise: Expense Agent Skills: validate-expense, check-expense-policy This provides a potential client with enough information to determine if the agent is relevant and how authentication should be established. Authenticated Extended Agent Card Some information may only be appropriate after the caller has authenticated. An authenticated extended Agent Card can expose additional capabilities or metadata intended for authorized clients. For example: Publicvalidate-expense check-expense-policy After authenticationsubmit-reimbursement approve-expense access-finance-workflow This is particularly useful in enterprise environments where an agent may reveal some capabilities depending on the caller's identity, permissions, or organizational context. The principle here is progressive disclosure: Discover → Understand Security Requirements → Authenticate → Retrieve Extended Agent Card → Access Authorized Capabilities This allows organizations to balance discoverability with controlled exposure. Agents can advertise enough information to participate in an ecosystem without unnecessarily exposing sensitive or privileged capabilities to unauthenticated clients. Security and Trust Begin at Discovery Discovery answers “Which agent can do this task?” But for enterprise environments, capability alone is not enough. When dealing with a discovered agent, a system also needs to determine if it can be trusted and accessed safely. Agent Cards support this process by exposing security-related metadata and capabilities. A client can understand the authentication mechanisms required by the agent and establish the appropriate credentials before starting an A2A interaction. This makes security part of the discovery workflow rather than something only considered after an agent has been selected: Discover → Evaluate Capability → Evaluate Security Requirements → Authenticate → Interact An important distinction is that an Agent Card is not proof that an agent is trustworthy. Trust also depends on how the card was obtained, the identity of the agent or organization behind it, transport security, credential validation, authorization policies, and governance controls surrounding the interaction. In enterprise environments, discovery should therefore combine two questions: Can this agent perform the task?Should this agent be allowed to perform the task? Agent Cards provide important metadata for making that decision, while enterprise identity, authorization, policy, and governance mechanisms establish the broader trust boundary around agent-to-agent communication. Securing Agent-to-Agent Communication Agent discovery establishes which agent to interact with. The next problem is to make sure the interaction itself is secure. Once an agent has been selected, requests, responses, credentials, context, and potentially sensitive enterprise data can flow between agents. Such A2A communication should therefore protect four fundamental properties: confidentiality, integrity, authentication, and authorization. Common enterprise controls include: HTTPS/TLS encryption to protect data in transit.Service-to-service authentication to verify the identity of communicating agents.Token-based authorization to ensure an agent can invoke only permitted capabilities.Identity propagation when the originating user's or agent's identity must be preserved across interactions.Mutual TLS (mTLS) where both communicating parties need strong cryptographic authentication.Audit logging to maintain traceability of agent interactions. These controls are particularly important because an agent may invoke another agent autonomously. A successful discovery should therefore not imply authorization. An agent might advertise a capability while the requesting agent is not permitted to use it. Securing Asynchronous Interactions Agent-to-agent interactions may also be long-running. Instead of keeping a connection open, an agent may complete a task asynchronously and inform the requesting agent by a callback or push notification. In such cases, the notification endpoint should be validated, incoming notifications authenticated, messages protected from tampering, replay attacks prevented, and communication performed over secure channels. A useful way to think about the flow is: Discover → Authenticate → Authorize → Establish Secure Channel → Invoke → Validate Response → Audit The Agent Card helps communicate the security requirements needed to establish this interaction, but it is not itself the security enforcement mechanism. Authentication, authorization, transport security, policy enforcement, and auditing must still be implemented by the surrounding agent platform and enterprise security architecture. This distinction is important: the Agent Card describes how secure interaction should occur; the runtime infrastructure enforces it. Agent Card Lifecycle and Governance An Agent Card should not be treated as a create-once, static metadata file. As an agent develops, its capabilities, skills, endpoints, authentication requirements, and operational constraints may also change. The Agent Card must evolve with it. This calls for an Agent Card lifecycle that governs how cards are created, validated, published, updated, and eventually retired. A typical lifecycle can be represented as: Define → Validate → Publish → Discover → Monitor → Update → Deprecate Define and Validate When an agent is created or modified, its Agent Card should accurately describe its identity, capabilities, skills, endpoints, and security requirements. Before publication, organizations should validate the card for: required metadata and schema correctness;valid and reachable endpoints;clear and unambiguous skill descriptions;correct authentication requirements;consistency between advertised capabilities and actual agent behavior. This prevents agents from publishing incomplete or misleading discovery information. Publish and Discover Once validated, the Agent Card can be published through the appropriate discovery mechanism. Consumers can then use it to evaluate whether the agent is suitable for a particular task. Publication should also consider visibility. As discussed earlier, not every skill or capability necessarily needs to be discoverable by every consumer. Public, internal, and privilege-specific views may require different governance policies. Monitor and Update Agent Cards should remain synchronized with the agents they describe. If a new skill, alternate endpoint, authentication mechanism, or ability is changed, the Agent Card should update accordingly. This is particularly important because Agent Card metadata can influence runtime agent selection and routing. Outdated metadata could cause an orchestrator to select an inappropriate agent or attempt to invoke a capability that no longer exists. Version and Deprecate Significant changes should be versioned so consumers can identify compatibility changes. Skills or capabilities that are being removed should be explicitly deprecated before they disappear where practical, allowing dependent agents and orchestrators time to adapt. When an agent itself is retired, its Agent Card should also be withdrawn or marked accordingly so that it is no longer selected during discovery. When the agent itself is retired, its Agent Card should also be withdrawn or marked so that it is no longer selected during discovery. Governance Across the Lifecycle As multi-agent environments grow, Agent Cards become part of the organization's agent governance surface rather than just documentation. Organizations should therefore establish ownership and controls around: Who can publish an Agent Card? Who can modify it? Who approves newly advertised capabilities? How are changes tracked? Which consumers can discover which skills? Governance should also provide traceability around agent interactions. The key principle is simple: The Agent Card lifecycle should follow the agent lifecycle. When an agent changes, its discoverable contract must change with it. This makes discovery accurate, secure, and trustworthy as the agent ecosystem evolves. Agent Card Design Best Practices A well-designed Agent Card should make an agent easy to discover, correctly understood, securely accessed, and safely selected. Because its metadata may affect runtime discovery and routing, clarity and accuracy matter. A few practices help keep Agent Cards effective: Make the purpose clear: Give a brief name and description that immediately describes what the agent is doing.Define skills precisely: Describe specific tasks and when they should be used. Avoid vague names such as Analyzer or Helper. Skills are an important signal for discovery and selection.Separate capabilities from skills: Capabilities describe general protocol-level functionality; skills describe the specific tasks the agent performs.Use meaningful metadata: Tags, descriptions, examples, and input/output modes should help consumers determine whether a skill matches their intent.Make security requirements explicit: Clearly advertise authentication and access requirements so consumers understand how to establish a secure interaction.Apply least-privilege discovery: Expose only the capabilities and skills appropriate for the consumer's identity and authorization level. Keep the card synchronized: Update the Agent Card whenever skills, capabilities, endpoints, or security requirements change.Validate and version: Validate cards before publication and version significant changes so consumers can manage compatibility and deprecation. A good Agent Card should make five questions easy to answer: Who are you? → What can you do? → When should I use you? → How do I access you? → Can I trust and safely invoke you? Agent Cards in an Enterprise Multi-Agent Architecture In an enterprise multi-agent environment, Agent Cards become part of the overall discovery and interaction architecture. Instead of having agents know in advance about each other, an orchestrator or agent router can use published Agent Cards to discover candidates, evaluate their skills and capabilities, apply security and policy requirements, and select the appropriate agent for the task. Agent Card acts as the bridge between an agent's implementation and the enterprise mechanisms that make it discoverable, selectable, secure, and governable. Conclusion As AI agents become more autonomous and interconnected, knowing where an agent is no longer enough. Agents need a structured way to understand what other agents can do, how to interact with them, and under what security and governance conditions. Agent Cards provide this metadata layer, enabling discovery, skill-based selection, secure interaction, and interoperability across multi-agent ecosystems. In combination with appropriate lifecycle management and governance, they can help transform the independently deployed agents into a discoverable, trustworthy, and manageable agent ecosystem. The future of multi-agent systems is not just connected agents, but agents that can discover, understand, and trust one another.

By Josephine Eskaline Joyce, Ph.D CORE

There are two realistic choices for building AI services in Go today: LangChainGo, the Go port of the Python framework everyone knows, and Genkit Go, Google's open-source Gen AI framework. LangChainGo is still the first result most Go developers find. It has 9,677 stars and the biggest provider catalog in the ecosystem. It is also eight months without a commit, eleven months without a release, and still on 0.1.x after three and a half years. This article compares them the way that actually matters: by building the same production service in both and looking at what each one costs you. Not a feature checklist, a real application with real requirements. If you have already decided and want the migration mechanics, that is the companion article: How to migrate from LangChainGo to Genkit Go. The State of Both Projects LangChainGoGenkit GoLatest releasev0.1.14, October 2025v1.13.1Last commit to mainJanuary 2026activeAge and version3.5 years, still 0.1.xGA since v1.0Open issues242Open pull requests171, oldest from June 2023Stars9,677 LangChainGo is not archived, and "dead" overstates it. But eleven months without a release is long enough for the ecosystem to move underneath a library, and it has. What Is Broken in LangChainGo Today Three concrete things, all of which affect a service in production right now. The quickstart returns a 404. DefaultOptions() hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) fails on the first call. The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, whose README states support ended on November 30, 2025. Streaming is broken on every current Gemini model. gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash all fail the same way: Plain Text error in stream mode: invalid character ']' looking for beginning of value The OpenAI provider streams fine, so this is the Google provider on the legacy SDK, not the library everywhere. There is a fourth consequence that follows from the third, and it is the nastiest one. LangChainGo installs a streaming function automatically whenever you attach a callbacks handler, in chains/options.go and again in agents/mrkl.go. Since streaming is broken, turning on observability breaks your Gemini agents: Plain Text WITHOUT callbacks handler: err=<nil> out="The current year is 2026." WITH callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value Feature by Feature CapabilityLangChainGoGenkit GoStructured outputFormat instructions in the prompt, parse the text backgenkit.GenerateData[T], real response schemaToolsCall(ctx, string) (string, error), no schemagenkit.DefineTool[In, Out], typed with JSON schemaAgentsReAct, driven by parsing Action: linesNative function calling, plus an agent primitive with sessions (experimental)CompositionChain interfaces over map[string]anyTyped Go functions (DefineFlow)RetryWrite it yourselfmiddleware.RetryCross-vendor fallbackWrite it yourselfmiddleware.FallbackObservability17 observe-only callback methodsBuilt-in tracing plus a Developer UIOpenTelemetryNoneBuilt inLocal vector storeNone, all 15 stores need a serverlocalvec, file-backedStreaming on GeminiBrokenWorksHTTP endpointnet/http boilerplategenkit.Handler(flow) The single most important row is retry and fallback, and the reason is architectural rather than cosmetic. A Genkit middleware wraps a call and receives a next(), so it can decide to call again, or call something else. LangChainGo's callbacks.Handler returns nothing from every method. You can watch a call, you cannot change it. That missing seam is why every resilience concern ends up in your application code. What We Are Building A support-ticket triage service. Every requirement here is something a real team asks for: Retrieve the relevant policy documents for an incoming ticket.Let the model call a tool to act on it.Return a typed struct, not free text.Retry transient model failures.Survive a vendor outage by falling back to a different provider.Trace every step, so somebody can debug it at 3 am. The Genkit Go Version The whole service, in one file: Go g := genkit.Init(ctx, genkit.WithPlugins( &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{}, )) // RAG store. embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001") store, retriever, err := localvec.DefineRetriever(g, "policy", localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil) localvec.Index(ctx, docs, store) // A tool the model can call. refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.", func(ctx *ai.ToolContext, a struct { OrderID string `json:"order_id"` AmountCents int `json:"amount_cents"` }) (string, error) { return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil }) // The flow. triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) { found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket)) if err != nil { return nil, err } out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)), ai.WithDocs(found.Documents...), ai.WithTools(refund), ai.WithPrompt("Triage this ticket using the policy documents, and act on it: %s", ticket), ai.WithUse( &middleware.Retry{MaxRetries: 2}, &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}, ), ) return out, err }) 73 lines, and it returns exactly what you asked for: Plain Text {Category:billing Severity:1 PolicyQuote:Duplicate charges must be refunded in full within 5 business days, no manager approval required. Action:Issued a refund of $25.50 (2550 cents) to resolve the duplicate charge on order A-1029.} The interesting part is which requirements are not code. Retry and fallback are two entries in a list. Tracing does not appear at all, because Genkit instruments flows, generate calls, tool calls, and retrieval itself. The RAG store is three calls because localvec ships in the box. The LangChainGo Version Requirements 1, 2, and 3 port directly. Requirements 4, 5, and 6 do not exist in the library, so they become your code: FileLinesWhy it existsmain.go110the actual applicationmemstore.go68all 15 vector stores need an external service, so the local one is yoursresilience.go58no middleware layer, so retry and fallback are yourstracer.go36observability is a 17-method interface to implementtotal272 None of those three extra files contain interesting code. memstore.go is cosine similarity and a sort. tracer.go is six methods that log and eleven empty ones that exist only to satisfy the interface: Go // Everything below exists only to satisfy the interface. func (t *tracer) HandleText(context.Context, string) {} func (t *tracer) HandleLLMStart(context.Context, []string) {} func (t *tracer) HandleLLMError(context.Context, error) {} func (t *tracer) HandleChainStart(context.Context, map[string]any) {} // ...seven more resilience.go wraps the primary model to add retries and a fallback, and it has a sharp edge worth seeing: Go clean := make([]llms.CallOption, 0, len(opts)) for _, o := range opts { var probe llms.CallOptions o(&probe) // Model names are vendor-specific, and gpt-5-mini rejects 'stop' // outright, which the ReAct agent always sets. Both must go. if probe.Model != "" || len(probe.StopWords) > 0 { continue } clean = append(clean, o) } return r.backup.GenerateContent(ctx, msgs, clean...) CallOption is an opaque function, so to find out what an option does you have to apply it to a probe struct and inspect the result. Miss the stop words and the fallback fails with Unsupported parameter: 'stop' is not supported with this model. Both versions work and produce equivalent results. So far this is a story about volume, not capability. The outage test is where that changes. What Happens When a Model Goes Down Point the primary at a retired model in both versions, which is a real 404 from the live API. Genkit heals and finishes: Plain Text WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND" WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND" {Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within 5 business days, no manager approval required. Action:Issued full refund of $25.50 ...} Two calls fail over, once for the initial generate and once for the turn after the tool call. The typed struct survives a vendor switch. LangChainGo heals at the model layer, then breaks one layer up: Plain Text 2026/09/11 20:01:45 primary failed (googleapi: Error 404 ...), falling back 2026/09/11 20:01:54 PARSE FAILED: input text should start with ```json and end with ``` The hand-written fallback did its job. The output parser then rejected the answer because outputparser.Defined.Parse hard-requires code fences: Go const opening = "```json" const closing = "```" if text[:len(opening)] != opening || text[len(text)-len(closing):] != closing { return target, fmt.Errorf("input text should start with %s and end with %s", opening, closing) } Gemini tends to emit those fences. OpenAI does not. The typed output layer is quietly coupled to one vendor's formatting habit, and the coupling surfaces at the exact moment you switch vendors, which is the worst possible time. That same function has a second problem. When the response is shorter than seven characters, it does not return an error; it panics: Plain Text input "ok" PANIC: runtime error: slice bounds out of range [:7] with length 2 input "" PANIC: runtime error: slice bounds out of range [:7] with length 0 An empty completion or a refusal becomes an uncaught panic in your request handler. Dependency Weight The same typed-extraction program, each in its own clean module: LangChainGoGenkit GoModules in the build graph6241Binary size38.0 MB28.7 MB The LangChainGo-only half of that list is worth reading: the deprecated generative-ai-go, plus cloud.google.com/go/vertexai and aiplatform, plus sprig, gonja (a Jinja2 template engine), go.starlark.net (the calculator tool evaluates expressions in Starlark), logrus, tiktoken-go, pkg/errors and json-iterator. A Python-shaped dependency tree, in Go. Where Genkit Go Shines Pulling the thread through all of it, the difference comes down to four things Genkit has, and LangChainGo does not. Middleware is the first. Because a Genkit middleware wraps a call and can decide to call it again or call something else, resilience becomes library code. Retry, cross-vendor fallback, and tool approval are list entries instead of wrapper types you maintain forever. Types are the second. Typed flows, typed tools and schema-backed output mean the compiler carries the contract. That is not just tidier: it is why the Genkit service survived the failover and the LangChainGo one did not. Nothing in the Genkit path was slicing strings out of a model response to find the data. Plugins are the third. A model is a string reference in a registry, so swapping Gemini for GPT is a configuration change. In LangChainGo, two providers are two concrete structs of two different types, which is why a hand-written fallback has to strip vendor-specific call options and still gets it wrong the first time. The Developer UI is the fourth, and it is the one you feel every day rather than during an incident. LangChainGo has no equivalent to it at all. One command starts it next to your app: Shell genkit start -- go run . It reads the registry your code built, so nothing has to be declared twice. On the triage service, it found the flow, the tool, 39 models, and 3 embedders without any configuration: Plain Text Flows (1) triageTicket Tools (1) issue_refund Models (39) googleai/gemini-2.5-flash, ... Embedders (3) googleai/gemini-embedding-001, ... Open a flow, and the input form is generated from your Go struct. A Ticket with Subject and Body fields gives you this, pre-filled and ready to edit: JSON { "body": "", "subject": "" } Press Run and you get the typed output back as a tree, not a blob of text: JSON { "category": "Billing", "urgency": 4, "action": "Refund issued" } The part that earns its place is the trace underneath. Every run is a span tree with per-span timings, and the tool call is a span of its own: Plain Text triageTicket 4.64s ├─ generate 2.52s │ ├─ googleai/gemini-2.5-flash 1.67s │ ├─ issue_refund 1ms │ └─ generate (2) 846ms │ └─ googleai/gemini-2.5-flash 844ms └─ generate 2.12s └─ googleai/gemini-2.5-flash 2.12s You can read the whole shape of the run off that: the first generate called the tool, the model went back for a second turn with the tool result, and a separate generate did the typed extraction. Nobody wrote a line of logging for it. Failures land in the same place. A run that hit a real Gemini constraint showed the error inline with a full Go stack trace, and marked every span in the tree that was affected: Plain Text Error 400, Message: Function calling with a response mime type: 'application/json' is unsupported, Status: INVALID_ARGUMENT, Details: [] That error is worth knowing on its own, and it is model-specific rather than a Genkit limitation. Running the same tools-plus-typed-output call across five models makes the line clear: Plain Text gemini-2.5-flash FAIL Function calling with a response mime type: 'application/json' is unsupported gemini-3-flash-preview OK gemini-3.5-flash OK gemini-3.6-flash OK gemini-3.8-flash OK Gemini 2.5 refuses function calling combined with a JSON response mime type, so on that model a tool call and a typed extraction have to be two steps. Every Gemini 3 model lifted the restriction, which is why the triage service above runs both in one call on gemini-3.5-flash. The useful part is that the Developer UI told me which span failed and why, instead of leaving me to bisect it. The sidebar also carries Traces, Evaluations and Datasets, so the same UI covers running an action by hand, reading what it did, and building a dataset out of it. In LangChainGo, the equivalent of all of this is a callbacks.Handler you write yourself, which returns nothing from every method and, as shown above, breaks your Gemini agents when you attach it. Underneath all four, someone is shipping. When gemini-2.0-flash was retired, that was a 404 to work around in LangChainGo and a fallback that healed itself in Genkit. What LangChainGo Still Does Well This is not a demolition. There are real reasons it is still in use: Provider catalog. Seventeen model providers, including Bedrock, Cloudflare, Ernie, HuggingFace, Maritaca, Mistral, watsonx, and llamafile. Genkit's plugin set is narrower.Vector store catalog. Fifteen integrations, including Dolt, MariaDB, and OpenSearch, which Genkit has no equivalent for.Conversation memory is less typing. chains.NewConversation(llm, memory.NewConversationBuffer()) beats passing resp.History() around by hand. Genkit's answer is an agent session store, which is more capable but sits behind an experimental flag, so on a plain generate call LangChainGo is still terser.The ReAct agent works. On a sequential two-tool task, it succeeds consistently. The cost is the code you own, not flakiness.Familiarity. If your team already thinks in chains, memories, and retrievers from Python, the vocabulary transfers directly. If you depend on a provider or a vector store only LangChainGo supports, that is a real reason to stay. Weigh it against an unmaintained dependency rather than against line counts. The Scorecard Non-blank, non-comment lines. RequirementGenkit GoLangChainGoFull service: RAG + tool + typed output + retry + fallback + tracing73, one file272, four filesBehavior under a vendor outageheals, typed output intactheals at model layer, output parser failsTool with two typed arguments3155 (hand-written unmarshalling)Retry and cross-vendor fallback2 config lines58 lines, vendor-specific options to stripLocal vector store for RAGships as localvec68 lines, no local option existsObservabilitybuilt-in, zero lines36 lines, and it breaks Gemini agentsStreaming on current Gemini modelsworksbroken on all four testedQuickstart with no optionsworks404, default model retiredModules/binary for the same program41 / 28.7 MB62 / 38.0 MB Conclusion If you are starting a Go AI project today, use Genkit Go. That part is not close. If you already have LangChainGo in production, the decision is about risk rather than elegance. The library is eight months without a commit, its Google provider is built on an SDK that reached end of life in November 2025, its quickstart returns a 404, its streaming is broken on every current Gemini model, and turning on observability breaks the agents it is meant to observe. Those are not problems that resolve themselves. The clearest signal is the outage test. Both versions were supposed to survive a vendor failure, and technically both did at the model layer. Only one came back with the data the caller asked for, because in the other one the layer above was matching on code fences. Further reading: How to migrate from LangChainGo to Genkit Go, the step-by-step guideGenkit Go vs ADK Go 2.0Building Agents in Go: Where Genkit Shines vs ADK 2.0Stop using Python for GenAI, use Genkit GoGenkit documentationLangChainGo repositoryMastering Genkit: Go Edition

By Xavier Portilla Edo CORE

If you are responsible for keeping a production system alive, you have probably been pitched an "AI SRE" in the last few months. Maybe more than one. Maybe more than one this week. When we sat down at Bronto to map that space, we ended up with 64 tools across seven camps: pure-play investigation agents, observability platforms with AI built in, incident management tools with AI on top, AIOps and alert correlation, hyperscaler agents like AWS DevOps Agent and Azure SRE Agent, Kubernetes copilots and open source projects, and a long tail of adjacent tools. All of them use the same two words on their landing page, and they solve very different problems. So the first question most teams ask is the classic one: do we buy, or do we build? In our landscape article, we argue that the choice isn't binary. You can go full DIY, build on open source foundations like HolmesGPT or K8sGPT, buy a product and extend it, or buy it as-is. Most teams land somewhere in the middle. In this article, I want to make a different argument for building one, at least once: build to understand what you buy. Even if you never run your own DIY agent in production, putting one together by hand shows you where its answers come from, what it can't see, and why it goes wrong when it does. After building one, you know which questions to ask a vendor, and you know how to get better results from whatever you end up running. That's the profit you should take away from this post. The fun is the building and seeing your agent try to investigate an incident: it might get it wrong, it might get it right, but you'll see how it interacts with your tools and your data. This post is based on a hands-on lab we ran at an AWS and Bronto evening, and everything is available through bronto-community/ai-sre-lab. By the end, you will have a small agent with the following capabilities: reads logs, traces, and metrics through an MCP server,reads the application's source code and commit history through GitHub's MCP server,writes its hypothesis, evidence, and confidence into a GitHub issue. Note that in the lab we use Bronto as the observability backend. I work there, so I am biased to think it is your best option, but you can of course use any other solution. What an AI SRE Is AI SRE is mostly a marketing term. In short, it's an agentic loop pointed at the production system that can answer questions about it and help you troubleshoot. Like any agentic loop, it has inputs on one side (an alert, a question from a human), a model in the middle, tools underneath (mostly MCP servers these days, for telemetry, code, cloud APIs, runbooks), and outputs on the other side (a Slack message, a ticket, a pull request). So, it's an agent like any other. It's what you connect to it that makes it an agent for SRE. The Patient: A Small Online Shop Our AI SRE needs something to investigate. The lab comes with a tiny storefront: four Python services instrumented with OpenTelemetry, plus a payment processor mock standing in for a third party. You can, of course, point the agent directly at your own data. It's read-only, so it is not able to harm your system. What You Need To run this lab, you need the following: A container engine, like Docker DesktopAn AWS account with access to a model on Amazon Bedrock.A Bronto API key (there is a free trial), or another observability backend with an MCP server.A GitHub account and a fine-grained personal access token. It needs to read the contents of your copy of the storefront repo and write issues to it.Optional, for nicer output: jq and glow. You can watch this video to follow along as we walk through each step: Start the Shop Clone the lab and generate the storefront repository. The script builds a small Git history with a handful of commits and tags two releases, v3.1.0 (healthy) and v3.2.0: Shell git clone https://github.com/bronto-community/ai-sre-lab cd ai-sre-lab ./build-storefront.sh Your agent will later read this code through GitHub, so push it to a repository of your own, tags included: Shell cd storefront gh repo create storefront --public --source . --push git push --tags cd .. Then point the Collector at your telemetry backend and bring everything up: Shell cd demo echo "BRONTO_API_KEY=<your key>" > .env make up Note, that demo/collector.yaml ships to Bronto's EU ingestion endpoint. If your account lives in another region, or you want a different backend, that file is the only thing you need to change. Next, run make up, which deploys v3.1.0 and starts the traffic. It prints a message that you should leave it about ten minutes before you do anything else. An investigator with no baseline has nothing to compare against. So, take your time going through the first few next steps so it can build up! Step 1: The Loop All agents start with a basic file that receives prompts, sends them to the model, and returns the results. It's a simple first agent, so here is more or less the code in full: Python import os from bedrock_agentcore import BedrockAgentCoreApp from strands import Agent from strands.models import BedrockModel model = BedrockModel( model_id=os.environ.get("BEDROCK_MODEL_ID", "us.anthropic.claude-haiku-4-5-20251001-v1:0"), region_name=os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION", "us-west-2"), ) app = BedrockAgentCoreApp() @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model) return {"result": str(agent(payload.get("prompt", "Hello!")))} if __name__ == "__main__": app.run() # POST /invocations, GET /ping on :8080 Two libraries do the work. Strands Agents runs the agent loop: send the prompt to the model, run any tool the model asks for, feed the result back, repeat until the model has an answer. The Bedrock AgentCore SDK wraps that in an HTTP server with the /invocations and /ping contract AgentCore Runtime expects. Today that runs on your laptop. Later, the same container can run on AgentCore Runtime without changes. Every step of the lab is a prebuilt image, so you don't need to build anything. Create a .env next to where you run the commands: Shell AWS_DEFAULT_REGION=us-west-2 AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... BEDROCK_MODEL_ID=us.anthropic.claude-haiku-4-5-20251001-v1:0 With this setup, you can start the agent for the first time: Shell docker run --rm -p 8080:8080 --env-file .env ghcr.io/bronto-community/ai-sre-agent:step-1 Open a second terminal and ask it something: Shell curl -s localhost:8080/invocations -d '{"prompt": "Who are you?"}' | jq -r .result It will say, "Claude, an AI assistant made by Anthropic." Correct, and completely useless for incident response. Right now this is a model behind an endpoint, with no identity and no tools. Step 2: Identity The second step changes only one line of code. Every Markdown file in prompts/ becomes part of the system prompt: Python SYSTEM_PROMPT = "\n\n".join(p.read_text() for p in sorted(Path("prompts").glob("*.md"))) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT) ... The image ships with two files. role.md: Markdown You are the on-call SRE for Storefront, an online shop. You investigate production incidents: what broke, where, and why. And rules.md: Markdown - Only state what you can back with evidence from your tools. - If you have no tool that can answer, say so and name what you would need. - Be brief. You also get to add your own. Create a my.md: Markdown - Answer in at most three bullet points. - End every answer with your confidence: low, medium or high. - Sign off as "your friendly on-call dino". Start step 2 as follows. The command contains your custom instructions as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-2 Shell curl -s localhost:8080/invocations -d '{"prompt": "Checkout is slow. What is wrong?"}' | jq -r .result You should get a short answer that says it has no access to telemetry, rates its own confidence as low, and signs off as a dino. That's a good result. The rule "say so and name what you would need" did its job: an agent that admits it can't see anything is far more useful than one that guesses. Step 3: Eyes Now the agent gets something to look at. Step 3 adds one tool source, Bronto's MCP server: Python bronto = MCPClient(lambda: streamable_http_client( os.environ.get("BRONTO_MCP_URL", "https://mcp.eu.bronto.io/mcp"), http_client=httpx2.AsyncClient( headers={"X-BRONTO-API-KEY": os.environ["BRONTO_API_KEY"]}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) ... That's the whole change. The MCP server's tools (search logs, list datasets, and so on) become tools the agent can call, and Strands handles the plumbing. This is also the point where MCP pays off: if your telemetry lives somewhere else, you swap this client for that vendor's MCP server and keep everything else. Add BRONTO_API_KEY to your .env (and BRONTO_MCP_URL if your account isn't in the EU region), then: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-3 Start with a question that tells you what it can see: Shell curl -s localhost:8080/invocations -d '{"prompt": "What services do you know about?"}' | jq -r .result It should list the storefront services. Go ahead and ask it some more questions about the state of those services: what HTTP endpoints are available? What is the average error rate right now? With this, you have a first working version. Next we want to investigate an incident. To do so, ship the next release of the storefront: Shell make release According to the Makefile, symptoms take about 60 seconds to show up in p95. So wait a minute or two and then ask it: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result It will come back with a hypothesis about what is wrong and what you might be able to do about it. Take a look at the telemetry yourself and check: is the analysis correct, or did the agent pick up some false information? Step 4: Mouth An answer in your terminal is nice for a demo, but in production you would want to have it sent to a Slack channel, a pager app, or an issue tracker. In our example, we send the answer as a GitHub issue. This would let you tag your coding agent to review the issue and open a pull request that fixes it. You now have two options for how this gets filed. Either you give your agent another tool, and it can decide for itself where to send the hypothesis, or you make the code file the issue, so every analysis gets pushed into a dedicated issue: Shell def file_issue(question: str, hypothesis: str) -> str: r = httpx2.post( f"https://api.github.com/repos/{os.environ.get('GITHUB_REPO', 'ai-sre-lab/storefront')}/issues", headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, json={"title": f"[investigation] {question}"[:120], "body": hypothesis}, timeout=30, ) r.raise_for_status() return r.json()["html_url"] @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) hypothesis = agent(question) return {"result": file_issue(question, hypothesis)} To make sure that what the agent provides fits the output, an additional new prompt file tells the agent how to write it: Markdown Your answer is filed as a GitHub issue for the on-call team. Write it as one: a one-line hypothesis, the evidence for it, your confidence, and what you could not check. Before running the next version of your agent, add your GitHub token and your repository to .env: Shell GITHUB_TOKEN=github_pat_... GITHUB_REPO=<your-user>/storefront Run step 4: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-4 Ask it again to investigate the issue: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result The response is now a link. Open it and read the hypothesis. Is it right? How confident is it? There's a good chance it's confidently wrong. While testing the lab, one run concluded that checkout was down because a query returned nothing: an empty result is not proof of an outage, but nothing told the agent that. Step 5: Better Instructions What you do next doesn't involve code. You do what you'd do with a new human on the rotation: you tell them how you investigate. Add these lines to your my.md: Markdown - Slow is broken too: compare latency (p50, p95) before and after, not only errors. - Compare against a quiet window earlier the same day. - Look for `deploy` spans near the onset: they name the commit that went live. Each line encodes a habit experienced SREs have, and models don't bring by default. Checkout can be very broken without throwing a single error. A number means little without a baseline. And the first question in most incidents is "what changed?" The prompt files are read when the container starts, so restart step 4 with the same command and ask a more realistic question: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Compare the new issue with the old one. Is it better? Worse? Did it fail outright? Step 6: Better Model As you might now, a different model can create a different output, so if you are not satisfied with the results yet, change one line in .env: Shell BEDROCK_MODEL_ID=us.amazon.nova-lite-v1:0 Restart your agent, ask the question again, and compare the outputs again: better or worse? Step 7: Code So far the agent can see that something changed and maybe even when. If your instructions worked, it found the deploy. What it can't do is look at what was deployed. Step 7 adds the second MCP server, GitHub's, restricted to read-only repository tools: Python github = MCPClient(lambda: streamable_http_client( os.environ.get("GITHUB_MCP_URL", "https://api.githubcopilot.com/mcp/x/repos/readonly"), http_client=httpx2.AsyncClient( headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto, github]) ... To make it useful to the agent, add one more prompt file, code.md. It connects the telemetry to the code: Markdown The shop's source code is in the GitHub repository `ai-sre-lab/storefront`. Every service reports the commit it runs as `deployment.commit`. Replace the repository path with your own, then mount it as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" \ -v "$PWD/code.md:/app/prompts/code.md" \ ghcr.io/bronto-community/ai-sre-agent:step-7 Ask the same question again: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Does it find the deploy? Does it open the release and pick the right commit? In our testing, it found the deploy marker and the commit, and then stopped. The release contains several commits, and the last hop, reading every one of them and figuring out which change explains the symptom, didn't happen on its own. If you experience the same, try to fix it yourself by refining your system prompts or by trying yet another model. When you're done, make rollback puts the healthy version back, and make down cleans everything up. If you want to give your agent a harder case, make psp-slow degrades the third-party payment processor. So, Buy or Build? Now you know what an AI SRE is and how it works, from the inside. That's a much better position from which to answer the question this post started with. If you buy, a lot of what hurt along the way becomes someone else's problem: confidently wrong answers that need endless prompt tuning, context windows overflowing with search results, wiring up every integration yourself. AWS DevOps Agent, for example, starts investigating when an alert fires and connects to the tools you likely already run, and you can still extend it with custom skills and your own tools. Take the four questions from the landscape article with you into every demo: Where does it get its data? Does it act, or only advise? Can you see its reasoning? What happens when it's wrong? After this lab, you'll know what good and bad answers to those look like. If you build, what you have now is a skeleton, and you can build more features towards a complete picture on top of it: Let alerts trigger it instead of curl.Run it on AgentCore Runtime instead of your laptop (agentcore deploy).Give it more tools: your cluster, your cloud APIs, your past incidents and postmortems.Observe the agent itself. Its own traces tell you which tools it called, how many tokens it burned, and where it went wrong, and that's the input you need to improve it. Either way, the data underneath matters more than the agent on top. Every improvement in this lab depended on the telemetry being there: a baseline from ten quiet minutes, latency and not just errors, a deployment.commit attribute that connects a span to a line of code. An agent can only reason about what your system tells it. That was true for humans long before agents showed up. So: what would your agent get wrong? The full lab, including the slides, the agent images for every step, and the storefront harness, is at github.com/bronto-community/ai-sre-lab. Content is CC BY 4.0; code is MIT. If you build something on top of it, I'd love to hear about it. I've shown how easy it is to build a simple AI SRE agent for fun. If you want to see what a production-ready one looks like, read our post on the Bronto blog, Introducing Bronto's Open-Source Alert Investigation Agent. It's an open-source agent that replicates the Alert Investigation feature in Bronto, and it's essentially our production version of this lab: a Bronto monitor triggers it instead of curl, it sticks to a constrained set of tools and reasoning paths instead of improvising, and it hands you a report with likely root causes and suggested next steps. The code is at bronto-community/bronto-alert-investigation-agent.

What does it take to build and sustain an open-source project for more than a decade? For Shai Almog, creator of Codename One, it takes a willingness to challenge conventional approaches to software development and a continued commitment to the developers who rely on the work. I recently caught up with Shai to discuss the inspiration behind Codename One, the realities of maintaining an open-source project, and his perspective on how the Java ecosystem has evolved over the years. Check it out below! 1. What is Codename One, and what originally inspired you to create it? Codename One is what we always wanted Java to be: write once, run anywhere, without accepting the lowest common denominator. It’s what you would get if Spring Boot, Flutter, Go, and Node.js had a baby that inherited the best qualities of each. You get Java’s mature ecosystem, Flutter-style control over the UI/portability, Go-like native binaries and footprint, and the full-stack client/server connection people associate with Node.js. With Codename One, one Java codebase can target iOS, Android, macOS, Windows on x64 and ARM64, Linux on x64 and ARM64 with either glibc or musl, and even the browser. A small self-contained desktop GUI application can be around 4–5 MB, with no JVM to install. Underneath, Codename One uses different pipelines for different targets. On our ahead-of-time compiled platforms, ParparVM analyzes the reachable Java bytecode, removes unused code, translates what remains to C, and hands the result to the platform’s native compiler. Codename One also provides the portability libraries, UI toolkit, tests, theming, and build system needed to handle the ugly platform-specific details. We don’t pretend every platform is identical. Our builder architecture assembles the native pieces each application needs, so developers can use platform-specific capabilities without fragmenting the entire codebase. The original inspiration was frustration with Java’s unfulfilled promise. Java gave us portability, but conventional Java application deployment became increasingly heavy. We wanted the language and ecosystem without requiring a large runtime or giving up native performance, platform access, or control of the UI. Now we’re taking that architecture to the backend. The goal is to bring the startup time, footprint, and native compilation work we developed for constrained devices into serverless and microservice workloads. I want Java developers to get the full-stack productivity they expect from Spring Boot, with deployment economics closer to Go and Node.js. 2. Codename One has been around for more than a decade. What has kept you motivated? The community is the biggest reason. People have built businesses and products on Codename One. When someone has trusted your work with something that important, walking away isn’t an option. For a long time, competing in this market was brutal. Our competitors were not normal startups. They were Google, Facebook, Microsoft, and companies with teams larger than our entire community. There was a limit to how much a small team could build, test, document, and support. Agentic development changed that equation. It lets us scale engineering work in a way that simply wasn’t possible a few years ago. We now maintain an enormous cross-platform test matrix and can move at a pace that is competitive with much larger organizations. On most feature dimensions, I would argue that we are already ahead of the big-name alternatives. We still have gaps. We’re a small team, and pretending otherwise would be ridiculous. But our size is also an advantage. The person reading a bug report can be the same person who changes the compiler. We can respond to developers in hours where a large company may need three teams, a quarterly roadmap, and permission from a product manager. The other reason is simpler: we love Java, and we use the competing tools. Every time we do, we run into one of the compromises Codename One was created to remove. 3. What’s one of the biggest misconceptions developers have about maintaining an open source project? That maintaining open source is mostly about accepting code contributions. A pull request looks like free work from the outside. For a maintainer, it can be the beginning of a large amount of work. Is the code correct? Does it meet the project’s standards? Does it work on every target? Who maintains it five years later? How do you tell a volunteer who spent a week on something that the design is wrong? How many rounds of changes can you reasonably ask them to make? We recently stopped accepting outside pull requests. AI wasn’t the reason. If anything, AI improved the baseline quality of many contributions. It also made some of them enormous, because generating another thousand lines became almost free while reviewing those thousand lines did not. The real problem was validation. Codename One currently has more than 60 GitHub Actions workflow files. A small framework change can affect Android, iOS, JavaScript, desktop, bytecode translation, native compilation, and screenshot baselines. That matrix represents roughly $15,000 worth of monthly CI capacity, which GitHub currently provides to us. At that point, validating and repairing an outside patch often took longer than implementing the confirmed change ourselves. So today we ask people to contribute through reproducible issues, benchmark counterexamples, design discussions, testing, and documentation feedback. Those contributions are often more valuable than code because they expose problems we cannot reproduce in our own environments. Also sorry for all the GitHub Actions downtime ;-) 4. You’ve watched Java and the broader development ecosystem evolve considerably. What changes have surprised you the most? The biggest surprise is that Java effectively surrendered the idea of being small. GraalVM is a remarkable piece of engineering, so we aren’t literally alone (although it's a very different project that hasn't really taken the same direction and hasn't taken it as far). But mainstream Java spent years treating bigness as evidence of seriousness: larger runtimes, larger frameworks, more dynamic machinery, longer warmups, and more memory. The industry began treating that weight as inevitable. I'm not against comprehensive/large; Codename One itself is HUGE... But not in runtime, just in API and scope. Most successful newer platforms moved in the opposite direction. Go made self-contained native binaries normal. Node.js made application deployment almost trivial. Mobile and serverless made startup time and memory consumption economically important again. Java largely responded by optimizing large systems rather than questioning why every system needed to be large. Modules are a good example. They help organize and encapsulate a large dependency graph. They do not eliminate that graph or turn it into a small, closed-world native program. Codename One puts a real Java GUI application or full server into a roughly 5 MB binary. What surprises me is not that this works. We have spent years proving that it works. What surprises me is how lonely that position still is. Java could be one of the best languages for small native applications and low-cost serverless workloads. Most of the ecosystem simply stopped trying to make that happen and let Go eat our lunch. 5. After spending so much of your career in technology, what do you enjoy doing when you completely step away from a screen? Completely step away from a screen? I’m probably the wrong person to ask. I just got back from Thailand with my kids. We had a wonderful time. I also had my laptop with me and was doing pull requests on the beach. This is what I enjoy doing. I don’t have an unwind button. Vacations often leave me more exhausted than work because sitting still doesn’t recharge me. If I have an idea and I can build it, fix it, or argue about it with another developer, that is how I unwind. Explore more of Shai's content here.

TL;DR: The Hidden Cost of Switching AI Models A new AI model arrives that is cheaper per token, scores better on benchmarks, and seems to require nothing more than a setting change. However, the token bill shows only part of the cost of switching AI models. There is also the effort to check whether your workflows, for example, the one to create status reports for stakeholders, still produce outputs in line with your AI Definition of Done. More often than not, this task remains unaligned with the decision to switch AI models. In the worst case, switching AI models behind a business workflow can raise operating costs despite cheaper tokens, because business cases can miss the revalidation effort. Thesis: This article explains the recurring operational cost of switching AI models behind a business workflow: why the revalidation effort may disappear from upgrade decisions, how to screen a switch, and how Agile teams use an AI Definition of Done, test evidence, and an accountable owner to stay in control. Why a Cheaper Model Can Cost You More Your vendor has announced a new model. It costs less per million tokens than the one behind your team's workflows, it beats its predecessor on every benchmark the release notes care to mention, and switching seems to require nothing more than a changed setting in a configuration file. The business case writes itself, which is exactly the problem, because it writes itself from the price list. A lower price per token is a fact about the model, whereas lower operating costs are a claim about your workflow. That claim depends on work no price list contains: somebody has to check whether the workflow still produces acceptable results after the switch, adapt whatever no longer does, and keep checking afterward. That effort recurs with every model change, and it vanishes from the decision whenever existing staff absorb it, because nobody "invoices" for their afternoons to create the desired transparency. The Upgrade That Looks Like a Configuration Change Take a hypothetical but ordinary case. A Product Manager runs a workflow that drafts the weekly stakeholder status report: its agent takes the export from the team's project tracker, the product roadmap, and customer input from a variety of sources. It builds on a skill that describes the report structure, including tone instructions, and the model produces a two-page draft that the Product Owner reviews, corrects, and sends out every Friday. (Yes, a written status report in an Agile organization; stakeholders who skip, for example, the Sprint Review still want to know what happened, and a page they read beats a meeting they avoid.) Over a few months, the review has settled into a quick pass that mostly fixes phrasing. Then the vendor releases a cheaper model, and the switch looks like a one-line change. When the Report Still Reads Well and Is Wrong After the switch, the reports still arrive on time, in the right structure, and in fluent prose. In the third week, however, the draft describes a blocked item as "on track," because the new model dropped the one sentence in the tracker comment explaining that the item waits for a decision from the legal department. The report reads smoothly; nevertheless, it would have told the stakeholders the opposite of the truth, and the Product Manager caught it only because they happened to know that particular item. If the team had written an AI Definition of Done for its status reports, with a verification level that requires checking every status claim against the project tracker, the report would have failed on the spot; without one, catching the error depended on the Product Owner's memory. (Download the complete A3 Delegation System template including the AI Definition of Done above.) This is the failure that can make switching AI models expensive: the workflow remains technically operational while it no longer produces acceptable output. A loud failure, say, a rejected parameter or a broken output format, stops the workflow with an error message, and somebody fixes it before lunch. The quiet failure passes every technical check and lands in front of the reviewer, who now reads every report with more suspicion than before, which is where the correction effort starts to climb. The same mechanism explains why a quick trial before the switch proves so little: model output can vary from one run to the next, so one successful run provides limited evidence, and the new model may well have produced a correct report on the day somebody tried it. Hence, every switch of the model behind a production workflow raises a revalidation question. By revalidation, I mean checking, before you approve the changed workflow for production use, that it still meets its agreed standard, and then inspecting its actual results after deployment. For teams that work with an AI Definition of Done, the trigger is already on paper: the standard names the model tier that suffices for each task class and carries a sign-off with a review date, so a model switch leaves that sign-off without a basis until somebody renews it. The depth of that check should depend on what a failure costs, as a brainstorming assistant and an agent that changes customer records do not deserve the same scrutiny. Where the Switching Cost Hides The explicit charges are easy to find: the new model's token price, perhaps a few more tokens per report if the new model writes longer drafts, and, in larger setups, the invoices for tooling or outside help with the migration. Additional costs include operational effort: the hours spent comparing old and new reports before the switch, the extra minutes of correction every Friday since, the test cases somebody has to maintain, the rework when a wrong report has already gone out, and whatever the Product Owner did not do in the meantime. (Have I mentioned the cost of losing stakeholder trust because of a faulty report?) Salaries are on the books, of course; however, existing staff budgets absorb this effort, and unless somebody attributes it to the migration, it stays out of the business case. Two days a team member spends validating a migration create no new cash expense, yet those two days are gone. That is what I mean by hidden: the cost is on the books, just not on the page where the decision gets made. Consuming more tokens per task can also outweigh a lower token price, more retries, higher latency, or more human correction. So the unit of comparison is the complete workflow at an acceptable quality level, not the token price list. A Screening Question Before You Switch For a switch motivated by cost savings, one division gives you a screening question: divide the one-off migration and revalidation cost by the net monthly savings to get the number of months it takes to recover the investment. "Net" is the word to watch, as it means the savings compared with the existing workflow after changes in review time, corrections, retries, and ongoing maintenance. Meeting the minimum standard, as such, is a precondition rather than a term in the formula: a cheaper workflow that sends wrong reports to stakeholders does not qualify, whatever the arithmetic says. Then ask whether the expected savings justify the investment, allowing for further migration and revalidation costs. A later change does not erase the savings you have already made. However, it brings its own revalidation cost, and vendors retire models on their own schedule: Anthropic, for example, retired Claude Sonnet 4 and Claude Opus 4 on June 15, 2026, and Claude Opus 4.1 on August 5, 2026. Treat the payback figure as a screen, not a forecast; it works when net savings are positive and reasonably stable, and it tells you whether the switch deserves a closer look. Beware, though: Upgrades motivated by quality or new capabilities need a different judgment: weigh that benefit against the additional cost directly, without squeezing every improvement into a monetary payback calculation (the attempt invites invented precision as a misleading metric). When the Vendor Decides for You Not every switch is voluntary. Anthropic commits to at least 60 days of notice before retiring a publicly released model, and on September 30, 2026, it notified developers that Claude Sonnet 4.5 is scheduled to retire on November 30, 2026. OpenAI gives generally available models at least 6 months of notice, specialized variants at least 3 months, and preview models much less, such as 2 weeks; all of these minimums yield when safety or compliance concerns call for a faster retirement, in which case OpenAI promises "as much notice as reasonably possible." A forced migration changes the question from whether switching is worthwhile to which viable alternative to choose: Moving to the successor modelChanging the providerReducing the workflow's scopeReverting temporarily to manual workRetiring the workflow altogether Teams that run an AI Delegation Audit will recognize the list, as the audit's decisions overlap with it (change the model tier, change the A3 category, update the AI Definition of Done, or retire the delegation). The audit counts retiring an automation that no longer justifies its audit cost as a successful outcome. Continuing the automation is itself a decision, and it deserves the same revalidation as a voluntary switch, with less time to do it. If the Product Manager first has to work out what an acceptable status report looks like, much of the notice period is gone before the first test runs, so the evidence has to exist before the notice arrives. Evidence That Survives the Switch The evidence consists of four things, kept together: examples of acceptable and unacceptable work, the criteria you used to judge them, the relevant workflow configuration (prompt, skills, instructions, data, and context), and the results the current workflow produces on those examples. For the status report, that might be a dozen past project tracker exports, the reports you consider correct, and a note on why the blocked-item report was wrong. If the workflow already has an A3 Handoff Canvas, its Records field has settled in advance what to keep (prompts, skills, inputs, outputs, edits, and approvals), which means the evidence accumulates while the workflow runs instead of being reconstructed under a deadline. Keep that evidence somewhere your organization controls, because testing tools are dependencies, too. OpenAI announced on June 3, 2026, that it is shutting down its Evals platform: existing evals become read-only on October 31, 2026, and the dashboard and API shut down on November 30, 2026. A team that kept its test cases only in that platform must migrate its testing setup, and if its model needs replacing at the same time, the two migrations compound the work. Passing the same cases supports the decision to deploy; however, a new model can pass every existing case and still fail in ways those cases never covered. So, retain the existing cases, add a case for every newly discovered failure (the blocked item is now case number 13), and inspect the actual results for a while after deployment. Who Approves Continued Use? Evidence that nobody acts on is an archive, so somebody has to be accountable for the check happening before a switch, for recording the decision to keep using the workflow, and for noticing when production results drift. Call that person the "workflow owner." They do not have to run every test personally, but they own the answer to "does this still work?" In the A3 Delegation System, the AI Delegation Audit inspects whether this practice works, much as a Retrospective inspects how the team works. One of its four checks, model fit, asks whether the assigned model still suits the task and its risk level. The audit does not gate individual switches, which a recurring inspection cannot do; the model change remains the trigger. One limit applies to all of this: you can only revalidate the workflows you know about, and an AI Workflow Inventory, even one that deliberately lists the shortcuts nobody approved, covers only what your discovery turns up. A workflow a team member runs privately through a personal account can change without the team knowing about it, let alone applying its agreed checks. Shadow IT always comes at a price. Conclusion: One Workflow Before Switching AI Models Pick one consequential recurring workflow, one where a wrong output would reach a stakeholder, a customer, or a decision before anybody noticed. Then do four things: Name the owner: The person who approves continued use after the next model change.Write down the standard: What counts as acceptable work for this workflow, ideally as an AI Definition of Done with the model tier it was signed off for and a review date. (If you work with Scrum, the parallel to the Definition of Done is deliberate.) If no such standard exists, writing it is your first task, and the hardest one, because without a standard there is nothing to revalidate against.Collect five starter cases: Real past inputs with outputs you consider acceptable and, if you have one, an output you consider wrong, together with the results the current workflow produces today.Record what runs it: The model and provider as your tool shows them, and where the evidence lives. (If you keep an AI Workflow Inventory, put this into a separate check record under the workflow's inventory number rather than into the inventory itself.) Five cases start the exercise; they do not validate anything on their own, and they are not supposed to. Then apply one test: if your vendor announced the retirement of that model tomorrow, could the owner demonstrate within the notice period that the workflow still meets its standard on the new one? If the honest answer is no, you have uncovered operational work that the model's price list leaves out. Key Questions This Article on Switching AI Models Answers Does a Cheaper AI Model Lower the Cost of a Business Workflow? A cheaper AI model does not necessarily lower a workflow's cost, because the token price is only one part of what the workflow costs to run. A cheaper model may need more tokens, more retries, or more human correction, and the effort to revalidate the workflow adds staff time that a business case built from token price lists may not capture. The unit of comparison is the complete workflow at an acceptable quality level. What Are the Hidden Costs of Switching AI Models? The hidden costs of switching AI models are the internal efforts that no vendor bill shows: comparing old and new outputs before the switch, extra review and correction afterward, maintaining test cases, and reworking wrong output that already went out. Existing staff budgets absorb this effort, so a business case built from token prices can miss it: two days of validation work creates no new invoice yet still consumes capacity, so it vanishes in the budget noise. How Do You Check Whether a Workflow Still Works After an AI Model Change? Revalidate the workflow: before approving it for production use, check it against an agreed standard, such as an AI Definition of Done, and a set of saved cases, then inspect its real results after deployment. Keep acceptable and unacceptable examples, judging criteria, workflow configuration, and previous results in a place your organization controls. Because model output varies between runs, one successful trial is weak evidence, and every newly found failure becomes a new case. What Should a Team Do When an AI Vendor Retires Its Model? Treat a vendor's model retirement as a choice between viable alternatives: moving to the successor model, changing the provider, reducing the workflow's scope, reverting temporarily to manual work, or retiring the workflow. Notice periods vary: Anthropic commits to at least 60 days for publicly released models, while OpenAI gives generally available models at least 6 months but may retire faster for safety or compliance reasons. Teams with prepared test evidence can revalidate within that window. Who Should Approve the AI Model Behind a Recurring Workflow? A named workflow owner should be accountable for checking the workflow before each model change and for approving its continued use, even if other people run the tests. In the A3 Delegation System, the AI Definition of Done supplies the standard the workflow is checked against, and the AI Delegation Audit inspects whether this practice works over time; the audit does not gate individual model switches.

By Stefan Wolpers CORE

Over the past few years, vision-language models (VLMs) have moved from experimental research projects to early production use. These systems take in both images and text, so they can read documents, interpret photographs, answer questions about visual content, and reason across modalities in ways that were not practical a few years ago. Better architectures and training methods explain part of this progress. Better ways of deploying models explain at least as much. In 2026, the defining question for enterprises is no longer whether VLMs work, but whether they can run reliably, safely and cost-effectively in production. The answer depends as much on cloud infrastructure, observability and governance as on the models themselves. This article traces how VLMs evolved, why cloud computing became the turning point for adoption, where reliability still falls short, and which cloud-native practices organizations need in order to deploy them responsibly at scale. From Research Breakthroughs to Foundation Models The modern VLM era began with CLIP (Contrastive Language–Image Pretraining). It showed that training on hundreds of millions of image-text pairs could align visual and textual representations in a shared embedding space. That alignment enabled zero-shot image classification and made large-scale multimodal search possible. DeepMind's Flamingo built on this foundation. It introduced cross-attention layers that connect a frozen vision encoder to a large language model, so a single model could learn new tasks from only a few examples. BLIP-2 then showed that a lightweight querying module could bridge frozen image encoders and frozen language models, which sharply reduced training cost. Instruction tuning was the next step. LLaVA (Large Language and Vision Assistant) showed that fine-tuning on visual instruction data produced conversational assistants that could follow complex multimodal prompts. OpenAI's GPT-4V brought these capabilities to a mass audience through a commercial product. Open-weight models such as Qwen2-VL later extended high-resolution perception to the open-source community. Taken together, these advances in cross-attention mechanisms, instruction fine-tuning, and large-scale multimodal pretraining allowed VLMs to combine perception and reasoning in a single system. The Cloud as the Deployment Turning Point The research milestones were essential, but the real turning point for adoption has been cloud deployment. VLMs are computationally demanding: they need high-memory GPUs, efficient batching, and low-latency serving. Few organizations can justify building that infrastructure in-house for workloads that are still evolving. Cloud platforms solve this problem. They offer GPU-backed compute on demand, managed model endpoints, and API-based access that fits directly into existing applications. As a result, VLMs now support practical enterprise workflows: Document intelligence: extracting data from invoices, contracts, and formsVisual inspection: spotting defects in manufacturingMultimodal search: retrieving content by combining images with natural-language queries Elastic scaling also lets organizations match capacity to demand, so they don't pay for idle hardware during quiet periods. Benchmarks Reveal Uneven Reliability Despite these advances, VLM reliability remains inconsistent. MMBench evaluates models across a wide range of perception and reasoning skills, and its results show large differences in performance from task to task. The VERIFY benchmark focuses on visual reasoning fidelity, testing whether models reason correctly about what they see rather than simply producing plausible answers. It exposes significant gaps even in leading systems. Industry reviews of recent computer vision research, including Voxel51's analysis of ICCV 2025 papers, point to the same ongoing weaknesses in fine-grained perception and consistency. A model that does well on one benchmark can still struggle on a closely related task. The practical lesson is clear: choose models based on the specific use case and workload, not on general leaderboard rankings. Choosing Between Open-Source and Managed Models Organizations face a strategic choice between open-source and managed VLMs. Open-source models such as LLaVA and Qwen2-VL can run on Kubernetes clusters or in a private cloud. They give full control over data residency, customization, and cost, which suits regulated industries and sensitive data. The trade-off is that they require in-house expertise in GPU orchestration, model serving, and ongoing maintenance. Managed services offer elastic scaling, built-in security, and much less operational complexity, at the cost of less control and possible vendor lock-in. Many enterprises use a hybrid approach. They route sensitive workloads to self-hosted models and send general-purpose tasks to managed endpoints. Hidden Failure Modes at Scale A major concern in production is hidden failure modes. The most widely discussed is hallucination, where a model describes objects or details that are not in the image, and research on object hallucination shows it remains a persistent problem. Other risks include: OCR errors on low-quality scansSmall objects being missed in dense scenesSpatial reasoning failures, such as confusing left and right or misjudging relative positions Inside automated pipelines, these errors are especially dangerous. One hallucinated invoice total or missed defect can repeat across thousands of transactions before anyone notices. At scale, small error rates add up to significant business and compliance risk. Cloud-Native Guardrails for Production To manage these risks, organizations need guardrails built into their cloud-native architecture: Confidence thresholds send uncertain outputs for further review instead of accepting them automatically.Human-in-the-loop review gives a person oversight of high-stakes decisions.Retrieval-based grounding ties model outputs to verified source data, which reduces hallucination.Audit logging records inputs, outputs, and decisions for traceability and compliance.API gateway policies enforce rate limits, access controls, and content rules consistently across applications. Frameworks such as the NIST AI Risk Management Framework give a useful structure for applying these controls. They help organizations map, measure, and manage AI risk across the lifecycle. Observability tools that track drift, latency, and error patterns complete the picture, so teams can detect problems before they spread. These controls are what move VLMs from impressive demonstrations to dependable production systems. Conclusion The VLM story is ultimately about systems as much as models. Breakthroughs from CLIP to Qwen2-VL made multimodal understanding possible, but cloud computing made it practical. Combining multimodal AI with cloud-native infrastructure has let organizations build AI workflows that are scalable, observable, and governed. VLMs are not yet perfect. Benchmarks show uneven reliability, and hallucination remains a real risk. Even so, they have matured enough for teams to deploy them at scale, provided those teams take careful model selection, rigorous evaluation, and strong guardrails seriously. The organizations that succeed in 2026 will treat VLMs as one part of a well-architected system with governance built in from the start.

By Srinivas Chippagiri CORE

Here's the security problem in enterprise AI I'd lose sleep over, and it isn't a model giving a wrong answer. It's a model connected to an identity with enough authority to act on that answer. Think about what an agent really is once it can query a database, call an infrastructure API, update a ticket, and kick off a workflow. It isn't a chat window anymore. It's software that executes things, with credentials, on its own. Most companies still secure it like a web app with a fancy text box. I think that gap is where a lot of future incidents will start. Who Actually Did That? Picture an incident response agent. A pager goes off. The agent pulls telemetry, queries a database, opens a ticket, calls a cloud API, and suggests a fix. Nobody clicked anything between the second step and the fifth. Now try to answer the question an investigator will ask later. Which principal did each of those things? Not who typed the prompt. Whose credential touched the database, with what scope, and who signed off? Microsoft's own guidance is refreshingly blunt about this. It calls out "identity ambiguity": agents running on shared secrets, service principals, or mixed "on behalf of user" setups, so nobody can say for certain which principal is accountable. It also points out that logs tend to record the chat response and miss the tool actions and downstream authorization decisions that matter. You can read it in Least privilege for AI agents with Microsoft Entra Agent ID, updated July 15, 2026. Authentication tells you the agent is who it says it is. It doesn't tell you whether a given action should be allowed. Those are different questions, and agents force us to ask the second one constantly. An App Has Paths. An Agent Makes Them Up. With a traditional app, the logic is fixed when you design it. A user acts, the app calls an API, the API touches data. You can map the paths on a whiteboard. An agent picks its path at runtime. It chooses tools, chains calls, reads results that shape the next call, and sometimes crosses system boundaries while doing it. Microsoft describes agents as planning and executing multistep workflows across systems, sometimes with little or no human involvement. Software like that needs what every other actor in the enterprise already has: an identity of its own, an accountable owner, scoped permissions, a lifecycle, and an off switch. That's the direction Microsoft's Entra Agent ID is heading, treating agents as a distinct identity type instead of recycling ordinary application identities. Here's an overview of agent identities in Microsoft Entra. I'll say the quiet part loudly: the LLM should never be your security boundary. A model can be talked into things, confused, or hijacked by a sentence buried in a retrieved document. Put the boundary where you can enforce it deterministically, around the identity, the tools, and the systems downstream. Authenticated and Still Dangerous An agent can log in flawlessly and still be wildly overprivileged. Take the incident agent again. It needs to read logs, metrics, tickets, and configuration, and it needs to write exactly one thing: an incident ticket. It has no business deleting infrastructure, editing IAM roles, creating admins, exporting customer records, or touching security policy. Microsoft's pattern makes that concrete. Keep the read role for gathering evidence separate from the write role for tickets. Allowlist only "create or update ticket." Block delete and admin actions. Ask a human before any bulk update. The Number Nobody Calculates Permission creep isn't new. Agents just make it sneakier. Microsoft names it "permission creep and excessive aggregate privileges." A team grants a broad role to get a pilot moving and never narrows it later. Or they stack lots of small roles without separating tasks, and the sum ends up far too powerful. Nobody adds it up, so everybody underestimates what the agent can really do. Every grant looks fine on its own. CRM read. Mailbox access. Ticketing. A cloud role. A deployment pipeline. Together they describe a high-privilege operator who never sat through onboarding. So stop asking "is this permission safe?" and start asking "what can this agent do, end to end, with everything combined?" Microsoft's first implementation step is exactly that: inventory your agents and tool integrations and work out their aggregate effective permissions. Every Tool Is a Door The dangerous failures here are boring ones. Microsoft warns that if an agent can call any tool it can find, one prompt injection or workflow bug can set off export, delete, or privilege changes, or chain actions across services. The industry is starting to name this properly. The OWASP GenAI Security Project announced its Top 10 for Agentic Applications on December 9, 2025, covering risks like goal hijack, tool misuse, and identity and privilege abuse. Its ASI03 entry is about agents running on borrowed, overscoped, or long-lived credentials, so a compromised agent inherits far more access than its job needs. Teleport has a readable breakdown. The rule I'd tattoo on every design review: a tool isn't trusted just because an agent asked for it. Microsoft's advice is to deny unreviewed tools, plugins, integrations, and cross-tenant or guest paths by default, and to check authorization again at the downstream system on every call instead of leaning on the orchestrator. Revalidate identity, role, and scope at each hop so one weak integration can't talk its way around the rest. There's a catch, and Microsoft admits it. All of this depends on downstream systems actually rechecking authorization. Where they don't, the risk walks right back in. "On Behalf Of" Is a Boundary, Not a Label When an agent acts as itself, the audit trail is simple. When it acts for Alice, things get interesting. Does it inherit everything Alice can do? It shouldn't. Alice may hold broad rights that one task will never need. The answer is explicit "on behalf of" context, narrower scopes, and a named owner for the agent. Microsoft recommends logging the "on behalf of" user wherever it applies. OWASP's ASI03 describes the flip side: delegated credentials, cached sessions, and implicit identity let an attacker use the agent to do something its owner never intended. Login Isn't the Finish Line Classic IAM asks, "can this identity reach this resource?" Agents need a second question: "should this exact action happen right now?" Microsoft's recommended pattern pairs a stable, lifecycle-managed agent identity with time-limited privileges. That means just-in-time entitlements: temporary role activation, short-lived tokens or approvals, so elevated access exists only for the length of one workflow. For remediation agents, it suggests JIT elevation or approval gates, scoping to specific resource groups, denying subscription-wide rights, keeping rollbacks ready, and requiring step-up controls for destructive actions. Here's the shape of the decision I'd want in front of every tool call: Python def authorize(agent_id, on_behalf_of, tool, action, resource, context): scope = effective_scope(agent_id, on_behalf_of) # computed, never assumed if tool not in allowlist(agent_id): return deny("tool not allowlisted") if action not in scope.allowed_actions(resource): return deny("action outside effective scope") if is_high_impact(action): # delete, export, privilege change return require_approval_or_jit(agent_id, action, resource) return allow(ttl_seconds=300, correlation_id=context.correlation_id) Don't focus on the syntax. Notice where the decision lives: outside the model, per action, with a short lifetime on whatever it grants. A Transcript Isn't an Audit Trail Microsoft spells out what a useful record holds: agent identity, role, effective scope, action, resource, correlation ID, and the "on behalf of" user when there is one. Its suggested success metrics include the share of agent actions that emit those fields and the share of workflows you can trace end to end across the orchestrator, the tools, and the downstream systems. If your SIEM can tell you an agent "answered a question" but can't tell you which API it called under which scope, you have chat history. You don't have telemetry. How Fast Can You Pull the Plug? Most agent programs I read about spend pages on capabilities and one sentence on shutdown. Microsoft treats the stop path as a first-class concern. Test revocation by disabling the agent, rotating credentials, invalidating tokens, and removing stale permissions. Its success factors include mean time to revoke or disable an agent identity, token invalidation included, and the share of environments with a tested kill switch and recovery procedure. Put mean time to revoke on the same dashboard as mean time to detect. If shutting down an agent takes a ticket, an approval, and two days, its autonomy was never really bounded. It was just unobserved. Legacy Makes Everything Harder Agents can't safely automate what an enterprise doesn't understand. Legacy estates love to hide undocumented business logic, creaky APIs, fragmented data, inconsistent identity models, and thin observability. Point an agent at that, and you're handing a fast, tireless actor a map nobody ever finished drawing. That's why identity, integration, observability and governance now sit beside model choice in modernization work, and why vendors in that space are repositioning around agents. On March 3, 2026, Sonata Software said it had been recognized as a Microsoft Frontier Partner, describing an approach that pairs AI agents with human engineering across cloud and AI platforms, AI business solutions and security. Sonata Software announced that it had earned the 2026–2027 Microsoft AI Business Solutions Inner Circle award, marking the company’s sixth Inner Circle recognition, with Microsoft's Niels Jensen saying in the release, "Inner Circle partners represent our top selling partners worldwide" (Sonata's press release). Both items are vendor announcements, so read them as a signal about where the services market is going, not proof of any particular control. The shift is what matters here: modernization is being pitched as preparation for governed autonomy, and governance starts with identity. What I'd Do on Monday Inventory every agent and tool integration, guest and cross-tenant paths included, and work out each agent's aggregate effective permissions.Give every production agent its own identity and a named owner. Retire shared credentials.Swap broad roles for task-scoped ones, and split read from write.Allowlist tools and actions. Put delete, export, and privilege changes behind approval or JIT elevation.Enforce authorization at the downstream system, not just the orchestrator.Emit identity, role, scope, action, resource, and correlation ID for every action, and send it to your SIEM.Run a revocation drill and write down how long it took. The Short Version Nobody's trying to stop companies from giving agents autonomy. The point is to make that autonomy bounded, attributable, and revocable. If an agent can behave like an employee, an application, and an automation system all at once, the real question is why we still secure it as if it were only an application.

By Igboanugo David Ugochukwu CORE

Agents Can Communicate, But Do They Understand Each Other? Two agents. Same customer ID. Different decisions. That is the semantic gap hiding inside most multi-agent enterprise deployments today — and protocols like MCP and A2A alone cannot close it (nor will they ever). AI agents are getting better at interacting through APIs, events, messages, and shared protocols. They can exchange data, invoke services, and coordinate tasks across enterprise systems. But communication doesn’t guarantee understanding. Imagine three agents dealing with the same customer request. One sees a customer as an account holder, another as a billing entity, and a third as a CRM contact. Data is exchanged, but each agent interprets it differently. This exposes a fundamental challenge in multi-agent enterprises: agents can exchange information without sharing the same meaning. Protocols define how information is exchanged. APIs define what data is exchanged. Neither defines what enterprise concepts such as customer, product, entitlement, incident, or approval actually mean. As organizations deploy specialized agents in sales, finance, ops, security, and engineering, semantic consistency is paramount. Agents need more than connectivity — they need a shared understanding of business concepts, relationships, and constraints. “Agent interoperability is not only a protocol problem; it is a semantic problem.” The Semantic Gap in Multi-Agent Enterprises In a multi-agent enterprise, different agents are built by different teams, trained on different data sets, connected to different systems, and designed around different business contexts. They may all exchange valid JSON messages, call the same APIs, or participate in the same workflow but still interpret the underlying business concepts differently. This is the semantic gap: what’s different between exchanging data and agreeing on what that data means? For example, the term customer may refer to a: CRM contact for a sales agentBilling account for a finance agentService subscriber for a support agentLegal entity for a compliance agent Similarly, priorities, incidents, owners, entitlements, or approvals may have different meanings in systems and domains. These differences are manageable when humans are involved because people infer context, ask clarifying questions, and reconcile ambiguity. Autonomous agents cannot do this reliably unless the semantics are explicitly represented. Without shared semantics, multi-agent systems can suffer from: Inconsistent interpretations of the same enterprise dataIncorrect agent-to-agent handoffsDuplicated or conflicting decisionsBrittle mappings between systemsGovernance rules that are applied differently across agents The challenge therefore moves beyond technical interoperability. Enterprises need semantic interoperability — a common model of concepts, relationships, constraints, and vocabulary that gives every participating agent the same frame of reference. That semantic foundation is provided by a shared enterprise ontology. What Is a Shared Enterprise Ontology? A shared enterprise ontology is a common semantic model that describes the concepts an enterprise uses (and how they are related to each other), as well as rules that give meaning to those concepts. It gives business knowledge a machine-readable format so that people, applications, and AI agents interpret enterprise information consistently. An ontology may define concepts such as: CustomerProductOrderServiceIncidentEntitlementEmployeeApproval More importantly, it defines how these concepts relate. For example: Customer → purchases → ProductCustomer → owns → AccountProduct → provides → ServiceIncident → affects → ServiceEmployee → approves → Request This is different from simply maintaining a common vocabulary. An ontology captures meaning, relationships, hierarchy, and constraints. A shared enterprise ontology thus becomes a semantic reference layer that agents can use to interpret data from different systems in the same way. Instead of each agent deciding what a customer, incident, or approval is, the enterprise defines those concepts once and reuses them across the enterprise. For multi-agent systems, this shared semantic model can become the common language that enables agents not only to exchange information but also to reason about it consistently. Why APIs, MCP, and A2A Are Not Enough on Their Own APIs, MCP, and A2A are examples of how interoperability problems are solved. Agents discover capabilities; they exchange messages; invoke tools; coordinate actions across systems. But they are more about connectivity and interaction than shared meaning. An API can have a field called customer_id. MCP can expose a tool to get customer data. An A2A interaction can send that information from one agent to another, but none of these mechanisms by themselves guarantees that every participating agent will interpret customer, account, service, or entitlement in the same way. This distinction matters. APIs specify how applications exchange data and invoke functions.MCP standardizes how models and agents access tools, resources, and context.A2A allows agents to communicate, delegate, and collaborate. These mechanisms provide technical interoperability, but semantic interoperability requires an additional layer: a common understanding of enterprise concepts and relationships. For example, two agents may exchange an incident object through an API or agent protocol, but one interprets an incident as a customer support ticket and the other an operational outage. Technically, they succeed in communication, but the decision made may still be wrong. An ontology for shared enterprise complements APIs, MCP, and A2A by specifying the semantic contract behind the information exchanged. In that model: APIs provide access.MCP provides tool and context interoperability.A2A provides agent collaboration.Ontology provides shared meaning. Together, these layers create a stronger foundation for reliable multi-agent enterprise systems. What Should an Agent-Aware Enterprise Ontology Describe? A traditional enterprise ontology describes business concepts (customers, products, services, orders, assets, and incidents). An agent-aware enterprise ontology should also go further by describing the agents that operate on those concepts. At minimum, it should capture five areas: Enterprise entities – customer, product, service, account, incident, policy, asset, employee.Relationships – owns, depends on, affects, approves, belongs to, consumes, manages.Agent capabilities – what an agent can search, analyze, create, modify, approve, or execute.Permissions and constraints – what an agent is allowed to access or change, under which conditions.Operational context – which system owns the data, its sensitivity, lifecycle state, provenance, and applicable governance rules. For example, the ontology could say that a Support Agent can read customer entitlement data; create an incident; recommend a resolution — but cannot approve a refund above some threshold. A Finance Agent can access billing records and authorize certain financial actions. This is what makes the ontology useful not only for understanding enterprise data but also for understanding who can act on that data, how they can act, and under what constraints. An agent-aware ontology can therefore answer questions like: What does this concept mean?Which agent owns this capability?What data can the agent access?Which actions are permitted?What policies apply before execution? This turns the ontology from a passive knowledge model into a semantic foundation for agent discovery, coordination, authorization, and governance. From Skill Discovery to Semantic Capability Discovery Most agent platforms can already discover skills or tools: search a catalog, inspect a description, and invoke a capability. The limitation is that skill discovery is usually done based on names, keywords, or interface descriptions. An agent may find a tool called create_ticket, but still not understand: What type of ticket it createsWhich business entity the ticket relates toWhat prerequisites applyWhat permissions are requiredWhether another capability is semantically equivalent This is where semantic capability discovery becomes important. Instead of asking only: “Which skill matches this request?” an agent can ask:“Which capability can perform this business action on this enterprise entity under these constraints?” For example, an ontology could describe that: RefundProcessingAgent → canPerform → IssueRefundIssueRefund → appliesTo → CustomerOrderIssueRefund → requires → ApprovedReturnIssueRefund → governedBy → RefundPolicy Now the agent isn’t finding a skill by name alone. It is finding a capability based on meaning, relationships, prerequisites, and policy context. This is especially useful when multiple agents expose overlapping capabilities (one agent may support refunds on retail orders, another on subscriptions, and another only for low-value transactions). Semantic metadata lets the orchestrating agent choose the right capability based on business context rather than just string matching. In this model, the ontology becomes a semantic capability graph connecting: Agent → Capability → Business Entity → Preconditions → Policies → Constraints This enables richer agent discovery, better routing, safer composition, and more reliable delegation across the enterprise. Ontology-Driven Agent Selection and Governance As enterprises deploy more specialized agents, choosing the right agent gets harder. A request may match several agents, but not every agent will have the same capability, authority, data access, or policy constraints. An enterprise ontology can make this selection more intelligent. Instead of routing a request based only on keywords or static configuration, an orchestrator can use semantic relationships such as: Agent → hasCapability → ApproveRefundApproveRefund → appliesTo → CustomerOrderAgent → authorizedFor → RegionCapability → constrainedBy → RefundPolicy When a task arrives, the system can evaluate not only which agent can perform it, but also: Whether the agent has the required capabilityWhether it can access the relevant enterprise dataWhether it is authorized for the requested actionWhich policies and constraints applyWhether human approval is required For example, two agents can both process refunds - one may be limited to transactions below a defined threshold, while another may handle high-value refunds but require managerial approval. The ontology provides the semantic context needed to route the request correctly. The same model can also support governance. Policies can be directly associated with business entities, capabilities, roles, and actions, so agents can reason about constraints before execution. This creates an ontology-driven decision flow: Request → Business Intent → Required Capability → Eligible Agents → Policy Evaluation → Selected Agent → Execution The result is more than smarter routing. It enables policy-aware, context-aware, and explainable agent selection across the enterprise. Building a Minimal Agent-Aware Enterprise Ontology AgentsCapabilitiesEnterprise entitiesPolicies and constraints Consider a simple refund-processing scenario. An enterprise has a support agent that can process refunds for customer orders, but that capability is governed by a refund policy. A minimal semantic model could look like this: This can be represented using RDF/OWL-style semantics. For example, using Turtle: Turtle @prefix ex: <http://example.com/enterprise#> ex:SupportAgent a ex:Agent ; ex:hasCapability ex:ProcessRefund ; ex:accesses ex:CustomerAccount ex:ProcessRefund a ex:Capability ; ex:appliesTo ex:CustomerOrder ; ex:governedBy ex:RefundPolicy . ex:RefundPolicy a ex:Policy ; ex:maxAutomaticRefund "500" Even this small ontology gives an orchestrating agent more information than a conventional tool catalog. Instead of discovering only process_refund(order_id), the agent can understand that the capability: Operates on a CustomerOrderIs provided by the SupportAgentDepends on access to customer informationIs governed by a specific refund policy The next step is to query the ontology. For example, an orchestrator could ask: SQL SELECT ?agent WHERE { ?agent ex:hasCapability ex:ProcessRefund . } The result identifies agents that semantically provide the required capability. A richer query could also include policy and business context: SQL SELECT ?agent ?policy WHERE { ?agent ex:hasCapability ?capability . ?capability ex:appliesTo ex:CustomerOrder . ?capability ex:governedBy ?policy . } This creates the foundation for semantic agent selection. The important point is that the ontology does not replace APIs, MCP tools, or agent protocols. It sits above them as a semantic decision layer. The ontology determines: What does the request mean?Which capability is required?Which agent provides it?Which enterprise entity is affected?Which policy applies? The execution layer can then use an API, MCP tool, or A2A interaction to perform the actual action. For a production environment, this minimal model can gradually expand to include concepts such as roles, data sensitivity, regions, approval thresholds, provenance, dependencies, risk levels, and human-approval requirements. The key is to start small. A handful of well-defined enterprise concepts and relationships can already move agent orchestration from simple tool matching toward semantic, policy-aware capability discovery. Where This Architecture Fits in the Emerging Agent Stack The ontology-driven architecture fits between the agent reasoning layer and the execution layer of the emerging enterprise agent stack. A typical enterprise agent stack may include: User or application layer – where requests originateAgent/orchestration layer – interprets intent and coordinates workSemantic layer – maps intent to enterprise concepts, capabilities, policies, and eligible agentsInteraction layer – MCP, APIs, A2A, events, and messagingEnterprise systems layer – CRM, ERP, ITSM, data platforms, and internal services The shared enterprise ontology belongs primarily in the semantic layer. Its role is not to replace the LLM, agent framework, MCP server, API gateway, or A2A protocol. Instead, it provides the semantic context needed before those components are used. This separation is important because each layer solves a different problem. The reasoning layer determines what the user is asking.The ontology layer determines what that request means within the enterprise.The orchestration layer determines which agent should handle it.The interaction layer determines how the selected agent or tool is invoked.The enterprise systems layer performs or records the actual business action. As agent ecosystems become more distributed, this semantic layer can act as connective tissue across otherwise independent agent platforms. Different agents may use different models, frameworks, protocols, or runtime environments while still relying on the same enterprise definitions of capabilities, entities, and policies. The architecture therefore adds a missing layer to the emerging agent stack: semantic interoperability between reasoning and execution. Challenges: Ontology Evolution, Ownership, Federation, and Governance A shared enterprise ontology improves semantic interoperability but also brings its own operational and governance problems. Enterprise concepts are not fixed: products change, organizational structures evolve, regulations shift, and new agents bring new capabilities; the ontology must therefore evolve without breaking existing consumers. Ontology Evolution In a federated ontology landscape, a change in one domain propagates to the entire semantic layer. Changing CustomerAccount in the Customer/CRM ontology, tightening the Entitlement definition in the Product ontology, or adding a new relationship between Incident and Risk in the IT and Operations domain can affect every agent that reasons over those concepts (sometimes not immediately visible). A database schema change usually only affects one system, whereas an ontology change can impact how a sales agent qualifies a lead, how a support agent resolves a ticket, how a finance agent authorizes a payment, and how a compliance agent evaluates a policy, all at the same time. The blast radius is wide because the semantic layer is shared. Ontology evolution therefore needs practices borrowed from API lifecycle management: Versioning – URI-based version identifiers on classes and relationships (for example, ex:v2/CustomerAccount) so that agents can declare which ontology version they depend on.Backward compatibility – additive changes are preferred; breaking changes require a deprecation period with the old and new definitions coexisting.Deprecation policies – a defined timeline for retiring concepts, with explicit successor mappings so consuming agents can migrate incrementally.Change impact analysis – before publishing a change, tooling should identify which agents, tools, and downstream applications reference the affected concepts.Validation before rollout – semantic regression tests that verify existing agent queries and capability graphs still resolve correctly against the new version. Agents should declare which ontology version they are using, and orchestrators should be able to route requests to agents that share a compatible semantic baseline. This prevents a situation where two agents with different ontology versions interpret the same concept differently — which is precisely the problem the ontology was introduced to solve. Ownership A central question is: who owns the meaning of enterprise concepts? The answer is rarely the team that builds the ontology infrastructure. A platform or architecture team may maintain tooling, graph store, versioning pipeline, and publishing process — but it shouldn’t define what Customer, Risk, Incident, Entitlement, or Approval mean independently of those concepts. Those definitions carry business, legal, and operational weight. Semantic ownership should be closer to the business domain. Customer/CRM ontology should belong to teams that run CRM processes and understand what a customer relationship means commercially; Finance ontology should be owned by finance and accounting because they know the difference between an Invoice, Payment, Cost Center, and Financial Account; Security ontology (Identity, Role, Permission, Policy, and Risk) should belong to security teams who understand the compliance implications of each concept. This model (central semantic governance with distributed domain ownership) is how platform engineering works at scale. The platform team sets standards, tooling, and alignment rules across domains. Domain teams set meaning within their area. Neither can override the other unilaterally. In practice, this means having ownership metadata on every ontology concept: who owns it (team), who is the subject-matter expert, and which review process applies when that definition needs to change. If not, the ontology becomes either a bottleneck controlled by one central team or an inconsistent patchwork where every team modifies concepts without coordination. Federation Trying to force all of this into one monolithic ontology isn’t practical. Governance bottleneck; slows down domain teams; model is too abstract to be useful for any specific domain. The diagram above shows a scalable alternative: federated ontology design. Each domain has its own ontology (Customer/CRM, Finance, Product, IT and Operations, Security and HR) — they have full autonomy over the concepts, relationships, and terminology in that domain. Ontologies for each domain link to an Enterprise Ontology Core shared by all domains via explicit semantic mappings and alignment relationships. The core contains cross-domain concepts, shared vocabularies, and governance and versioning rules that make federation work. This architecture also extends outward in two directions. On one side, the enterprise core aligns with external industry ontologies — schema.org for general web semantics, GS1 for product and supply chain standards, FIBO for financial industry terminology, SNOMED CT for healthcare — allowing the enterprise to adopt established standards rather than reinventing definitions. On the other side, partner and ecosystem ontologies for suppliers, regulatory bodies, and industry-specific frameworks can federate with the core through controlled extension points. The goal is not to eliminate domain-specific meaning. It is to make the relationships between those meanings explicit and machine-readable, so that an agent operating in the Finance domain and an agent operating in the Customer domain can each reason within their own semantic context while still communicating consistently when their work intersects. Federation also means that new domains can be onboarded incrementally. A new AI agent introduced for HR or procurement does not require a redesign of the entire ontology. It connects through the same alignment and mapping pattern, inheriting cross-domain concepts while adding its own domain vocabulary. Governance Because AI agents use ontology definitions to make autonomous operational decisions, semantic changes are no longer just documentation updates — they are configuration changes with real business consequences. Changing the definition of Entitlement may cause a support agent to deny services it should approve. Altering the Policy concept may cause a finance agent to bypass a control it previously enforced. The ontology is live operational infrastructure. Governance must therefore be treated with the same rigor applied to production software releases. At minimum, it should define: Who can propose changes – any domain team can propose additions or modifications within their domain, but cross-domain changes require joint review.Who approves semantic changes – changes to core enterprise concepts should require approval from the relevant domain owners and a central semantic governance board.How conflicts between domains are resolved – when two domain teams define the same concept differently (for example, both Finance and CRM define Account), there must be a defined arbitration process that results in either a canonical definition or an explicit mapping.How provenance is recorded – every concept should carry metadata about who introduced it, when, why, and which version it appeared in, so that an audit trail is always available.How policy-related concepts are reviewed – concepts that directly govern agent behavior, such as approval thresholds, permission levels, and compliance rules, should require review by legal, compliance, or risk teams before they are published.How changes are tested against existing agent workflows – before a semantic change is released, it should be validated against a test suite of representative agent queries and orchestration flows to confirm that existing behavior is not unintentionally altered. For high-impact concepts, ontology updates should require the same level of review as a production policy change or a regulated configuration update. The bar is not arbitrary: if an agent can take a financial action, restrict access, or escalate an incident based on an ontology definition, then that definition carries the same operational weight as the code or policy that governs the same action. Ultimately, the challenge is not just building a shared ontology. It is maintaining a trusted semantic contract across teams, systems, and agents as the enterprise continuously changes. The federated architecture distributes that responsibility appropriately — each domain team owns what they understand, the core team maintains coherence, and governance ensures that changes are deliberate, reversible, and traceable. “A shared ontology is only valuable if its meaning remains governed, traceable, and trustworthy as the enterprise evolves.” Conclusion In the multi-agent enterprise, communication is only the first step. True collaboration requires a shared understanding of business meaning. Protocols connect agents; ontologies align them. As AI ecosystems grow, semantic interoperability will become as important as technical interoperability. The challenge will no longer be whether agents can communicate, but whether they understand enterprise concepts well enough to collaborate safely, consistently, and effectively. A practical starting point is simple: model three things, your key business entities, the agents that operate on them, and the policies that govern those interactions. The ontology can evolve incrementally as the agent ecosystem grows. The future of enterprise AI will depend not only on smarter agents, but on agents that share a common understanding of the enterprise they serve.

By Josephine Eskaline Joyce, Ph.D CORE

I got interested in the AI Software Factory concept because it asks a much bigger question than “How can I write code faster?” It asks how AI agents can help move an idea through the entire software development lifecycle, from planning and implementation to testing, deployment, monitoring, and learning. That difference matters. Coding assistants are already useful, but generating code is only one part of building software. I still need to understand the service, validate the change, get the right approvals, and keep production healthy. When I started researching how companies like Uber approach agent-driven development, the infrastructure behind the agents fascinated me more than the code generation itself. The real opportunity is not a collection of clever tools working independently. It is a coordinated system that gives those tools context, boundaries, and a reliable path to production. What I Mean by an AI Software Factory I think of an AI Software Factory as an assembly line for software, with specialized agents handling different jobs and humans controlling the decisions that carry meaningful risk. The factory starts with developer intent and ends with verified software running in production, with operational feedback returning to the beginning. The manufacturing analogy helps me explain it. A car factory does not rely on one machine to perform every task. Different stations have specific responsibilities, and the assembly line determines how work moves between them. Software needs the same kind of coordination. An AI Software Factory can include agents for planning, coding, review, testing, security, release verification, deployment, and monitoring. But the agents alone do not make it a factory. The operating layer needs to connect their work, enforce policies, and expose what happened at each stage. Uber was one of the examples that drew me into this topic. The research I explored described agents handling more than 70% of the company’s pull requests. What interested me most was the supporting infrastructure: agents need to understand systems, run validation, and help maintain software at scale, not simply produce changes. That is the distinction I keep coming back to. The factory is the system around the agents, not just the agents themselves. Faster Coding Does Not Automatically Mean Faster Delivery I am excited about tools like GitHub Copilot and Cursor, but I do not confuse faster syntax generation with faster software delivery. If coding gets dramatically faster while everything after it stays the same, the bottleneck simply moves downstream. More code arrives in testing. More pull requests land in review queues. Senior engineers spend more time evaluating changes, and deployment processes still have the same constraints. The team can feel incredibly productive while the amount of reliable software reaching production barely changes. This is the problem an AI Software Factory is supposed to address. Optimizing one stage is not enough when delivery depends on the whole pipeline. I find it useful to separate two questions: How quickly can I generate a change? Coding assistants can help significantly here.How reliably can I deliver that change? This depends on context, testing, review, approvals, deployment, and operations. The second question is where the broader architecture becomes important. A bigger pile of generated code is not the outcome I want. I want useful changes that pass validation, fit the system, and reach production safely. For me, the value of an AI Software Factory lies in improving that complete flow rather than celebrating raw code volume. Governance Is What Separates a Factory from Agentic Chaos There is a risky version of this story: I add several autonomous agents to the development lifecycle, give them access to tools, and assume the result will be efficient. Without a common operating layer, that can become agentic chaos. When something breaks, I need answers. Which agent changed the service? What requirements and business context did it use? Which checks passed? Who approved the release? If those answers are missing, faster execution has made the system harder to control. A governed AI Software Factory makes that information visible and makes its boundaries enforceable. I want explicit inputs, defined output destinations, security rules, and approval gates that agents cannot quietly bypass. Earlier approaches such as infrastructure as code and GitOps helped automate infrastructure and deployment workflows. Agent-driven development extends automation into planning and implementation, which makes governance across the lifecycle even more important. My preferred division of responsibility is straightforward: agents perform focused work, while humans define the rules and authorize consequential decisions. An agent should not decide what business risk is acceptable. That is why I do not see human governance as the opposite of automation. In an AI Software Factory, governance is what makes broader automation usable. I Start with Service Context, Then Route Work to Specialists Before generating a single line of code, I want the factory to fetch the service context. That means knowing who owns the service, which systems it depends on, what requirements apply, and which policies govern the change. Without that foundation, an agent can produce an implementation that looks reasonable in isolation but does not fit the organization’s actual system. The initial context is not administrative overhead. It is part of the engineering input. I also would not hand an entire feature request to one giant prompt and ask a single model to do everything. Planning, implementation, review, and operations are different responsibilities. Mixing them together makes it harder to isolate mistakes and understand the output. In an AI Software Factory, I prefer a constrained handoff: Gather context: Resolve ownership, dependencies, requirements, and applicable rules.Create a plan: A planning agent maps the architecture and implementation approach.Approve the plan: A human checks the proposed direction before implementation begins.Build the change: A coding agent executes the approved, bounded plan.Validate the output: Review, testing, and security responsibilities follow through their own stages. These boundaries help contain errors rather than allowing an unsupported assumption to travel unchecked through every phase. They do not make hallucinations impossible, but they create clearer places to detect and stop them. That narrow division of labor is one of the most useful design ideas in the AI Software Factory model. CI and Human Approval Must Be Real Gates Once generated code moves forward, I still want the traditional continuous integration machinery to do its job. AI does not remove the need for builds, automated tests, or security validation. My rule is simple: a failed CI build blocks progression toward deployment. It should not become an advisory message that an agent can ignore because it thinks the change is probably fine. An AI Software Factory needs those checks wired into workflow control. For background on the underlying practice, GitHub’s continuous integration documentation explains how automated builds and tests support validation of repository changes. Passing CI is necessary, but it is not the same as permission to release. After automated validation, the workflow reaches a human review gate and pauses until someone explicitly authorizes the next step. At that gate, I want to evaluate architectural integrity, release readiness, and acceptable risk. The emphasis shifts away from spending all my attention on routine syntax checks and toward the questions that require engineering judgment. That does not mean I stop examining code when necessary. It means the factory should handle repetitive validation so that my review can focus on whether the change belongs in the system and whether it is safe to proceed. For me, an AI Software Factory protects speed by putting human judgment exactly where the stakes justify it. The Factory Continues After Deployment I do not consider deployment the finish line. Once a human approves the release, a delivery agent can move the change into production, but the workflow needs to continue into operations. A monitoring agent then checks live service health and looks for degradation after deployment. If health drops, incident response should follow predefined rules rather than depend on an agent improvising a recovery strategy. In the AI Software Factory model I explored, those responses include creating incident tickets, notifying the responsible team in Slack, and triggering predefined rollbacks. Service ownership and incident policies matter here just as much as they did during planning. The final piece is the feedback loop. Production signals and deployment outcomes need to flow back into the service context used for future work. I find this especially important because a pipeline that only moves forward is still largely a static automation sequence. A pipeline that brings operational results back to its starting point can improve the information available for the next plan. When I describe an AI Software Factory as learning, this is what I mean: deployment results and live system behavior inform subsequent work. I am not suggesting that agents should rewrite their own policies or expand their authority. The feedback improves context. Humans still own the boundaries. A Practical AI Software Factory Playground With Port To make the architecture concrete, I used a playground built around a customer support API. The developer intent was to build an API that summarizes tickets, stores conversation history, exposes REST endpoints, and can be safely deployed to production. I like this example because the request is more than a coding task. It includes functionality, persistence, an interface, and a delivery expectation. An AI Software Factory has to carry all of that intent through the lifecycle. In the playground, Port sits above the factory floor as the agentic SDLC layer. It supplies organizational context and connects agent and skill registries, workflows, policies, guardrails, and human gates. The visible stages run from intent and context through planning, building, reviewing, testing, security, release verification, deployment, post-deployment verification, and monitoring. An agent console and activity log expose the sequence of actions. When I ran the demo, the workflow encountered a human gate after planning and another before deployment. After approval, it continued through post-deployment verification and monitoring. The completed run showed 10 agents completed, 13 checks passed, two human gates, and a low release-risk indicator. I treat those numbers as the outcome of that illustrative run, not evidence that every release will be low risk. The playground demonstrates the coordination pattern; it is not proof that a real customer support API has been built and validated in a production environment. What I find useful is the visibility. I can see which specialist acted, where approval happened, and how work progressed through the AI Software Factory. The Guardrails I Want Before Giving Agents More Authority The factory floor is interesting, but I pay just as much attention to its guardrails. A workflow can look beautifully organized and still be unsafe if its permissions and release conditions are too loose. The playground’s AI Software Factory includes a concrete set of controls: Least privilege access to toolsNo secrets committed to codeDependency and license scanningStatic application security testing and injection-related checksRequired tests before releaseHuman approval for productionAn audit trail for every agent actionRollback support and post-deployment verification I do not think of these as decorations around the workflow. They determine what an agent may do, what evidence is required, and when work must stop. Least privilege limits tool access. Secret and dependency checks constrain what can enter the codebase. Audit trails help explain actions later. Production approval keeps release authority with a human, while rollback and verification extend safety beyond the initial deployment. For an AI Software Factory to be useful, these controls need to participate in execution, not just appear in a checklist. I want the orchestrator to enforce them consistently across stages. That is also why centralized service context matters. The factory needs to know the service owner, its policies, and the expected incident response before it can route work responsibly. How I Would Start Without Automating Everything on Day One My instinct as a developer is to connect the entire pipeline immediately. Planning agents, coding agents, deployment agents, monitoring agents: the possibilities are exciting. But I would resist that instinct when introducing an AI Software Factory to a team. I would begin with one small, reliable path: requirements, coding, CI, and a human review gate. The initial workflow should stop there. Before adding deployment authority, I want confidence that the inputs are clear, the checks block failures, and the approval boundary behaves as expected. Only after that loop is stable would I expand into deployment and monitoring orchestration. The point is not to have the most agents on day one. It is to establish a trustworthy handoff and then extend it. This changes how I think about the developer’s role. I am not just writing implementation details. I am defining system rules, approval criteria, service context, and the structure through which automated work moves. That is what excites me most about an AI Software Factory. The goal is not to remove engineering judgment. It is to free more of my time for that judgment by coordinating repetitive work around it. I want a factory that ships reliable software, explains its actions, and keeps meaningful decisions accountable to people. Faster code generation is useful. A governed delivery system is the bigger opportunity.

By Pavan Belagatti CORE