Integration

In the early days of the web, monitoring availability was simple: a server either responded to a ping, or it didn't. HTTP checks tightened that up a little — a 200 OK meant the dashboard turned green, and everyone assumed things were fine. That assumption doesn't really hold anymore, though. A modern app can return a picture-perfect 200 OK and still be completely unusable to an actual customer. Take an e-commerce site where the web server is healthy and responding in milliseconds. Somewhere behind it, a third-party inventory service has quietly died, or a CSS change buried the checkout button under a promo banner nobody tested for. Nobody can buy anything. Server's up. Business is down. Legacy monitoring can't see any of this — it was built to check the plumbing, not whether the person standing at the sink can actually get water out of the tap. Uptime Isn't an Infrastructure Metric Anymore In a monolithic architecture, the app and the database lived on one server, and uptime was basically a binary infrastructure question. That's not how most applications get built anymore. A typical SaaS product today is a single-page application backed by dozens of independent microservices spread across regions, plus a stack of external dependencies — an identity provider, a payment processor, a CDN, whatever else. If any one of those goes down, your own servers can be perfectly healthy while your users still can't get through a core workflow. Uptime, in that world, has to mean the continuous availability of the actual business workflow, not a response code. What Synthetic Monitoring Actually Does Synthetic monitoring uses automated clients to simulate real user traffic on a schedule, from multiple locations, around the clock — instead of waiting for a human to hit a broken flow and file a ticket. These aren't simple URL pingers, either. A synthetic monitor opens a real browser, renders the DOM, executes JavaScript, fills out forms, clicks through multi-step flows, and checks that the right data shows up on screen, all while watching the underlying API calls to make sure the backend agrees with what the UI is claiming. If a login flow that normally takes two seconds suddenly takes ten, or a button just stops responding, the monitor flags it right away, typically with a video of the failed session and enough diagnostic detail attached that someone can actually act on it, routed straight into whatever incident tool the team already uses. That's a fundamentally faster loop than "someone tweeted that checkout is broken." Where This Overlaps With QA: Shifting Right QA and production monitoring have traditionally been separate worlds — different teams, different tools, a handoff at the deployment line. That divide doesn't have much justification anymore. If a team's already built solid automated functional tests for CI/CD, there's no real reason to throw those away once code ships. The same script that validates a checkout flow pre-deploy can get repurposed to run every few minutes in production as a synthetic monitor — generally called "shifting right." Done well, it cuts duplicated engineering effort and gets QA and SRE working off the same definition of "healthy" instead of two different ones. Testing Beyond the Front Door: Complex User Journeys Basic uptime monitoring tells you the front door is open. Synthetic monitoring actually walks through the door, picks something up, applies a promo code, checks shipping, completes a transaction — the whole path, not just the entrance. That requires handling state, not just static checks. A monitor testing a healthcare portal needs to log in with synthetic credentials, get through MFA, pull a specific record, and confirm it belongs to the test account and nothing else. One testing a fintech transfer needs to confirm the UI shows success and then separately query the backend to make sure the balances actually moved, because a UI that says "success" while the ledger disagrees is arguably worse than an honest failure. Validating both the interface and the underlying state is what makes this useful for anything regulatory or revenue-critical. The Self-Healing Problem Running scripts against a live production environment is harder than running them in staging, because production changes constantly — new banners, UI experiments, shifting layouts. A rigid script breaks on cosmetic changes it shouldn't even care about, and that's how you end up with false alarms nobody trusts. This is where AI-assisted self-healing has become genuinely useful, rather than just a buzzword bolted onto a monitoring dashboard. If a button's ID changes from submit-order to confirm-purchase, a brittle script just fails. A self-healing monitor uses visual and semantic signals to relocate the element, finishes the check, and logs a low-priority note for someone to review later, instead of paging an engineer at 3 a.m. over what amounts to a rename. Alert Fatigue Is a Design Problem, Not a Tooling Problem Poorly tuned monitoring trains engineers to ignore it, and static thresholds are a big part of why. If an alert fires whenever a page takes longer than three seconds, a one-off network blip pages someone for a problem that resolves itself before anyone even looks at it. A better approach builds a dynamic baseline from historical performance data — per time of day, per day of week — and only escalates when something deviates meaningfully from that baseline. Often it's worth requiring confirmation from more than one geographic location before paging anyone at all, so a regional network hiccup doesn't wake someone up for nothing. Where This Matters Most E-commerce is the obvious one — downtime there is measured in dollars per second, and synthetic checks on cart logic, discount calculation, and payment gateway responses catch the silent revenue leaks a green uptime dashboard would never surface. Multi-tenant SaaS is a quieter version of the same problem: a single shared microservice failing can degrade the experience for every tenant at once, sometimes without anyone noticing for a while. Synthetic scripts that log in under different tenant configurations help confirm data isolation is actually holding and that SLAs are being met in practice, not just assumed on paper because nothing's screamed yet. Healthcare and fintech carry real regulatory weight on top of the operational risk. Synthetic checks that confirm patient records render correctly, or that a banking handshake with a clearing house completes securely, end up functioning as both an operational safeguard and a rough form of continuous compliance evidence — useful when an auditor eventually asks how you know. The Takeaway A green uptime dashboard doesn't mean much anymore if all it's checking is whether a server responds. The failures that actually cost money and trust — a hidden checkout button, a silently failing third-party integration, a broken multi-step flow — live above the infrastructure layer. Only something that behaves like a real user is going to catch them.

Learn how attackers enumerated Salesforce Experience Cloud and ServiceNow portals — and how defenders can detect and prevent the same abuse. When Guest Access Becomes an Attack Surface Modern enterprise portals increasingly expose APIs to unauthenticated users. The problem is not necessarily that those APIs are vulnerable. The problem is that the anonymous identity behind them may have been granted more access than the organization realizes. By now, the existence of the campaign covered in this piece isn't news. SecurityWeek, BleepingComputer, Dark Reading, and Help Net Security have all reported on it in the last few days, drawing on research published by SaaS security firm Reco. What none of that coverage had room for is the protocol-level mechanics: exactly how the enumeration works against Salesforce's two different component frameworks, exactly where ServiceNow's authorization decision actually lives, and exactly what a defender should pull from logs to tell this apart from ordinary traffic. That's the gap this article fills. In an interview arranged through Reco, I spoke with security researcher Nitay Bachrach — one of the researchers behind the original investigation — about how his team built that distinction, endpoint by endpoint. What follows combines his answers with Reco's published indicators and current Salesforce and ServiceNow platform documentation. What the City-Forum Campaign Actually Found Reco calls the activity the City-Forum campaign, after a domain tied to the operator's infrastructure. A single source has been interacting with Salesforce Experience Cloud and ServiceNow Service Portal deployments through guest-accessible interfaces since at least March 2025 — over seventeen months of continuous activity, still climbing in volume as of Reco's publication. On Salesforce, the activity spans Aura enumeration, LWR UI-API and GraphQL requests, and self-registration probing. On ServiceNow, the same infrastructure repeatedly targets the native Service Portal search endpoint. Targets span telecommunications, banking and financial services, enterprise software vendors — including security and data-privacy companies — and public-sector portals; Reco has not named individual organizations. Critically, Reco is explicit that none of this exploits a platform vulnerability. Every record retrieved was something a site owner had already exposed to anonymous users, through sharing rules, permissions, or portal search-source configuration. One Infrastructure Source, Two Enterprise Platforms Everything traces to a single IP address: 158.220.87.79, on a Contabo VPS (ASN 51167, Germany). Passive DNS ties that IP to the domain city-forum.com, registered in 2002 and long abandoned before being repurposed for this infrastructure, resolving to the operator's server since at least March 12, 2025. That's an unusually long, unrotated run for this kind of activity. Campaigns like the previously reported ShinyHunters Experience Cloud campaign have typically drawn on multiple machines and rotating IP ranges. This one hasn't — the same box has carried the same domain for the entire observed window. Verifiable indicators, independently confirmable via dig: IP: 158.220.87.79 — ASN 51167 (Contabo GmbH), reverse DNS vmi2213719.contaboserver.netDomain: city-forum.com and active subdomains www.city-forum.com, server.city-forum.com, www.server.city-forum.com, mail.city-forum.com, www.mail.city-forum.comAn SPF record explicitly authorizing the IP to send mail as the domain Reco's own guidance is worth repeating for anyone hunting this: resolve the domain rather than browsing to it. There's no legitimate reason to load attacker-adjacent infrastructure in a browser. Every request across both platforms carries the same user-agent: Go-http-client/1.1, Go's default net/http string. On its own, that identifies a client library, not a threat actor — as Bachrach put it, "it doesn't say much, except that they wrote their tools in Golang. Go is one of the two 'go-to' languages hackers use for their toolset — the other one being Python." What makes it meaningful is context: Experience Cloud sites and ServiceNow portals are built to be driven by browsers. A guest session arriving via Go-http-client is unusual enough to warrant investigation. Salesforce Aura: Enumerating the Guest Context Every Experience Cloud site has a persistent Guest User — a real identity that unauthenticated visitors execute as. It cannot be deleted, and requiring login on the site doesn't remove the underlying profile, its sharing rules, or any code running in its context. Whatever the guest identity is authorized to read may be reachable by an unauthenticated internet caller. Aura, Salesforce's older Experience Cloud framework, has a single endpoint — /aura (also /s/sfsites/aura) — that accepts a POST containing a descriptor and parameters. Reco observed high-volume guest requests against two actions: HostConfigController/ACTION$getConfigData — enumerates the objects reachable from the guest context (Account, Contact, Case, Lead, and so on).SelectableListDataProviderController/ACTION$getItems — pages through records for each object surfaced by the first call. One target generated more than 560,000 events from the campaign IP across the observation window, almost entirely attributable to guest Aura enumeration via these two actions. At that volume, the activity is consistent with systematic enumeration and potential large-scale extraction rather than ordinary application use. LWR and GraphQL: The Surface Aura Tooling Misses Lightning Web Runtime is Salesforce's newer Experience Cloud framework, and its /aura endpoint is disabled entirely. Tooling built to detect Aura enumeration — which describes most public and open-source Experience Cloud scanners — finds nothing on a pure LWR site. Not because the site is safer. Because the tooling wasn't built to look at the surface LWR actually exposes. That surface is the UI-API, under /webruntime/api/services/data/{version}/, backing both REST and GraphQL. Guest access to the entire surface is governed by one Experience Builder preference — "Allow guest users to access public APIs" — distinct from both the guest profile's "API Enabled" permission and the site's general login-required visibility toggle. Confusing these three is a common misconfiguration; disabling the wrong one leaves the UI-API fully reachable while an admin believes the site is locked down. The chain: Plain Text Guest User → LWR site → /webruntime/api/services/data/{version}/ → GraphQL or REST UI-API → Object / Field-Level Security / Sharing Rules → Returned records Reco observed guest POST requests to /webruntime/api/services/data/vNN.0/graphql, with the operator's tool stepping through consecutive API versions — v56.0 through v66.0 — against every LWR site it discovered. A representative schema-enumeration query: Plain Text query { uiapi { query { EntityDefinition(first: 2000) { edges { node { QualifiedApiName { value } KeyPrefix { value } } } } } } } That returns every object name the guest context can query — the LWR equivalent of Aura's object map, but more complete. Record queries then follow the same authorization model as Aura: object permissions, field-level security, and sharing rules on the guest profile determine what comes back. Salesforce's own GraphQL documentation confirms this directly: queries are evaluated against the object- and field-level permissions of the executing user, which for a guest session means the guest profile. Proportionally, LWR traffic was lighter than the Aura flood — a handful of requests per version per subsite. Reco reads this as the operator treating LWR as a secondary technique, consistent with Aura sites still being more common across Experience Cloud generally. How to Distinguish Automation From Legitimate API Traffic I asked Bachrach how Reco distinguished this from a legitimate, if unusual, frontend implementation calling the UI-API directly. His answer is a detection principle worth generalizing: individual indicators are weak alone, but decisive in combination. First, GraphQL activity from a guest user is unusual to begin with — a frontend component could in theory call it directly, but it's rare enough to warrant a second look on its own. Second, the requests carried Go-http-client/1.1 throughout, never a browser string, across the entire campaign window. Third, the request stream lacked everything a browser normally generates alongside API calls — HTML page loads, JavaScript asset retrieval, the general traffic a human session produces. Fourth — what Bachrach called the "final nail" — the operator systematically walked API versions from v56.0 through v66.0, a sequence no legitimate client has a reason to produce. Individually, each observation is explainable in isolation. Together, on the same source, against the same endpoint, they leave little room for an innocent explanation. That's the model worth adopting for your own detection engineering: correlate client fingerprint, endpoint sensitivity, request sequence, and surrounding traffic pattern — don't let any single one carry the conclusion. Self-Registration as a Second-Stage Opportunity Alongside enumeration, the tool appended /SiteRegister and /CommunitiesSelfReg to nearly every Experience Cloud path it discovered — consistently, across most Salesforce targets, which is what makes it a deliberate part of the methodology rather than incidental noise. The objective: determine whether self-registration is enabled. If it is, an anonymous guest can promote itself into an authenticated external user, and external users routinely see meaningfully more than the guest profile does. The relevant defensive question isn't only whether self-registration exists — it's what a successfully registered identity actually gains. If registration unlocks additional records, search sources, files, or workflow access, the registration flow is part of the attack surface, not a separate concern. ServiceNow's Hidden Search Surface The second major surface is ServiceNow's Service Portal. The operator's tool first loads the portal landing page — GET /$sp.do?...&id=landing — then concentrates nearly all remaining volume against one endpoint: HTML POST /api/now/sp/search?sysparm_cancelable=true This is native platform Java. It doesn't appear in any customization table, isn't visible in Studio, and ServiceNow publishes no API reference for it. It is, however, exactly what the stock Service Portal typeahead widget calls. Reco reverse-engineered the request shape from that widget's client controller: JSON POST /api/now/sp/search?sysparm_cancelable=true Content-Type: application/json { "query": "password", "portal": "sp", "page": "homepage", "source": ["kb", "sc"], "include_facets": false, "searchType": "typeahead", "count": 5 } The source field determines which search sources are invoked and is required — omit it, and the endpoint returns zero results with no error explaining why. I asked Bachrach what initially drew Reco's attention to an endpoint this undocumented. The trigger was correlation, not the endpoint in isolation: "After discovering the Salesforce attack, we checked that IP and its activity. Seeing the same IP hammering a specific ServiceNow API was interesting, and we knew we had to investigate it." As with LWR, the endpoint can be used entirely legitimately in a normal browser session; the user-agent is what separated this traffic from that baseline. Why HTTP 201 Is Not an Access-Control Signal This is the finding I'd flag as most operationally important for ServiceNow admins. The endpoint does not gate on authentication at the transport layer. An authenticated request and a fully anonymous one both return HTTP 201. What differs is the response body and two headers — X-Is-Logged-In and X-Is-Visitor — not the status code — a distinction Reco's own captures, shown below, make directly. An authenticated request against a readable catalog source returns real results: JSON { "result": { "results": [ { "name": "Password Reset", "type": "sc", "table": "sc_cat_item", "sys_id": "29a39e830a0a0b27007d1e200ad52253", "short_description": "Request a reset of a password for a service or an application." } ], "total_number_results": 3 } } The identical request with no Authorization header and no session cookie also returns 201, with X-Is-Logged-In: false and X-Is-Visitor: false, and an empty result set: JSON { "result": { "results": [], "additionalResults": [], "facets": {}, "$$uiNotification": [], "total_number_results": 0 } } I asked Bachrach whether any telemetry resolves the resulting ambiguity — response time, payload size, anything deterministic separating "nothing matched" from "you were blocked." He was direct about the limit: "there's no deterministic way to conclude that except for checking the configuration of that instance or, better yet, running it yourself on that endpoint." The empty 201 is genuinely uninformative in both directions. To an operator sweeping the endpoint with varying query terms, an access-denied empty result and a genuinely-no-matches empty result look identical — so they learn what's exposed by watching which queries eventually come back non-empty. To a defender watching status codes alone, a portal returning 201 all day to anonymous callers looks the same whether it's leaking data or fully locked down. Where ServiceNow Authorization Actually Happens The access decision lives entirely behind the endpoint, in the search sources wired to a portal. Three tables matter: sp_portal – the Service Portals themselves; note which are reachable without login.m2m_sp_portal_search_source – the join between a portal and the search sources it actually exposes.sp_search_source – the source definitions, either table-backed or scripted (is_scripted_source). ServiceNow's current documentation confirms this architecture directly: search sources can be configured against tables or built with custom data-fetch scripts, and administrators can apply user criteria to control who is permitted to view a given search source. Reco's comparison of two stock sources illustrates the range of outcomes. The Catalog source (sc) opens with an unambiguous, code-level gate, then re-checks per item: JavaScript var results = []; if (!gs.isLoggedIn()) return results; // ... then, per candidate item: if (catalog_item.canViewOnSearch()) { /* include */ } The Knowledge Base source (kb) has no equivalent gs.isLoggedIn() check anywhere in its script. It calls directly into new KBPortalServiceImpl().getResultData(request), and the only control between an anonymous request and KB content is whatever "Can Read" user criteria are attached to that knowledge base — a data configuration decision, not a code-level gate, and the script gives no indication either way of whether that configuration is safe. The specific pattern Reco recommends hunting for in user_criteria: any record that is active = true, advanced = false, with every scoping field — role, user, group, company, department, location — left empty. That combination resolves to true for the guest identity exactly as if public access had been explicitly granted. The built-in Any User and Any user for KB seed records that ship on every instance, with the same fixed sys_id values across deployments, are precisely this pattern. One caveat from Reco's methodology: a criteria record with advanced = true and empty scoping fields is governed by its script rather than unconstrained, and shouldn't be flagged on the empty-fields heuristic alone. Correlating Activity Across Platforms I asked Bachrach how confidently Reco could tie Aura activity, LWR activity, and ServiceNow activity to a single operator and toolset. His answer was direct: "This one was actually very easy in this case — they all originated from the same IP, a VPS, which had no legitimate activity." That's the basis for treating this as one operation rather than three unrelated anomalies: one Go binary, from one box, hitting Salesforce over two distinct frameworks and ServiceNow over a third native endpoint. Public and open-source scanning tools — AuraInspector, S-RET, CirrusGo, including the modified AuraInspector variant used in the earlier ShinyHunters campaign — don't touch webruntime at all. Whoever built this evidently researched both platforms' guest-access surfaces independently rather than adapting an existing public tool. What the Evidence Says About Attribution Reco is explicit that it doesn't know who is behind this campaign and isn't ruling anyone in or out — a position echoed in the broader reporting on the campaign as well.[^1] That restraint is worth preserving rather than reading more into the pattern than the evidence supports. On the surface, the activity resembles the previously reported ShinyHunters Experience Cloud campaign — guest enumeration of Salesforce over Aura and GraphQL. It also diverges: this operator built custom tooling rather than running a modified public scanner, and ShinyHunters has not been publicly linked to ServiceNow targeting. The Contabo infrastructure itself is generic commodity hosting, tied to no named group and absent from public threat feeds. Neither similarity nor divergence settles the question. A campaign that doesn't match a group's last observed fingerprint tells you nothing on its own — actors rewrite tooling and rent new infrastructure constantly. Reasoning from "this doesn't resemble their previous campaign" to "this must be a different actor" is a common way confident, wrong attribution gets made. One operational detail is worth noting as a soft signal, not an attribution claim: this campaign's infrastructure hasn't rotated once across the entire seventeen-month window, a different pattern from the multi-machine, rotating-range approach typically reported for other groups. Passive scanning of the box shows only SSH and a CUPS print-sharing service — no web panel, nothing dashboard-like, consistent with the box functioning purely as a scanner. Its SSH build has sat unpatched across the observation window, roughly a year and a half behind current. That's poor hygiene on infrastructure the operator evidently isn't worried about protecting, though it says little about skill either way — there's limited reason to harden a box intended to eventually be burned. Building Detections From Behavior, Not IOCs No single indicator in this campaign is sufficient, and building detection around one — an IP, a domain, a user-agent string — is fragile by design. The IP can be replaced. The domain can change. The user-agent is one line of code away from a browser string. What's harder to hide is the underlying behavior pattern. Signals worth correlating, drawn directly from this campaign's request patterns: Guest identity combined with GraphQL access on SalesforceGuest identity combined with any /webruntime/api/services/data/ trafficNon-browser client fingerprints against /aura, the UI-API, or /api/now/sp/searchSequential API-version probing across consecutive vNN.0 valuesHigh-volume getItems/getConfigData activity from a single guest sessionRepeated /SiteRegister or /CommunitiesSelfReg probing across many subsitesGuest-attributed POST /api/now/sp/search activity at a cadence inconsistent with human typeahead behaviorRows in syslog_transaction where Created by is guest against /api/now/sp/search, grouped and trended over time For Salesforce, this requires Event Monitoring (Shield or the standalone add-on) to pull AuraRequest and Sites event log files: SQL SELECT Id, LogDate, Interval, LogFile, LogFileLength FROM EventLogFile WHERE EventType IN ('AuraRequest', 'Sites') Within those logs, the columns that matter are USER_AGENT, CLIENT_IP, ACTION_MESSAGE on AuraRequest rows, and the request URI on Sites rows — any guest URI containing /webruntime/api/services/data/v is the LWR tell that detection built around Aura alone will miss entirely. For ServiceNow, the relevant data lives in syslog_transaction. Filtering on IP Address is 158.220.87.79 and URL starts with /api/now/sp/search, combined with AND or OR depending on whether you're isolating this actor or surveying all guest traffic against the endpoint, surfaces the pattern directly. Created by reading guest, Type as REST, and request volume climbing from tens per day into the hundreds are the markers Reco's investigation used. Output length is a useful secondary signal — rows returning meaningfully more than the empty-result baseline are the searches that returned content, worth investigating first. The One-Hour Exposure Assessment I asked Bachrach what he'd check first with limited time and nothing else to go on. Salesforce: Pull every guest-user sharing rule, list them, and check the conditions on each individually. Justify each one on its own merits, and assume by default that any share makes the underlying data public — even on a site believed to be configured securely. ServiceNow: Review Knowledge Base user criteria and scripted search sources specifically. Confirm every scripted source gates on gs.isLoggedIn() before touching data and uses GlideRecordSecure rather than a bare GlideRecord, and check whether any unscoped "Any User"-pattern criteria record is attached to a knowledge base that shouldn't be public. Neither check requires reproducing the campaign's traffic. Both require someone actually reading configuration that, in most organizations, hasn't been reviewed since the site or portal went live. As Bachrach told Dark Reading separately, "seeing an indicator does not mean sensitive data was stolen... that being said, whether it shows up or not, it's crucial to audit the environment." Remediation Salesforce. Work the guest profile down to least privilege: audit and strip guest sharing rules to the minimum the site genuinely needs to serve to anonymous visitors; remove object- and field-level access on anything the site doesn't render publicly; remove "Access Activities" from the guest profile; disable self-registration unless the site requires it; disable guest file access and member visibility. On LWR specifically, disable "Allow guest users to access public APIs" under Experience Builder → Workspaces → Administration → Preferences — a single toggle that closes both GraphQL and REST UI-API access at once, distinct from the guest's "API Enabled" permission (also worth disabling, but insufficient alone) and from the site's login-required visibility setting (which governs page access, not API access). ServiceNow. Map every guest-facing portal in sp_portal to its search sources via m2m_sp_portal_search_source, and detach anything a public portal doesn't need. For every remaining scripted source, read the actual data_fetch_script: confirm it gates on login state and uses GlideRecordSecure. For table-backed sources, check source_table, condition, and roles — a source pointing at a sensitive table with no role requirement is directly reachable by the guest. Audit kb_uc_can_read_mtom for unscoped grants, and when found, detach the specific join record rather than editing the shared user_criteria record — that record is reused across the instance, and direct edits carry blast radius well beyond the one knowledge base being fixed. What AI Agents Change I asked Bachrach whether the growing use of AI agents against Salesforce, ServiceNow, MCP servers, CI/CD systems, and internal workflows could turn these guest-accessible surfaces into an indirect attack path for autonomous systems never intended to go looking for exposed data. "This is almost guaranteed," he said. "AI agents often try anything they can. They see a Salesforce site or a ServiceNow portal — they will try to scan it using the relevant tools or methods." That's an expert assessment of emerging risk, not a claim that agents are currently exploiting this specific campaign's exposure — worth being precise about. An agent given a browsing tool, an HTTP client, and a task doesn't inherently understand an organization's intended boundary between "guest" and "authenticated" — it understands what a given request returns. The same access model becomes more significant as organizations deploy autonomous agents capable of discovering and interacting with enterprise applications on their own initiative, without a human deciding in advance which endpoints are safe to query. That's a meaningful shift in the threat model, even though it's forward-looking rather than something this campaign's evidence directly demonstrates. A guest misconfiguration that today requires a deliberately built Go tool and seventeen months of patient infrastructure could, going forward, be discovered incidentally by an agent doing something entirely unrelated to reconnaissance. Conclusion Nothing in the City-Forum campaign broke either platform. Every request behaved exactly as Salesforce's and ServiceNow's own documentation describes — GraphQL and UI-API calls evaluated against the executing user's object and field permissions, search sources returning whatever their configured user criteria allow. That's precisely what makes the finding worth taking seriously rather than filing away as a routine scanning report. The question defenders need to keep asking isn't "is this endpoint vulnerable?" It's "what is the guest identity behind this endpoint actually authorized to do, as configured today" — and that answer needs to be re-verified on a schedule, not assumed once at launch and left alone. An attacker with a single Go binary and over a year of undisturbed infrastructure found the answer to that question across a wide range of organizations before those organizations found it themselves. As guest-accessible interfaces become a surface that autonomous agents may reach independently, closing that gap stops being a lower-priority audit item. IOCs/Defensive References IP: 158.220.87.79 (ASN 51167, Contabo GmbH; rDNS vmi2213719.contaboserver.net)Domain: city-forum.com (resolving to the above IP since at least 2025-03-12; registered 2002, since abandoned)Active subdomains: city-forum.com, www.city-forum.com, server.city-forum.com, www.server.city-forum.com, mail.city-forum.com, www.mail.city-forum.comUser-agent: Go-http-client/1.1Salesforce: guest /aura calls to getItems/getConfigData; guest requests to /webruntime/api/services/data/vNN.0/graphql sweeping v56.0–v66.0; guest hits on /SiteRegister and /CommunitiesSelfRegServiceNow: guest POST /api/now/sp/search?sysparm_cancelable=true at escalating volume, Created by = guest Research and indicators referenced in this piece are drawn from Reco's City-Forum campaign investigation. Interview quotes from Nitay Bachrach were obtained in an interview arranged through Reco's PR representative. Sources: Long-running Data Theft Campaign Targeting Salesforce, ServiceNow — Dark Reading"City-Forum" data-theft attacks target Salesforce, ServiceNow portals — BleepingComputerThe "City-Forum" Campaign — Reco (original research)A stranger has been reading Salesforce and ServiceNow portals worldwide for 17 months — Help Net SecurityStealthy 'City-Forum' Attacks Target Salesforce and ServiceNow With Custom Toolset — SecurityWeekQuery Objects | Query Records | GraphQL API — Salesforce DevelopersDefine a search source — ServiceNow DocumentationApply user criteria to a search source — ServiceNow Documentation

By Igboanugo David Ugochukwu CORE

In this blog, you will take a closer look at the different exchange types that can be used in RabbitMQ. All are demonstrated by means of examples in a Spring Boot application. Enjoy! Introduction In the previous blog, you learned the basic concepts of RabbitMQ and how to use it in a Spring Boot application. However, you only scratched the surface of it, so now it is time to dig a bit deeper into the different exchange types. If you are not yet familiar with the basic concepts, it is advised to read the previous blog. The official RabbitMQ documentation also provides detailed information that is worth reading. Sources used in this blog can be found on GitHub. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose;Basic knowledge of RabbitMQ. Topics The code can be found in the topics module. In the previous blog, you created two consumers A and B. Consumer A was bound to Queue A with routing key event.general.*. Consumer B was bound to Queue B with routing keys event.general.* and event.specific.*. The asterisk (*) wildcard was used and is a substitute for exactly one word. In the examples, the routing keys event.general.message and event.specific.message were used. You can also use the hash (#) wildcard, and this is a substitute for zero or more words. This is visualized in the figure below. In the RabbitMqConfig, you declare queue C and bind it to the TopicExchange with routing key event.general.#. Java public static final String QUEUE_CONSUMER_C = "consumer-c.queue"; public static final String ROUTING_KEY_NESTED_GENERAL_MESSAGE = "event.general.#"; @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } @Bean public Queue queueConsumerC() { return new Queue(QUEUE_CONSUMER_C, false); } @Bean Binding bindingConsumerCNestedGeneral(Queue queueConsumerC, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerC).to(exchange).with(ROUTING_KEY_NESTED_GENERAL_MESSAGE); } In the MessageController, you create an endpoint for sending a message with routing key event.general.message.nested. This routing key will not match the bindings of consumers A and B. Java @RequestMapping( method = RequestMethod.POST, value = "send-nested-general" ) public ResponseEntity<Void> sendNestedGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message.nested", message); return new ResponseEntity<>(HttpStatus.CREATED); } The ReceiverC listens to messages received in queue C and prints a message. Java @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_C) public void receiveMessage(String message) { System.out.println("Queue Consumer C received <" + message + ">"); } } Start the application from within the topics module. Shell mvn spring-boot:run First, post a general message; this should be received by all consumers. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" In the application console log, you notice that all consumers receive the message. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Queue Consumer C received <This is a general message> Now, post a nested general message, which should be received only by consumer C. Shell curl -X POST http://localhost:8080/send-nested-general \ -H "Content-Type: text/plain" \ -d "This is a nested general message" In the application console log, you notice that the message is only received by consumer C. Plain Text Queue Consumer C received <This is a nested general message> Work Queues The code can be found in the work module. With work queues, you can publish a message and dispatch it to a pool of consumers. One of the consumers will pick up the message and start processing it. This is especially useful for dispatching long-running tasks. You use the default direct exchange in this case, and the queue name is used as the routing key. No need to use a custom exchange. This is visualized in the figure below. The RabbitMqConfig is quite small; you only define the queue. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_TASK = "task.queue"; @Bean public Queue queueTask() { return new Queue(QUEUE_TASK, false); } } When sending a message via an endpoint, you use the queue name as the routing key. Java @RequestMapping( method = RequestMethod.POST, value = "send-work" ) public ResponseEntity<Void> sendWorkMessage(@RequestBody String message) { messageService.sendMessage(RabbitMqConfig.QUEUE_TASK, message); return new ResponseEntity<>(HttpStatus.CREATED); } Every consumer listens to the queue. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer A <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer B <" + message + ">"); } } @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer C <" + message + ">"); } } Start the application from within the work module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-work \ -H "Content-Type: text/plain" \ -d "This is a work message" The message is processed by one consumer. Plain Text Task picked up by Consumer A <This is a work message> Fanout The code can be found in the fanout module. With fanout, you want to broadcast messages to all queues. You send messages to the exchange, but there is no need to specify a routing key. You can also ensure that temporary queues are used. When temporary queues are used, the queue name will be generated. In the RabbitMqConfig, you define a FanoutExchange. The queues are defined as an AnonymousQueue. This creates a non-durable, exclusive, auto-delete queue with a generated name. You bind the queues to the exchange. Java @Configuration public class RabbitMqConfig { public static final String FANOUT_EXCHANGE_NAME = "fanout.exchange"; @Bean FanoutExchange fanoutExchange() { return new FanoutExchange(FANOUT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new AnonymousQueue(); } @Bean Binding bindingConsumerA(Queue queueConsumerA, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange); } @Bean public Queue queueConsumerB() { return new AnonymousQueue(); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } } In order to send messages, you only need to send them to the exchange. This can be seen in the MessageService. Java public void sendMessage(String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.FANOUT_EXCHANGE_NAME, "", message); } On the receiving side, you listen to the generated queue name (thus not a specific one in this case). Java @Component public class ReceiverA { @RabbitListener(queues = "#{queueConsumerA.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = "#{queueConsumerB.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Start the application from within the fanout module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-to-all \ -H "Content-Type: text/plain" \ -d "This is a fanout message" In the application console log, you notice that the message is consumed by all queues. Plain Text Queue Consumer B received <This is a fanout message> Queue Consumer A received <This is a fanout message> RPC The code can be found in the RPC module. Remote Procedure Call (RPC) can be used when you need to execute a function on a remote application and wait for the result. The event is sent to the queue and is processed by Consumer A. The result is sent to a queue in the replyTo field of the request. The publisher waits for data to be returned on this callback queue. When the message appears, it checks the correlationId. If it matches the value of the request, the response is returned to the publisher. All of this is done automatically by the RabbitTemplate. In the RabbitMqConfig, a DirectExchange is used. With a DirectExchange, you match exactly on events; you cannot use wildcards here, just like a TopicExchange. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String DIRECT_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_RPC_MESSAGE = "event.rpc"; @Bean DirectExchange eventsExchange() { return new DirectExchange(DIRECT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, DirectExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_RPC_MESSAGE); } } The MessageController contains an endpoint for sending the event. Java @RequestMapping( method = RequestMethod.POST, value = "send-rpc" ) public ResponseEntity<Void> sendRpcMessage(@RequestBody String message) { messageService.sendMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you use convertSendAndReceive and process the response. Java public void sendMessage(String message) { Object response = rabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } } In the receiver, you receive the message and send a response. Do note that some additional processing is added in order to trigger a timeout. More on that in a moment. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public String receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); if (message.equals("This is an rpc message")) { return "success"; } else if (message.equals("This is a timeout message")) { try { Thread.sleep(10000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "success"; } else { return "failure"; } } } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is an rpc message" In the application console log, you notice that the message is consumed by consumer A, and that a successful response is received by the publisher. Plain Text Queue Consumer A received <This is an rpc message> Sender received response: success But what if it takes too long to process the message? In real life, the remote application can be unreachable for one reason or another. Send a timeout message. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the MessageService, the response will return null, and a timeout exception is raised. Plain Text Queue Consumer A received <This is a timeout message> No response received 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] o.s.amqp.rabbit.core.RabbitTemplate : Reply received after timeout for 2 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed. org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted 2026-04-25T14:50:16.790+02:00 ERROR 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] .l.DirectReplyToMessageListenerContainer : Failed to invoke listener org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted How to solve this? In this case, you are better off using the AsyncRabbitTemplate. This template is not automatically autowired, so you have to define it as a bean. Let's do so in the RabbitMqConfig. Java @Bean public AsyncRabbitTemplate asyncRabbitTemplate(RabbitTemplate rabbitTemplate) { return new AsyncRabbitTemplate(rabbitTemplate); } In the MessageController, you define an endpoint to trigger the async template. Java @RequestMapping( method = RequestMethod.POST, value = "send-async" ) public ResponseEntity<Void> sendAsyncMessage(@RequestBody String message) { messageService.sendAsyncMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you autowire the AsyncRabbitTemplate. And because it is an async call, you catch the response by means of a CompletableFuture. Java public void sendAsyncMessage(String message) { CompletableFuture<Object> future = asyncRabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); future.thenAccept(response -> { if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } }); } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-async \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the application log, you see the same result: the response is null, but no timeout exception anymore. Conclusion In this post, you learned different exchange types. Each serves its own use case. It is up to you to choose the right pattern for your use case.

By Gunter Rotsaert CORE

Head-based sampling makes a decision the instant a trace starts, before anyone knows whether that trace is boring or the one you will spend Friday night chasing. That is the wrong time to decide. At that point the request has not failed yet, and the slow dependency call that will define it is still milliseconds away. Head sampling commits before any of that is visible, so it discards a random slice of exactly the traces you will later wish you had kept. Tail-based sampling flips the order. It buffers the spans of a trace until the trace is complete, then decides once the errors and timing are actually on the record. The OpenTelemetry Collector ships a tail_sampling processor that does this well. It also has one operational trap that most tutorials skip, and getting it wrong quietly corrupts every decision the processor makes. This walks through a policy set that keeps the traces worth keeping, and then through the trap. How the Processor Actually Works The tail_sampling processor groups incoming spans by trace ID and holds them in memory. It waits decision_wait seconds for more spans in the same trace to arrive, then evaluates the buffered trace against your policies. If the decision is to sample, the whole trace is exported. Otherwise it is dropped. A minimal configuration that keeps every error and a baseline of everything else: processors: tail_sampling: decision_wait: 10s num_traces: 100000 expected_new_traces_per_sec: 1000 policies: - name: errors type: status_code status_code: status_codes: [ERROR] - name: baseline type: probabilistic probabilistic: sampling_percentage: 5 One thing to internalize early: policies are not first-match-wins. By default, every policy votes, and if any policy votes to keep, the trace is kept. The config above does not mean "errors, otherwise 5 percent." It means "keep all errors, and independently keep 5 percent of everything (including errors)." That OR behavior is usually what you want, but it surprises people who read the list top-down like an if/else. The one exception is an inverted or drop policy, which votes to drop and overrides the keep votes, though none of the policies here use that. A Policy Set That Keeps What Matters The point of tail sampling is to encode "interesting" in policy. In practice, four categories cover most of it: errors, slow requests, business-critical paths, and a low baseline so healthy traffic is still visible. YAML processors: tail_sampling: decision_wait: 15s num_traces: 200000 expected_new_traces_per_sec: 10000 policies: - name: errors type: status_code status_code: status_codes: [ERROR] - name: slow type: latency latency: threshold_ms: 1000 - name: critical-routes type: string_attribute string_attribute: key: http.route values: - /api/v1/checkout - /api/v1/payment - name: baseline type: probabilistic probabilistic: sampling_percentage: 2 This keeps every errored trace, every trace slower than a second, every trace through checkout or payment, and 2 percent of the rest. You can go further with numeric_attribute (keep transactions over a value, or traces with more than N database calls, a cheap way to catch N+1 queries), span_count (keep unusually complex traces), and and composite policies when a single condition is too blunt, for example "slow AND in production AND an API call." Reach for the composite policy when a plain latency rule would sweep in noise from batch jobs or health checks. The Trap: A Trace Decided On Half Its Spans Here is the part that breaks silently. The processor can only make a correct decision if it can see the whole trace. A trace is not correct or incorrect in isolation; a checkout trace might have twenty spans across six services. If those spans are split across two collector instances, each instance sees a fragment, evaluates a fragment, and decides on a fragment. The instance that never received the errored span happily drops the trace. You do not get an error. You get a slow, steady loss of exactly the traces your policies were written to keep, and it looks like the policies are just not matching. A single collector sidesteps this, because it sees everything, but a single collector does not scale and is a single point of failure. The first time I ran into this, error traces started disappearing the day we scaled the sampling collector from one replica to three, and nothing alerted, because the fragments that survived still parsed as valid traces. The moment you run more than one tail_sampling instance, you have to guarantee that all spans of a given trace land on the same instance. The Collector solves this with a two-tier layout. A first tier receives spans and routes them by trace ID using the load_balancing exporter (older configs call it loadbalancing, now a deprecated alias). A second tier runs the actual tail_sampling processor. Tier one, the router: YAML exporters: load_balancing: routing_key: traceID protocol: otlp: tls: insecure: true resolver: dns: hostname: otel-sampling.observability.svc.cluster.local port: 4317 service: pipelines: traces: receivers: [otlp] exporters: [load_balancing] The routing_key: traceID setting is the whole point. It hashes on trace ID so every span with the same trace ID is sent to the same downstream instance. The DNS resolver watches a headless service and keeps the backend list current as sampling pods come and go, rehashing when the set changes. Tier two, the sampler, is a normal tail_sampling pipeline that receives the already-grouped spans and exports the survivors to your backend: YAML service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch] exporters: [otlp/backend] Run tier two as a StatefulSet or a stable set of replicas behind that headless service. Put batch after tail_sampling, not before, so you are batching the survivors rather than shuffling spans ahead of the grouping. Sizing Decision_wait and Memory Two settings decide whether this is stable. decision_wait has to be longer than your slowest realistic trace, or you will evaluate traces before their tail spans arrive and drop good data. Rough starting points: 5 to 10 seconds for a monolith, 15 to 20 for microservices, 30 or more when traces cross regions. If you see "interesting" traces getting dropped, this is the first knob to turn. num_traces is the in-memory buffer, and memory is the constraint people hit. A workable estimate: YAML num_traces ≈ expected_new_traces_per_sec × decision_wait × 1.2 memory ≈ average_trace_size × num_traces At 10,000 traces per second, a 15-second wait, and 10 KB per trace, you are holding roughly 180,000 traces and around 1.8 GB before headroom. Size the pods for it and add 20 to 30 percent buffer, because an out-of-memory kill on a sampling collector drops whatever it was holding. Confirm It Is Actually Working Do not trust it because it started. The processor emits metrics that tell you the truth: otelcol_processor_tail_sampling_count_traces_sampled breaks down kept-versus-dropped by policy. If your errors policy is sampling almost nothing, either you have very few errors or your status codes are not set the way you think.otelcol_processor_tail_sampling_sampling_trace_removal_age is how old a trace is when it leaves the buffer. At steady state, it sits near decision_wait, and that is healthy: a trace waits, gets decided, and is removed. The warning sign is the opposite. If it drops well below decision_wait, the buffer is full, and traces are being evicted before they can be decided, so raise num_traces or add replicas.otelcol_processor_tail_sampling_sampling_decision_timer_latency shows how long decisions take. It is your early warning that the instance is overloaded. The end-to-end check that matters: trigger a known error and a known slow request in a test environment, then confirm both traces show up complete in your backend. If they arrive whole, your routing is correct. If they arrive missing spans, the load-balancing tier is not doing its job, and every decision above it is suspect. Tail sampling earns its place because it keeps the traces you will actually open: the failures and the slow paths, not a random 2 percent that probably misses both. But the processor is only as good as the traces it can see in one place. Get the trace-ID routing right first. The policies are the easy part.

Google's transition from Manifest V2 to Manifest V3 has been one of the most significant architectural overhauls in the history of browser extension development. For developers building ad blockers, privacy shields, or developer tools, the biggest impact is the deprecation of the blocking capabilities of the chrome.webRequest API. In its place is the chrome.declarativeNetRequest (DNR) API. Instead of letting extensions intercept and inspect network traffic in real-time, the browser now executes filtering on behalf of the extension using declarative rules. Understanding how to design, register, and optimize these declarative rules is essential for building modern web-filtering software. Here is a technical breakdown of the DNR API architecture, rule structure, dynamic rule updates, and current platform constraints. The Architectural Shift: Interception vs. Declaration In Manifest V2, network filtering occurred within the extension's background page or service worker. The extension registered a listener that executed JavaScript on every request before it was sent: JavaScript // The MV2 blocking request pattern (deprecated) chrome.webRequest.onBeforeRequest.addListener( (details) => { if (shouldBlock(details.url)) { return { cancel: true }; } }, { urls: ["<all_urls>"] }, ["blocking"] ); While highly flexible, this design introduced two major problems: Performance Overhead: The browser had to pause network requests, spin up the extension's background process, serialize the request metadata, run the extension's custom JavaScript, and wait for a response.User Privacy: Extensions required the broad <all_urls> permission, giving them access to read every request header, URL query parameter, and POST payload. Manifest V3 solves this by moving the execution engine into the browser itself. The extension defines what needs to be blocked or redirected beforehand. The browser reads these rules and applies them natively during the network stack lifecycle. The extension’s code is never executed during the request, which reduces memory consumption and protects user privacy. The Anatomy of a Declarative Rule Under the DNR model, everything is defined using rules. Each rule is a JSON object that specifies an action and the conditions under which that action should execute. Here is the standard structure of a declarative rule: JSON { "id": 1, "priority": 1, "action": { "type": "block" }, "condition": { "urlFilter": "||doubleclick.net", "resourceTypes": ["script", "sub_frame"] } } Every rule requires four primary keys: id: A unique integer (1 or greater) that identifies the rule.priority: An integer indicating order of execution. Rules with higher priority numbers override lower priority rules.action: Specifies what the browser should do when a match occurs. Valid types include block, redirect, allow (bypasses other blocks), allowAllRequests (bypasses all rules on a page), and modifyHeaders.condition: The criteria that must be met to trigger the action. This can filter by domain, URL pattern, initiator origin, request method, or resource type (such as image, xmlhttprequest, or stylesheet). Implementing Static Rulesets Extensions can bundle pre-defined rule lists within their distribution package. These are defined as static JSON files and declared in the manifest.json: JSON { "name": "Custom Focus Blocker", "version": "1.0", "manifest_version": 3, "permissions": ["declarativeNetRequest"], "declarative_net_request": { "rule_resources": [{ "id": "ruleset_social", "enabled": true, "path": "rules/social.json" }] } } The referenced social.json file contains an array of rules: JSON [ { "id": 101, "priority": 1, "action": { "type": "block" }, "condition": { "urlFilter": "||facebook.com", "resourceTypes": ["main_frame"] } } ] Managing Dynamic Rules Programmatically Static rulesets are read-only once compiled into the extension package. To allow users to add custom blocked domains or configure personal schedules, you must update the extension's dynamic rules at runtime. Chrome provides chrome.declarativeNetRequest.updateDynamicRules to modify rules programmatically. This method accepts arrays of rules to remove and rules to add. Here is a JavaScript helper class to manage dynamic site blocking: JavaScript class BlocklistManager { // Add a domain to the dynamic blocklist static async addDomain(ruleId, domain) { const newRule = { id: ruleId, priority: 1, action: { type: 'block' }, condition: { urlFilter: `*://${domain}/*`, resourceTypes: ['main_frame', 'sub_frame'] } }; await chrome.declarativeNetRequest.updateDynamicRules({ removeRuleIds: [ruleId], // Remove old rule with same ID to prevent duplicates addRules: [newRule] }); } // Remove a rule from the active dynamic set static async removeRule(ruleId) { await chrome.declarativeNetRequest.updateDynamicRules({ removeRuleIds: [ruleId] }); } // Retrieve all currently active dynamic rules static async getActiveRules() { return await chrome.declarativeNetRequest.getDynamicRules(); } } Session Rules vs. Dynamic Rules In addition to dynamic rules, Manifest V3 introduces Session Rules via the chrome.declarativeNetRequest.updateSessionRules API. Dynamic Rules: Persist across browser restarts and extension updates. They are stored in Chrome's internal extension storage.Session Rules: Saved purely in memory. They are cleared when the browser session ends, or the extension is reloaded. Session rules are ideal for temporary focus sessions, one-time study blocks, or incognito mode rules that should not write data permanently to the disk. Modifying HTTP Headers The DNR API also supports modifying HTTP request and response headers natively using the modifyHeaders action. This is useful for removing tracking cookies, injecting authentication tokens, or overriding Referrer headers. Here is a rule structure that strips the Cookie header from requests sent to a third-party tracking domain: JSON { "id": 201, "priority": 2, "action": { "type": "modifyHeaders", "requestHeaders": [ { "header": "cookie", "operation": "remove" } ] }, "condition": { "urlFilter": "||tracker-domain.com", "resourceTypes": ["xmlhttprequest", "sub_frame"] } } Platform Constraints and Rule Limits Because the browser must parse and evaluate all active rules in linear time to avoid latency, Google enforces strict limits on the number of rules you can register: Static Rulesets: An extension can declare up to 100 static rulesets, but only a limited number can be enabled simultaneously (typically 50).Dynamic and Session Rules: Extensions are limited to 5,000 dynamic rules and 5,000 session rules.Regex Filter Performance: You can use regular expressions in the regexFilter key under conditions, but the regex patterns must conform to a restricted syntax. Lookaheads, lookbehinds, backreferences, and lazy quantifiers are disabled to guarantee that matching runs in linear time. If a regex pattern is too complex, the API will fail to register the rule. Conclusion and Best Practices When building extensions under Manifest V3: Use Priorities Wisely: Use higher priority values for user-defined whitelists to ensure they override system-level blocklists.Minimize Rule Count: Instead of creating separate rules for sub.domain.com and domain.com, use wildcard patterns or regex expressions to group matches into single rules.Optimize Storage: Clean up unused dynamic rule IDs periodically. Retrieve active rules using getDynamicRules() to prevent collisions. By moving execution to the browser engine, Manifest V3 requires developers to change their approach to web filtering. Designing within these declarative constraints ensures your extension runs efficiently without compromising user privacy.

Every B2B SaaS product team knows this moment. You're trying to close a deal, and the prospect says, "We just need you to sync with our CRM. And our HRIS. Oh, and these three other tools. You can do that, right?" Your roadmap takes a hit, and your engineering backlog doubles overnight. And eventually someone says, "What about a unified API?" It sounds like the answer — one normalized schema, one auth model, and one point of connection for a dozen or more apps in a vertical. You buy it, hook it up, and ship the integrations before the quarter ends, the integration checkbox gets checked, and you move on. For a while, it works. But there's a problem most teams don't see until they start moving upmarket. For many SaaS teams, a unified API is the right first move. It's rarely the right last one. Unified APIs Exist for a Reason, and They're Good at What They Do Most apps in a category share the same data objects. CRMs have contacts, accounts, opportunities, and activities. HRIS platforms store employee, department, and compensation data. Ticketing systems track tickets, users, and statuses. A unified API vendor abstracts the data models for an app category into a common schema so that, rather than learning a dozen APIs, your devs learn one. For startups under pressure to ship quickly, that abstraction is valuable. You can launch integrations faster, reduce engineering work, and simplify auth across the board. If your customers need common objects and standard workflows, a unified API can meaningfully accelerate your roadmap. That's all positive. The negative shows up down the road. The Lowest Common Denominator Problem A normalized data model (which is what a unified API is based on) is, by definition, a reduced or simplified data model. To present a single schema across N apps, a unified API must identify the fields they have in common. The result is a model built on the smallest shared dataset. Anything that's app-specific is abstracted away, and anything proprietary is dropped. Unified APIs work until your customers stop being generic. Enterprise customers have Salesforce custom objects built for their unique processes. They have Workday compensation structures that don't fit a normalized HRIS schema. They have vertical-specific fields that are critical to their business processes. And, it's increasingly common for them to be running systems that the unified API vendor has never heard of. The moment a prospect asks you to sync a custom object, access a proprietary field, or connect to an app outside your unified API vendor's supported list, the abstraction layer is no longer sufficient. You either tell your prospect "No" or you build a custom, one-off integration anyway, which largely defeats the point of a unified API. At first, these seem like edge cases. Then you realize enterprise customers are the edge cases. And that they are bringing the highest-value deals in your pipeline. The "Zero Maintenance" Promise Doesn't Hold Up The biggest marketing claim of a unified API is that upstream API changes are no longer your problem: "They update their API, we handle the change." In reality, you're trading one type of maintenance for another. With native APIs, you worry about endpoint deprecations, auth updates, and rate limits. With a unified API, you worry about data lost in translation or debugging through an abstraction layer. When that happens for an enterprise customer, you can't just look at the target system's logs. You have to work through the unified API provider's black box. If the root cause is a nuance in how they handle a specific app's rate-limiting rules, your engineering team is now waiting on someone else's support ticket queue. The maintenance didn't go away. It just moved down the street. Complexity Comes Later The full cost of a unified API strategy rarely appears during implementation. Instead, it waits until things have settled into a steady rhythm and then shows up as operational complexity. Dual integration architectures – Once you need custom integrations alongside your unified API (and you will), your team will maintain two separate integration layers with different auth flows, error handling, retry logic, and monitoring. Every integration request now needs to go through a decision tree to determine which of these patterns (or perhaps even a new one) you should use for development.Data model constraints – Your app connects with the unified API's schema rather than to the underlying apps. When customers ask for fields the schema doesn't expose, your team builds manual workarounds, relocating rather than reducing the complexity.Vendor roadmap dependency – If your unified API provider doesn't support a specific endpoint, a webhook behavior, an advanced API feature, or a vertical SaaS platform your customer uses, you wait (or you build around it). Either way, the original value proposition isn't holding up to the rigors of reality.Escalation cost – Enterprise prospects bring technical evaluators. When those evaluators discover that your integration can't provide the specific data they depend on, the deal may end right there. That's not good for your bottom line. What the Workaround Trap Looks Like Most teams respond the same way when they hit these limits. They start building custom integrations in addition to those handled through the unified API. What began as a simplification strategy is starting to look like this: a unified API for common integrations, direct API connections for exceptions, custom middleware for unsupported workflows, separate auth handling, multiple sync models, and one-off transformation logic wherever it's needed. In short, that neatly ordered integration layer is no longer. The abstraction created to reduce maintenance has, in fact, increased it. Teams find they're burning an appreciable portion of their integration budget maintaining low-value integrations and working around the things their unified API vendor can't support. That's engineering time that isn't being devoted to your core product. Vertical SaaS Is the Forcing Function The continued fragmentation of B2B software makes this worse every year. Beyond mainstream CRMs and HR platforms, companies increasingly rely on industry-specific applications: systems narrowly designed and built for healthcare, manufacturing, financial services, and a score of other verticals. These systems rarely conform to standardized schemas. Many of them don't appear in any unified API vendor's list of supported apps. A unified API might help you connect to ten generic CRMs. It won't help much when your largest prospect is running Epic, Procore, or a heavily customized NetSuite instance. Those are the integrations that determine whether enterprise deals close. What Happens at Scale Unified APIs are usually evaluated based on how fast they help teams launch. However, the more important question is: "What happens when integration requirements grow more complex?" Because they always do. Every single time. As SaaS products mature, integration requests shift from "Can you connect to this category?" to "Can you support this exact workflow?" That move exposes the architectural limits of a unified API. And the teams that hit the limit mid-deal (or mid-contract) feel the immediate pain. Why Embedded iPaaS Is the Durable Foundation This is where embedded iPaaS platforms fundamentally differ from unified APIs: they aren't constrained to a single simplified schema. An embedded iPaaS gives your team a flexible integration foundation that handles both ends of the spectrum: the common apps that benefit from productized integrations, and the complex, vertical-specific, niche apps that don't fit any standardized model. Some of your customers need a basic CRM sync. Others need multi-flow orchestration, conditional business logic, extensive data mapping, and more. A rigid abstraction model breaks under those requirements. An embedded iPaaS doesn't. This Isn't "Unified APIs vs. Embedded iPaaS" Unified APIs still have value. For early-stage validation or straightforward category integrations at scale, they can accelerate time-to-market. Many mature teams use them alongside a more flexible platform for the scenarios where standardization works. But for most B2B SaaS teams, they are a way-station, not the destination. The mistake teams make is assuming the abstraction can scale indefinitely as customer complexity increases. But that's not true. It can't, and it doesn't. The bigger and more complex your customers get, the more a lowest-common-denominator approach becomes an obstacle instead of a shortcut.

Artificial intelligence has rapidly become a core capability in modern software development. For Java developers, integrating these capabilities into existing enterprise applications no longer requires learning entirely new frameworks or interacting directly with complex AI APIs. Spring AI bridges this gap by providing a familiar Spring programming model for working with large language models (LLMs) from providers such as OpenAI, Google Gemini, and others. In this article, we will build a simple AI-powered REST API using Spring Boot and Spring AI while exploring practices that help move beyond proof-of-concept implementations toward production-ready enterprise applications. A Typical Enterprise Architecture Rather than allowing clients to communicate directly with an AI provider, enterprise applications usually introduce a service layer responsible for security, validation, business logic, and monitoring. Plain Text Client Application │ ▼ Spring Boot REST API │ Validation & Business Logic │ ▼ Spring AI ChatClient │ ▼ Large Language Model (OpenAI / Gemini / Azure) This architecture keeps AI interactions behind your own APIs, allowing you to enforce authentication, authorization, logging, rate limiting, and governance without exposing provider-specific details to consumers. Creating the Spring Boot Project Getting started with Spring AI is straightforward. The application requires Spring Web, Validation, and the Spring AI starter. XML <properties> <java.version>21</java.version> <spring-ai.version>1.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-validation</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-openai</artifactId> </dependency> </dependencies> Configuring the AI Model One security practice I strongly recommend is avoiding hard-coded API keys or model names inside the application. Instead, configure them using environment variables or an enterprise secrets manager. YAML spring: ai: openai: api-key: ${OPENAI_API_KEY} chat: options: model: ${OPENAI_MODEL} temperature: 0.2 The lower temperature value encourages more deterministic responses, which is generally preferable for technical or business APIs where consistency matters. Designing the API Contract Rather than exposing raw AI requests directly, I prefer defining explicit request and response models. This keeps the REST API independent of the underlying AI provider and makes future changes much easier. Java import jakarta.validation.constraints.NotBlank; import jakarta.validation.constraints.Size; public record AIQuestionRequest( @NotBlank @Size(max = 2000) String question, String audience ) {} Response model: Java public record AIAnswerResponse( String answer ) {} Configuring the ChatClient Spring AI's ChatClient is responsible for interacting with the configured language model. Rather than repeating the same instructions in every request, we can configure a default system prompt once. Java @Configuration public class AIConfiguration { @Bean ChatClient chatClient(ChatClient.Builder builder) { return builder .defaultSystem(""" You are an experienced Java architect. Provide concise, accurate, production-ready answers. Never invent APIs. If uncertain, clearly state your assumptions. """) .build(); } } The system prompt establishes the overall behavior of the assistant. It ensures that every request follows the same guidelines, resulting in more predictable responses. Implementing the AI Service One architectural decision I recommend is keeping AI interactions inside a dedicated service layer rather than calling the language model directly from a controller. This separation makes the code easier to test, improves maintainability, and keeps business logic independent of the web layer. Java @Service public class TechnicalAssistantService { private final ChatClient chatClient; public TechnicalAssistantService(ChatClient chatClient) { this.chatClient = chatClient; } public AIAnswerResponse answer(AIQuestionRequest request) { String audience = request.audience() == null ? "Java Developer" : request.audience(); String response = chatClient.prompt() .user(user -> user .text(""" Explain the following question. Audience: {audience} Question: {question} Keep the answer under 300 words. """) .param("audience", audience) .param("question", request.question())) .call() .content(); return new AIAnswerResponse(response); } } Creating the REST Controller With the service layer complete, exposing the AI functionality through a REST endpoint becomes straightforward. Java @RestController @RequestMapping("/api/ai") public class AIController { private final TechnicalAssistantService assistantService; public AIController(TechnicalAssistantService assistantService) { this.assistantService = assistantService; } @PostMapping("/ask") public ResponseEntity<AIAnswerResponse> ask( @Valid @RequestBody AIQuestionRequest request) { return ResponseEntity.ok( assistantService.answer(request)); } } The endpoint accepts a JSON request, validates the input, invokes the service layer, and returns a structured response. Returning Structured AI Responses Many AI examples simply return text. While that's useful for chat applications, enterprise APIs usually need predictable JSON responses. For example, suppose we want AI to review Java code. Instead of receiving one long paragraph, we can ask the model to return structured data. Java public record CodeReviewResponse( String summary, List<String> strengths, List<String>issues, List<String>recommendations, String riskLevel ){} Now Spring AI can map the model response directly into a Java object. Java public CodeReviewResponse review(String sourceCode){ return chatClient.prompt() .system(""" You are a Senior Java Architect. Review the code for correctness, performance, security and maintainability. """) .user(sourceCode) .call() .entity(CodeReviewResponse.class); } This approach is much cleaner than parsing raw JSON or trying to interpret free-form responses manually. It also keeps the rest of the application strongly typed. Streaming AI Responses Some AI responses can take several seconds to complete. Rather than waiting until the entire response has been generated, Spring AI allows responses to be streamed back to the client. Java @RestController @RequestMapping("/api/ai") public class StreamingController { private final ChatClient chatClient; public StreamingController(ChatClient chatClient) { this.chatClient = chatClient; } @GetMapping( value="/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) public Flux<String> stream( @RequestParam String question){ return chatClient.prompt() .user(question) .stream() .content(); } } Streaming significantly improves the user experience because clients can begin displaying the answer immediately instead of waiting for the complete response. This is especially useful for chat applications and AI assistants. Cache Responses When Appropriate AI requests introduce additional latency and cost because every request communicates with an external model. If the same prompt is frequently submitted, consider caching the response. Spring Cache makes this simple. Java @Service public class TechnicalAssistantService { @Cacheable("aiResponses") public AIAnswerResponse answer(AIQuestionRequest request) { // AI Call } } Caching works particularly well for frequently asked questions, product descriptions, technical explanations, and internal knowledge articles. Dynamic or user-specific responses generally should not be cached unless the cache key includes the relevant context. Final Thoughts What stands out to me is that Spring AI allows AI capabilities to become a natural extension of an existing Spring Boot application rather than requiring an entirely new architecture. Whether the goal is building an internal knowledge assistant, generating summaries, reviewing code, or automating repetitive tasks, the development experience remains consistent with the rest of the Spring ecosystem. That said, building a production-ready AI application involves much more than calling an LLM. Prompt design, security, validation, observability, performance, and cost management all play a critical role in delivering reliable solutions.

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs

The evolution from monolithic applications to microservices transformed enterprise software by decomposing business capabilities into independently deployable services. REST APIs, asynchronous messaging, and service discovery enabled systems that scaled both organizationally and technically. Although this model remains effective for deterministic business logic, the emergence of AI agents introduces a different execution paradigm. Instead of invoking predefined endpoints, an agent receives an objective, reasons about available capabilities, selects appropriate services, and dynamically composes a workflow. This shift changes service boundaries from business functionality to decision-making and capability orchestration. Why This Matters Traditional microservices assume that applications already know which services to invoke. An Order Service calls Inventory, Payment, and Shipping because the workflow is explicitly encoded during development. An AI agent, however, begins with an intent rather than an execution path. A request such as "purchase the least expensive laptop available and deliver it tomorrow" requires evaluating inventory, pricing, promotions, shipping constraints, and fraud policies before any API is called. The workflow is determined during execution instead of implementation. A conventional orchestration service typically resembles the following implementation. Java public OrderResponse checkout(OrderRequest request) { Inventory inventory = inventoryClient.reserve(request); Payment payment = paymentClient.authorize(request); Shipping shipment = shippingClient.schedule(request); return new OrderResponse(payment, shipment); } The implementation is deterministic because every dependency is known beforehand. Adding another payment gateway or shipping provider requires modifying orchestration logic, gradually increasing coupling between services. As enterprises integrate AI-driven workflows, continuously extending predefined execution paths becomes increasingly difficult. Agent Services replace hardcoded dependencies with capability discovery. Rather than directly invoking an Inventory Service, the runtime identifies which registered capability satisfies the current intent. Java public Tool resolve(Intent intent) { return toolRegistry.stream() .filter(tool -> tool.supports(intent)) .findFirst() .orElseThrow(() -> new ToolNotFoundException(intent.name())); } The registry enables services to advertise capabilities instead of exposing only procedural APIs. Existing microservices remain responsible for inventory reservation, payment authorization, or shipment scheduling, but the responsibility for deciding which capability should execute moves into an intelligent coordination layer. New business capabilities can therefore be introduced without rewriting orchestration code. This distinction fundamentally changes API design. Traditional REST endpoints expose operations such as /reserveInventory or /authorizePayment. Agent-oriented systems instead expose semantic capabilities like "find lowest cost supplier," "recommend shipping option," or "detect payment risk." These descriptions allow planning engines to reason about business objectives instead of matching endpoint names. Reasoning requires an additional architectural component capable of translating natural language into executable plans. This responsibility belongs to an Intent Router, which functions similarly to an API Gateway but routes requests based on semantic meaning rather than URLs. Java public ExecutionPlan plan(String goal) { Intent intent = classifier.classify(goal); Tool tool = registry.resolve(intent); return planner.create(tool, goal); } The classifier converts an objective into structured intent, the registry discovers an appropriate capability, and the planner generates an execution strategy. Once planning completes, downstream execution remains deterministic. Large language models participate only during reasoning, while conventional microservices continue enforcing validation rules, transactional consistency, and domain constraints. Separating planning from execution preserves enterprise reliability while introducing adaptive behavior. This separation also dispels a common misconception that AI agents replace microservices. Business logic continues to belong inside deterministic services because payment authorization, inventory consistency, pricing calculations, and compliance rules require predictable execution. Agent Services instead provide an intelligent layer responsible for selecting, coordinating, and sequencing those services according to business objectives. Rather than replacing existing architectures, they extend them with decision-making capabilities that previously existed only inside application code. Consequently, service boundaries begin shifting away from business entities toward reusable decision engines. Instead of embedding procurement, logistics, or fraud decisions inside multiple applications, organizations can expose these responsibilities as independent Agent Services that orchestrate existing microservices. The underlying APIs remain stable while reasoning evolves independently, enabling enterprise systems to become progressively more adaptive without sacrificing the deterministic foundations that made microservice architectures successful. Taking Memory Into Account Memory becomes the next architectural concern once planning is separated from execution. Stateless REST requests work well for isolated transactions, but agents frequently solve objectives through multiple reasoning cycles. Intermediate decisions, retrieved knowledge, user preferences, and execution history must persist beyond a single request. This context is operational rather than transactional. Business entities continue residing in relational databases, while the agent memory layer preserves reasoning state that enables future decisions to remain consistent. Java public AgentContext update(String sessionId, Observation observation) { AgentContext context = repository.load(sessionId); context.append(observation); repository.save(context); return context; } Rather than storing business records, the memory layer continuously enriches execution context with observations generated during planning. Future reasoning cycles consume this accumulated context instead of repeatedly querying downstream services, reducing redundant tool execution while maintaining continuity across long-running workflows. As objectives become more sophisticated, a single agent rarely owns every required capability. Instead of directly invoking multiple APIs, an agent can delegate specialized responsibilities to another agent while maintaining overall coordination. This interaction is based on expertise rather than ownership, allowing procurement, logistics, compliance, or fraud agents to evolve independently while sharing the same underlying microservices. Java AgentResponse response = logisticsAgent.execute( new AgentTask( "Optimize shipping route", context)); Delegation transfers structured objectives instead of procedural API calls. Each agent independently plans its assigned task before returning a deterministic result. Existing Inventory, Payment, and Shipping services remain unchanged, while the coordination layer becomes modular and extensible. Observability Implications Observability must also evolve because traditional distributed tracing explains service execution but not decision making. Understanding why an agent selected one capability over another is equally important as measuring latency or availability. Reasoning traces therefore become first-class telemetry alongside conventional application metrics. Java Span span = tracer.nextSpan() .name("agent.plan"); span.tag("goal", goal); span.tag("selectedTool", tool.name()); span.tag("confidence", score.toString()); span.end(); Capturing planning metadata allows engineering teams to correlate business outcomes with reasoning quality. An operation may succeed technically while producing an incorrect recommendation because the planner selected an unsuitable capability. Monitoring therefore expands beyond response times to include tool selection, planning confidence, execution cost, and reasoning latency. Autonomous planning also introduces governance challenges. Traditional services authorize callers before executing business logic, whereas Agent Services must additionally validate that planners invoke only approved capabilities. Every tool should expose explicit permissions and execution policies so that reasoning engines remain constrained by enterprise governance regardless of how plans are generated. Java public ToolResult execute(AgentTask task) { policyEngine.authorize(task.agent(), task.tool()); return toolExecutor.run(task); } Separating authorization from planning ensures deterministic policy enforcement around probabilistic reasoning. Existing identity providers, audit systems, and compliance frameworks remain applicable because execution ultimately flows through governed business capabilities rather than unrestricted model outputs. A Final Word The transition from microservices to Agent Services is therefore not a replacement of proven architectural principles but their natural evolution. Microservices continue delivering transactional consistency, persistence, and deterministic business logic, while Agent Services introduce planning, semantic routing, capability discovery, memory, and adaptive orchestration. The architectural boundary shifts from exposing operations to exposing decisions, allowing intelligent planners to compose existing services according to business objectives rather than predefined workflows. Enterprise platforms adopting this layered approach preserve the reliability of mature microservice ecosystems while gaining the flexibility required for AI-native applications, making Agent Services the next logical abstraction for software systems where reasoning becomes as important as execution.

By Uthej Mopathi CORE

Every on-call engineer understands this situation well. An alert fires at 2 a.m., engineers spend the first five minutes figuring out what it means, the next few minutes searching Confluence for the relevant runbook, and finally start doing something useful. By that point, an automated system that could have classified the alert and retrieved the right procedure, proposed a remediation plan, and opened a ticket in thirty seconds has saved you nothing because it didn’t exist. That’s the problem this article addresses. We are going to build a working incident triage agent using .NET 10 and .NET Aspire 9 that does exactly that chain of steps automatically. The agent receives an HTTP alert payload, which classifies it using a Groq-hosted LLM, retrieves the matching runbook section from a Qdrant vector store, asks the LLM to propose remediation steps, and escalates to PagerDuty (through a local stub). If the severity warrants it, it writes a full audit record. The system will automatically follow all these without human involvement. What makes this integration not have any single component? It’s the combination of MCP as the tool contract, Aspire as the wiring layer, and a small eval harness that prevents the agent from quietly drifting over time. Let's go through how it's built. What are we Actually Solving? Before we bring AI into this, let’s be honest about what the actual problem is because “AI for incident triage” sounds impressive but means nothing without a clear picture of what exactly the AI is doing. When an alert goes out, the on-call engineer has three jobs Is this serious? (figuring out the severity before doing anything else)What do I do about it? (find the right procedure and follow it)Who else needs to know? (escalate the right people and open a ticket) Here are the things: Job one is mostly pattern matching, Job two is where it gets interesting, and Job three is completely mechanical but needs to be well understood, including failure modes, database connection pool exhaustion, memory leaks, and disk pressure. So, the answer is already written down somewhere in your runbook. Because the engineer is not thinking; they are searching. An LLM is genuinely good at steps one and two when given the right context. It can classify alerts and turn a runbook excerpt into a clear list of actions. The tricky part is making sure it gets the right runbook excerpt in the first place. If you ask an LLM to fix a memory leak without giving it the memory runbook, you’ll get a generic answer. So, give the right context, and you get something genuinely useful. Solution Architecture The solution is split into six focused .NET Aspire projects. Each project has a single, well-defined responsibility. App Host is the entry point to run. It doesn’t serve HTTP traffic or business logic. Its only job is to tell Aspire what services exist, which one needs to start, and which configuration needs to be injected into each of them. Think of it as the framework that describes the whole system. Services Defaults is a shared library that all other projects reference. It sets up the things every service should have, including structured logging with Serilog, distributed tracing, health check endpoints, and service discovery. So, these are all wired up with a single builder.AddServiceDefaults() call and never think about it again. Agent Service is the front door. It exposes one endpoint POST/triage and drives the five-step pipeline from start to finish. It doesn’t classify alerts, talk to Qdrant, and doesn’t know what pager duty is. It just calls the right tools in the right order and assembles the final response. MCP Tool Server is where the actual work happens. It hosts four MCP tools (alert classification, runbook lookup, PagerDuty escalation, and audit writing) and exposes them over HTTP using the Model Context Protocol. The Agent Service calls these tools by name without knowing anything about their internal implementation. PagerDuty Stub is a throwaway stand-in for the real PagerDuty API. In development, you do not want to fire real pagers or need a PagerDuty account just to test the escalation step. The stub accepts the same payload, logs it, and returns a synthetic ticket. Swap it for the real endpoints in production by changing one config value. Evals Harness is a safety check. It fires six carefully chosen alerts at the live agent and checks that the responses match expectations. If fewer than five pass, the process exits with a non-zero code, and your continuous integration pipeline fails. It is the thing that tells you when a model update or a config change has quietly broken something. The data flow for a single alert looks like this. The AgentService and McpToolServer are deliberately separate processes. The agent knows nothing about embeddings, Qdrant, or PagerDuty. It only knows how to call MCP tools by name. This is the core benefit of MCP. In the future, if we update the MCP server, the agent doesn’t change at all. MCP Tool Server The McpToolServer is an ASP.NET Core minimal API that exposes four tools over the MCP streamable HTTP transport. Each tool is a static class annotated with `McpServerToolType` and `McpServerTool`. C# [McpServerToolType] public static class AlertClassifierTool { [McpServerTool, Description("Classify an alert and return severity, category, and confidence.")] public static async Task<AlertClassification> ClassifyAsync( [Description("The raw alert text to classify")] string alertText, IChatClient chatClient, ILogger<AlertClassifierTool> logger, CancellationToken ct) { var prompt = $""" You are an incident classifier. Classify the following alert: {alertText} Respond with JSON only: {{ "severity": "Critical|High|Medium|Low", "category": "short category label", "confidence": 0.0-1.0, "reasoning": "one sentence" } """; var response = await chatClient.GetResponseAsync(prompt, new ChatOptions { ResponseFormat = ChatResponseFormat.Json }, ct); return JsonSerializer.Deserialize<AlertClassification>(response.Text) ?? throw new InvalidOperationException("LLM returned empty classification"); } } The `IChatClient` and `ILogger` parameters are injected by the MCP framework via ASP.NET Core’s dependency injection container. The tool itself is stateless, a plain static method. This keeps unit testing straightforward and allows you to pass in a mock `IChatClient`, call the method, and assert on the result. The `RunbookLookupTool` follows the same pattern but takes an `IEmbeddingGenerator<string, Embedding<float>>` and a `QdrantClient` instead of a chat client. C# [McpServerTool, Description("Find the most relevant runbook excerpts for a given incident category.")] public static async Task<List<RunbookExcerpt>> LookupAsync( [Description("Incident category from classification")] string category, IEmbeddingGenerator<string, Embedding<float>> embedder, QdrantClient qdrant, IConfiguration config, CancellationToken ct) { var topK = int.Parse(config["Qdrant:TopK"] ?? "3"); var colName = config["Qdrant:CollectionName"] ?? "runbooks"; var embedResult = await embedder.GenerateAsync([category], cancellationToken: ct); var vector = embedResult[0].Vector.ToArray(); var hits = await qdrant.SearchAsync(colName, vector, limit: (ulong)topK, cancellationToken: ct); return hits.Select(h => new RunbookExcerpt( Title: h.Payload["title"].StringValue, Content: h.Payload["content"].StringValue, Score: (float)h.Score)).ToList(); } The vector query uses cosine similarity, so (high memory usage on API node) still finds the memory-pressure runbook even though the wording doesn’t match. The embeddings capture semantic meaning, not keyword overlap. Custom Embeddings with Nomic AI Nomic AI `nomic-embed-text-v1.5` model produces 768-dimensional vectors at very low cost. The only catch is that Nomic uses a non-standard API path (`POST/v1/embedding/text` rather than the OpenAI-compatible `/V1/embeddings`), so we can’t use the default OpenAI embedding adapter from `Microsoft.Extensions.AI`. Instead, we implement `IEmbeddingGenerator<string, Embedding<float>>` directly. C# internal sealed class NomicEmbeddingGenerator( IHttpClientFactory httpClientFactory, string model, ILogger<NomicEmbeddingGenerator> logger) : IEmbeddingGenerator<string, Embedding<float>> { public EmbeddingGeneratorMetadata Metadata { get; } = new("nomic", providerUri: null, defaultModelId: model); public async Task<GeneratedEmbeddings<Embedding<float>>> GenerateAsync( IEnumerable<string> values, EmbeddingGenerationOptions? options = null, CancellationToken cancellationToken = default) { var client = httpClientFactory.CreateClient("nomic"); var requestBody = new NomicEmbedRequest(model, values.ToList(), "search_document"); using var response = await client.PostAsJsonAsync( "embedding/text", requestBody, NomicJsonContext.Default.NomicEmbedRequest, cancellationToken); response.EnsureSuccessStatusCode(); var result = await response.Content.ReadFromJsonAsync( NomicJsonContext.Default.NomicEmbedResponse, cancellationToken) ?? throw new InvalidOperationException("Nomic returned an empty response body"); return new GeneratedEmbeddings<Embedding<float>>( result.Embeddings.Select(v => new Embedding<float>(v)).ToList()); } public object? GetService(Type serviceType, object? serviceKey = null) => null; public void Dispose() { } } This class implements the full `IEmbeddingGenerator<string, Embedding<float>>` contract from `Microsoft.Extensions.AI`, so the rest of the codebase, including the `RunbookLookupTool`, sees a standard interface and never needs to know it’s talking to Nomic rather than OpenAI. The `JsonSerializable` source generation at the bottom of the file `NomicJsonContext` is important for trimming-safe serialization and for performance in hot paths. Both the request and response records must be at namespace scope (not nested inside the generator class) for the source generator to work correctly. This is a common mistake that produces `SYSLIB1032` at compile time. The Agent Service The Agent Service is where the triage pipeline is assembled. It uses Semantic Kernel to handle the remediation step (where we need prompt rendering and the injection filter) and calls all other steps via `McpClient.CallToolAsync`. The pipeline in `DotNetAspireTriageAgentService.cs` looks like this. C# // Step 1 — Classify var classification = await _mcpClient.CallToolAsync<AlertClassification>( "ClassifyAsync", new { alertText = payload.AlertText }, ct); // Step 2 — Runbook lookup (skip for Medium/Low) List<RunbookExcerpt> runbooks = []; if (_lookupSeverities.Contains(classification.Severity)) { runbooks = await _mcpClient.CallToolAsync<List<RunbookExcerpt>>( "LookupAsync", new { category = classification.Category }, ct); } // Step 3 — Remediation (via Semantic Kernel for prompt filter support) var proposal = await _kernel.InvokePromptAsync<RemediationProposal>( RemediationPromptTemplate, new KernelArguments { ["alert"] = payload.AlertText, ["runbooks"] = JsonSerializer.Serialize(runbooks), ["severity"] = classification.Severity }, cancellationToken: ct); // Step 4 — Escalate var escalation = await _mcpClient.CallToolAsync<EscalationResult>( "EscalateAsync", new { classification, correlationId = payload.CorrelationId }, ct); // Step 5 — Audit await _mcpClient.CallToolAsync( "WriteAuditAsync", new { classification, proposal, escalation }, ct); Defending Against Prompt Injection Prompt injection is a real concern in agentic systems where user-supplied text ends up literally inside an LLM prompt. An attacker who controls the alert body could try to override the system prompt and redirect the agent’s behavior. Prevent here uses Semantic Kernel’s `IPromptRenderFilter`, which fires after the prompt template is rendered but before the rendered string is sent to the model. C# public sealed class PromptInjectionFilter( InjectionDetectionContext context, ILogger<PromptInjectionFilter> logger) : IPromptRenderFilter { // Matches common injection patterns: "ignore previous instructions", // "disregard your system prompt", role-switching attempts, etc. private static readonly Regex InjectionPattern = new( @"(?i)(ignore\s+(all\s+)?(previous|prior|above)\s+instructions?" + @"|disregard\s+(your\s+)?(system\s+prompt|instructions?)" + @"|you\s+are\s+now\s+(?:a\s+)?(?:an?\s+)?\w+" + @"|act\s+as\s+(if\s+you\s+are\s+)?(?:a\s+)?(?:an?\s+)?\w+)", RegexOptions.Compiled | RegexOptions.CultureInvariant); public async Task OnPromptRenderAsync( PromptRenderContext context, Func<PromptRenderContext, Task> next) { await next(context); // let the template render first if (context.RenderedPrompt is not null && InjectionPattern.IsMatch(context.RenderedPrompt)) { context.RenderedPrompt = InjectionPattern.Replace( context.RenderedPrompt, "[SANITISED]"); this.context.InjectionDetected = true; logger.LogWarning( "Prompt injection attempt detected and sanitised — correlationId={CorrelationId}", context.Arguments["correlationId"]); } } } The filter doesn’t abort the request. It sanitizes the offending text and sets a flag that the agent includes in the response. This is a deliberate choice where failing silently is worse than completing with a sanitized prompt, because a failed triage means a missed escalation. The response `injectionDetected` field lets downstream systems know that something suspicious happened without stopping the pipeline. Handle Everything Together with .NET Aspire The AppHost is where everything comes together. Every service, dependency, and API key is declared in one place. When we run this project, Aspire reads those declarations and automatically starts the entire system in the correct order. C# var builder = DistributedApplication.CreateBuilder(args); // API keys from user-secrets or appsettings.json var groqApiKey = builder.AddParameter("GroqApiKey", secret: true); var nomicApiKey = builder.AddParameter("NomicApiKey", secret: true); // Qdrant container — persisted between restarts var qdrant = builder.AddQdrant("vectorstore") .WithLifetime(ContainerLifetime.Persistent); // PagerDuty development stub var pagerDutyStub = builder.AddProject<Projects.DotNetAspireTriageAgent_PagerDutyStub>( "pagerduty-stub"); // MCP Tool Server — waits for Qdrant and the PagerDuty stub var pagerDutyStubEndpoint = pagerDutyStub.GetEndpoint("http"); var mcpServer = builder.AddProject<Projects.DotNetAspireTriageAgent_McpToolServer>("mcp-tools") .WithReference(qdrant) .WithReference(pagerDutyStub) .WaitFor(qdrant) .WaitFor(pagerDutyStub) .WithEnvironment("Groq__ApiKey", groqApiKey) .WithEnvironment("Nomic__ApiKey", nomicApiKey) .WithEnvironment("PagerDuty__StubEndpoint", ReferenceExpression.Create($"{pagerDutyStubEndpoint}/pagerduty-stub/incidents")); // Agent Service — waits for the MCP server builder.AddProject<Projects.DotNetAspireTriageAgent_AgentService>("agent-service") .WithReference(mcpServer) .WaitFor(mcpServer) .WithEnvironment("Groq__ApiKey", groqApiKey); builder.Build().Run(); Three things in this code are worth understanding properly before moving on. .WithReference() vs .WithEnvironment(): These two look similar but do various jobs. When you call .WithReference(Qdrant), you are telling Aspire to figure out Qdrant’s host, port, and credentials at runtime and automatically inject the full connection string into McpToolServer. We do not need to mention it hardcoded anywhere. ReferenceExpression.Create. This one trips people up the first time. When McpToolServer needs to call the PagerDuty stub, it needs the stub’s full URL including the path (like domain/pagerduty-stub/incidents). The problem is you do not know the port number at the time you write the code; in this case, Aspire assigns it dynamically at startup. So instead of hardcoding a URL that will break on someone else’s machine, for this we write ReferenceExpression.Create($"{pagerDutyStubEndpoint}/pagerduty-stub/incidents") and let Aspire fill in the real address when it starts up. WaitFor This tells Aspire not to start McpToolServer until Qdrant and the PagerDuty stub are fully up and ready. Without it, McpToolServer would try to connect before they are ready and crash on the very first run. Once everything is running, the Aspire dashboard gives you a live view of the whole system. The resources tab shows all four services with their current health status and the URLs Aspire assigned to each one. The graph tab is even more useful when you are onboarding someone new to the project. It draws the exact dependency map you declared in the codebase, which service depends on which, which API keys go where, and how everything connects. Note: if a service fails to start, this graph tells you immediately which dependency in the chain is the problem instead of you having to read through logs across four different console windows. PagerDuty Stub Rather than mocking PagerDuty calls in covers or requiring a real PagerDuty account, the solution includes a lightweight stub service. It is a genuine Aspire project registered in Apphost. C# app.MapPost("/pagerduty-stub/incidents", async (HttpRequest request) => { // ... read and log the body ... var response = new PagerDutyStubResponse( Incident: new StubIncident( Id: correlationId, Status: "triggered", Number: Random.Shared.Next(1000, 9999))); return Results.Created( $"/pagerduty-stub/incidents/{correlationId}", response); }); Because the stub is a real Aspire project, its URL is dynamically allocated by Aspire and injected into McpToolServer via `ReferenceExpression.Create`. This means there are no hardcoded ports that break when someone else is already using that port, and the stub starts and stops with the rest of the solution. Swapping it for the real PagerDuty events API in Production means changing a single config value, the URL injected via `WithEnvironment`. Runbook Seeding on Startup The McpToolServer seeds its Qdrant collection on startup using a hosted service. It checks whether the collection already exists before doing any work, which means subsequent restarts are near-instant. C# public sealed class RunbookSeeder( QdrantClient qdrant, IEmbeddingGenerator<string, Embedding<float>> embedder, IConfiguration config, ILogger<RunbookSeeder> logger) : IHostedService { public async Task StartAsync(CancellationToken ct) { var collectionName = config["Qdrant:CollectionName"] ?? "runbooks"; var exists = await qdrant.CollectionExistsAsync(collectionName, ct); if (exists) { logger.LogInformation("Runbook collection already exists — skipping seed"); return; } await qdrant.CreateCollectionAsync(collectionName, new VectorsConfig(new VectorParams(size: 768, distance: Distance.Cosine)), ct); } } The runbooks test the most common failure categories, including high CPU, memory pressure, database connection exhaustion, disk saturation, network timeout, and pod restart loops. Each is stored as a Qdrant point with title and content payload fields that `RunbookLookupTool` reads back on retrieval. Eval Harness AI systems have a subtle problem that unit tests don’t catch. So, the agent can quietly get worse over time. A model version bumps, someone tweaks a prompt, a config value changes, and suddenly your critical alerts are coming back as medium with no error thrown anywhere. You only find out when a real incident gets missed. The Evals project is the safety net for exactly this. It fires six alert payloads at the live agent and checks that each response matches the expected severity, category, and escalation behavior. If fewer than five pass, the build fails. It is the same idea as a unit test suite, except it is testing the intelligence of the agent, not just the correctness of the code. Key Takeaways .NET Aspire service coordination makes it practical to run a multi-service AI agent system, including a vector database, an MCPToolServer, and an LLM-backed agent, locally with a single `dotnet run` command.The Model Context Protocol (MCP) gives you clean, language-agnostic control for exposing agent tools over HTTP, so the agent and its capabilities can evolve independently without tight coupling.Combining Nomic AI embeddings with a Qdrant vector store lets you attach a runbook knowledge base to an AI agent without fine-tuning a model that will help semantic search retrieve the right production even when the alert wording doesn’t match the runbook text exactly.Groq’s OpenAI-compatible API with `llama-3.3-70v-versatile` provides sub-second structured JSON responses, which is fast enough to complete a full five-step triage pipeline including classify, retrieve, remediate, escalate, and audit in under three seconds on most workloads.Adding a Semantic Kernel `IPromptRenderfilter` to scan every prompt render before it reaches the LLM is a lightweight, zero-overhead way to defend against prompt injection in agentic pipelines. Prerequisites To follow along with the code in this article, you will need: Visual Studio 2026 (17 or later) with the .NET Aspire package installed, or the .NET 10 SDK (10.0.300 or later) if you prefer using a terminal.Docker Desktop (4.x or later) must be running before you start because .NET Aspire automatically starts a Qdrant container.Groq API key (free get from console.groq.com) used for the alert classification and remediation via `llama-3.3-70v-versatile`.Nomic AI API key (free get from atlas.nomic.ai) used for runbook text embeddings via `nomic-embed-text-v1.5`. Note: No cloud subscription is required. Both API keys have generous free quotas that comfortably cover development and testing. Conclusion What we have built is a working blueprint for an AI triage agent that respects software engineering discipline, clean boundaries between components, a tool contract that survives dependency changes, prevents misuse, and a regression harness that makes model-level drift a continuous integration failure rather than a surprise. The combination of .NET Aspires coordination, Mcp tool abstraction, Groq’s low-latency inference, and Nomic embeddings means you can stand up a full agentic pipeline locally, with realistic dependencies, in the time it takes to run `dotnet run`. The development experience matters because it determines how quickly you can experiment, iterate, and validate changes. The next natural extensions are a persistent audit store, a document ingestion pipeline for runbooks, and a feedback loop that uses closed incidents to refine the classification prompts. All three can be added as new MCP tools without changing the agent. Appendix The complete source code for this article, including all six projects, runbook seed data, eval harness cases, and configuration examples, is available in the GitHub repository. You can clone it, run it locally with a single command, and use it as a starting point for your own incident triage pipeline. Full source code is available at the GitHub Repository.