DZone Spotlight

Learn how attackers enumerated Salesforce Experience Cloud and ServiceNow portals — and how defenders can detect and prevent the same abuse. When Guest Access Becomes an Attack Surface Modern enterprise portals increasingly expose APIs to unauthenticated users. The problem is not necessarily that those APIs are vulnerable. The problem is that the anonymous identity behind them may have been granted more access than the organization realizes. By now, the existence of the campaign covered in this piece isn't news. SecurityWeek, BleepingComputer, Dark Reading, and Help Net Security have all reported on it in the last few days, drawing on research published by SaaS security firm Reco. What none of that coverage had room for is the protocol-level mechanics: exactly how the enumeration works against Salesforce's two different component frameworks, exactly where ServiceNow's authorization decision actually lives, and exactly what a defender should pull from logs to tell this apart from ordinary traffic. That's the gap this article fills. In an interview arranged through Reco, I spoke with security researcher Nitay Bachrach — one of the researchers behind the original investigation — about how his team built that distinction, endpoint by endpoint. What follows combines his answers with Reco's published indicators and current Salesforce and ServiceNow platform documentation. What the City-Forum Campaign Actually Found Reco calls the activity the City-Forum campaign, after a domain tied to the operator's infrastructure. A single source has been interacting with Salesforce Experience Cloud and ServiceNow Service Portal deployments through guest-accessible interfaces since at least March 2025 — over seventeen months of continuous activity, still climbing in volume as of Reco's publication. On Salesforce, the activity spans Aura enumeration, LWR UI-API and GraphQL requests, and self-registration probing. On ServiceNow, the same infrastructure repeatedly targets the native Service Portal search endpoint. Targets span telecommunications, banking and financial services, enterprise software vendors — including security and data-privacy companies — and public-sector portals; Reco has not named individual organizations. Critically, Reco is explicit that none of this exploits a platform vulnerability. Every record retrieved was something a site owner had already exposed to anonymous users, through sharing rules, permissions, or portal search-source configuration. One Infrastructure Source, Two Enterprise Platforms Everything traces to a single IP address: 158.220.87.79, on a Contabo VPS (ASN 51167, Germany). Passive DNS ties that IP to the domain city-forum.com, registered in 2002 and long abandoned before being repurposed for this infrastructure, resolving to the operator's server since at least March 12, 2025. That's an unusually long, unrotated run for this kind of activity. Campaigns like the previously reported ShinyHunters Experience Cloud campaign have typically drawn on multiple machines and rotating IP ranges. This one hasn't — the same box has carried the same domain for the entire observed window. Verifiable indicators, independently confirmable via dig: IP: 158.220.87.79 — ASN 51167 (Contabo GmbH), reverse DNS vmi2213719.contaboserver.netDomain: city-forum.com and active subdomains www.city-forum.com, server.city-forum.com, www.server.city-forum.com, mail.city-forum.com, www.mail.city-forum.comAn SPF record explicitly authorizing the IP to send mail as the domain Reco's own guidance is worth repeating for anyone hunting this: resolve the domain rather than browsing to it. There's no legitimate reason to load attacker-adjacent infrastructure in a browser. Every request across both platforms carries the same user-agent: Go-http-client/1.1, Go's default net/http string. On its own, that identifies a client library, not a threat actor — as Bachrach put it, "it doesn't say much, except that they wrote their tools in Golang. Go is one of the two 'go-to' languages hackers use for their toolset — the other one being Python." What makes it meaningful is context: Experience Cloud sites and ServiceNow portals are built to be driven by browsers. A guest session arriving via Go-http-client is unusual enough to warrant investigation. Salesforce Aura: Enumerating the Guest Context Every Experience Cloud site has a persistent Guest User — a real identity that unauthenticated visitors execute as. It cannot be deleted, and requiring login on the site doesn't remove the underlying profile, its sharing rules, or any code running in its context. Whatever the guest identity is authorized to read may be reachable by an unauthenticated internet caller. Aura, Salesforce's older Experience Cloud framework, has a single endpoint — /aura (also /s/sfsites/aura) — that accepts a POST containing a descriptor and parameters. Reco observed high-volume guest requests against two actions: HostConfigController/ACTION$getConfigData — enumerates the objects reachable from the guest context (Account, Contact, Case, Lead, and so on).SelectableListDataProviderController/ACTION$getItems — pages through records for each object surfaced by the first call. One target generated more than 560,000 events from the campaign IP across the observation window, almost entirely attributable to guest Aura enumeration via these two actions. At that volume, the activity is consistent with systematic enumeration and potential large-scale extraction rather than ordinary application use. LWR and GraphQL: The Surface Aura Tooling Misses Lightning Web Runtime is Salesforce's newer Experience Cloud framework, and its /aura endpoint is disabled entirely. Tooling built to detect Aura enumeration — which describes most public and open-source Experience Cloud scanners — finds nothing on a pure LWR site. Not because the site is safer. Because the tooling wasn't built to look at the surface LWR actually exposes. That surface is the UI-API, under /webruntime/api/services/data/{version}/, backing both REST and GraphQL. Guest access to the entire surface is governed by one Experience Builder preference — "Allow guest users to access public APIs" — distinct from both the guest profile's "API Enabled" permission and the site's general login-required visibility toggle. Confusing these three is a common misconfiguration; disabling the wrong one leaves the UI-API fully reachable while an admin believes the site is locked down. The chain: Plain Text Guest User → LWR site → /webruntime/api/services/data/{version}/ → GraphQL or REST UI-API → Object / Field-Level Security / Sharing Rules → Returned records Reco observed guest POST requests to /webruntime/api/services/data/vNN.0/graphql, with the operator's tool stepping through consecutive API versions — v56.0 through v66.0 — against every LWR site it discovered. A representative schema-enumeration query: Plain Text query { uiapi { query { EntityDefinition(first: 2000) { edges { node { QualifiedApiName { value } KeyPrefix { value } } } } } } } That returns every object name the guest context can query — the LWR equivalent of Aura's object map, but more complete. Record queries then follow the same authorization model as Aura: object permissions, field-level security, and sharing rules on the guest profile determine what comes back. Salesforce's own GraphQL documentation confirms this directly: queries are evaluated against the object- and field-level permissions of the executing user, which for a guest session means the guest profile. Proportionally, LWR traffic was lighter than the Aura flood — a handful of requests per version per subsite. Reco reads this as the operator treating LWR as a secondary technique, consistent with Aura sites still being more common across Experience Cloud generally. How to Distinguish Automation From Legitimate API Traffic I asked Bachrach how Reco distinguished this from a legitimate, if unusual, frontend implementation calling the UI-API directly. His answer is a detection principle worth generalizing: individual indicators are weak alone, but decisive in combination. First, GraphQL activity from a guest user is unusual to begin with — a frontend component could in theory call it directly, but it's rare enough to warrant a second look on its own. Second, the requests carried Go-http-client/1.1 throughout, never a browser string, across the entire campaign window. Third, the request stream lacked everything a browser normally generates alongside API calls — HTML page loads, JavaScript asset retrieval, the general traffic a human session produces. Fourth — what Bachrach called the "final nail" — the operator systematically walked API versions from v56.0 through v66.0, a sequence no legitimate client has a reason to produce. Individually, each observation is explainable in isolation. Together, on the same source, against the same endpoint, they leave little room for an innocent explanation. That's the model worth adopting for your own detection engineering: correlate client fingerprint, endpoint sensitivity, request sequence, and surrounding traffic pattern — don't let any single one carry the conclusion. Self-Registration as a Second-Stage Opportunity Alongside enumeration, the tool appended /SiteRegister and /CommunitiesSelfReg to nearly every Experience Cloud path it discovered — consistently, across most Salesforce targets, which is what makes it a deliberate part of the methodology rather than incidental noise. The objective: determine whether self-registration is enabled. If it is, an anonymous guest can promote itself into an authenticated external user, and external users routinely see meaningfully more than the guest profile does. The relevant defensive question isn't only whether self-registration exists — it's what a successfully registered identity actually gains. If registration unlocks additional records, search sources, files, or workflow access, the registration flow is part of the attack surface, not a separate concern. ServiceNow's Hidden Search Surface The second major surface is ServiceNow's Service Portal. The operator's tool first loads the portal landing page — GET /$sp.do?...&id=landing — then concentrates nearly all remaining volume against one endpoint: HTML POST /api/now/sp/search?sysparm_cancelable=true This is native platform Java. It doesn't appear in any customization table, isn't visible in Studio, and ServiceNow publishes no API reference for it. It is, however, exactly what the stock Service Portal typeahead widget calls. Reco reverse-engineered the request shape from that widget's client controller: JSON POST /api/now/sp/search?sysparm_cancelable=true Content-Type: application/json { "query": "password", "portal": "sp", "page": "homepage", "source": ["kb", "sc"], "include_facets": false, "searchType": "typeahead", "count": 5 } The source field determines which search sources are invoked and is required — omit it, and the endpoint returns zero results with no error explaining why. I asked Bachrach what initially drew Reco's attention to an endpoint this undocumented. The trigger was correlation, not the endpoint in isolation: "After discovering the Salesforce attack, we checked that IP and its activity. Seeing the same IP hammering a specific ServiceNow API was interesting, and we knew we had to investigate it." As with LWR, the endpoint can be used entirely legitimately in a normal browser session; the user-agent is what separated this traffic from that baseline. Why HTTP 201 Is Not an Access-Control Signal This is the finding I'd flag as most operationally important for ServiceNow admins. The endpoint does not gate on authentication at the transport layer. An authenticated request and a fully anonymous one both return HTTP 201. What differs is the response body and two headers — X-Is-Logged-In and X-Is-Visitor — not the status code — a distinction Reco's own captures, shown below, make directly. An authenticated request against a readable catalog source returns real results: JSON { "result": { "results": [ { "name": "Password Reset", "type": "sc", "table": "sc_cat_item", "sys_id": "29a39e830a0a0b27007d1e200ad52253", "short_description": "Request a reset of a password for a service or an application." } ], "total_number_results": 3 } } The identical request with no Authorization header and no session cookie also returns 201, with X-Is-Logged-In: false and X-Is-Visitor: false, and an empty result set: JSON { "result": { "results": [], "additionalResults": [], "facets": {}, "$$uiNotification": [], "total_number_results": 0 } } I asked Bachrach whether any telemetry resolves the resulting ambiguity — response time, payload size, anything deterministic separating "nothing matched" from "you were blocked." He was direct about the limit: "there's no deterministic way to conclude that except for checking the configuration of that instance or, better yet, running it yourself on that endpoint." The empty 201 is genuinely uninformative in both directions. To an operator sweeping the endpoint with varying query terms, an access-denied empty result and a genuinely-no-matches empty result look identical — so they learn what's exposed by watching which queries eventually come back non-empty. To a defender watching status codes alone, a portal returning 201 all day to anonymous callers looks the same whether it's leaking data or fully locked down. Where ServiceNow Authorization Actually Happens The access decision lives entirely behind the endpoint, in the search sources wired to a portal. Three tables matter: sp_portal – the Service Portals themselves; note which are reachable without login.m2m_sp_portal_search_source – the join between a portal and the search sources it actually exposes.sp_search_source – the source definitions, either table-backed or scripted (is_scripted_source). ServiceNow's current documentation confirms this architecture directly: search sources can be configured against tables or built with custom data-fetch scripts, and administrators can apply user criteria to control who is permitted to view a given search source. Reco's comparison of two stock sources illustrates the range of outcomes. The Catalog source (sc) opens with an unambiguous, code-level gate, then re-checks per item: JavaScript var results = []; if (!gs.isLoggedIn()) return results; // ... then, per candidate item: if (catalog_item.canViewOnSearch()) { /* include */ } The Knowledge Base source (kb) has no equivalent gs.isLoggedIn() check anywhere in its script. It calls directly into new KBPortalServiceImpl().getResultData(request), and the only control between an anonymous request and KB content is whatever "Can Read" user criteria are attached to that knowledge base — a data configuration decision, not a code-level gate, and the script gives no indication either way of whether that configuration is safe. The specific pattern Reco recommends hunting for in user_criteria: any record that is active = true, advanced = false, with every scoping field — role, user, group, company, department, location — left empty. That combination resolves to true for the guest identity exactly as if public access had been explicitly granted. The built-in Any User and Any user for KB seed records that ship on every instance, with the same fixed sys_id values across deployments, are precisely this pattern. One caveat from Reco's methodology: a criteria record with advanced = true and empty scoping fields is governed by its script rather than unconstrained, and shouldn't be flagged on the empty-fields heuristic alone. Correlating Activity Across Platforms I asked Bachrach how confidently Reco could tie Aura activity, LWR activity, and ServiceNow activity to a single operator and toolset. His answer was direct: "This one was actually very easy in this case — they all originated from the same IP, a VPS, which had no legitimate activity." That's the basis for treating this as one operation rather than three unrelated anomalies: one Go binary, from one box, hitting Salesforce over two distinct frameworks and ServiceNow over a third native endpoint. Public and open-source scanning tools — AuraInspector, S-RET, CirrusGo, including the modified AuraInspector variant used in the earlier ShinyHunters campaign — don't touch webruntime at all. Whoever built this evidently researched both platforms' guest-access surfaces independently rather than adapting an existing public tool. What the Evidence Says About Attribution Reco is explicit that it doesn't know who is behind this campaign and isn't ruling anyone in or out — a position echoed in the broader reporting on the campaign as well.[^1] That restraint is worth preserving rather than reading more into the pattern than the evidence supports. On the surface, the activity resembles the previously reported ShinyHunters Experience Cloud campaign — guest enumeration of Salesforce over Aura and GraphQL. It also diverges: this operator built custom tooling rather than running a modified public scanner, and ShinyHunters has not been publicly linked to ServiceNow targeting. The Contabo infrastructure itself is generic commodity hosting, tied to no named group and absent from public threat feeds. Neither similarity nor divergence settles the question. A campaign that doesn't match a group's last observed fingerprint tells you nothing on its own — actors rewrite tooling and rent new infrastructure constantly. Reasoning from "this doesn't resemble their previous campaign" to "this must be a different actor" is a common way confident, wrong attribution gets made. One operational detail is worth noting as a soft signal, not an attribution claim: this campaign's infrastructure hasn't rotated once across the entire seventeen-month window, a different pattern from the multi-machine, rotating-range approach typically reported for other groups. Passive scanning of the box shows only SSH and a CUPS print-sharing service — no web panel, nothing dashboard-like, consistent with the box functioning purely as a scanner. Its SSH build has sat unpatched across the observation window, roughly a year and a half behind current. That's poor hygiene on infrastructure the operator evidently isn't worried about protecting, though it says little about skill either way — there's limited reason to harden a box intended to eventually be burned. Building Detections From Behavior, Not IOCs No single indicator in this campaign is sufficient, and building detection around one — an IP, a domain, a user-agent string — is fragile by design. The IP can be replaced. The domain can change. The user-agent is one line of code away from a browser string. What's harder to hide is the underlying behavior pattern. Signals worth correlating, drawn directly from this campaign's request patterns: Guest identity combined with GraphQL access on SalesforceGuest identity combined with any /webruntime/api/services/data/ trafficNon-browser client fingerprints against /aura, the UI-API, or /api/now/sp/searchSequential API-version probing across consecutive vNN.0 valuesHigh-volume getItems/getConfigData activity from a single guest sessionRepeated /SiteRegister or /CommunitiesSelfReg probing across many subsitesGuest-attributed POST /api/now/sp/search activity at a cadence inconsistent with human typeahead behaviorRows in syslog_transaction where Created by is guest against /api/now/sp/search, grouped and trended over time For Salesforce, this requires Event Monitoring (Shield or the standalone add-on) to pull AuraRequest and Sites event log files: SQL SELECT Id, LogDate, Interval, LogFile, LogFileLength FROM EventLogFile WHERE EventType IN ('AuraRequest', 'Sites') Within those logs, the columns that matter are USER_AGENT, CLIENT_IP, ACTION_MESSAGE on AuraRequest rows, and the request URI on Sites rows — any guest URI containing /webruntime/api/services/data/v is the LWR tell that detection built around Aura alone will miss entirely. For ServiceNow, the relevant data lives in syslog_transaction. Filtering on IP Address is 158.220.87.79 and URL starts with /api/now/sp/search, combined with AND or OR depending on whether you're isolating this actor or surveying all guest traffic against the endpoint, surfaces the pattern directly. Created by reading guest, Type as REST, and request volume climbing from tens per day into the hundreds are the markers Reco's investigation used. Output length is a useful secondary signal — rows returning meaningfully more than the empty-result baseline are the searches that returned content, worth investigating first. The One-Hour Exposure Assessment I asked Bachrach what he'd check first with limited time and nothing else to go on. Salesforce: Pull every guest-user sharing rule, list them, and check the conditions on each individually. Justify each one on its own merits, and assume by default that any share makes the underlying data public — even on a site believed to be configured securely. ServiceNow: Review Knowledge Base user criteria and scripted search sources specifically. Confirm every scripted source gates on gs.isLoggedIn() before touching data and uses GlideRecordSecure rather than a bare GlideRecord, and check whether any unscoped "Any User"-pattern criteria record is attached to a knowledge base that shouldn't be public. Neither check requires reproducing the campaign's traffic. Both require someone actually reading configuration that, in most organizations, hasn't been reviewed since the site or portal went live. As Bachrach told Dark Reading separately, "seeing an indicator does not mean sensitive data was stolen... that being said, whether it shows up or not, it's crucial to audit the environment." Remediation Salesforce. Work the guest profile down to least privilege: audit and strip guest sharing rules to the minimum the site genuinely needs to serve to anonymous visitors; remove object- and field-level access on anything the site doesn't render publicly; remove "Access Activities" from the guest profile; disable self-registration unless the site requires it; disable guest file access and member visibility. On LWR specifically, disable "Allow guest users to access public APIs" under Experience Builder → Workspaces → Administration → Preferences — a single toggle that closes both GraphQL and REST UI-API access at once, distinct from the guest's "API Enabled" permission (also worth disabling, but insufficient alone) and from the site's login-required visibility setting (which governs page access, not API access). ServiceNow. Map every guest-facing portal in sp_portal to its search sources via m2m_sp_portal_search_source, and detach anything a public portal doesn't need. For every remaining scripted source, read the actual data_fetch_script: confirm it gates on login state and uses GlideRecordSecure. For table-backed sources, check source_table, condition, and roles — a source pointing at a sensitive table with no role requirement is directly reachable by the guest. Audit kb_uc_can_read_mtom for unscoped grants, and when found, detach the specific join record rather than editing the shared user_criteria record — that record is reused across the instance, and direct edits carry blast radius well beyond the one knowledge base being fixed. What AI Agents Change I asked Bachrach whether the growing use of AI agents against Salesforce, ServiceNow, MCP servers, CI/CD systems, and internal workflows could turn these guest-accessible surfaces into an indirect attack path for autonomous systems never intended to go looking for exposed data. "This is almost guaranteed," he said. "AI agents often try anything they can. They see a Salesforce site or a ServiceNow portal — they will try to scan it using the relevant tools or methods." That's an expert assessment of emerging risk, not a claim that agents are currently exploiting this specific campaign's exposure — worth being precise about. An agent given a browsing tool, an HTTP client, and a task doesn't inherently understand an organization's intended boundary between "guest" and "authenticated" — it understands what a given request returns. The same access model becomes more significant as organizations deploy autonomous agents capable of discovering and interacting with enterprise applications on their own initiative, without a human deciding in advance which endpoints are safe to query. That's a meaningful shift in the threat model, even though it's forward-looking rather than something this campaign's evidence directly demonstrates. A guest misconfiguration that today requires a deliberately built Go tool and seventeen months of patient infrastructure could, going forward, be discovered incidentally by an agent doing something entirely unrelated to reconnaissance. Conclusion Nothing in the City-Forum campaign broke either platform. Every request behaved exactly as Salesforce's and ServiceNow's own documentation describes — GraphQL and UI-API calls evaluated against the executing user's object and field permissions, search sources returning whatever their configured user criteria allow. That's precisely what makes the finding worth taking seriously rather than filing away as a routine scanning report. The question defenders need to keep asking isn't "is this endpoint vulnerable?" It's "what is the guest identity behind this endpoint actually authorized to do, as configured today" — and that answer needs to be re-verified on a schedule, not assumed once at launch and left alone. An attacker with a single Go binary and over a year of undisturbed infrastructure found the answer to that question across a wide range of organizations before those organizations found it themselves. As guest-accessible interfaces become a surface that autonomous agents may reach independently, closing that gap stops being a lower-priority audit item. IOCs/Defensive References IP: 158.220.87.79 (ASN 51167, Contabo GmbH; rDNS vmi2213719.contaboserver.net)Domain: city-forum.com (resolving to the above IP since at least 2025-03-12; registered 2002, since abandoned)Active subdomains: city-forum.com, www.city-forum.com, server.city-forum.com, www.server.city-forum.com, mail.city-forum.com, www.mail.city-forum.comUser-agent: Go-http-client/1.1Salesforce: guest /aura calls to getItems/getConfigData; guest requests to /webruntime/api/services/data/vNN.0/graphql sweeping v56.0–v66.0; guest hits on /SiteRegister and /CommunitiesSelfRegServiceNow: guest POST /api/now/sp/search?sysparm_cancelable=true at escalating volume, Created by = guest Research and indicators referenced in this piece are drawn from Reco's City-Forum campaign investigation. Interview quotes from Nitay Bachrach were obtained in an interview arranged through Reco's PR representative. Sources: Long-running Data Theft Campaign Targeting Salesforce, ServiceNow — Dark Reading"City-Forum" data-theft attacks target Salesforce, ServiceNow portals — BleepingComputerThe "City-Forum" Campaign — Reco (original research)A stranger has been reading Salesforce and ServiceNow portals worldwide for 17 months — Help Net SecurityStealthy 'City-Forum' Attacks Target Salesforce and ServiceNow With Custom Toolset — SecurityWeekQuery Objects | Query Records | GraphQL API — Salesforce DevelopersDefine a search source — ServiceNow DocumentationApply user criteria to a search source — ServiceNow Documentation

By Igboanugo David Ugochukwu CORE

Convolutional neural network workloads rarely fail because the forward pass is mathematically difficult. They fail because modern training and inference pipelines are distributed systems: datasets arrive late, GPU workers disappear, validation jobs stall, model registration breaks halfway through, and long-running executions need to resume without corrupting state. Temporal is designed for exactly that class of problem. A Temporal Workflow Execution is durable, reliable, and scalable, and Temporal defines durable execution as the ability of a workflow to maintain state and progress through crashes or outages. That makes it a strong fit for CNN pipelines whose control plane must survive for hours, days, or even longer while the actual tensor computation runs elsewhere. Why Temporal Fits CNN Pipelines A useful way to think about Temporal in ML systems is as a durable control plane rather than a replacement for PyTorch, CUDA, or a model server. Temporal Workflows hold orchestration state, issue commands, wait on results, and recover by replaying event history. The workflow code itself must stay deterministic, while failure-prone and non-deterministic work belongs in Activities. Temporal’s own documentation is explicit on that boundary: API calls, file I/O, database access, and other external interactions belong in Activities, while workflows should remain replay-safe. That separation aligns naturally with CNN systems, where data staging, training job submission, checkpoint writes, validation, and registry updates interact with external systems constantly. That design matters because a CNN training run is not one step. Even a modest image-classification job usually has a training phase, a validation phase, best-model selection, a checkpointing loop, and a final artifact publication step. The PyTorch transfer-learning tutorial illustrates that pattern directly by alternating train and validation phases, persisting the best state_dict, and loading the best weights at the end; the quickstart tutorial likewise uses model.eval() and torch.no_grad() before prediction. Temporal does not change the math of those stages. It makes the sequence durable, observable, and restartable. A concise workflow can therefore stay almost entirely orchestration-focused: Python @workflow.defn class CnnTrainingWorkflow: @workflow.run async def run(self, req: TrainRequest) -> ModelArtifact: dataset = await workflow.execute_activity( prepare_dataset, req.dataset_ref, start_to_close_timeout=timedelta(minutes=20), ) trained = await workflow.execute_activity( train_cnn, TrainJob(dataset_uri=dataset.uri, config=req.config), start_to_close_timeout=timedelta(hours=8), heartbeat_timeout=timedelta(minutes=1), retry_policy=RetryPolicy(maximum_attempts=3), ) metrics = await workflow.execute_activity( evaluate_cnn, EvalJob(model_uri=trained.best_model_uri, dataset_uri=dataset.val_uri), start_to_close_timeout=timedelta(minutes=30), ) return await workflow.execute_activity( register_model, RegisterRequest(trained.best_model_uri, metrics), start_to_close_timeout=timedelta(minutes=5), ) The important detail in this snippet is not syntax but placement. The workflow issues durable commands, while every side effect lives inside an activity. That matches Temporal’s execution model, where workflows await activity results, activities carry retry policies, and activity timeouts define how long a unit of external work is allowed to run. For long GPU jobs, heartbeat_timeout is especially important because an activity heartbeat tells Temporal the worker is still alive and making progress. Making Long Training Jobs Resumable The natural temptation with CNN training is to keep a Python process alive for hours and hope that the host, container runtime, and storage path all behave. Temporal offers a more robust approach. Activities can be retried automatically after transient failure, and Temporal recommends Start-To-Close timeouts for activity executions. If heartbeats stop arriving inside the heartbeat timeout, the activity can be considered failed and retried according to policy. For training jobs that run on flaky GPU nodes or preemptible infrastructure, that is a meaningful improvement over ad hoc retry shells. The training activity itself should then follow framework-native checkpoint discipline instead of trying to serialize the full training loop into workflow state. PyTorch’s guidance is centered on saving and loading model state with state_dict, and the transfer-learning tutorial shows a production-relevant pattern: save the best model parameters during validation, then reload them at the end. Temporal’s own ML Ops example follows the same philosophy by highlighting checkpoint-aware fine-tuning and resumable inference, with deterministic orchestration in workflows and non-deterministic ML work in activities. A minimal activity therefore looks more like this: Python @activity.defn def train_cnn(job: TrainJob) -> TrainingResult: state = restore_checkpoint(job.resume_uri) model = build_model(job.config, state) best_acc = state.best_acc if state else 0.0 for epoch in range(state.next_epoch if state else 0, job.epochs): train_one_epoch(model, job.train_loader) val_acc = validate(model, job.val_loader) save_checkpoint(job.resume_uri, model, epoch, val_acc, best_acc) if val_acc >= best_acc: save_best_weights(job.best_model_uri, model) best_acc = val_acc activity.heartbeat({"epoch": epoch, "best_acc": best_acc}) return TrainingResult(best_model_uri=job.best_model_uri, best_acc=best_acc) This pattern is deliberately boring, which is a strength. The epoch loop stays in the activity, checkpoints remain in object storage or a shared filesystem, and each heartbeat advertises progress to Temporal. If a retry occurs, the activity can resume from the latest checkpoint rather than restarting from epoch zero. That is the same operational idea highlighted in Temporal’s ML sample repository, and it matches PyTorch’s recommendation to persist model parameters through state_dict-based saves and reloads. When training runs are submitted to an external batch scheduler instead of executing directly inside the worker process, Temporal’s asynchronous activity completion becomes especially useful. An activity can submit the job, capture the task token, and return without marking the activity complete. The external system can later heartbeat and complete the activity through a Temporal client. Temporal documents this explicitly and notes that asynchronous completion is preferable when the external process needs heartbeats or cancellation. Python @activity.defn async def submit_training(job: TrainJob): token = activity.info().task_token launch_gpu_job(job, token) activity.raise_complete_async() Treating Inference as a Workflow Only When It Is One Temporal is not a substitute for a low-latency online model server. A single image classification request that must return in milliseconds usually belongs in the serving layer. Temporal becomes valuable when inference is part of a larger durable process, such as nightly batch scoring, asynchronous document-image analysis, model fallback, approval gates, or multi-stage post-processing. That recommendation follows directly from Temporal’s model of workflows as durable, stateful executions that communicate through activities, child workflows, signals, queries, and schedules. Inside the inference activity, the framework rules remain unchanged. PyTorch examples set the model to evaluation mode and disable gradient tracking during inference. That is essential for CNNs that use dropout- or batchnorm-sensitive behavior and for avoiding unnecessary autograd overhead. Python @activity.defn def score_batch(req: BatchScoreRequest) -> BatchScoreResult: model = load_model(req.model_uri) model.eval() with torch.no_grad(): return predict_batch(model, req.input_uri) Temporal’s message-passing model then makes the surrounding orchestration easier to operate. The Python SDK documents that a workflow can act like a stateful web service receiving queries, signals, and updates. For a batch scoring workflow, a query can expose current shard progress, while a signal can switch the canary model version for the remaining work without restarting the execution. Recurring inference or retraining can be started through Temporal Schedules, which the documentation describes as a more flexible and user-friendly approach than cron jobs. Python @workflow.query def status(self) -> dict: return {"phase": self.phase, "completed": self.completed, "model": self.model_uri} @workflow.signal def switch_model(self, model_uri: str) -> None: self.model_uri = model_uri Keeping ML Workflows Operable as They Grow CNN platforms do not stay small for long. A single training run becomes a hyperparameter sweep, then a retraining program, then a fleet of region-specific models. Temporal scales that expansion through composition. A parent workflow can start child workflows for each experiment, fold, or dataset shard, and the child workflow APIs guarantee that the child has actually started before the call resolves. That makes fan-out training and batch inference easier to reason about than out-of-band job launchers with partial status tracking. Long-running ML control planes also run into two operational realities: event history growth and code evolution. Temporal addresses the first with Continue-As-New, which closes the current execution successfully and starts a new run with the same workflow ID and a fresh event history. It addresses the second with versioning support and, in production, Worker Versioning. Temporal recommends Worker Versioning as the default way to deploy changes safely, and pinned workflows can stay on the worker deployment version where they started. That is especially relevant for training or evaluation flows that may stay active across multiple application releases. Conclusion Temporal brings discipline to CNN systems by separating durable orchestration from non-deterministic GPU work. The workflow owns state, retries, waiting, composition, and observability. Activities own data movement, training submission, checkpointing, evaluation, and inference execution. PyTorch continues to provide the familiar mechanics of state_dict checkpoints, evaluation mode, and gradient-free inference, while Temporal turns those stages into a resilient end-to-end process that can survive infrastructure faults, external scheduler delays, and repeated deployment cycles. For teams building CNN platforms that have outgrown shell scripts and brittle job glue, that combination is not merely convenient. It is often the boundary between a model pipeline that occasionally works and one that can be operated confidently in production.

By Akhil Madineni CORE

A Temporal Workflow that appears stuck is rarely “stuck” in the conventional process sense. Temporal persists Workflow state through Event History and resumes execution through replay, so an open execution can remain healthy while waiting for a timer, Signal, Activity, or external condition. The operational problem is therefore not simply lack of completion; it is lack of expected progress. Effective diagnosis starts by establishing what event should have happened next, why it did not happen, and whether remediation can preserve the Workflow’s business invariants. Temporal’s history model makes that analysis unusually tractable because commands, task transitions, Activity attempts, failures, timers, and external interactions are durably represented as Events. Progress Is Visible in the Event History The first diagnostic artifact should be the execution description and raw history, not application logs. temporal workflow describe exposes current execution information and pending Activity state, while temporal workflow show --output json returns Event History in a form suitable for programmatic replay or analysis. A Workflow Query can additionally expose application-defined state without mutating the execution. Shell temporal workflow describe --workflow-id order-7814 temporal workflow show \ --workflow-id order-7814 \ --output json History should be read as a state-transition trace. A WorkflowTaskScheduled event with no corresponding start suggests that work is waiting for a Worker. A started Workflow Task that repeatedly times out can indicate blocked Workflow code, Worker instability, or excessive work inside a task. Repeated WorkflowTaskFailed events can indicate replay or deterministic-compatibility failures after code deployment. Workflow Task failures are retried by Temporal rather than governed by an Activity-style Retry Policy, so a Workflow can remain open while repeatedly failing to make application-level progress. Activity sequences reveal a different failure surface. ActivityTaskScheduled without ActivityTaskStarted points toward dispatch capacity, missing pollers, queue mismatch, or backlog. Temporal persists Workflow and Activity Tasks in Task Queues, and worker-health guidance identifies Schedule-to-Start latency and approximate backlog count as key signals when tasks wait for Workers. ActivityTaskStarted without completion requires inspection of Start-to-Close and Heartbeat behavior because Temporal relies on Start-to-Close timeout to detect a Worker crash after an Activity has started. Not every long pause is pathological. A timer that has not fired, a Workflow waiting for a Signal, or an Activity still inside a valid timeout window can represent correct durable waiting. Conversely, very large histories can become an operational risk. Temporal warns after 10,240 events or 10 MB and enforces a limit of 51,200 events or 50 MB; Continue-As-New creates a new run with a fresh history while carrying forward relevant state. Triage Works Best as Deterministic Evidence Before Model Judgment LangGraph is useful for automating this analysis, but the safest design keeps Temporal facts deterministic and uses an LLM only for classification, hypothesis ranking, and explanation. LangGraph explicitly supports graphs that mix deterministic nodes with model-driven nodes, while structured output can constrain routing decisions into a defined schema rather than free-form text. A compact analyzer can first reduce raw history into evidence that is difficult to hallucinate: the last completed Workflow Task, consecutive Workflow Task failures, pending Activity IDs, the latest Activity attempt, the timeout type, the last Signal, the last timer, the history size, the task queue, and deployment/version metadata. The model then receives that normalized evidence instead of thousands of raw events. Python def extract_facts(state): events = state["events"] return { "facts": temporal_fact_extractor(events), "tail": events[-60:], } def classify(state): result = triage_model.with_structured_output(TriageResult).invoke({ "facts": state["facts"], "tail": state["tail"], "allowed_causes": [ "worker_unavailable", "activity_retrying", "workflow_task_failure", "intentional_wait", "history_pressure", "unknown", ], }) return {"triage": result} That separation matters operationally. Event parsing can enforce hard rules such as “scheduled but never started,” while the model can correlate several weak signals and produce an explanation. Conditional edges can then route low-risk cases to observation, ambiguous cases to deeper diagnostics, and recovery candidates to an approval gate. LangGraph’s graph API supports conditional routing, and persistence stores checkpoints so triage state survives interruptions or process failures. Recovery Must Preserve Temporal and Business Semantics Diagnosis and remediation should remain separate graph stages. A model-generated recommendation must not directly issue cancellation, reset, or termination. LangGraph interrupts provide a natural control boundary because execution can pause with persisted state and resume only after external approval. Python def approval_gate(state): decision = interrupt({ "workflow_id": state["workflow_id"], "cause": state["triage"].cause, "action": state["triage"].recommended_action, "evidence": state["triage"].evidence, }) return {"approved": decision == "approve"} The remediation choice depends on the failure mode. A transient Worker outage usually requires restoring Worker capacity rather than mutating Workflow state because queued tasks persist until Workers can process them. An Activity repeatedly failing on a recoverable dependency can often be left to its Retry Policy, while permanent errors should be made non-retryable in application design to avoid pointless retries. Activity side effects should be idempotent because Activity attempts may execute more than once under retry and recovery behavior. Cancellation is the preferred stop mechanism when Workflow cleanup logic must run. Temporal records a cancellation request and schedules a Workflow Task so Workflow code can react. Termination is forceful: Workflow code does not receive a chance to clean up, and the terminated event closes the history. That makes termination an escalation path for executions that cannot process cancellation normally. Reset is more powerful and more dangerous. Temporal terminates the current execution and creates a new execution that copies history through a selected reset point, then replays forward using current Workflow code. Progress after the reset point is discarded. Reset is therefore appropriate only after the underlying cause has been corrected and after downstream side effects are reviewed for possible re-execution beyond the reset boundary. Shell temporal workflow reset \ --workflow-id order-7814 \ --event-id 42 \ --reason "Recovered after deterministic-compatibility fix" For history pressure rather than a fault, Continue-As-New is generally the safer lifecycle mechanism because it preserves logical continuity under the same Workflow ID while starting a fresh Event History with a new Run ID. It should be designed into long-lived or high-volume Workflow logic instead of used as an improvised emergency action. Safe Automation Requires an Explicit Remediation Envelope A production triage graph should treat remediation as a constrained transaction. The evidence snapshot, selected run ID, candidate reset event, intended action, reason, approval identity, and execution result should all be persisted before any mutation. The action node should re-read the Workflow immediately before execution and reject the operation if the run has changed or the observed condition no longer matches the diagnosis. This is an engineering safeguard rather than a Temporal requirement, but it reduces time-of-check/time-of-use errors when active Workflows continue progressing during investigation. LangGraph’s checkpoint model supports durable approval state, but resumed graph nodes can re-execute from checkpoint boundaries. Its documentation therefore recommends isolating side effects and designing them to be idempotent. A remediation executor should consequently use an operation ID, record completion externally, and refuse duplicate destructive actions. Recovery Without Guesswork Reliable recovery of a stuck Temporal Workflow is fundamentally an event-history problem, not a process-restart problem. The strongest diagnostic path reconstructs expected progress from Workflow Tasks, Activity attempts, timers, Signals, queue state, timeouts, and history growth before considering mutation. LangGraph can turn that evidence into a durable triage pipeline by combining deterministic extraction, constrained model reasoning, conditional routing, and interrupt-based approval. Safe remediation then follows Temporal semantics: restore Workers when dispatch is the issue, allow bounded retries for transient Activities, cancel when cleanup matters, terminate only as a last resort, reset only after the root cause is fixed, and use Continue-As-New to control long-running history growth. The result is automation that accelerates incident response without allowing probabilistic diagnosis to become an unchecked control plane.

By Akhil Madineni CORE

In a chapter of The SingleStore Cookbook, there is a complete sentiment analysis pipeline using Rust compiled to WebAssembly and loaded directly into SingleStore via its Code Engine. The result was clean: one CLI command to deploy, sentiment scoring running inside the database engine alongside the data and a full stock-price-plus-headlines analytical pipeline built on top of it. Can we do the same thing in Neo4j? Neo4j has a fully documented, officially supported extensibility model that lets us write custom functions and procedures in Java and register them directly with the database engine. Java also has a port of Valence Aware Dictionary and sEntiment Reasoner (VADER), the same lexicon-based sentiment analyzer used in the SingleStore Rust implementation. The pieces are all there. The question is how well they would fit together and what the resulting pipeline would look like compared to the SingleStore Wasm approach. This article documents an experiment from start to finish: the UDF implementation, the graph schema, a complete data loading and scoring pipeline, and a full set of analytical queries. Along the way, we also discovered that Neo4j has a second path to sentiment analysis via NLP procedures, and the choice between the two turns out to be an interesting engineering decision in its own right. The goal here isn't to claim a new sentiment-analysis technique. It's to explore what Neo4j's extension model makes possible and how the result compares with the equivalent SingleStore implementation. The full source code is available on GitHub. What We Are Building Figure 1 shows how data moves through the pipeline. CSV files are loaded into Neo4j via LOAD CSV or the Python loader. As each Headline node is created, sentiment.score() is called inline in the same Cypher statement — scoring happens inside the database at ingestion time, not in a separate application step. The resulting graph is then available for the analytical queries covered later in the article. Figure 1. Pipeline data flow The pipeline mirrors the one in the SingleStore book chapter: A VADER-based sentiment function registered with the system and callable from queriesA graph containing synthetic stock price ticks and news headlinesA set of analytical queries: per-headline scoring, daily aggregation, sentiment-vs-price joins, most positive and most negative ranking, and a live consistency check For the example in this article, we'll need a local install of Neo4j, a Docker container, or a server where we can place files and restart the process. How Neo4j Extensibility Works Neo4j lets us extend Cypher with custom Java code packaged as a .jar file. This is a fully documented and supported extensibility path. Neo4j publishes official guidance on setting up a plugin project and maintains a Neo4j Procedure Template on GitHub. Neo4j provides this extensibility model for building custom extensions. There are several extension types: User-defined functions (UDFs) – take inputs, return a single value, called inline in a query like a built-in functionUser-defined aggregation functions (UDAs) – group-level aggregation, analogous to SUM or COLLECTProcedures – more flexible, can return multiple rows and perform side effects, called with CALL For our sentiment use case, a UDF is the right fit. We pass in a string and get back a map of polarity scores. In SingleStore, the equivalent was a Table-Valued Function (TVF) that returned a row set. A Neo4j UDF returning a Map<String, Double> is the closest structural equivalent. One practical note on naming is that Neo4j maintains a list of reserved and deprecated procedure namespaces, such as db.*, dbms.*, graph.* and others. These are off-limits. The sentiment.* namespace is not reserved or deprecated, so it's a safe choice. Check User-defined procedures before choosing a namespace for any new plugin to confirm it doesn't conflict with a built-in namespace. What to Know Before We Build Because a Neo4j UDF runs inside the same JVM as the database engine, it's worth understanding a few practical considerations before diving in. These are the same considerations that apply to any extension of a running JVM process — Neo4j's own plugin authors deal with them too — and being aware of them upfront makes for a smoother build experience. Memory. If a plugin allocates more memory than the JVM has available — for example, loading a very large model file or accumulating state across calls — it can trigger an OutOfMemoryError. The VADER UDF we build here loads a compact lexicon and holds no state, so this is not a concern in practice. For more complex plugins that allocate significant heap memory, Neo4j provides a preview ProcedureMemory API where we can register allocations against the configured transaction memory limits, which prevents uncapped growth from causing database restarts. Uncaught exceptions. An unhandled RuntimeException in a UDF propagates up through the Neo4j query execution engine. Good error handling in the UDF code keeps this from becoming a problem. Infinite loops and thread starvation. A UDF that hangs — waiting on a network call, deadlocked or stuck in a loop — ties up a JVM thread from Neo4j's shared pool. The VADER UDF makes no network calls, holds no state and performs a relatively small amount of computation per call, so this is not a concern here, but it matters for more complex plugins. Dependency conflicts. Because the plugin jar shares the classpath with the database engine, any library bundled into the fat jar must not conflict with libraries Neo4j already ships. This problem was encountered during development and more on that in the build section below, including a straightforward fix. Startup failures. A jar that fails to load prevents the system from starting. The solution is always to test in a development environment first, such as Neo4j Desktop or a local Docker container, before deploying anywhere more critical. Security. A Java plugin has full access to the JVM, filesystem and network. This is the same trust model as Neo4j's own plugins and is appropriate for code we've written and reviewed. For third-party plugins from untrusted sources, the same caution applies as for any third-party code running inside a critical process. AuraDB. AuraDB supports plugins provided and certified by Neo4j, such as APOC, GDS and GenAI, but not arbitrary third-party or custom jars. The Java UDF approach in this article requires self-managed Neo4j, such as Desktop, Docker or a server install. If AuraDB is the target, the Java UDF approach described here is not available; the GenAI plugin or an external service are the alternatives. None of this should discourage us from building a Java UDF. The VADER UDF we build here is small, does one thing, makes no network calls, holds no state and uses a well-tested library. The sensible approach, which applies to any plugin development, is to build and test on a local development instance first, then deploy with confidence. In Neo4j, the steps to deploy our UDF are: Build a fat jarStop the serverCopy the jar file to the server's plugins directoryAdd an allowlist entry to neo4j.confRestart the server The deployment model differs from the Wasm approach — more on that in the build and deploy section below. Setting Up the Project Prerequisites We'll need the following before starting: Java 21 – check with java -version. Java 21 is the version used by the official Neo4j plugin template and by this articleMaven 3.8+ – check with mvn -versionNeo4j 2026.06.0 – the version used for this article, running in one of the ways described below Choosing a Neo4j Install For this experiment, we'll use either Neo4j Desktop or Docker. Neo4j also supports server installs on Linux and Windows — the plugin mechanism is the same — but we did not test that path and don't provide instructions for it here. Neo4j Desktop is the easiest starting point. Download it from Neo4j for Desktop, create a new project and start a local database server. Find the exact path to the plugins directory by clicking Open folder > plugins. Docker is convenient for a clean, throwaway environment. The command below starts Neo4j 2026.06.0 with a plugins volume mounted to a local directory, which is where we'll drop the jar: Shell mkdir -p ~/neo4j/plugins ~/neo4j/data docker run \ --name neo4j-sentiment \ -p 7474:7474 -p 7687:7687 \ -v ~/neo4j/plugins:/plugins \ -v ~/neo4j/data:/data \ -e NEO4J_AUTH=neo4j/password \ -e NEO4J_dbms_security_procedures_allowlist="sentiment.*" \ neo4j:2026.06.0 With Docker we pass the allowlist as an environment variable rather than editing neo4j.conf directly. The jar goes into ~/neo4j/plugins/ on the host. Creating the Project Structure Create a new Maven project directory: Shell mkdir neo4j-sentiment-udf cd neo4j-sentiment-udf The full directory tree should look like this when finished: Plain Text neo4j-sentiment-udf/ ├── pom.xml └── src/ ├── main/ │ └── java/ │ └── sentiment/ │ └── Sentimentable.java └── test/ └── java/ └── sentiment/ └── SentimentableTest.java The sections below cover each part in turn. Next, we'll create both source directories: Shell mkdir -p src/main/java/sentiment mkdir -p src/test/java/sentiment Maven Dependencies We'll create a pom.xml file in the project root. The structure follows the official Neo4j procedure template at Neo4j Procedure Template, with three adjustments specific to this project that are explained below. XML <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.neo4j.example</groupId> <artifactId>sentimentable</artifactId> <version>1.0.0-SNAPSHOT</version> <packaging>jar</packaging> <name>Neo4j Sentiment UDF</name> <description>VADER sentiment analysis as a Neo4j user-defined function</description> <properties> <java.version>21</java.version> <maven.compiler.release>${java.version}</maven.compiler.release> <neo4j.version>2026.06.0</neo4j.version> </properties> <!-- ADJUSTMENT 1: JitPack required for VaderSentimentJava --> <repositories> <repository> <id>jitpack.io</id> <url>https://jitpack.io</url> </repository> </repositories> <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <!-- ADJUSTMENT 2: VaderSentimentJava runtime dependency --> <dependency> <groupId>com.github.apanimesh061</groupId> <artifactId>VaderSentimentJava</artifactId> <version>v1.1.1</version> </dependency> <!-- Test dependencies — let neo4j-harness manage JUnit version --> <dependency> <groupId>org.neo4j.test</groupId> <artifactId>neo4j-harness</artifactId> <version>${neo4j.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> <version>6.0.2</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.5.4</version> </plugin> <plugin> <artifactId>maven-shade-plugin</artifactId> <version>3.5.1</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <!-- ADJUSTMENT 3: relocate commons-lang3 to avoid version conflict with Neo4j's internal copy --> <relocations> <relocation> <pattern>org.apache.commons.lang3</pattern> <shadedPattern>sentiment.shaded.org.apache.commons.lang3</shadedPattern> </relocation> </relocations> <artifactSet> <excludes> <exclude>org.neo4j:*</exclude> </excludes> </artifactSet> <shadedArtifactAttached>false</shadedArtifactAttached> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> The three adjustments from the official template are called out inline as comments. Everything else — groupId convention, provided scope for the Neo4j dependency, the shade plugin structure and the test dependency pattern — follows the official guidance. Writing the UDF We'll create the file src/main/java/sentiment/Sentimentable.java and paste in the following: Java package sentiment; import com.vader.sentiment.analyzer.SentimentAnalyzer; import com.vader.sentiment.analyzer.SentimentPolarities; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.UserFunction; import java.util.Map; public class Sentimentable { @UserFunction("sentiment.score") @Description("Score a string with VADER. Returns compound, positive, negative, neutral.") public Map<String, Double> score(@Name("text") String text) { if (text == null || text.isBlank()) { return Map.of("compound", 0.0, "positive", 0.0, "negative", 0.0, "neutral", 1.0); } final SentimentPolarities polarities = SentimentAnalyzer.getScoresFor(text); return Map.of( "compound", (double) polarities.getCompoundPolarity(), "positive", (double) polarities.getPositivePolarity(), "negative", (double) polarities.getNegativePolarity(), "neutral", (double) polarities.getNeutralPolarity() ); } } The following implementation details are worth highlighting. The v1.1.1 API uses a static method — SentimentAnalyzer.getScoresFor(text) — rather than a mutable instance. This means there is no shared state between calls, which is what we want in a Neo4j UDF where multiple Cypher queries may invoke the function concurrently. The VADER lexicon is loaded internally by the library on first call and cached for subsequent calls. The @UserFunction("sentiment.score") annotation registers the method as callable from Cypher under that name. The @Name annotation on the parameter provides the argument name for Neo4j's function metadata and documentation — UDFs are always called with positional arguments in Cypher, as shown throughout this article: sentiment.score(row.headline). The return type is Map<String, Double>. In Cypher, this surfaces as a map literal, so callers can destructure it with dot notation: sc.compound, sc.positive and so on. In the SingleStore version, the TVF returned a row set and was used in a FROM clause. Here the UDF is called inline in a WITH or RETURN clause instead. Writing the Tests Following the official Neo4j procedure template pattern, we'll use neo4j-harness to spin up a lightweight embedded Neo4j instance in JUnit, register our UDF with it and run Cypher queries against it — all without deploying to a running database. This is the recommended testing approach in Neo4j's own documentation. We'll create the file src/test/java/sentiment/SentimentableTest.java and paste in the following: Java package sentiment; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestInstance; import org.neo4j.driver.Driver; import org.neo4j.driver.GraphDatabase; import org.neo4j.driver.Session; import org.neo4j.harness.Neo4j; import org.neo4j.harness.Neo4jBuilders; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertTrue; @TestInstance(TestInstance.Lifecycle.PER_CLASS) public class SentimentableTest { private Neo4j embeddedDatabaseServer; private Driver driver; @BeforeAll void initializeNeo4j() { this.embeddedDatabaseServer = Neo4jBuilders.newInProcessBuilder() .withDisabledServer() .withFunction(Sentimentable.class) .build(); this.driver = GraphDatabase.driver(embeddedDatabaseServer.boltURI()); } @AfterAll void closeNeo4j() { this.driver.close(); this.embeddedDatabaseServer.close(); } @Test void scorePositiveSentence() { try (Session session = driver.session()) { var scores = session.run( "RETURN sentiment.score('The movie was great') AS scores" ).single().get("scores").asMap(); assertTrue((Double) scores.get("compound") > 0.5); assertTrue((Double) scores.get("positive") > 0.0); assertEquals(0.0, (Double) scores.get("negative")); } } @Test void capitalizationIncreasesScore() { try (Session session = driver.session()) { var normal = session.run( "RETURN sentiment.score('The movie was great') AS scores" ).single().get("scores").asMap(); var caps = session.run( "RETURN sentiment.score('The movie was GREAT!') AS scores" ).single().get("scores").asMap(); assertTrue((Double) caps.get("compound") > (Double) normal.get("compound")); } } @Test void emptyStringReturnsNeutral() { try (Session session = driver.session()) { var scores = session.run( "RETURN sentiment.score('') AS scores" ).single().get("scores").asMap(); assertEquals(0.0, (Double) scores.get("compound")); assertEquals(1.0, (Double) scores.get("neutral")); } } @Test void nullStringReturnsNeutral() { try (Session session = driver.session()) { var scores = session.run( "RETURN sentiment.score(null) AS scores" ).single().get("scores").asMap(); assertEquals(0.0, (Double) scores.get("compound")); assertEquals(1.0, (Double) scores.get("neutral")); } } } The four tests mirror the tests we'll run manually in Neo4j Browser, but now they run automatically as part of the build. Neo4jBuilders.newInProcessBuilder() starts a lightweight embedded instance with the Sentimentable function registered; .withDisabledServer() skips the HTTP server since we only need the Bolt connection. The structure follows the official JoinTest.java pattern. Building and Deploying Step 1: Install the Maven Wrapper and build The official Neo4j procedure template uses the Maven Wrapper (mvnw), which means we only need Java installed, not a separate Maven installation. To add the wrapper to the project: Shell mvn wrapper:wrapper Then build and run the tests: Shell ./mvnw clean package Or to skip the tests during development: Shell ./mvnw clean package -DskipTests To use a globally installed Maven directly, mvn clean package -DskipTests works equally well — the wrapper is a convenience, not a requirement. Maven compiles the Java source, runs the Shade plugin and writes two jar files to target/. The one we want is sentimentable-1.0.0-SNAPSHOT.jar — the fat jar with VADER bundled inside. The original-sentimentable-1.0.0-SNAPSHOT.jar is the plain jar without dependencies, so we'll ignore it. If the build fails with a package org.neo4j.procedure does not exist error, check that the pom.xml has <scope>provided</scope> on the Neo4j dependency and that the version matches the running Neo4j instance. Step 2: Copy the Jar to the Plugins Directory Neo4j Desktop: Stop the serverOpen folder > plugins and copy sentimentable-1.0.0-SNAPSHOT.jar into that folderOpen folder > conf > neo4j.conf, find dbms.security.procedures.allowlist= and uncomment the line if it is commented outAdd sentiment.* to the end of the line Docker: Copy to the host directory mounted as /plugins: Shell cp target/sentimentable-1.0.0-SNAPSHOT.jar ~/neo4j/plugins/ Step 3: Whitelist the Function Namespace Neo4j's default dbms.security.procedures.allowlist is *, which loads all plugins. If an allowlist is configured with specific entries, any custom namespace must be included or the function will silently be unavailable — no error on startup, it simply won't exist. It's good practice to configure an explicit allowlist following the principle of least privilege. Our UDF uses only the public Neo4j procedure API, which means it doesn't require the separate dbms.security.procedures.unrestricted setting — that's only needed for extensions that access internal APIs. Step 4: Restart Neo4j Neo4j Desktop: Restart the server using the button in the Desktop UI. If Desktop shows "stopped" immediately after starting, open http://localhost:7474 directly — the server may be running before the UI reflects it. Docker: If this is the initial launch, no restart is needed — the docker run command in the Choosing a Neo4j Install section already starts Neo4j with the jar in place from the mounted plugins directory. If updating the jar after the container is already running, stop the container, replace the jar in ~/neo4j/plugins/ and then restart: Shell docker stop neo4j-sentiment cp target/sentimentable-1.0.0-SNAPSHOT.jar ~/neo4j/plugins/ docker start neo4j-sentiment The clearest confirmation that the plugin loaded correctly is to run the verification queries in step 5 below — if sentiment.score() is visible and returns results, the jar was picked up successfully. Verifying the Function We can interact with Neo4j by entering http://localhost:7474 in the browser. Step 5: Confirm the Function Loaded First, we'll check that Neo4j can see the function at all: Cypher SHOW FUNCTIONS YIELD name WHERE name STARTS WITH 'sentiment' RETURN name; Expected output: Plain Text +-----------------+ | name | +-----------------+ | sentiment.score | +-----------------+ If this returns zero rows, the jar is either not in the plugins directory, the allowlist entry is missing or misspelled or Neo4j was not fully restarted. Step 6: Run the Tests Run the following tests: Cypher RETURN sentiment.score('The movie was great') AS scores; Expected output: JSON { neutral: 0.4230000078678131, negative: 0.0, positive: 0.5770000219345093, compound: 0.6248999834060669 } Now we'll test that VADER's capitalization awareness is working: Cypher RETURN sentiment.score('The movie was GREAT!') AS scores; Expected output: JSON { neutral: 0.36899998784065247, negative: 0.0, positive: 0.6309999823570251, compound: 0.7289999723434448 } The compound score rises with the capitalized GREAT!, exactly as in the Wasm version. For the examples we tested, the Java port produces scores consistent with the Rust crate used in the book chapter. Now, we'll test the null guard. Passing an empty string should return a neutral result rather than an exception: Cypher RETURN sentiment.score('') AS scores; Expected output: JSON { neutral: 1.0, negative: 0.0, positive: 0.0, compound: 0.0 } If all three return the expected values, the UDF is working and we're ready to build the graph schema and load data. Designing the Graph Schema The graph model for this pipeline has three node labels, as shown in Figure 2. A central Stock node connects to Tick nodes via HAS_TICK relationships and to Headline nodes via HAS_HEADLINE relationships. VADER polarity scores are stored directly on each Headline node at ingestion time, making them available to any Cypher query without recomputing. Figure 2. Graph data model Plain Text (:Stock {symbol}) -[:HAS_TICK]-> (:Tick {symbol, ts, open, high, low, close, volume}) -[:HAS_HEADLINE]->(:Headline {id, symbol, ts, headline, url, publisher, compound, positive, negative, neutral}) The Stock node acts as the join key. In SingleStore the queries join tick and stock_sentiment on (symbol, DATE(ts)); in Neo4j that same co-reference is expressed by traversing from a shared Stock node to both Tick and Headline nodes with a date predicate. The relationship replaces the foreign key. Let's now run these commands to create constraints and indexes: Cypher CREATE CONSTRAINT tick_pk IF NOT EXISTS FOR (t:Tick) REQUIRE (t.symbol, t.ts) IS NODE KEY; CREATE CONSTRAINT headline_id IF NOT EXISTS FOR (h:Headline) REQUIRE h.id IS UNIQUE; CREATE CONSTRAINT stock_id IF NOT EXISTS FOR (s:Stock) REQUIRE s.symbol IS UNIQUE; CREATE INDEX tick_symbol_ts IF NOT EXISTS FOR (t:Tick) ON (t.symbol, t.ts); CREATE INDEX headline_symbol_ts IF NOT EXISTS FOR (h:Headline) ON (h.symbol, h.ts); Loading Data and Scoring Headlines Getting the Datasets The datasets, notebook and SQL files for the original SingleStore book chapter are all publicly available in the book's GitHub repository. The two CSV files we need are in the datasets subdirectory: fictitious_stocks.csv – synthetic daily OHLCV stock prices (random-walk model, fictitious symbols)raw_fictitious_headlines.csv – programmatically generated news headlines (templates + ticker symbols + financial events) We'll download both files into our local working directory. Dataset Format fictitious_stocks.csv has seven columns. The date and Name columns are renamed to ts and symbol, respectively, to match the graph schema: Plain Text date,open,high,low,close,volume,Name 2013-01-02,743.98,756.93,736.15,745.68,9142645,BBRQ-FX 2013-01-03,764.41,779.16,757.72,765.16,1208771,BBRQ-FX ... raw_fictitious_headlines.csv has five columns that map directly to the Headline node properties: Plain Text headline,url,publisher,ts,symbol BBRQ-FX stock record revenues after analyst update,http://www.hill.net/,The Stock Chronicle,2014-10-22,BBRQ-FX ... No preprocessing is needed beyond what the loader already does, such as dropping nulls, filtering the one extreme volume outlier and sorting by date. The Python Loader The data_loader.py below reads the two CSV files and writes them into Neo4j via the Python driver. Install the dependencies first if not already done so: Shell pip install -r requirements.txt Then run the loader, substituting the actual paths to the downloaded CSV files. Also replace your_password_here with your actual password. Python # data_loader.py import pandas as pd from neo4j import GraphDatabase from tqdm import tqdm URI = "bolt://localhost:7687" AUTH = ("neo4j", "your_password_here") TICK_CSV = "fictitious_stocks.csv" RAW_CSV = "raw_fictitious_headlines.csv" driver = GraphDatabase.driver(URI, auth=AUTH) def chunks(df, size): for i in range(0, len(df), size): yield df.iloc[i:i+size].to_dict("records") # load tick data tick_df = (pd.read_csv(TICK_CSV) .dropna() .query("volume <= 2_147_483_647") .rename(columns={"date": "ts", "Name": "symbol"}) .sort_values(["ts", "symbol"])) tick_batches = list(chunks(tick_df, 1000)) print(f"Loading {len(tick_df):,} tick rows in {len(tick_batches)} batches...") with driver.session() as session: for batch in tqdm(tick_batches, desc="Ticks", unit="batch"): session.run(""" UNWIND $rows AS row MERGE (s:Stock {symbol: row.symbol}) CREATE (t:Tick {symbol: row.symbol, ts: date(row.ts), open: row.open, high: row.high, low: row.low, close: row.close, volume: toInteger(row.volume)}) CREATE (s)-[:HAS_TICK]->(t) """, rows=batch) # load headlines and score at ingestion time raw_df = pd.read_csv(RAW_CSV) raw_batches = list(chunks(raw_df, 1000)) print(f"Loading {len(raw_df):,} headline rows in {len(raw_batches)} batches...") with driver.session() as session: for batch in tqdm(raw_batches, desc="Headlines", unit="batch"): session.run(""" UNWIND $rows AS row MATCH (s:Stock {symbol: row.symbol}) WITH s, row, sentiment.score(row.headline) AS sc CREATE (h:Headline { id: randomUUID(), symbol: row.symbol, ts: datetime(row.ts), headline: row.headline, url: row.url, publisher: row.publisher, compound: sc.compound, positive: sc.positive, negative: sc.negative, neutral: sc.neutral }) CREATE (s)-[:HAS_HEADLINE]->(h) """, rows=batch) print("Done.") driver.close() Run the Python program: Shell python data_loader.py The key line is sentiment.score(row.headline) AS sc inside the Cypher. This is doing what the sentimentable(i.headline) TVF call does in the SingleStore INSERT ... SELECT — computing scores at the database level in the same operation that writes the record, with no round-trip to the application layer. One important note if we need to re-run the loader is that the script uses CREATE for Tick and Headline nodes, so running it a second time without clearing the database will create duplicates rather than overwriting. Clear the database first with the following Cypher, using the Query tab: Cypher MATCH (n) CALL { WITH n DETACH DELETE n } IN TRANSACTIONS OF 100 ROWS; The batch size of 100 is deliberate — larger values can exceed the default transaction memory limit and fail. After clearing, re-run the schema constraints and indexes before running the loader again. Alternative Loading Directly From GitHub With LOAD CSV To stay entirely within Cypher and avoid Python, Neo4j's LOAD CSV command can fetch the files directly from GitHub over HTTPS. No file copying, no import directory, no Python dependencies. Run both queries using the Query tab in order — ticks first, then headlines, since the headlines query does a MATCH on Stock nodes created by the tick query. Cypher LOAD CSV WITH HEADERS FROM 'https://...' AS row CALL { WITH row MERGE (s:Stock {symbol: row.Name}) CREATE (t:Tick { symbol: row.Name, ts: date(row.date), open: toFloat(row.open), high: toFloat(row.high), low: toFloat(row.low), close: toFloat(row.close), volume: toInteger(row.volume) }) CREATE (s)-[:HAS_TICK]->(t) } IN TRANSACTIONS OF 1000 ROWS; LOAD CSV WITH HEADERS FROM 'https://...' AS row CALL { WITH row MATCH (s:Stock {symbol: row.symbol}) WITH s, row, sentiment.score(row.headline) AS sc CREATE (h:Headline { id: randomUUID(), symbol: row.symbol, ts: datetime(row.ts), headline: row.headline, url: row.url, publisher: row.publisher, compound: sc.compound, positive: sc.positive, negative: sc.negative, neutral: sc.neutral }) CREATE (s)-[:HAS_HEADLINE]->(h) } IN TRANSACTIONS OF 1000 ROWS; LOAD CSV WITH HEADERS reads the first row as column names, so the original names (row.Name, row.date) are mapped directly to the graph property names inline — the same column renaming the Python loader does with rename(). The IN TRANSACTIONS OF 1000 ROWS batching is required for the tick file at ~600,000 rows to avoid the transaction memory limit. The same delete-before-reload rule applies here: re-running either query without clearing the database first will create duplicates. The only requirement is that Neo4j has outbound HTTPS access to reach GitHub, which is the case for Desktop and local Docker. In a network-restricted server environment the Python loader with local files is the safer fallback. Next, some example queries to test using the Query tab. Headline-Level Sentiment Cypher MATCH (h:Headline) RETURN h.symbol AS symbol, date(h.ts) AS ts, left(h.headline, 30) AS headline, round(h.positive, 3) AS positive, round(h.negative, 3) AS negative, round(h.neutral, 3) AS neutral ORDER BY h.symbol, h.ts LIMIT 10; Aggregate Sentiment by Stock and Day Cypher MATCH (h:Headline) WITH h.symbol AS symbol, date(h.ts) AS ts, avg(h.positive) AS avg_positive, avg(h.negative) AS avg_negative, avg(h.neutral) AS avg_neutral, count(h) AS num_headlines RETURN symbol, ts, round(avg_positive, 3) AS avg_positive, round(avg_negative, 3) AS avg_negative, round(avg_neutral, 3) AS avg_neutral, num_headlines ORDER BY symbol, ts LIMIT 10; Join Sentiment With Closing Price In Cypher, the shared Stock node makes the symbol join implicit and we only need a date predicate. Cypher MATCH (t:Tick)<-[:HAS_TICK]-(s:Stock)-[:HAS_HEADLINE]->(h:Headline) WHERE date(t.ts) = date(h.ts) RETURN t.symbol AS symbol, date(t.ts) AS ts, round(t.close, 2) AS close, round(h.positive, 3) AS positive, round(h.negative, 3) AS negative, round(h.neutral, 3) AS neutral ORDER BY t.symbol, t.ts LIMIT 10; Most Positive Headlines Cypher MATCH (h:Headline) RETURN h.symbol, date(h.ts) AS ts, left(h.headline, 30) AS headline, round(h.positive, 3) AS positive ORDER BY h.positive DESC LIMIT 10; Most Negative Headlines Cypher MATCH (h:Headline) RETURN h.symbol, date(h.ts) AS ts, left(h.headline, 30) AS headline, round(h.negative, 3) AS negative ORDER BY h.negative DESC LIMIT 10; In the SingleStore book, CEO scandal headlines dominated the negative ranking across multiple stocks. We see the same pattern here because the underlying VADER lexicon is identical. Validate Stored Scores Against Live UDF Calls This mirrors the consistency check from the SingleStore book, where stored stock_sentiment values were compared against a fresh JOIN LATERAL sentimentable(...) call to confirm the ingestion pipeline was deterministic. Cypher MATCH (h:Headline {symbol: 'BBRQ-FX'}) WITH h, sentiment.score(h.headline) AS live RETURN h.symbol AS symbol, date(h.ts) AS ts, left(h.headline, 30) AS headline, CASE WHEN round(h.positive, 3) = round(live.positive, 3) AND round(h.negative, 3) = round(live.negative, 3) AND round(h.neutral, 3) = round(live.neutral, 3) THEN 'match' ELSE 'not match' END AS comparison LIMIT 10; Daily Average Sentiment vs. Closing Price The CTE-style aggregation from the book translates naturally to Cypher's WITH chaining. Cypher MATCH (h:Headline) WITH h.symbol AS symbol, date(h.ts) AS ts, avg(h.positive) AS avg_positive, avg(h.negative) AS avg_negative, avg(h.neutral) AS avg_neutral MATCH (t:Tick {symbol: symbol}) WHERE date(t.ts) = ts RETURN symbol, ts, round(t.close, 2) AS daily_close, round(avg_positive, 3) AS avg_positive, round(avg_negative, 3) AS avg_negative, round(avg_neutral, 3) AS avg_neutral ORDER BY symbol, ts LIMIT 10; What We Learned The experiment was a clear success. VADER runs inside Neo4j, scores headlines at ingestion time via a simple Cypher call and all the analytical queries from the SingleStore book have direct equivalents in Cypher. For the examples we tested, the Java port produces scores consistent with the Rust crate used in the SingleStore book — although independent language ports may differ in edge cases due to differences in tokenization or floating-point handling. The graph model handles the stock-tick-plus-headlines domain naturally and in several respects the Cypher queries are more expressive than their SQL counterparts — the relationship traversal from a shared Stock node replaces a keyed SQL join in a way that reflects the actual structure of the domain rather than just being an implementation detail. The graph model is a genuine advantage for the join queries. Replacing JOIN tick ON (symbol, DATE(ts)) with a graph traversal through a shared Stock node is not just syntactic preference — it reflects the actual structure of the domain. A stock symbol connects ticks and headlines naturally as a graph entity and Cypher expresses that more directly than a keyed SQL join. In-database scoring works. Calling sentiment.score(row.headline) inside the Cypher CREATE statement means scoring and ingestion happen in the same operation, with no round-trip to an application layer. This is the same goal the SingleStore Wasm pipeline achieves and the Java UDF delivers it cleanly. The dependency conflict is a one-time fix. We hit the commons-lang3 version conflict during development and it stopped the server from starting. The fix — relocating the bundled classes to a private namespace using the Maven Shade plugin — is straightforward once we know what to look for and the solution is baked into the pom.xml in this article. There are also honest differences from the SingleStore Wasm approach. Deployment requires a restart. SingleStore uses a tool that loads a function into a live database with no downtime. Neo4j requires a jar build, a file copy, a config edit and a restart. For an initial Docker launch, the jar is picked up automatically — but any subsequent update to the jar requires a container restart. The Maven Wrapper and the clear deployment steps in this article make the process repeatable. No execution sandbox. SingleStore runs each Wasm function instance in its own isolated process with a hard memory boundary. The Neo4j UDF runs in the same JVM as the server. For a small, well-behaved plugin like the VADER UDF this makes no practical difference, but it's a meaningful architectural distinction for more complex or heavyweight plugins. Language is JVM-based. The Wasm approach accepts any language that compiles to the Wasm core spec. Neo4j's extensibility model is JVM-only. For teams that want to bring existing Python or Rust models into the database, that is worth knowing about upfront. Alternative Approaches The Java UDF is the focus of this article, but it's not the only way to bring sentiment scoring close to Neo4j data. We considered several alternatives during the experiment. Some are compelling for specific use cases and others less so. Knowing the options helps us choose the right tool for our situation. Pre-scoring outside the database. Score all headlines before loading. Add the polarity scores as columns in the CSV and load everything with LOAD CSV. Nothing custom runs inside Neo4j at all. For a batch pipeline like this one, where data are loaded once and queried many times, this is entirely practical and requires no Java knowledge. The only thing we give up is the ability to call sentiment.score() inline in Cypher at query time. For many teams this will be the right answer and it's the simplest path to a working pipeline. External microservice. Deploy a small Python or Rust service that runs VADER and exposes an HTTP endpoint. An external microservice can expose VADER through an HTTP API, with the application layer calling the service before or during ingestion. This gives us complete process isolation — a crash in the sentiment service cannot touch the database — and works with AuraDB. The tradeoff is network latency on every call and the operational overhead of running a separate service. For lower-volume or interactive use cases it's a clean, flexible pattern. Neo4j GenAI plugin. Neo4j's GenAI plugin supports calling embedding and LLM APIs — OpenAI, Azure OpenAI and compatible endpoints — directly from Cypher. It's fully managed by Neo4j, works on AuraDB and requires no Java. To use a cloud LLM for sentiment classification rather than VADER’s lexicon is a well-supported, low-friction path. The tradeoff is API cost and the opacity of a large language model compared to VADER's fully transparent, inspectable lexicon — which matters in regulated domains where we need to explain a score. GraalVM native compilation. GraalVM can ahead-of-time compile Java UDFs to native binaries, reducing JVM startup overhead and memory footprint. This is a performance optimization rather than an architectural change — the code still runs inside the Neo4j process — and adds significant build complexity for modest gain in this use case. It is worth knowing about for larger, more heavyweight plugins, but not the right choice here. Wasm runtime embedded inside a Java UDF. Theoretically, we could embed a Wasm runtime such as wasmtime inside a Java UDF and execute the VADER Wasm module from within Neo4j, getting Wasm's sandbox guarantees inside Neo4j's plugin model. It's technically feasible but no published working example appears to exist and the complexity cost is high relative to the alternatives. An interesting idea to watch, but not practical today. The table below shows how these approaches compare on the dimensions that matter most. ApproachCompute locationAuraDBLanguage choiceOperational complexityPre-score outside DBCompleteYesAnyLowExternal microserviceCompleteYes (via APOC)AnyMediumAPOC NLP (cloud API)Remote serviceNo (APOC Extended required)N/ALowGenAI pluginRemote serviceYesN/ALowJava UDF (this article)Shared JVMNoJVM-basedMediumWasm-in-Java (theoretical)Wasm sandboxNoAny (via Wasm)Very high The Java UDF sits in the middle of this table — it's uniquely capable of calling sentiment.score() inline from any Cypher query without application-layer involvement and it runs entirely within the system without external API calls or network latency. Whether that inline, self-contained capability is what our use case needs is the key question. For development, experimentation and pipelines where the data and team are well understood, it's a compelling and practical approach. For other situations, the alternatives above offer different but equally valid tradeoffs. A Second Path Is APOC NLP Procedures The two approaches differ in where the computation happens, as shown in Figure 3. With the Java UDF, the VADER lexicon is bundled in the jar and scoring runs inside the Neo4j JVM — no network call, no external dependency, no per-call cost. With APOC NLP, Neo4j orchestrates calls to an external cloud API and receives scores back over the network. That single architectural difference drives most of the tradeoffs covered in this section. Figure 3. Java UDF vs. APOC NLP Neo4j already has sentiment analysis capability — it just works quite differently and it lives not in GDS but in APOC Extended, a separate component from APOC Core. APOC's NLP procedures act as wrappers around cloud-based Natural Language APIs. The supported providers are AWS Comprehend, Azure Cognitive Services and Google Cloud Natural Language. The calling pattern is straightforward. With AWS, for example: Cypher MATCH (h:Headline {symbol: 'BBRQ-FX'}) CALL apoc.nlp.aws.sentiment.stream(h, { key: $apiKey, secret: $apiSecret, nodeProperty: 'headline' }) YIELD value RETURN h.headline, value.sentiment, value.sentimentScore; And with Azure: Cypher MATCH (h:Headline {symbol: 'BBRQ-FX'}) CALL apoc.nlp.azure.sentiment.stream(h, { key: $apiKey, url: $apiUrl, nodeProperty: 'headline' }) YIELD value RETURN h.headline, value.sentiment, value.sentimentScore; The graph variant goes one step further and writes the sentiment result back as a node property automatically, with write: true in the config map. Choosing Between the Two Java VADER UDFAPOC NLP (AWS / Azure / GCP)Where scoring runsInside Neo4j JVMExternal cloud APINetwork call per batchNoYesCost per callNo API chargeAPI pricing appliesModel qualityLexicon-based (VADER)Cloud NLP / ML modelsAuraDB compatibleNoNo (APOC Extended not available in AuraDB)Java knowledge neededYesNoOffline / air-gappedYesNoDeterministic resultsYesProvider-dependentDomain tuningLimited (lexicon)Better (ML models handle context) The Java UDF is the stronger choice when scoring volume is high, API costs matter, the text is short social-media-style content that VADER was designed for, or an offline/air-gapped environment is required. The VADER lexicon is fully transparent — we can inspect why a string received a given score, which matters in regulated domains. APOC NLP is the stronger choice when Java knowledge is limited, the text requires linguistic nuance beyond VADER’s lexicon (negation, sarcasm, domain-specific vocabulary), or cloud NLP APIs are already in use for other workloads. One important constraint applies to both: APOC NLP is part of APOC Extended, not APOC Core. AuraDB includes APOC Core by default, but APOC Extended is not available in AuraDB — so neither the Java UDF nor APOC NLP works there. The GenAI plugin or an external microservice are the practical AuraDB paths. GDS, Neo4j's Graph Data Science library, does not include text-level sentiment analysis — it's graph-algorithm-oriented. Text scoring in Neo4j is either in-database via a Java UDF or delegated to a cloud NLP service via APOC. Summary The experiment confirms that Neo4j's Java extensibility model is a capable platform for in-database compute. The VADER UDF works, the graph model is a natural fit for the stock-tick-plus-headlines domain and the analytical queries translate cleanly from SQL to Cypher — in some cases more expressively, because the relationship between prices and headlines is explicit in the graph schema rather than inferred at query time through a join predicate. The more interesting engineering question is when to use a Java UDF versus the alternatives. The answer depends primarily on four factors: Deployment model (self-managed Neo4j only for UDFs)Latency and network requirements (the UDF has none; APOC NLP and external microservices introduce both)Model sophistication (VADER's lexicon is transparent and fast but limited; cloud NLP APIs offer better linguistic coverage)Operational constraints (Java knowledge, plugin management and the restart-on-update requirement all have a cost) There is no universally correct choice — the table in the APOC NLP section lays out the tradeoffs and reasonable teams will land in different places depending on their priorities. What the article does establish is that the approach works and is officially supported. Building a plugin is documented and templated. For development, experimentation and well-understood production pipelines, it's a practical and interesting path. To go further, the official Neo4j Procedure Template is an excellent starting point, neo4j-harness makes unit testing UDFs straightforward without needing a running database instance and the full Neo4j Java Reference covers procedures, aggregation functions and the complete extensibility API in depth. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Software engineers often view soft skills as secondary, considering them relevant mainly for managers, recruiters, or those frequently in meetings, rather than essential for technical roles. However, as your career advances, this perspective becomes harder to maintain. Greater impact requires you to explain ideas, influence decisions, manage disagreements, build trust, exchange feedback, and communicate with those outside your technical context. While leadership roles highlight this need, these skills are integral to effective software engineering well before any formal leadership title. Practicing soft skills alone is challenging. While you can develop technical abilities like Java, databases, or system design independently, communication and influence require real interaction. Open source provides this environment, offering opportunities for discussions, code reviews, proposals, community meetings, documentation, conferences, and collaboration across companies, cultures, and experience levels. This article explores how open source can serve as a practical training ground for the communication and interpersonal skills essential for technical leaders. How Open Source Builds Soft Skills Through Real Collaboration This article will not revisit the importance of soft skills for software engineers, as that topic has been addressed elsewhere. Instead, it focuses on practical ways to develop these skills. While hard skills can be practiced independently, soft skills such as communication, influence, trust, empathy, and collaboration require interaction with others. Open source offers a natural and consistent environment for this development. Collaborate Across Cultures and Perspectives A key benefit of open source is the opportunity to collaborate with individuals from diverse cultures, organizations, backgrounds, and perspectives. Contributors may disagree due to differing technical opinions, communication styles, risk priorities, or problem-solving contexts. Learning to work productively in this environment builds a key leadership skill: transforming diverse perspectives into better decisions rather than unnecessary friction. This experience is especially valuable for technical leaders, as broader responsibilities mean working with people who think, communicate, and operate differently. Learn to Present and Defend Your Ideas Open source also requires you to communicate your ideas clearly. A proposal is rarely accepted on technical merit alone. You must explain the problem, provide context, outline trade-offs, answer questions, address criticism, and often revise your proposal before gaining community approval. This is remarkably similar to proposing an architectural or design decision inside an organization. Translating your ideas into concepts others can understand, discuss, and support is a critical skill in technical leadership. Navigate Politics and Build Agreements Wherever people collaborate, interests, priorities, relationships, and politics will be present. This is not inherently negative. Politics often arises because people value different aspects, such as backward compatibility, developer experience, performance, or long-term maintainability. Technical leaders must understand others’ perspectives before advancing decisions. This is especially important in software architecture, where there is rarely a single correct answer. Many decisions depend on context; what works well in one situation may be unsuitable in another. When no clear answer exists, technical knowledge alone is not enough to resolve the discussion. You must negotiate trade-offs, understand others’ priorities, reach agreements, build consensus, and sometimes accept solutions that differ from your initial preference. Open-source communities regularly present these situations. Advancing technical initiatives often requires understanding both the architecture and the people involved. Navigating these dynamics without escalating disagreements into conflicts is essential for technical leadership. Improve Your Spoken Communication Technical leadership extends beyond written proposals. Community calls, working groups, meetups, podcasts, workshops, and conferences provide opportunities to communicate technical ideas verbally. If you need to advocate for an architectural decision within your company, you will use many of the same skills practiced in open source: structuring arguments, explaining complexity, adapting to your audience, answering challenging questions, and remaining constructive under scrutiny. Speaking is integral to engineering leadership. It helps transform technical ideas into organizational decisions. Learn to Coordinate Software Delivery Open source can also develop strong project and delivery skills. Mature projects require releases, versioning strategies, roadmap planning, issue prioritization, dependency coordination, estimation, and clear communication about deliverables and timelines. Participating in these activities teaches that software delivery is as much about coordination as technical execution. As projects grow, understanding dependencies, priorities, expectations, and others' contributions becomes increasingly important. These concerns are common for staff engineers, architects, and technical leaders as their responsibilities expand beyond a single team. Learn Leadership Without Formal Authority One of the most notable aspects of open source is that many contributors are volunteers. You cannot rely on hierarchy, salary, or reporting structures to motivate contributions. You need to build trust. You need to make people feel that their contribution matters. You need empathy when someone cannot complete a task. You must communicate expectations respectfully, recognizing others’ autonomy over their time. You also need to foster an environment that encourages people to return. That makes open source a particularly interesting leadership laboratory. If you can help create momentum among people who are free to walk away at any moment, you are practicing a form of leadership based on influence rather than authority. Build Trust and Empathy Trust is a key currency in open-source communities. People gradually assess whether you review fairly, listen before disagreeing, acknowledge contributions, keep commitments, and argue in good faith. This reputation is built through consistent interactions. The same principle applies inside an organization. Engineers are much more likely to follow somebody they trust than somebody who merely has a more senior title. Open source offers repeated opportunities to learn how trust is earned, maintained, and sometimes lost. Practice Soft Skills Through Real Situations This is what makes open source especially valuable for developing soft skills. You are not practicing communication through hypothetical exercises; you are communicating because an actual proposal requires approval. You are not simply reading about conflict resolution; you are addressing real disagreements. You are not studying influence theoretically; you are persuading people who have no obligation to agree with you. This distinction is important. You can practice hard skills alone. Soft skills require interaction, and open source provides real people, real challenges, and ongoing opportunities to develop leadership. Conclusion Soft skills are not an optional layer added on top of technical expertise; they are part of what allows that expertise to create impact. Open source gives software engineers a real environment to practice communication, persuasion, negotiation, collaboration, trust, empathy, public speaking, and even the politics that naturally emerge when people with different priorities need to make decisions together. This is especially important in areas such as software architecture, where many choices live in a gray area, and the final direction depends not only on technical knowledge, but also on context, trade-offs, and the ability to build agreement. For engineers who want to grow into staff engineer, principal engineer, software architect, or technology leadership roles, this kind of practice is invaluable. You can study communication theory, but eventually you need people to communicate with; you can read about influence, but eventually you need a real disagreement to navigate. Open source creates those opportunities repeatedly and at scale. It helps transform soft skills from abstract concepts into practical leadership capabilities that can make your technical knowledge more understandable, more trusted, and ultimately more influential.

By Otavio Santana CORE

TL; DR: The AI Delegation Lifecycle Your team ships AI outputs that nobody fully trusts; you needed to be quick, and “dirty” tagged along. Too bad that ungoverned automation becomes AI debt when a stakeholder asks who owns it. But do not despair: The AI Delegation Lifecycle turns skills you already use into six decisions you can apply this week to govern that work and prove it audit-ready and suited for agent harnesses. Popular Traps When Creating AI Output All teams can show you what their AI produces: status reports sent without anyone touching them, release notes drafted in seconds, a customer-facing FAQ that updates itself. Far fewer teams can answer the question posed by a prospective customer or by compliance: how do you govern your own internal AI use? Often, in the frenzy past to make of AI, nobody decided. Outputs without decisions are expensive, as nobody: Decided that the status report should run unattendedWrote down what a good output looks like,Analyzed the effects of a recent model change, orChecked last month whether it still produces one. The work grew that way, one helpful shortcut at a time, until it became a system nobody could explain, and nobody owned. As we know, complex systems always start as complicated systems that, at least to some people, still seem understandable. That approach of avoiding the creation of that gap, or “evolution”, is to address one decision at a time systematically. I call the practice the Delegation Lifecycle, and you already have most of the skills it requires. How Outputs Pile Up Without a Single Decision Behind Them The mechanism is ordinary yet still a shortcut: a Scrum Master pastes Retrospective data into ChatGPT to save 20 minutes. Next Sprint, the team does it again, as it worked the first time. By the third month, the summary goes straight into the team wiki, and nobody reads the raw notes anymore. The team charter does not cover it, but it is convenient; “accepted” as an amendment to the working agreement by not opposing it. The shortcut became infrastructure while everyone was busy delivering. Now multiply that by every person on the team and every task an AI can touch. You get what I call AI debt: a pile of useful, undocumented, unowned automation that works right up until the moment someone asks who is responsible for it. The “we ship it now and fix it later” habit that got you through the funding round, the launch, the reorg, and the last crisis in general, becomes the liability that shows up the moment you least expect it. The problem is not that the team uses AI, but that it uses AI without deciding which decisions belong to it. Agile practitioners are good at making decisions. We make working agreements, we set acceptance criteria, we run Retrospectives, and we refine backlogs. The Delegation Lifecycle takes those habits and points them at the work you have started handing to a model. What Counts as Delegation in the Delegation Lifecycle First, a boundary, because not every AI interaction needs governing. By delegated work, I mean recurring work in which an AI output is incorporated into a team artifact, stakeholder communication, operational workflow, or customer-facing surface. Asking a model for five ideas before you write the update yourself is not delegation. That is assistance, and it needs prompt discipline, a skill, but not a governance record. The model drafting the update from your tracker every Friday is a form of delegation. And so is sending it without a human in the loop. The lifecycle applies when AI use becomes recurring, consequential, externally visible, or embedded in how the work runs. Not every prompt needs a record, but every delegated workflow does. One Decision, Six Stages Take a single piece of work your team has delegated to AI. Not the whole AI strategy. One task: the status report, the test generation, and the first-draft release notes. That one decision has a complete path, from “should AI touch this at all” to “what do our records prove when someone asks?” The Delegation Lifecycle places a decision at each of the six points along that path. Decide Question it answers: Should AI touch this work, and how autonomously?Agile skill you already have: Decision-making with explicit categories. Route and Boundaries Question it answers: Which model, tool, environment, data boundary, and sufficiency bar are appropriate?Agile skill you already have: Acceptance criteria and process design. Hand Over Question it answers: How does the work transfer, and who owns it?Agile skill you already have: Working agreements. Define Done Question it answers: What must be true before the output leaves the team, and how is it evaluated?Agile skill you already have: Definition of Done. Inspect Question it answers: Is the delegation still safe and useful?Agile skill you already have: Retrospective facilitation. Roll Up Question it answers: What evidence can we show the people who ask?Agile skill you already have: Stakeholder communication. Before the first stage, one rule sets up the rest. The A3 Framework is the entry gate: Assist when AI supports human judgment, and you own the outcome,Automate when AI executes a bounded task under human-owned responsibility, andAvoid when the work is too consequential, ambiguous, or sensitive to hand over. If you followed my work here on the blog or took the earlier version of my AI4Agile course, you already use it. A3 decides whether work enters the lifecycle at all; the six stages sketched above govern what happens once it does. Stage 1, Decide: Should AI do this work, and at what level of autonomy? The skill underneath is decision-making with explicit categories. Where teams get stuck: Assist work quietly becomes Automate work when the review habit disappears. You start by checking every output, then most of them, then none, and nobody decided that on purpose. Stage 2, Route and Boundaries: Which model, tool, and environment run this, what data may enter, and what counts as good enough? Not every task needs your most expensive model, and not every task can run on your cheapest. Model routing means designing the process and acceptance criteria for model, tool, and data choices. The skill is the one you use every time you define done for a Product Backlog item. Where teams get stuck: they default to the priciest model for everything, never set a sufficiency bar, and then cannot explain the monthly bill. A cost nobody can account for — better: a low return on invested tokens — is a Stage 2 failure. Stage 3, Hand Over: How does the work transfer, and who owns it? Task split, owner, inputs, outputs, validation, stop rules, and a record: this is a working agreement, written for a collaborator who happens to be a model. The skill is the same one you use to set team norms. Where teams get stuck: the handoff lives in one person’s head, and no human owner is named. When that person changes teams or leaves, the system leaves with them. Without stop rules, nothing halts the work when the output starts drifting. Stage 4, Define Done: What must an AI-assisted output meet before it leaves the team? Stage 3 is how the work transfers; Stage 4 is the release gate it has to pass: the verification level, provenance disclosure, data hygiene, and the sufficiency tier from Stage 2. This is your Definition of Done, extended to work that a model touched. Where teams get stuck: “looks good” becomes the only “standard.” Approval gets mistaken for review. Someone clicks send on a procurement email the model wrote after skimming it, and now the team’s name is on a claim nobody verified. Approval is not review, and once an external audience is involved, that gap is what throws your team under the proverbial bus. Stage 5, Inspect: Is the delegation still working, or has it drifted? This is a Retrospective focused on delegated work rather than the team: Has the output quality slipped?Has Assist crept into Automate? Admittedly, applying “evals” sounds fancier, but the skill is Retrospective facilitation, which you run every Sprint. Inspection does not mean reviewing every output forever. It means agreeing on a sampling rate, the drift signals worth watching, and the trigger that brings the work back under tighter human review. Where teams get stuck is a set-and-forget mentality: Nobody scheduled the inspection, so the drift compounds unseen until it becomes an incident. Stage 6, Roll Up: What does all of this prove to the people who ask? Leadership, enterprise procurement, and increasingly, regulators want evidence of controlled AI adoption. The skill is stakeholder communication. This stage needs no separate governance artifact. The records from Stages 1 through 5 should already aggregate into what those people ask for: a delegation inventory, an autonomy distribution across Assist and Automate, and an inspection trail. Where teams get stuck is in governance theater. They build a separate leadership deck full of confident claims, disconnected from what the team actually does, and a single sharp question from a CFO collapses it. The Stages of the Delegation Lifecycle Are a Loop, Not a Checklist The six stages are a teaching order, not a strict sequence. In practice, your team will agree on the Definition of Done while filling out the A3 handoff canvas, and a finding from an inspection will send a task straight back to Stage 1 for re-classification. That is the system working, not failing. The stages also depend on each other. A Stage 1 decision that never gets inspected becomes the most dangerous kind of automation: confident, unattended, and unowned. A Definition of Done with no handoff behind it has no teeth, because nobody agreed on who applies the standard or when. Just count how many of these six stages your team has a real decision behind right now; it will make a good starting point for a team discussion on how you are using AI at the moment. Two points on the path are deliberately set to have no artifact. Before Stage 1 sits, know which work your team does, at what frequency, and at what stakes, which is a forensic analysis of your own workflow. Around all six sits your AI working agreement, the team norm layer. Neither needs a new canvas. The AI Delegation Lifecycle adds a document only where a recurring decision genuinely had no home. Why This Is Not Optional Anymore The approach the Delegation Lifecycle proposes is not just about internal hygiene. AI use is moving from personal productivity into organizational accountability. Since February 2, 2025, Article 4 of the EU AI Act has required providers and deployers to ensure a sufficient level of AI literacy among staff and others operating AI systems on their behalf. (Which, interestingly, seems to be largely ignored by many players.) Enforcement through national market surveillance authorities will take effect on August 3, 2026. NIST organizes AI risk management around the four steps: govern, map, measure, and manage. Anthropic’s first Economic Index found that real-world Claude usage already splits between augmentation and automation: 57% augmentation, 43% automation. The practical question underneath it all is simpler: can you show the decisions behind the work you delegated? The Roll-Up Is the Quiet Payoff Most teams miss the byproduct of every stage, producing a valuable record as part of normal use: The A3 decisions become a portfolio of what you deliberately automated, assisted, and kept human.The routing records become AI spend by task and tier, with a reason attached.The Definition of Done sign-offs and inspection logs serve as an audit trail of controlled, inspected adoption. Nobody fills in an extra report: operational work generates governance evidence as it runs. So when a prospect asks, “How do you govern your own internal AI use?” the team running this lifecycle does not shrug. It answers with its records. That is the difference between a team that merely uses AI and a team that can be trusted with it, and that trust is becoming a line item in enterprise procurement. Your Turn Pick one task your team has handed to an AI. Just one. Walk it through the six questions out loud in your next Retrospective: did we decide this, who owns it, what does done mean, when did we last check it, and what would we show someone who asked. I guess that you will find at least one stage where the honest answer is “nobody decided that.” That is your starting point: adopt one stage at a time, wherever your pain is sharpest. Knowledge that walked out with a departing colleague points to Stages 3 and 4. A token bill nobody can explain to the CFO points to Stage 2. An output that embarrassed you in front of a stakeholder points to Stages 4 and 5. Conclusion I turned this lifecycle into a working method teams can apply immediately: how to decide what to delegate, hand it over safely, inspect for drift, and produce evidence without building a governance theater. Which of the six stages does your team actually have a decision for? I am curious, and I suspect the answer is fewer than you would like.

By Stefan Wolpers CORE

Goose — the open-source, Rust-based AI developer agent from Block (donated to the Linux Foundation’s Agentic AI Foundation) — interacts natively with your local development environment via the Model Context Protocol (MCP). In this tutorial, you will learn how to build stateless, cloud-native Java microservices using Quarkus LangChain4j and expose them as governed MCP extensions that Goose can discover and run seamlessly. Autonomous AI coding agents like Goose go far beyond simple code autocompletion. Built in Rust for speed and portability, Goose runs on your local machine, inspects files, runs terminal commands, and uses tools over MCP to automate complex engineering tasks. However, when developers want an AI agent to query enterprise microservices, trigger database migrations, or fetch internal API metrics, writing custom local scripts or ad-hoc wrappers is brittle and dangerous. The solution is to build a stateless MCP Tool Server in Java using Quarkus LangChain4j. Quarkus provides near-zero startup time and low memory footprint, while LangChain4j makes exposing @Tool methods via standard MCP HTTP/JSON-RPC trivial. Architecture: How Goose Integrates With Quarkus MCP Markdown ┌────────────────────────────────────────────────────────┐ │ Goose AI Agent (Rust Runtime) │ │ (Local CLI / Desktop App / ACP Server) │ └───────────────────────────┬────────────────────────────┘ │ Model Context Protocol (MCP) │ JSON-RPC over Stateless HTTP ▼ ┌────────────────────────────────────────────────────────┐ │ Quarkus LangChain4j MCP Server │ │ - @Tool Annotations & Bean Validation │ │ - Reactive SmallRye Mutiny Execution │ │ - GraalVM Native Image Ready │ └───────────────────────────┬────────────────────────────┘ │ Reactive Clients ▼ Enterprise APIs / Databases / Dev UI Goose Agent (Client): Executes on the developer machine, orchestrating LLM tool loops via MCP.MCP HTTP Transport: Goose sends structured tool calls to the Quarkus backend as stateless HTTP POST requests using standardized MCP methods (tools/list, tools/call).Quarkus Microservice: Validates parameters with Jakarta Bean Validation, executes reactive business logic, and returns structured data to Goose. Step 1: Configuring Dependencies in Quarkus Create a new Quarkus project or update your pom.xml to include quarkus-langchain4j-mcp and Reactive: Note: Find the completed demo application here: https://github.com/danieloh30/governed-mcp-tools.git. XML <dependencyManagement> <dependencies> <dependency> <groupId>${quarkus.platform.group-id}</groupId> <artifactId>${quarkus.platform.artifact-id}</artifactId> <version>${quarkus.platform.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-arc</artifactId> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-rest-jackson</artifactId> </dependency> <dependency> <groupId>io.quarkiverse.mcp</groupId> <artifactId>quarkus-mcp-server-http</artifactId> <version>2.0.0.CR2</version> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-hibernate-validator</artifactId> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-junit</artifactId> <scope>test</scope> </dependency> </dependencies> Step 2: Implementing Hardened MCP Tools We will create a Customer Services MCP Tool that Goose can call when an engineer asks: "Goose, check the database status for customer CUST-4091 and fetch their recent telemetry." By placing @Tool annotations on CDI beans, Quarkus LangChain4j automatically registers the class as an MCP server endpoint: Embedded Javascript @ApplicationScoped public class CustomerServiceTools { @Tool(description = "Retrieve the current account status, service tier, and primary deployment region for a given customer.") public Uni<CustomerStatusResponse> getCustomerStatus( @ToolArg(description = "Customer ID formatted as CUST-XXXX") @NotNull @Pattern(regexp = "^CUST-[0-9]{4,8}$") String customerId) { CustomerStatusResponse response = switch (customerId) { case "CUST-4091" -> new CustomerStatusResponse("CUST-4091", "ACTIVE", "ENTERPRISE_TIER", "US-EAST-1"); case "CUST-2187" -> new CustomerStatusResponse("CUST-2187", "ACTIVE", "BUSINESS_TIER", "EU-WEST-1"); case "CUST-7734" -> new CustomerStatusResponse("CUST-7734", "SUSPENDED", "STARTER_TIER", "AP-SOUTH-1"); default -> new CustomerStatusResponse(customerId, "NOT_FOUND", "UNKNOWN", "UNKNOWN"); }; return Uni.createFrom().item(response); } @Tool(description = "Retrieve recent health-check logs and diagnostic metrics for a specified availability zone.") public Uni<List<String>> getZoneHealthLogs( @ToolArg(description = "Zone identifier, e.g., US-EAST-1") @Size(max = 20) String zoneId) { return Uni.createFrom().item(List.of( "[" + zoneId + "] CPU utilization: 42% (healthy)", "[" + zoneId + "] Memory pressure: 31% (normal)", "[" + zoneId + "] Network I/O: 1.2 Gbps ingress / 0.8 Gbps egress", "[" + zoneId + "] Disk IOPS: 12,400 read / 8,300 write (within SLA)", "[" + zoneId + "] Active connections: 18,230 (capacity: 50,000)", "[" + zoneId + "] Last incident: none in past 72 hours" )); } @Tool(description = "Track the current status, item count, and estimated delivery for an enterprise order.") public Uni<OrderStatusResponse> getOrderStatus( @ToolArg(description = "Order ID formatted as ORD-XXXXXXXX") @NotNull @Pattern(regexp = "^ORD-[0-9]{8}$") String orderId) { OrderStatusResponse response = switch (orderId) { case "ORD-20240815" -> new OrderStatusResponse("ORD-20240815", "SHIPPED", 12, "$48,750.00", "2024-08-22", "US-EAST-1"); case "ORD-20240901" -> new OrderStatusResponse("ORD-20240901", "PROCESSING", 5, "$12,300.00", "2024-09-10", "EU-WEST-1"); case "ORD-20241003" -> new OrderStatusResponse("ORD-20241003", "DELIVERED", 28, "$134,500.00", "2024-10-08", "AP-SOUTH-1"); default -> new OrderStatusResponse(orderId, "NOT_FOUND", 0, "$0.00", "N/A", "UNKNOWN"); }; return Uni.createFrom().item(response); } @Tool(description = "Retrieve SLA compliance metrics including uptime, latency, and violation count for a service.") public Uni<SLAComplianceResponse> getSLACompliance( @ToolArg(description = "Service identifier, e.g., api-gateway, auth-service") @NotNull @Size(max = 40) String serviceId) { SLAComplianceResponse response = switch (serviceId) { case "api-gateway" -> new SLAComplianceResponse("api-gateway", 99.97, "45ms", 99.99, 0, "2024-Q3"); case "auth-service" -> new SLAComplianceResponse("auth-service", 99.82, "120ms", 99.95, 3, "2024-Q3"); case "data-pipeline" -> new SLAComplianceResponse("data-pipeline", 98.50, "340ms", 99.80, 12, "2024-Q3"); case "notification-hub" -> new SLAComplianceResponse("notification-hub", 99.91, "78ms", 99.97, 1, "2024-Q3"); default -> new SLAComplianceResponse(serviceId, 0.0, "N/A", 0.0, -1, "N/A"); }; return Uni.createFrom().item(response); } ... } Step 3: Enabling the MCP Extension in application.properties Configure your Quarkus MCP server settings: Properties files quarkus.mcp-server.server-info.name=customer-tools quarkus.mcp-server.server-info.version=1.0.0 quarkus.mcp-server.http.root-path=/mcp quarkus.log.category."io.quarkiverse.mcp".level=DEBUG Launch Quarkus in dev mode: Shell ./mvnw quarkus:dev Step 4: Connecting Goose to Your Quarkus MCP Server Goose can be extended with any MCP server over stdio or HTTP. Configure Goose by editing its YAML configuration file or using the Goose CLI. Option A: Using the Goose CLI Register the Quarkus MCP server directly in your terminal: Shell goose extension add customer-tools \ --type http \ --uri http://localhost:8080/mcp Option B: Editing ~/.config/goose/config.yaml Add the Quarkus backend to your Goose extensions configuration: YAML extensions: customer-tools: enabled: true type: http uri: http://localhost:8080/mcp headers: Content-Type: "application/json" Step 5: Testing the Developer Workflow Launch Goose via CLI or the Desktop App: Shell goose session Prompt Goose: Developer: "I'm debugging customer CUST-4091. Use customer-tools to fetch their account tier, and then check the health logs for their primary region." Frontend UI: Developer: Choose one of the Tool explorers. Select the “Run tool” button on the right panel. Verify the audit events. What Happens Under the Hood Discovery: Goose sends an HTTP POST /mcp JSON-RPC tools/list request. Quarkus responds with JSON schema definitions derived from getCustomerStatus and getZoneHealthLogs.Tool Invocation 1: Goose parses the prompt, formats a tools/call JSON payload with {"customerId": "CUST-4091"}, and posts it to Quarkus.Execution and validation: Quarkus executes Hibernate Bean Validation. Since CUST-4091 matches ^CUST-[0-9]{4,8}$, it runs getCustomerStatus and returns primaryRegion: US-EAST-1.Tool Invocation 2: Goose sees US-EAST-1, triggers getZoneHealthLogs("US-EAST-1"), receives the green health metrics, and summarizes the complete diagnostic report back to you in the CLI. Summary and Next Steps By wrapping Java business logic in Quarkus LangChain4j @Tool beans, you give local AI developer agents like Goose secure, validated access to enterprise backend systems. However, when hundreds of developers run local Goose agents against shared backend microservices in production, connecting them directly creates security and governance risks. Coming up in Part 2: We will introduce agentgateway — the Linux Foundation data plane proxy —to sit between Goose and Quarkus. We will configure OAuth2/OIDC authentication, fine-grained tool-level RBAC, and rate limiting to harden our enterprise AI infrastructure.

By Daniel Oh CORE

A step-by-step guide to grounding a LangGraph agent in Microsoft Foundry IQ agentic retrieval — without rebuilding your RAG pipeline. Why This Integration Is Worth Doing If you build agents on LangGraph and your enterprise content lives in Azure, you have probably written the same code twice: a chunker, an embedding job, a vector store, a retriever, a reranker, and a permissions filter bolted on at the end. Every new agent gets its own copy. Every copy drifts. Foundry IQ moves that work behind a single endpoint. A knowledge base wraps one or more knowledge sources, and the agentic retrieval engine handles query planning, parallel execution, semantic reranking, and (optionally) answer synthesis. Crucially for anyone outside the Microsoft agent stack: every knowledge base is also a standalone MCP server exposing one tool, knowledge_base_retrieve. Any MCP-compatible client can call it — including LangGraph, via langchain-mcp-adapters. That is the whole integration. The interesting parts are the four places it does not behave like a normal retriever, which this tutorial covers in detail: The MCP tool result has a different shape from the REST/SDK retrieve response.Bearer tokens expire, and a static headers dict will fail an hour into a long-running graph.Per-user permission filtering requires a second token, distinct from your service credential.The knowledge base is itself a planner, so you have two planners per turn and need to decide who does what. By the end, you will have a working LangGraph agent grounded in a Foundry IQ knowledge base, with citations preserved in graph state and a token provider that survives long sessions. What you should already know: LangGraph basics (StateGraph, ToolNode, the ReAct loop), and enough Azure to create a resource and assign a role. Architecture Figure 1 — The knowledge base owns retrieval. LangGraph owns orchestration. MCP is the contract between them. Three things are worth noticing before you write any code. The knowledge base is reusable. It is not scoped to one agent. The same knowledge base can ground a LangGraph agent, a Foundry Agent Service agent, and a Copilot integration simultaneously. That is the point of the abstraction, and it changes how you name things — name knowledge bases after topics (hr-policy-kb, product-docs-kb), not after the agent that happens to consume them first. Sources come in two flavors. Indexed sources (Blob, OneLake, an existing search index) are ingested, chunked, and vectorized into an index on your search service. Federated sources (remote SharePoint, web, MCP servers) are queried live at retrieval time and never ingested. This distinction matters for permissions, which we return to in Step 7. Identity flows in two channels. Your application authenticates to the search service with a service identity. Optionally, you also pass the end user's identity in a separate header so the engine filters documents that user may not see. Conflating these two is the most common source of "why is everyone seeing everything" bugs. Prerequisites An Azure AI Search service with agentic retrieval available in your region. Basic tier or higher if you want managed identity support.A Microsoft Foundry project and resource, with an LLM deployment (e.g. gpt-5-mini) and an embedding model (e.g. text-embedding-3-large).The Search Index Data Reader role assigned to the identity that will query the knowledge base.If your knowledge base specifies an LLM, the search service needs a managed identity with Cognitive Services User on the Foundry resource.Python 3.10+. Install the client libraries: Shell # Preview SDK — required for answer synthesis, configurable reasoning effort, # document-level permissions and multi-turn retrieve. pip install --pre azure-search-documents # Stable SDK is enough if you only need GA features on 2026-04-01: # pip install azure-search-documents pip install azure-identity langchain-mcp-adapters langgraph langchain-openai httpx A Word on API Versions Before You Start This is the decision that will bite you later if you get it wrong, so make it deliberately now. Agentic retrieval is generally available in the 2026-04-01 REST API. The 2026-05-01-preview adds answer synthesis, configurable reasoning effort, the messages input, document-level permissions, and sensitivity-label metadata. Both the Azure portal and the Microsoft Foundry portal expose preview-only behavior regardless of what your code uses, which means the portal is not a reliable preview of what your production code will do. The API version also changes MCP behavior directly. With 2026-05-01-preview, the knowledge base can return synthesized answers when it is configured with an LLM and a compatible reasoning effort. With 2026-04-01, MCP retrieval is always minimal and extractive, and the connection returns grounding data only. Pick 2026-04-01 if you are shipping to production now and can live with extractive grounding. Pick 2026-05-01-preview if you need synthesis or permission filtering, and accept there is no SLA. Step 1: Create the Knowledge Source and Knowledge Base A knowledge source points at your content. A knowledge base wraps one or more sources with retrieval configuration. Create the source first. Python import os from azure.identity import DefaultAzureCredential from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( KnowledgeBase, KnowledgeSourceReference, KnowledgeBaseAzureOpenAIModel, AzureOpenAIVectorizerParameters, SearchIndexKnowledgeSource, SearchIndexKnowledgeSourceParameters, ) SEARCH_ENDPOINT = os.environ["AZURE_SEARCH_ENDPOINT"] # https://<svc>.search.windows.net AOAI_ENDPOINT = os.environ["AZURE_OPENAI_ENDPOINT"] CHAT_DEPLOYMENT = "gpt-5-mini" credential = DefaultAzureCredential() index_client = SearchIndexClient(endpoint=SEARCH_ENDPOINT, credential=credential) # A knowledge source over an index you already have. knowledge_source = SearchIndexKnowledgeSource( name="product-docs-ks", description=( "Product documentation, release notes and API reference. " "Use for questions about product behaviour, configuration and limits." ), search_index_parameters=SearchIndexKnowledgeSourceParameters( search_index_name="product-docs-index", source_data_select="id,title,content,url,updated_at", ), ) index_client.create_or_update_knowledge_source(knowledge_source=knowledge_source) The description is not decoration. The retrieval engine uses it when deciding which sources to query for a given subquery, so write it the way you would explain the source to a new colleague: what is in it, and what kinds of questions it answers. Now the knowledge base: Python knowledge_base = KnowledgeBase( name="product-docs-kb", description="Grounding for product support questions.", knowledge_sources=[ KnowledgeSourceReference(name="product-docs-ks"), ], models=[ KnowledgeBaseAzureOpenAIModel( azure_open_ai_parameters=AzureOpenAIVectorizerParameters( resource_url=AOAI_ENDPOINT, deployment_name=CHAT_DEPLOYMENT, model_name=CHAT_DEPLOYMENT, ) ) ], retrieval_instructions=( "Prefer the most recently updated documents when versions conflict. " "For questions about limits or quotas, always consult product-docs-ks." ), ) index_client.create_or_update_knowledge_base(knowledge_base=knowledge_base) print("knowledge base ready") retrieval_instructions steers the planner's source selection. It is the highest-leverage knob in the whole configuration, and the one most people leave empty. Index requirements. If you point at an existing index, it needs a semantic configuration — agentic retrieval uses L2 semantic ranking. If the index has vector fields, it also needs a valid vectorizer so the engine can vectorize subqueries; otherwise vector fields are silently ignored. Step 2: Verify With the Retrieve API Before You Touch MCP Do not debug two systems at once. Confirm retrieval works over the SDK first, where you get the full response envelope including the query plan. Python from azure.search.documents.knowledgebases import KnowledgeBaseRetrievalClient from azure.search.documents.knowledgebases.models import ( KnowledgeBaseMessage, KnowledgeBaseMessageTextContent, KnowledgeBaseRetrievalRequest, ) kb_client = KnowledgeBaseRetrievalClient( endpoint=SEARCH_ENDPOINT, knowledge_base_name="product-docs-kb", credential=credential, ) request = KnowledgeBaseRetrievalRequest( messages=[ KnowledgeBaseMessage( role="user", content=[KnowledgeBaseMessageTextContent( text="What are the rate limits on the ingestion API, and did they change in the last release?" )], ) ], include_activity=True, # gives you the query plan ) result = kb_client.retrieve(request) print(result.response[0].content[0].text[:800]) # grounding data for entry in result.activity: # what the planner actually did print(entry.type, getattr(entry, "elapsed_ms", None)) The activity array is your observability surface. It reports the planner's token usage, the subqueries that were issued to each source, elapsed time per source, and reasoning-token consumption. Read it now, because — as we will see in Step 5 — you do not get it back over MCP. If the response is empty but activity shows matches were found, a document probably exceeded the output budget. Increase max_output_size, or chunk large source documents more aggressively. Step 3: Understand What the MCP Endpoint Gives You (and What It Does Not) Every knowledge base is automatically an MCP server. There is nothing to deploy. The endpoint is: Plain Text https://<your-search-service>.search.windows.net/knowledgebases/<your-knowledge-base>/mcp?api-version=<api-version> It exposes exactly one tool, knowledge_base_retrieve. Clients cannot see index management or source configuration through it — the surface is deliberately narrow. The trap is that the MCP tool result is not the retrieve action's response. It is a plain MCP tool result, which most clients surface under result.content[]: JSON { "result": { "content": [ { "type": "text", "text": "[{\"ref_id\":\"0\",\"title\":\"Ingestion limits\",\"terms\":\"rate limit, throttling\",\"content\":\"<chunk>\"}]" } ] } That text field is a JSON-encoded string, not a JSON object. You have to parse it twice. And the activity and references arrays you relied on in Step 2 are simply absent. Here is the comparison in full — this table is the thing to keep open while you build: AspectRetrieve action (REST / SDK)MCP endpoint (knowledge_base_retrieve)Payload locationresponse[0].content[0].textresult.content[0].textGrounding data formatJSON-encoded stringJSON-encoded string (same inner shape)activity array (query plan, tokens, per-source timings)Returned when includeActivity is setNot returnedreferences array (ref_id → docKey, activitySource)Returned, controllable per sourceNot returnedSensitivity label metadataPer-reference + response-level aggregateSame fields surfaced when configuredAnswer synthesisAvailable on 2026-05-01-previewAvailable on 2026-05-01-preview only; 2026-04-01 is always extractivePer-request tuning (filterAddOn, maxOutputDocuments, failOnError)Full control per knowledge sourceNot exposed — set defaults on the knowledge base insteadAuth mechanismSDK credential objectAuthorization: Bearer header, or api-key headerBest forDeterministic pipelines, evaluation harnesses, observabilityAgent frameworks, tool-calling loops, cross-runtime reuse The practical consequence: use MCP for the agent loop, and keep a direct retrieve client around for evaluation and debugging. They point at the same knowledge base, so there is no duplication of configuration — only of client code, and only where it earns its keep. Step 4: Authenticate, Properly Two options. Only one belongs in production. Admin key (api-key header) grants full read-write access to the search service. Use it for a five-minute spike, never beyond that. Bearer token (Authorization header) is the recommended path. The identity behind the token needs Search Index Data Reader on the search service, and the token must be scoped to https://search.azure.com/.default. The naive version looks like this, and it works — for about an hour: Python from azure.identity import DefaultAzureCredential, get_bearer_token_provider credential = DefaultAzureCredential() search_token_provider = get_bearer_token_provider( credential, "https://search.azure.com/.default" ) MCP_URL = ( f"{SEARCH_ENDPOINT}/knowledgebases/product-docs-kb/mcp" "?api-version=2026-05-01-preview" ) connection = { "foundry_iq": { "transport": "http", # streamable HTTP "url": MCP_URL, "headers": {"Authorization": f"Bearer {search_token_provider()}"}, } } Note the parentheses: search_token_provider() is evaluated once, at construction time, and frozen into a dict. A long-running graph, a checkpointed conversation resumed the next morning, or a service that builds its client at startup will all start returning 401s once that token expires. The Fix: An httpx.Auth That Refreshes langchain-mcp-adapters uses the official MCP SDK underneath, which accepts a custom authentication mechanism implementing the httpx.Auth interface. That is where token refresh belongs. Python import httpx from azure.identity import DefaultAzureCredential, get_bearer_token_provider class EntraBearerAuth(httpx.Auth): """Attaches a fresh Entra ID bearer token to every MCP request. azure-identity caches the token internally and only round-trips to the IdP when it is close to expiry, so calling the provider per request is cheap. """ def __init__(self, credential, scope: str = "https://search.azure.com/.default"): self._provider = get_bearer_token_provider(credential, scope) def auth_flow(self, request: httpx.Request): request.headers["Authorization"] = f"Bearer {self._provider()}" yield request auth = EntraBearerAuth(DefaultAzureCredential()) connection = { "foundry_iq": { "transport": "http", "url": MCP_URL, "auth": auth, # instead of a frozen headers dict } } This single change is the difference between a demo and something you can leave running. Step 5: Load the Tool Into LangGraph With authentication sorted, wiring the tool in is short. Python from langchain_mcp_adapters.client import MultiServerMCPClient client = MultiServerMCPClient(connection) tools = await client.get_tools() print([t.name for t in tools]) # ['knowledge_base_retrieve'] For a quick check, hand the tool straight to a prebuilt agent: Python from langgraph.prebuilt import create_react_agent # (LangChain v1 equivalent: from langchain.agents import create_agent) agent = create_react_agent("azure_openai:gpt-5-mini", tools) response = await agent.ainvoke( {"messages": [{"role": "user", "content": "What changed about ingestion rate limits in the last release?"}]} ) print(response["messages"][-1].content) If that returns a grounded answer, the integration works. Two operational notes before you build the real graph: MultiServerMCPClient is stateless by default — each tool invocation opens a fresh MCP session and tears it down. For a stateful server, you would use client.session(), but for Foundry IQ retrieval, stateless is correct and cheaper.If you register several MCP servers, be aware that a single failing server has historically been able to take down get_tools() for all of them. Register Foundry IQ in its own client if the rest of your tool estate is flaky. Step 6: Build the Graph, and Parse the Response The prebuilt agent hides the thing you most need to control: what happens to the grounding data on its way into state. Here is the explicit version. Figure 2 — Two planners run per turn. Step 2 decides whether to retrieve; step 5 decides how. Python import json from typing import Annotated, TypedDict from langchain_core.messages import ToolMessage from langchain_openai import AzureChatOpenAI from langgraph.graph import StateGraph, START, END from langgraph.graph.message import add_messages from langgraph.prebuilt import ToolNode class AgentState(TypedDict): messages: Annotated[list, add_messages] citations: list # accumulated across the conversation SYSTEM = ( "You answer questions about our product documentation. " "Always call knowledge_base_retrieve before answering a factual question. " "Cite sources using the ref_id values in the grounding data. " "If the grounding data does not contain the answer, say you do not know." ) llm = AzureChatOpenAI(azure_deployment="gpt-5-mini", api_version="2025-04-01-preview") llm_with_tools = llm.bind_tools(tools) async def agent_node(state: AgentState): messages = [{"role": "system", "content": SYSTEM}] + state["messages"] return {"messages": [await llm_with_tools.ainvoke(messages)]} def parse_grounding(state: AgentState): """Pull ref_id/title/url out of the last tool message into state. The MCP tool result is a JSON-encoded string inside a text content block, so it needs a second json.loads(). """ last = state["messages"][-1] if not isinstance(last, ToolMessage): return {} raw = last.content if isinstance(raw, list): # content-block form raw = next((b.get("text", "") for b in raw if isinstance(b, dict) and b.get("type") == "text"), "") try: docs = json.loads(raw) except (json.JSONDecodeError, TypeError): return {} # synthesized answer, not extractive found = [ { "ref_id": d.get("ref_id"), "title": d.get("title"), "url": d.get("url"), } for d in docs if isinstance(d, dict) ] return {"citations": state.get("citations", []) + found} def should_continue(state: AgentState): last = state["messages"][-1] return "tools" if getattr(last, "tool_calls", None) else END builder = StateGraph(AgentState) builder.add_node("agent", agent_node) builder.add_node("tools", ToolNode(tools)) builder.add_node("parse", parse_grounding) builder.add_edge(START, "agent") builder.add_conditional_edges("agent", should_continue, {"tools": "tools", END: END}) builder.add_edge("tools", "parse") builder.add_edge("parse", "agent") graph = builder.compile() The parse node is what keeps citations alive. Without it, the grounding JSON passes through the message history as an opaque blob, the model cites ref_id values, and your UI has nothing to resolve them against. Note the defensive try/except: if you switch the knowledge base to answer-synthesis output mode, the tool returns prose rather than a JSON array, and a parser that assumes JSON will crash on a configuration change made by someone else in the Azure portal. Step 7: Enforce Per-User Permissions Everything so far runs as a single service identity, which means every user sees every document the service can see. For most enterprise deployments, that is unacceptable. Permission enforcement has two halves. At ingestion time, indexed sources need ingestionPermissionOptions set so that ACLs, RBAC scopes, or Purview sensitivity labels are ingested alongside content. If you skip this, results come back unfiltered no matter what you send at query time — and the only fix is to recreate the knowledge source. Federated sources work differently: remote SharePoint queries through the Copilot Retrieval API using the user's own token and never ingests anything, and Fabric and Work IQ sources exchange the user's token for a scoped one. At query time, you pass the end user's access token — scoped to https://search.azure.com/.default, separate from your service credential, and requiring no search-service permissions of its own — in the x-ms-query-source-authorization header. Over the SDK, that is a named parameter: Python result = kb_client.retrieve( retrieval_request=request, x_ms_query_source_authorization=user_token, # the end user, not the service ) Over MCP, it is a per-request header, which means it varies per user while your client is long-lived. Extend the auth class rather than rebuilding the client: Python import contextvars current_user_token = contextvars.ContextVar("current_user_token", default=None) class EntraBearerAuthWithUser(EntraBearerAuth): def auth_flow(self, request: httpx.Request): request.headers["Authorization"] = f"Bearer {self._provider()}" user_token = current_user_token.get() if user_token: request.headers["x-ms-query-source-authorization"] = user_token yield request Set the context variable at the edge of your application — in the FastAPI dependency or middleware that already validates the caller — and every downstream MCP call in that request inherits it, including calls made deep inside a graph. Verify this end to end in your own environment. Microsoft documents the header explicitly for the retrieve action, and notes that MCP clients configure custom headers differently. Test with two users who have genuinely different document access and confirm the result sets differ — do not assume it works because it did not error. Step 8: Decide Who Plans This is the design question the tutorial format tends to bury, so it gets its own step. Look again at Figure 2. Your LangGraph agent node runs an LLM to decide whether to retrieve and how to phrase the query. Then the knowledge base runs another LLM to decompose that query into subqueries and choose sources. Two planners, two model calls, two chances to lose the user's intent. The failure mode is specific: the agent node paraphrases the user's question before handing it over, dropping a constraint ("in the 2026 release", "for part XYZ2B"), and the knowledge base then plans excellent subqueries for the wrong question. Microsoft's own evaluation work identifies exactly this — constraint preservation in the handoff from orchestrator to retriever — as the thing that correlates with retrieval quality. Three rules that follow from it: Pass the question through; do not summarize it. Instruct the agent node to forward the user's wording, including qualifiers, rather than composing a "better" search query. The knowledge base is better at query formulation than your agent node is; that is what you are paying it for.Tune reasoning effort, not prompts. minimal skips LLM planning entirely and runs keyword or hybrid search on the query as given — the right choice for lookups. low and medium add planning; medium adds iterative search, where the engine reviews its own results and issues follow-ups. Answer synthesis requires low or medium. Route cheap questions to a minimal knowledge base and hard ones to a medium one, and you have turned a latency/quality trade-off into a graph edge.Prefer extractive output inside an agent. Answer synthesis produces a finished natural-language answer, which is what you want when retrieval output goes straight to a user. Inside a LangGraph agent, the agent is going to reason over the content anyway — synthesizing first costs tokens and latency, and flattens the structure your parse node wants. Troubleshooting SymptomLikely causeWhat to do403 from Azure AI SearchIdentity lacks Search Index Data Reader on the search serviceAssign the role; confirm you are signed in to the right tenant and subscription401 after roughly an hourBearer token frozen into a static headers dictSwitch to the httpx.Auth provider from Step 4400 Bad RequestA knowledgeSourceName is not attached to the knowledge base, or its kind does not match; or one option requires another that is not enabledRead the top-level error — it names the offending property206 Partial ContentAt least one source failed, none of them marked requiredInspect the activity entries carrying an error; process partial results or mark the critical source failOnError502 Bad GatewayEvery selected source failed, or a source marked failOnError failedDo not assume an outage — read the underlying source failure firstEmpty response, but activity shows matchesMost relevant document exceeded the output budgetRaise maxOutputSize, or chunk large documents at ingestionEvery user sees every documentingestionPermissionOptions was not set when the knowledge source was createdRecreate the knowledge source with the right options; the header alone will not fix itAnswers ignore recent documentsScoring profiles are not applied by agentic retrievalUse freshness-aware retrieval rather than an index scoring profileTool list comes back emptyOne failing server in a multi-server clientGive Foundry IQ its own MultiServerMCPClient Where to Go Next The same knowledge base you just built is reachable from Microsoft Agent Framework, Foundry Agent Service, GitHub Copilot, Claude, and Cursor without any change to its configuration. That is the real payoff of putting retrieval behind MCP rather than inside your agent: when your team standardizes on a different runtime next year, the knowledge layer does not move. Two things worth building next: an evaluation harness that calls the retrieve API directly (so you get the activity array and can measure whether your retrieval_instructions are actually steering source selection), and a second knowledge base at minimal reasoning effort so you can route by question difficulty. References Query a knowledge base using the retrieve action or MCP endpoint — the authoritative reference for the MCP endpoint URL, authentication, response shapes, permission headers, and troubleshooting status codes.Agentic retrieval in Azure AI Search — overviewModel Context Protocol (MCP) — LangChain docs — transports, custom httpx.Auth, session lifecycle.langchain-mcp-adapters on GitHubMultiServerMCPClient API referenceFoundry IQ: build smarter agents faster with unified knowledge and serverless retrieval — Build 2026 announcement; GA scope and the MCP server.Foundry IQ: improve recall by up to 54% with knowledge bases — the constraint-preservation evaluation behind Step 8.

By Jubin Soni, FBCS CORE

The term software factory is getting a lot of attention right now, and for a good reason. AI coding assistants can generate code much faster than before. But faster coding alone does not mean faster, safer delivery. In many teams, it simply moves the bottleneck to review, testing, deployment, and operations. A software factory is a way to organize the entire software development life cycle as one connected, repeatable system. Think of a car manufacturing assembly line. Each station has a clear job, work moves forward in a predictable order, quality checks happen at the right moments, and the finished product is inspected before it leaves the factory. An agentic software factory applies that same idea to software delivery. AI agents do focused work across planning, coding, testing, deployment, monitoring, and feedback. Humans remain in charge of specifications, security, policies, approvals, and the decisions that should never be delegated blindly. Key Takeaways An agentic software factory coordinates AI agents across the complete software delivery lifecycle.Faster AI-assisted coding can create review bottlenecks unless downstream stages also improve.Humans retain ownership by defining guardrails, specifications, approvals, and security requirements.Workflow orchestration connects context, automation, observability, incident handling, and feedback loops. What Is a Software Factory? A software factory is not just a collection of developer tools. It is an operating model where software delivery is designed as a smooth, observable workflow from idea to production and back into improvement. In a car factory, a vehicle moves through assembly, painting, quality inspection, final assembly, and delivery. People are involved at important checkpoints, but the process does not restart from scratch at every station. It is structured, repeatable, and connected. The same model works for software. In an agentic software factory, the flow can look like this: Requirements and design: clarify what needs to be built and collect service context.Planning: turn requirements into a feasible implementation plan.Build: generate or modify code for a feature or bug fix.Testing and continuous integration: validate the change and stop failures early.Human review: approve, reject, or request changes before a risky next step.Deployment: deliver the approved change through continuous delivery.Monitoring and operations: check service health, create incidents, notify teams, and roll back when needed.Feedback loop: feed production signals and outcomes back into future planning. The important point is simple: agents perform work, while humans provide the gates. A software factory is not about removing people from the loop. It is about putting people at the moments where their judgment matters most. How Software Delivery Evolved Into the Software Factory Model The software factory did not appear suddenly. It is the next step in a long evolution toward more reliable software delivery. From Manual Deployment to Automated Delivery In the 1990s, developers often wrote the code, prepared servers, and deployed software manually. A release could take weeks or months. Testing and deployment were labor-intensive, and repeatability depended heavily on individual knowledge. Then continuous integration tools such as Hudson and Jenkins helped teams automate builds and tests. The rise of DevOps brought development and operations closer together, reducing the handoff gap between teams. Continuous delivery, continuous deployment, and infrastructure as code tools such as Terraform pushed automation further. GitOps and platforms such as Docker and Kubernetes added a strong operational model where Git could serve as the source of truth for application and infrastructure changes. Each stage made delivery more repeatable. From Automation to Agentic Engineering After AI agents and coding assistants became practical, teams began using them across more parts of the SDLC. An agent could help gather requirements, propose a plan, write code, prepare tests, review pull requests, monitor a service, or summarize feedback. That is where the software factory becomes agentic. Instead of treating AI as a single chat window or code-completion tool, I treat it as a coordinated group of specialized workers within a governed delivery system. The software factory model gives those agents a place, a sequence, boundaries, and clear outputs. Without that structure, adding more agents can create more confusion rather than more throughput. Why AI Coding Assistants Create New Bottlenecks Before coding assistants, the time required across planning, coding, review, and shipping was comparatively balanced. Writing code often took a large part of the cycle, but every stage had its own workload. Now coding can accelerate dramatically. Tools such as Cursor, GitHub Copilot, Claude Code, and Codex can help teams generate and change code faster. The problem is that the rest of the system does not automatically become faster. When code arrives faster, code review queues can become overloaded. Senior engineers get stuck reviewing a growing number of pull requests. Testing may become backed up. Deployment approvals may take longer. Only a small portion of the increased output may actually reach production. This is exactly why a software factory matters. It looks at the whole system, not only the coding stage. A good software factory improves the flow across the entire lifecycle so that one accelerated step does not jam everything downstream. Preventing Agentic Chaos There is another issue. Developers are already using many tools across the SDLC. Add multiple AI agents without a common operating layer, and it becomes difficult to answer basic questions: Which agent changed this service?What context did the agent use?Which policy or guardrail applied?Who approved the deployment?What should happen if the health check fails? That is agentic chaos: lots of autonomous activity, but little visibility, control, governance, or accountable decision-making. A software factory makes the agent workflow explicit. It creates a visible path for work, controls access to actions, and places checks before high-impact changes. Humans Still Own the System It is tempting to say that an agentic software factory can automate everything end-to-end. Technically, many tasks can be automated. In reality, giving unrestricted authority to agents is risky. A poorly constrained agent can make the wrong decision, trigger the wrong action, or cause damage in production. Humans and developers still own the system. In a well-designed software factory, my role is not to manually do every repeated task. My role is to define the rules of the factory. That includes: Defining product requirements and technical specifications.Setting security checklists and guardrails.Deciding which actions agents may take automatically.Creating human approval gates for important decisions.Reviewing plans, pull requests, release readiness, and incident responses.Maintaining accountability for production systems. This is the right division of responsibility. Agents can gather context, plan work, implement changes, run tests, check health, and notify teams. Developers decide what good looks like, which risks are acceptable, and whether a change should proceed. The Building Blocks of an Agentic Software Factory A practical software factory breaks broad lifecycle phases into smaller, focused responsibilities. Rather than relying on one giant agent to do everything, I can use agents for specific jobs and connect them through workflow orchestration. Plan The planning stage starts with human input and service context. A requirements agent can gather the feature request, identify the affected service, and collect relevant information. A planning agent can then turn that into an implementation plan. A feedback digest or product improvement agent can provide useful context from previous issues and outcomes. Build and Review The build stage can include a feature builder and bug-fixer agent. The review stage can include a pull request reviewer, automated CI checks, and other quality actions. The key is that a failed CI build blocks the workflow. It should not quietly move toward deployment. After CI succeeds, a human review gate can decide whether the change is ready to continue. This is where the software factory protects speed with judgment. Deploy and Operate After approval, a continuous delivery agent can deploy the service or feature. A monitoring agent can then assess the health of the service. If health is degraded or a critical issue appears, the workflow can create an incident, notify the relevant team through Slack, and, where appropriate, perform an automated rollback. The final piece is the feedback loop. Production data should not disappear into dashboards. It should update service context and help inform future planning. That loop is what turns a set of automation steps into an evolving software factory. Building a Software Factory Workflow With Port To put this into practice, I used Port as the context layer for an agentic SDLC. Port brings together workflow orchestration, agent management, service context, and governance so I can automate delivery without losing control. Inside the platform, I can create services, agents, dashboards, self-service actions, and workflows. The workflow is the backbone of the software factory because it makes the entire path visible and enforceable. Here is the workflow I built for a software factory agentic SDLC: Fetch service context: identify what the selected service is, its ownership, and relevant details.Gather requirements: use a requirements agent to understand the feature request.Create a plan: have a planning agent prepare the implementation approach.Build the change: use a coding agent to implement the requested work.Test and run CI: validate the change through testing and continuous integration.Block failures: stop the workflow immediately if CI fails.Request human review: let a developer approve or reject progression to deployment.Deploy through CD: release the approved change with a continuous delivery agent.Monitor health: inspect the health of the deployed service.Respond to degradation: create incidents, notify the right team, and roll back when required.Collect feedback: send outcomes back into the service catalog and planning context.Use a final deploy gate: keep a human decision point before final release or publishing. I can trigger this software factory through self-service by choosing a service and describing a feature, such as adding an API gateway to a fraud detection service. The workflow begins by retrieving context, then moves through requirements, planning, code generation, testing, CI, review, deployment, and monitoring. I can also trigger the workflow through Port AI. For example, I can request an agentic SDLC pipeline for a service and ask to add OpenTelemetry distributed tracing. The system can locate the service, find the appropriate software factory workflow, trigger it, and provide a live path to track the run. That does not mean the workflow is a black box. I can inspect its stages, check the run state, see whether it is currently planning, coding, or testing, and review the workflow configuration. The software factory becomes both automated and observable. Build Your Own Software Factory A software factory can orchestrate AI agents, workflows, service context, and human approval gates across the SDLC. Start Small, Then Expand the Factory You do not need to automate every part of delivery on day one. A software factory can begin with one valuable and repeatable path. For example, start with requirements, coding, CI, and a human review gate. Once that flow is stable, add deployment automation, monitoring, incident creation, rollback rules, and feedback loops. The goal is not automation for its own sake. The goal is a better system for delivering software: faster where tasks are repetitive, safer where risks are high, and clearer at every stage. A mature software factory gives every agent a defined responsibility, every workflow a visible path, and every human a meaningful control point. That is how I can take advantage of agentic engineering without turning the SDLC into chaos.

By Pavan Belagatti CORE

Headless CMS architecture solved a real development problem. It separated content from presentation, gave frontend teams control over frameworks and deployment, and made structured content available to websites, apps, and other channels through APIs. The friction often appears later, when content operations become more complex. Routine publishing changes can still depend on engineering, especially when editors need more control over layout, preview, or page composition. That gap is why some teams consider a different architectural pattern: hybrid headless CMS. It keeps the structured, API-based approach of headless while adding a visual authoring layer to assemble approved components. The Authoring Problem Behind Pure Headless In a pure headless setup, the CMS manages structured content while the frontend controls how that content is rendered. For developers, that separation is valuable. Teams can use React, Vue, Svelte, native applications, or another presentation layer without tying the frontend directly to the CMS. The tradeoff becomes more visible when presentation changes frequently. A CMS may contain a hero title, image, CTA, and product description, but the frontend still determines how those elements become a page. Supporting visual preview, flexible layouts, and reusable page composition can therefore require additional engineering around preview APIs, component mapping, draft rendering, routing, deployment, etc. None of this is inherently a weakness in headless architecture. It is implementation work that teams need to account for. For applications with stable layouts and highly structured content, the model can work extremely well. For enterprises running many sites, markets, and campaigns, the amount of presentation-related work can become an operational bottleneck. When Content Work Becomes Engineering Work The clearest signal is the backlog. Consider a marketing team launching ten regional campaign pages. The content already exists, and no new application behavior is required. But several regions need a different component order, one needs an additional promotional block, and another needs a temporary landing page. In a tightly controlled pure headless implementation, those requests may still require developers to modify templates or component configuration. The workflow can become: Content request → development ticket → code change → review → build → deployment → editor validation That process makes sense when the requested change affects application behavior. It becomes expensive when the request is simply to rearrange approved components. Preview creates a similar issue. Headless systems can support preview, but developers often have to connect draft content with the rendering application so editors can see the actual result before publication. The CMS provides structured data. The frontend provides the presentation context. The distinction matters because the application still owns rendering, routing, accessibility, performance, and browser behavior. MDN provides useful background on the separation between server-side systems and client-facing application behavior. What Hybrid Headless Changes Hybrid headless keeps the API-based content model but adds visual composition capabilities for editors. Instead of letting editors create arbitrary frontend code, developers define the available building blocks. A content team can then assemble approved components through the CMS while the frontend remains responsible for how those components render. For example, developers might provide: HeroProduct gridCustomer quotePricing blockCTAFAQ Editors can change the order or selection of those components without changing the underlying application. The key difference is where composition happens. capabilitypure headlesshybrid headless Structured content Yes Yes API delivery Yes Yes Framework freedom Yes Yes Page composition Usually implemented in frontend logic Can be exposed through CMS authoring tools Visual preview Possible, often requires integration Commonly integrated into the authoring workflow Editor-controlled layouts Depends on implementation Typically a core capability Component governance Application specific Central to the model Definitions vary between CMS vendors, so engineering teams should evaluate the architecture rather than the label. A platform described as hybrid should still expose a clean delivery API that applications can consume independently. If the frontend becomes dependent on proprietary page rendering behavior, teams may reintroduce some of the coupling they were trying to remove. Developers Still Own the Architecture Hybrid headless changes who handles routine page composition, but developers still control the technical boundaries. They define components, validation, accessibility, performance, and application behavior. They also own the delivery contract between the CMS and frontend, including the security implications of new integrations and features. For teams adopting AI-powered capabilities, resources with AI security explained in practical terms can help clarify some of those risks. Overall, that means the architecture still depends on disciplined component and API design. Components that are too rigid send editors back to development tickets. Too many overlapping components create governance problems. The goal is simple: editors control approved composition, while developers retain control over how the application works. When Pure Headless Is Still the Better Fit Pure headless remains a strong choice when presentation is primarily application logic. A product dashboard is a good example. Developers may control nearly every screen because layout, state, permissions, and application behavior are closely connected. Pure headless also fits well when content changes are mostly structured data changes rather than page composition. Typical signals include: A small number of highly custom applicationsStable page structuresLimited need for editor-controlled layoutsContent reused heavily across channelsStrong frontend engineering capacityPresentation decisions that should remain in code In these environments, adding visual composition may introduce complexity without solving a real problem. When Hybrid Headless Becomes More Practical Hybrid approaches become more attractive when content operations generate repeated frontend work. Common signals include: Many sites, markets, or brands using the same component libraryFrequent campaign pagesEditors who need reliable visual previewRegular requests to rearrange approved page componentsEngineering queues filled with presentation changes that contain little new logicTeams that need stronger separation between component development and page assembly A useful test is to pull the previous quarter's engineering backlog and count how many tickets were created primarily to move an existing content block, change a layout, build a campaign page from existing components, or make another presentation change that required no new application behavior. Then look at who filed those tickets. If the same content or marketing teams repeatedly depend on developers for short-lived campaign changes, the organization may need more authoring autonomy rather than more frontend capacity. The Tradeoffs Hybrid Headless Does Not Remove Visual composition shifts work rather than eliminating it. Component governance becomes more important because shared components now act as an interface between engineering and content teams. Someone needs to own versioning, accessibility, documentation, budgets, and backward compatibility. Preview also needs production-quality engineering. A visual editor is useful only when what the editor sees accurately reflects what users will receive. Teams also need to decide how much flexibility to expose. Unlimited layout freedom can create inconsistent pages and undermine a design system. Too little flexibility recreates the ticket backlog the architecture was meant to reduce. The goal is controlled composition. Developers create safe building blocks. Editors assemble them within defined constraints. Evaluate the Workflow, Not the Label The architecture decision should start with the actual publishing workflow. Map who creates content, who changes layouts, who builds components, how preview works, what triggers a deployment, and which requests currently require engineering involvement. Then examine the CMS boundary. Can content be consumed independently through APIs? Can developers control component behavior? Can editors perform routine composition without changing application code? Can teams preview changes accurately? Can the architecture support additional channels without rebuilding the content model? Pure headless and hybrid headless preserve the same core idea: separating content from presentation. The practical difference is how much controlled presentation capability the platform gives back to content teams. For developers, the goal is to keep engineering focused on work that actually requires engineering. If developers are building components, integrations, and application behavior, the architecture is doing useful work. If they are repeatedly moving existing blocks around landing pages, the boundary probably needs another look.

By Alex Vakulov CORE