Databases Resources

DZone's Featured Databases Resources

One Stolen Key, One Stolen Token: Why Machine Identity Is Cloud-Native's Quietest Crisis — and the Only Fix That Actually Holds

By Igboanugo David Ugochukwu

CORE

On December 2, 2024, a security vendor called BeyondTrust noticed something wrong inside its own AWS account. By the time the investigation closed, the story that emerged was almost absurdly simple for something with this much fallout: an attacker — later attributed to the Chinese state-sponsored group Silk Typhoon — had used a software flaw to reach into a BeyondTrust cloud account and pull out an API key. Not a password. Not a phishing victim's login. A string of characters that a piece of software used to talk to another piece of software. With that one key, the attacker walked straight into the U.S. Department of the Treasury, reset internal passwords, accessed workstations inside the Office of Foreign Assets Control, and read unclassified documents before anyone noticed. The Treasury disclosed it to Congress on December 30. The Department of Justice indicted the alleged operators in March 2025. If you've never worked in security, here's the plain-English version of what happened: somewhere inside the machinery that runs modern software, there's almost always a "key" — a credential one computer program shows another to prove it's allowed to be there. Humans log in with passwords and, increasingly, a second factor on their phone. Software mostly doesn't. It just holds a key, often for months or years at a time, and whoever holds that key gets treated as trustworthy, no questions asked. The Treasury breach happened because one of those keys ended up in the wrong hands and nothing else stood between that key and a federal agency's internal documents. Two months later, a different flavor of the same problem produced the largest theft of digital assets in history. $1.5 Billion, One Developer's Laptop In February 2025, the cryptocurrency exchange Bybit lost approximately $1.5 billion in Ethereum in a single operation. Palo Alto Networks' Unit 42 threat research team later tied the attack to Slow Pisces, a North Korean state-linked group also known as Lazarus or TraderTraitor, and traced the entry point back to a developer at a third-party vendor that managed Bybit's multi-signature wallet infrastructure. The attackers didn't break Ethereum's cryptography. They stole that developer's AWS session tokens — another form of machine credential — and used them to gain administrative access to cloud infrastructure that could authorize transactions, then quietly altered what a routine-looking transaction actually did before it executed. Unit 42 then found the same pattern at a second cryptocurrency exchange later in 2025, this time running through Kubernetes, the orchestration system that now runs much of the cloud-native world. The attackers phished a developer, used the access on the developer's machine to drop a malicious workload directly into the exchange's production Kubernetes cluster, and had that workload expose its own service account token — a credential Kubernetes automatically hands to every running pod so it can talk to the cluster's control plane. The stolen token happened to belong to a CI/CD management identity with sweeping permissions. From there, the intruders queried secrets across namespaces, planted a backdoor, and pivoted into the exchange's cloud-hosted backend, reaching the financial systems behind it. Unit 42's broader research found suspicious activity consistent with service-account-token theft in 22 percent of cloud environments analyzed in 2025, and recorded a 282 percent year-over-year jump in Kubernetes-directed attacks overall. Different industries, different attackers, same root cause: a non-human credential that was both long-lived and broader in scope than the task in front of it ever needed. Why This Keeps Happening Identity and access management, as a discipline, was built for people. People have managers, onboarding dates, performance reviews, and an HR system that flags them the day they leave. A workload has none of that. A microservice can spin up, do its job, and disappear thousands of times a day; a service account, by contrast, often gets created once and never revisited again. CyberArk's research has been blunt about the resulting imbalance: machine identities now outnumber human ones by more than 80 to 1 in the average enterprise, and the security architecture protecting most of them still assumes the old, human-shaped world — an org chart, not a fleet of ephemeral containers. That mismatch is exactly why static secrets sprawl the way they do. A developer hardcodes a key during a deadline crunch, intending to externalize it "later." A Terraform state file ends up holding plaintext cloud credentials because nobody flagged it in review. A default Kubernetes service account token, more permissive than anyone realized, gets mounted into a pod by default because turning that off requires deliberate configuration most teams never get around to. None of these are exotic mistakes. They're the ordinary residue of moving fast, and they accumulate the way unpaid debt does — quietly, until the day someone calls it in. The structural fix has a name by now, even if adoption is uneven: frameworks like SPIFFE and its production runtime SPIRE replace the static key with a short-lived, cryptographically attested identity — something closer to a backstage pass that's reissued before every single show rather than a master key cut once and handed out forever. A workload proves what it actually is — which Kubernetes service account launched it, which container image it's running — and receives an identity document valid for minutes, not months. Steal that, and an attacker is racing a clock that resets automatically rather than one that only resets when a human notices something is wrong. Cloud providers offer narrower versions of the same idea for their own platforms — AWS's IAM Roles for Service Accounts, Google's Workload Identity Federation — letting a workload trade a short-lived token for cloud access instead of carrying a standing key in the first place. But identity alone doesn't close the loop, and this is the part most "zero trust" conversations skip past. None of it matters if nothing in your pipeline actually enforces it. Security By Design Is a Promise. CI/CD Is Where You Find Out If It's Kept. Plenty of organizations will tell you, with complete sincerity, that they practice "security by design." Most of them mean it stopped at an architecture review months before the first line of code shipped. That's not a fix, it's a memory of one. Code that deploys daily — sometimes hourly — doesn't wait for an annual audit to catch a misconfigured token or an over-privileged service account, and by the time a quarterly review would have caught the BeyondTrust-style key or the Bybit-style session token, the damage in both real cases was already done. The only version of "security by design" that survives contact with a real production pipeline is the one written as code and enforced automatically, at every stage, by something that can actually say no. Picture the pipeline this way: Plain Text Developer commits code | v CI build triggers | +--> SAST (code flaws) + SCA (dependency CVEs) + secrets scan | | | fail? -----> build blocked, developer notified | | | pass v Generate SBOM + sign artifact (Cosign) + build provenance (SLSA) | v Policy-as-code gate (OPA / Kyverno) | +--> checks: image from approved registry? running as non-root? | signature valid? provenance matches expected builder? | service account scoped to least privilege? | | fail? -----> deployment rejected, logged, alert raised | pass v Deploy to production | v Runtime monitoring + short-lived workload identity (SPIFFE/SPIRE, IRSA) | v Continuous re-verification — nothing trusted indefinitely Every box in that chain is a place where the Treasury breach or the Bybit breach could have stopped instead of escalating. A policy-as-code rule using Open Policy Agent's Rego language, or Kyverno's Kubernetes-native YAML equivalent, can flatly refuse to schedule a pod requesting broader RBAC permissions than its declared task needs — which would have directly undercut the over-privileged CI/CD identity that the crypto-exchange attackers rode into the cluster. A signing and attestation step using Cosign, tied to SLSA provenance, means a deployed artifact has to prove which build system actually produced it before it runs at all — closing exactly the kind of trust gap that let a single compromised AWS asset cascade into a stolen infrastructure API key at BeyondTrust. None of this is theoretical tooling. Red Hat's own Enterprise Contract documentation describes signing as tying an image to a specific builder identity precisely so an attacker can't substitute a malicious binary without the signature itself breaking and announcing the tampering. The Uncomfortable Bottom Line I don't think either of this year's headline breaches happened because anyone involved was careless in some obvious, fireable way. They happened because the credential — not the firewall, not the encryption, not the cleverness of the malware — was the actual asset under attack the entire time, and almost nothing downstream of "the key worked" was built to ask a second question. Gartner named non-human identity management a top strategic security trend for exactly this reason in 2025, and OWASP followed with a dedicated Non-Human Identity Top 10 the same year, an overdue acknowledgment that the tooling built for human logins was never going to be enough. My honest prediction, watching this pattern repeat across a federal agency and two of the largest crypto exchanges on earth within twelve months of each other: the organizations that treat policy-as-code enforcement and short-lived machine identity as default infrastructure — not optional hardening bolted on after an incident — are the ones that won't end up writing the next version of this story. Everyone else is currently running on borrowed time, secured by a key that, statistically, is already older than it should be. More

Mac Native Builds, Live Protocols, And Open Issues Under 350

By Shai Almog

CORE

Our focus was all over the place this week with work that targeted many different directions: desktop, monetization, communication, media, and more. This fits with our roadmap of one platform that delivers the promise Java never delivered: WORA for Everything Everywhere. But before we dig into the new features, there's one number I'm particularly proud of… Open Issues Are Under 350 The open issue count is now below 350 (332 at the moment of this writing). That is the result of a deliberate pass through the tracker, closing things that were already fixed, reproducing the ones that were not, and fixing a batch of them outright. Some of the reports we closed this week had been open since 2015. We got a little side-tracked in the process (it is hard to read an old report without wanting to fix it on the spot), but the direction is set: we want this number to keep dropping, and by a lot. We went over the issue tracker, starting with the oldest issues and working our way back. You may recall that when we started tracking this, going under the 500-issue mark was a major milestone, and that was just a few weeks back! What Shipped Every one of the bigger items has its own deep-dive tutorial. Here is the tour, with the links to the full posts. Your App Is Now a Native Mac App Your existing Codename One app can ship as a 100% native Mac app today, with zero porting effort. No rewrite, no bundled JVM, no Electron shell: the project you already have produces a lean native Mac binary the same way it produces your iPhone app, on the same Metal renderer and battle-tested native pipeline. And it arrives feeling like a real Mac app, not a phone in a window: native title bar, native menu bar, interactive scrollbars, and desktop notifications come with it. Two of this week's features in one shot: the sample below uses the new advertising API covered later in this post, running as a native Mac app from the same Java code that produces the iOS and Android builds: The full tutorial, including the new desktop menu and shortcut APIs and the Mac signing hints, is in Your Codename One App, Now A Native Mac App. WebSockets in the Core com.codename1.io.WebSocket is now part of the framework with no cn1lib required, implemented natively on every port. A live connection is a fluent one-liner: Java WebSocket ws = WebSocket.build("wss://chat.example.com/room/general") .onConnect(() -> Log.p("connected")) .onTextMessage(text -> Display.getInstance() .callSerially(() -> addBubble(text, false))) .connect(); This is the foundation the GraphQL subscriptions below ride on, and it is trusted enough that our own screenshot CI uses it as the transport for device renders. The hands-on tutorial that builds a live chat with it is WebSockets, gRPC, and GraphQL In The Core. A Typed GraphQL Client From Your Schema cn1:generate-graphql turns a GraphQL schema into a typed client: you declare an interface against your operations and the build emits the implementation, with zero HTTP plumbing on your side. A @Subscription gets you live server-pushed updates over the new core WebSocket: Java @GraphQLClient("https://swapi.example.com/graphql") public interface StarWarsApi { @Query("query HeroName($episode: Episode) { hero(episode: $episode) { name } }") void hero(@Var("episode") Episode episode, OnComplete<GraphQLResponse<HeroData>> callback); @Subscription("subscription OnReview($ep: Episode!) { reviewAdded(episode: $ep) { stars } }") GraphQLSubscription onReview(@Var("ep") Episode ep, GraphQLSubscription.Handler<ReviewData> handler); } Note that Episode is a real enum: enum binding landed in the JSON/XML mapper alongside this work, fixing a long-standing gap for @Mapped users too. The full tutorial is in WebSockets, gRPC, And GraphQL In The Core. gRPC Clients With No Protoc cn1:generate-grpc does the same for proto3. Point it at your .proto files and it generates the messages, the binary protobuf codecs, and the client, speaking the standard gRPC-Web wire protocol that works with Envoy and modern gRPC servers. There is no protoc to install and no native dependency; calling a service looks like this: Java GreeterGrpc g = GreeterGrpc.of("https://api.example.com"); HelloRequest req = new HelloRequest(); req.name = "world"; g.sayHello(req, "Bearer " + token, response -> { if (response.isOk()) { renderGreeting(response.getResponseData().message); } }); Together with cn1:generate-openapi and cn1:generate-graphql, this means a typed client for practically any backend is one Maven goal away. The walkthrough from proto file to running call is in WebSockets, gRPC, And GraphQL In The Core. A New Advertising API Advertising support had quietly rotted across three dead-end legacy mechanisms. A pluggable, format-complete com.codename1.ads subsystem replaces all of them, with a modern AdMob reference provider, GDPR consent and iOS App Tracking Transparency built in, and a simulator placeholder provider so you can exercise every format without a device. A rewarded ad, the format people most often ask about, is now this short: Java RewardedAd ad = new RewardedAd("your-rewarded-ad-unit-id"); ad.setAdListener(new AdListener() { public void onLoaded() { ad.show(reward -> grantCoins(reward.getType(), reward.getAmount())); } }); ad.load(); The full story, including banners, native ads, app-open ads, and the provider SPI, is in A New Advertising API, Built From The Ground Up. Background Execution and Push Constraint-based background work, foreground services, push topics, shared-content handling, and a much richer local notification API, all of it usable in the simulator so you can debug these flows on your desktop. You describe what the work needs, not when to poll: Java WorkRequest req = WorkRequest.builder("daily-sync", SyncWorker.class) .setRequiresNetwork(true) .setRequiresCharging(true) .setPeriodic(6 * 60 * 60 * 1000L) .build(); BackgroundWork.schedule(req); The walkthrough, from progress notifications and inline replies to push topics and shared content, is Background Work, Push Topics, and Richer Notifications. Building Screens From Screenshots, and a Simpler Initializr Generated apps ship with a codename-one agent skill under .claude/skills/, so an AI agent working in your project already knows how to build, test, and screenshot a Codename One UI. PR #5161 teaches it the single most common design task: "make this screen look like this mockup." The hard part of that task was that the agent had no objective measure of how close it had gotten. The existing screenshot test only compares a render to a baseline the system produced itself, which measures consistency, not correctness. The new CompareToMockup tool is a single-file, pure-JDK CLI that scores a render against a target image and prints a similarity percentage: a STRUCTURAL score (an SSIM-style perceptual measure, robust to font and anti-aliasing noise against a vector mockup) and a PIXEL score (the fraction of pixels within the framework's own three-channel "same pixel" tolerance). It has a region mode so device chrome does not sabotage the score, a --diff heatmap, and a --min gate. That gives the agent a real signal: render, score, read the heatmap, adjust, repeat until the number stops climbing. A companion DesignImport tool turns a Figma, Sketch, or Adobe XD file (and, after PR #5168, the tokens.css from an HTML or React mockup) into a starter theme.css so the agent adjusts rather than starting from a blank page. The skill can also update itself now through an UpdateSkills tool, so a project generated months ago can pull the current guidance instead of carrying a frozen copy. Agents can now automatically update the skills to the latest versions and also describe the content of Codename One GUIs. This is valuable as they review their work and don't need to use vision, which is both more expensive and not as accurate. We also added the ability to check component alignment, which is often a problem that LLMs find difficult. There is also a new linter that I think we should expose to the human developers as well in the future. Right now, you can see all of these tools and use them just as an agent would, but they are more CLI-oriented. PR #5168 also rebuilt the Initializr, the tool that scaffolds a new project, around the Codename One design language, and trimmed it so it is easier to approach. It leads with the essentials (main class, package, and a Java or Kotlin toggle) and tucks IDE, localization, Java version, and current settings into collapsible cards, with a live preview and a single generate bar at the bottom. The four-template picker became the Java/Kotlin toggle, and the accent, rounded-buttons, and custom-CSS controls were dropped. The project model behind it is unchanged, so generated projects are the same; this is purely about lowering the barrier to getting started. Smaller Fixes Worth Knowing About Several of these came straight out of the issue-tracker pass: cubic-bezier() motion now matches CSS. PR #5122 fixes #1524: Motion.createCubicBezierMotion was feeding its control points into a 1D polynomial directly, so the curve did not match the CSS cubic-bezier() it was modeled on. It does now so animations might act differently in some cases.Always-tensile on the X axis. PR #5112 closes #1399, the next-oldest open issue (filed March 2015): setAlwaysTensile(true) now applies horizontally as well as vertically. This means you would see the rubber-band effect also on X axis scrolling.Validation highlights on tap-away. PR #5123 closes #1459: a field with an invalid value is now highlighted when you tap into a different field, not only when you press the virtual keyboard's next/enter.EncodedImage.dispose() actually frees memory. PR #5127 makes dispose() release the decoded image and the encoded bytes (it was a no-op before) and adds isDisposed(). Closes #3733.NetworkManager.ping(). PR #5130 adds ping(url, timeoutMillis), a real server-reachability probe to pair with the device-side isConnected(). Closes #3669.ImageViewer drag bubbling. PR #5132: a vertical drag on an ImageViewer at zoom 1 now scrolls the parent container instead of being swallowed. Closes #3700. This means you can now include many image viewers in a scrollable Y container.Graphics.isVisible(). PR #5129 adds a clip-intersection primitive so a zoomed canvas can cull off-screen content and skip the decode/scale. Closes #3846.Screenshot block now covers peers. PR #5107: ios.blockScreenshotsOnEnterBackground=true was hiding the render surface but leaving peer components (such as a BrowserComponent's WKWebView) visible in the app-switcher snapshot. Fixed.Better site search. PR #5090 sorts the on-site search index newest-first and stops the giant developer-guide page from crowding out every result. We also have dates visible next to blog post results in the search, and a new highlight explaining we don't search the developer guide/Javadoc. A Note on Contributions We stopped accepting community pull requests. We want to be precise about why, because it would be easy to read more into this than is there. This is not about AI-generated PRs. We are not policing how a contribution was written. The real reason is mechanical: our CI does not run correctly against pull requests from forks. The screenshot pipeline, the device runners, and the protocol tests all need credentials and a setup that a forked PR cannot get, so a community PR cannot actually be validated by the same gates we hold our own work to. This isn't something we can easily solve without introducing major security vulnerabilities to our process. Recently, a change that looked completely safe slipped through and triggered a CI regression that took the builds down. That was on us, not on the contributor, but it convinced us that merging code we cannot fully run is the wrong trade. So the door is open the other way. Please keep filing issues. A clear issue with a test case is genuinely no trouble for us to pick up, and as the number above shows, we are actively working on the tracker. If you have a fix in mind, describe it in the issue, attach the failing case, and we will carry it through CI ourselves. To everyone who has sent us patches over the years, and especially the people who contributed recently: thank you. The effort was real and appreciated, and this decision is about our pipeline, not your work. Upcoming Attractions Four deep-dive tutorials follow this one, one per day: Saturday. Native Mac builds and deeper desktop integration. PRs #5053, #5136, #5170.Sunday. WebSockets, gRPC, and GraphQL in the core. PRs #5133, #5099, #5141.Monday. The new advertising API. PR #5169.Tuesday. Background work, push topics, and richer notifications. PR #5142. Wrapping Up The issue tracker is here, and it is the best place to reach us right now. The discussion forum is here, and the Build Cloud console is at /console/. The Playground, Initializr, and Skin Designer are where they have always been. More

Code and Connect: MCP + MuleSoft

By Ajay Singh

Selective Deployment in Azure Data Factory: A Practical Blueprint for Safer CI/CD

By Sauhard Bhatt

REST-Assured Configuration and Specifications: Writing Maintainable API Tests

By Faisal Khatri

CORE

AI, OAuth, and Other Platform APIs in the Core

This is the second follow-up to June 5's release post. It covers the platform APIs that moved into the framework core this release. There are two headline pieces (AI/LLM and the modern OAuth/OIDC stack) and two smaller pieces (WiFi/connectivity and share-sheet result callbacks). This continues the direction the previous release set when we moved NFC, biometrics, and cryptography into the framework core. The full background on that earlier set is in NFC, Crypto, Biometrics, And A New Build Cloud. AI: A First-Class LLM Client and a ChatView Component PR #5035 lands the com.codename1.ai package, the ChatView UI component, the speech and TTS additions, and the build-time dependency injection that wires the native pieces in. PR #5057 lands the developer-guide chapter and the agent-skill addition, so any project generated from the Initializr inherits the new APIs through its bundled AGENTS.md. LlmClient: The Basic Chat Request com.codename1.ai.LlmClient is the entry point. The simplest possible use: Java LlmClient client = LlmClient.openai(apiKey); ChatRequest req = new ChatRequest.Builder() .model("gpt-4o-mini") .system("You are a helpful assistant.") .user("What is the capital of France?") .temperature(0.7) .build(); client.chat(req).onResult((resp, err) -> { if (err != null) { Log.e(err); return; } Log.p(resp.firstChoice().content()); LlmClient.openai(...), LlmClient.anthropic(...), LlmClient.gemini(...), LlmClient.ollama(...), and LlmClient.openAiCompatible(baseUrl, apiKey) are the factories. All five are fully implemented native clients. The OpenAI client also drives Ollama, vLLM, llama.cpp, and any other endpoint that speaks the OpenAI wire format, so most local-model stacks plug in through LlmClient.openAiCompatible(...) without a separate driver. Streaming Chat (What You Actually Want for Chat UIs) For any UI that types responses out token-by-token, the streaming entry point is the one to reach for. The callback fires on the EDT, so you can append directly to a text component: Java client.chatStream(req, new ChatStreamListener() { @Override public void onDelta(ChatDelta d) { responseLabel.setText(responseLabel.getText() + d.contentDelta()); responseLabel.getParent().revalidateLater(); } @Override public void onComplete(ChatResponse fin) { sendButton.setEnabled(true); } @Override public void onError(Throwable t) { Log.e(t); sendButton.setEnabled(true); } Under the hood this is a custom ConnectionRequest subclass that parses SSE line-by-line and dispatches each delta through Display.callSerially. AsyncResource.cancel() kills the socket. So a chat UI that has a cancel button is a one-line cancellation. Tool Calls If you want the model to call back into your app, Tool / ToolChoice give you OpenAI-style function calling. Define the tool, hand the model your model and the available tools, and the response surfaces structured ToolCall objects you dispatch: Java Tool getWeather = Tool.builder() .name("get_weather") .description("Look up the current weather for a city.") .parameter("city", "string", "The city name, e.g. \"Paris\".") .build(); ChatRequest req = new ChatRequest.Builder() .model("gpt-4o-mini") .user("Is it raining in Tel Aviv right now?") .tool(getWeather) .toolChoice(ToolChoice.AUTO) .build(); client.chat(req).onResult((resp, err) -> { if (err != null) return; for (ToolCall call : resp.firstChoice().toolCalls()) { if ("get_weather".equals(call.name())) { String city = call.argument("city").asString(); String json = lookupWeather(city); // Loop the result back into the conversation client.chat(req.replyWithToolResult(call, json)) .onResult((followUp, e) -> updateUi(followUp)); } } The shape mirrors the OpenAI function-calling contract one for one, so anything you have written against the OpenAI API directly maps across without rethinking. Embeddings LlmClient.embed(...) returns a vector for any input string. Useful for similarity search against a local SQLite store (tomorrow's post will cover the new ORM that pairs with this): Java EmbeddingRequest er = new EmbeddingRequest.Builder() .model("text-embedding-3-small") .input("Codename One is a cross-platform mobile framework.") .build(); client.embed(er).onResult((emb, err) -> { float[] vector = emb.firstVector(); // store, search, compare Image Generation DALL-E and a Replicate scaffold are surfaced through ImageGenerator: Java ImageGenerator gen = ImageGenerator.openAiDallE(apiKey); gen.generate("A red bicycle leaning against an olive tree", "1024x1024") .onResult((img, err) -> { if (err != null) return; myImageComponent.setIcon(img); Working Against Ollama in the Simulator (No API Charges) JavaSEPort pings localhost:11434 at startup. If it finds Ollama, it sets the cn1.ai.ollamaDetected property. With cn1.ai.simulatorRedirect=auto (or =ollama) every LlmClient.openai(...) call routes through the local Ollama endpoint instead of OpenAI's. Production code does not change. The iteration loop, your tests, and your offline debugging stop costing money and stop needing an internet connection. In common/codenameone_settings.properties: Properties files simulator.cn1.ai.simulatorRedirect=auto (The simulator. prefix scopes the property to the JavaSE simulator path.) Then run Ollama locally with whichever model your code expects (ollama run llama3.2 or similar) and your existing LlmClient.openai(...) calls go to localhost. How to Handle API Keys A direct word on credentials before any of the above sees production. LLM provider API keys (OpenAI, Anthropic, Gemini, your Auth0 / Firebase configs) are bearer tokens with a budget attached. They must never be checked into source control, embedded in your app binary, or hard-coded in code. A leaked key can be extracted from any APK or IPA in minutes and used to drain your account. The correct shape is to fetch the key from your own backend over an authenticated request, then store it on the device using the platform's keychain / keystore. The framework provides both pieces: com.codename1.crypto.SecureStorage (from the previous release) is the cross-platform wrapper over iOS Keychain Services and Android EncryptedSharedPreferences. Values are encrypted at rest using the platform's hardware-backed protection class where one is available.This release adds a single-argument get / set / remove(account, ...) overloads next to the existing biometric-gated methods. The new overloads store the value without a per-read Face ID / Touch ID prompt, which is what you want for an LLM API key (you read it on every network call; a biometric prompt every time is not workable). The biometric-gated methods are still there for credentials you do want to gate per use. A reasonable shape: Java private static AsyncResource<String> getOpenAiKey() { String cached = SecureStorage.get("openai_api_key"); if (cached != null) { return AsyncResource.complete(cached); } return Rest.get(myServer + "/v1/credentials/openai") .bearerToken(userSessionToken()) .fetchAsString() .onResult((key, err) -> { if (err == null) { SecureStorage.set("openai_api_key", key); } }); Your server gates the credential request behind the user's session, your app caches the result on the keychain, and the key never sits anywhere a reverse-engineering pass could find it. If your server rotates the key, invalidate the cache and refetch. Existing biometric-gated SecureStorage calls keep working unchanged. The new overloads are additive. ChatView: A Ready-Made Streaming Chat UI com.codename1.components.ChatView is the matching UI component. Scrollable message list, ChatBubble for the per-message bubble (theme-aware UIIDs so it picks up the iOS Modern / Material 3 native themes consistently), ChatInput for the bottom input bar, and a one-line bindToLlm(...) that wires the input to a streaming chat request: Java ChatView view = new ChatView(); getOpenAiKey().onResult((key, err) -> { view.bindToLlm(LlmClient.openai(key), new ChatRequest.Builder() .model("gpt-4o-mini") .system("You are a friendly tutor for " + "Codename One developers.") .build()); }); Form f = new Form("Chat", new BorderLayout()); f.add(BorderLayout.CENTER, view); The result is a standard mobile chat layout, picked up from whichever native theme the project uses: If you want more control than bindToLlm(...) gives you (custom message styling, a "thinking" placeholder, hand-rolled retry, persistence to your own model class), drive the view by hand: Java ChatView view = new ChatView(); ConversationStore store = ConversationStore.open("tutor-thread"); view.setMessages(store.load()); LlmClient client = LlmClient.openai(apiKeyFromKeychain); view.setInputListener(userText -> { ChatMessage userMsg = ChatMessage.user(userText); view.appendMessage(userMsg); store.append(userMsg); ChatMessage assistant = ChatMessage.assistant(""); view.appendMessage(assistant); ChatRequest req = new ChatRequest.Builder() .model("gpt-4o-mini") .messages(store.load()) .build(); client.chatStream(req, new ChatStreamListener() { @Override public void onDelta(ChatDelta d) { view.appendToLastMessage(d.contentDelta()); } @Override public void onComplete(ChatResponse fin) { store.append(ChatMessage.assistant(view.lastMessage().content())); view.setInputEnabled(true); } @Override public void onError(Throwable t) { view.appendToLastMessage(" [error: " + t.getMessage() + "]"); view.setInputEnabled(true); } }); appendToLastMessage(...) is the streaming entry point; it marshals through callSerially so deltas land on the EDT in order. ConversationStore persists the thread (the default backing is Storage; pluggable via a custom implementation if you would rather keep it in SQLite or push it to your server). The AI cn1libs The core LLM stack is paired with a set of opt-in cn1libs that wrap specific on-device capabilities: Google ML Kit features, the TensorFlow Lite runtime, a local Whisper transcription engine, and an on-device Stable Diffusion model. Thirteen new cn1libs ship this release. These cn1libs are not yet listed in the Codename One Preferences cn1lib picker, so for the moment they are added by hand. Drop the matching dependency block into your project's common/pom.xml and rebuild. The build-time scanner does the rest: the iOS pod or Swift Package, the Android Gradle dependency, the plist usage strings (NSCameraUsageDescription for the vision libraries, NSSpeechRecognitionUsageDescription for Whisper, etc.), and the Android permissions (android.permission.RECORD_AUDIO for audio capture) are all injected automatically the first time the scanner sees the matching class on the classpath. For each cn1lib below, the dependency block is identical in shape; only the <artifactId> changes. The shared pattern is: XML <dependency> <groupId>com.codenameone</groupId> <artifactId></artifactId> <version>${cn1.version}</version> </dependency> cn1-ai-mlkit-text: Text Recognition (OCR) TL;DR. Pull printed or handwritten text out of an image (a photo of a page, a sign, a receipt) entirely on-device. Platforms. iOS bridges to GoogleMLKit/TextRecognition. Android bridges to com.google.mlkit:text-recognition. The JavaSE simulator returns an unsupported error. Use cases. Receipt scanning, sign translation pipelines (combine with cn1-ai-mlkit-translate), accessibility tools that read printed text aloud, automated form ingestion. Java byte[] jpeg = capturePhotoBytes(); TextRecognizer.recognize(jpeg).onResult((text, err) -> { if (err == null) Log.p("OCR: " + text); cn1-ai-mlkit-barcode: Barcode and QR Scanning TL;DR. Decodes QR, EAN, UPC, Data Matrix, PDF417, and the rest of the common 1D / 2D code families from a captured image. Platforms. iOS bridges to MLKitBarcodeScanning. Android bridges to com.google.mlkit:barcode-scanning. The JavaSE simulator returns an unsupported error. Use cases. Inventory scanning, ticket / boarding-pass readers, QR-driven onboarding flows, retail loyalty cards. Java byte[] jpeg = capturePhotoBytes(); BarcodeScanner.scan(jpeg).onResult((codes, err) -> { if (err == null) { for (String code : codes) Log.p("Found: " + code); } }); cn1-ai-mlkit-face: Face Detection TL;DR. Returns bounding boxes for human faces detected in an image. Each face is reported as a packed int[4] (x, y, width, height). Platforms. iOS bridges to MLKitFaceDetection. Android bridges to com.google.mlkit:face-detection. Use cases. Auto-crop a contact photo, mosaic / blur bystanders in a group shot, drive a face-tracked overlay for AR-lite filters. Java FaceDetector.detect(jpeg).onResult((boxes, err) -> { if (err != null) return; for (int i = 0; i < boxes.length; i += 4) { Log.p("face at " + boxes[i] + "," + boxes[i + 1] + " " + boxes[i + 2] + "x" + boxes[i + 3]); } }); cn1-ai-mlkit-labeling: Image Labeling TL;DR. "What is in this picture." Returns a list of descriptive labels for the image content. Platforms. iOS bridges to MLKitImageLabeling. Android bridges to com.google.mlkit:image-labeling. Use cases. Auto-tagging uploaded photos, content moderation pre-filters, content-based image search. Java ImageLabeler.label(jpeg).onResult((labels, err) -> { if (err == null) Log.p("labels: " + String.join(", ", labels)); }); cn1-ai-mlkit-translate: On-Device Translation TL;DR. Translate short text between supported language pairs entirely on-device; no server round-trip, no API key, works offline. Platforms. iOS bridges to MLKitTranslate. Android bridges to com.google.mlkit:translate. Languages are identified by their ISO 639-1 codes (en, fr, es, ...). Use cases. Offline travel assistants, chat translation, accessibility readers for foreign signage (combine with cn1-ai-mlkit-text). Java Translator.translate("Where is the train station?", "en", "fr") .onResult((fr, err) -> { if (err == null) Log.p(fr); // "Où est la gare ?" }); cn1-ai-mlkit-smartreply: Short Reply Suggestions TL;DR. Generates short suggested replies for chat conversations, similar to Gmail's Smart Reply chips. Platforms. iOS bridges to MLKitSmartReply. Android bridges to com.google.mlkit:smart-reply. The input is a JSON array of {role, message, timestamp, userId} objects. Use cases. A "quick reply" row above the keyboard in your in-app chat, response suggestions in a CRM inbox. Java String thread = "[{\"role\":\"remote\",\"message\":\"See you at 6?\"," + "\"timestamp\":" + System.currentTimeMillis() + "," + "\"userId\":\"u42\"}]"; SmartReply.suggest(thread).onResult((suggestions, err) -> { if (err == null) { for (String s : suggestions) Log.p("suggestion: " + s); } }); cn1-ai-mlkit-langid: Language Identification TL;DR. Returns the most likely ISO 639-1 code for a given text, or und (undetermined) when the input is too short or ambiguous. Platforms. iOS bridges to MLKitLanguageID. Android bridges to com.google.mlkit:language-id. Use cases. Auto-route a customer-support message to the right team, pick the correct TTS voice for an arbitrary string, pre-screen input before running an expensive translation. Java LanguageIdentifier.identify("Bonjour le monde").onResult((code, err) -> { if (err == null) Log.p(code); // "fr" }); cn1-ai-mlkit-pose: Pose Detection TL;DR. Returns 33 skeletal landmarks per detected pose as a packed float[3 * 33] (x, y, confidence triples). Platforms. iOS bridges to MLKitPoseDetection. Android bridges to com.google.mlkit:pose-detection. Use cases. Fitness apps with form correction, dance/yoga timing analysis, gesture-driven controls. Java PoseDetector.detect(jpeg).onResult((landmarks, err) -> { if (err != null || landmarks.length < 99) return; float noseX = landmarks[0], noseY = landmarks[1], noseConf = landmarks[2]; Log.p("nose at (" + noseX + ", " + noseY + ") conf=" + noseConf); }); cn1-ai-mlkit-segmentation: Selfie Segmentation TL;DR. Returns a per-pixel mask separating the person in the foreground from the background as byte[width * height] (0 = background, 255 = foreground). Platforms. iOS bridges to MLKitSegmentationSelfie. Android bridges to com.google.mlkit:segmentation-selfie. Use cases. Background replacement for video calls, sticker / portrait-mode effects, blur-the-background privacy filters. Java SelfieSegmenter.segment(jpeg).onResult((mask, err) -> { if (err == null) applyBackgroundReplacement(mask); }); cn1-ai-mlkit-docscan: Document Scanner TL;DR. Detects a rectangular document in a photo, perspective-corrects it, and writes the cropped JPEG to a temporary file. Returns the file path. Platforms. iOS uses Apple's VisionKit + Core Image rectangle detection (no extra pod). Android uses com.google.android.gms:play-services-mlkit-document-scanner. Use cases. "Scan to PDF" flows, expense apps that capture receipts, contract signing flows, ID-document capture. Java DocumentScanner.scanToFile(jpeg).onResult((path, err) -> { if (err == null) uploadDocument(path); }); cn1-ai-tflite: TensorFlow Lite Interpreter TL;DR. A general-purpose on-device inference engine. Bring your own .tflite model and run it against a float32 input tensor. Platforms. iOS uses TensorFlowLiteSwift (Pods or Swift Package). Android uses org.tensorflow:tensorflow-lite + tensorflow-lite-support. Use cases. Any custom on-device ML model your team trains or pulls from TF Hub. Image classification, simple regression, recommendation pre-filters. Java byte[] modelBytes = Util.readFully(Display.getInstance().getResourceAsStream(null, "/model.tflite")); float[] input = featureVector(); Interpreter.run(modelBytes, input).onResult((output, err) -> { if (err == null) Log.p("model returned " + output.length + " values"); }); cn1-ai-whisper: Speech-to-Text via whisper.cpp TL;DR. On-device transcription of a 16 kHz mono WAV file using a ggml-format Whisper model. The cn1lib bundles libwhisper.a. Platforms. iOS uses the Accelerate framework; Android uses a JNI build of the same whisper.cpp core. Models (e.g. ggml-base.bin) are not bundled; ship the one your app expects under the app's resources or download on first launch. Use cases. Voice notes, accessibility transcription, offline dictation, podcast indexing. Java String modelPath = SecureStorage.getFilePath("ggml-base.bin"); String audioPath = recordWavToFile(); WhisperRecognizer.transcribe(modelPath, audioPath) .onResult((text, err) -> { if (err == null) Log.p("heard: " + text); }); cn1-ai-stablediffusion: On-Device Image Generation TL;DR. Generates a JPEG from a text prompt using a bundled Stable Diffusion model. Multi-gigabyte payload, local build only. Platforms. iOS uses Core ML pipelines compiled from the bundled model. Android uses ONNX Runtime. Both configurations exceed the cloud build server's 2 GB upload limit, so this cn1lib triggers the cn1.ai.requiresBigUpload guard and the cloud build aborts with a "build this one locally" message. Add it to a project you build via mvn cn1:buildAndroid / mvn cn1:buildIosXcodeProject on the developer machine. Use cases. Avatar generation in apps where shipping to a cloud API is undesirable (offline-first apps, regulated industries, privacy-sensitive products). Java StableDiffusion.generate("a teal hot-air balloon over Lisbon, watercolour", 512, 512, /* steps */ 25) .onResult((jpeg, err) -> { if (err == null) display(Image.createImage(jpeg, 0, jpeg.length)); }); Why These Are cn1libs and Not Part of the Core The core gets the AI plumbing every app that adopts AI at all wants: the LLM client, streaming, the chat UI, the secure storage primitive for credentials, the simulator Ollama redirect for offline iteration. The cn1libs above are specialized verticals. Barcode scanning, document scanning, face detection, smart reply, pose detection, on-device translation, transcription, and on-device image generation are genuinely useful, but only for some apps. They also each bring a non-trivial native dependency. The Google ML Kit Android frameworks are large; the iOS pods carry their own weight; the bundled libwhisper.a and the Stable Diffusion model are big. Pulling all of them into the core would tax every app, whether the feature is used or not. The Stable Diffusion cn1lib in particular is large enough that the cloud build server cannot accept the upload at all (it trips the 2 GB pre-upload guard). That kind of opt-in does not belong in a dependency every app inherits. The corresponding chapter, including the full LlmClient API table, the ChatView reference, the SecureStorage overloads, the simulator Ollama redirect, and the full cn1lib coverage, is at AI, Chat UI, and Speech in the developer guide. OAuth and OIDC: The Modern Identity Stack The in-app-WebView Oauth2 flow that Codename One has shipped since approximately forever was the way every cross-platform mobile framework solved "sign in with Google / Facebook / Microsoft" in the 2010s. It is also the way every one of those identity providers stopped wanting you to solve it. Google has been blocking embedded user agents for years. Apple does not want third-party apps wrapping the Apple ID flow in a WKWebView. Microsoft and Facebook joined the chorus. The right answer is the system browser: ASWebAuthenticationSession on iOS, Custom Tabs on Android, with PKCE on the wire. That is what PR #5018 lands. PR #5039 adds a portable WebAuthn / passkey client on top. Sign In With Google (or Any OIDC Provider) com.codename1.io.oidc.OidcClient is the entry point. Point it at the discovery URL of an OIDC provider, hand it the client id and the redirect URI you registered with the provider, ask for tokens: Java OidcConfiguration cfg = OidcConfiguration.discover("https://accounts.google.com"); OidcClient client = OidcClient.builder() .configuration(cfg) .clientId("123-abc.apps.googleusercontent.com") .redirectUri("com.example.myapp:/oauthredirect") .scopes("openid", "email", "profile") .build(); client.signIn().onResult((tokens, err) -> { if (err != null) { OidcException oe = (OidcException) err; if (oe.getCode() == OidcException.USER_CANCELLED) return; Log.e(oe); return; } String idToken = tokens.getIdToken().raw(); String email = tokens.getIdToken().getClaim("email").asString(); proceed(email, idToken); Discovery JSON parsed and cached. PKCE S256 challenge generated and verified. State and nonce checked on the callback. ID-token claims decoded for you (we deliberately do not verify the signature client-side; the dev guide is explicit about why and points at the "re-validate on your backend" remedy). Refresh and revoke are first-class. The token store is pluggable via TokenStore; the default is Storage-backed, but a Keychain-backed or in-memory variant is a small class. On iOS the system-browser piece routes through ASWebAuthenticationSession. On Android through androidx.browser.customtabs with a plain ACTION_VIEW fallback for the rare device with no Custom Tabs provider. AuthenticationServices.framework and androidx.browser:browser are auto-linked when the classpath scanner sees OidcClient in use. Provider Wrappers: Google, Apple, Microsoft, Facebook, Auth0, Firebase If you would rather not configure OIDC by hand, the existing social classes get a signIn(...) method that drives the same stack with the provider's issuer URL pre-wired: Java GoogleConnect.signIn(googleClientId, "com.example.myapp:/oauthredirect", "openid", "email", "profile") .onResult((tokens, err) -> { /* ... */ }); MicrosoftConnect.signIn(entraClientId, "msauth.com.example.myapp://auth", "User.Read") .onResult((tokens, err) -> { /* ... */ }); Auth0Connect.signIn("tenant.auth0.com", clientId, redirectUri, "openid profile email") .onResult((tokens, err) -> { /* ... */ }); FacebookConnect.signIn(...) follows the same shape against the Facebook OIDC endpoint. FirebaseAuth covers the REST-based Firebase auth surface (email/password, IdP token exchange, refresh) which sits underneath any provider hand-off you might want to drive from app code. Sign In With Apple Sign in with Apple is required on iOS for apps that offer any other social login, and on Android it must fall through to a web flow. com.codename1.social.AppleSignIn handles both transparently: Java AppleSignIn.signIn() .onResult((result, err) -> { if (err != null) return; String idToken = result.getIdToken(); String code = result.getAuthorizationCode(); proceedToBackend(idToken, code); }); On iOS 13 and later this drops directly into the native Apple sheet via ASAuthorizationAppleIDProvider. On non-iOS platforms it falls through to the same OIDC web flow as everything else, so a single line of app code does the right thing on every port. The Maven plugin injects the com.apple.developer.applesignin entitlement on iOS when it sees AppleSignIn in use; Android does not see it because it is not there. Migration From the Legacy Oauth2 com.codename1.io.Oauth2 is now deprecated. Existing code still compiles, but the migration is short and almost always shorter than what it replaces: Java // Before Oauth2 oauth = new Oauth2("https://accounts.google.com/o/oauth2/auth", clientId, redirectUri); oauth.setClientSecret(clientSecret); oauth.setScope("openid email profile"); oauth.setBrowserComponent(myBrowserComponent); // tied to a WKWebView String token = oauth.authenticate(); // blocks, opens the web view Java // After OidcClient.builder() .configuration(OidcConfiguration.discover("https://accounts.google.com")) .clientId(clientId) .redirectUri(redirectUri) .scopes("openid", "email", "profile") .build() .signIn() .onResult((tokens, err) -> proceed(tokens.getIdToken().raw())); You stop owning the browser. The OS owns it. The cookies live in the platform's authentication session. The user gets the same login experience they have everywhere else on their device. WebAuthn/Passkeys PR #5039 layers a portable WebAuthn client on top: Java WebAuthnClient client = WebAuthnClient.getInstance(); if (!client.isAvailable()) { fallbackToPassword(); return; } PublicKeyCredentialCreationOptions opts = PublicKeyCredentialCreationOptions.fromServerJson(serverJson); client.create(opts).onResult((cred, err) -> { if (err == null) postToRelyingParty(cred.toJson()); }); W3C JSON wire format in both directions, so the response can be POSTed verbatim to any standard server-side WebAuthn library. iOS 16+ routes through ASAuthorizationPlatformPublicKeyCredentialProvider; Android API 28+ through androidx.credentials.CredentialManager. Provider helpers: Auth0Connect.signInWithPasskey(...) / .registerPasskey(...) and FirebaseAuth.signInWithPasskey(...) / .registerPasskey(...). One thing worth pulling out before you reach for it: if you sign in via OIDC against Google, Apple, Microsoft, Auth0, or Firebase, you usually already get passkeys for free. The identity provider runs the WebAuthn ceremony inside the system browser; OIDC just hands you the resulting tokens. So you do not need WebAuthnClient for that case. You need it for apps that run their own relying-party backend, and for apps driving the Auth0 or Firebase passkey grants directly. Full chapter: Authentication and Identity. Connectivity: WiFi, Bonjour, USB, network-type listeners PR #5021 lands four packages for apps that need to do more with the network than open an HTTP socket. The shape: Java WiFi wifi = WiFi.getInstance(); String ssid = wifi.getCurrentSSID(); String bssid = wifi.getBSSID(); String gateway = wifi.getGateway(); String ip = wifi.getIp(); wifi.scan(new ScanOptions().setTimeoutMillis(5000)) .onResult((results, err) -> { /* ... */ }); wifi.connect("MyNetwork", "hunter2", Security.WPA2_PSK) .onResult((success, err) -> { /* ... */ }); com.codename1.io.wifi for WiFi info, scan, and connect. com.codename1.io.wifi.WiFiDirect for peer-to-peer (Android only by platform reality). com.codename1.io.bonjour for mDNS / Zeroconf via BonjourBrowser and BonjourPublisher. com.codename1.io.usb for USB host (Android only). And NetworkManager.addNetworkTypeListener(...) plus NETWORK_TYPE_* constants so an app can react to a transition between cellular, WiFi, ethernet, or "none": Java NetworkManager.getInstance().addNetworkTypeListener(evt -> { int type = evt.getNetworkType(); if (type == NetworkManager.NETWORK_TYPE_NONE) showOfflineBanner(); else if (type == NetworkManager.NETWORK_TYPE_CELLULAR) suppressLargeBackgroundDownloads(); else clearOfflineBanner(); }); iOS does not expose programmatic WiFi scanning to third-party apps; scan() throws UnsupportedOperationException on iOS. iOS also does not expose WiFi Direct or general USB host. None of those are Codename One limitations; they are Apple's. The dev guide is explicit about each platform's limits. Three new compile-time defines (CN1_INCLUDE_WIFI_INFO, CN1_INCLUDE_HOTSPOT, CN1_INCLUDE_BONJOUR) wrap the iOS native code, set only when the classpath scanner sees the matching Java API in use. Apps that do not use these APIs do not pay for them at App Store review time. Same pattern as the NFC gating from the previous release. Full reference: Network Connectivity. Share-Sheet Result Callbacks PR #5036 closes a small but persistent gap: Display.share(...) and ShareButton finally tell you what the user did with the share sheet: Java ShareButton btn = new ShareButton(); btn.setTextToShare("Look at this fox"); btn.setImageToShare("/fox.jpg"); btn.setShareResultListener(result -> { switch (result.getStatus()) { case SHARED_TO: track("share_completed", result.getTargetPackage()); break; case DISMISSED: track("share_dismissed"); break; case FAILED: track("share_failed", result.getError()); break; } }); iOS routes through UIActivityViewController.completionWithItemsHandler; Android through Intent.createChooser with an IntentSender callback (API 22+). The framework normalizes the platform values into SHARED_TO(packageName), DISMISSED, or FAILED. Appearing in Other Apps' Share Menus The other half of sharing is the inverse direction: not "let the user share from your app", but "let your app receive content other apps share". If a user is in Safari, Photos, or Mail and taps the share icon, your app should be able to appear as a target there alongside Messages, WhatsApp, and Instagram. On iOS that requires a separate Share Extension target inside the .ipa, with its own bundle, its own Info.plist, an App Group string that links it to the host app, and a ShareViewController that handles the incoming payload. Historically the recommendation was to bootstrap that target by hand in Xcode, copy the resulting files into the Codename One project under ios/app_extensions/, and let the build server's extractor consume them. It worked, but it was a workflow most teams put off because the setup is fiddly. The same PR ships an IOSShareExtensionBuilder Mojo that does all of that for you. A typical setup is one Maven command and a one-time configuration block: XML <plugin> <groupId>com.codenameone</groupId> <artifactId>codenameone-maven-plugin</artifactId> <configuration> <iosShareExtension> <bundleIdentifier>com.example.myapp.share</bundleIdentifier> <displayName>MyApp</displayName> <appGroup>group.com.example.myapp</appGroup> <acceptedContent> <content>PUBLIC_URL</content> <content>PUBLIC_IMAGE</content> <content>PUBLIC_TEXT</content> </acceptedContent> </iosShareExtension> </configuration> </plugin> Run mvn cn1:generate-ios-share-extension and the Mojo writes a complete .ios.appext bundle into ios/app_extensions/: the Info.plist with the right NSExtension activation rules for the content types you declared, the App Group entitlement, a minimal ShareViewController.swift that lands the payload in the App Group's UserDefaults(suiteName:), and the matching buildSettings.properties. The result feeds straight into the existing IPhoneBuilder.extractAppExtensions pipeline, so apps that already have a hand-rolled extension keep working unchanged. On the host-app side, you read the payload on launch: Java // Anywhere after Display.init has run String shared = Storage.getInstance() .readObject("ios.shareExtension.lastPayload"); if (shared != null) { handleSharedPayload(shared); } After the next cloud or local build, your app appears in the iOS share sheet for the content types you declared. No Xcode work, no hand-rolled plist, no App Group string typed in three places. The build-time tooling owns it. Wrapping Up Tomorrow's post covers the architectural change in this release: a build-time bytecode annotation framework, the declarative router that is its first consumer, the SQLite ORM and JSON / XML mappers and component binder built on the same SPI, and the build-time SVG / Lottie transcoder that ships in the same release for related reasons. Back to the weekly index.

By Shai Almog

CORE

Implementing Asynchronous Communication Between Microservices Using Kafka and Spring Boot

In a microservices system, that tight coupling turns a small hiccup into a cascading slowdown. Thread pools fill, retries amplify traffic, and suddenly your simple request is blocked on half the fleet. My executive summary: asynchronous messaging with Kafka helps systems keep moving when individual components inevitably slow down or fail. It does this by decoupling producers from consumers, absorbing traffic spikes, and allowing services to evolve without tying their availability directly to one another. Code Patterns in Spring Boot With Kafka Spring for Apache Kafka gives me two primitives that feel pleasantly old Spring KafkaTemplate for sending and @KafkaListener for receiving. That template/listener model is intentionally similar to other Spring integration tech, which keeps application code focused on domain logic instead of raw client plumbing. Below is a compact (but production-shaped) pattern: externalized config via @ConfigurationProperties, a service port for publishing, a REST command endpoint, a consumer with a real error strategy (DLT), and a REST error advice. Java // === Messaging config (externalized, type-safe) === @ConfigurationProperties(prefix = "messaging.orders") @Validated record OrdersMessagingProps( @NotBlank String topic, @NotBlank String dltTopic ) {} // === DTO (event contract) === public record OrderCreatedEvent(UUID orderId, UUID userId, BigDecimal total, Instant createdAt) {} // === Service port (keeps domain testable, Kafka swappable) === public interface OrderEventPublisher { void publishOrderCreated(OrderCreatedEvent event); } // === Adapter: Kafka producer === @Component class KafkaOrderEventPublisher implements OrderEventPublisher { private final KafkaTemplate<String, OrderCreatedEvent> template; private final OrdersMessagingProps props; KafkaOrderEventPublisher(KafkaTemplate<String, OrderCreatedEvent> template, OrdersMessagingProps props) { this.template = template; this.props = props; } @Override public void publishOrderCreated(OrderCreatedEvent event) { // Keying by orderId keeps per-order ordering and drives partitioning decisions. template.send(props.topic(), event.orderId().toString(), event); } } // === REST command API (synchronous edge, async core) === @RestController @RequestMapping("/v1/orders") class OrdersController { private final OrderService orderService; // domain port OrdersController(OrderService orderService) { this.orderService = orderService; } @PostMapping public ResponseEntity<Map<String, Object>> create(@Valid @RequestBody CreateOrderRequest req) { UUID orderId = orderService.create(req.userId(), req.total()); // persists + publishes event return ResponseEntity.accepted().body(Map.of("orderId", orderId, "status", "ACCEPTED")); } record CreateOrderRequest(@NotNull UUID userId, @NotNull @Positive BigDecimal total) {} } // === Domain service port (implementation can use outbox, transactions, etc.) === public interface OrderService { UUID create(UUID userId, BigDecimal total); } // === Consumer: downstream service reacts to events === @Component class BillingListener { @KafkaListener(topics = "${messaging.orders.topic}", groupId = "${spring.kafka.consumer.group-id}") void onOrderCreated(OrderCreatedEvent event) { // Idempotency belongs here: process-by-key + store processed eventId/orderId to avoid duplicates. // Do work (charge card, create invoice, etc.) } } // === Kafka consumer error handling: retries + DLT === @Configuration class KafkaErrorHandlingConfig { @Bean DefaultErrorHandler defaultErrorHandler(KafkaTemplate<Object, Object> template, OrdersMessagingProps props) { var recoverer = new DeadLetterPublishingRecoverer(template, (rec, ex) -> new TopicPartition(props.dltTopic(), rec.partition())); // Backoff and retry policy are configurable; keep it finite to avoid poison-pill loops. return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); } } // === REST error handling (ProblemDetail) === @RestControllerAdvice class ApiErrors { @ExceptionHandler(IllegalArgumentException.class) @ResponseStatus(HttpStatus.BAD_REQUEST) ProblemDetail badRequest(IllegalArgumentException ex) { var pd = ProblemDetail.forStatusAndDetail(HttpStatus.BAD_REQUEST, ex.getMessage()); pd.setTitle("Invalid request"); return pd; } } A few been-burned-before notes on the code above. Spring Kafka’s reference docs are explicit that KafkaTemplate is the convenience wrapper for producing, and DefaultErrorHandler + DeadLetterPublishingRecoverer is a first-class way to route failed records to dead-letter topics after retries. If we want non-blocking retries, Spring Kafka also provides @RetryableTopic, which orchestrates retry topics and a DLT automatically useful when transient failures are common and you want predictable retry delay semantics. Containers and Local Dev With Docker Compose When I’m chasing down event flow bugs, I like local environments that feel like the old days: one command, deterministic startup order, and no mystery dependencies. Docker Compose is still the quickest way to stand up Kafka alongside your services, and Confluent publishes straightforward Docker-based tutorials and compose examples for running Kafka locally. For the service image itself, multi-stage builds are the modern classic compile in a builder stage, and copy the artifact into a slimmer runtime stage. Docker documents multi-stage builds as a way to reduce the final image contents and keep build dependencies out of production. Dockerfile # Multi-stage Dockerfile for a Spring Boot service (orders-service) FROM eclipse-temurin:21-jdk AS build WORKDIR /workspace COPY mvnw pom.xml ./ COPY .mvn .mvn RUN ./mvnw -q -DskipTests dependency:go-offline COPY src src RUN ./mvnw -q -DskipTests package FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=build /workspace/target/*.jar app.jar EXPOSE 8080 ENTRYPOINT ["java","-jar","/app/app.jar"] And here’s a Compose file that wires up Kafka and Schema Registry, plus an example Spring Boot service. The exact image choices are illustrative. Your production choices are unspecified and should reflect your standards and security posture. YAML # compose.yaml (local/dev) services: zookeeper: image: confluentinc/cp-zookeeper:7.6.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka:7.6.0 depends_on: [zookeeper] ports: ["9092:9092"] environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 schema-registry: image: confluentinc/cp-schema-registry:7.6.0 depends_on: [kafka] ports: ["8081:8081"] environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092 orders: build: ./orders-service depends_on: [kafka] ports: ["8080:8080"] environment: SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092 MESSAGING_ORDERS_TOPIC: orders.events MESSAGING_ORDERS_DLTTOPIC: orders.events.dlt SCHEMA_REGISTRY_URL: http://schema-registry:8081 Deploying on Kubernetes or AWS On AWS, the Kafka decision is usually managed or self-managed. If you choose Amazon MSK, the cluster lives in your VPC, pick subnets across distinct Availability Zones, and connect clients using the cluster’s bootstrap brokers. That’s the networking baseline, and it’s not optional. MSK is VPC-first by design. For authentication/authorization, MSK supports IAM access control. AWS documents the client configuration for IAM mechanisms. In EKS, I typically pair MSK IAM with IRSA so pods can obtain AWS credentials the AWS way, while ECS services would use task roles instead. Both patterns are documented by AWS, and your choice here is unspecified. Kubernetes service discovery is usually the easy part. Services and Pods get DNS names so workloads can call each other by name rather than IP. Kafka itself is reached via bootstrap broker endpoints or via internal Services, but either way, you want the strings in externalized config, not hardcoded. Here’s a minimal Kubernetes Deployment/Service for a Kafka client service. Values like region, account IDs, and MSK endpoints are unspecified placeholders. YAML apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: apps spec: replicas: 2 selector: matchLabels: { app: orders } template: metadata: labels: { app: orders } spec: serviceAccountName: orders-sa # IRSA-bound (role ARN unspecified) containers: - name: orders image: <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:<TAG> ports: [{ containerPort: 8080 }] env: - name: SPRING_KAFKA_BOOTSTRAP_SERVERS value: "<UNSPECIFIED_MSK_BOOTSTRAP_BROKERS>" - name: MESSAGING_ORDERS_TOPIC value: "orders.events" - name: MESSAGING_ORDERS_DLTTOPIC value: "orders.events.dlt" readinessProbe: httpGet: { path: /actuator/health/readiness, port: 8080 } initialDelaySeconds: 10 --- apiVersion: v1 kind: Service metadata: name: orders namespace: apps spec: selector: { app: orders } ports: - port: 80 targetPort: 8080 Operationally, MSK exposes metrics into CloudWatch (AWS/Kafka), and broker logs can be delivered to CloudWatch Logs (or S3/Firehose). That combination gives you the classic visibility loop: throughput, lag, under-replicated partitions, and error logs without running your own monitoring plane. For distributed tracing in async flows, OpenTelemetry is my default vocabulary now. Spring Boot supports OpenTelemetry export via OTLP, and OpenTelemetry defines Kafka semantic conventions so your producer/consumer spans and attributes stay consistent across tools. CI/CD and the Hard-Earned Field Notes For CI/CD, I keep it boring: build once, push an immutable image, deploy via a declarative mechanism. AWS Prescriptive Guidance provides a clear GitHub Actions pattern for building Docker images and pushing to Amazon ECR, which is a solid baseline when your region/account is unspecified until configured. YAML # .github/workflows/orders.yml name: orders on: push: branches: ["main"] jobs: build_push_deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: "21" - name: Build & test run: ./mvnw -q test package - name: Configure AWS credentials (OIDC) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<UNSPECIFIED_AWS_ACCOUNT_ID>:role/<UNSPECIFIED_GHA_ROLE> aws-region: <UNSPECIFIED_REGION> - name: Login to ECR run: | aws ecr get-login-password --region <UNSPECIFIED_REGION> \ | docker login --username AWS --password-stdin <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com - name: Build & push image run: | IMAGE=<UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:${{ github.sha } docker build -t $IMAGE ./orders-service docker push $IMAGE - name: Deploy to EKS (example) run: | aws eks update-kubeconfig --name <UNSPECIFIED_EKS_CLUSTER> --region <UNSPECIFIED_REGION> kubectl -n apps set image deploy/orders orders=$IMAGE Now, the part I wish someone had handed me in 2016: Kafka gives you strong tools, but it does not remove distributed-systems truths. You still need safeguards on the consumer side: idempotent processing, disciplined schema management, and clearly defined retry and dead-letter topic behavior. Kafka’s documentation is careful about the limits of “exactly once” guarantees. Idempotent producers and transactions can strengthen delivery semantics, but achieving true end-to-end exactly-once behavior, especially when external side effects are involved, still depends on deliberate system design. For schema governance, Kafka itself doesn’t ship a schema registry, but acknowledges third-party registries; in practice, Confluent Schema Registry and Apicurio Registry are common choices. Both store schemas out-of-band, so messages carry only a schema identifier, and both support evolvable contracts across Avro/JSON Schema/Protobuf depending on your ecosystem. Conclusion and Best Practices If you take one lesson from my legacy brain into modern event-driven systems, let it be this: asynchrony is a reliability feature, not a performance trick. Kafka’s durable log and consumer group model decouples uptime and absorbs spikes, but you only get the real benefit when you treat schemas as contracts, consumers as idempotent processors, and failure handling as first-class application behavior. On AWS, the operational baseline is non-negotiable. MSK lives in your VPC across AZ subnets, clients connect via bootstrap brokers, IAM auth is configured explicitly, and observability lives in CloudWatch. Do those fundamentals early, and Kafka stops feeling like a mysterious black box and starts feeling like the dependable workhorse it was built to be.

By Mallikharjuna Manepalli

Phantom APIs Are Eating Your Attack Surface, and Most Security Teams Are Still Looking the Other Way

I've spent the better part of fifteen years staring at API traffic logs for a living, and I can tell you the job has changed twice. The first shift came with microservices, when a handful of monolithic endpoints became thousands of small, chatty interfaces, and nobody could agree on who owned the inventory. The second shift is happening right now, and it's worse because this time the endpoints aren't even being written by people who can explain why they exist. Call them phantom APIs: routes, handlers, and parameters that show up in production but never appear in a spec, a ticket, or a design review. Some get hand-built by a developer in a hurry and are forgotten. Increasingly, though, they're a byproduct of AI code generation — Copilot, Cursor, an internal fine-tuned assistant, whatever your shop has standardized on — quietly scaffolding an admin route, a debug handler, or a permissive query path because that pattern showed up often enough in training data to feel "normal." Nobody asked for it. Nobody reviewed it with fresh eyes, because by the time a human glances at the diff, the suggestion already looks plausible. That's the part that should worry you more than any single CVE: plausibility, not malice, is now the main vector. How a Phantom Gets Born Here's the mechanism, stripped of drama. An engineer asks an AI assistant to "add an endpoint that lets support staff look up account status." The model, trained on millions of internal admin panels, often reaches for the path of least resistance: broad object access, no granular scope check, maybe a debug flag left wired to a query parameter "for testing." It compiles. It passes the smoke tests because the smoke tests check that the feature works, not that it's bounded. It ships. None of that shows up in your OpenAPI document because nobody updated the spec — the AI didn't know one existed, and the human reviewing the pull request was scanning for logic bugs, not authorization boundaries. Your API gateway, meanwhile, is busy enforcing policy on the routes it knows about. A path it has never seen just rides along on the same TLS termination and the same network ACLs as everything else, because from the network's point of view, there's nothing unusual happening. The gateway isn't broken. It's just answering a question nobody thought to ask it. I've heard versions of this story from engineers at a logistics platform, a healthcare billing vendor, and a fintech, all in the last year, none of whom wanted their names anywhere near a public postmortem — which is its own data point. Shame keeps these incidents quiet, and quiet incidents are exactly what let the pattern repeat across the industry instead of getting fixed once. The Numbers Stopped Being Theoretical in 2025 If you've been treating "API security" as a slide in next year's budget deck rather than this quarter's incident response calendar, the data from the past twelve months should change your mind. Wallarm's 2026 API ThreatStats Report, which pulled from 67,058 published vulnerabilities and 60 disclosed API breaches across 2025, found that API-related flaws made up 17% of all published vulnerabilities and 43% of the entries CISA added to its Known Exploited Vulnerabilities catalog that year. The technical profile of those flaws is the part that should keep API owners up at night: 97% exploitable with a single request, 99% remotely reachable, and 59% requiring no authentication at all. This isn't an attack surface that rewards patience and tradecraft. It rewards speed, and speed is exactly what AI tooling hands to attackers as readily as it hands to developers. That same report tracked AI-related vulnerabilities jumping from 439 in 2024 to 2,185 in 2025 — a 398% increase — with 315 of those tied specifically to Model Context Protocol implementations, the connective tissue between AI agents and the tools they're allowed to call. MCP didn't exist as a meaningful attack surface two years ago. Now it's 14% of all AI-related vulnerability disclosures in a single annual report. I don't think I've watched a category go from nonexistent to material that fast since the early days of container orchestration. IBM's X-Force Threat Intelligence Index 2026 adds the macro view: exploitation of public-facing applications became the single most common initial access vector in 2025, up 44% year over year, and 56% of the roughly 40,000 vulnerabilities X-Force tracked required no authentication to exploit. CybelAngel's own 2025 API threat reporting found that 95% of API attacks that year originated from sessions that were already authenticated — meaning the front door wasn't the problem; what happened after someone walked through it was. Put those two findings side by side, and you get a fairly bleak picture: getting in is easy, and once an attacker is in, the API layer rarely stops them from going sideways. And CrowdStrike's 2026 Global Threat Report puts a number on how little time defenders now have to notice. Average eCrime breakout time — the gap between initial access and lateral movement — fell to 29 minutes in 2025, down from 48 minutes the year before and 98 minutes in 2021. The fastest breakout CrowdStrike observed clocked in at 27 seconds. AI-enabled adversary operations rose 89% year over year, and the company recorded prompt-injection or AI-tool abuse incidents at more than 90 organizations. As Adam Meyers, CrowdStrike's head of counter adversary operations, put it when the report landed, breakout time is now the clearest signal of how intrusions have changed. A phantom API sitting outside your monitoring isn't a slow-burning liability anymore. It's a 27-second one. GraphQL Made This Worse, Not Better GraphQL was supposed to reduce shadow API risk by giving clients one well-documented entry point instead of dozens of REST routes. In practice, it concentrated the risk instead of eliminating it. Roughly 70% of organizations now run GraphQL in some form, according to Wallarm's Q2 2025 ThreatStats data, and the same report flagged something that should sound familiar to anyone who's done incident response: zero GraphQL-specific breaches were publicly disclosed that quarter, despite the technology's deep reach into production systems. That's not a sign GraphQL is safe. It's a sign almost nobody is looking closely enough to catch what's happening inside a single, deeply nested query that can touch a dozen resolvers and a dozen authorization decisions in one round trip. A REST endpoint that's missing an authorization check is one bug. A GraphQL resolver tree with the same gap can be a dozen bugs wearing one URL. Shadow and zombie APIs compound the problem from the other direction. Salt Security's 2025 CISO report found that only 19% of CISOs globally have full visibility into their API inventory — just 27% among large enterprises, and a thin 12% among smaller organizations — despite 73% ranking API security as a high or critical priority. Two-thirds of organizations audit for shadow APIs only monthly or quarterly, which leaves a four-to-twelve-week window every single cycle during which an undocumented route can sit there, fully reachable, before anyone goes looking. Salt Labs' own Q1 2025 data found that 99% of organizations had encountered an API security issue in the prior twelve months, and BOLA and injection flaws together accounted for more than a third of everything reported. None of this is exotic. It's the same handful of failure modes, recurring at a scale that AI-assisted development is now accelerating rather than fixing. The Failure Chain, Step by Step Strip away the vendor-report statistics for a second and walk through how this actually plays out on a single team, because the abstraction is where people lose the thread. A developer asks an AI assistant for a quick internal tool: pull account status for support staff, fast, no fuss. The assistant generates a working route, and because "working" was the only bar anyone set, it also generates a second, undocumented path the model added on its own initiative — a debug variant that accepts a raw account ID with no scope check, left over from however the model's training data tends to structure admin tooling. The pull request gets reviewed for logic, not for the existence of a route nobody asked for, because nobody is in the habit of reading a diff looking for endpoints that shouldn't exist. It merges. The OpenAPI spec doesn't change because nothing in the toolchain forces it to. The API gateway keeps doing its job — rate limiting, TLS, routing — on every path it's configured to recognize, and the new one simply isn't on that list, so it inherits whatever the underlying framework allows by default rather than anything the security team actually decided. For months, nothing happens because nobody is sending traffic to a path nobody knows about. Then someone does. Maybe it's a script kiddie running a wordlist against common admin paths, maybe it's a scraper, maybe it's one of the AI-driven reconnaissance tools the CrowdStrike and Wallarm data above describe as increasingly common. The request lands. There's no auth check to fail, so there's no log entry resembling a failed login — the kind of signal most SOC dashboards are tuned to catch. There's just a 200 response and a payload of account data. Given that CrowdStrike clocked the fastest 2025 breakout at 27 seconds and the average at 29 minutes, the gap between "endpoint found" and "data gone" is no longer a window anyone can rely on noticing in real time. By the time it surfaces — an anomaly report, a customer complaint, a researcher's disclosure email — the honest answer to "how long has this been exposed" is usually some shrug-worthy variant of "the logs only go back so far." That's the chain: AI suggestion → unreviewed scope gap → silent spec drift → gateway blind spot → silent exploitation → discovery after the fact. Every link in it is mundane. None of it requires a sophisticated attacker. That's exactly why it keeps happening. What I'd Actually Build to Catch It Description is cheap. Here's the shape of a pipeline I'd put in front of a team that wanted to stop shipping phantom routes instead of just talking about the risk: Plain Text CI/CD LAYER (pre-merge, blocking) → Generate live OpenAPI spec from the build → Diff against the last approved spec → Any new route not explicitly annotated/reviewed → FAIL build → Flag missing auth decorators, missing rate-limit config, wildcard scopes RUNTIME LAYER (continuous, post-deploy) → Traffic profiler sits behind the gateway, fingerprints every path actually receiving requests → Cross-reference live traffic against the approved spec, on a rolling window (hours, not quarters) → Anything serving 200s that isn't in the spec → page on-call, not a quarterly report GATEWAY LAYER (enforcement) → Default-deny for any path not present in the signed spec → Schema validation on request/response shape, not just route existence → Auth/scope check enforced at the gateway, independent of what the service itself does The CI step is the cheapest control here, and the one most teams skip, because it requires someone to decide that an undocumented route is a build failure, not a Slack message for later. The runtime layer catches what gets past CI anyway — config drift, routes added outside the normal deploy path, anything a human forgot to annotate. The gateway layer is the backstop: even if the first two fail, a default-deny policy means an unrecognized path doesn't get served at all, rather than getting served and merely logged. None of these three layers is sufficient alone. Together, they convert "we hope someone notices" into "the system refuses to let this happen quietly," which is the actual point. What Actually Works, and What's Mostly Marketing The vendor response has been predictably fast and not entirely cynical. Akamai's $450 million acquisition of Noname Security, announced in May 2024 and closed that June, folded one of the better-regarded API discovery platforms directly into a CDN-and-edge company's security stack — a clear bet that API visibility belongs as close to the traffic as possible, not bolted on afterward. Salt Security's 1H 2026 report introduced what it calls Agentic Security Posture Management, aimed squarely at mapping the relationships between LLMs, MCP servers, and the APIs underneath them, specifically to catch what the industry has started calling "Shadow MCP." Whether that label sticks or fades in eighteen months, the underlying instinct is correct: you cannot secure an API layer you can't continuously enumerate, and static documentation reviewed once a quarter is no longer a serious control. The defenses that actually move the needle, based on what I've watched, hold up under real incident response, aren't glamorous: Runtime discovery over documentation trust. Treat your OpenAPI spec as a claim to be verified against live traffic, not a source of truth. If traffic is hitting a path that isn't in the spec, that's an incident, not a documentation gap.Spec-diffing in CI, not just in security review. A pull request that introduces a new route should fail a build if that route doesn't appear in an updated, reviewed spec. This is cheap to automate and catches the AI-generated-endpoint problem at the exact moment it's introduced.Authorization checks that don't trust the session. Given that 95% of API attacks in CybelAngel's 2025 dataset started from an authenticated session, the perimeter check matters far less than the per-object, per-field authorization decision happening on every single call.AI-assisted review aimed at AI-generated code specifically. Ironically, the same pattern-matching that produces phantom endpoints can be turned around to flag them — diff-aware tooling that specifically interrogates new routes for missing rate limits, missing auth decorators, or unscoped data access, rather than general-purpose linting.Treat MCP and agent tool definitions as part of your API attack surface, full stop. They're not a side project. They're API endpoints with extra steps, and the ThreatStats data says they're already 14% of AI-related disclosures. None of these are silver bullets, and I'd be lying if I said any vendor has fully solved this. What I will say, after watching this category for a year now, is that the organizations doing well are the ones that stopped treating "shadow API discovery" as a once-a-quarter audit and started treating it as a property of the deployment pipeline itself — something that gets checked on every merge, the same way a linter or a test suite does. The ones still relying on a documentation review process built for a world where humans wrote every route are going to keep finding out about their phantom APIs the way most teams still do: during an incident, not before one. The question worth sitting with isn't whether your API inventory has gaps — every inventory does. It's whether you could currently produce, on demand, a complete list of every endpoint serving production traffic right now, including the ones nobody remembers approving. If the honest answer is no, you don't have an API security posture. You have an API security guess, and AI-generated code is making the guess bigger every sprint.

By Igboanugo David Ugochukwu

CORE

Connect Existing Data to AI Retrieval: How to Build Production-Ready Search Without Rebuilding Core Systems

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads. Most teams that want to add AI retrieval already have the data they need in databases, document stores, ticketing systems, and lakehouse tables that serve their purpose well. You usually do not need to centralize or rebuild this data; you can add retrieval as a thin layer over the systems you already run. This tutorial takes one workflow from start to finish, creating one artifact at each step. By the end, you will have a small program that runs, returns cited results, and that you can extend. All examples are tool-agnostic. If you would like to follow along as you read, a companion repository with the full runnable code, sample data, and tests is available at github.com/jubins/dzone-tr-ai-retrieval-starter. The figure above shows the full flow. Data moves from the source systems, through ingestion, normalization, and chunking, into the index or vector store, and out through the retrieval API to the application. Permissions, evaluation, and monitoring apply across the whole path. Step 1: Pick One Retrieval Workflow Worth Shipping The first step is to choose one workflow and define it clearly: one with a clear user need, known sources, output a person can review, and a result you can measure in a few weeks. Support knowledge retrieval, runbook search, and policy search are good examples. Write the scope down before any code. The workflow.yaml file below records it in one place: name, user need, output pattern, sources, success metrics, and reviewer. YAML # workflow.yaml workflow: support_knowledge_retrieval user_need: "Agents answer billing questions correctly and quickly" output_pattern: cited_answer # ranked_results | cited_answer | draft | context_packet sources: [help_center, ticket_history, billing_db] success_metrics: top1_relevance: ">= 0.85" citation_accuracy: ">= 0.95" median_latency_ms: "<= 800" escalation_reduction: ">= 15%" review: human_in_loop # who signs off on output quality Choose the output pattern with care because it sets how much accuracy and citation detail you need. Do not start with a broad, company-wide search that has no clear owner, since you cannot tell whether a result is correct. Step 2: Map Existing Data Sources and Boundaries Next, list every source the workflow will use. For each one, record the owner, how often it changes, any sensitive fields, the permission rule, and a handling decision. Keep the list in a file so your program and your team read the same information. The sources.yaml file below shows three sources for a support workflow with these fields filled in. YAML # sources.yaml - name: help_center owner: docs_team freshness: weekly sensitive_fields: [] access_rule: all_agents handling: index # index | live_fetch | metadata_only | summarize | exclude - name: ticket_history owner: support_ops freshness: near_real_time sensitive_fields: [pii_email, account_id] access_rule: role_scoped_by_queue handling: live_fetch - name: billing_db owner: finance freshness: real_time sensitive_fields: [card_last4, balance] access_rule: strict_per_account handling: exclude # never copy; live lookup only The handling decision is the most important field. You can index stable content, fetch volatile or sensitive data live, use some sources only as filters, or exclude a source. For permissions, either reuse the existing source rules at query time (simpler but slower) or build a separate access layer (faster but harder to keep in sync). As a rule, index stable, widely readable content and fetch live anything that changes often. This is the tradeoff between a central index and a per-domain fit. Table 1 below shows the inventory, one row per source. Table: Sample Source Inventory for a Support-Knowledge Workflow Data SourceOwnerFreshnessAccess RuleIndexing DecisionHelp-center articlesDocs teamWeeklyAll agentsIndex (keyword + vector)Ticket history (CRM)Support opsNear real timeRole-scoped by queueLive fetch + metadata filterProduct spec wikiEngineeringAd hocInternal onlyIndex; exclude restricted spacesBilling records (DB)FinanceReal timeStrict, per-accountExclude from index; live lookup only Step 3: Define the Retrieval Contract Before Indexing Before building the index, define the retrieval contract: the fixed interface between your application and everything behind it. Once fixed, you can change the chunking, embeddings, or storage layer without breaking the callers. The contracts.py file below shows the request and response types. The request carries the query, the user and role, filters, a freshness requirement, the result count, and the mode. The response carries a status and results, each with a source id, snippet, citation, timestamp, and score. Python # contracts.py from dataclasses import dataclass, field from enum import Enum class Status(str, Enum): OK = "ok"; NO_RESULTS = "no_results"; STALE = "stale"; DENIED = "denied" SOURCE_UNAVAILABLE = "source_unavailable" LOW_CONFIDENCE = "low_confidence"; CONFLICT = "conflict" @dataclass class RetrievalRequest: query: str user_id: str roles: list[str] filters: dict[str, str] = field(default_factory=dict) freshness: str | None = None # e.g. "<=24h" max_results: int = 5 # mode: ranked_results | cited_answer | context_packet mode: str = "cited_answer" @dataclass class Result: source_id: str; snippet: str; citation: str timestamp: str | None = None; score: float = 0.0 @dataclass class RetrievalResponse: status: Status results: list[Result] = field(default_factory=list) retrieval_version: str = "2026.06.1" Treat each failure state as a normal return value, not an error. The states no_results, stale, denied, source_unavailable, low_confidence, and conflict each need different handling. Add a retrieval_version field from the start so your logs show which version produced each result. The JSON below shows one request and one matching response, the exact shape your callers will work with. JSON // request and response (JSON) // request { "query": "refund a duplicate charge", "user": { "id": "u_812", "roles": ["agent"] }, "filters": { "product": "billing" }, "freshness": "<=24h", "max_results": 5, "mode": "cited_answer" } // response { "status": "ok", "results": [ { "source_id": "help_center/refunds#duplicate", "snippet": "...", "citation": "Help Center > Billing > Refunds", "timestamp": "2026-05-20T09:14:00Z", "score": 0.83 } ] } Step 4: Prepare Data Without Disrupting Source Systems Now, prepare the data as a read-only copy for search, without moving or changing the original data. You break the content into small pieces called chunks, each with a link back to its source. The chunking.py file below shows the Chunk type, which lists the fields a chunk needs: a stable id, a source reference, the text, an optional embedding, and metadata such as source, permissions, last update time, and content hash. Python # chunking.py @dataclass class Chunk: chunk_id: str # stable id: hash(source_ref + version + offset) source_ref: str # e.g. "kb/refunds#duplicate" text: str embedding: list[float] = field(default_factory=list) # metadata: source, permissions, updated_at, content_hash metadata: dict = field(default_factory=dict) def stable_id(source_ref: str, version: str, offset: int) -> str: raw = f"{source_ref}|{version}|{offset}".encode("utf-8") return hashlib.sha1(raw).hexdigest()[:16] Use a stable id so an update replaces the old chunk instead of creating a copy. A database row becomes one short chunk with labeled fields, while a long document is split into overlapping passages. A few hundred tokens with a small overlap usually works better than very small pieces, which lose context, or full documents, which reduce accuracy. The indexer.py file below shows the indexer, which reads a source, creates chunks with stable ids, stores them, and handles reprocessing and deletion. When a record is deleted or its access is removed, delete its chunks too. Python # indexer.py REPROCESS_EVENTS = {"source_update", "schema_change", "embedding_change", "chunking_change", "policy_change"} def index_source(index: Index, records: list[dict]) -> Index: for record in records: for chunk in chunk_record(record): # row->1 chunk; doc->passages index.upsert(chunk) # upsert by id: no duplicates return index def on_event(index, event, record=None, records=None): if event in REPROCESS_EVENTS and records is not None: index_source(index, records) # reindex affected scope elif event == "record_deleted" and record is not None: index.delete_by_source(record["id"]) # delete chunks + embeddings return index Step 5: Build the Indexing and Retrieval Path This step builds the main retrieval path. Write it as one function with a clear order: validate the request, apply permissions and filters first, fetch candidates, combine and rerank them, and return cited results. Applying permissions first keeps restricted content out of the candidate list. The figure above shows the same path as a single function. The request is validated, permissions are applied first, keyword and vector search run, their lists are fused and reranked, and a confidence and freshness check produces a cited result. Each stage can instead return a failure state, which the application handles. The retrieval.py file below shows the retrieve function, which runs this order: the permission filter, then keyword and vector search, fusion, reranking, and the confidence and freshness checks. Python # retrieval.py CONFIDENCE_MIN = 0.02 # set from evaluation data, not by guessing def retrieve(request: RetrievalRequest, index: Index) -> RetrievalResponse: if not request.query.strip(): return RetrievalResponse(status=Status.DENIED) # invalid query allowed = permission_filter(index.all_chunks(), request.roles) # first by_id = {c.chunk_id: c for c in allowed} kw = keyword_search(request.query, allowed) # exact terms, ids vec = vector_search(request.query, allowed) # meaning fused = rrf(kw, vec) # reciprocal rank fusion if not fused: return RetrievalResponse(status=Status.NO_RESULTS) fused = rerank_with_recency(fused, by_id) # old ranks lower top = fused[: request.max_results] if top[0][1] < CONFIDENCE_MIN: return RetrievalResponse(status=Status.LOW_CONFIDENCE) return RetrievalResponse( status=Status.OK, results=with_citations(top, by_id) ) Use hybrid retrieval, meaning keyword search and vector search together. Keyword search is good for exact terms such as ids and error codes, while vector search matches meaning. Fetch a wide candidate set, about 50 from each method, and keep only what you need. Set a freshness method per source, the tradeoff between freshness and latency, and set the confidence threshold from your evaluation data. The table below describes each stage: the decision, the way it can fail, and the evidence to record. Table: Retrieval Stages — Decisions, Failures, and Evidence StageDecisionFailure ModeEvidence CapturedRequest validationReject malformed or over-scoped queriesInvalid filter; missing identityQuery, filters, caller IDPermission filterApply role and source rules firstOver-broad access; leakRoles, rules applied, excluded sourcesCandidate retrievalHybrid keyword + vector, then fusedZero recall; missed exact termPattern used, candidate countRerank / scoreOrder by relevance; drop low scoresConfident but wrong rankingScores, threshold, cutoff appliedResponse assemblyAttach citations, timestamps, statusMissing or wrong citationSource IDs, timestamps, status Step 6: Evaluate Retrieval Quality Before Generating Answers Before letting a model generate answers, measure the quality of retrieval on its own. Build a small evaluation set of real questions, and for each one, record the source that should answer it. Include negative cases such as restricted or unanswerable questions. The eval_set.json file below shows three cases, each with the question, the expected source, the type, and the pass rule. JSON // eval_set.json [ { "q": "refund a duplicate charge", "expect": "help_center/refunds", "type": "conceptual", "pass": "relevant_cited_top3" }, { "q": "account 8821 balance", "expect": null, "type": "restricted", "pass": "status==denied" }, { "q": "end of life date for plan v2", "expect": "docs/plans#v2", "type": "outdated", "pass": "latest_version_only" } ] The evaluate function below runs each case through the retrieve function, checks it against the pass rule, and records the cause when a case fails, such as source quality, metadata, chunking, embedding, ranking, or permissions. Python # the evaluate() runner def evaluate(eval_set): report = [] for case in eval_set: res = retrieve(to_request(case)) ok = check(case.pass, res) # pass rule written as code cause = classify_failure(case, res) if not ok else None report.append({case, ok, cause}) return aggregate(report) # top1_relevance, citation_accuracy, ... Test the difficult cases: restricted records denied, old documents ranked lower, unclear short forms clarified, conflicting sources shown, and empty results reported honestly. Around 30–50 real questions is enough to begin and is cheap to re-run after any change. Use human review first, then add automatic checks. The table below lists the main query types and the expected behavior for each. Table: Evaluation Cases by Query Type Query TypeExpected SourcePass CriteriaFailure SignalExact ID lookupSpecific recordCorrect record in top 1Wrong or missing recordConceptual questionReference doc(s)Relevant passage in top 3, citedOff-topic or uncited resultRestricted queryNone (denied)Permission-denied returnedRestricted content exposedOutdated topicCurrent documentLatest version returnedStale version returnedUnanswerable queryNoneHonest no-answerFabricated answer Step 7: Connect Retrieval to the Application or AI Layer Now, connect the retrieval layer to the application or the model. The status field from the contract decides what the application should do. The app.py file below shows render_for_status, which maps each status to a behavior: ok shows the answer with citations, low_confidence shows only the search results, conflict shows the conflicting sources, and denied shows a safe message without revealing any titles or content. Python # app.py — render_for_status() VIEW = { Status.OK: "answer_with_citations", Status.LOW_CONFIDENCE: "results_only", Status.CONFLICT: "conflicting_sources", Status.STALE: "answer_with_freshness_warning", Status.NO_RESULTS: "no_reliable_answer", Status.DENIED: "access_denied", # no titles or snippets Status.SOURCE_UNAVAILABLE: "source_unavailable", } def render_for_status(response) -> dict: view = VIEW.get(response.status, "source_unavailable") if view in ("access_denied", "no_reliable_answer"): return {"view": view} # show nothing sensitive return {"view": view, "results": response.results} When a model writes the final answer, give it only the retrieved evidence and ask it to cite the source ids. The same app.py file builds the context packet, which has this structure: the question, the evidence with ids and citations, and an instruction to answer only from the evidence and to say so when it is not enough. Python // app.py — build_context_packet() def build_context_packet(query: str, response: RetrievalResponse) -> dict: return { "question": query, "evidence": [{"id": r.source_id, "text": r.snippet, "cite": r.citation} for r in response.results], "instruction": ("Answer only from the evidence; cite source ids; " "if the evidence is not enough, say so."), } Give users simple controls, such as opening the source, refining the query, or running a manual search. Use the same confidence threshold from Step 5 to decide between an answer and results only, and place citations next to the claims they support. Generating an answer over low-confidence retrieval costs more and can hide the uncertainty, the tradeoff between cost and quality. Step 8: Add Monitoring, Rollout Controls, and Production Guardrails The last step adds monitoring and controls for production. Record a log line for every retrieval so you can reconstruct any answer later. The monitoring.py file below shows log_retrieval, which records fields such as the query, filters, permissions applied, sources returned, retrieval version, status, latency, and top score, followed by useful dashboard signals. Python // monitoring.py DASHBOARD_SIGNALS = [ "latency_p95", "index_lag", "source_freshness", "empty_result_rate", "low_confidence_rate", "top_failing_queries", ] def log_retrieval(request, response, latency_ms: float) -> str: return json.dumps({ "query": request.query, "filters": request.filters, "user_roles": request.roles, "pattern": "hybrid", "source_ids": [r.source_id for r in response.results], "retrieval_version": response.retrieval_version, "status": response.status.value, "latency_ms": round(latency_ms, 1), "top_score": response.results[0].score if response.results else None, }) Control the rollout with feature flags that you can turn on for one team at a time, and keep switches that let you turn off an index, a source, the retrieval configuration, or the generated answers. The rollout.py file below shows the rollout settings: the flags, the audience, the kill switches, and the owner for each type of incident. Python // rollout.py @dataclass class Rollout: flags: dict = field(default_factory=lambda: { "generated_answers": "team_only", # off | team_only | on "hybrid_rerank": "on" }) audience: list[str] = field(default_factory=lambda: ["support_team"]) kill_switches: list[str] = field(default_factory=lambda: [ "index", "source:ticket_history", "retrieval_config", "generated_answers" ]) incident_owners: dict = field(default_factory=lambda: { "wrong_source": "search_eng", "unauthorized_exposure": "security", # page immediately "stale_answer": "data_eng", "reduced_quality": "search_eng" }) Name the incident types in advance, such as wrong source, stale answer, unauthorized exposure, missing data, and reduced quality, and give each an owner. Start with one team and watch the empty-result rate, the low-confidence rate, and the latency, widening the audience only when these stay stable. Many search problems come from data problems in the source, so send retrieval failures back to the data and governance teams. Put It Together: A Small Program You Can Run All the parts above fit into one small program. The retriever.py program below is plain Python and uses only the standard library, so it runs without installing anything. It keeps a few documents in memory, builds a keyword score and a vector score, combines them with reciprocal rank fusion, applies a permission filter, checks a confidence threshold, and prints cited results. Save it and run python retriever.py. Python // retriever.py (runnable, standard library only) import re, math from collections import Counter # A tiny in-memory index. Each document has an id, the roles allowed to read # it, and some text. In a real system this comes from your sources. DOCS = [ {"id":"kb/refunds#dup", "roles":["agent"], "text":"How to refund a duplicate charge to a customer."}, {"id":"kb/refunds#late", "roles":["agent"], "text":"Refund policy for late or delayed payments."}, {"id":"kb/billing#cycle","roles":["agent"], "text":"When the monthly billing cycle starts and ends."}, {"id":"kb/internal#sla", "roles":["admin"], "text":"Internal SLA targets for billing escalations."}, ] CONFIDENCE_MIN = 0.02 # set this from your evaluation data, not by guessing def tokens(t): return re.findall(r"[a-z0-9]+", t.lower()) def keyword_search(query, docs): # count of shared words q = set(tokens(query)) s = [(d["id"], len(q & set(tokens(d["text"])))) for d in docs] return sorted([x for x in s if x[1] > 0], key=lambda x: -x[1]) def cosine(a, b): c = set(a) & set(b) num = sum(a[t] * b[t] for t in c) da = math.sqrt(sum(v * v for v in a.values())) db = math.sqrt(sum(v * v for v in b.values())) return num / (da * db) if da and db else 0.0 def vector_search(query, docs): # cosine over word counts qv = Counter(tokens(query)) s = [(d["id"], cosine(qv, Counter(tokens(d["text"])))) for d in docs] return sorted([x for x in s if x[1] > 0], key=lambda x: -x[1]) def rrf(a, b, k=60): # reciprocal rank fusion scores = {} for ranked in (a, b): for rank, (doc_id, _) in enumerate(ranked): scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1) return sorted(scores.items(), key=lambda x: -x[1]) def retrieve(query, user_roles, max_results=3): allowed = [d for d in DOCS if set(d["roles"]) & set(user_roles)] # permissions first fused = rrf(keyword_search(query, allowed), vector_search(query, allowed)) if not fused: return {"status": "no_results", "results": []} top = fused[:max_results] if top[0][1] < CONFIDENCE_MIN: return {"status": "low_confidence", "results": top} by_id = {d["id"]: d for d in DOCS} results = [ { "source_id": i, "citation": i, "snippet": by_id[i]["text"], "score": round(s, 3), } for i, s in top ] return {"status": "ok", "results": results} if __name__ == "__main__": out = retrieve("refund a duplicate charge", user_roles=["agent"]) print("status:", out["status"]) for r in out["results"]: print(f" [{r['source_id']}] score={r['score']} -> {r['snippet']}") When you run it, the program prints status: ok with the matching documents and their scores, and does not return the admin-only document for an agent. The example is small on purpose. To grow it into a real system, replace the simple vector with a real embedding model, point the documents at your sources, move the index into a real keyword and vector store, and add the controls from Step 8. The structure stays the same. The full project, with each of these functions wired together, sample data, and tests, is in the companion code repository. Production-Ready Retrieval Without a Rebuild To summarize, you can add production retrieval without a rebuild: scope one workflow, list your sources, define a contract, prepare a read-only copy of the data, build one retrieve function, evaluate it, connect it to the application, and ship with monitoring and controls. Each step extends systems you already run, and the program above is a starting point you can expand for your own data. The references below are the canonical sources behind each technique used in this tutorial, mapped to the step where it appears. OpenAPI Initiative. (n.d.). OpenAPI specification (latest version). — Used in Step 3: Retrieval contract as a versioned API. Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389. — Used in Step 5: Keyword search and BM25 baseline. Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’09). — Used in Step 5: Fusing ranked lists from keyword and vector retrieval. Karpukhin, V., et al. (2020). Dense passage retrieval for open-domain question answering. arXiv. — Used in Steps 4–5: Dense retrieval foundation for the vector path. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). — Used in Step 4: Embeddings for indexing and vector retrieval. Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv. — Used in Step 7: Grounding generated answers in retrieved evidence. Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval-augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). — Used in Step 6: Evaluation approach for retrieval and generation. National Institute of Standards and Technology. (n.d.). SP 800-162: Guide to attribute based access control (ABAC) definition and considerations. — Used in Steps 2, 5, 8: Permissions and governance framing for access rules. This is an excerpt from DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads.Read the Free Report

By Jubin Abhishek Soni

CORE

I Built a VS Code Extension to Debug Azure AI Foundry Agents Without Leaving My Editor

The Problem Azure AI Foundry has a genuinely great portal. You can see your agent runs, the tools it calls, the messages it sends and receives, and even a breakdown of token usage — all in a clean UI. But here's what actually happens when you're building an agent locally: Write some code, trigger a runSwitch to the browser, open the Foundry portalNavigate to your project → your agent → Traces tabFind the right runClick through to see what happenedSwitch back to VS Code to make a fixRepeat That context switch sounds minor. But when you're iterating fast — tweaking a system prompt, adjusting tool call logic, debugging why an agent handed off to the wrong sub-agent — it adds up. You're constantly pulling your attention out of your editor and into the browser and back again. What I wanted was simple: see the trace right where I'm working. What Foundry Trace Inspector Does The extension connects to your Azure AI Foundry project and gives you three views for every agent run, all inside a VS Code panel: Trajectories: The Full Span Tree A Gantt-style collapsible tree showing the full execution: Session → Invoke Agent → Chat turns → Tool calls. Every span shows duration, token counts, and cost. Click any span to open a detail drawer with the model, status, token breakdown, and raw input/output. Duration Per-span timing bars — see exactly how long each step took. Tokens Input vs output token breakdown per span. This is the view I use most during debugging. At a glance, I can see: did the tool call happen? How long did it take? What did the LLM actually receive as input? User View: Readable Conversation Replay A chat-bubble timeline of the full conversation: user messages and assistant replies rendered the way a human reads them, with the agent name and model on each assistant turn. Each assistant bubble has a "View Trace" button that jumps directly to the corresponding response in the sidebar — so you can go from "something looked off in this reply" to the raw span in one click. Token and Cost Chart A stacked bar chart (input vs output tokens per LLM turn) so you can instantly spot which turns are burning the most tokens — useful when you're trying to understand why a multi-turn conversation is getting expensive. Per span cost breakdown for both input and output tokens consumed. How It Works Under the Hood Azure AI Foundry agents use the OpenAI Responses API internally. Every agent reply produces a resp_... response ID that's visible in the Foundry portal's Traces tab. The extension fetches those responses directly via the same API and reconstructs the full conversation timeline locally. When a session spans multiple turns, each response links to the previous one via previous_response_id. Load any response in the chain and the extension walks the chain automatically — you don't need to manually track down every ID. Conversation IDs (conv_...) are discovered automatically from your saved responses, so once you track one response, the whole conversation surfaces. No intermediate server. The extension makes API calls only to the Azure endpoint you configure. Your API key is stored in VS Code's encrypted SecretStorage — it never touches settings.json and never leaves your machine. Setting It Up You need two things: An Azure AI Foundry project endpoint URL (found in the Foundry portal under your project → Overview)Either an API key or Azure CLI auth (az login) via DefaultAzureCredential Once configured, grab a conv_... conversation ID from the portal's Traces tab, paste it into the sidebar, and the extension fetches all responses in that conversation automatically. What's Next A few things I want to add in v0.2: Auto-discovery of recent runs – instead of pasting IDs manually, list recent conversations directly from the panelSide-by-side diff – compare two runs of the same agent to see what changed between runsExport to Markdown – generate a readable trace report you can paste into a PR or incident note Further Reading What is Foundry Agent Service? – official overview of the service this extension connects toUse the Azure OpenAI Responses API – the underlying API the extension fetches trace data fromMicrosoft Foundry Pricing – understand what your agents actually cost to runVS Code Webview API – how the timeline panel is builtVS Code Extension API – full reference if you want to contribute or build on top of this

By Jubin Abhishek Soni

CORE

How to Set MX Records via API: Automate Email Routing Programmatically

Every domain you register for a user without setting MX records just creates broken email configurations. At five domains, it’s a minor annoyance. At five hundred, it’s a support backlog. At five thousand, it’s a full-time job. If your platform provisions domains for users (whether that’s a website builder, a multi-tenant SaaS, or a developer tool that provides domain-at-checkout), email routing belongs in your provisioning pipeline, executed immediately after domain registration, without any user involvement. This guide covers the complete implementation of MX records via API: how MX records work, what each field actually means, how to authenticate with the name.com API, and how to write the curl commands that create and verify MX records against the sandbox before you touch production. Why Manually Managing MX Records Doesn’t Scale When you don’t automate MX records, the failure mode is predictable: a user registers a domain through your platform, sets up their email with Google Workspace or Microsoft 365, and then waits. Email doesn’t arrive. They open a support ticket. Your team investigates. The problem is more than likely that nobody set the MX records. It’s an easy fix, but only if you’ve wired it into your pipeline. If the fix requires a human (your team or the user), it might get missed. At scale, “gets missed sometimes” and “breaks at scale” are the same thing. Fire off a [POST call to /core/v1/domains/{domainName}/records](https://docs.name.com/api/v1/reference/dns/create-record#create-record) immediately after your domain registration call returns successfully. One HTTP request, with a fixed payload containing your standard MX configuration, timed to run before the user ever sees the “domain registered” confirmation. No manual steps, no UI navigation, no user action required. MX Record Anatomy: What the Fields Actually Mean An MX record has three required fields that your API call needs to supply. These fields are pretty straightforward. There are also MX-only fields, like priority. Finally, TTL should be set according to how often you think the record might change. If it’s going to change frequently, you’ll want a lower TTL to lower propagation times. type: Always "MX" for mail exchanger records. Required.host: The hostname relative to the domain zone. For apex routing (mail to [email protected]), use an empty string "" or "@". Most platforms route email at the apex. Required.answer: The target for the MX record. Required.priority: An integer. Lower number = higher preference. DNS resolvers try the lowest number first.ttl: Time to live in seconds. Minimum 300 (5 minutes) on name.com’s API. A value of 300 to 3600 is reasonable for most setups. RFC 5321 (the spec that defines how SMTP works) explicitly states that MX records must point to a fully qualified domain name, not an IP address. If your email provider gives you an IP address rather than a hostname, don’t put it in the answer field. Create an A record pointing to that IP first (e.g., mail.yourdomain.com pointing to 203.0.113.42), then set your MX record’s answer to mail.yourdomain.com. Priority controls failover order. Set priority: 10 and priority: 20 on two different records, and resolvers will try the 10 server first, falling back to the 20 server only if the first is unreachable. Two records at the same priority value split traffic randomly between them, which suits some setups but isn’t what most people mean by “primary and backup.” Use distinct priority values if you want predictable failover. Authenticating With the name.com DNS API name.com uses HTTP Basic Authentication. Your credentials are your API username and a generated API token (your account password won’t work here). Generate a token at https://www.name.com/account/settings/api under API Tokens. You’ll get a username/token pair that every API call requires. Test in the sandbox first. The sandbox endpoint is https://api.dev.name.com. Your sandbox credentials differ slightly: append -test to your username (so yourname becomes yourname-test) and use the separate sandbox token shown on the same API Tokens page. The production endpoint is https://api.name.com, with your regular credentials. Store these as environment variables in your codebase from day one. There’s one gotcha with 2FA that’s easy to miss. If your name.com account has two-factor authentication enabled, you must explicitly toggle “name.com API Access” on at Account Settings → Security Settings. Without it, every API call returns an authentication error (HTTP Response 401), but not the exact reason why. If you prefer iterating on requests interactively before scripting, httpie or Postman both work well for testing individual calls. curl is what we’ll use here because it’s available practically everywhere and makes requests fully reproducible. Creating an MX Record: The Actual API Call The endpoint for programmatic MX record creation is POST https://api.dev.name.com/core/v1/domains/{domainName}/records. Replace {domainName} with the domain you’re targeting, for example yourdomain.com. Shell curl -u "yourusername-test:your-sandbox-token" \ -X POST \ -H "Content-Type: application/json" \ -d '{ "type": "MX", "host": "", "answer": "mail.yourmailprovider.com", "priority": 10, "ttl": 300 }' \ "https://api.dev.name.com/core/v1/domains/yourdomain.com/records" A successful response returns HTTP 200 with the created record object: JSON { "id": 12345678, "domainName": "yourdomain.com", "host": "", "fqdn": "yourdomain.com.", "type": "MX", "answer": "mail.yourmailprovider.com", "ttl": 300, "priority": 10 } A 401 means your credentials are wrong or the 2FA toggle mentioned above is misconfigured. A 404 on the domain means the domain isn’t registered under the account tied to your API credentials. Routing to Google Workspace looks slightly different because Google supplies specific MX hostnames with pre-defined priority values. The primary MX record call looks like this: Shell curl -u "yourusername-test:your-sandbox-token" \ -X POST \ -H "Content-Type: application/json" \ -d '{ "type": "MX", "host": "", "answer": "aspmx.l.google.com", "priority": 1, "ttl": 300 }' \ "https://api.dev.name.com/core/v1/domains/yourdomain.com/records" Use Google’s priority values verbatim (1, 5, 10, 20, 30) rather than values you invent. This more than likely applies to any managed provider. Their onboarding docs give you the exact hostnames and priorities, and those values reflect their infrastructure’s routing logic. You can verify the record was created with a GET request: Shell curl -u "yourusername-test:your-sandbox-token" \ "https://api.dev.name.com/core/v1/domains/yourdomain.com/records" The response includes all DNS records for the domain. Your new MX record should appear in the array: JSON { "records": [ { "id": 12345678, "domainName": "yourdomain.com", "host": "", "fqdn": "yourdomain.com.", "type": "MX", "answer": "aspmx.l.google.com", "ttl": 300, "priority": 1 } ] } Configuring Failover: Multiple MX Records With Priority A primary server plus two fallbacks means three API calls, one per record: Shell # Primary mail server — priority 10 curl -u "yourusername-test:your-sandbox-token" \ -X POST \ -H "Content-Type: application/json" \ -d '{ "type": "MX", "host": "", "answer": "mail-primary.yourmailprovider.com", "priority": 10, "ttl": 300 }' \ "https://api.dev.name.com/core/v1/domains/yourdomain.com/records" # Secondary — priority 20 curl -u "yourusername-test:your-sandbox-token" \ -X POST \ -H "Content-Type: application/json" \ -d '{ "type": "MX", "host": "", "answer": "mail-secondary.yourmailprovider.com", "priority": 20, "ttl": 300 }' \ "https://api.dev.name.com/core/v1/domains/yourdomain.com/records" # Tertiary — priority 30 curl -u "yourusername-test:your-sandbox-token" \ -X POST \ -H "Content-Type: application/json" \ -d '{ "type": "MX", "host": "", "answer": "mail-tertiary.yourmailprovider.com", "priority": 30, "ttl": 300 }' \ "https://api.dev.name.com/core/v1/domains/yourdomain.com/records" DNS resolvers traverse priority in ascending order. A sending mail server looks up your domain’s MX records, sorts them by priority value, and tries the lowest first. If priority: 10 times out or refuses the connection, it falls back to priority: 20, then priority: 30. This is standard SMTP failover behavior defined in RFC 5321. For Google Workspace, Microsoft 365, and most managed email providers, the full list of MX hostnames and required priority values appears during setup. Copy those values exactly. Consolidating to a single record or reassigning priorities will break their infrastructure’s routing logic. Wiring MX Record Creation into Your Domain Provisioning Pipeline After your domain registration API call returns a success response, fire your MX record creation calls immediately before returning control to the user. Store your standard MX payload as a configuration constant (provider FQDN, priority, and TTL) rather than hardcoding it inline per request. When you switch email providers, you change the relevant values in one place. Plain Text # pseudocode MX_RECORDS = [ { type: "MX", host: "", answer: "aspmx.l.google.com", priority: 1, ttl: 300 }, { type: "MX", host: "", answer: "alt1.aspmx.l.google.com", priority: 5, ttl: 300 }, { type: "MX", host: "", answer: "alt2.aspmx.l.google.com", priority: 10, ttl: 300 } ] # On successful domain registration: for record in MX_RECORDS: POST /core/v1/domains/{domainName}/records with record name.com’s API is built for platforms that need to embed domain registration and DNS management directly into their products, without redirecting users to a registrar dashboard. It follows the OpenAPI specification, which integrates cleanly with AI code generation tools and produces consistent, predictable results. To go live from here: Generate a sandbox token at https://www.name.com/account/settings/apiRun the POST /core/v1/domains/{domainName}/records curl command from Section 4 against an already-registered sandbox domainConfirm with the GET call that the record appears correctlySwitch your base URL from api.dev.name.com to api.name.com, update your credentials to the production token and standard username, and you’re live You can have all of this working in under an hour. Final Thoughts MX records belong in your domain provisioning pipeline. Manual checklists and user documentation will only cause you and your users headaches. The two mistakes most likely to break things quietly are pointing answer at an IP address instead of an FQDN, and missing the 2FA API access toggle. Both are easy to catch in the sandbox before they reach production. The full endpoint reference and API token generation are available at docs.name.com/docs/api-overview, with no subscription required to get started.

By Jakkie Koekemoer

Foxit MCP Server: Give AI Agents Direct Access to 30+ PDF Tools via Model Context Protocol

Wiring a document automation agent directly to REST endpoints forces you to repeat the same plumbing for every operation: push a file up, poll until the task finishes, pull the result down, catch failures, and juggle auth tokens across several services. With PDFs, that cycle runs again for each conversion, OCR pass, or merge in your pipeline. The Foxit PDF API MCP Server replaces all of that with 30+ tools an agent can invoke directly, while the MCP Server absorbs the upstream REST mechanics behind the scenes. This article walks through registering the server, the full tool catalog it advertises, how Foxit’s eSign and DocGen REST APIs carry the same agent session forward into signing and document generation, and a concrete four-step workflow you can reproduce with your own files. MCP Architecture in 90 Seconds The MCP specification splits responsibility across three roles. The Host is the LLM runtime, such as Claude Desktop, VS Code with GitHub Copilot, or Cursor, which owns the conversation and chooses when a tool should run. The Server is the capability provider, a process that publishes tools over the MCP protocol and runs them against an underlying service. Tools are the individual operations a server makes callable, each described by a JSON schema so the host knows what goes in and what comes out. Foxit sits on both ends of this picture. Foxit PDF Editor ships as an MCP Host, the first PDF application to take that role, reaching outward to external MCP servers such as Gmail or Salesforce so its built-in AI assistant can use those services. The Foxit PDF API MCP Server points the other way, publishing Foxit’s cloud PDF Services API as 30+ tools that any MCP Host can invoke. The operations the MCP Server surfaces span format conversion, content extraction, OCR, merge, split, compress, flatten, linearize, compare, watermark, form data import/export, security, and property inspection. Foxit’s eSign API and DocGen API sit outside the MCP Server as independent REST services, which means they never appear as MCP tools. An agent workflow can still call them within the same session, just through the agent’s own code-execution layer instead of the MCP protocol itself, a difference the eSign section unpacks fully. PDF processing belongs to the MCP tools; signing and template generation belong to code the agent executes. Prerequisites and Configuration Three things need to be in place before you register the server: A Foxit developer account to obtain a client_id and client_secret (the free plan at developer-api.foxit.com needs no credit card)Python 3.11+ alongside the uv package manager, or Node.js 18+ with pnpm if you prefer the TypeScript versionAny MCP-compatible host, such as Claude Desktop, VS Code, or Cursor Grab the repo from github.com/foxitsoftware/foxit-pdf-api-mcp-server and add it to your host’s MCP configuration. Claude Desktop is the host used in the walkthrough below, but the identical command, args, and env values carry over to any MCP host. In Claude Desktop, open Settings, switch to the Developer tab, and choose Edit Config. Next, open claude_desktop_config.json in any text editor. The file lives at ~/Library/Application Support/Claude/ on macOS or %APPDATA%\Claude\ on Windows. Register the Foxit server beneath the mcpServers key: JSON { "mcpServers": { "foxit-pdf": { "command": "uv", "args": [ "--directory", "/path/to/foxit-pdf-api-mcp-server", "run", "foxit-pdf-api-mcp-server" ], "env": { "FOXIT_CLOUD_API_HOST": "https://na1.fusion.foxit.com/pdf-services", "FOXIT_CLOUD_API_CLIENT_ID": "your_client_id", "FOXIT_CLOUD_API_CLIENT_SECRET": "your_client_secret" } } } } Define FOXIT_CLOUD_API_CLIENT_ID and FOXIT_CLOUD_API_CLIENT_SECRET as system environment variables before the host process starts. Feeding credentials in through prompt context is a security exposure that any production setup should close off. The client_id and client_secret from your developer portal cover authentication for every MCP tool call against the PDF Services API. Bringing eSign into the same agent session means performing its own OAuth2 token exchange (detailed in the next section), so the two credential scopes never mix. Once you save the file, quit Claude Desktop entirely and relaunch it. On startup, it reads the config and spawns the server as a local subprocess communicating over standard input and output, which is the transport the Foxit server speaks. After the restart, the Foxit MCP server should appear as Running under local MCP servers in the Developer tab. Head to the Customize tab, open Connectors, and click foxit-pdf to inspect the tools the Foxit MCP server provides; the full set of 30+ registered tools should be listed there. If the connector never appears, the server failed to launch. Claude’s logs at ~/Library/Logs/Claude/mcp*.log usually reveal why, most often a missing uv binary or an incorrect --directory path. Invoking a tool is as simple as typing a natural-language request like “Convert this Word file to PDF and compress it.” The agent picks pdf_from_word and pdf_compress, and before each call executes, Claude Desktop displays an approval prompt listing the exact tool name and arguments; the tool’s JSON result then streams back into the chat. That per-call approval doubles as your audit point, because it shows precisely which tool the agent selected and the arguments it supplied. To run the server in VS Code instead, place the equivalent entry in .vscode/mcp.json under a top-level servers key, adding a "type": "stdio" field, so VS Code launches the process the same way: JSON { "servers": { "foxit-pdf": { "type": "stdio", "command": "uv", "args": [ "--directory", "/path/to/foxit-pdf-api-mcp-server", "run", "foxit-pdf-api-mcp-server" ], "env": { "FOXIT_CLOUD_API_HOST": "https://na1.fusion.foxit.com/pdf-services", "FOXIT_CLOUD_API_CLIENT_ID": "your_client_id", "FOXIT_CLOUD_API_CLIENT_SECRET": "your_client_secret" } } } } An alternative path is running MCP: Add Server from the Command Palette (Cmd+Shift+P or Ctrl+Shift+P), selecting Command (stdio), then choosing Workspace to store the entry in .vscode/mcp.json or Global to keep it in your user profile. After saving, VS Code displays inline Start, Stop, and Restart actions above the server entry and adds it to the MCP SERVERS - INSTALLED view, where a green indicator and the discovered tool count confirm everything is connected. PDF Services MCP Tools: Full Catalog The 30+ tools fall into seven functional categories. Nearly all of them expect a documentId produced by an earlier upload_document call and hand back a resultDocumentId you can feed to download_document whenever you need the output on disk. The one exception is pdf_from_url, which takes a URL directly. Document Lifecycle upload_document: push a PDF, Office file, image, HTML file, or plain text file to the cloud; returns a documentId used by every later operationdownload_document: pull a processed result down to a local file pathdelete_document: remove stored files from cloud storage when you are done with them PDF Creation (File to PDF) pdf_from_word, pdf_from_excel, pdf_from_ppt: turn Office documents into PDFspdf_from_text, pdf_from_image, pdf_from_html: turn plaintext, image files, or HTML into PDFspdf_from_url: fetch a live URL and render the page as a PDF PDF Conversion (PDF to File) pdf_to_word, pdf_to_excel, pdf_to_ppt: recover editable Office formats from a PDFpdf_to_text, pdf_to_html, pdf_to_image: produce text, HTML, or image representations Manipulation pdf_merge: join multiple PDFs into a single filepdf_split: divide a PDF by page ranges, page count, or one file per pagepdf_extract: lift a subset of pages out of a PDFpdf_compress: shrink file size by 30-70% depending on content typepdf_flatten: bake form fields and annotations into static content (a requirement for compliance archiving workflows)pdf_linearize: prepare a file for Fast Web View so browsers can stream pages as they loadpdf_watermark: stamp text or image watermarks with configurable position, opacity, and rotationpdf_manipulate: rotate, delete, or rearrange pages Analysis pdf_compare: diff two PDFs and produce a color-coded annotation document highlighting the changespdf_ocr: turn scanned or image-based PDFs into searchable text, with multi-language supportpdf_structural_analysis: detect document structure (titles, headings, paragraphs, tables with cell grids, images, form fields, hyperlinks, and metadata) with bounding boxes, following the Foxit PDF structural extraction engine schema. The output is JSON delivered inside a downloadable ZIP rather than a set of named business entities; it describes layout and structure only, and converting that into fields such as party names falls to the agent’s LLM, which performs the semantic extraction over the JSON Security and Forms pdf_protect: lock a document with password protection using 128-bit or 256-bit AES encryption plus granular permission flagspdf_remove_password: lift password protection off a documentexport_pdf_form_data: read form field values out as JSONimport_pdf_form_data: fill form fields from a JSON payload Properties get_pdf_properties: report page count, page dimensions, PDF version, encryption status, digital signature info, embedded files, font inventory, and document metadata In production document pipelines, the operation that gets called most is pdf_from_word. The agent uploads a DOCX, receives a documentId, then invokes pdf_from_word with that ID. Under the hood the PDF Services API performs the conversion asynchronously, but the MCP Server takes care of polling internally and hands the finished result straight back to the agent. MCP tool call: JSON { "name": "pdf_from_word", "input": { "documentId": "doc_abc123" } } MCP tool response: JSON { "success": true, "taskId": "task_xyz789", "resultDocumentId": "doc_result456", "message": "Word document converted to PDF successfully. Download using documentId: doc_result456" } From here, hand doc_result456 to download_document to save the PDF locally, or pipe it straight into the next tool in a chain, such as pdf_structural_analysis or pdf_compress. Extending to eSign: Foxit’s Signing API as a Complementary REST Layer Once the MCP tools finish PDF processing, the workflow’s next stage sends a document out for signature through Foxit’s eSign REST API, hosted at https://na1.foxitesign.foxit.com. Everything in this guide targets the na1 (US) region. Foxit also runs regional eSign hosts for the EU (eu1.foxitesign.foxit.com), Canada (na2.foxitesign.foxit.com), and Australia (au1.foxitesign.foxit.com). Payloads and endpoints stay identical across regions; only the host differs, so select whichever host satisfies your data residency requirements. The eSign API lives outside the Foxit MCP Server, so it is not an MCP tool, and that detail shapes how the agent gets to it. Most MCP hosts have no ability to fire arbitrary HTTP requests themselves, which means eSign is never reached “through MCP.” The agent instead calls eSign from its own code-execution layer, whether that takes the form of a host-provided code interpreter, an agent framework executing Python, or a custom tool you register that wraps the eSign endpoints. The cleanest pattern for production is wrapping the eSign operations you need as custom MCP tools so the host invokes them exactly as it invokes the PDF tools; the production considerations section comes back to this. The code below is what runs inside that layer. Authentication relies on OAuth2 client_credentials. This eSign token exchange is a separate flow from the PDF Services header auth that powers your MCP tools: Python import requests resp = requests.post( "https://na1.foxitesign.foxit.com/api/oauth2/access_token", data={ "client_id": ESIGN_CLIENT_ID, "client_secret": ESIGN_CLIENT_SECRET, "grant_type": "client_credentials", "scope": "read-write" } ) access_token = resp.json()["access_token"] “Folder” is the term the Foxit eSign API developer guide uses throughout its documentation. An automated signing flow centers on these endpoints: POST /api/folders/createfolder: build a signing folder from one or more PDF documents, including signers, subject, and messagePOST /api/folders/sendDraftFolder: send a draft folder out to its signersPOST /api/templates/createtemplate: store a reusable template from a PDF with pre-placed signature fields (later instantiate a folder from it via POST /api/templates/createFolder)GET /api/folders/viewActivityHistory?folderId={id}: fetch the activity audit trail for a folder after it has been sent (a draft that was never shared returns an error)Webhook channels for status callbacks: register a callback URL to get real-time events whenever signers view, sign, or decline A createfolder call accepts the PDF produced by your MCP pipeline, uploaded into eSign’s document storage after download_document fetches it, and configures the signing workflow: POST /api/folders/createfolder Authorization: Bearer {access_token} Content-Type: application/json JSON { "folderName": "Acme Corp Contract - Q3 2025", "sendNow": false, "fileUrls": ["https://your-storage.example.com/acme_contract_final.pdf"], "fileNames": ["acme_contract_final.pdf"], "parties": [ { "firstName": "John", "lastName": "Smith", "emailId": "[email protected]", "permission": "FILL_FIELDS_AND_SIGN", "sequence": 1 } ] } With sendNow at false, the call creates a draft folder you dispatch later through a separate request to /api/folders/sendDraftFolder. Setting sendNow to true instead creates and sends in one step. When a file cannot be reached by URL, include "inputType": "base64" and supply the documents as a base64FileString array in place of fileUrls; leaving out inputType causes the API to reject the base64 payload as empty. Foxit’s eSign API comes with HIPAA, eIDAS, ESIGN Act, UETA, 21 CFR Part 11, FERPA, and FINRA compliance built in. Each audit trail record captures signer location, IP address, recipient identity, event timestamp, consent confirmation, security level, and the complete folder history. If legal defensibility matters in your regulated industry, persist those fields in your own data layer as well, since depending entirely on Foxit’s folder history API for compliance record-keeping leaves a single point of failure in your audit chain. End-to-End Workflow: AI Agent Automates a Sales Contract Imagine a sales ops agent handed one natural language goal, “Generate a contract for Acme Corp, $48,000 ARR, and send it for signature.” No part of the tool sequence is hard-coded. Because the MCP Server advertises its PDF tools to the host at connection time, the agent can interpret the goal, recognize it has a template to render and a document to route for signature, and choose which operations to run and in what order. The PDF steps execute as MCP tool calls, while the DocGen and eSign steps execute from the agent’s code layer. The sequence shown below is one plausible run the agent could produce, not a fixed script assembled ahead of time. The agent starts with MCP tools to get a PDF in hand. It uploads the DOCX contract template through upload_document, gets documentId: "doc_abc" back, and runs pdf_from_word. The MCP Server manages the async conversion internally and reports resultDocumentId: "doc_pdf" when the job finishes. To understand what the PDF contains, the agent runs pdf_structural_analysis against documentId: "doc_pdf". The tool never returns named entities such as “party” or “ARR.” What comes back is a resultDocumentId pointing at a ZIP archive, so the agent fetches it with download_document, unpacks it, and reads the structural JSON describing headings, paragraphs, and table cells along with their positions. Semantic extraction is the job of the agent’s LLM, which reads that structural JSON and lifts “Acme Corp” from a heading or a contract value from a table cell, verifying the fields it needs exist. Structure comes from the tool; meaning comes from the model. If you would rather have an API return business entities directly instead of relying on the model to interpret layout, that capability belongs to Foxit’s iDox.ai Document API, a separate service purpose-built for entity and PII extraction. Holding the field values, the agent produces the finished contract via the DocGen API, posting to /document-generation/api/GenerateDocumentBase64 so the values merge into the template through {{dynamic_tags} syntax. Because DocGen is synchronous, the finalized PDF arrives in the response body with Acme Corp’s name, the $48,000 ARR figure, and the right dates filled in. There is no polling step. The last move is routing the document for signature. The agent authenticates against the eSign OAuth2 endpoint, uploads the DocGen output, builds a signing folder through /api/folders/createfolder with [email protected] as the signer, and sends it via /api/folders/sendDraftFolder. The thread running through all of this is that the model derives the order from the goal rather than following a script. PDF steps resolve to MCP tool calls the host already knows about, while DocGen and eSign steps pass through the agent’s code layer because those APIs are not MCP tools. Each step’s output feeds the next step’s input, and the only orchestration left for you to maintain is whatever exposes that code layer to the model, ideally a set of custom tools rather than ad hoc scripting. Production Considerations: Error Handling, Rate Limits, and Data Governance Calling PDF Services through the MCP Server means async polling stays inside the server process, and your agent only ever sees the final resultDocumentId once the task completes. Calling the raw PDF Services REST API directly is different, since every operation hands back a taskId you must poll yourself. The pattern below uses exponential backoff capped at 10 seconds per interval with a 30-second overall timeout: Python import time, requests API_HOST = "https://na1.fusion.foxit.com/pdf-services" auth_headers = { "client_id": "your_client_id", "client_secret": "your_client_secret" } def poll_task(task_id: str, max_wait: int = 30) -> str: delay = 1 elapsed = 0 while elapsed < max_wait: resp = requests.get( f"{API_HOST}/api/tasks/{task_id}", headers=auth_headers ) data = resp.json() if data["status"] == "COMPLETED": return data["resultDocumentId"] time.sleep(delay) elapsed += delay delay = min(delay * 2, 10) raise TimeoutError(f"Task {task_id} timed out after {max_wait}s") Since eSign and DocGen are not MCP tools, be deliberate about how the agent reaches them. Allowing the model to emit raw HTTP from a free-form code interpreter is fragile and difficult to audit. The sturdier approach is wrapping the specific eSign and DocGen operations you actually use, such as create-folder, send-folder, and generate-document, as custom MCP tools with typed inputs. The host then invokes them over the same protocol it uses for the PDF tools, credentials remain inside the tool process instead of the prompt, and the agent’s decisions surface as inspectable tool calls rather than opaque scripts. The output of pdf_structural_analysis warrants a caution of its own. For a long contract, the structural JSON can contain many thousands of elements, and pushing the whole file into the model can silently exceed its context window, a failure that usually shows up as truncated or confused extraction instead of a clean error. The code that unzips the archive should filter the JSON before the model ever sees it, retaining only the element types and pages that matter (for a contract, typically the heading blocks and the relevant table) instead of forwarding the entire document. The free developer plan at developer-api.foxit.com is sized for development and testing volumes. Production workloads beyond the free-tier threshold call for a volume plan requested through the Developer Portal. On the data governance side, every API call travels over TLS 1.2+, and documents at rest are protected with AES-256 encryption. Foxit’s API security documentation details SOC 2 Type II audit status, HIPAA BAA support, GDPR, CCPA, eIDAS, ESIGN Act, UETA, 21 CFR Part 11, FERPA, and FINRA requirements. Customer data is kept in logically segmented environments. Teams in healthcare, legal, or financial services should confirm data residency requirements before wiring up production document flows, then pick the matching regional eSign host described earlier, because the host you call determines where the data gets processed. Run Your First Tool Call Now A working MCP tool call is under 15 minutes away: Sign up for a free developer account at developer-api.foxit.com (no credit card, instant access), then copy your client_id and client_secret from the dashboard.Set the three environment variables: Shell export FOXIT_CLOUD_API_HOST="https://na1.fusion.foxit.com/pdf-services" export FOXIT_CLOUD_API_CLIENT_ID="your_client_id" export FOXIT_CLOUD_API_CLIENT_SECRET="your_client_secret" Clone the repo, register it with the config block from the Prerequisites section, restart your MCP host, and call pdf_from_url against any public URL. A confirmed PDF lands in your working directory. The Developer Portal also offers a live API Playground where you can validate request payloads against the PDF Services API before connecting them to an agent. To extend toward a full signing workflow, the smallest useful addition on top of the MCP setup is authenticating against the eSign OAuth2 endpoint and posting a static PDF to /api/folders/createfolder. From there, DocGen field population, pdf_structural_analysis extraction, and webhook callbacks build on the same pattern step by step. Claim your free API access at developer-api.foxit.com.

By Lucien Chemaly

When Valid SQL Was Still the Wrong Answer

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads. I started working on a personal project with a simple question: If AI can analyze a database schema and generate SQL, what still makes the answer hard to trust? The first version of my prototype worked at a surface level. A user could ask a business question, and the system would retrieve the relevant schema, generate SQL, run the query against a sample analytics database, and return a result. Technically, that felt like progress. But then I tested a question like, What is monthly revenue? The SQL ran. The database returned an answer. Still, I could not say the answer was truly correct because the meaning of revenue was not clear enough. After that test, I stopped treating the prototype as a text-to-SQL demo. I started handling it as an experiment in the database context that an AI assistant needs: metric definitions, semantic retrieval, validation rules, and governance signals. The semantic registry and validation layer help move the prototype from raw text-to-SQL toward governed, context-aware analytics. The Problem: Context Is Not the Same as Understanding The model could write SQL; the problem was that SQL execution alone did not prove the answer was right. In my database, monthly revenue could mean net revenue after refunds, gross order value, or paid revenue. Active customer could mean a customer with a login, a purchase, or both. Even if the model retrieved the right tables, it still needed to understand which business definition to use. The main failure signals were: Metric names that had more than one possible meaningMultiple date columns that could change the answerJoins that were technically possible but not analytically safeMissing filters like date range, status, or regionQuestions that needed clarification before execution The Constraints: Keep It Small, But Make It Reliable Because this was a personal project, I was not trying to build a full BI platform or enterprise data catalog. I wanted to focus on one narrow and realistic piece: how to make an AI-generated database answer more trustworthy before it reaches the user. Table: Prototype Boundaries ConstraintDesign ResponseSmall project scopeFocused on a few high-risk metricsAmbiguous business termsCreated explicit metric contractsThe schema alone was not enoughAdded semantic retrieval over definitionsNo analyst review stepAdded validation before SQL executionSimple user experienceUsed clarification instead of exposing schema complexity The hard part was keeping the prototype lightweight without making it too shallow. I wanted enough structure to make the answers safer, but not so much complexity that the project turned into a full governance platform. The Tradeoffs: What I Changed in the Pipeline The first major decision was to add a lightweight semantic registry. Each metric contract included the metric name, definition, grain, default date column, required filters, safe dimensions, and approved join path. YAML metric: net_revenue definition: paid_amount - refunded_amount grain: month date_column: payment_date required_filters: - payment_status = 'completed' clarify_if_missing: - date_range I almost relied only on schema retrieval because it was easier to build. But the schema only tells the model what exists. It does not tell the model what is correct for a specific business question. The second decision was to retrieve both schema metadata and metric definitions. This made the retrieval step more useful because the model was both matching a question to tables and grounding the answer in business meaning. The third decision was to validate SQL before execution. The generated query had to pass checks for allowed tables, approved joins, required filters, and selected dimensions. If it failed, the system either regenerated the query with stronger constraints or asked the user a clarification question. Table: Decisions, Tradeoffs, and Outcomes ChoiceTradeoffOutcomeAdd metric contractsMore setup workClearer business meaningRetrieve semantic context, not just schemaMore retrieval complexityBetter groundingValidate before executionSlightly slower responseFewer misleading answers What Changed The biggest lesson was that query execution is not the same as analytical correctness. A query can run successfully and still answer the wrong question. The prototype became more reliable when I stopped treating the database as just tables and columns. The model needed more context: what the metric means, which joins are safe, which filters are required, and when it should ask the user for clarification. My main takeaway was practical: Before tuning prompts, define the meaning layer the prompt is expected to respect. For intelligent data systems, the interesting work is not only faster retrieval or cleaner SQL. It is the connection between data, definitions, governance rules, and answers people can trust. This is an excerpt from DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads.Read the Free Report

By Anusha Kovi

CORE

Keeping AI-Powered BI Honest: A Human-in-the-Loop (HITL) Playbook

A few months ago, I led a BI project with a deceptively simple pitch: let business users ask questions in plain English, and hand back the answer. We wired an LLM to our warehouse, got SQL generation working, and ran a pilot. It did not go well. The model was actually right a lot of the time, and that wasn’t the problem. The problem was that nobody on the business side could tell when it was right. Prompts came in tangled; the model would interpret one clause subtly wrong, and we’d return a clean-looking number sitting on top of a clean-looking SQL query. The users couldn’t read the SQL. When we tried to surface the model’s reasoning, it was a wall of CTEs and join keys that helped no one. We had humans in the loop. Reviewers, too. The catch is they weren’t operating as a loop; they were operating as a relay. They’d glance at the SQL, agree it looked plausible, and forward the answer along. The user nodded. Two weeks later, finance would surface a number that didn’t reconcile, and by the time we traced it back, the decision had already been made. That was the painful version of the lesson I want to share. HITL is not a checkbox between the model and production. It’s a translation layer. The model produces SQL and rows; the user needs an answer they trust. A human has to do the work of turning one into the other, and the system around that human has to make the work possible. Show a reviewer raw SQL plus a confidence score, and you’ve built a relay, not a loop. Below is the playbook I wish I’d had on day one. 1. Confidence Threshold Routing Score every generated query before it runs. Self-consistency sampling is the cleanest version: generate 5 candidate SQL statements for the same prompt and check how many agree on the join logic. If 4 out of 5 join to dim_employee and one joins to dim_customer, your agreement ratio is 0.8. If your threshold is 0.85, that query gets routed to review even though it looks correct on the surface. Agreement across multiple generations is a stronger signal than any single model’s confidence score, which is famously well-calibrated for the wrong things. Aggregated log-probabilities are another option. The choice matters less than the discipline: anything below the threshold goes into a queue, never straight to execution. The threshold itself becomes a tunable lever you tighten over time as you learn which query patterns deserve more scrutiny. 2. Staged Execution With Approval Gates Confidence on its own isn’t enough for high-stakes domains. Define a list of high-impact tables such as revenue facts, employee dimensions, compliance event logs, and require human approval for any query that touches them, regardless of confidence score. The model might be certain. The business context still demands validation. In practice, the table list is governance work, not engineering work. The data team should own it, finance and HR should ratify it, and you should revisit it the same way you revisit access controls. If you let engineering pick the list alone, the list will be wrong, and nobody outside engineering will know it’s wrong until it’s too late. 3. Reviewer Tooling: The Part Everyone Underinvests In This is where my pilot fell over. Showing a non-technical reviewer raw SQL and asking “looks good?” is worse than no review at all, as it produces fake assurance. Reviewer tooling has to bridge SQL and business context. On a single screen, the reviewer needs the original natural-language prompt, the semantic-model entities the query touches (measures and dimensions, not table names), the filters being applied in human terms, and the expected shape of the result. The reviewer’s job is to validate intent, not parse syntax. Build the interface around that. If your reviewers are reading SQL out loud in their heads to figure out what a query does, you’ve shipped a relay. 4. Audit-Linked Approval Records Every reviewer decision to approve, reject, or edit has to write back to an audit log alongside the original prompt, the generated SQL, the reviewer’s identity and a timestamp. That log is the dataset you’ll need months later. It’s how you explain a number when finance comes asking. It’s how you recalibrate thresholds based on what actually shipped versus what got bounced. It’s how you find the query patterns that consistently trip up the model. Skip this step and the program loses its memory. You keep paying the human-review cost without ever compounding the learnings, which is the worst of both worlds. 5. Escalation Paths Reviewers will get stuck. They’ll sense a query is doing something odd without being able to articulate why, especially when it crosses domain boundaries. Give them a one-click route to a domain expert such as a finance lead, HR ops, or compliance, along with their concern, without freezing the user who originally submitted the query. The whole point is to prevent reluctant approvals. A reviewer who isn’t sure should never feel pressured to sign off because they have no other option. In my pilot, “no other option” was the silent failure mode. Reviewers approved because rejecting felt rude, and the loop swallowed the doubt instead of routing it. 6. HITL Bypass Logging When a query clears the confidence threshold and isn’t flagged as high-impact, it just runs. Log that bypass anyway with the score that justified it, the prompt, and the SQL. This is the data that surfaces threshold drift, model regressions, and good training examples for the next iteration. It also closes the audit gap between “approved by a human” and “approved by silence”. Without it, you can’t tell the two apart, which means you can’t defend either. Wrapping Up Shipping AI-generated SQL straight to production is reckless. The model will be wrong, and it will be wrong in ways that look right. A single bad number in a board deck can outlive whoever wrote the prompt. HITL isn’t a nice-to-have here. It’s the only thing standing between a useful BI assistant and a very fast way to make confident, well-formatted and completely wrong decisions. The lesson from my pilot wasn’t that humans should validate SQL. It’s that humans have to translate. The model speaks in joins; the business speaks in outcomes. Build the loop so the people in the middle have a real chance of bridging the two, like tooling that surfaces intent, processes that protect them from reluctant sign-off, audit trails that turn every decision into future training data. Do that, and you get a BI assistant that’s actually trusted. Skip it, and you get a relay that breaks quietly until it doesn’t. Key Takeaways Confidence threshold routing using self-consistency sampling catches semantic errors that a model’s own confidence scores miss. Generate multiple candidates and measure agreement.Staged execution with approval gates protects high-stakes queries such as revenue, headcount, and compliance regardless of model confidence.Reviewer tooling has to bridge SQL and business context. Show the prompt, the semantic entities, and the expected output shape, and never just the raw query.Audit-linked approval records are the dataset you’ll need to recalibrate thresholds and explain numbers when finance comes asking months later.Escalation paths prevent reluctant approvals. Make it one click to route to a domain expert.HITL bypass logging turns silent successes into a feedback loop and closes the gap between “approved by a human” and “approved by silence.”

By Nithish Shetty

Automating Power Automate: How to Ensure Cloud Flows Are Active After Every Pipeline Deployment

You've spent hours — maybe days — building and testing a Dynamics 365 Power Platform solution. Your Azure DevOps pipeline runs clean. The managed solution imports successfully into the target environment. All green. Then the business calls. Nothing is working. The automations aren't firing. You log into Power Automate in the target environment and find the same scene every time: every single cloud flow is turned off. Not broken. Not errored. Just off. And every connection reference is sitting there unresolved, pointing at nothing, waiting for someone to manually wire it up. If your solution has 5 flows, that's annoying. If your solution has 50 or 100 flows, that's a half-day of manual work — clicking into each flow, assigning the connection, saving, turning it on, and moving to the next one. In a team doing frequent releases across multiple environments (Test, UAT, Production), this compounds quickly. It turns what should be a 10-minute deployment into an hours-long chore, introduces human error, and makes your pipeline feel like it only does half the job. This is one of the most common pain points in Power Platform DevOps, and it's almost never solved cleanly out of the box. This article explains exactly why it happens and how to fix it so that flows are on, connections are wired, and the environment is fully operational the moment the pipeline finishes. Why This Happens Understanding the root cause is important because there are actually three separate things that go wrong, and you need to address all three. 1. Flows Are Exported in Whatever State They Were In When a Power Platform solution is exported from a source environment, every cloud flow is embedded in the solution package in its current state. If a flow was turned off in the source environment at the time of export — even briefly, for testing or debugging — it ships in that state. When the managed solution is imported into the target environment, the flow arrives and stays off. There is no automatic activation step built into the standard import process. 2. Connection References and Actual Connections Are Different Things This is the conceptual point that trips up most teams new to Power Platform ALM. A connection is the actual authenticated link to a service — a specific Dataverse instance, an Outlook mailbox, a SharePoint site. Connections are environment-specific, created manually or via admin tools, and they live outside any solution. They should never be part of a solution package. A connection reference is a pointer. It's a solution component that says "this flow uses a connection of type X." The connection reference lives inside the solution, travels with it across environments, and is what the flow binds to at runtime. The connection reference itself has no credentials — it just points to whichever actual connection in the environment is assigned to it. The correct setup is: In the source environment (DEV): The actual connections exist and are assigned to the connection references. The solution contains only the connection references, not the connections themselves. In the target environment (Test, UAT, Production): The actual connections are pre-created by an administrator and given appropriate access. The service principal used by the pipeline to deploy the solution must have read/write access to these connections. When the solution is imported, the deployment settings file maps each connection reference in the solution to the correct pre-existing connection in that environment. If this mapping is not done correctly, flows that depend on unresolved connection references will remain in a draft state after import, regardless of any other settings. 3. The Standard Import Task Does Not Activate Flows Power Platform Build Tools for Azure DevOps includes an ActivatePlugins flag on the import task. Despite what the name implies, this activates Dataverse plugins and custom workflow activities only — it has no effect on Power Automate cloud flows. There is no built-in flag on the standard import task that activates cloud flows. This means that even a perfectly configured import, with all connection references resolved and all tokens substituted, will still leave flows in a deactivated state unless you add an explicit activation step. Prerequisites: What Must Be in Place Before the Pipeline Runs Before the pipeline can solve this problem end-to-end, two things must be true in every target environment. First, the actual connections must already exist. For every service your flows connect to — Dataverse, Outlook, SharePoint, Teams, or any other connector — an administrator must have already created a connection in the target environment. These connections are not part of the solution and should never be included in the solution export. They are environmental infrastructure, created once and maintained independently of deployments. Second, the service principal must have access. The Azure Active Directory app registration used as the service principal for the pipeline (the account that authenticates the import) must be granted access to read and write in the target environment. This includes having sufficient Dataverse security roles and, where applicable, being designated as an owner or co-owner of the connections so that the deployment settings file can map connection references to those connections during import. Once these are in place, the pipeline can take over the rest automatically. The Pipeline Approach The pipeline is split into two phases: a build phase that runs against the source environment and packages the solution, and a release phase that deploys the packaged solution to each target environment. Build Phase The build phase exports the managed solution from the source environment and generates a DeploymentSettings.json file. This file is the key to automating connection reference mapping. It is generated by the PAC CLI from the solution ZIP and contains a structured list of every connection reference and environment variable in the solution. Out of the box, the generated file has empty ConnectionId fields. The build pipeline post-processes this file by replacing those empty fields with placeholder tokens in the format @@token_name@@. For example, a connection reference with the logical name shared_commondataserviceforapps becomes @@shared_commondataserviceforapps@@ in the deployment settings file. The file is then published as part of the build artifact. The critical point is that connection reference logical names often include a random trailing suffix added by the platform (e.g., shared_commondataserviceforapps_8ca1f). The build script normalizes these by stripping the suffix, so the token is deterministic and consistent across builds. Release Phase The release pipeline picks up the build artifact for each environment stage and runs the following sequence: Step 1: Replace Tokens A token replacement task reads DeploymentSettings.json and substitutes each @@token@@ with the corresponding pipeline variable for that stage. The pipeline variables for each stage hold the actual connection IDs of the pre-existing connections in that environment. For example, the Test stage has a variable shared_commondataserviceforapps with the value of the Dataverse connection ID in the Test environment. After this step, the deployment settings file is fully resolved with no remaining placeholders. Step 2: Import Solution The Power Platform Import Solution task imports the managed solution ZIP using the resolved DeploymentSettings.json. This wires up every connection reference to its corresponding connection in the environment automatically, with no manual intervention. Step 3: Activate Flows This is the step that closes the gap. A PowerShell task runs after the import and queries the Dataverse API for all cloud flows in the solution that are not currently active. It then activates each one programmatically. The Activation Script This PowerShell script uses the Dataverse Web API, authenticated with the same service principal credentials used by the rest of the pipeline. It queries specifically for Modern Flow entities (category eq 5) in the target solution and activates any that are in a stopped or draft state. PowerShell # Activate all Power Automate cloud flows in the solution post-import $tenantId = "$(TenantId)" $clientId = "$(CRMClientId)" $clientSecret = "$(CRMClientSecret)" $environmentUrl = "$(CRMEnvironmentUrl)" $solutionName = "$(CRMSolutionName)" # Obtain an OAuth token for the Dataverse API $tokenUrl = "" $tokenBody = @{ grant_type = "client_credentials" client_id = $clientId client_secret = $clientSecret resource = $environmentUrl } $tokenResponse = Invoke-RestMethod -Method Post -Uri $tokenUrl -Body $tokenBody $token = $tokenResponse.access_token $headers = @{ "Authorization" = "Bearer $token" "OData-MaxVersion" = "4.0" "OData-Version" = "4.0" "Content-Type" = "application/json" } $apiBase = "$environmentUrl/api/data/v9.2" # Query for cloud flows (category = 5) in this solution that are not Active (statecode != 1) $queryUrl = "$apiBase/workflows?`$filter=category eq 5 and statecode ne 1" + " and _solutionid_value eq (select solutionid from solutions where uniquename eq '$solutionName')" + "&`$select=workflowid,name,statecode,statuscode" $flows = (Invoke-RestMethod -Uri $queryUrl -Headers $headers).value if ($flows.Count -eq 0) { Write-Host "All flows are already active. Nothing to do." } else { Write-Host "Found $($flows.Count) flow(s) to activate." foreach ($flow in $flows) { $patchUrl = "$apiBase/workflows($($flow.workflowid))" $payload = @{ statecode = 1; statuscode = 2 } | ConvertTo-Json Invoke-RestMethod -Method Patch -Uri $patchUrl -Headers $headers -Body $payload Write-Host "Activated: $($flow.name)" } Write-Host "Done. $($flows.Count) flow(s) activated successfully." } Note on category eq 5: Power Platform stores multiple automation types in the same workflow entity. Category 0 is classic workflows, category 4 is business process flows, and category 5 is Modern Flow (Power Automate cloud flows). The filter ensures only cloud flows are touched. Guarding Against Unresolved Connections A common failure mode is a new connection reference being added to the solution in DEV, the build generating a new @@token@@ for it, but the corresponding pipeline variable not being added to the release stage yet. The import will succeed, but the flow that depends on that connection will remain inactive — and the activation script will fail to activate it because the connection reference is still unresolved. To catch this early, add a validation step before the import that checks for any remaining @@token@@ placeholders in the deployment settings file and fails the pipeline immediately if any are found: PowerShell $settingsPath = "$(System.DefaultWorkingDirectory)/$(Build.DefinitionName)/$(Build.BuildNumber)/DeploymentSettings.json" $content = Get-Content $settingsPath -Raw $unresolved = [regex]::Matches($content, '@@[^@]+@@') | Select-Object -ExpandProperty Value if ($unresolved.Count -gt 0) { Write-Error "Unresolved connection tokens found in DeploymentSettings.json:`n$($unresolved -join "`n")" Write-Error "Add the missing pipeline variables for this stage and re-run." exit 1 } Write-Host "All connection tokens resolved. Proceeding with import." Failing fast here is far better than a silent partial deployment where some flows activate, and others don't. The Result Once this is in place, the deployment experience changes completely. A pipeline run that previously required a human to log into each target environment, open Power Automate, navigate to each flow, assign connections, and manually toggle flows on — a process that scales linearly with the number of flows — becomes fully automated. For a solution with 100 cloud flows deploying across three environments, that might be 300 individual manual actions eliminated per release cycle. The environment is fully operational the moment the pipeline is completed. No follow-up tickets. No forgotten flows. No production incidents because someone missed one. The key insight is that the platform gives you all the pieces — connection references for portability, deployment settings for environment-specific mapping, and the Dataverse API for programmatic activation — but it does not wire them together for you automatically. Once you do, your Power Platform deployments become as reliable and hands-off as any other enterprise application deployment.

By karthik nallani chakravartula

Databases

DZone's Featured Databases Resources

Top Databases Experts

The Latest Databases Topics