Databases

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/why-openai-upgrading-api. If you’ve ever built a large-language-model application, you’ve most likely started with this endpoint: HTTP POST /v1/chat/completions In the era of GPT-3.5 and GPT-4, this endpoint was practically synonymous with the OpenAI API. Developers would pass in a set of messages, and the model would generate the next response based on the context. But as applications have evolved from “chatbots” to “agents capable of invoking tools, executing tasks, and processing multimodal content,” the structure of the API has also begun to change. OpenAI has introduced a more unified approach: HTTP POST /v1/responses This doesn’t mean Chat Completions are obsolete; rather, it provides a more appropriate abstraction for the more complex workflows of agents. Chat Completions: Conversation Messages at the Center The core data structure of Chat Completions is messages. In each request round, the client must submit the context required for the model to understand the current task. For example, a user requests the weather in Beijing: JSON { "model": "gpt-5.6", "messages": [ { "role": "user", "content": "帮我查询北京天气" } ], "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "查询天气", "parameters": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] } } } ] } After the model decides to call a tool, it will return a result similar to the following: JSON { "choices": [ { "message": { "role": "assistant", "content": null, "tool_calls": [ { "id": "call_weather_001", "type": "function", "function": { "name": "get_weather", "arguments": "{\"city\":\"北京\"}" } } ] } } ] } After the application executes get_weather, the next request must include the previous conversation, the tool calls initiated by the model, and the results of those tool executions: JSON { "model": "gpt-5.6", "messages": [ { "role": "user", "content": "帮我查询北京天气" }, { "role": "assistant", "content": null, "tool_calls": [ { "id": "call_weather_001", "type": "function", "function": { "name": "get_weather", "arguments": "{\"city\":\"北京\"}" } } ] }, { "role": "tool", "tool_call_id": "call_weather_001", "content": "北京晴，25°C" } ] } This approach is intuitive, well-established, and still suitable for most chat scenarios. However, it has one obvious engineering shortcoming: context management is primarily handled by the client. As conversations grow longer and tool calls increase, the application must continuously maintain and replay historical messages. Responses: Centered Around a Single “Task Response” The Responses API takes a different approach: it treats model output not merely as a piece of text, but as a “response” that may include text, reasoning, tool calls, images, or structured results. Let’s use the weather query as an example again: JSON { "model": "gpt-5.6", "input": "帮我查询北京天气", "tools": [ { "type": "function", "name": "get_weather", "description": "查询天气", "parameters": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] } } ] } The model returns a responsecontaining a function call: JSON { "id": "resp_123", "output": [ { "type": "function_call", "call_id": "call_weather_001", "name": "get_weather", "arguments": "{\"city\":\"北京\"}" } ] } After the tool completes execution, the next round only needs to submit the new results and reference the previous response: JSON { "model": "gpt-5.6", "previous_response_id": "resp_123", "input": [ { "type": "function_call_output", "call_id": "call_weather_001", "output": "北京晴，25°C" } ] } OpenAI can use previous_response_id to associate the previous context with the tool call. The client does not need to manually replay the entire message history each time, making the Agent’s orchestration code more concise. However, note that this does not mean “context no longer incurs costs.” Using previous_response_idreduces the complexity for the client in constructing and maintaining the message history; previous input tokens in the response chain will still be billed as input tokens. Why Does the Agent Need the Responses API More? In a question-and-answer scenario, messagesare natural; but Agents often need to constantly switch between conversations, tool calls, tool results, and structured data. Chat Completions can also handle these tasks, but as the number of steps increases, the client must maintain a complex messages history on its own and ensure that tool calls are correctly mapped to their results. The focus of the Responses API is not on adding a new capability, but on unifying these elements into response items and supporting the continuation of tasks based on the previous response, making it better suited for complex Agent workflows. How Should an API Gateway Be Designed? If the Gateway integrates models such as OpenAI, Claude, Gemini, and DeepSeek simultaneously, the key is not to rewrite all requests as Responses. A more practical approach is to retain client-familiar interfaces  —  such as Chat Completions and Responses  —  for external use; once requests enter the system, they are parsed by the corresponding converters and routed into the same processing pipeline. OwlVigil adopts precisely this approach: rather than replacing Chat with Responses, it allows different protocols to share the same set of gateway capabilities. Plain Text Client ├─ Chat Completions ├─ Responses ├─ Anthropic Messages └─ Gemini API ↓ Inbound Converter ↓ Unified LLM Request Model ↓ Model mapping, routing, rate limiting, retries ↓ Outbound converter ↓ OpenAI ├─ Claude ├─ Gemini └─ DeepSeek The term “unified” here does not mean forcing a binding to a single vendor’s protocol, but rather placing messages, tool calls, tool results, model parameters, and streaming responses into a single processing pipeline.

Every recorded meeting your organization has ever held is already a knowledge base. It just happens to be stored in the least queryable format imaginable, which is a wall of MP4 files sitting in a storage account that nobody opens twice. The good news is that the gap between that wall of files and a working question-answering agent is now much shorter than it used to be, because Microsoft Foundry ships the two halves you need in one place. Fast transcription turns the audio into diarized text in seconds rather than in real time, and Foundry IQ turns that text into a permission-aware knowledge base that any agent can query through a single endpoint. This walkthrough builds the whole thing end to end. By the end you will have a pipeline that watches a blob container for new recordings, transcribes them with speaker labels, chunks them into speaker turns with enough metadata to make citations useful, indexes them as a Foundry IQ knowledge source, and exposes a Foundry agent that answers questions like "what did we decide about the pricing migration in Q2 and who pushed back" with real references back to the moment in the recording. A quick naming note before we start, because the ground has moved. At Ignite 2025, Microsoft renamed Azure AI Foundry to Microsoft Foundry, and the rename was formalized in the January 2026 Product Terms. The platform is the same platform, but there are now two portal experiences and two generations of SDK. The 2.x preview of azure-ai-projects targets the new Foundry portal and API, and the 1.x GA line targets what the docs call Foundry classic. Everything in this article uses the 2.x line and the Responses-based agent surface. What We Are Building, and the Shape of the Data Flow The pipeline has two independent halves that meet at a blob container of curated transcripts. The ingestion half is batch and event-driven. It cares about throughput and about not losing files. The retrieval half is synchronous and user-facing. It cares about latency and about grounding quality. Keeping them decoupled through storage means you can reindex, re-chunk, or swap the retrieval strategy without touching a byte of audio again. The flow is worth reading left to right once. A recording lands in raw-recordings. Event Grid picks up the Blob Created event and drops a message on a queue, which gives you retry semantics and a dead letter path for free. A queue-triggered Function pulls the message, POSTs the audio to the Foundry Speech fast transcription endpoint, and gets back a synchronous response containing diarized phrases. A second stage groups those phrases into speaker turns, attaches timestamps and meeting metadata, and writes JSONL into curated-transcripts. Foundry IQ indexes that container on a schedule. Why a queue between Event Grid and the Function rather than a direct trigger? Because fast transcription is synchronous and the audio files are large. A direct blob trigger gives you very little control over concurrency, and the first time somebody bulk-uploads six months of archived recordings, you will saturate your Speech resource and start collecting 429s. The queue lets you cap batchSize in host.json and shape the load. Standing up the Foundry Project and the Speech Resource Create a Foundry project first. In the portal, make sure the New Foundry toggle is on, then create or select a project. The thing you need out of the portal is the project endpoint, which has the form https://<resource-name>.services.ai.azure.com/api/projects/<project-name>. Install the preview packages. Shell pip install "azure-ai-projects>=2.4.0" azure-identity openai azure-storage-blob requests az login Entra ID is the only authentication method the project client supports, so there is no key-based escape hatch here. Give yourself the Azure AI User role on the project resource for development work. For the pipeline itself, use a user-assigned managed identity and grant it Azure AI User plus Storage Blob Data Contributor. Two environment variables carry the rest of the article. Shell export FOUNDRY_PROJECT_ENDPOINT="https://your-account.services.ai.azure.com/api/projects/meetings" export SPEECH_RESOURCE_NAME="your-speech-resource" Confirm the project client talks to the service before you build anything on top of it. Python import os from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential with ( DefaultAzureCredential() as credential, AIProjectClient( endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=credential, ) as project, ): openai = project.get_openai_client() r = openai.responses.create( model="gpt-5-mini", input="Reply with the single word ready.", ) print(r.output_text) get_openai_client() returns an authenticated client from the openai package configured to run Responses operations against your Foundry project endpoint. That is the pattern to internalize. You use the project client for setup, configuration, agents, and evaluations, and the OpenAI-compatible client for the actual model calls. Turning an Hour of Audio Into Diarized Speaker Turns Fast transcription is the right tool for recorded meetings. It returns results synchronously and much faster than real time, which is exactly the tradeoff you want for a file that already exists. Batch transcription is the alternative, and it wins on very long archives and on advanced customization, but for a one-hour standard-format recording, fast transcription gets you a result in a small number of seconds with predictable latency. The endpoint is /speechtotext/transcriptions:transcribe and the current generally available API version is 2025-10-15. It takes multipart/form-data with the audio in one part and a JSON definition in another. Diarization is configured with a diarization object carrying maxSpeakers, and the service can separate up to 35 distinct speakers in a single channel before it errors out. Here is the worker in full, with the retry behavior that you will absolutely need. Python import json import os import time import requests from azure.identity import DefaultAzureCredential SPEECH_ENDPOINT = ( f"https://{os.environ['SPEECH_RESOURCE_NAME']}" ".cognitiveservices.azure.com/speechtotext/transcriptions:transcribe" "?api-version=2025-10-15" ) SCOPE = "https://cognitiveservices.azure.com/.default" RETRYABLE = {408, 429, 500, 502, 503, 504} def transcribe(audio_path, locales=("en-US",), max_speakers=8, max_attempts=5): """Fast transcription with diarization and bounded exponential backoff.""" credential = DefaultAzureCredential() definition = { "locales": list(locales), "diarization": {"enabled": True, "maxSpeakers": max_speakers}, "profanityFilterMode": "None", } for attempt in range(max_attempts): token = credential.get_token(SCOPE).token with open(audio_path, "rb") as fh: response = requests.post( SPEECH_ENDPOINT, headers={"Authorization": f"Bearer {token}"}, files={"audio": (os.path.basename(audio_path), fh)}, data={"definition": json.dumps(definition)}, timeout=600, ) if response.status_code == 200: return response.json() if response.status_code not in RETRYABLE: raise RuntimeError( f"Fast transcription failed {response.status_code} {response.text[:400]}" ) wait = float(response.headers.get("Retry-After", 2 ** attempt)) time.sleep(min(wait, 60)) raise RuntimeError(f"Giving up on {audio_path} after {max_attempts} attempts") A few things in there earn their place. The Retry-After header is honored when the service sends one, which matters a lot under throttling because blind exponential backoff on a shared Speech resource just means every worker retries in lockstep. Profanity filtering is set to None because the default is Masked and masked words in a transcript quietly damage retrieval, since the asterisks become tokens that match nothing. The 600-second timeout is generous on purpose, because a large file uploading over a constrained egress path can spend a long while before the service even starts work. The response contains a phrases array where each entry carries speaker, offsetMilliseconds, durationMilliseconds, and text. Phrases are the wrong chunk size for retrieval. They are usually a sentence or two, which means an embedding of a phrase carries almost no context, and a citation to a phrase drops the reader into the middle of a thought. Group them into speaker turns instead. Python from dataclasses import dataclass, asdict @dataclass class Turn: meeting_id: str meeting_title: str meeting_date: str speaker: str start_ms: int end_ms: int text: str @property def chunk_id(self): return f"{self.meeting_id}-{self.start_ms:09d}" def to_turns(result, meta, max_chars=2400, gap_ms=4000): """Collapse diarized phrases into speaker turns, splitting very long ones.""" turns, current = [], None for p in result.get("phrases", []): speaker = f"Speaker {p.get('speaker', 'unknown')}" start = p["offsetMilliseconds"] end = start + p["durationMilliseconds"] same_speaker = current and current.speaker == speaker contiguous = current and (start - current.end_ms) < gap_ms room = current and (len(current.text) + len(p["text"])) < max_chars if same_speaker and contiguous and room: current.text += " " + p["text"] current.end_ms = end continue if current: turns.append(current) current = Turn( meeting_id=meta["meeting_id"], meeting_title=meta["title"], meeting_date=meta["date"], speaker=speaker, start_ms=start, end_ms=end, text=p["text"], ) if current: turns.append(current) return turns The gap_ms guard is the part people leave out. Without it, a speaker who talks at minute three and again at minute forty gets merged into one chunk if nobody else spoke in between, which is rare but produces a chunk whose timestamp range is meaningless. Four seconds of silence is a reasonable turn boundary for meeting audio. Making Chunks That Are Worth Citing Retrieval quality on meeting transcripts lives or dies on what surrounds the raw text. A bare speaker turn like "yeah I think that's fine, let's go with option two" is nearly unretrievable, because it contains no nouns. The fix is to write a small amount of generated context into each record and let the hybrid search match on that. Python def contextualize(openai, turn, neighbors): """Prepend a one-line situating summary so short turns stay retrievable.""" window = "\n".join(f"{n.speaker}: {n.text}" for n in neighbors) r = openai.responses.create( model="gpt-4.1-mini", input=( "Write one sentence, under 25 words, situating the final utterance " "inside this meeting excerpt. Name the topic and any decision. " "Do not editorialize.\n\n" f"Meeting: {turn.meeting_title} ({turn.meeting_date})\n\n" f"{window}\n\nFinal utterance: {turn.speaker}: {turn.text}" ), ) return r.output_text.strip() def to_records(openai, turns): for i, turn in enumerate(turns): neighbors = turns[max(0, i - 3): i + 1] context = contextualize(openai, turn, neighbors) yield { **asdict(turn), "chunk_id": turn.chunk_id, "context": context, "content": f"{context}\n\n{turn.speaker}: {turn.text}", "timecode": f"{turn.start_ms // 60000:02d}:{(turn.start_ms // 1000) % 60:02d}", } This costs one small model call per turn, which, in a one-hour meeting, is a few hundred calls of a couple hundred tokens each. Run it concurrently with a semaphore rather than serially. The timecode field is what makes citations feel like a product feature rather than a footnote, because you can render it as a deep link into your video player. Write the records as JSONL to curated-transcripts, one file per meeting, and you are done with audio forever. Wiring the Transcripts Into a Foundry IQ Knowledge Base Foundry IQ is the knowledge and retrieval layer built on Azure AI Search. The mental model is two nested objects. A knowledge source points at searchable content, and a knowledge base wraps one or more knowledge sources behind a single endpoint that agents query. For indexed sources, Foundry IQ manages the whole indexing pipeline, so content gets ingested, chunked, vectorized, and prepared for hybrid retrieval without you standing up a skillset by hand. Agentic retrieval features are generally available in the 2026-04-01 REST API. The 2026-05-01-preview version exposes the fuller feature set, including preview knowledge source kinds and the ability to attach an LLM to non-web sources. Blob Storage is a generally available indexed source kind, which is exactly what we need. Point a knowledge source at the curated container. Python from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( KnowledgeBase, KnowledgeSourceReference, AzureBlobKnowledgeSource, AzureBlobKnowledgeSourceParameters, ) from azure.identity import DefaultAzureCredential index_client = SearchIndexClient( endpoint=os.environ["SEARCH_ENDPOINT"], credential=DefaultAzureCredential(), ) source = AzureBlobKnowledgeSource( name="meeting-transcripts", description=( "Diarized speaker turns from recorded internal meetings, 2024 onward. " "Each chunk carries meeting title, date, speaker label, and timecode." ), azure_blob_parameters=AzureBlobKnowledgeSourceParameters( connection_string=os.environ["BLOB_CONNECTION"], container_name="curated-transcripts", embedding_model=..., # your deployed text embedding model chat_completion_model=..., # optional, enables verbalization ), ) index_client.create_or_update_knowledge_source(source) That description field is not decoration. When a knowledge base holds several sources, the retrieval engine plans which sources to query, and the description is the primary signal it uses to route. Write it like you are briefing a colleague who has never seen your data. Now the knowledge base. Python kb = KnowledgeBase( name="meetings-kb", knowledge_sources=[ KnowledgeSourceReference(name="meeting-transcripts", always_query_source=False), ], retrieval_instructions=( "Meeting transcripts. When the user asks who said or decided something, " "return the speaker turns that contain the statement plus the surrounding turns. " "Prefer recent meetings when the question is about current state." ), ) index_client.create_or_update_knowledge_base(kb) The retrieval engine plans which sources to query and performs iterative search when the first pass does not clear its relevance bar. Iterative search depends on setting a medium retrieval reasoning effort, either on the knowledge base or per request. That single knob is also the biggest lever on both latency and spend, so treat it as a tuning parameter rather than a set-and-forget value. Reasoning effortWhat the engine doesGood fit forMinimalSingle pass, extractive results, no query planningLookup-style questions where the user names the meetingLowLight query decomposition across sourcesMost interactive chat trafficMediumIterative search plus richer planning over sourcesAnalytical questions spanning many meetings Giving the Agent a Knowledge Base and a Personality With the knowledge base in place, the agent is short. Agent operations in the 2.x SDK are built on the Responses protocol, and agents are versioned objects created with create_version. Python from azure.ai.projects.models import PromptAgentDefinition INSTRUCTIONS = """You answer questions about internal meetings using only the meeting transcript knowledge base. Rules you follow without exception. 1. Every factual claim carries a citation naming the meeting title, date, and timecode. 2. When you cannot find support in the transcripts, say so plainly and stop. 3. Attribute statements to the speaker label exactly as it appears. Never guess a real name. 4. When speakers disagreed, surface the disagreement rather than flattening it into consensus. 5. Distinguish a decision from a suggestion. Quote the language that makes it one or the other. """ agent = project.agents.create_version( agent_name="meeting-analyst", definition=PromptAgentDefinition( model="gpt-5-mini", instructions=INSTRUCTIONS, tools=[{"type": "knowledge_base", "knowledge_base": {"name": "meetings-kb"}], ), ) print(agent.id, agent.version) Rule three is doing real work. Diarization gives you stable speaker identifiers within a recording, not identities, so you get generic labels rather than names. If the instructions do not forbid it, a capable model will cheerfully infer that Speaker 2 is the person whose name appears in the meeting title, and it will be wrong roughly as often as it is right. If you need real names, map them yourself in the chunking stage from calendar metadata or from multichannel capture, and write the resolved name into the record. Calling the agent looks like any Responses call. Python def ask(openai, agent_name, question, previous_response_id=None): return openai.responses.create( extra_body={"agent": {"name": agent_name, "type": "agent_reference"}, input=question, previous_response_id=previous_response_id, ) first = ask(openai, "meeting-analyst", "What did we decide about the pricing migration, and did anyone object?") print(first.output_text) follow_up = ask(openai, "meeting-analyst", "Which of those objections were ever resolved?", previous_response_id=first.id) print(follow_up.output_text) Threading through previous_response_id keeps the conversation server-side, which means you are not shipping a growing transcript of the chat on every turn and you are not writing your own history store. Failing Well When Retrieval or the Model Does Not Cooperate Two failure classes matter in production, and they want different handling. Transient service errors want retries. Empty or weak retrieval wants a different answer, not a retry, because running the same query again against the same index returns the same nothing. Python import random from openai import APIStatusError, APITimeoutError TRANSIENT = {408, 409, 429, 500, 502, 503, 504} def ask_resilient(openai, agent_name, question, attempts=4, **kwargs): last = None for i in range(attempts): try: return ask(openai, agent_name, question, **kwargs) except APITimeoutError as exc: last = exc except APIStatusError as exc: if exc.status_code not in TRANSIENT: raise retry_after = exc.response.headers.get("retry-after") last = exc if retry_after: time.sleep(min(float(retry_after), 30)) continue time.sleep(min(2 ** i + random.random(), 30)) raise last Full jitter on the backoff is not optional at any real concurrency. Without it, your retries synchronize into a thundering herd, and you turn a brief throttle into a sustained one. For the retrieval side, the answer is to make the agent's failure visible rather than silent. Instruction two above tells the model to say it found nothing, and you should assert on that in your evaluation set. A grounded system that admits ignorance is far more valuable than one that produces confident prose from three irrelevant chunks, and the second failure mode is much harder to notice in production because the output looks fine. Measuring Whether the Thing Actually Works Two separate quality questions live in this pipeline, and they need separate measurement. The transcription layer has an accuracy problem measured in word error rate. The retrieval and generation layer has a groundedness problem measured by a judge model. A regression in either one looks identical from the outside, which is a good argument for measuring them apart. Build a golden set first. A hundred or so questions written against meetings you have actually listened to is worth more than a thousand synthetic ones, because the value is in the expected answers and only a human who sat through the meeting can write those. Cover the awkward shapes deliberately. Include questions whose answer is genuinely absent so you can measure refusal behavior. Include questions that span two meetings. Include questions where two people disagreed. JSON {"question": "Who owned the migration rollback plan after the March review?", "expected": "Speaker 3 accepted ownership at 41:12 in Platform Review 2026-03-04.", "must_cite": "Platform Review 2026-03-04", "kind": "attribution"} {"question": "What was the agreed SLA for the batch job?", "expected": "Not discussed in any recorded meeting.", "must_cite": null, "kind": "refusal"} The evaluation operations live on the project client in the 2.x SDK, under properties such as evaluators, evaluation_rules, and schedules. For groundedness and relevance, you use built-in judge evaluators. For word error rate, you register a custom evaluator, because that one is arithmetic rather than judgment. Python import jiwer def transcript_wer(reference_text, hypothesis_text): transform = jiwer.Compose([ jiwer.ToLowerCase(), jiwer.RemovePunctuation(), jiwer.RemoveMultipleSpaces(), jiwer.Strip(), jiwer.ReduceToListOfListOfWords(), ]) return jiwer.wer(reference_text, hypothesis_text, truth_transform=transform, hypothesis_transform=transform) Hand-correct twenty minutes of audio across three or four recordings and keep it as your reference. Twenty minutes sounds thin, and it is, but it catches the failures that matter, which are domain vocabulary and acronyms coming back as phonetic mush. If your WER on product names is bad, the fix is a phrase list rather than a better model. Phrase lists let you hand the recognizer a set of words likely to appear, and they move the needle hard on proper nouns and internal jargon. The metrics worth gating a deploy on are these four. MetricWhat it catchesWhere it comes fromWord error rate on domain termsVocabulary drift, new product names, bad audioCustom evaluator against hand-corrected referenceGroundednessAnswers not supported by retrieved chunksBuilt-in judge evaluatorCitation validityFabricated meeting titles, timecodes outside the recordingDeterministic check against chunk metadataRefusal rate on absent answersConfident invention when nothing was retrievedGolden set questions with no supporting content Citation validity is the cheap one everyone skips. You already have the chunk metadata, so parsing the citations out of the answer and asserting that each meeting title exists and each timecode falls inside that recording's duration is maybe thirty lines of code. It catches a specific and embarrassing failure that judge models are surprisingly forgiving of. Getting This to Production Without Regrets Reindex on a schedule and expect churn. Foundry IQ triggers indexing and data synchronization automatically for indexed sources, but your curated container is the contract. If you change chunking strategy, you are rewriting every record, and a full reindex of a large corpus is not instant. Version your chunking logic and write the version into each record so you can tell mixed-generation content apart during a migration. Decide the permission model before you index anything. Meeting recordings are among the most sensitive content an organization has. Retrieval in Foundry IQ respects user permissions for supported knowledge source types, and for the remote SharePoint source, Purview sensitivity labels and data classifications flow through the indexing and retrieval pipeline. Blob-backed sources do not give you that for free. If access control per meeting matters, either enforce it with security filters at query time using a field on each chunk, or keep recordings in SharePoint and use the remote source, where content never leaves SharePoint, and SharePoint enforces permissions. Retrofitting this later means reindexing everything and auditing every conversation that already happened. Instrument with tracing from day one. The projects SDK ships GenAI tracing instrumentation, currently an experimental preview where spans and attributes may change between versions. Turn it on anyway. When a user says the agent gave a bad answer, you want the retrieved chunk IDs and the query plan from that exact response, and reconstructing them after the fact from logs you did not write is miserable. Watch the two meters. Retrieval bills token usage for subquery execution and semantic reranking, and the model you attach for query planning and answer synthesis bills separately on the model side. Reasoning effort, source count, and how much content you route into synthesis are the levers, in that order. Plan the migration if you are on the old pattern. If you are still using Azure OpenAI On Your Data, the "Add your data" flow in the classic chat playground, it is deprecated and retires on October 14, 2026. The official migration target is exactly the stack in this article, which is Foundry Agent Service plus Foundry IQ. How This Compares to Rolling the Pipeline Yourself The obvious alternative is a hand-built stack. Whisper for transcription behind your own GPU or an inference endpoint, pyannote for diarization, your own chunker, a vector database, and LangChain or a custom orchestrator on top. That stack is genuinely good, and it is genuinely more work. The honest comparison looks like this. ConcernFoundry with fast transcription and Foundry IQSelf-hosted Whisper plus pyannote plus a vector DBAmazon Transcribe plus Bedrock Knowledge BasesGoogle Speech-to-Text plus Vertex AI SearchDiarizationBuilt into the same call, up to 35 speakersSeparate model, separate tuning, best-in-class quality achievableBuilt into the transcription jobBuilt into the recognizerTime to first working answerHoursDays to weeksHoursHoursRetrieval planningAgentic, multi-query, iterative at higher effortWhatever you writeManaged retrieval, less query planningManaged retrieval with good semantic rankingPermission-aware retrievalNative for supported sources, Purview labels honored for remote SharePointYou build itIAM-scoped, coarser at the chunk levelIAM-scopedWhere the audio goesYour Azure regionWherever you run it, including fully on-premisesYour AWS regionYour GCP regionEscape hatchKnowledge bases callable from any app through the Search APIsTotal controlBedrock APIsVertex APIs The self-hosted path wins on two things, and they are not small. One is cost at very high volume, because at some point per-minute transcription pricing loses to a GPU you already own. The other is data residency in the strict sense, meaning audio that legally cannot leave your premises. If neither applies to you, the managed path buys back weeks of work you would otherwise spend on chunking heuristics and retry logic. Within Azure, there is also a smaller decision, which is fast transcription against batch transcription. Fast wins on latency and simplicity for files under the size limit. Batch wins when you need to process very large archives asynchronously, when you want webhook notifications on completion, or when you want to bring your own storage account for the outputs. Where to Take It Next The pipeline above is the spine. The interesting extensions hang off the chunking stage, because that is where you decide what the retrieval layer is even capable of answering. Extracting action items into a structured field lets you answer "what did I commit to last month" without any retrieval creativity. Writing a sentiment or disagreement flag onto each turn lets the agent find contested moments directly. Adding a second knowledge source pointed at your specs and design docs turns "what did we decide" into "what did we decide and does the shipped code match", and because a knowledge base fronts multiple sources behind one endpoint, that is a configuration change rather than an architecture change. The part worth protecting as you extend is the evaluation loop. Meeting corpora grow continuously and unevenly, and a retrieval strategy tuned on six months of transcripts behaves differently on three years. The golden set is what tells you when that has happened. References Use the fast transcription APISpeech-to-text REST API referenceWhat is Foundry IQCreate a knowledge base in Azure AI SearchConnect agents to Foundry IQ knowledge basesQuickstart: Get started with the Microsoft Foundry SDKAzure AI Projects client library for Python

By Jubin Soni, FBCS CORE

Originally, back-end and front-end Site Reliability Engineering (SRE) were owned by teams. They code the programs, set up databases and infrastructure, and quickly spring to action at the beep of any anomaly. The advent of code vs no-code infrastructure, SaaS, API dependencies, third parties, and other modern systems seems to be eroding this authority. Mainstream and underdog companies now often leverage the significant advantages of outsourcing, collaboration, or delegation, which are usually accompanied by a silent clause: no or partial control. Unlike in previous systems, modern production is largely assembled rather than built from scratch. For example, a conventional SaaS product is built on interdependencies among payment processors, outsourced data infrastructure such as Amazon Web Services (AWS), messaging services, web hosting, design, AI inference APIs, authentication providers like Google, and more. These useful platforms and products are essentially outside teams' control stations, even though they critically impact users' experience. When they function effectively, you share the glory with the platforms. But when there is a system blackout, your users put you on your toes, even though you have no direct access to resolve the problem on time. Therefore, we shall be exposing SRE practices in platform-SaaS and API-dependent systems and how reliability is getting beyond the control of engineering teams and companies. Why Classical SRE Practices May Fail One major downside of SaaS and dependency on external platforms is that reliability control is often assumed to be in a team's hands, whereas it has been bargained. However, teams must reckon with the fact that the case is reversing. For example, traditional SRE models once alleged that: Service Level Indicators (SLIs) focus on availability or internal uptime and latency.Error budgets arise from changes teams make or deploy.Runbooks still suggest that teams can immediately reconfigure or directly work on faulty components. All these are becoming past cases, especially in platform-SaaS systems. You can have a system indicating 99.99% or even 100% uptime on the back end, while new users are struggling to sign up, probably because an authenticator provider is not fully functional. Dashboards and control panels may indicate green, but in reality, third-party payment APIs have been degraded. A New Definition of Reliability in Operating SRE Practices To resolve the new problem in site reliability engineering (SRE), there needs to be a conceptual shift from component health to an integrated, continuous user experience. Therefore, teams need to undergo a paradigm shift away from questions such as "Is our CPU working maximally?" “Is our API up?” “What are the error rates?” Instead, we should inquire: “Are users checking out seamlessly?” “How fast can they authenticate?” “Can they use the SaaS product to perform its key function?” These types of outcome-based questions span interdependent platforms beyond your full control. The login SLI needs to work with the identity provider; otherwise, its output is meaningless. If the checkout SLO skips payment authorization, then it's both fishy and unreliable. True, there may be some internal errors in a reliable system, but what really matters is an integrated multiplatform experience that the user enjoys. Error Budgets? An SRE Practice to Revisit How many teams would love error budgets to disappear when they give up control? But that’s not so. Instead, they are molecularized. When components of your systems are outsourced, the error budget doesn’t just fade away; it is instead transferred to the interdependent platforms. So, it’s better to plan for the fact that SaaS and API providers will consume some of your reliability budget. Doing so keeps you a few steps ahead and protects your business in the long run. Reliable SRE teams make decisions such as allocating part of their error budget to certain dependencies, setting acceptable parameters for degradation, and defining specific steps to take when a dependency exceeds the stipulated budgets. Here’s an example you can adapt: “We will accept payment authorization failure of 0.0% to 0.2% if it is caused by dependency instability. If it goes above that, we will turn on delayed capture or turn off promotions.” This SRE approach keeps you ready for downtime, as your systems automatically switch to planned or budgeted actions rather than relying solely on integrated platforms. What to Do When Failures Beyond Your Control Arise Actually, some failures may seem beyond your control. The more you attempt to resolve them, the more amplified they become. At this point, your team must adapt to the savvy absorption of such situations. Instead of focusing solely on retrial in an SRE approach, your team needs to design its processes and platforms. This could include failing selectively through circuit breakers, failing fast with timeouts, or failing visibly by keeping users informed. Some core settings should always remain non-negotiable and on standby. These could include the following: Read-only modes/cachesBulkheads that prevent a failure avalanche.Automated circuit breakersDeferred processing These reliable practices ensure there is some form of controlled uptime even when operations seem interrupted. Laser Observability That Proves Reliability In traditional SRE observability, the service boundary is usually the ultimate, but in most modern integrated SaaS platforms, this could be insufficient or worse, dangerous. Operators need to be aware of the actual dependency that is failing, how it is failing (e.g., errors or throttling), and how the failure affects the user experience. Accurate observability for platform-SaaS and API-dependent systems requires these four provisions: Specific dashboard and internal metrics for each vendor.SLI monitoring at the dependency level.Parallel tracing of all outbound calls.Simulation of real-time user experience and workflows. Essentially, whenever there is an emergency, operators should be able to promptly identify whether the source is internal or external. Accuracy and clarity facilitate swift response. Responding to Incidents Without Ownership Another distinct characteristic of modern SRE practice in platform-SaaS is how incidents are responded to. Without ownership, you often cannot debug on your own, roll back a bad deploy, or directly manage other issues. However, you can choose how your system responds by identifying when certain features are disabled, when signals to activate degraded modes are sent, when high traffic is redirected or shed, or when to notify users. To maintain reliability, incident response relies on runbooks to inform decisions. The following questions could help convert the technicality of runbooks to practical solutions: What is the impact on the customer?In what ways can we respond harmlessly?What can we reverse?What should we communicate externally? These questions help resolve incidents, mitigate losses, and intertwine reliability with sound judgment. Is Safety an Illusion in SLAs? SLA providers often readily contract for financial compensation when losses arise, but seldom give absolute reliability guarantees. You may not always expect vendors to consistently meet your availability goals or resolve an avalanche of outages. Safety is a critical consideration when building systems, because when users lose trust in a brand, compensation may not be able to redeem it. Therefore, advanced teams do not consider SLAs as safety nets but as risk pricing. They understand that contractual credits cannot replace trust, brand image, and some almost irredeemable damages. Human Factors in Platform-SaaS and API-Dependent Systems Dependency failures often escalate when cognitive load increases. There could be degraded performance, timeouts without error indicators, partial success, or inconsistent system behavior. Operators may not only focus on machines when dashboards lag or seem to lie. They examine the logs, failure history, or commands. Teams have to design systems with overrides and predictable degradation paths, and observability tools are beyond the failure systems. Reliability goes beyond the correct function of software; it's also about human operations. How Your SaaS and API Platforms Can Imbibe “Good” SRE Practice Effective SRE practices are modern. The following attributes know saas products and API-dependent platforms: Acknowledgment of lack of control very early.Ensuring reliability is embedded in the design.Measuring the outcomes of each SRE criterion or target, instead of just the components.Giving priority to clarity instead of trying to model or control everything because you do not own all the components.Making engineering and operations decisions and products as an integrated whole.Preparing for degradations as inevitable procedures when things fail. Your systems can be reliable if you anticipate failure and accept the reality. Conclusion Modern platform-as-a-service (SaaS) operates in a reliability-without-control manner, leading solid SRE teams to accept that they need to adapt when failures occur. It's simple logic: if you don't absolutely own everything end-to-end, then prepare for the worst: each dependency might fail. It's all about keeping the trust of your users and protecting your brand image.

Temporal is designed to preserve Workflow state through process crashes and infrastructure failures, but durable state does not remove ordinary capacity limits. In production, the control plane can remain healthy while throughput collapses because Worker slots are saturated, Task Queues mix incompatible workloads, or a failover activates a region without enough Worker capacity. Temporal Workers run outside the Temporal Service and execute Workflow and Activity code, so production scalability depends as much on Worker and routing design as on the service itself. The Worker Fleet Is Usually the First Capacity Boundary Schedule-to-Start latency is best treated as queueing delay rather than application execution time. It measures the interval between a Task being enqueued and a Worker starting it. Rising Schedule-to-Start latency, growing approximate backlog, and exhausted Worker task slots indicate that Tasks are arriving faster than the fleet can consume them. Temporal Cloud exposes temporal_cloud_v1_approximate_backlog_count, while SDK metrics expose Workflow and Activity Schedule-to-Start latency and available task slots. Temporal guidance recommends watching these signals together because backlog depth alone does not identify whether the limit is Worker count, Worker configuration, or polling behavior. Worker scaling has two layers. Horizontal scaling adds Worker processes, while concurrency tuning changes how many Tasks each process can execute simultaneously. For well-benchmarked workloads, fixed slot limits place a predictable ceiling on local resource consumption. The Java SDK exposes separate concurrency controls for Workflow Tasks and Activities, and a server-side Activity rate limit can cap dispatch across all Workers polling the same Task Queue. Java WorkerOptions options = WorkerOptions.newBuilder() .setMaxConcurrentWorkflowTaskExecutionSize(120) .setMaxConcurrentActivityExecutionSize(80) .setMaxTaskQueueActivitiesPerSecond(250) .build(); The values in this example are capacity-test outputs, not universal defaults. A CPU-heavy Activity fleet may need a lower Activity slot count than an I/O-heavy fleet. Newer Worker tuners can allocate slots dynamically from CPU and memory signals, while fixed-size suppliers remain more predictable when task resource cost is well understood. Temporal also recommends poller autoscaling for most workloads because too few pollers constrain ingestion and too many waste connections and reduce efficiency. Task Queue Topology Determines Isolation and Backpressure Adding replicas cannot repair a Task Queue topology that couples unrelated bottlenecks. A shared Task Queue is reasonable when Workflows and Activities have similar latency and resource characteristics, but it becomes risky when fast orchestration work shares capacity with slow database calls, GPU jobs, tenant bursts, or Activities constrained by a downstream API. Temporal supports specialized routing through separate Task Queues, and Activity-level server-side throttling applies to the entire queue. A throttled Activity therefore should not share a queue with work that must remain unrestricted. A Workflow can route a costly Activity to a dedicated fleet without changing the Workflow’s own Task Queue. The separation creates an independent scaling and backpressure boundary. Java ActivityOptions options = ActivityOptions.newBuilder() .setTaskQueue("payments-io") .setStartToCloseTimeout(Duration.ofSeconds(20)) .build(); PaymentActivities payments = Workflow.newActivityStub(PaymentActivities.class, options); With payments-io isolated, replicas, concurrency, credentials, network placement, and queue-wide rate limits can be tuned for payment traffic without changing the Worker pool that advances Workflow Tasks. The same principle applies to multi-tenant systems. Temporal documents per-tenant Task Queues as a strong isolation pattern and also supports fairness keys when many tenants share one queue. Priority and fairness operate within Task Queue partitions, so they manage contention inside a queue rather than replacing isolation when resource requirements differ fundamentally. Task Queue partitioning should also be distinguished from application-level queue proliferation. Temporal Task Queues are lightweight and scale internally through partitions; current documentation states that Task Queues use four partitions by default. Multiple partitions increase throughput but relax strict FIFO behavior because Tasks are distributed among partitions. Separate named queues should therefore be created for routing, isolation, or rate-control reasons, not merely to manufacture throughput that Temporal’s matching layer can already scale internally. Autoscaling Should Follow Queue Pressure, Not CPU Alone CPU-based autoscaling is insufficient for many Temporal workloads. An I/O-bound Activity can leave CPU utilization low while all Activity slots are occupied and backlog grows. Conversely, high CPU with near-zero Schedule-to-Start latency may mean that the fleet is efficiently utilized. A stronger autoscaling policy combines queue delay, backlog trend, slot availability, and host resource saturation. Temporal’s Worker health guidance treats Schedule-to-Start latency as a primary symptom of insufficient processing capacity and recommends correlating it with sync-match behavior and available slots before changing fleet size. On Kubernetes, Temporal’s Worker Controller can attach HPA or KEDA resources to versioned Worker deployments and scale from CPU, memory, Task Queue backlog, slot utilization, or custom metrics. Current guidance recommends HPA with a Prometheus adapter as the general default, while KEDA is positioned for scale-to-zero, long idle periods, or faster event-driven reactions. This matters because old and new Worker versions can coexist during safe rollout, so autoscaling should follow each active Worker Deployment Version rather than treating the fleet as a single anonymous pool. Scale-down deserves the same attention as scale-up. Backlog can reach zero while Activities are still running, and terminating aggressively can create retries or latency spikes. Worker shutdown should therefore be graceful, minimum replica counts should reflect availability requirements, and cooldowns should account for Activity duration and startup time. Pre-production tests should include Worker termination, burst recovery, and partial failure because Temporal durability preserves state but does not guarantee that an undersized replacement fleet will meet latency objectives. Regional Failover Has to Include Workers and Dependencies Regional failover is often mis-scoped as a Temporal Service feature. Temporal Cloud High Availability replicates a Namespace to a secondary region and can automatically promote the replica during an outage, but application Workers remain separately operated compute. Temporal documents a 20-minute RTO and sub-one-minute RPO for its HA service, yet application recovery can still be slower when the secondary region lacks ready Worker capacity, network access to the active Namespace, or available downstream systems. For latency-sensitive systems, Active/Hot-Passive is the most deterministic failover model: a full Worker fleet runs in both regions, the secondary fleet stays warm, and only the fleet local to the active replica processes Tasks. On failover, the warm fleet begins processing without a Worker cold start. Active/Passive costs less but requires starting or scaling Workers after failover, while Active/Active runs Workers in multiple regions even though the HA Namespace still has one active replica underneath. Connectivity must be tested as part of the failover path. For HA Namespaces, the Namespace Endpoint follows the active region through DNS; Temporal documents a 15-second TTL and roughly 30 seconds for clients to converge when resolvers honor that TTL. Private connectivity requires routes and DNS design that allow Workers to reach the promoted region. A test that switches only the Namespace but omits Worker connectivity, database promotion, queue access, secrets, codec servers, or proxies validates only part of the production path. Self-hosted multi-cluster deployments require explicit planning as well. Temporal’s Global Namespace model uses asynchronous cross-cluster replication and eventual conflict resolution, and successful failover requires Worker Processes to poll the Namespace in clusters that may become active. Replication versions determine which cluster can mutate Workflow history after failover, but they do not provision Worker compute or external dependencies. Conclusion Temporal becomes a production bottleneck when durable orchestration is treated as a substitute for capacity engineering. Stable performance comes from measuring queue delay and slot saturation, scaling Worker fleets from demand signals rather than CPU alone, separating Task Queues where workloads need independent isolation or rate control, and designing regional failover around ready Workers and reachable dependencies. With those boundaries in place, Temporal remains the durable coordination layer rather than the slowest component in the execution path.

By Akhil Madineni CORE

Every few weeks, someone on my team, or in a client meeting, asks me the same question: "Which cloud should we use for our AI workloads?" I have been building enterprise integrations for over fourteen years now, and lately most of my time goes into RAG pipelines, vector databases, and agentic orchestration on top of these platforms. So I get this question a lot, and honestly, there is no single right answer. The right cloud depends on where your data already lives, what your compliance team will accept, and which models your architecture actually needs. In this article, I want to walk through the three big players, AWS Bedrock, Google Vertex AI, and Microsoft Azure AI Foundry, and share what I have learned working with these platforms in real enterprise settings, not just from reading marketing pages. AWS Bedrock Bedrock started as a model marketplace back in 2023, and it has grown into a full platform with Guardrails for content filtering, Knowledge Bases for RAG, and AgentCore for building agentic workflows. What I like most about Bedrock is the sheer breadth of models available behind a single API. You get Claude from Anthropic, Llama from Meta, Mistral, Cohere's Command models, and Amazon's own Nova family, all through one consistent interface. If your architecture needs to swap models without rewriting your integration layer, Bedrock makes that easier than the other two. Pros: Broadest model catalog of the three, so you are not locked into one vendor's models.Strong identity and governance story if you are already running on AWS, since it plugs directly into IAM, CloudTrail, and Macie.Bedrock is one of the few places where you get Claude with enterprise indemnification, which matters a lot when legal teams get involved.Provisioned throughput options give you predictable latency for production workloads that cannot tolerate spikes. Cons: If your organization is not already AWS-native, the onboarding curve is steeper than it looks.Cross-cloud portability is basically nonexistent. A model you fine-tune on Bedrock does not export cleanly to Vertex AI or Foundry. That is a real switching cost you should plan for on day one, not something to figure out later.Some of the newer agentic tooling is still maturing, so documentation gaps show up more than I would like. Google Vertex AI Vertex AI feels different from the other two because Google's DNA here is research first. If your team cares about multimodal capability, or you want access to Gemini models the moment they ship, Vertex AI tends to be ahead. It is also the strongest option if your data already lives in BigQuery, because the integration between Vertex and BigQuery for feature engineering and MLOps pipelines is genuinely smooth. Pros: Best fit for teams doing custom model training, not just calling a hosted API. AutoML and the broader MLOps tooling cut training time noticeably compared to the other two.Tight coupling with BigQuery is a huge advantage if your organization already runs its analytics there. You avoid a lot of data movement overhead.Gemini-first multimodal workflows, plus Google Search grounding for agents, which is something neither Bedrock nor Foundry offers natively.TPU support gives real throughput advantages for heavy batch processing. Cons: If your organization is not GCP-centric already, the value proposition weakens fast. You end up paying a data-gravity tax to move information into Google's ecosystem.Governance and compliance tooling, while solid, is not as battle-tested across regulated industries as AWS's certifications.The agent ecosystem, while improving, still trails Bedrock's AgentCore and Foundry's Azure AI Agents in terms of enterprise adoption stories I have personally seen. Azure AI Foundry Foundry, formerly Azure AI Services, is Microsoft's rebranded and expanded platform, and it is the one I have written about before because it is what my own recent client work has centered on. If your enterprise already lives inside Microsoft 365, Entra ID, and Azure infrastructure, Foundry removes almost all of the identity and governance friction you would otherwise deal with. That matters more than people expect once you are past the proof of concept stage and into actual production rollout with security review. Pros: Deep Microsoft 365 and Entra ID integration means your existing enterprise approvals and identity workflows extend naturally into your AI layer.Strong OpenAI-led model access, since Microsoft's partnership with OpenAI gives Foundry early and deep access to GPT-family models.Hybrid deployment options are genuinely better here than on the other two platforms, which matters if you have on-prem systems you are not ready to fully cloud-migrate.Roughly three-quarters of Fortune 500 companies already run on Microsoft's stack, so for a lot of enterprises Foundry is simply the path of least resistance. Cons: Model breadth is narrower than Bedrock's catalog, so if you need a specific non-OpenAI model family, you may find yourself stitching together a secondary platform anyway.Because it is tied so closely to Azure compute pricing, cost predictability requires more upfront modeling than teams expect.Some newer agentic and orchestration features are still catching up to what AWS has shipped with AgentCore. So Which One Should You Actually Pick? Here is the honest answer I give in client meetings: do not choose based on a benchmark screenshot or a features table. Choose based on where your data already lives and where your governance and compliance story already works. If you are AWS-first and want maximum model flexibility, go with Bedrock. If you are Microsoft-heavy and need your AI layer to inherit existing Entra ID and 365 approvals without a fight, Foundry is the path of least resistance. If your analytics already lives in BigQuery and multimodal Gemini capability is core to your roadmap, Vertex AI earns its place. What I am increasingly seeing among the teams I work with is a hybrid pattern. A primary cloud handles the bulk of regulated workloads, and a secondary cloud gets called in only when a specific model family is not well supported on the primary platform. It is not the cleanest architecture on paper, but it reflects how fast this space is still moving. None of these three platforms is standing still, and the leader on any given feature this quarter is not guaranteed to hold that spot by next year. My suggestion, whichever cloud you land on: build your RAG and orchestration layer with enough abstraction that swapping the underlying model provider is a configuration change, not a rewrite. That single decision will save you more pain than picking the "right" cloud ever will.

It started as a fleeting thought while I was heads-down building agentic AI systems: somewhere between "just call the API" and "let's train our own model," we've quietly ended up with three completely different ways to solve the same problem. Most teams treat that as a single decision, made once, early, and never revisited. It isn't. It's a portfolio you manage for the life of the product. Here's the framework, and why I think most teams have the sequencing backward. The Three Tiers 1. Model API reliance. You call the frontier model, Claude, GPT, Gemini, whichever lab is ahead this quarter, and let its R&D absorb the part of the problem you don't understand yet. This is the right default when you genuinely don't know the shape of the task: when "correct" is still being defined, when volume is low, when the fastest way to learn is to ship and watch what breaks. 2. Fine-tuning open-source models. Once a use case turns out to be repeatable, same shape of input, same shape of output, high enough volume that you're paying real money for it every month, you stop renting intelligence and start owning it. You fine-tune an open-weight model on your own data. You don't have to chase every new open-source release to stay current; you can do this on a slow, deliberate cadence while gradually weaning that specific use case off the frontier API. 3. Migrating to declarative software. Eventually, for the use cases you understand well enough, you don't need a model call at all; you need code. Once you've mapped the edge cases, you write the deterministic pipeline: rules, retrieval, control flow, maybe a small model bolted onto the one genuinely ambiguous step. This is the least glamorous option and the most durable one: reliable, cheap, testable, and not a black box. Why This Feels Backward (and Why It Isn't) Andrej Karpathy's "Software 3.0" framing has been everywhere in AI circles since his 2025 "Software Is Changing (Again)" talk: software moved from Software 1.0 (humans hand-write code) to Software 2.0 (humans train neural network weights) to Software 3.0 (humans write natural-language prompts, treating the model itself as a new kind of programmable computer, with everything in its context window acting as the program). At the frontier, that arc is real; natural language keeps unlocking categories of software that used to require a full engineering team. But zoom into any single feature inside an actual product, and the maturity curve runs the other way. You start at 3.0, a prompt against a frontier model, because that's the fastest way to find out if the idea works at all. Once it works and repeats, you climb down to 2.0: weights you own. Once you fully understand it, you climb down further to 1.0: code you can read. Both arcs are true at the same time. Karpathy's arc is about what becomes possible. This arc is about what becomes worth hardening, once you've learned the actual shape of the problem. The frontier keeps pushing the ceiling up. Underneath it, mature teams keep pushing their own floor down. The Receipts This isn't just a personal theory; it's showing up everywhere once you look for it. Stanford University's DSPy framework is this pattern turned into an actual engineering discipline. Instead of hand-tuning prompt strings forever, you write a declarative "signature" of what a step should do, and a compiler decides, and re-decides, every time the underlying model or data changes, whether that step should run as a prompt, a set of few-shot examples, or fine-tuned weights. The program is code. The model call becomes just one swappable implementation detail inside it. Token prices, meanwhile, keep collapsing. One 2026 analysis of pricing across hundreds of models estimated something like a 600x drop in token costs since 2020, with cheaper model tiers now halving in price faster than Moore's Law ever moved. That actually complicates a naive cost argument for fine-tuning low-stakes, high-volume tasks; the API might already be close to free. What fine-tuning and code increasingly buy you isn't just savings; it's control, latency, and moat. Specialization keeps beating generality on narrow, well-defined tasks. A recent study on structured contract extraction found domain-trained small models matching or beating frontier general-purpose LLMs, at a fraction of the cost and deployable entirely inside enterprise infrastructure. That's tier 2, working exactly as advertised. And not everyone agrees on the timing, which is worth holding onto rather than smoothing over. Some sharp voices in AI investing argue the opposite case: frontier labs will keep out-improving your custom fine-tune faster than you can maintain it, so unless you're sitting on genuinely proprietary data, the better bet is to keep riding the API and pour your effort into the product wrapped around it. That's a real, unresolved tension. It's exactly why this is a portfolio decision and not a fixed rule. The Part Nobody's Actually Managing Here's what I think most roadmaps get wrong: this isn't three sequential stages for your product. It's three tiers running simultaneously, for different capabilities, all the time. Your onboarding flow might already be sitting at tier 3 because you nailed it a year ago. Your newest agentic feature is at tier 1 because you shipped it three weeks ago and don't know its failure modes yet. Something in the middle just crossed the volume threshold where fine-tuning finally pays for itself. That's not a one-time build-vs-buy fork. That's a resource allocation problem, a live one, shifting every quarter as usage patterns, model prices, and your own understanding of the task all move independently of each other. Most AI roadmaps are still built like it's a single decision made once at kickoff. A few questions I've found useful for figuring out where a given capability actually belongs: How often does it run? Low volume, sporadic — stay on the API. The fixed cost of owning it isn't worth paying yet.Is "correct" still moving? If your own definition of a good output changed last month, don't freeze it into weights or code. You'll just have to redo the work.Could a competitor replicate this with the same API call you're making? If yes, it was never your moat. Don't over-invest in owning it.What's your tolerance for a black box? Audit, compliance, and debuggability needs can pull a capability toward code even before the economics demand it.Do you actually have the data? You can't responsibly fine-tune or hard-code what you can't yet describe with real, labeled examples. Where This Leaves Us Having three ways to solve a problem instead of one is genuine abundance. A few years ago, "write the code yourself" was the only option on the table. That's insane! But abundance isn't free; it converts every roadmap into a standing allocation problem: what stays on the frontier, what gets pulled in-house, what gets frozen into something boring and reliable. Decided over and over, forever, as the ground shifts under all three tiers at once. Which of your product's capabilities do you think is sitting at the wrong tier right now?

Let's begin with the definition of an AI agent. Agents are software entities that perform tasks autonomously on behalf of a user or another program. Another way to say it is that agents can perceive the environment, think, and act to achieve a specific goal with minimal human intervention. Action is the key here. For example, if I ask my agent to book a flight from Bengaluru to Delhi. The agent will perform the following tasks. Check the flight availabilityCompare priceAsk for confirmation (Human in the loop)Book the ticket (Action) Now, can we use the same agent for every kind of action? The answer is no. It will be akin to building a monolithic application. Rather, we will prefer an architecture similar to microservices or multiple APIs designed for different functionalities. We will create multiple agents specialized for acting on specific tasks. Let's extend our previous example and think about multiple agents to build a complete travel solution. We have agents such as: Travel Agent → books flightsHotel Agent → reserves hotelFinance Agent → checks budget Now, if we have to achieve a common business goal (booking a flight and hotel after comparing the price), there will be a need for agents' collaboration and interaction. This is where the A2A protocol comes in. A2A is an open protocol that complements Anthropic's Model Context Protocol (MCP). This means MCP standardizes how AI applications connect to data sources, databases, and APIs. A2A focuses on how specialized, autonomous agents (e.g., a "Sales Agent" and a "Finance Agent") "talk" and exchange information to achieve a goal, even if they are built by different providers (OpenAI, Anthropic, Google) and on different frameworks. Agent Card is one of the key capabilities that facilitates communication between Client Agent and Remote Agent. In other words, Agent Card makes A2A possible. Agents can advertise their capabilities using an “Agent Card” in JSON format, allowing the client agent to identify the best agent that can perform a task and leverage A2A to communicate with the remote agent. We can understand agent card with an analogy. You might have seen WSDL file when there is a soap web service is exposed or open api specification for RESTFul apis. WSDL or Open API Specification describes the operations, methods, input, output etc. Similar to this Agent Card make the Agent discoverable which means the agent can actively broadcast its presence, capabilities, and endpoints so that other AI agents or orchestrators can find it and use it automatically, without a human developer having to manually hardcode the connection. (This is analogy is completely from two different software architecture. I have used this for simplifying the visualisation of Agent Card). Agent Card defines the following: What does the agent do?When should this agent be used?What input does this agent expect?What output does it return?What security schemes are supported by the agent?What is the endpoint to call this agent? If we take the previous analogy of an API, each API has a contract that defines input, output, endpoints, methods, etc. Similarly, you can understand an Agent Card as a clear contract for an Agent. JSON { "url": "https://api.travelbot-ai.com/v1/a2a", "documentationUrl": "https://docs.travelbot-ai.com/guide", "capabilities": { "streaming": true, "pushNotifications": true, "stateTransitionHistory": false }, "authentication": { "type": "bearer", "description": "JWT token obtained via OAuth2 client credentials flow." }, "defaultInputModes": ["text"], "defaultOutputModes": ["text", "data"], "skills": [ { "id": "skill-find-flights", "name": "Search Flights", "description": "Finds available flights based on origin, destination, and dates.", "tags": ["travel", "flights", "search"], "InputModes": ["text", "data"], "OutputModes": ["data"], "examples": [ "Find me a one-way flight from JFK to LAX on October 12th." ] }, { "id": "skill-book-hotel", "name": "Reserve Hotel Room", "description": "Books a specific hotel room for given check-in/check-out dates.", "tags": ["travel", "hotels", "booking"], "InputModes": ["data"], "OutputModes": ["text", "data"], "examples": [ "Book the Deluxe King Room at The Grand Hotel from Nov 1 to Nov 5." ] } ] } To see exactly how an Agent Card operates, it helps to look at its structure. In an Agent-to-Agent (A2A) workflow, a client agent requests this card from a server agent before sending a task, establishing exactly how they will interact. The key fields of the agent card are: URL: Where to connect to the agentDocumentationUrl: The user manual/guideCapabilities: What special features it supports (like live streaming or notifications)Authentication: How to securely log in (e.g., passwords, tokens)DefaultInputModes / DefaultOutputModes: How it talks and listens by default (text, audio, data)Skills: A list of specific jobs the agent can do, including details on how each job works To demonstrate this, we can build an agent with an agent card. I will use MuleSoft A2A Task Listener to demonstrate this. Do remember, Agent Card makes Agent-to-agent communication seamless; however, it is not limited to a2a. Any client that we want to connect to an agent and use it will be utilizing the Agent Card to understand the capabilities and skills of the agent. Step 1: Create a project in MuleSoft using the A2A Task Listener. Step 2: Configure A2A. Step 3: Configure the HTTP Listener. Step 4: Deploy the server. Step 5: Retrieve the agent-card using the local URL (http://localhost:8081/support-agent/.well-known/agent-card.json). Step 6: Deploy the code to CloudHub and test it again. You will receive the response as provided below: JSON { "name": "Travel Agent", "description": "Handles flight and hotel booking task.", "url": "https://travel-agent-of3h9v.5sc6y6-3.usa-e2.cloudhub.io/support-agent", "provider": { "organization": "MuleSoft", "url": "https://www.mulesoft.com" }, "version": "1.0.0", "capabilities": { "streaming": false, "pushNotifications": false, "stateTransitionHistory": false }, "defaultInputModes": [ "application/json", "text/plain" ], "defaultOutputModes": [ "application/json", "text/plain" ], "skills": [ { "id": "skill-find-flights", "name": "Search Flights", "description": "Finds available flights based on origin, destination, and dates.", "tags": [ "Flight Booking" ] }, { "id": "skill-book-hotel", "name": "Reserve Hotel Room", "description": "Books a specific hotel room for given check-in/check-out dates.", "tags": [ "Hotel Booking" ] } ], "supportsAuthenticatedExtendedCard": false, "preferredTransport": "JSONRPC", "protocolVersion": "0.3.0" } This will be used by the Client Agent to discover the skills of other agents and send the task request. Please watch the video for step-by-step implementation: I hope this helps. Let me know if you liked it.

Google's transition from Manifest V2 to Manifest V3 has been one of the most significant architectural overhauls in the history of browser extension development. For developers building ad blockers, privacy shields, or developer tools, the biggest impact is the deprecation of the blocking capabilities of the chrome.webRequest API. In its place is the chrome.declarativeNetRequest (DNR) API. Instead of letting extensions intercept and inspect network traffic in real-time, the browser now executes filtering on behalf of the extension using declarative rules. Understanding how to design, register, and optimize these declarative rules is essential for building modern web-filtering software. Here is a technical breakdown of the DNR API architecture, rule structure, dynamic rule updates, and current platform constraints. The Architectural Shift: Interception vs. Declaration In Manifest V2, network filtering occurred within the extension's background page or service worker. The extension registered a listener that executed JavaScript on every request before it was sent: JavaScript // The MV2 blocking request pattern (deprecated) chrome.webRequest.onBeforeRequest.addListener( (details) => { if (shouldBlock(details.url)) { return { cancel: true }; } }, { urls: ["<all_urls>"] }, ["blocking"] ); While highly flexible, this design introduced two major problems: Performance Overhead: The browser had to pause network requests, spin up the extension's background process, serialize the request metadata, run the extension's custom JavaScript, and wait for a response.User Privacy: Extensions required the broad <all_urls> permission, giving them access to read every request header, URL query parameter, and POST payload. Manifest V3 solves this by moving the execution engine into the browser itself. The extension defines what needs to be blocked or redirected beforehand. The browser reads these rules and applies them natively during the network stack lifecycle. The extension’s code is never executed during the request, which reduces memory consumption and protects user privacy. The Anatomy of a Declarative Rule Under the DNR model, everything is defined using rules. Each rule is a JSON object that specifies an action and the conditions under which that action should execute. Here is the standard structure of a declarative rule: JSON { "id": 1, "priority": 1, "action": { "type": "block" }, "condition": { "urlFilter": "||doubleclick.net", "resourceTypes": ["script", "sub_frame"] } } Every rule requires four primary keys: id: A unique integer (1 or greater) that identifies the rule.priority: An integer indicating order of execution. Rules with higher priority numbers override lower priority rules.action: Specifies what the browser should do when a match occurs. Valid types include block, redirect, allow (bypasses other blocks), allowAllRequests (bypasses all rules on a page), and modifyHeaders.condition: The criteria that must be met to trigger the action. This can filter by domain, URL pattern, initiator origin, request method, or resource type (such as image, xmlhttprequest, or stylesheet). Implementing Static Rulesets Extensions can bundle pre-defined rule lists within their distribution package. These are defined as static JSON files and declared in the manifest.json: JSON { "name": "Custom Focus Blocker", "version": "1.0", "manifest_version": 3, "permissions": ["declarativeNetRequest"], "declarative_net_request": { "rule_resources": [{ "id": "ruleset_social", "enabled": true, "path": "rules/social.json" }] } } The referenced social.json file contains an array of rules: JSON [ { "id": 101, "priority": 1, "action": { "type": "block" }, "condition": { "urlFilter": "||facebook.com", "resourceTypes": ["main_frame"] } } ] Managing Dynamic Rules Programmatically Static rulesets are read-only once compiled into the extension package. To allow users to add custom blocked domains or configure personal schedules, you must update the extension's dynamic rules at runtime. Chrome provides chrome.declarativeNetRequest.updateDynamicRules to modify rules programmatically. This method accepts arrays of rules to remove and rules to add. Here is a JavaScript helper class to manage dynamic site blocking: JavaScript class BlocklistManager { // Add a domain to the dynamic blocklist static async addDomain(ruleId, domain) { const newRule = { id: ruleId, priority: 1, action: { type: 'block' }, condition: { urlFilter: `*://${domain}/*`, resourceTypes: ['main_frame', 'sub_frame'] } }; await chrome.declarativeNetRequest.updateDynamicRules({ removeRuleIds: [ruleId], // Remove old rule with same ID to prevent duplicates addRules: [newRule] }); } // Remove a rule from the active dynamic set static async removeRule(ruleId) { await chrome.declarativeNetRequest.updateDynamicRules({ removeRuleIds: [ruleId] }); } // Retrieve all currently active dynamic rules static async getActiveRules() { return await chrome.declarativeNetRequest.getDynamicRules(); } } Session Rules vs. Dynamic Rules In addition to dynamic rules, Manifest V3 introduces Session Rules via the chrome.declarativeNetRequest.updateSessionRules API. Dynamic Rules: Persist across browser restarts and extension updates. They are stored in Chrome's internal extension storage.Session Rules: Saved purely in memory. They are cleared when the browser session ends, or the extension is reloaded. Session rules are ideal for temporary focus sessions, one-time study blocks, or incognito mode rules that should not write data permanently to the disk. Modifying HTTP Headers The DNR API also supports modifying HTTP request and response headers natively using the modifyHeaders action. This is useful for removing tracking cookies, injecting authentication tokens, or overriding Referrer headers. Here is a rule structure that strips the Cookie header from requests sent to a third-party tracking domain: JSON { "id": 201, "priority": 2, "action": { "type": "modifyHeaders", "requestHeaders": [ { "header": "cookie", "operation": "remove" } ] }, "condition": { "urlFilter": "||tracker-domain.com", "resourceTypes": ["xmlhttprequest", "sub_frame"] } } Platform Constraints and Rule Limits Because the browser must parse and evaluate all active rules in linear time to avoid latency, Google enforces strict limits on the number of rules you can register: Static Rulesets: An extension can declare up to 100 static rulesets, but only a limited number can be enabled simultaneously (typically 50).Dynamic and Session Rules: Extensions are limited to 5,000 dynamic rules and 5,000 session rules.Regex Filter Performance: You can use regular expressions in the regexFilter key under conditions, but the regex patterns must conform to a restricted syntax. Lookaheads, lookbehinds, backreferences, and lazy quantifiers are disabled to guarantee that matching runs in linear time. If a regex pattern is too complex, the API will fail to register the rule. Conclusion and Best Practices When building extensions under Manifest V3: Use Priorities Wisely: Use higher priority values for user-defined whitelists to ensure they override system-level blocklists.Minimize Rule Count: Instead of creating separate rules for sub.domain.com and domain.com, use wildcard patterns or regex expressions to group matches into single rules.Optimize Storage: Clean up unused dynamic rule IDs periodically. Retrieve active rules using getDynamicRules() to prevent collisions. By moving execution to the browser engine, Manifest V3 requires developers to change their approach to web filtering. Designing within these declarative constraints ensures your extension runs efficiently without compromising user privacy.

Model Context Protocol (MCP) servers that work perfectly in development can fail intermittently once they are deployed across multiple replicas behind a load balancer. The failure mode is a stream of "session not found" errors that appear at random, and the cause is a mismatch between how certain MCP transports hold session state and how load balancers distribute requests. This article explains why the problem occurs, when it applies, and a concrete pattern for solving it using a shared session store. The problem is easy to miss in early development because it only appears once there is more than one server instance. A single-instance deployment holds every session in local memory, so every request naturally finds its session. Add replicas, and that assumption quietly breaks. The Failure Pattern Consider a deployment with four MCP server replicas behind a round-robin load balancer, serving agents that connect over the Server-Sent Events (SSE) transport. In this configuration, roughly three out of four follow-up requests fail with a "session not found" error. That ratio is not random. With four replicas and round-robin distribution, a follow-up request has only a one-in-four chance of returning to the replica that created the session. The other three times it lands on a replica that has no record of that session. The reason the failures look random at first is that success depends entirely on which replica the load balancer happens to select. The distribution of failures tracks the replica count directly, which is the clearest signal that the load balancer, not application logic, is the source of the problem. Why MCP Sessions and Load Balancers Conflict Not every MCP deployment has this problem, so it helps to be precise about when it applies. A tools-only MCP server can be stateless. Under the streamable HTTP transport, the client caches tool schemas after discovery, and each tool call is a self-contained request that carries everything the server needs to process it. Any replica can handle any request, and load balancing works without special handling. Two situations make a deployment session-bound. The first is the SSE transport. SSE was the only remote transport available for a long time and remains widely deployed. It is stateful by design: the client opens a long-lived connection that the server holds open as a stream, and the server delivers responses back through that open stream rather than through the response to each individual request. The stream physically lives on one replica. When a follow-up request is routed to a different replica, that replica is not holding the stream and cannot associate the request with the session. The result is the "session not found" error. The second is stateful MCP features. Even on a transport that supports stateless operation, an MCP server that must retain per-client state needs sessions. MCP resource subscriptions that push updates when server-side data changes, long-running operations where a client may disconnect and reconnect expecting to resume, and per-client authorization context established at initialization all require the server to hold state across requests. That state must be reachable regardless of which replica receives the next request. The conflict reduces to a single sentence, which is that the session lives on one replica, but the load balancer distributes requests across all of them. How the Connection Is Established The session originates at connection time. The following example uses the Koog framework to connect an agent to an MCP server over SSE, which illustrates where the session comes from: Kotlin import ai.koog.agents.core.agent.AIAgent import ai.koog.agents.mcp.McpToolRegistryProvider import ai.koog.prompt.executor.llms.all.simpleAnthropicAIExecutor import ai.koog.prompt.llm.AnthropicModels import kotlinx.coroutines.runBlocking fun main() = runBlocking { // Open an SSE transport to the MCP server val transport = McpToolRegistryProvider.defaultSseTransport("http://mcp-server:3000/sse") // Build a tool registry from the tools the MCP server val mcpRegistry = McpToolRegistryProvider.fromTransport( transport = transport, name = "records-client", version = "1.0.0" ) val agent = AIAgent( executor = simpleAnthropicAIExecutor(), llmModel = AnthropicModels.Claude.SONNET, toolRegistry = mcpRegistry ) val result = agent.run("Look up the status of record 12345") println(result) } The relevant detail is the transport and the roles it establishes. The client, Koog in this case, opens the SSE connection, and the session lives on the MCP server. Opening an SSE transport creates a stateful connection: the MCP server creates a session bound to that open stream, and from that point the client and server communicate through a channel anchored to one specific server instance. With a single instance, this is invisible. Behind a load balancer, it is the entire problem. The fix belongs on the server side, not in the client. The Fix: A Shared Session Store The solution is to stop storing session state in a replica's local memory and move it to a shared store that every replica can reach. This is an addition to the MCP server implementation. Neither the MCP specification nor the client library provides a distributed session store; the specification defines that sessions exist but does not prescribe how to persist them across instances, so the server-side session handling is the implementer's responsibility. Redis is a natural fit for this role because the access pattern is a simple keyed lookup and the added latency is negligible relative to the rest of an agent request. The mechanism is straightforward. When any replica creates a session, it writes the session record to the shared store rather than to local memory. When any replica receives a request, it reads the session from the shared store before processing. The session no longer belongs to a replica; it belongs to the store, and every replica can reach it. The session record contains what the server would otherwise hold in memory - the session identifier, the negotiated capabilities, any accumulated per-client state, and timestamps for expiry. Assigning each entry a time-to-live allows idle sessions to expire automatically rather than accumulating. The change in the server's request handling can be reduced to the difference between a local map and a shared lookup: Kotlin // Before: the session lives in this replica's memory. // Other replicas have no record of it. val localSessions = mutableMapOf<String, McpSession>() fun handleRequest(sessionId: String, request: McpRequest): McpResponse { val session = localSessions[sessionId] ?: error("session not found") // fails on any other replica return session.process(request) } Kotlin // After: the session lives in a shared store every replica can read. suspend fun handleRequest(sessionId: String, request: McpRequest): McpResponse { val session = sessionStore.get(sessionId) // shared lookup ?: error("session expired or unknown") val response = session.process(request) sessionStore.put(sessionId, session) // persist any state change return response } The SSE transport adds one further requirement. Because the response must travel back through the stream held by a specific replica, the shared store also records which replica holds the stream, and a publish-subscribe channel routes the response to that replica when a request is handled elsewhere: Kotlin // The replica holding the SSE stream subscribes for its sessions sessionBus.subscribe("mcp:response:$sessionId") { payload -> sseStream.send(payload) } // Any replica that processes a request publishes the response sessionBus.publish("mcp:response:$sessionId", response) In this arrangement, the shared store serves two purposes. It is the session store that allows any replica to handle a request, and it is the message bus that routes each response to the replica holding the open stream. A request may arrive at any replica, while the response is delivered to the connection the client is actually listening on. Why Not Sticky Sessions The most immediate alternative is sticky sessions: configuring the load balancer to pin each client to the replica that created its session. This works and is a reasonable temporary measure, but it carries three drawbacks that make it unsuitable as a durable solution. Sticky sessions undermine load distribution, because a high-volume client is concentrated on a single replica while others remain underused. They reintroduce the single point of failure that multiple replicas were intended to eliminate: if the pinned replica fails, every session on it is lost. And they complicate scaling, because newly added replicas receive no existing traffic and take on load only gradually. A shared session store avoids all three. The load balancer can use plain round-robin distribution. Any replica can fail without affecting sessions held by the others. A new replica can serve existing sessions immediately, because it reads them from the same shared store as every other replica. Results With the shared session store in place, the "session not found" errors are eliminated for active sessions, and requests distribute evenly across replicas. Deliberately terminating a replica no longer interrupts active agents, and their requests are absorbed by the remaining replicas. Adding a replica requires no special handling. The shared lookup adds a small step to each request, but the cost is minor in context. A session read is well under a millisecond, while an agent request already spends hundreds of milliseconds or more on model inference and downstream calls. The overhead is not observable in practice. Summary For teams deploying MCP servers at scale, three points are worth carrying forward. Keep the MCP server stateless where possible. A tools-only server on the streamable HTTP transport scales horizontally without any of this complexity. Sessions should be introduced only when genuinely required, for subscriptions, resumable operations, or server-held per-client context. When sessions are required, do not store them on the replica. Move them to a shared store so that any replica can serve any request. This mirrors the lesson web applications settled on years ago for HTTP session state, now recurring in the context of MCP. Account for the SSE response-routing requirement. A shared session store resolves request handling, but the response must still reach the replica holding the open stream, which a publish-subscribe channel provides. Session persistence behind a load balancer is a common example of the operational gaps teams encounter when deploying MCP in production, and the shared-store pattern described here is a direct and durable solution.

In a previous article, we built a static supply chain graph in Neo4j using Apache Spark, with suppliers, warehouses, distribution centers, and retailers connected by shipping routes. That gave us a snapshot of the network at a point in time. In this article, we'll add the streaming layer: shipment events flow through Confluent Cloud Kafka in real time, land in Neo4j as enriched graph properties, and a live dashboard shows network health updating as events arrive. The full source code is available on GitHub. The Stack Each tool in the stack does what it does best: ToolRoleConfluent Cloud (free tier)Managed Kafka cluster and topicPython producer (Jupyter)Generates and publishes synthetic shipment eventsPython consumer (Jupyter)Consumes events and writes them into Neo4jNeo4j AuraDBGraph database storing the supply chain and shipment eventsPlotlyLive dashboard visualization One deliberate omission is that we aren't using the Neo4j Kafka Sink Connector, which is available as a managed connector on Confluent Cloud. That connector handles the consumer side automatically but carries a per-task hourly charge. For this article, we'll keep everything free by writing a Python consumer that does the same job. This also has a practical benefit: all the pipeline logic is visible in Python rather than hidden inside a managed connector configuration, which makes it easier to understand and adapt. The managed connector is a natural next step for production workloads. Setting Up Confluent Cloud Sign up at confluent.io and create a free cluster.Once the cluster is running, create a topic named shipment-events with 1 partition and default settings.Create an API key and secret under API Keys.Note the bootstrap server address from the cluster settings. Export these as environment variables in your shell: Shell export CONFLUENT_BOOTSTRAP_SERVERS=your_cluster.confluent.cloud:9092 export CONFLUENT_API_KEY=your_api_key export CONFLUENT_API_SECRET=your_api_secret Setting Up Neo4j AuraDB AuraDB is Neo4j's fully managed cloud database. A free tier is available with no credit card required. Sign up at console.neo4j.io/graphacademy.Create a new AuraDB Free instance.When the instance is created, download or note the credentials — the connection URI, username, and password. Neo4j only shows the password once, so save it somewhere safe.Once the instance is running, open the built-in Query tab and verify connectivity: MATCH (n) RETURN count(n). This should return 0. We are ready to load data. Before starting Jupyter, export the connection details as environment variables in your shell: Shell export NEO4J_URI=neo4j+s://xxxx.databases.neo4j.io export NEO4J_USERNAME=your_username_here export NEO4J_PASSWORD=your_password_here export NEO4J_DATABASE=your_database_name_here The Data Model Each shipment event represents a single status update for a shipment at a point in time. A shipment does not generate a sequence of events as it progresses — each event is an independent snapshot, which keeps the producer simple and the consumer stateless. The event structure is: JSON { "shipment_id": "c60eb761-f153-4840-8427-17fa9e34c56c", "supplier_id": "S013", "warehouse_id": "W005", "dist_center_id": "DC004", "retailer_id": "R025", "status": "delayed", "timestamp": "2026-08-04T12:57:15Z", "delay_minutes": 34 } Status follows one of four values — departed, in_transit, delayed or delivered, with a configurable delay probability. We use 15% delayed to make the dashboard interesting without overwhelming it. When the consumer writes an event into Neo4j, it creates a Shipment node and links it to the existing supply chain nodes via four relationship types: Cypher MERGE (sh:Shipment {shipment_id: $shipment_id}) SET sh.status = $status, sh.timestamp = $timestamp, sh.delay_minutes = $delay_minutes WITH sh MATCH (s:Supplier {id: $supplier_id}) MATCH (w:Warehouse {id: $warehouse_id}) MATCH (dc:DistributionCenter {id: $dist_center_id}) MATCH (r:Retailer {id: $retailer_id}) MERGE (s)-[:HAS_SHIPMENT]->(sh) MERGE (sh)-[:VIA_WAREHOUSE]->(w) MERGE (sh)-[:VIA_DIST_CENTER]->(dc) MERGE (sh)-[:DESTINED_FOR]->(r) MERGE on shipment_id means re-running the consumer never creates duplicate nodes. The Producer The producer notebook uses a fixed random seed to generate reproducible shipment events using IDs drawn from the existing supply chain and publishes them to Confluent Cloud via the confluent-kafka library: Python producer = Producer({ "bootstrap.servers": BOOTSTRAP_SERVERS, "security.protocol": "SASL_SSL", "sasl.mechanisms": "PLAIN", "sasl.username": API_KEY, "sasl.password": API_SECRET, "log_level": 0, }) Setting "log_level": 0 suppresses the librdkafka telemetry messages that appear otherwise. The producer supports both batch and continuous modes. For example: Python produce_events(num_events = -1) # stream continuously produce_events(num_events = 100) # publish exactly 100 events The display refreshes every PRINT_EVERY events using clear_output, showing the latest event and a running status breakdown — so the cell output stays manageable even when streaming thousands of events. The Consumer and Live Dashboard Rather than two separate notebooks, we combine the consumer and dashboard into a single pipeline. On each cycle, the loop: Polls Kafka for up to POLL_BATCH events and writes them to Neo4jQueries Neo4j for the current graph stateRebuilds and redraws the dashboardSleeps for REFRESH_INTERVAL seconds before repeating Rebuilding the full dashboard on every cycle is straightforward and works well at demo event rates. At higher throughput, a more efficient approach would be to update only the changed data rather than redrawing all eight panels on each refresh. The consumer uses its own Kafka group ID (supply-chain-dashboard) so it reads the topic independently, catching up on all existing events first before staying live: Python consumer = Consumer({ "bootstrap.servers": BOOTSTRAP_SERVERS, "security.protocol": "SASL_SSL", "sasl.mechanisms": "PLAIN", "sasl.username": API_KEY, "sasl.password": API_SECRET, "group.id": "supply-chain-dashboard", "auto.offset.reset": "earliest", "log_level": 0, }) The Live Dashboard The dashboard uses Plotly's make_subplots in a 4x2 grid, rebuilt on every refresh cycle using clear_output. Eight panels give a complete picture of network health: Row 1 – Overall Health Network status table: Total shipments, delayed count, delay rate, Kafka events consumed, refresh count, and any disabled nodesShipment status distribution: Donut chart showing the split between departed, in transit, delayed, and delivered, as shown in Figure 1 Figure 1. Shipment Status Distribution Row 2 – Warehouse View Delayed shipments by warehouse: Which warehouses are handling the most delayed shipments right nowWarehouse health score: A heatmap scoring each warehouse from 0.0 (everything delayed) to 1.0 (fully healthy), colored red through orange to green, as shown in Figure 2 Figure 2. Warehouse Health Score Row 3 – Origin and Destination Supplier performance: Which suppliers are generating the most delayed shipmentsRetailer impact: Which retailers are receiving the most delayed shipments — the downstream effect of any disruption Row 4 – Mid-Network and Flow Average delay by distribution center: Where in the middle layer delays are accumulatingShipment flow: A Sankey diagram (Figure 3) showing which suppliers are routing through which warehouses Figure 3. Shipment Flow - Suppliers to Warehouses The warehouse health score is the most immediately readable panel. The Cypher behind it computes the score directly in the graph: Cypher MATCH (sh:Shipment)-[:VIA_WAREHOUSE]->(w:Warehouse) WHERE w.active IS NULL OR w.active <> false WITH w.id AS warehouse, count(sh) AS total, count(CASE WHEN sh.status = 'delayed' THEN 1 END) AS delayed RETURN warehouse, round(1.0 - toFloat(delayed) / total, 3) AS health_score ORDER BY warehouse Simulating a Network Disruption One of the more compelling features of the graph model is how easy it is to simulate and visualize a disruption. Setting active = false on any node excludes it from the dashboard queries and the dashboard immediately reflects the simulated disruption on the next refresh cycle. We can do this before the dashboard starts: Python REMOVE_NODE = "W007" # mark this warehouse as inactive Or live, while the dashboard is running, using the Neo4j AuraDB Query tab: Cypher // Disable a node MATCH (n {id: "W007"}) SET n.active = false // Re-enable a node MATCH (n {id: "W007"}) REMOVE n.active // Check what is currently disabled MATCH (n) WHERE n.active = false RETURN labels(n)[0] AS label, n.id AS id Within 5 seconds, the dashboard reflects the change. The warehouse health heatmap shows the gap, the delayed shipments bar shifts to other warehouses as traffic reroutes, and the network status table shows the node as disabled. Re-enabling it and watching the metrics recover completes the disruption and recovery story. Standalone Operation At startup, the consumer notebook creates the supply chain nodes using MERGE. This operation is idempotent, so any existing nodes from the previous article are left unchanged. Note that this step creates nodes only — the relationships between supply chain nodes (supplier -> warehouse -> distribution center -> retailer) are assumed to exist from the previous article, or can be added separately if running this notebook in isolation. Python with driver.session(database = NEO4J_DATABASE) as session: for i in range(20): session.run("MERGE (:Supplier {id: $id})", id = f"S{i:03d}") for i in range(12): session.run("MERGE (:Warehouse {id: $id})", id = f"W{i:03d}") for i in range(10): session.run("MERGE (:DistributionCenter {id: $id})", id = f"DC{i:03d}") for i in range(30): session.run("MERGE (:Retailer {id: $id})", id = f"R{i:03d}") Gotchas and Lessons Learned Suppress librdkafka Logging Without "log_level": 0 in the producer and consumer config, Confluent's underlying librdkafka library prints telemetry messages to the cell output every time a connection is established. The messages are harmless. Suppress Neo4j Property Warnings Querying a property that does not yet exist on any node produces a GqlStatusObject warning from Neo4j for every query that references it. The active property falls into this category when no node has been disabled. The fix is one line to set notifications to "OFF" on the driver, as follows: Python driver = GraphDatabase.driver( NEO4J_URI, auth = (NEO4J_USERNAME, NEO4J_PASSWORD), notifications_min_severity = "OFF", ) Consumer Group Isolation Kafka distributes partitions across consumers in the same group, so each consumer processes only its assigned partitions. If we run multiple consumers using the same group ID against the same topic, each will only process a subset of the events. The dashboard uses supply-chain-dashboard as its group ID, and the tip is to run only one instance of this notebook at a time against the same topic and cluster. auto.offset.reset = earliest Without this setting, a consumer that starts after events have been published will miss everything that arrived before it connected. Setting earliest means the consumer always catches up on the full history of the topic before going live, which is essential if we stop and restart the dashboard mid-session. Clear Shipment Nodes Between Runs Each run of the consumer creates new Shipment nodes. Since the producer generates synthetic demo data, it's safe to clear these between runs; otherwise, successive runs would accumulate all historical shipments, and the dashboard counts would grow unbounded. The notebook clears all Shipment nodes at startup: Cypher MATCH (sh:Shipment) CALL (sh) { DETACH DELETE sh } IN TRANSACTIONS OF 10000 ROWS Summary We've built a real-time supply chain event streaming pipeline using Confluent Cloud Kafka and Neo4j. The producer generates synthetic shipment events continuously, the consumer writes them into the graph, and a live dashboard shows network health updating in near real-time. The disruption simulation — marking a node inactive mid-run and watching the dashboard respond — demonstrates one of the most compelling aspects of the graph model: the ability to ask structural questions about a network as it evolves. The same architecture adapts naturally to real logistics, IoT, or manufacturing event streams where understanding network structure matters as much as raw throughput. The full source code is available on GitHub.

By Akmal Chaudhri CORE