Culture and Methodologies

AI coding assistants are becoming increasingly capable at generating code, explaining systems, and accelerating development workflows. But in real engineering environments, the biggest blocker is often not the model’s ability to write code. The bigger issue is whether the assistant has the right context before it starts making changes. A developer rarely works from a single source of truth. A Jira ticket may describe the implementation task. A Google Doc may contain the detailed requirements. A slide deck may explain the business goal. A meeting summary may include key decisions, open questions, and next steps that never made it back into the ticket. For a human developer, this creates friction. For an AI coding assistant, it creates risk. The assistant may generate code that looks correct, passes basic syntax checks, and follows existing patterns - but still implements the wrong behavior because the actual feature context was fragmented across multiple places. This is where a PARA-style context workspace becomes useful. PARA - Projects, Areas, Resources, and Archives is commonly used to organize knowledge by actionability. Applied to AI-assisted software development, it can become a practical architecture pattern for preparing scattered engineering knowledge before an AI coding assistant touches code. The goal is not to dump every document into the model. The goal is to organize scattered context so the assistant can reason with the right information for the task. The Problem: AI Coding Assistants Often See Only Part of the Work Consider a developer asked to build a new data pipeline that calculates a generic quality score. The implementation sounds straightforward: Build a pipeline that joins multiple input tables, applies business rules, and produces a quality score output table. But the actual context may be spread across several sources: SourceWhat It May ContainTicketImplementation scope, acceptance criteria, due dateRequirements docBusiness rules, scoring logic, data definitionsSlide deckBusiness goal, stakeholder alignment, expected impactMeeting summaryFinal decisions, open questions, changed thresholdsExisting codePipeline patterns, naming conventions, dependency structureOlder documentsPrevious decisions, deprecated approaches, known constraints If the AI coding assistant only sees the ticket, it may miss the deeper context needed to implement the feature correctly. This is especially risky for data pipelines and analytics features, where correctness depends not only on code structure but also on interpretation: which source tables to use, how freshness should be handled, how business rules are applied, and how downstream consumers will use the output. What Can Go Wrong If the Agent Only Reads the Ticket? A ticket often captures the visible work, but not the full reasoning behind the work. If the assistant only uses the ticket, it may: Implement the task but miss business rules from the requirements documentIgnore key decisions captured in meeting summariesUse a technically available source table that is not the approved source for this featureMiss freshness expectations for the output tableProduce a score that does not match how downstream dashboards or reports will consume itFollow an outdated implementation pattern because it found old but similar codeGenerate a pull request that looks reasonable but fails product or data-quality expectations This is the core issue: The AI assistant may know how to write code, but it may not know which code should be written. That distinction matters. For coding agents to become more reliable, developers need a better way to prepare context before code generation begins. Reframing PARA for AI Coding Agents PARA can be adapted from a personal knowledge organization method into a context classification pattern for AI-assisted development. In a PARA-style context workspace: PARA CategoryEngineering MeaningAgent Context RoleProjectsActive work being deliveredCurrent feature scope, ticket, task goalAreasOngoing responsibilitiesStandards, ownership, governance, quality expectationsResourcesReusable knowledgeDocs, runbooks, design patterns, pipeline examplesArchivesCompleted or inactive knowledgeHistorical decisions, old approaches, past incidents This structure helps the AI assistant understand the role of each piece of information. A current requirement should not be treated the same way as an old design decision. A meeting decision should not be buried behind a generic document search. A reusable pipeline pattern should be available to guide implementation, while archived material should be used carefully as historical context. The value of PARA is not just an organization. It gives the assistant a way to distinguish between active task context, long-running rules, reusable references, and historical information. This flow changes how the assistant approaches implementation. Instead of asking: “What code should I generate from this ticket?” The assistant can reason from a richer question: “What is the active feature goal, what rules must be followed, what reusable references apply, and what historical context should be considered before changing code?” That shift is small, but important. Applying PARA to a Quality Score Pipeline Now apply this to the quality score pipeline example. The feature requires a pipeline that joins multiple input tables, applies business rules, and writes a quality score output table. The exact business logic is intentionally generic, but the pattern is common across analytics engineering, data engineering, machine learning platforms, and reporting systems. A PARA-style workspace could organize the context like this: Project Context This is the active feature work. It may include: The current ticketFeature scopeAcceptance criteriaCurrent implementation statusTarget output tableExpected delivery milestoneKnown blockers or open questions For the coding assistant, this answers: “What am I being asked to build right now?” Area Context This represents ongoing expectations that apply beyond this one feature. It may include: Data quality standardsFreshness expectationsOwnership rulesPrivacy or compliance constraintsNaming conventionsRelease processTesting expectations For the coding assistant, this answers: “What rules and standards must this implementation follow?” Resource Context This is reusable technical knowledge. It may include: Existing pipeline patternsSimilar transformation logicData model documentationDashboard dependency notesCommon test patternsRunbooksData validation examples For the coding assistant, this answers: “What reusable references should guide the implementation?” Archive Context This is historical information that may still be useful, but should not automatically drive the implementation. It may include: Older design decisionsDeprecated scoring logicPast pipeline migrationsPrevious quality metric experimentsHistorical meeting notesOld RCA or incident learnings For the coding assistant, this answers: “What historical context may explain why the system works this way?” The critical point is that archived context should be used for awareness, not blindly copied into the current implementation. Why Meeting Summaries Matter Meeting summaries are often underestimated in AI-assisted development. In many teams, the final decision is not always reflected immediately in the ticket or requirements document. A meeting summary may contain important details such as: A threshold was changed after stakeholder discussionA source table was rejected because of data freshness concernsA metric definition was clarifiedA downstream dashboard dependency was identifiedA launch decision was postponedAn open question was assigned to another teamA temporary workaround was approved only for the first release For a human developer, these details may be remembered from the meeting. For an AI coding assistant, they are invisible unless they are included in context. This is one reason a PARA-style workspace can be valuable. It gives meeting summaries a place in the feature context without treating them as random notes. A meeting summary tied to an active feature belongs in the Project context. A recurring decision about data freshness may become the Area context. A reusable explanation of metric calculation may become the Resource context. Once the feature is complete, the same meeting summary may eventually move into the Archive context. How the Coding Assistant Should Use Context Before Changing Code Before generating code, the AI coding assistant should use the structured context to form an implementation understanding. For a quality score pipeline, it should first understand: What the feature is trying to accomplishWhich input data sources are approvedWhich business rules define the scoreWhich decisions were finalized in meetingsWhat freshness or latency expectations existWhich existing pipeline patterns should be followedWhat downstream dashboards, reports, or consumers depend on the outputWhich historical approaches should be avoided Only after that should it propose an implementation plan or modify code. This changes the assistant’s role. It is no longer simply a code generator responding to a ticket. It becomes a context-aware engineering assistant that can reason across requirements, decisions, standards, and existing system patterns. The Bigger Shift: From Prompting to Context Preparation Prompting is still useful, but it is not enough for complex engineering work. A good prompt cannot fully compensate for missing requirements, outdated context, or scattered decisions. For AI coding assistants, the quality of the result depends heavily on the quality of the context that comes before the prompt. This is especially true when the task involves business logic, analytics definitions, data contracts, or cross-team decisions. In those cases, the question is not: “How do we write a better prompt?” The better question is: “How do we prepare the right engineering context before asking the assistant to write code?” For developers building with AI coding agents, this may become one of the most important habits: do not ask the agent to write code first. Prepare the context first. Because the future of AI-assisted development will not belong only to teams with the most powerful coding models. It will belong to teams that know how to structure knowledge so those models can make better engineering decisions.

For years, search technology meant one thing: type in a keyword, and the system goes hunting for an exact match. That works fine for product SKUs or error codes, but it falls apart the moment someone asks a real question. If your knowledge base is full of manuals, support tickets, transcripts, and reports, a person searching for "why does the machine shut down during startup" shouldn't have to guess the exact phrase the original author used. This is the gap that vector search closes. Instead of matching words, it matches meaning. And on Databricks, building this kind of system is more accessible than most teams expect, once you understand the moving pieces. Why Vector Databases Work Differently A vector database doesn't store text the way a traditional database does. It stores text as numbers, specifically, as long lists of numerical values that represent the meaning of a piece of content. Two sentences that say the same thing in different words end up with similar number patterns, even if they don't share a single word in common. This unlocks three distinct ways of searching: Similarity search finds content that's conceptually related, even when the wording is completely different. Hybrid search blends that conceptual matching with traditional keyword scoring, so you get the best of both worlds. Full-text search sticks to exact matches, which still matters when precision is non-negotiable. Together, these give developers the tools to build something that feels less like a search box and more like a colleague who actually understands what you're asking. Getting Your Data Ready Before any of this works, your data needs to be in the right shape. On Databricks, that means your source table needs Change Data Feed turned on. Think of this as a way for the vector index to "listen" for changes, so when documents get updated, added, or removed, the index stays in sync automatically rather than going stale. You'll also need a unique identifier for every row. This becomes the primary key that ties each chunk of text back to its source, which matters later when you're filtering or tracing results back to the original document. Turning Text Into Embeddings Embeddings are the numerical fingerprints mentioned earlier, the representations that let the system compare meaning instead of matching strings. Databricks gives you two paths here. With managed embeddings, Databricks handles the entire process: it generates the embeddings and keeps them updated as your data changes. With manual embeddings, you generate them yourself using an external tool and store the results in a column. For the vast majority of projects, managed embeddings are the easier and more reliable choice. There's less to maintain, and compatibility with the platform is guaranteed out of the box. One question that comes up constantly: what does it mean when someone says an embedding has 1,024 dimensions? It simply means each chunk of text is represented by 1,024 numbers. That number isn't arbitrary; it's baked into whichever embedding model you choose, such as GTE-large. If you want a different dimensionality, you'd need to switch models entirely; it's not a setting you can tweak independently. Building the Index Once your embeddings are in place, you create the actual vector search index. Databricks gives you two routes: the SDK, using the databricks-vectorsearch library for programmatic, repeatable setups, or the UI, which walks you through configuration visually. A few decisions matter most here. The index type determines whether you're doing pure semantic search or hybrid search; for most real-world use cases, hybrid is the safer default since it catches both conceptual matches and exact terminology. The embedding model, like databricks-gte-large-en, determines how your text gets converted into vectors. And the sync mode controls how fresh your index stays: continuous sync keeps things updated automatically, while triggered sync gives you manual control over when refreshes happen. Choosing the Right Search Method With the index built, you have three retrieval modes to choose from, and picking the right one depends entirely on what your users are asking. Similarity search shines when people ask natural-language questions or when the same concept might be described using different terminology across documents. Hybrid search becomes valuable when domain-specific terms carry real weight, think compliance codes or technical standards like ISO 13849-1, where an exact match matters just as much as conceptual relevance. Full-text search is your fallback when precision trumps everything else, and you need exact keyword hits, no exceptions. Don't Skip Metadata Filtering Here's a piece of advice that's easy to overlook: don't make your search work harder than it needs to. If a user only cares about PDFs from the last quarter, let the system know that upfront. Filtering by document path, page number ranges, or document type narrows the search space before the heavy lifting even starts. The result is faster queries and more relevant results, because the system isn't wasting effort sifting through content that was never going to be useful anyway. When Re-Ranking Earns Its Keep Sometimes the top results from a semantic search are technically "close" in meaning but miss the point of the question. That's where re-ranking comes in, a second pass that re-scores your top candidates using something more sophisticated, like a cross-encoder or an LLM. This extra step is worth the computational cost when queries are nuanced, when domain context really matters, or when the stakes for getting the right answer are high. It's not something you need everywhere, but used selectively, it can be the difference between a good answer and the right one. A Few Practical Tips A handful of best practices can save you headaches down the road. Don't over-invest in embedding dimensionality. If a smaller model performs nearly as well as a larger one, take the smaller one and enjoy the lower latency. Keep your num_results parameter reasonable; pulling back 10 to 100 results is usually plenty, and larger sets just slow things down. Match your endpoint SKU to your scale; standard tiers work fine under roughly 2 million vectors, while storage-optimized tiers make sense beyond that. And lean on metadata filters wherever possible; they're one of the simplest ways to boost both speed and relevance. The Bigger Picture Vector search isn't just a buzzword bolted onto a database. It's the connective tissue between how humans naturally ask questions and how systems find answers. Get the fundamentals right- solid embeddings, a well-configured index, smart filtering, and selective re-ranking- and you're not just building a search feature. You're building something that genuinely understands what people are looking for.

TL;DR: The AI Definition of Done Your team has a Definition of Done for a product increment. It has none for the 20-plus AI-supported outputs that leave the team each week: status reports, stakeholder emails, release notes, and updates for the C-level. Each one carries your team’s name. “I know quality when I see it” is the standard most teams actually run by, and you cannot audit it, teach it to a new colleague, or defend it when a claim turns out to be wrong. The AI Definition of Done fixes that with one page per task class, agreed by the team, before the output ships. Your Increment Has a Standard; Does Your AI Output? A model turns the Jira board into a Friday status update, and the update tells an enterprise prospect that the security feature is in production. Unfortunately, it is not. The feature was descoped three months ago, but the old ticket title persisted because no one felt responsible. So the model reported the title instead of the reality. Nobody checked the claim against the release notes because nobody had agreed that someone should. The email was sent with the team’s name on the cover. A functioning agile team should be able to tell you what “done” means for a product increment. Few can tell you what “done” means for that status update. No agreed standard governs it, and it ships every week. The product increment passes through a standard that the team argued over and agreed on. The AI-assisted output passes through one person’s gut feeling at the moment they clicked send. One of those you can defend to a stakeholder, an auditor, or a new hire. The other you cannot. The AI Definition of Done closes that gap without adding a governance department, which is exactly why it survives in organizations where “AI governance” earns eye rolls. It takes a practice every agile practitioner already owns and points it at the work you have started handing to a model. It is not for everything: skip it for private brainstorming, throwaway prompts, or personal sensemaking, unless the output later informs a decision or leaves the team. The Four Questions Every AI Definition of Done Answers The Concept Verification Level Which claims get checked, by whom, against what source, and how? “Looks good” is not a method. A method names the claim, the checker, the source, and the test: every factual claim about product status gets checked against the release notes by the sender before sending, every time. Where teams get stuck: approval gets mistaken for review. Someone skims a draft, clicks send, and the team’s name now sits on a claim nobody verified. Provenance Disclosure What does the team declare about how the output was produced? Three labels cover practice: a) Human means no material AI contribution to the content, claims, or structure (a spellchecker does not count), b) AI-assisted means AI contributed to drafting, summarizing, or analysis, and a named human reviewed the output and decided, and c) AI-automated means AI produced and sent the output under predefined rules, without human review before release, audited at a set cadence. The line that matters runs through “reviewed”: clicking send on an unread draft is approval, never review. An output approved without reading is AI-automated, whatever the team tells itself. Data Hygiene What never enters a model on the way to this output? Name the exclusions concretely: personal data from team surveys, customer-identifiable information, anything your organization’s AI policy restricts. If the input rules in your A3 Handoff Canvas already cover this, point to them. Do not keep two versions of the same rule. Where teams get stuck: nobody wrote the exclusions down, so each person guesses, and the guesses differ. Sufficiency Tier and Environment Which model, plan, and data boundary are good enough for this task class, and why? A top-notch frontier model drafting calendar invitation may fail in this regard. The cheapest model, run locally on an old Mac mini, can write a board update but likely fails in the other. Capability is only half of it: a board update may need an enterprise plan with a no-training guarantee or an approved connector, even when a mid-tier model is plenty. If your team has a routing policy, point to the tier and the environment it mandates. If it does not yet, name the model and the plan, and explain in one sentence why both are enough. The AI Definition of Done Template Four questions, plus two operating controls, one page. Here is the template a team fills in per task class: DimensionYour Standard for This Task ClassTask classVerification level: What is checked, by whom, against what, howProvenance label: Human (Avoid) / Assist / Automate from the A3 Delegation Framework, and where the label appearsData hygiene: What never enters the modelSufficiency tier and environment: Wich model, plan, and data boundary, and why they are enoughSign-off: Who agreed, on what date, and the review dateStop rule: When the delegation is paused, downgraded, or returned to manual work The last two rows are operational, not definitional: Sign-off records who agreed and when, and the stop rule names the condition that pauses the delegation, because this standard should say not only when an output may ship but when the task class stops being eligible for AI at all. Without it, teams keep tuning the prompt or skill long after the delegation has proven unfit. A Worked Example: External Status Communication The status update failure that opened this article maps to one task class, status communication, leaving the company. Here is the team’s first AI Definition of Done for it: DimensionStandardTask classStatus communication leaving the companyVerification levelEvery claim about feature status is checked against the release notes by the sending manager, before sending, every timeProvenance labelAI-assisted; footer states “Drafted with AI, reviewed by [name]”; Assist is not permitted for this task classData hygieneNo customer names, no security-finding details, no internal financials enter the modelSufficiency tier and environmentMid-tier model on an enterprise plan with no model training; drafting from structured release data needs no frontier modelSign-offTeam agreed, dated; review after the next four status updatesStop ruleIf two updates in a review cycle need a factual correction after sending, the task class returns to manual drafting until the standard is revised The standard costs the sending manager about four minutes a week, set against an error that can put a flagship deal at risk. Write Your AI Definition of Done in 75 Minutes An AI Definition of Done that one person downloads and pastes into the wiki doesn’t change anything. The argument over the standard is where the standard takes hold. Run it as a workshop: Pick three task classes (10 minutes): Choose from work the team actually shipped in the last two weeks, never hypotheticals. The best candidates are outputs that leave the team.Draft in pairs (20 minutes): Each pair fills the template for one task class. Pairs work without comparing notes; divergence is the point.Argue the differences (25 minutes): Compare drafts. Where pairs disagree on verification level or provenance, the team has found an unspoken assumption. Resolve each disagreement with a decision, never with “both are fine.”Set the labels (10 minutes): Agree where provenance labels appear: email footers, document headers, report covers. Visible beats buried.Adopt and date (10 minutes): Sign off each AI Definition of Done with a review date, and add the adoption to your AI working agreement. Ownership stays with the team running the delegation. Compliance, security, or legal may constrain the standard, but they do not write it for the team. When someone says, “We do not need this for internal outputs,” ask what happened the last time an internal draft got forwarded outside the team. Every team has that story. The Record You Get for Free Each signed-off AI Definition of Done is a dated, versioned, one-page record. Stack them, and they answer the due diligence question enterprise buyers increasingly ask, “How do you control AI-generated output?” with documents instead of assurances. Nobody wrote a governance report. The records came out of normal work. That answer is already part of procurement and due diligence conversations. Article 4 of the EU AI Act has been applied since February 2, 2025, and requires providers and deployers to ensure a sufficient level of AI literacy among staff and others operating AI systems on their behalf. The EU Commission’s Q&A places supervision and enforcement under national market surveillance authorities, with the enforcement rules applying from early August 2026. The practical question underlying the regulation is simpler, and a prospect’s procurement team will ask it before any regulator does: can you show the standard that underlies the output you sent us? Three Ways It Fails The downloaded standard: A template adopted without the workshop. Nobody argued, so nobody owns it. An AI Definition of Done that nobody argued about is one nobody will follow. The universal standard: One AI Definition of Done for all work. Verification that aligns with external communication suffocates internal brainstorming, and the team abandons the practice within a month. One page per task class. Contrary to the classic Definition of Done, there is no one-size-fits-all in our use case. The static standard: Written once, reviewed never. Models change, people change, task classes change. The review date is part of the artifact, and your next delegation inspection enforces it. Conclusion: Pick One Output This Week Pick one AI-assisted output your team ships regularly. The Friday status update, the Sprint summary, or the stakeholder email. Walk it through the four questions out loud in your next Retrospective: what gets checked and by whom, how we label it, what never enters the model, and which tier is enough. You will likely find at least one question where the honest answer is “nobody decided that.” Write the one-page response for that task class, argue it, sign it, and date it. One standard, agreed by the team, is the difference between a team that uses AI and a team that a customer can trust with it. Which of your AI-assisted outputs has a standard behind it right now, and which one is merely a habit? Key Questions This Article Answers What Is an AI Definition of Done? An AI Definition of Done is a one-page, team-agreed standard that an AI-assisted output must meet before it leaves the team. Teams write one per task class, such as external status communication or data analysis summaries, never one per task. It answers four questions: what gets verified, how the output is labeled, what data never enters the model, and which model and environment are sufficient. It borrows the discipline of the Scrum Definition of Done and applies it to work on a model touched. What Is the Difference Between Approval and Review for AI Output? Review means a named human reads the AI-generated output and checks its claims against a source before it ships. Approval means someone clicked send. Clicking send on an unread draft is approval, not review, whatever the team calls it. An output approved without reading is effectively AI-automated, and it should carry that provenance label rather than the AI-assisted label, which implies a human verified it. How Do You Write an AI Definition of Done? Run a 75-minute team workshop, not a solo download. Pick three task classes from work shipped in the last two weeks, draft the standard in pairs, then compare and resolve every disagreement with a decision. Agree where provenance labels appear, set a stop rule that returns the task class to manual drafting when outputs repeatedly fail, sign off each standard with a review date, and add the adoption to your AI working agreement. The argument over the standard is what makes the team own it. How Do Agile Teams Prove They Govern AI Output? Each signed-off AI Definition of Done is a dated, one-page record. Together, a team’s standards answer the procurement and due diligence question “how do you control AI-generated output” with documents rather than assurances. The records are a byproduct of normal work, so no separate governance report is needed. This matters because buyers and regulators, including under the EU AI Act Article 4, increasingly require evidence of controlled AI adoption. What Are the Four Dimensions of an AI Definition of Done? Verification level (which claims get checked, by whom, against what source, and how), provenance disclosure (Human, AI-assisted, or AI-automated, and where the label appears), data hygiene (what never enters the model), and sufficiency tier and environment (which model, plan, and data boundary are good enough and why). Each dimension fits on one line of a one-page template, signed off with an adoption date and a stop rule that pauses the delegation when outputs repeatedly fail.

By Stefan Wolpers CORE

Long-running, distributed business processes often require careful coordination, state management, and fault handling. Temporal offers a code-first approach to durable workflows: developers write ordinary code for orchestration, and the Temporal service persists state, retries failed tasks, and resumes execution after failures. This shifts focus from plumbing (queues, retries, timeouts) to domain logic, but it also encourages reuse of proven patterns. The Temporal community and documentation highlight several orchestration patterns — for example, sagas, state machines/actors, polling strategies, fan-out/fan-in, and versioning patterns — that solve recurring problems in workflow design. This article surveys these patterns, explaining when and how to use them, with concise code snippets to illustrate their implementation in Temporal. A classic pattern in distributed transactions is the Saga (compensating transaction). In a saga, a business process is broken into a sequence of steps, each with its own “undo” compensation. If any step fails, the saga executes compensations in reverse order to restore consistency. In Temporal, this maps naturally to a try/catch around activities or to the built-in Saga helper. For example, a vacation booking workflow might book a hotel, then a flight, then an excursion. Each step registers a compensation action before invoking the activity. If a failure occurs, the catch block calls saga.compensate() to run all registered compensations in reverse. The following Java-like snippet shows this approach: Java public void bookVacation(BookingInfo info) { Saga saga = new Saga(new Saga.Options.Builder().build()); try { saga.addCompensation(activities::cancelHotel, info.getClientId()); activities.bookHotel(info); saga.addCompensation(activities::cancelFlight, info.getClientId()); activities.bookFlight(info); saga.addCompensation(activities::cancelExcursion, info.getClientId()); activities.bookExcursion(info); // If all succeed, method returns normally. } catch (TemporalFailure e) { saga.compensate(); // undo previous steps throw e; // propagate failure } } If any book* activity throws an exception, the catch invokes saga.compensate(), which calls cancelExcursion, cancelFlight, and cancelHotel in reverse order. This pattern ensures that even if the workflow crashes after partial work, Temporal’s durable execution will eventually resume the compensation sequence. Because Temporal workflows are persistent, the saga logic itself is recoverable – the service records each step and its compensation in the history. In effect, workflows become distributed state machines where try/catch embodies the saga pattern. Polling and External Events Workflows often need to wait for external processes or inputs. In Temporal, there are two main polling strategies. Frequent polling (short interval) is implemented inside an activity loop: the activity repeatedly attempts a call, sleeps briefly, and heartbeats after each iteration. Because long-running activities must heartbeat to show liveness, the loop invokes Activity.getExecutionContext().heartbeat(null) each cycle. For example, a polling activity might look like this: Java @Override public String doPoll() { ActivityExecutionContext context = Activity.getExecutionContext(); while (true) { try { return service.getServiceResult(); } catch (TestServiceException e) { // Service not ready; will retry } // Heartbeat to prevent timeout, then sleep briefly context.heartbeat(null); sleep(POLL_DURATION_SECONDS); } } In this snippet, service.getServiceResult() is retried until it succeeds. Each loop iteration heartbeats and sleeps for a fixed interval. If the worker or service crashes, Temporal will resume the loop exactly where it left off. This pattern is ideal for rapid retries or waiting on resources that become available shortly. For infrequent polling, Temporal relies on activity retry options instead of a custom loop. A workflow can call an activity once, but configure its retry backoff so that failures trigger re-execution after longer delays. In practice, one sets a high initial retry interval and backoff coefficient in the ActivityOptions at workflow time. The workflow code itself is just a single activity call (no loop needed). If the activity throws an error, Temporal automatically retries it later, waiting longer each time. This approach leverages the built-in retry policy (e.g., exponential backoff) for occasional checks. To handle arbitrary external signals or time delays, Temporal workflows can also use Workflow.await(timeout, condition) or Workflow.newTimer(). For instance, a workflow might await a boolean flag that is set by a signal handler, or await a fixed timeout for human input. This avoids busy-wait loops at the workflow level. Signals themselves can come at any time; Temporal’s messaging system lets running workflows be interrupted by signals without polling. In short, Temporal workflows mix timers (Workflow.await) and external signals to wait efficiently. Frequent polling lives in an activity with heartbeats, whereas infrequent or one-off waits can use activity retry or workflow timers. Parallel and Batch Processing When processing large data sets or issuing many operations in parallel, Temporal’s fan-out/fan-in pattern is useful. A parent workflow can spawn multiple child workflows or activities concurrently and then wait for all to complete. This is commonly used for batch jobs, bulk queries, or any parallel computations. The following example shows a “page-by-page” batch processing workflow. For each batch of records, it spawns a child workflow per record and then uses Promise.allOf() to wait for all children. When a batch is done, it can optionally continue-as-new to process the next page without growing history indefinitely: Java @Override public int processBatch(int pageSize, int offset) { List<SingleRecord> records = recordLoader.getRecords(pageSize, offset); List<Promise<Void>> results = new ArrayList<>(); for (SingleRecord record : records) { String childId = Workflow.getInfo().getWorkflowId() + "/" + record.getId(); RecordProcessorWorkflow processor = Workflow.newChildWorkflowStub(RecordProcessorWorkflow.class, ChildWorkflowOptions.newBuilder().setWorkflowId(childId).build()); results.add(Async.procedure(processor::processRecord, record)); } // Wait for all child workflows to finish Promise.allOf(results).get(); // If no more records, return result and finish if (records.isEmpty()) { return offset; } // Otherwise continue as new for the next batch (to reset history) return nextRun.processBatch(pageSize, offset + records.size()); } In this code, each child workflow processes one record. The parent collects a list of Promise<Void> and calls Promise.allOf(...).get(), which blocks the parent until all child workflows complete. Using children allows highly parallel processing without overloading a single worker. After finishing a batch, the code checks if (records.isEmpty()) and returns; otherwise it calls a continueAsNew stub (nextRun) with an updated offset. This continueAsNew effectively starts a fresh workflow execution with a new history, avoiding unbounded history growth for long-running loops. As shown, Temporal’s Async and Promise primitives make parallel fan-out/fan-in straightforward. Beyond paging, fan-out can apply to any use case needing parallel work (bulk updates, scatter-gather queries, etc.). Conversely, gathering results into a list or aggregation is just collecting activity/child results into a shared variable, which Temporal safely persists in the history. Actor-Like Workflows and Event-Driven Patterns Temporal workflows are naturally stateful and can run indefinitely, making them suitable for actor or process-manager patterns. A workflow can “sleep” or wait for signals, maintain in-memory state, and react to external events. Clients can use signals (@SignalMethod) to send events into a running workflow and queries (@QueryMethod) to read its state without affecting it. This allows workflows to act like autonomous entities. For example, imagine a subscription service workflow. It starts with a customer on trial, waits for either trial expiration or a cancellation signal, then proceeds to billing periods. Signals like cancelSubscription() can interrupt the main flow. Meanwhile, queries like queryCustomerId() can retrieve the workflow’s state from outside. Temporal’s event system handles all this without polling: “a running workflow can receive external messages without polling, and clients can inspect workflow state at any time”. Internally, the workflow code can use Workflow.await(...) to pause until a signal sets a flag. Here’s a conceptual sketch (TypeScript/JavaScript style) of using signal and query definitions: TypeScript const abortSignal = defineSignal<[string]>('abort'); const updateSignal = defineSignal<[number]>('update'); const getStateQuery = defineQuery<State>('getState'); export async function statefulWorkflow(config: Config): Promise<Result> { let state: State = {...initial...}; let aborted = false; setHandler(abortSignal, (reason: string) => { aborted = true; }); setHandler(getStateQuery, () => state); // Main workflow logic: await condition(() => aborted, '1 minute'); if (aborted) { // cleanup or compensation return { status: 'aborted' }; } // ... continue normal processing return { status: 'completed' }; } In this pattern, external callers would workflow.signal(abortSignal, reason) or workflow.query(getStateQuery). Temporal’s signal-and-query features implement a process manager-style pattern: a workflow can behave like an event-driven state machine, reacting to signals in real time and allowing external inspection. This is more robust than polling, and since all state changes happen in the workflow code, consistency is guaranteed. (If a query is issued while the workflow is mid-activity, it will reflect the last completed state.) Note that newer Temporal releases also support Workflow Updates, which are like synchronous signals that can return values. In environments where Update is available, a workflow can reply to a message directly. Otherwise, a client can query state as a two-step “signal then query” process. Either way, this pattern empowers long-lived processes and human-in-the-loop steps. Versioning and Evolving Workflows Temporal requires workflow code to be deterministic, so changing logic in running workflows must be done carefully. The community and docs describe versioning strategies. For short-lived or rare workflows, one can deploy a new workflow definition (e.g. MyWorkflowV2) or use a new task queue for new versions. For long-lived workflows, Temporal’s Workflow.getVersion API lets the code branch on a version number recorded in the history. This is often called the “patch” strategy. For example: Java int version = Workflow.getVersion("checksumAdded", Workflow.DEFAULT_VERSION, 1); if (version == Workflow.DEFAULT_VERSION) { activities.upload(targetBucket, targetFilename, data); } else { long checksum = activities.calculateChecksum(data); activities.uploadWithChecksum(targetBucket, targetFilename, data, checksum); } Here, on first execution getVersion("checksumAdded", DEFAULT, 1) returns DEFAULT_VERSION and runs the original upload() call. When a new worker with updated code runs getVersion("checksumAdded", DEFAULT, 1) again, Temporal records version = 1 in the history. Future runs hit the else branch and use the new uploadWithChecksum() code. This ensures deterministic replay: workflows that started before the code change continue on the original branch, and newer executions use the new logic. After all old executions finish, the branching logic can often be removed. Overall, versioning patterns let developers evolve workflows without breaking running executions. Temporal offers multiple options — definition names, task queues, and the getVersion API — each with trade-offs. (Using separate definitions or queues isolates versions at the cost of more infrastructure, while getVersion keeps a single codebase but requires planned version markers.) Regardless, versioning is a key pattern to safely deploy workflow updates in production. Conclusion Temporal’s durable workflow engine incorporates many built-in aids for complex process patterns. By applying established designs — such as sagas for compensating transactions, retry and heartbeat loops for polling, fan-out/fan-in via child workflows, and event-driven actors with signals/queries — engineers can build robust systems without manual boilerplate. Each pattern leverages Temporal features: workflows and activities, promises, signals, queries, and continuations. The examples above show how little code is needed: a few method calls and standard control structures achieve what would otherwise be elaborate orchestration logic. In practice, adopting these patterns means that failures are handled gracefully and state is managed cleanly. For example, the saga code snippet illustrates reversing partial work on error, while the parallel batch example shows how to process unbounded data safely with continueAsNew. In summary, understanding Temporal’s idioms — as documented by the Temporal team and community — empowers developers to focus on business logic while the platform ensures reliability. Mastery of these workflow patterns leads to systems that are easier to reason about, easier to maintain, and resilient in production.

WebSocket debugging is one of those things that sounds simple until you actually have to do it. The connection looks fine in DevTools, but messages are malformed, timing is off, or the server is behaving unexpectedly — and you have no easy way to inspect what's happening at the frame level without setting up a proxy or installing something heavy. Here's a practical workflow that requires nothing beyond a browser, illustrated with a real debugging scenario. The Problem With WebSocket Debugging HTTP requests are easy to inspect. DevTools shows you the full request and response, you can replay them with curl, mock them with interceptors, and diff payloads in seconds. WebSocket connections are different. Once the handshake completes, it's a persistent bidirectional channel, and most tooling treats frames as an afterthought. The Chrome DevTools WebSocket panel shows you raw frames, but it doesn't let you filter, transform, or replay them. You can see that a frame was sent with a 400-byte payload — but you can't easily extract it, modify it, and resend it to see how the server responds. The common workarounds all have friction: console.log on both sides – requires access to server code, adds noise, and still doesn't let you test edge cases without changing the clientCharles Proxy or mitmproxy – heavyweight, requires SSL certificate setup, and adds a network hop that can change timing behaviorCustom proxy server – takes time to build and maintain, and is overkill for a one-off debugging session None of these is fast when you just need to understand what's happening right now. A Real Scenario: Debugging a Real-Time Chat Feature To make this concrete, here's a situation that comes up often in practice. You're building a chat feature on top of a WebSocket backend. The UI looks fine in testing, but in production, some users report that messages occasionally appear out of order or that a specific type of system message causes the client to crash. You can't reproduce it reliably in your local environment, and you don't have direct access to the production server's logs. The questions you need to answer: What does the actual message payload look like when the crash happens?Is the issue in the message structure (missing field, unexpected type), or is it a timing problem (two messages arriving within milliseconds of each other)?How does the server respond if you send a deliberately malformed message? This is exactly the kind of debugging that browser-only tooling handles well — if you have the right tools. Step 1: Validate the Endpoint With an Online Tester Before anything else, confirm that the WebSocket endpoint is reachable and responding correctly. The tests.ws WebSocket tester is a browser-based tool that lets you connect to any ws:// or wss:// server, send arbitrary messages, and see server responses in real time. No install, no configuration, no account. For the chat scenario: connect directly to your production WebSocket endpoint, send a message that matches the format your client normally sends, and verify the server acknowledges it correctly. If this works as expected, the issue is likely in how the client processes incoming messages, not in the connection itself. The site also provides a free public echo server at wss://echo.tests.ws. Anything you send comes back immediately. This is useful for validating your client-side message serialization — connect to the echo server, send your payload, and confirm what comes back matches what you sent. If there's a mismatch, you've found a serialization bug before you even involve a real server. For the real-time testing step, the interface also shows frame-level details: message direction, payload size, timestamp, and raw content. This is enough to identify structural issues in isolation. Step 2: Intercept Live Traffic With the Chrome Extension Once you've validated the endpoint in isolation, the next step is observing what actually happens in your running application. The tests.ws Chrome extension adds a WebSocket proxy layer directly into Chrome DevTools, without modifying your application code or network configuration. Install the extension, open your application, and open DevTools. A new panel appears that logs every WebSocket frame — direction (sent/received), timestamp, payload size, and raw content — for all connections on the page simultaneously. Unlike the built-in DevTools WebSocket view, you can filter frames by content, copy payloads, and see a cleaner timeline. For the chat scenario, reproduce the conditions where messages go out of order. In the extension panel, you can see the exact sequence of frames with millisecond timestamps. If two messages are arriving 3ms apart and your client processes them synchronously, you'll see the problem immediately in the frame log — even if your application-level logging shows them in the wrong order. Step 3: Modify Outgoing Messages to Test Edge Cases This is where the extension's real value shows up. The extension lets you write JavaScript transform rules that intercept outgoing frames and modify them before they're transmitted to the server. For the crash scenario: you suspect the crash happens when a system message arrives with a missing userId field. Instead of waiting for it to happen in production, you write a transform rule: JavaScript if (message.type === 'system') { delete message.userId; } The extension applies this rule to matching outgoing frames. The server receives the malformed payload, you observe its response in the frame log, and you can immediately see whether it sends back an error, silently drops the message, or sends something that would cause the client to crash. This replaces a workflow that would otherwise require: modifying client code, building a new bundle, deploying to a test environment, and hoping you can reproduce the right conditions. With the extension, the iteration loop is: write a rule, trigger the action in the UI, observe the server response. No code changes, no deployment. Step 4: Test Protocol Edge Cases Beyond the immediate crash scenario, the transform approach is useful for systematic protocol testing: Missing required fields – remove fields one at a time to see which ones the server validatesType mismatches – send a string where the server expects an integer, or an array where it expects an objectOversized payloads – test the server's behavior when message size exceeds expected limitsRapid sequences – send the same message 10 times in quick succession to test for race conditions server-sideMalformed JSON – send a syntactically invalid payload to verify error handling Each of these can be tested in minutes, directly against a running server, without writing test harnesses or modifying application code. When This Approach Has Limits Browser-based WebSocket debugging works well for: Front-end debugging when you don't have server accessQA validation of message formats and server behaviorSecurity testing and input validation checksLearning how a third-party service's WebSocket protocol works It doesn't replace load testing tools. If you need to simulate 10,000 concurrent connections or measure throughput under sustained load, you need something like k6 or Artillery running outside the browser. Similarly, for server-side issues — memory leaks, connection pool exhaustion, handler bugs — you need server-side observability tools. But for the class of problems that are most common during development and integration — "why is the client behaving unexpectedly when it receives this specific message?" — the browser-only workflow gets you to an answer faster than any other approach. Summary The debugging workflow for the chat scenario above: Validate the endpoint – use the online WebSocket tester at tests.ws to confirm the server responds correctly to well-formed messagesObserve live traffic – install the Chrome extension, open the application, and capture the actual frame sequence that leads to the problemReproduce and test – write a transform rule that simulates the malformed message, trigger it in the UI, observe the server's response Total time to go from "users are reporting a crash" to "here's the exact server response that causes it": under 15 minutes, with no infrastructure changes, no deployments, and no server access required. WebSocket tooling has historically lagged behind HTTP tooling. The gap is smaller than it used to be.

Analytics pipelines tend to scale in both cost and the age of their data sources: costs increase with data volume growth, while data freshness decreases due to longer batch jobs. The common approach, scaling out the cluster, addresses the symptom rather than the architectural issue. In this tutorial, we will look at an alternative solution that addresses both problems at their root: using Netflix Maestro, a horizontally scalable workflow orchestrator open-sourced by Netflix in July 2024, along with Apache Iceberg, a standard table format for analytics on object storage. The former helps by shifting from time-based scheduling to event-driven, whereas the latter removes the overhead of listing files that slows down queries on large datasets and increases their costs. We will cover all aspects of creating a full-fledged pipeline, including code examples, explanations of why each component reduces costs, and real metrics showing what results to expect. What You'll Need ComponentPurposeNotesApache Iceberg + a catalogTable format and metadata managementREST catalog (Polaris, Nessie, Lakekeeper, Unity Catalog) recommended for new deployments; Glue/Hive also fineA compute engineReads and writes Iceberg tablesSpark 3.5+, Flink, Trino, or DuckDB via PyIcebergNetflix MaestroWorkflow orchestrationRequires Java 21, Docker, and Postgres or CockroachDB for stateCloud object storageData files and metadataS3, GCS, ADLS, or S3-compatible (MinIO works for local dev)Python 3.10+Lightweight tasks and ingestionPyIceberg 0.11+, PyArrow Terminology note: there are several products named "Maestro" in the data space. This guide is about Netflix's Maestro and is different from Maestro by Conductor, AWS Maestro, etc. Netflix's Maestro executes hundreds of thousands of workflows and up to 2 million jobs per day inside Netflix, so the scalability claim is valid — although some practitioners consider Maestro overengineered for small teams, so keep that in mind. The Problem Statement The standard stack on Hive tables stored in S3 has three structural inefficiencies: File listing dominates query planning. Listing operations on S3 are slow and rate-limited. For a query on a partitioned Hive table, listing might take more time than reading data itself.Small-file proliferation. Continuous or micro-batch writing produces thousands of Parquet files. Each query suffers from open-file overhead, and each list operation brings in additional results.Time-based scheduling wastes compute. Jobs are triggered based on a fixed schedule, not data availability. If upstream data is late, the job processes stale inputs. If the data is early, the job idles until the next scheduled run. Iceberg solves (1) and (2) in the storage tier. Maestro solves (3) in the orchestration tier. Let's see how. Why Iceberg Shifts the Cost Model Iceberg takes the table metadata out of the filesystem and puts it into a metadata tree. In response to the query "what files are part of this table?", the engine looks up a single metadata entry, follows the path to the manifest list, and gets back an exact list of data files, along with file-level statistics such as min/max values, null count, and row count. File discovery turns from an O(n) directory listing to O(1) metadata lookup. As a result, we get a chain reaction: Hidden partitioning. Declare a table PARTITIONED BY days(event_time), and queries filter on event_time directly. Partition transform happens automatically. No more WHERE year=2026 AND month=05 AND day=18, and no risk of analysts forgetting.Partition evolution. You can change the partitioning of the table from monthly to daily without rewriting old data. The metadata keeps track of it, and the engine routes queries correctly.Time travel and rollback. Writes produce immutable snapshots. If a bad load happens, you don't need to restore from backups – just roll the catalog pointer back to the previous snapshot. It matters operationally – recovery time goes from hours to seconds.Snapshot isolation and ACID. Writers operate concurrently; readers always see the consistent state, never a partial commit. The cost angle: manifest statistics can prune scans by an order of magnitude in time-filtered queries. With S3 list operations removed entirely, query costs on warehouse engines like Trino, Athena, or BigQuery (which charge per byte scanned) go down proportionally. Why Maestro Helps With Freshness and Costs The killer feature of Maestro in the context of our use case is the signal service — an event-driven trigger mechanism. Instead of scheduling "run this job at 02:00 every day", you tell Maestro to execute the job "when user_events_raw table receives a new snapshot". The trigger may originate from another Maestro workflow, an S3 event, a database table modification, or even from any external system capable of sending a request to the signal API endpoint. The gap between data arrival and data availability closes from hours (the worst-case batch window) to seconds or minutes. Other notable features of Maestro: Support for both DAGs and cyclic workflows. Unlike Airflow, Maestro allows loops and re-execution, which is useful for retry-with-backoff and convergence scenarios.ForEach loops and subworkflows as native concepts. Reduces the YAML sprawl common in large Airflow setups.At-least-once triggering with built-in deduplication leads to effective exactly-once execution.Mixed task types. A single workflow can combine Python, Spark, SQL (Trino/Presto), bash, notebook, Docker container, and Kubernetes jobs.100x performance improvement of the engine announced in September 2025 brings a step transition time from seconds to milliseconds, which is important for workflows with hundreds of steps. Step 1: Create the Iceberg Table With Sensible Defaults Begin with a definition of the table such that partitioning is done correctly from the start. By far the most frequent problem when adopting Iceberg is to overlook partitioning. SQL CREATE TABLE analytics.user_events ( user_id BIGINT, event_type STRING, event_time TIMESTAMP, session_id STRING, properties MAP<STRING, STRING> ) USING iceberg PARTITIONED BY (days(event_time), bucket(16, user_id)) TBLPROPERTIES ( 'format-version' = '2', 'write.target-file-size-bytes' = '134217728', -- 128 MB target 'write.parquet.compression-codec' = 'zstd', 'write.metadata.delete-after-commit.enabled' = 'true', 'write.metadata.previous-versions-max' = '20', 'history.expire.max-snapshot-age-ms' = '604800000', -- 7 days 'history.expire.min-snapshots-to-keep' = '10' ) LOCATION 's3://your-bucket/iceberg-tables/user_events'; Some interesting choices that should be explained: days(event_time) is a partitioning transform. Queries filtering by event_time will receive automatic partition pruning.bucket(16, user_id) is a bucket transform that evenly spreads writes among 16 buckets per day partition. It helps with hot spot prevention when one user produces disproportionately high amounts of traffic and provides better parallelism for joining on user_id.format-version = '2' allows for row-level deletions through delete files. V3 is a more recent version that adds many features, including deletion vectors, but make sure your engine supports it first.zstd provides better compression ratio by 10-20% compared to snappy with the same performance when reading.Expiring snapshot properties help avoid metadata explosion, which is one of the most frequent causes of costs silently accumulating in an Iceberg environment. Without this, each write would retain all previous snapshots indefinitely. Step 2: Ingest Data There are two reasonable options for ingesting data from Python into Iceberg: Spark (in case you already have a Spark cluster and need the scale provided by it) and PyIceberg (low overhead, no JVM required). Python from pyspark.sql import SparkSession from pyspark.sql.functions import to_timestamp, col spark = ( SparkSession.builder .appName("IcebergIngestion") .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog") .config("spark.sql.catalog.my_catalog.type", "rest") .config("spark.sql.catalog.my_catalog.uri", "https://your-rest-catalog/api/v1") .config("spark.sql.catalog.my_catalog.warehouse", "s3://your-bucket/iceberg-tables/") .config("spark.sql.catalog.my_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") .getOrCreate() ) raw = spark.read.json("s3://your-bucket/raw/events/2026-05-18/") events = ( raw .withColumn("event_time", to_timestamp(col("event_time"))) .select("user_id", "event_type", "event_time", "session_id", "properties") ) # MERGE INTO supports idempotent ingestion — important for replay safety events.createOrReplaceTempView("staging_events") spark.sql(""" MERGE INTO my_catalog.analytics.user_events t USING staging_events s ON t.user_id = s.user_id AND t.event_time = s.event_time AND t.event_type = s.event_type WHEN NOT MATCHED THEN INSERT * """) Two important aspects. First, the REST catalog should be used for any new deployment, as it allows accessing the same table via Spark, Trino, Flink, Snowflake, BigQuery, and PyIceberg without having to deal with catalog configurations drifting per engine. Second, using MERGE INTO instead of INSERT ensures that the ingestion becomes idempotent, especially when the step fails and Maestro tries to retry it. PyIceberg Ingestion (Lightweight Path) For lighter loads or ingestion processes executed as part of an orchestrator step, PyIceberg is quicker to initialize and has no dependency on the JVM. Currently, the library requires tables in PyArrow format, not pandas DataFrames: Python import pyarrow as pa from pyiceberg.catalog import load_catalog catalog = load_catalog( "my_catalog", type="rest", uri="https://your-rest-catalog/api/v1", warehouse="s3://your-bucket/iceberg-tables/", ) table = catalog.load_table("analytics.user_events") new_rows = pa.table({ "user_id": [3, 4], "event_type": ["purchase", "click"], "event_time": pa.array( ["2026-05-18T12:10:00", "2026-05-18T12:15:00"], type=pa.timestamp("us"), ), "session_id": ["sess-001", "sess-002"], "properties": [{"sku": "A123"}, {"page": "/home"}], }) table.append(new_rows) By default, PyIceberg uses "fast append" optimization, which reduces per-commit metadata operations but creates more manifest files than other optimizations. This is good for frequent micro-batch processing as long as you perform regular compaction (see below). Step 3: Define the Maestro workflow Maestro workflows can be defined using either JSON or YAML format. The following example defines a workflow that loads raw events, applies transformation, performs data quality checks, and updates the aggregate. Steps are connected by signals to start processing as soon as their dependencies are available. YAML name: user-events-pipeline description: Ingest, transform, validate, and aggregate user events trigger: signal: name: raw_events_landed match: bucket: your-raw-bucket prefix: events/ nodes: - name: ingest-events task: type: python script: ingest.py params: partition_date: ${execution_date} retry: max_attempts: 3 backoff_seconds: 60 - name: validate-schema dependencies: [ingest-events] task: type: python script: validate.py - name: transform-events dependencies: [validate-schema] task: type: spark class: com.yourorg.transforms.SessionizeEvents params: input_table: analytics.user_events output_table: analytics.user_sessions partition_date: ${execution_date} - name: dq-checks dependencies: [transform-events] task: type: trino query_file: dq_checks.sql fail_on: any_row_returned - name: refresh-daily-aggregate dependencies: [dq-checks] task: type: trino query: | INSERT INTO analytics.daily_user_metrics SELECT CAST(event_time AS DATE) AS event_date, event_type, COUNT(*) AS event_count, APPROX_DISTINCT(user_id) AS unique_users FROM analytics.user_events WHERE event_time >= DATE '${execution_date}' AND event_time < DATE '${execution_date}' + INTERVAL '1' DAY GROUP BY 1, 2 - name: emit-completion-signal dependencies: [refresh-daily-aggregate] task: type: signal emit: name: daily_metrics_ready params: date: ${execution_date} The last step, emitting a completion signal, makes pipelines composable. The downstream pipeline, such as the feature engineering task for ML, subscribes to the daily_metrics_ready topic and kicks off right away upon completion of this one without polling or any delay period.Ingestion Script Python # ingest.py import os import pyarrow as pa import pyarrow.parquet as pq from pyiceberg.catalog import load_catalog PARTITION_DATE = os.environ["partition_date"] catalog = load_catalog("my_catalog") table = catalog.load_table("analytics.user_events") raw_path = f"s3://your-raw-bucket/events/{PARTITION_DATE}/" arrow_table = pq.read_table(raw_path) # Schema enforcement before write — fail loudly on drift expected = table.schema().as_arrow() arrow_table = arrow_table.select(expected.names).cast(expected) table.append(arrow_table) print(f"Appended {arrow_table.num_rows} rows for {PARTITION_DATE}") The cast is intentional. Schema drift — upstream system silently adds or modifies a column – is one of the most frequent pipeline failures. Early detection through an error at ingestion is far less expensive than debugging further down the line. Step 4: Make Queries Cheap There are three main optimizations that account for the majority of savings. Each one is worth comprehending rather than blindly copying. Compaction: The Single Most Important Maintenance Activity Real-time or micro-batch ingestions result in lots of small files. The smaller files lead to larger metadata, inefficient query planning, and unnecessary storage of Parquet footers and row-group overheads. Compaction periodically merges them into files of the desired size (128 MB for our table definition above). With Spark: SQL -- Rewrite small files using bin-packing CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', options => map( 'min-input-files', '5', 'target-file-size-bytes', '134217728' ) ); -- Rewrite manifests so a query reads fewer manifest files CALL my_catalog.system.rewrite_manifests('analytics.user_events'); -- Expire old snapshots beyond the retention configured in TBLPROPERTIES CALL my_catalog.system.expire_snapshots( table => 'analytics.user_events', older_than => TIMESTAMP '2026-05-11 00:00:00', retain_last => 10 ); -- Remove orphan files (files in storage not referenced by any snapshot) CALL my_catalog.system.remove_orphan_files(table => 'analytics.user_events'); Schedule as part of a Maestro workflow that runs either daily or weekly. The remove_orphan_files command is particularly crucial — without this, any failures in writing will result in untracked files in S3, which you continue to pay for storing. Sorting Within Partitions for Skipping Efficiency If you know that your analysts always filter by event_type and user_id, sort your files so that Iceberg’s file-by-file statistics can skip entire files: SQL CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', strategy => 'sort', sort_order => 'event_type ASC, user_id ASC' ); For higher-dimensional access patterns, use Z-order: SQL CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', strategy => 'sort', sort_order => 'zorder(event_type, user_id, session_id)' ); Let Hidden Partitioning Do Its Job The query below requires no partition predicate — Iceberg derives the partition filter from event_time: SQL SELECT user_id, COUNT(*) AS event_count FROM analytics.user_events WHERE event_time >= TIMESTAMP '2026-05-17 00:00:00' AND event_time < TIMESTAMP '2026-05-18 00:00:00' AND event_type = 'purchase' GROUP BY user_id; In Hive, we would have to do AND year=2026 AND month=5 AND day=17 to enable pruning. In Iceberg, the transformation days(event_time) happen automatically, and the extra predicate event_type enables more pruning based on min/max statistics at the file level; files that don’t cover 'purchase' in their event_type range will not be opened. Step 5: Execute the Pipeline Execute the pipeline from the Maestro command-line interface: Shell # Trigger a manual run with parameters maestro start user-events-pipeline \ --param partition_date=2026-05-18 # Check workflow status and last N runs maestro status user-events-pipeline --last 10 # Inspect a specific run maestro instance describe user-events-pipeline <run_id> # Replay a failed run from a specific step maestro instance restart user-events-pipeline <run_id> \ --from-step transform-events Maestro exports metrics on queue depth, step latency, and failure rates via /metrics. Use this together with engine metrics (Spark UI, Trino query stats) to correlate any delays in orchestration with query performance. What Kind of Savings Should You Really Be Expecting? There is the old story about 90 percent savings when making such migrations that needs to be taken with a grain of salt. The real truth is highly dependent on your source. ScenarioRealistic savingsSource of savingsHive tables on S3 → Iceberg, same engine20–50% on query costsEliminated S3 listing, file pruning via stats, fewer small filesCron-scheduled batch → Maestro signalsVariable on compute, large on freshnessCompute drops only if jobs were over-running their window; freshness improves from hours to minutesProprietary warehouse → Iceberg + open engines40–80% on storage and licenseStorage decoupled from compute; engine competition on the same dataStreaming with no compaction → Iceberg + scheduled maintenance30–60% on query costsCompaction collapses small-file overhead The 90% number is realistic if the starting point is truly pathological, say a highly partitioned Hive table on S3 with no file size management that is being queried by a byte-scanned engine. Most organizations should budget for 30%-60% improvements and view anything higher as upside. Freshness improvements, by contrast, are reliably dramatic. Upgrading from a 4-hour cron job to an event-driven pipeline that fires within seconds of completion of its upstream is a structural win, not an incremental one. Comparing Maestro to Other Options Maestro is not the only option. The lay of the land as of 2026: Airflow has the broadest deployment and the most extensive provider ecosystem. Strengths: DAG construction; weaknesses: high-frequency triggering. Airflow's scheduler is traditionally been the bottleneck when operating at very high workflow volumes.Dagster has better data-aware abstractions (assets, partitions, software-defined assets) and integrates well with dbt and modern data tooling. The scale ceiling is lower than Maestro's.Prefect is native-Python and developer-friendly, offering good dynamic workflow capabilities. Still immature for very large scale.Temporal is the best general-purpose orchestrator for application workflows, less specialized for data pipelines.Maestro beats competitors on scale and on the signal/cyclic workflow paradigm. Cost factors: smaller community, steeper operational overhead, fewer out-of-the-box integrations. If you are already using Airflow and have fewer than a few thousand workflows per day, the migration costs to Maestro probably don't justify themselves through orchestration improvements alone — Iceberg adoption can be decoupled. However, if you are hitting Airflow scheduler limitations or have highly interdependent workflows across teams, Maestro's signal paradigm deserves a serious look. Common Mistakes Some recurring pitfalls in production: Deferment of catalog selection. Setting up Iceberg with a Hadoop or filesystem catalog "as a temporary solution" creates a future migration burden. Choose a REST catalog (Polaris, Nessie, Lakekeeper, or vendor-managed) from the start.No snapshot expiration policy. Snapshots persist indefinitely by default. High-volume tables generate gigabytes of metadata each month. Set expiration policies in table properties and run expire_snapshots periodically.No orphan file removal. Failing writes leave behind Parquet files not referenced by any snapshot. Remove orphan files weekly.Over-partitioning. Partitioning by the hour on a low-volume table results in more partitions than rows. Partition by the resolution of your query filters and target file sizes, not finer.Using signals as a free pass on idempotency. Workflow execution triggered by signals can be replayed or backfilled. Make every step idempotent — use MERGE INTO for writes, de-dupe on natural keys, and never make assumptions about "this only runs once."Skipping compaction. Streaming pipelines without compaction gradually degrade query performance until someone notices that the queries are 10x slower than at launch time. Conclusion Iceberg and Maestro solve two aspects of the same problem. Iceberg makes the data layer cheap to query by converting filesystem state into metadata state. Maestro makes the orchestration layer responsive by substituting signals for clocks. Adopting either technology creates tangible value, while adoption of both yields a pipeline that is inherently cheaper to operate and inherently fresher than a cron-based/Hive setup. If your current challenge is query cost or small file issues, start with Iceberg. If you are plagued with data staleness or unreliable scheduling, start with Maestro (or any other modern orchestrator). But eventually aim to adopt both if your goal is a data platform that scales without scaling your cloud bill. Where to learn more: Netflix Maestro: github.com/Netflix/maestroApache Iceberg: iceberg.apache.orgPyIceberg: py.iceberg.apache.orgApache Polaris (Iceberg REST catalog): polaris.apache.org

When I first started building AI applications, I kept hearing the same words everywhere: workflows, agents, and multi-agent systems. At first, they all sounded like different labels for the same thing. After all, in every case, you are still calling an LLM, sending some context, and getting something back. That assumption turns out to be one of the easiest ways to design the wrong system. Once you start building real projects, the difference becomes very obvious. Some systems need strict control. Some need flexibility. Some need multiple specialized roles. If you choose the wrong model, you usually pay for it in cost, reliability, debugging pain, or unnecessary complexity. This is the explanation I wish I had when I started. I want to keep it beginner-friendly, but also useful enough that you can apply it in real projects without walking away with the usual “everything is an agent” confusion. Workflow vs Agent vs Multi-Agent System The simplest way to understand the whole topic is this: A workflow is when you decide the steps in advance. An agent is a model that decides what to do next. A multi-agent system is one in which multiple agents, usually with different roles, coordinate to solve a larger problem. That core distinction aligns closely with external references: workflows follow predefined code paths, while agents dynamically direct their own tool usage and execution flow. That sounds simple, but it becomes much clearer with a relatable example. Imagine you are ordering pizza. In a workflow, the restaurant follows a script. They ask for size, toppings, crust, and address in a fixed sequence. It is fast, reliable, and predictable. In an agent-style system, you might say, “I’m hungry, and I want something good for movie night,” and the system figures out whether you usually order vegetarian, whether you want something quick, whether it should ask a follow-up question, and what option best fits your past behavior. In a multi-agent setup, one specialist handles the order, another checks ingredient availability, and another optimizes delivery timing. Each one does a narrower job, but together they solve a broader problem. That is the real difference. The question is not whether all three use AI. The question is who is controlling the process. What a Workflow Really Is A workflow is the most structured option. You define the steps, the order, and often the failure points. The model may still do useful work inside the system, but the system itself is not making open-ended decisions about how to proceed. Think of it like a recipe. Step one happens first. Step two happens second. If something goes wrong, you usually know where it happened. A simple example is a blog post generator that deliberately separates outline generation, introduction writing, body drafting, and final assembly. TypeScript import Anthropic from '@anthropic-ai/sdk'; const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); async function generateBlogPost(topic: string) { const outlineResponse = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 1024, messages: [ { role: 'user', content: `Create a blog post outline about: ${topic}` } ] }); const outline = outlineResponse.content[0].text; console.log('Step 1: Outline created'); const introResponse = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 1024, messages: [ { role: 'user', content: `Based on this outline, write an introduction:\n\n${outline}` } ] }); const intro = introResponse.content[0].text; console.log('Step 2: Introduction written'); const bodyResponse = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 2048, messages: [ { role: 'user', content: `Based on this outline, write the body:\n\n${outline}` } ] }); const body = bodyResponse.content[0].text; console.log('Step 3: Body written'); return `${intro}\n\n${body}`; } The reason workflows dominate production is not that teams lack ambition. It is that predefined orchestration is easier to reason about. Predictable systems are easier to test, monitor, certify, and price. That is exactly why guidance around production AI systems keeps steering builders toward workflows first, especially for reliability-critical environments. The referenced material also repeatedly points out that workflows are the better fit when requirements are stable, boundaries are clear, and reliability matters more than open-ended autonomy. That makes workflows a very strong fit for document processing, onboarding, report generation, fixed moderation pipelines, approval chains, and regulated systems. What an Agent Really Is An agent changes one important thing. Instead of hardcoding the order of operations, you give the model a goal, a set of tools, and enough context to decide what should happen next. That is where the flexibility comes from. The model can inspect the task, choose a tool, look at the result, decide whether another tool is needed, and continue until it reaches a stopping point. That pattern is what makes an agent feel more like a smart assistant than a pipeline. The external guides describe this clearly as dynamic decision-making, autonomous tool selection, reasoning, and self-directed task execution. A simple research assistant is a good example for beginners. TypeScript import Anthropic from '@anthropic-ai/sdk'; const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); const tools = [ { name: 'search_web', description: 'Search the web for information about a topic', input_schema: { type: 'object', properties: { query: { type: 'string' } }, required: ['query'] } }, { name: 'save_notes', description: 'Save research notes to a file', input_schema: { type: 'object', properties: { notes: { type: 'string' } }, required: ['notes'] } } ]; async function searchWeb(query: string): Promise<string> { return `Results for ${query}`; } async function saveNotes(notes: string): Promise<void> { console.log(`Saved notes: ${notes.slice(0, 80)}...`); } async function researchAgent(topic: string) { const messages: any[] = [ { role: 'user', content: `Research ${topic} and save comprehensive notes.` } ]; let done = false; while (!done) { const response = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 4096, tools, messages }); if (response.stop_reason === 'tool_use') { const toolUse = response.content.find( (block: any) => block.type === 'tool_use' ); if (toolUse.name === 'search_web') { const results = await searchWeb(toolUse.input.query); messages.push({ role: 'assistant', content: response.content }); messages.push({ role: 'user', content: [ { type: 'tool_result', tool_use_id: toolUse.id, content: results } ] }); } if (toolUse.name === 'save_notes') { await saveNotes(toolUse.input.notes); done = true; } } else { done = true; } } } What matters here is not the SDK syntax. What matters is that you did not hardcode “search first, summarize second, save last.” The agent decides that. It may search once. It may search five times. It may decide it has enough information early. That is precisely why agents are useful for research, support, exploratory planning, and other tasks where you cannot fully predict the required path ahead of time. The trade-off is that you lose some of the certainty that workflows give you. The number of tool calls can vary. The runtime can vary. The cost can vary. If something behaves strangely, you often need stronger logs and better observability to understand why. Seeing the Difference Side by Side One of the best parts of your attached draft was the side-by-side review analysis example, because it shows the difference without abstract theory. That absolutely deserves to stay. Suppose the task is to analyze a customer review and generate a response. The workflow version might look like this. TypeScript async function analyzeReviewWorkflow(review: string) { const sentiment = await callLLM( `Analyze sentiment of this review as positive, negative, or neutral: ${review}` ); const topics = await callLLM( `Extract the main topics from this review: ${review}` ); const response = await callLLM( `Generate a customer support response for a ${sentiment} review about ${topics}` ); return { sentiment, topics, response }; } This is clean and efficient. It makes the same three calls every time. The cost is predictable. The behavior is stable. It is also rigid. A weird review gets handled through the same path as a normal one. Now compare that with an agent version. TypeScript async function analyzeReviewAgent(review: string) { return await runAgent({ task: `Analyze this review and generate a support response: ${review}`, tools: [ 'check_sentiment', 'extract_topics', 'search_knowledge_base', 'generate_response' ] }); } Now the system can decide whether a highly emotional complaint requires a knowledge base lookup before responding, while a simple positive review may only require sentiment classification and a thank-you response. That flexibility is exactly what makes agents attractive. It is also what makes them less predictable. This is one of the most important beginner lessons in the whole topic. A workflow handles every case with the same planned path. An agent adapts its path to the case. When Workflows Are the Better Choice This is where most of the production reality sits. If you know the exact steps, a workflow is almost always the first thing you should build. If predictability matters, a workflow is usually safer. If cost matters, workflows are easier to manage because you know roughly how many model calls happen per run. For debugging, workflows are easier because every state transition is explicit. That is also why modern workflow-oriented systems emphasize type safety, checkpointing, durable execution, human-approval steps, and clear routing. Those capabilities are not flashy, but they are exactly what real teams need when a system runs in production for weeks or months. A customer onboarding pipeline is a simple example. TypeScript async function onboardCustomer(email: string) { await sendWelcomeEmail(email); await createAccount(email); await setupDefaultPreferences(email); await sendTutorial(email); } A document processing pipeline is another. TypeScript async function processDocument(pdfPath: string) { const text = await extractText(pdfPath); const summary = await summarize(text); const keywords = await extractKeywords(text); await saveToDatabase({ text, summary, keywords }); await notifyUser(); } A content moderation flow is another good fit. TypeScript async function moderatePost(post: string) { const isSpam = await checkSpam(post); const isToxic = await checkToxicity(post); return isSpam || isToxic ? 'reject' : 'approve'; } None of these tasks benefits much from letting the model invent the control flow on the fly. They benefit from clean orchestration. When Agents Are the Better Choice Agents make more sense when the task is open-ended, when the path cannot be fully predefined, or when adaptability matters more than deterministic execution. Customer support is a classic example because every issue arrives in a different way. Research is another reason because you do not know in advance which leads will be useful. Trip planning is another challenge because different users, constraints, budgets, dates, and preferences change the best route through the task. A travel helper captures this nicely. TypeScript async function travelAgent(request: string) { return await runAgent({ task: `Help the user with this travel request: ${request}`, tools: [ 'search_flights', 'search_hotels', 'get_weather', 'suggest_itinerary', 'ask_followup_question' ] }); } The system may begin by asking a clarifying question. It may check the weather before hotels. It may avoid hotel search entirely if the user says they are staying with friends. This is exactly the sort of context-dependent behavior that agents are designed for. The guides also specifically call out use cases like deep research, agentic RAG, customer support, virtual assistants, and coding assistants as agent-friendly territory. What Multi-Agent Systems Add Multi-agent systems take the idea one step further. Instead of having one agent handle everything, you split the work among multiple specialists. This matters when specialization actually improves the result. One agent might research. Another might write. Another might review or validate. The Inkeep article makes an important distinction: true multi-agent systems are not just a sequential workflow with different names for each step. The key idea is autonomous coordination between specialized agents, often through direct communication or delegated responsibilities. A simple content team example makes this concrete. TypeScript async function researchAgent(topic: string) { return callLLM(`Research ${topic}. Return key facts, trends, and context.`); } async function writerAgent(research: string, topic: string) { return callLLM(`Using this research, write an article about ${topic}:\n${research}`); } async function editorAgent(article: string) { return callLLM(`Edit this article for clarity, accuracy, and flow:\n${article}`); } async function contentCreationTeam(topic: string) { const research = await researchAgent(topic); const draft = await writerAgent(research, topic); const final = await editorAgent(draft); return final; } This is still a simple coordinator-led version, but it shows the value of specialization. A more advanced system might allow the editor to request a revision from the writer, or the writer to request more supporting evidence from the researcher. That is where multi-agent systems start to feel like collaborative problem-solving rather than a chain of prompts. The caution here is important. Multi-agent systems are not “the next level” you should jump to just because they sound advanced. They introduce more moving parts, more coordination overhead, more debugging complexity, and higher cost. They are useful when the problem actually needs multiple kinds of expertise, not when you are just trying to make a simple app look more impressive. The Practical Decision Model A good beginner question is not “which one is the smartest?” It is “how much uncertainty does this task have, and who should own the decision-making?” If the task is well-defined and stable, start with a workflow. If the task is open-ended and the system needs to choose how to proceed, consider an agent. If the task genuinely benefits from multiple specialists with separate responsibilities, consider multiple agents. That decision model lines up closely with the source material as well. Use workflows when requirements are clear, control is important, cost matters, and debugging stays simple. Use agents when tasks are exploratory, human-like reasoning is valuable, and adaptability matters more than fixed control flow. Use multi-agent systems when a single reasoning unit is no longer sufficient to capture the problem's diversity. The Beginner Mistakes That Cost Time and Money The first mistake is using agents for simple tasks that should be handled by normal code or a fixed workflow. If you want to add two numbers, do not build an agent. If you want to categorize simple support tickets with a stable schema, start with a workflow. Not every AI problem needs autonomy. TypeScript function addNumbers(a: number, b: number) { return a + b; } The second mistake is forcing a workflow onto a task that clearly needs adaptation. Creative writing, research, and support escalation often branch in ways that are hard to encode cleanly in advance. If you keep adding if-statements and exception paths to rescue a rigid workflow, that is often a sign the task wants agent behavior. The third mistake is building multi-agent systems too early. Three agents for a simple email writer is usually just an expensive ceremony. You should earn that complexity by hitting a real need first. These mistakes sound obvious when written down, but they are very common because the AI space rewards novelty in demos more than maintainability in products. The Cost Conversation Matters More Than People Admit A workflow-based newsletter creator might always make three model calls, one for the intro, one for the main copy, and one for the closing section. That means the cost per run is fairly easy to estimate. TypeScript async function createNewsletter(topics: string[]) { const intro = await generateIntro(topics); const articles = await generateArticles(topics); const outro = await generateOutro(); return { intro, articles, outro }; } An agent-based newsletter creator might decide it needs extra research, then rewrite one section twice, then call another tool to validate tone. Sometimes that flexibility is useful, but it also means cost and latency can move around more than you expect. TypeScript async function newsletterAgent(topics: string[]) { return runAgent({ task: `Create a newsletter about these topics: ${topics.join(', ')}`, tools: ['research_topic', 'draft_section', 'revise_section', 'validate_tone'] }); } That does not automatically make agents bad. It just means the operational model is different. The broader production guidance on workflows versus agents keeps coming back to exactly this point: deterministic systems are easier to budget for, observe, and control. The Hybrid Model Is Usually the Best Answer This is probably the most useful real-world takeaway in the entire topic. You do not have to choose one pattern forever. Many successful systems use workflows to structure the outer system and agents only where flexibility is genuinely needed. The Prompt Engineering Guide explicitly recommends hybrid approaches, such as using workflows for structure and agents for open-ended subtasks. That pattern looks like this. TypeScript async function smartCustomerSupport(message: string) { const category = await categorize(message); if (category === 'simple_faq') { return faqWorkflow(message); } if (category === 'complex_issue') { return supportAgent(message); } return escalateToHuman(message); } This is a very practical architecture. The workflow gives you control, routing, and predictability. The agent only appears where variability is too high for rigid orchestration. That means you keep the system understandable while still benefiting from adaptive behavior. If you are building beginner-to-intermediate AI products, this is one of the best mental models to adopt early. A Cleaner Way to Think About Real Projects A document processor usually wants a workflow because the same stages repeat every time. A support assistant may want an agent because issues differ, and tool selection depends on context. A software delivery assistant might eventually become a multi-agent system if planning, implementation, testing, and review are separate responsibilities that benefit from specialization. Here is a simplified example of that last case. TypeScript async function developFeature(requirement: string) { const specs = await productManagerAgent(requirement); const code = await developerAgent(specs); const testResults = await qaAgent(code); if (!testResults.passed) { return developerAgent(`Fix these issues:\n${testResults.issues}`); } return code; } This kind of setup can make sense, but only if the complexity is real. It should come from the nature of the work, not from the desire to use more agents. Conclusion If you are just starting, build a workflow first. That advice is not anti-agent. It is pro-clarity. Workflows teach you how to decompose tasks, define boundaries, measure outcomes, and understand where AI actually adds value. Once you understand the stable parts of your system, it becomes much easier to identify the unstable parts that may benefit from an agent. Once you understand where one agent becomes overloaded, it becomes much easier to justify multiple specialized agents. That progression is healthier than starting with maximum autonomy and then trying to reverse-engineer stability later. So my practical rule is simple. If the task can be described as a sequence of reliable steps, use a workflow. If the system needs to decide the steps as it goes, use an agent. If the problem truly needs multiple specialized minds working together, then and only then reach for a multi-agent design. The best AI systems are not the ones with the most autonomy. They are the ones that stay understandable when something goes wrong.

"Vibe coding" tweaking a prompt, running it once, and seeing if it looks okay does not scale for enterprise software. Here is how to build a rigorous verification pipeline to audit, bench, and evaluate your Claude agent's behavior over time. If you are building autonomous agents with the Claude API, you have likely experienced the trap of "vibe coding." It usually goes like this: you write a prompt, give Claude access to a tool, run a single test execution in your terminal, and watch it succeed. You think you're ready for production. Then, you deploy. Within hours, a customer inputs an unexpected edge case, Claude gets trapped in an infinite tool-calling loop, consumes 5 million tokens, and fails the task entirely. As the software development lifecycle shifts toward long-running autonomous workflows, engineers must stop evaluating agents like chat logs and start treating them like production software systems. Moving an agentic system from an experimental script to enterprise-grade software requires a deterministic engineering framework: an Automated Evaluation (Evals) Loop. The Core Architecture of an Agentic Eval Loop Unlike traditional software test suites that evaluate a single inputs-to-outputs assertion, agentic evaluations are fundamentally trajectory-based. Your evaluation infrastructure must run the agent through a stateful "agent loop," collect its execution steps, capture its tool requests, and grade the final environmental impact. Step 1: Building a Rigorous Evaluation Dataset An effective eval suite doesn't require thousands of abstract test cases to start. The absolute best way to begin is by curating 20 to 50 complex tasks directly inspired by real-world user failures, support tickets, and edge cases. A production-grade eval dataset item requires three concrete pillars: The User Intent Prompt: An open-ended instruction containing real-world noise or partial context.The Initial System State: A clean configuration file, a localized repository footprint, or a mock database snapshot that resets before every run.The Gold Standard Reference Solution: The unambiguous target state that confirms success. Avoid vague task criteria. Vague metrics generate noisy, inconsistent evaluation data. Vague Task Spec (Prone to Failure) "Look at the customer account records, find the ones with high spending, and generate an alert script." Unambiguous Task Spec (Production-Grade) JSON { "task_id": "mcp_analytics_042", "intent": "Parse the CSV located at /data/q2_raw.csv. Identify all client IDs whose cumulative transaction value exceeds $50,000. Write an executable python script at /scripts/alerts.py that formats these IDs into a clean JSON list.", "environment_setup": "copy_fixture('q2_raw_unfiltered.csv', '/data/q2_raw.csv')", "evaluation_criteria": { "type": "unit_test_and_state_verification", "target_file": "/scripts/alerts.py", "expected_output_contains": ["10425", "10982", "11034"] } } By explicitly stating target file paths, expected data keys, and environment variables, you ensure the agent fails because its reasoning broke, not because the evaluation test harness itself was poorly specified. Step 2: Utilizing a "Reviewer" Claude Agent for Quality Control Not every agentic outcome can be evaluated by a binary file assertion or a hardcoded regex pattern. If your production agent generates human-facing code documentation, structures a complex customer email response, or proposes an architecture blueprint, verifying correctness requires qualitative reasoning. To handle this at scale without manual human review bottlenecks, deploy a separate "Reviewer" Claude Agent to act as a structured quality control judge (often called an LLM-as-a-Judge architecture). Python import anthropic def evaluate_agent_trajectory(task_intent, final_output, execution_log): client = anthropic.Anthropic() # Use a reasoning-optimized model for evaluation, like Claude 3.5 Opus response = client.messages.create( model="claude-3-5-opus", max_tokens=2000, temperature=0.0, # Lock down stochastic variation system="You are an expert Quality Assurance Judge. Your task is to evaluate an agent's trajectory against a true user intent.", messages=[ { "role": "user", "content": f""" ### CRITERIA FOR SUCCESS The agent's final text summary must address the core issue, maintain professional tone guidelines, and explicitly note any API errors encountered. ### ORIGINAL USER INTENT {task_intent} ### AGENT TRAJECTORY (LOGS) {execution_log} ### FINAL OUTPUT GENERATED BY PRODUCTION AGENT {final_output} Analyze the trajectory step-by-step. Output a JSON object containing your 'reasoning' string, an explicit 'score' integer from 1 to 5, and a binary 'pass_verdict' boolean. """ } ] ) return response.content Critical Rules for Model-Based Grading Isolate your models: Never use the exact same agent system prompt or model instance to grade its own output.Enforce zero temperature: Set your grading agent's temperature to 0.0 to maximize consistency across identical test cycles.Provide negative anchor examples: Give your Reviewer Agent concrete examples of what a "Fail" or "Partial Pass" looks like in its system instructions to anchor the scoring boundaries. Step 3: Tracking Production Metrics That Matter To successfully benchmark your system modifications over time, stop relying on subjective impressions and track three critical system performance indicators across every execution run: 1. Task Completion Success Rate (pass@1) The total percentage of test evaluations where the agent successfully reaches the objective on its first complete run. If you run multiple iterations to account for variance, map the divergence carefully. A sharp drop in your pass@1 metrics combined with high variance is a direct indicator of brittle system instructions or ambiguous tool documentation. 2. Tool Execution Accuracy Track how accurately Claude invokes your functions against your schemas. Calculate these two sub-metrics: Tool call precision: The number of valid tool敲 invocations divided by the total tool attempts made by Claude. A lower score indicates Claude is hallucinating parameter properties or passing corrupted syntax values.Redundant loop count: The number of times Claude executes the exact same tool with the exact same inputs consecutively. High redundancy means your system isn't feeding errors back into the context correctly, leaving the agent trapped in a loop. 3. Comprehensive Token Cost Accounting An agent that completes a task successfully but takes 120 sequential steps and handles 4,000,000 raw input tokens might be too slow and financially expensive to deploy to production. Track the full consumption curve across your evaluation runs: Test Run IDModel VersionSuccess RateAvg. Agent Turn StepsTotal Input TokensTotal Output TokensFinancial Cost / Runv1.0-baselineClaude 3.5 Sonnet74%8.2 turns340,00022,000$1.35v1.1-fixed-toolsClaude 3.5 Sonnet92%4.1 turns185,00011,500$0.71v2.0-heavy-reasoningClaude 3.5 Opus96%3.9 turns420,00038,000$3.20 Synthesizing Your Metrics into Actionable Systems Engineering Building an evals loop alters your entire day-to-day workflow. When you update tool definitions, rewrite an orchestration script, or test a brand-new model variation, you no longer guess if the system improved. You simply run your evaluation test runner, observe the changes across your dashboard, and deploy with confidence. Stop vibe coding. Build a robust, data-backed evaluation loop today, and ensure your Claude-powered agentic systems remain stable, efficient, and aligned at enterprise scale.

Part 1 dived into what to trace in an agentic system and why. How the traditional tracing and metrics, such as latency, scale, cost, uptime, and throughput, need to be redefined. And how to define the new metrics that are at the core of an agentics system, such as response quality, accuracy, and task completion. This part is about the mechanics: how a trace is structured, how context propagates across agent boundaries, and how to make sense of it all. And then, how to automate it all with an observability agent. Anatomy of a Trace Every event in the system that's triggered when processing a member's request — from the initial user request to receiving a response back (and any subsequent tracking and orchestration) — belongs to a single trace. The structure of a trace is hierarchical, built from three key identifiers. These three fields travel in the headers of every internal request and response - traceId: Generated once per request, traceId persists throughout and connects all the operations in the entire execution together.spanId: Generated for each "unit of work", e.g., agent invocation, LLM call, memory persistence, response chunk streaming. SpanId is what uniquely identifies an operation in the trace.parentSpanId: When one operation triggers another, the child span stores the parent’s spanId as its parentSpanId. ParentSpanId is what helps define the relationship between operations, which can then be used to create the execution tree. This is how these IDs get logged in a trace: XML traceId: 8f3a... [A1] Orchestrator Plan traceId = 8f3a... spanId = A1 parentSpanId = null //root span [B1] Fetch Profiles traceId = 8f3a... spanId = B1 parentSpanId = A1 [C1] Profile Ranking traceId = 8f3a... spanId = C1 parentSpanId = B1 [D1] LLM Synthesis traceId = 8f3a... spanId = D1 parentSpanId = A1 Every service, agent, and skill that receives a request reads these headers, generates a new spanId for its own work, and passes its spanId as the parentSpanId in any downstream calls it makes. The result is a complete, traversable execution tree where every node knows its parent. Along with these three fields, a trace should include all important artifacts that need to be tracked for a particular span, such as timestamp, token usage, API, or model version, etc. When collated into a DAG as explained below, whatever can be collected in a span can be optimized in turn. And that is pretty much how tracing works. From a Single Trace to a System DAG A single trace captures one execution. That alone enables replay — debugging and validating system behavior on a per-request basis. For example, it can help confirm whether the intended tool was triggered, whether graceful degradations kicked in as expected, whether the system took the intended path, or where the workflow faltered to produce an inaccurate or low-quality response. Many traces sampled together — across users, queries, surfaces, and over time — produce something much more useful: an empirical Directed Acyclic Graph (DAG) of how the system is actually running in production. This aggregated graph shows what is happening, and can be sliced by invocation frequency, timestamp, and cost. A few critical things become visible only at this level: Critical Paths A critical path is a sequence of operations that consistently dominates the end-to-end latency of a system. Tracing surfaces insights such as - which spans sit on the critical path versus which run in parallel without affecting wall-clock time, which calls block downstream work, and where serialization could be replaced with concurrent fan-out. This is where latency optimization actually pays off. Speeding up off-critical-path operations changes nothing for the user. Resource and Cost-Intensive Workflows In agentic systems, token consumption is the dominant cost driver, and it can be orchestrated to show up in aggregated traces. Tracing makes spending attributable by agent, by query type, by model variant, by use case, or by workflow. Common patterns it exposes: An agent quietly pulls oversized context on every invocation.A frontier model used for trivial classification or routing where a smaller, cheaper one would suffice.Retry behavior or speculative calls that multiply token cost without measurably improving response quality.Tools whose outputs are fetched but never end up in the final prompt. System Inefficiencies Aggregated tracing surfaces some common pitfalls in the agentic workflows Dead branches – skills or agents that exist on paper but rarely fire, or only fire on one narrow query type, signaling candidates for removal or consolidation.Outdated paths – code paths intended to be deprecated that still receive traffic from a handful of callers, blocking cleanup.Unexpected fan-out – an orchestrator that, under certain prompts, triggers far more downstream calls than its design suggests, inflating both latency and cost.Loops and near-loops – repeated transitions between the same agents that indicate stalled reasoning rather than forward progress, often invisible until you look at transition counts across many traces. And to reiterate, it can be measured, it can be improved. The system DAG helps highlight specific, addressable engineering gaps — backed by data rather than intuition. Tracing Overhead Tracing doesn’t come free. Naive instrumentation adds real latency and ships sensitive payloads through the observability stack. A few defaults worth setting early to minimize these overheads: Sample judiciously in production so as not to oversample without losing critical signals. For example, depending on the traffic, one can choose to sample 100% of traces with errors and slow response time, and 5% of the rest.Be wary of what’s actually being logged to avoid misuse — redact PII at the span boundary rather than at query time,Emit spans asynchronously, as a blocking exporter on the request path defeats the purpose. Closing the Loop: Agents Observing Agents The structural primitives that render traces queryable for humans similarly make them consumable by machine intelligence. When spans are emitted in a consistent, structured format, an observer agent can ingest the stream to compute rolling metrics and detect drifts, such as rising p95 on a specific tool, token consumption creeping up on an untouched agent, or transition counts that signal an emerging loop. This observer doesn't have to stop at detection and alerting. A higher-order agent can subscribe to these signals to automate corrective actions: replacing a failing node, shifting traffic to faster model variants, tightening retry limits, disabling misbehaving skills, or falling back to cached responses. By attaching offending trace IDs to an auto-generated ticket, the trace becomes the input for the system's next decision: detect → diagnose → mitigate, automated end-to-end. The tractability of this approach lies in the contract between layers: traceId, spanId, parentSpanId, and a payload. Observer agents require no bespoke integrations or complex plumbing; they read the same spans humans monitor in dashboards. Ultimately, the agentic system observes and corrects itself using the very substrate it produces. Conclusion Done well, the same trace stream that engineers query for debugging becomes the input layer for observer agents that monitor, diagnose, and remediate the system in flight, turning observability from a human dashboard into a closed loop that the system runs on itself.

Zero-downtime deployment is often described as a rollout strategy, but in production, it is more accurately a coordination problem. Traffic must remain on healthy instances while new ones warm up, controllers must wait for readiness before shifting load, and promotion must stop cleanly when metrics degrade. Kubernetes rolling updates already replace Pods incrementally and wait for new instances to start before removing old ones, while readiness probes determine when a Pod should receive traffic. Progressive delivery systems such as Argo Rollouts add weighted traffic shifts, pauses, and analysis gates. The difficult part is not the individual primitive, but the stateful control flow around all of them when retries, human approvals, controller restarts, and rollback decisions intersect. Stateful Release Logic Temporal fits this problem because a Workflow Execution is a durable, reliable, and scalable function execution that persists state and resumes from the latest recorded event after failure. A workflow can wait on timers, external messages, or child workflows without turning those waits into a fragile in-memory state. Temporal also persists durable timers, so a canary soak period or a maintenance window survives worker restarts and infrastructure interruptions instead of being tied to the lifetime of a CI runner or a shell script. That property changes the nature of deployment logic. Instead of treating a release as a short-lived pipeline job, the release can be modeled as a long-running control loop with explicit state such as requested version, current traffic weight, observed health, approval status, and rollback reason. Temporal also guarantees that at most one open Workflow Execution can exist for a given Workflow ID, which makes a fixed ID such as payments-prod a practical concurrency control mechanism for serializing production rollouts and preventing overlapping deploys to the same environment. A Long-Lived Environment Workflow A particularly effective pattern is a long-lived environment workflow that receives release requests by Signal, exposes current status by Query, and periodically uses Continue-As-New to keep its event history fresh. Temporal message handlers operate on workflow state, Signals can be sent from clients or other workflows, and Continue-As-New starts a fresh run in the same chain with the same Workflow ID when history grows. That combination turns a deployment lane into a durable queue and a durable mutex at the same time. If the lane is not already running, Signal-With-Start can start it and enqueue the first release in a single atomic client call. Java @WorkflowInterface public interface EnvironmentDeploymentWorkflow { @WorkflowMethod void run(String service, String environment); @SignalMethod void enqueue(ReleaseCandidate release); @SignalMethod void approve(String releaseId); @QueryMethod DeploymentView current(); } private final Deque<ReleaseCandidate> queue = new ArrayDeque<>(); private boolean approved; @Override public void run(String service, String environment) { while (true) { Workflow.await(() -> !queue.isEmpty()); ReleaseCandidate release = queue.removeFirst(); approved = false; deployRelease(release); if (Workflow.getInfo().isContinueAsNewSuggested()) { Workflow.continueAsNew(service, environment); } } } This pattern keeps rollout ownership inside the workflow rather than in an external scheduler. Approval is a state transition, not a webhook race. Waiting is explicit through Workflow.await, not an ad hoc sleep in a pipeline stage. The workflow can remain open for months, continue across runs when suggested, and still preserve a single logical identity for the service and environment being managed. Activities Encode the Real Work The workflow should not talk directly to Kubernetes, Argo Rollouts, load balancers, or telemetry backends. Temporal workflow code must remain deterministic, and direct I/O belongs in Activities. Activity executions can be retried with explicit retry options, and Temporal recommends designing activities to be idempotent because they may be retried if failures happen before completion is recorded. That requirement has an immediate impact on deployment APIs: methods such as setCanaryWeight(10) or applyManifest(version) are far safer than imperative operations such as increaseTrafficBy(10) or deployAgain(), because retries converge on a desired state instead of amplifying side effects. Java private final RolloutActivities rollout = Workflow.newActivityStub( RolloutActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(5)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(2)) .setMaximumAttempts(5) .build()) .build()); private void deployRelease(ReleaseCandidate release) { rollout.applyManifest(release.service(), release.version()); rollout.waitForAvailable(release.service(), release.version()); rollout.setCanaryWeight(release.service(), 10); Workflow.sleep(Duration.ofMinutes(5)); HealthSnapshot health = rollout.measureHealth(release.service(), release.version()); if (health.errorRate() > 0.01 || health.p95LatencyMs() > 250) { rollout.rollback(release.service(), release.previousVersion()); return; } Workflow.await(() -> approved); rollout.setCanaryWeight(release.service(), 100); rollout.waitForStable(release.service(), release.version()); } The snippet is intentionally narrow: the workflow owns orchestration, while the activity layer owns interaction with external systems. waitForAvailable usually maps to deployment status checks and readiness conditions. In Kubernetes, readiness probes determine when a Pod is ready to accept traffic, Pods that are not Ready are removed from Service endpoints, and a stalled rollout surfaces through progress conditions such as ProgressDeadlineExceeded. If Argo Rollouts is the execution layer, the activity boundary often maps cleanly to its setWeight, pause, and inline analysis steps. One additional design constraint matters here: activity inputs and results are recorded in workflow history, so deployment activities should return compact state, such as health verdicts or revision identifiers, rather than whole manifests or large telemetry payloads. Parallel Waves Without Fragile Fan-Out Many deployments are not single-cluster events. Regional waves, cluster cohorts, and dependency checks often need to run in parallel but still report into one release decision. Temporal child workflows are a natural fit because they are started from a parent workflow, they have their own histories, and they can be invoked asynchronously. This keeps failure domains separate and prevents one large release workflow from becoming an unbounded event log. Java RegionDeploymentWorkflow east = Workflow.newChildWorkflowStub( RegionDeploymentWorkflow.class, ChildWorkflowOptions.newBuilder() .setWorkflowId("payments-prod-" + release.version() + "-us-east") .build()); RegionDeploymentWorkflow west = Workflow.newChildWorkflowStub( RegionDeploymentWorkflow.class, ChildWorkflowOptions.newBuilder() .setWorkflowId("payments-prod-" + release.version() + "-eu-west") .build()); Promise<Void> p1 = Async.procedure(east::deploy, release); Promise<Void> p2 = Async.procedure(west::deploy, release); Promise.allOf(p1, p2).get(); Abort handling also becomes more disciplined in this model. Temporal distinguishes cancel from terminate, and cancel is usually the safer operator action because the workflow receives a cancellation request and can still execute cleanup logic, such as traffic restoration or stable version re-pinning. Terminate stops execution immediately and gives the workflow no chance to run rollback code, which makes it the right tool only for genuinely stuck executions. For deployment orchestration, graceful cancellation aligns with operational reality because rollback is part of the business logic, not an afterthought. The Deployer Must Remain Deployable There is a second deployment problem hidden inside the first one: release workflows often stay open while Temporal workers themselves are being upgraded. Temporal addresses that are directly related to workflow versioning. In the Java SDK, Patching allows a workflow definition to branch safely so that existing executions remain compatible, while newer executions use updated logic. Temporal’s production guidance now recommends Worker Versioning as the default approach for most teams, because worker deployments can be tagged into versions so that old workers continue running old code paths and new workers take new paths, enabling gradual traffic ramps and fast rollback for workflow code itself. Java int v = Workflow.getVersion("post-canary-health-v2", Workflow.DEFAULT_VERSION, 1); boolean accepted = v == Workflow.DEFAULT_VERSION ? health.errorRate() < 0.02 : health.errorRate() < 0.01 && health.p95LatencyMs() < 250; That capability matters because deployment orchestration is rarely static. Health thresholds change, additional gates appear, and new regions get introduced. Without safe workflow versioning, the deployment controller eventually becomes the source of deployment risk. Temporal’s own pre-production guidance is aligned with that concern: deliberately killing all workers and restarting them validates at-least-once semantics, idempotent activities, and clean replay behavior. A zero-downtime deployer should therefore be tested under the same failure patterns it is supposed to absorb on behalf of the application being released. Conclusion Zero-downtime deployment is not achieved by replacing Pods slowly or by adding a canary percentage alone. It is achieved when the full release process can survive restarts, wait safely for readiness and analysis, accept approvals without race conditions, and roll back deterministically when health degrades. Kubernetes and progressive delivery controllers provide the runtime primitives for availability, but Temporal provides the durable control plane that turns those primitives into a reliable deployment application. With stable workflow identities, idempotent activities, durable timers, child workflows for regional waves, and safe versioning for the orchestrator itself, deployment logic stops behaving like a fragile CI episode and starts behaving like production software.