Team Management Resources

DZone's Featured Team Management Resources

How to Diagnose and Recover Stuck Temporal Workflows

By Akhil Madineni

CORE

A Temporal Workflow that appears stuck is rarely “stuck” in the conventional process sense. Temporal persists Workflow state through Event History and resumes execution through replay, so an open execution can remain healthy while waiting for a timer, Signal, Activity, or external condition. The operational problem is therefore not simply lack of completion; it is lack of expected progress. Effective diagnosis starts by establishing what event should have happened next, why it did not happen, and whether remediation can preserve the Workflow’s business invariants. Temporal’s history model makes that analysis unusually tractable because commands, task transitions, Activity attempts, failures, timers, and external interactions are durably represented as Events. Progress Is Visible in the Event History The first diagnostic artifact should be the execution description and raw history, not application logs. temporal workflow describe exposes current execution information and pending Activity state, while temporal workflow show --output json returns Event History in a form suitable for programmatic replay or analysis. A Workflow Query can additionally expose application-defined state without mutating the execution. Shell temporal workflow describe --workflow-id order-7814 temporal workflow show \ --workflow-id order-7814 \ --output json History should be read as a state-transition trace. A WorkflowTaskScheduled event with no corresponding start suggests that work is waiting for a Worker. A started Workflow Task that repeatedly times out can indicate blocked Workflow code, Worker instability, or excessive work inside a task. Repeated WorkflowTaskFailed events can indicate replay or deterministic-compatibility failures after code deployment. Workflow Task failures are retried by Temporal rather than governed by an Activity-style Retry Policy, so a Workflow can remain open while repeatedly failing to make application-level progress. Activity sequences reveal a different failure surface. ActivityTaskScheduled without ActivityTaskStarted points toward dispatch capacity, missing pollers, queue mismatch, or backlog. Temporal persists Workflow and Activity Tasks in Task Queues, and worker-health guidance identifies Schedule-to-Start latency and approximate backlog count as key signals when tasks wait for Workers. ActivityTaskStarted without completion requires inspection of Start-to-Close and Heartbeat behavior because Temporal relies on Start-to-Close timeout to detect a Worker crash after an Activity has started. Not every long pause is pathological. A timer that has not fired, a Workflow waiting for a Signal, or an Activity still inside a valid timeout window can represent correct durable waiting. Conversely, very large histories can become an operational risk. Temporal warns after 10,240 events or 10 MB and enforces a limit of 51,200 events or 50 MB; Continue-As-New creates a new run with a fresh history while carrying forward relevant state. Triage Works Best as Deterministic Evidence Before Model Judgment LangGraph is useful for automating this analysis, but the safest design keeps Temporal facts deterministic and uses an LLM only for classification, hypothesis ranking, and explanation. LangGraph explicitly supports graphs that mix deterministic nodes with model-driven nodes, while structured output can constrain routing decisions into a defined schema rather than free-form text. A compact analyzer can first reduce raw history into evidence that is difficult to hallucinate: the last completed Workflow Task, consecutive Workflow Task failures, pending Activity IDs, the latest Activity attempt, the timeout type, the last Signal, the last timer, the history size, the task queue, and deployment/version metadata. The model then receives that normalized evidence instead of thousands of raw events. Python def extract_facts(state): events = state["events"] return { "facts": temporal_fact_extractor(events), "tail": events[-60:], } def classify(state): result = triage_model.with_structured_output(TriageResult).invoke({ "facts": state["facts"], "tail": state["tail"], "allowed_causes": [ "worker_unavailable", "activity_retrying", "workflow_task_failure", "intentional_wait", "history_pressure", "unknown", ], }) return {"triage": result} That separation matters operationally. Event parsing can enforce hard rules such as “scheduled but never started,” while the model can correlate several weak signals and produce an explanation. Conditional edges can then route low-risk cases to observation, ambiguous cases to deeper diagnostics, and recovery candidates to an approval gate. LangGraph’s graph API supports conditional routing, and persistence stores checkpoints so triage state survives interruptions or process failures. Recovery Must Preserve Temporal and Business Semantics Diagnosis and remediation should remain separate graph stages. A model-generated recommendation must not directly issue cancellation, reset, or termination. LangGraph interrupts provide a natural control boundary because execution can pause with persisted state and resume only after external approval. Python def approval_gate(state): decision = interrupt({ "workflow_id": state["workflow_id"], "cause": state["triage"].cause, "action": state["triage"].recommended_action, "evidence": state["triage"].evidence, }) return {"approved": decision == "approve"} The remediation choice depends on the failure mode. A transient Worker outage usually requires restoring Worker capacity rather than mutating Workflow state because queued tasks persist until Workers can process them. An Activity repeatedly failing on a recoverable dependency can often be left to its Retry Policy, while permanent errors should be made non-retryable in application design to avoid pointless retries. Activity side effects should be idempotent because Activity attempts may execute more than once under retry and recovery behavior. Cancellation is the preferred stop mechanism when Workflow cleanup logic must run. Temporal records a cancellation request and schedules a Workflow Task so Workflow code can react. Termination is forceful: Workflow code does not receive a chance to clean up, and the terminated event closes the history. That makes termination an escalation path for executions that cannot process cancellation normally. Reset is more powerful and more dangerous. Temporal terminates the current execution and creates a new execution that copies history through a selected reset point, then replays forward using current Workflow code. Progress after the reset point is discarded. Reset is therefore appropriate only after the underlying cause has been corrected and after downstream side effects are reviewed for possible re-execution beyond the reset boundary. Shell temporal workflow reset \ --workflow-id order-7814 \ --event-id 42 \ --reason "Recovered after deterministic-compatibility fix" For history pressure rather than a fault, Continue-As-New is generally the safer lifecycle mechanism because it preserves logical continuity under the same Workflow ID while starting a fresh Event History with a new Run ID. It should be designed into long-lived or high-volume Workflow logic instead of used as an improvised emergency action. Safe Automation Requires an Explicit Remediation Envelope A production triage graph should treat remediation as a constrained transaction. The evidence snapshot, selected run ID, candidate reset event, intended action, reason, approval identity, and execution result should all be persisted before any mutation. The action node should re-read the Workflow immediately before execution and reject the operation if the run has changed or the observed condition no longer matches the diagnosis. This is an engineering safeguard rather than a Temporal requirement, but it reduces time-of-check/time-of-use errors when active Workflows continue progressing during investigation. LangGraph’s checkpoint model supports durable approval state, but resumed graph nodes can re-execute from checkpoint boundaries. Its documentation therefore recommends isolating side effects and designing them to be idempotent. A remediation executor should consequently use an operation ID, record completion externally, and refuse duplicate destructive actions. Recovery Without Guesswork Reliable recovery of a stuck Temporal Workflow is fundamentally an event-history problem, not a process-restart problem. The strongest diagnostic path reconstructs expected progress from Workflow Tasks, Activity attempts, timers, Signals, queue state, timeouts, and history growth before considering mutation. LangGraph can turn that evidence into a durable triage pipeline by combining deterministic extraction, constrained model reasoning, conditional routing, and interrupt-based approval. Safe remediation then follows Temporal semantics: restore Workers when dispatch is the issue, allow bounded retries for transient Activities, cancel when cleanup matters, terminate only as a last resort, reset only after the root cause is fixed, and use Continue-As-New to control long-running history growth. The result is automation that accelerates incident response without allowing probabilistic diagnosis to become an unchecked control plane. More

Orchestrating CNN Training and Inference Workflows With Temporal

By Akhil Madineni

CORE

Convolutional neural network workloads rarely fail because the forward pass is mathematically difficult. They fail because modern training and inference pipelines are distributed systems: datasets arrive late, GPU workers disappear, validation jobs stall, model registration breaks halfway through, and long-running executions need to resume without corrupting state. Temporal is designed for exactly that class of problem. A Temporal Workflow Execution is durable, reliable, and scalable, and Temporal defines durable execution as the ability of a workflow to maintain state and progress through crashes or outages. That makes it a strong fit for CNN pipelines whose control plane must survive for hours, days, or even longer while the actual tensor computation runs elsewhere. Why Temporal Fits CNN Pipelines A useful way to think about Temporal in ML systems is as a durable control plane rather than a replacement for PyTorch, CUDA, or a model server. Temporal Workflows hold orchestration state, issue commands, wait on results, and recover by replaying event history. The workflow code itself must stay deterministic, while failure-prone and non-deterministic work belongs in Activities. Temporal’s own documentation is explicit on that boundary: API calls, file I/O, database access, and other external interactions belong in Activities, while workflows should remain replay-safe. That separation aligns naturally with CNN systems, where data staging, training job submission, checkpoint writes, validation, and registry updates interact with external systems constantly. That design matters because a CNN training run is not one step. Even a modest image-classification job usually has a training phase, a validation phase, best-model selection, a checkpointing loop, and a final artifact publication step. The PyTorch transfer-learning tutorial illustrates that pattern directly by alternating train and validation phases, persisting the best state_dict, and loading the best weights at the end; the quickstart tutorial likewise uses model.eval() and torch.no_grad() before prediction. Temporal does not change the math of those stages. It makes the sequence durable, observable, and restartable. A concise workflow can therefore stay almost entirely orchestration-focused: Python @workflow.defn class CnnTrainingWorkflow: @workflow.run async def run(self, req: TrainRequest) -> ModelArtifact: dataset = await workflow.execute_activity( prepare_dataset, req.dataset_ref, start_to_close_timeout=timedelta(minutes=20), ) trained = await workflow.execute_activity( train_cnn, TrainJob(dataset_uri=dataset.uri, config=req.config), start_to_close_timeout=timedelta(hours=8), heartbeat_timeout=timedelta(minutes=1), retry_policy=RetryPolicy(maximum_attempts=3), ) metrics = await workflow.execute_activity( evaluate_cnn, EvalJob(model_uri=trained.best_model_uri, dataset_uri=dataset.val_uri), start_to_close_timeout=timedelta(minutes=30), ) return await workflow.execute_activity( register_model, RegisterRequest(trained.best_model_uri, metrics), start_to_close_timeout=timedelta(minutes=5), ) The important detail in this snippet is not syntax but placement. The workflow issues durable commands, while every side effect lives inside an activity. That matches Temporal’s execution model, where workflows await activity results, activities carry retry policies, and activity timeouts define how long a unit of external work is allowed to run. For long GPU jobs, heartbeat_timeout is especially important because an activity heartbeat tells Temporal the worker is still alive and making progress. Making Long Training Jobs Resumable The natural temptation with CNN training is to keep a Python process alive for hours and hope that the host, container runtime, and storage path all behave. Temporal offers a more robust approach. Activities can be retried automatically after transient failure, and Temporal recommends Start-To-Close timeouts for activity executions. If heartbeats stop arriving inside the heartbeat timeout, the activity can be considered failed and retried according to policy. For training jobs that run on flaky GPU nodes or preemptible infrastructure, that is a meaningful improvement over ad hoc retry shells. The training activity itself should then follow framework-native checkpoint discipline instead of trying to serialize the full training loop into workflow state. PyTorch’s guidance is centered on saving and loading model state with state_dict, and the transfer-learning tutorial shows a production-relevant pattern: save the best model parameters during validation, then reload them at the end. Temporal’s own ML Ops example follows the same philosophy by highlighting checkpoint-aware fine-tuning and resumable inference, with deterministic orchestration in workflows and non-deterministic ML work in activities. A minimal activity therefore looks more like this: Python @activity.defn def train_cnn(job: TrainJob) -> TrainingResult: state = restore_checkpoint(job.resume_uri) model = build_model(job.config, state) best_acc = state.best_acc if state else 0.0 for epoch in range(state.next_epoch if state else 0, job.epochs): train_one_epoch(model, job.train_loader) val_acc = validate(model, job.val_loader) save_checkpoint(job.resume_uri, model, epoch, val_acc, best_acc) if val_acc >= best_acc: save_best_weights(job.best_model_uri, model) best_acc = val_acc activity.heartbeat({"epoch": epoch, "best_acc": best_acc}) return TrainingResult(best_model_uri=job.best_model_uri, best_acc=best_acc) This pattern is deliberately boring, which is a strength. The epoch loop stays in the activity, checkpoints remain in object storage or a shared filesystem, and each heartbeat advertises progress to Temporal. If a retry occurs, the activity can resume from the latest checkpoint rather than restarting from epoch zero. That is the same operational idea highlighted in Temporal’s ML sample repository, and it matches PyTorch’s recommendation to persist model parameters through state_dict-based saves and reloads. When training runs are submitted to an external batch scheduler instead of executing directly inside the worker process, Temporal’s asynchronous activity completion becomes especially useful. An activity can submit the job, capture the task token, and return without marking the activity complete. The external system can later heartbeat and complete the activity through a Temporal client. Temporal documents this explicitly and notes that asynchronous completion is preferable when the external process needs heartbeats or cancellation. Python @activity.defn async def submit_training(job: TrainJob): token = activity.info().task_token launch_gpu_job(job, token) activity.raise_complete_async() Treating Inference as a Workflow Only When It Is One Temporal is not a substitute for a low-latency online model server. A single image classification request that must return in milliseconds usually belongs in the serving layer. Temporal becomes valuable when inference is part of a larger durable process, such as nightly batch scoring, asynchronous document-image analysis, model fallback, approval gates, or multi-stage post-processing. That recommendation follows directly from Temporal’s model of workflows as durable, stateful executions that communicate through activities, child workflows, signals, queries, and schedules. Inside the inference activity, the framework rules remain unchanged. PyTorch examples set the model to evaluation mode and disable gradient tracking during inference. That is essential for CNNs that use dropout- or batchnorm-sensitive behavior and for avoiding unnecessary autograd overhead. Python @activity.defn def score_batch(req: BatchScoreRequest) -> BatchScoreResult: model = load_model(req.model_uri) model.eval() with torch.no_grad(): return predict_batch(model, req.input_uri) Temporal’s message-passing model then makes the surrounding orchestration easier to operate. The Python SDK documents that a workflow can act like a stateful web service receiving queries, signals, and updates. For a batch scoring workflow, a query can expose current shard progress, while a signal can switch the canary model version for the remaining work without restarting the execution. Recurring inference or retraining can be started through Temporal Schedules, which the documentation describes as a more flexible and user-friendly approach than cron jobs. Python @workflow.query def status(self) -> dict: return {"phase": self.phase, "completed": self.completed, "model": self.model_uri} @workflow.signal def switch_model(self, model_uri: str) -> None: self.model_uri = model_uri Keeping ML Workflows Operable as They Grow CNN platforms do not stay small for long. A single training run becomes a hyperparameter sweep, then a retraining program, then a fleet of region-specific models. Temporal scales that expansion through composition. A parent workflow can start child workflows for each experiment, fold, or dataset shard, and the child workflow APIs guarantee that the child has actually started before the call resolves. That makes fan-out training and batch inference easier to reason about than out-of-band job launchers with partial status tracking. Long-running ML control planes also run into two operational realities: event history growth and code evolution. Temporal addresses the first with Continue-As-New, which closes the current execution successfully and starts a new run with the same workflow ID and a fresh event history. It addresses the second with versioning support and, in production, Worker Versioning. Temporal recommends Worker Versioning as the default way to deploy changes safely, and pinned workflows can stay on the worker deployment version where they started. That is especially relevant for training or evaluation flows that may stay active across multiple application releases. Conclusion Temporal brings discipline to CNN systems by separating durable orchestration from non-deterministic GPU work. The workflow owns state, retries, waiting, composition, and observability. Activities own data movement, training submission, checkpointing, evaluation, and inference execution. PyTorch continues to provide the familiar mechanics of state_dict checkpoints, evaluation mode, and gradient-free inference, while Temporal turns those stages into a resilient end-to-end process that can survive infrastructure faults, external scheduler delays, and repeated deployment cycles. For teams building CNN platforms that have outgrown shell scripts and brittle job glue, that combination is not merely convenient. It is often the boundary between a model pipeline that occasionally works and one that can be operated confidently in production. More

Multi-Agent Software Engineering: Can AI Teams Build Production Systems?

By Uthej Mopathi

CORE

AI Assist vs AI Complete: The Real Gap in Most AI Workflows Today

By Muralidharan Lakshmanan

AI-Augmented React Development: How I Rebuilt My Workflow Without Losing Control of the Code

By Sathwik Nagulapati

Building Internal Developer Platforms as Products: A Practical Guide for IDP Architects

Why Most Platforms Fail to Become Products Many companies are heavily investing in internal developer platforms (IDPs) with the expectation that they will speed up delivery and governance, and increase developer productivity. Despite significant investment in Kubernetes, CI/CD, observability, security tooling, and cloud infrastructure, many platforms struggle to gain adoption. The reason is simple: they are built and operated like infrastructure projects, not products. Infrastructure teams are often very focused on technical excellence: automation, scalability, reliability, and compliance. Developers, on the other hand, are interested in a different goal — getting their applications into production quickly and safely without having to go through so much complexity. IDP is successful when developers choose it voluntarily because it makes their lives easier. That shift requires platform architects to think less like infrastructure engineers and more like product managers. Building an IDP is like operating an airport. Nobody travels because they love airports. They travel because they want to reach a destination efficiently. Similarly, developers do not care about Kubernetes clusters, pipelines, secrets management, or observability stacks. They care about shipping features to customers. The platform's job is to make the journey smooth, fast, and safe. This article explores the core practices that differentiate successful product-centric platforms from infrastructure-centric ones. Practice 1: Start With Developer Journeys, Not Technology Choices Imagine constructing a shopping mall by selecting elevators, security systems, and air-conditioning units before understanding customer traffic patterns. The result is often technically impressive but operationally frustrating. The same happens with developer platforms. Architects should first map the customer journey (developer journey) before designing platform capabilities. Many platform initiatives begin with questions like: Which Kubernetes distribution should we use?Which GitOps framework is best?Which CI/CD tool should be standardized? These are important questions, but they should not be the starting point. Successful platform architects begin by understanding developer workflows: How does a new service get created?How long does environment provisioning take?Where do deployment delays occur?What causes support tickets?Which activities are repetitive and manual? The goal is to identify friction and eliminate it. Organizations using platforms based on technologies like Red Hat OpenShift, IBM Cloud Kubernetes Service, or other cloud-native platforms have found that developers adopt only when the platform team focuses on reducing the friction in workflow rather than adding more infrastructure features to the platform. Practice 2: Create Golden Paths, Not Golden Handcuffs A highway encourages drivers to use the fastest route while still allowing exits when necessary. Successful IDPs behave like highways. Developers naturally choose the Golden Path because it is easier and safer than building everything from scratch. One of the most powerful concepts in modern platform engineering is the Golden Path. A Golden Path provides: Recommended architecturesStandard deployment patternsPre-approved security controlsBuilt-in observabilityAutomated CI/CD workflows Developers should be able to move fast along a paved road while retaining flexibility for unique requirements. Platform teams that leverage services from cloud provider environments often realize that standardized self-service templates drive significantly higher adoption than restrictive governance models. Practice 3: Make Self-Service the Primary Interface Every banking transaction once required a visit to a physical branch. Today, customers expect to do everything from a mobile app. Developers hope for the same experience from inside their own software. Nothing kills developer productivity faster than dependency queues. Consider a common case of dependency queues. Open a ticket for infrastructure.Wait for approval.Wait for provisioning.Request secrets.Request monitoring.Request deployment access. Weeks can pass before development even begins. Modern platforms must provide self-service experiences where developers can do the following without opening tickets. Create environmentsProvision databasesConfigure pipelinesAccess observability dashboardsRequest infrastructure resources An IDP should function like a digital banking application—secure, streamlined, and available on demand. Below is the Product-Centric IDP reference architecture. Developers consume platform capabilities through self-service experiences, while the platform embeds security, observability, governance, and delivery capabilities and exposes them through Golden Paths. Practice 4: Treat Platform APIs as Products A power drill might have sophisticated engineering in it. Users judge it by a very simple standard: “Can I drill a hole fast and reliably?" Many platform teams are focused on infrastructure automation and not developer experience. Each API, template, workflow, and portal interaction is a product interface. Questions worth asking include: Is the API predictable?Is documentation clear?Are error messages actionable?Is onboarding intuitive?Can developers discover capabilities easily? Developers evaluate IDPs the same way. They are not interested in the complexity underneath. They care about usability. This principle is especially important when integrating observability services, cloud provisioning layers, or deployment automation platforms. For example, IBM Cloud's managed services can significantly simplify operational complexity, but value is realized only when developers experience that simplicity through intuitive platform workflows. Practice 5: Build Observability into the Platform, Not Around It Imagine when you are driving a car without any speedometer, fuel gauge or warning indicators. You may still reach your destination but the risk increases dramatically. Observability is the dashboard for software systems. Observability is often treated as an afterthought. A team deploys an application and later attempts to add: MetricsLogsTracesDashboardsAlerting This approach creates inconsistency and operational blind spots. Platform teams should embed observability from day one. Every service created through the platform should automatically include: Logging standardsDistributed tracingMetrics collectionHealth monitoringService dashboards Whether organizations use IBM Cloud Observability, Instana, OpenTelemetry, Prometheus, Grafana, or other solutions, the platform should make observability automatic rather than optional. Practice 6: Make Security Invisible but Ubiquitous When entering a modern office building, people rarely think about security. Access badges, surveillance, and emergency controls are built into the environment — the building is secure without requiring employees to become security experts. The same principle applies to IDPs. In immature environments, security is seen as a series of checkpoints, review meetings, manual compliance approvals, vulnerability assessments, and audit evidence collection. Developers find it as friction because it arrives late in the delivery lifecycle. Traditional security models operate as gates. Platform-centric security operates as guardrails. The objective is not fewer security controls — it is fewer manual interactions. Build Secure-by-Default Golden Paths Every new service created through the platform should automatically inherit: Secure CI/CD pipelines with dependency and container image scanningSecret detection and policy enforcementAccess control standards and audit loggingEncryption best practices Automate Policy Enforcement Manual compliance verification is one of the biggest sources of deployment delays. Platform teams should adopt policy-as-code (PaC) approaches that automatically validate deployment configurations, infrastructure standards, and regulatory controls. Instead of asking, "Did someone review this configuration?" the platform asks, "Does this configuration satisfy our policies?" Reduce Security Cognitive Load Developers should not need deep expertise in every security domain. The platform should abstract identity management, secrets management, certificate management, and vulnerability remediation workflows—particularly in hybrid and multi-cloud environments where security complexity grows rapidly. A useful measure of progress: the percentage of security controls inherited from the platform versus manually implemented by application teams. The higher the inheritance rate, the lower the cognitive load. Practice 7: Measure Platform Success Like a Product A gym owner does not measure success by counting treadmills—they measure it by member outcomes. Platform teams should apply the same logic. Traditional infrastructure metrics like cluster utilization, pipeline counts, and resource consumption tell you whether the platform is running. They do not tell you whether it is working for developers. Product-oriented platform teams focus on: Developer satisfactionPlatform adoptionTime to first deploymentDeployment frequencyLead time for changes If developers still circumvent the platform, no amount of technical sophistication matters. The Developer Experience Scorecard Measuring developer experience requires balancing sentiment, effort, and adoption. High-performing platform teams track four key measures: Metric What It Measures How to Collect Developer Satisfaction Score (DSS) Overall platform sentiment Quarterly survey, 1–10 scale Platform NPS Willingness to recommend the platform "How likely are you to recommend this platform?" scored 0–10 Ease-of-Use Score How intuitive common workflows feel Per-task rating, 1–5 scale Developer Effort Score How much work is required to achieve an outcome Survey question on effort per task Together, these reveal not just whether developers are using the platform but whether they genuinely value it. Satisfaction Is a Leading Indicator Most delivery metrics lag behind—deployment frequency (e.g., lead time, incident count) and other metrics. Developer satisfaction is a leading indicator. Developers discover friction long before it is observable from the data. A declining DSS today will result in a decline in productivity and adoption tomorrow. Listening early allows platform teams to respond before problems grow into organizational challenges. The real measure of success is not how many developers use the platform—it is how they feel while using it. The IDP Health Dashboard High-performing platform teams monitor a balanced set of metrics across four categories: Category Metrics Sentiment DSS, Platform NPS, Ease-of-Use ratings Adoption Golden Path adoption, self-service usage, onboarding rates Friction Support ticket volume, documentation search failures, manual approval requests Productivity Time to First Deployment (TTFD), environment provisioning time, lead time for changes A platform succeeds not when developers are forced to use it, but when they prefer to use it. Practice 8: Reduce Cognitive Load Relentlessly The automotive industry spent decades simplifying the driving experience so drivers could focus on reaching their destination rather than understanding the mechanics of their vehicles. IDPs should do the same. As organizations evolve into cloud-native architectures, developers are expected to navigate containers, Kubernetes, CI/CD, IaC, security policies, service meshes, observability tools, and compliance requirements all at once. Each one solves a very important problem individually. As a whole, they overwhelm developers and take focus away from developing business capabilities. A successful platform is not one that exposes every infrastructure capability. It is one that hides unnecessary complexity while providing simple, intuitive paths to outcomes. The goal of platform engineering is not to eliminate complexity. It is to absorb complexity so developers don't have to. Common indicators of excessive cognitive load: Developers struggling to find documentationFrequent support requests for routine tasksLong onboarding times for new servicesMultiple handoffs between teamsTool sprawl across the engineering ecosystem Reduce Tool Sprawl Every tool a developer must learn introduces new interfaces, terminology, documentation, and configuration models. Platform teams should create a unified experience through a developer portal, service catalog, or platform API, that minimizes the number of decisions and interfaces developers encounter. Minimize Context Switching Every transition between tools, teams, or approval processes introduces cognitive overhead. Platform teams should ask: Can this be automated? Can these steps be consolidated? Can approvals be replaced with automated guardrails? The goal is fewer interruptions between code creation and deployment. Platform Teams Are Complexity Brokers Complexity never disappears — it moves. Organizations can either push complexity onto every development team, or centralize and manage it within the platform. High-performing platform teams choose the latter, absorbing operational, security, infrastructure, and compliance complexity so application teams can focus on features. Practice 9: Obsess Over Time to First Deployment The first experience developers have with a platform often determines whether they embrace it or avoid it. Imagine a shopping mall where opening a new store requires twelve forms, multiple approval queues, and manual setup of every utility. Store owners would go elsewhere. The best malls provide ready-made spaces where businesses can start operating almost immediately. Developer platforms should do the same. High-performing platform teams focus relentlessly on Time to First Deployment (TTFD) — the time between creating a service and successfully deploying it. The Biggest Contributors to Poor TTFD Bottleneck Root Cause Fix Manual infrastructure provisioning Ticket-driven approval chains Self-service IaC, service catalogs, platform portals CI/CD pipelines built from scratch No standard templates Pre-built, reusable pipeline templates Security reviews at the end Late-stage compliance gates Shift left — embed scans and policy checks in Golden Paths Observability setup delays Manual metrics/dashboard configuration Auto-provision logging, tracing, and health checks by default Too many decisions Choice overload at onboarding Provide Golden Paths with sensible defaults Measure Every Stage Stage Target Service creation < 5 mins Repository creation Automated Pipeline creation Automated Infrastructure provisioning < 10 mins First build < 5 mins First deployment < 15 mins Observability enablement Automatic TTFD = Provisioning Time + Setup Time + Approval Time + Deployment Time Many organizations discover that approval time is larger than all technical activities combined. The fastest platforms replace approvals with automated guardrails. Practice 10: Build a Platform Community, Not Just a Platform Team Cities flourish when residents contribute feedback and shape growth. Cities planned entirely from a central authority often struggle to meet citizen needs. IDPs are no different. The best platforms evolve through continuous collaboration. Platform teams should create feedback loops through office hours, community forums, developer councils, internal documentation reviews, and experience surveys. Developers become co-creators rather than consumers. Community Health Metrics Running community mechanisms is not enough — each one needs a way to know whether it is working. Track these six indicators to measure community health: Metric What It Measures Healthy Signal Monthly Active Community Members Developers engaging in forums, channels, or office hours Steady growth quarter over quarter Developer-to-Developer Answer Rate % of forum questions answered by non-platform-team members Above 40% indicates a self-sustaining community External Contributions per Quarter Pull requests or documentation edits from application teams Increasing trend Roadmap Items from Community Input % of platform backlog items originating from developer feedback Above 50% signals product-centric culture Office Hours Repeat Attendance Rate % of attendees who return across multiple sessions Above 60% indicates ongoing value Support Ticket Deflection Rate % of issues resolved via community before a ticket is opened Rising deflection reduces platform team toil The ultimate sign of a mature platform community is a change in how developers talk about the platform—from something that happens to them to something they help shape. Practice 11: Think in Products, Roadmaps, and Customer Value Smartphones succeeded because manufacturers continuously improved user experience. Customers did not buy phones because of processor specifications. They bought outcomes—better communication, productivity, and convenience. Developers adopt platforms for the same reason. The strongest indicator that a platform is becoming a product is a change in language. Instead of asking: What infrastructure should we standardize? Platform teams begin asking: What developer problems should we solve next? Which user journeys create the most friction?Which capabilities deliver the highest value?What does our product roadmap look like? Features matter only when they improve the developer experience. Practice 12: Design for Platform Reliability, not Just Application Reliability Imagine a city that invests heavily in building roads, bridges, and public transport for its citizens, but has no maintenance crew, no traffic monitoring, and no plan for when a bridge closes. The infrastructure exists, but without reliability commitments, citizens cannot depend on it. Internal developer platforms face exactly the same risk. Most platform engineering conversations focus on the reliability of applications running on the platform — uptime, error rates, latency SLOs for customer-facing services. What is rarely discussed is the reliability of the platform itself. Yet the platform is load-bearing infrastructure for every engineering team in the organisation. When the CI/CD pipeline degrades, every team's delivery stops. When the service catalog is unavailable, no new services can be provisioned. The platform's reliability is a multiplier — a single failure can simultaneously impact dozens of teams. Define Platform SLOs Before Developers Define Them for You Platform teams that do not define their own Service Level Objectives will find that developers define them informally — through frustration, workarounds, and loss of trust. Effective platform SLOs cover the experiences developers depend on most: Pipeline availability — what percentage of CI/CD pipeline executions succeed without infrastructure-related failures?Provisioning latency — how long does environment or resource provisioning take at the 95th percentile?Portal availability — is the developer portal and service catalog accessible during working hours?Golden Path build time — how long does a standard pipeline template take to complete? These are the experience metrics developers encounter every day. A platform team that publishes and tracks these SLOs operates as a reliable internal service provider. A team that does not is invisible until something breaks. IDP Maturity Model Stage Characteristics Infrastructure Platform Standardized infrastructure, clusters, CI/CD tooling Self-Service Platform Service catalogs, automation, infrastructure on demand Developer Platform Golden Paths, integrated observability and security, DevEx focus Platform Product Platform roadmaps, adoption metrics, developer satisfaction measurement Adaptive Platform Continuous feedback loops, AI-assisted operations, continuous platform evolution Most organizations do not start with a Platform Product. They evolve toward it. The goal of the maturity model is not to reach the highest stage overnight, but to identify the next set of capabilities that will improve developer experience and platform adoption. High-performing platform teams treat platform maturity as a journey rather than a destination. Assessing Your Current Stage To identify where your platform currently sits, ask three diagnostic questions: How do developers access platform capabilities today? If the answer is "by opening a ticket," the platform is at the infrastructure stage. If developers provision resources on demand without human approval, they are at the self-service stage or beyond.Do developers choose the platform voluntarily or use it because they must? Voluntary adoption driven by speed and simplicity signals a developer platform or platform product. Mandatory usage with frequent workarounds signals an earlier stage.Does the platform team maintain a product roadmap prioritized by developer feedback? A yes here is the clearest indicator of a platform product. The absence of a roadmap almost always reflects an infrastructure or self-service mindset. Moving to the Next Stage Each stage has a single dominant unlock that drives progression: Infrastructure → Self-Service: Replace ticket-driven provisioning with self-service automation and a service catalog.Self-Service → Developer Platform: Introduce Golden Paths that embed security, observability, and CI/CD by default.Developer Platform → Platform Product: Establish a formal platform roadmap, measure developer satisfaction (DSS, NPS), and treat developer feedback as a product backlog.Platform Product → Adaptive Platform: Build continuous feedback loops, introduce AI-assisted operations, and invest in platform telemetry that proactively surfaces friction before developers report it. The most common mistake is attempting to skip stages. Teams that build Golden Paths before self-service exists create well-designed paths nobody can access independently. Teams that adopt satisfaction metrics before Golden Paths exist measure friction without the tools to address it. Progress through the stages in order. The IDP Architect's Checklist Before launching any new platform capability, ask: ✅ Does this feature remove friction from a developer workflow? ✅ Can developers access it through self-service? ✅ Is it aligned with a Golden Path? ✅ Is observability included by default? ✅ Is security built into the platform? ✅ Is governance automated rather than manual? ✅ Can success be measured through developer outcomes? ✅ Does it reduce cognitive load? ✅ Does it improve Time to First Deployment? ✅ Would developers choose this platform if they had alternatives? If the answer to several of these questions is "no," the capability is probably infrastructure-focused rather than product-focused. Final Thoughts The future of platform engineering is not about building more infrastructure. It is about delivering better developer experiences. The most successful IDPs combine the discipline of site reliability engineering (SRE), the automation of cloud-native technologies, and the mindset of product management. Whether your foundation runs on IBM Cloud, OpenShift, hyperscaler cloud services, or a hybrid environment, the winning formula remains the same: Treat developers as customers. Treat the platform as a product. Treat developer productivity as the ultimate business metric. When platform architects embrace this mindset, platforms stop being collections of tools and start becoming accelerators of innovation—and that's when platforms truly become products.

By Josephine Eskaline Joyce, Ph.D

CORE

How I Built a Star Wars Grogu Product Research Agent With Codex, Lark, and SerpApi

Cross-border e-commerce sellers often spend hours comparing the same products across different Amazon marketplaces. Prices, reviews, and seller signals vary by country, but the process is still largely manual. I wanted to see how far I could automate it with a small AI agent built using Codex, SerpApi, and Lark. Suppose I ask in Codex: Compare Grogu products in the US and Japan The workflow is straightforward: Search Amazon marketplacesFetch product detailsSummarize seller signalsSend a report to Lark I kept the stack intentionally simple: Python 3.12uvPydanticSerpApi Python SDKOpenAI structured outputsLark custom bot webhookCodex for both development and as the conversational interface Running the Agent From Codex Once everything was wired together, I could simply ask Codex: Compare Grogu products in the US and Japan Codex understood the request, triggered the CLI workflow, collected marketplace data via SerpApi, and delivered a structured report to Lark. Here’s the entire flow in action: The Workflow The overall flow looks like this: Starting with a natural-language request in Codex, the agent first validates that the question is related to cross-border Amazon research. OpenAI then translates the request into a structured command. SerpApi handles both product discovery and detailed product retrieval, while OpenAI extracts seller-focused insights from the collected data. Finally, the results are packaged into a Lark card and delivered via a Custom Bot webhook. I intentionally did not start with FastAPI, background jobs, or a database. For the MVP, the important question was simpler: Can I go from a natural-language product question to a useful cross-border product card? Translating Natural Language into Commands The natural-language entry point uses OpenAI to translate user requests into structured commands. For example: Compare Grogu products in the US and Japan becomes: YAML query="Grogu" marketplaces=["us","jp"] output_mode="send_lark" Using a strict schema keeps the pipeline predictable and much easier to debug. Instead of letting the model orchestrate everything, I only ask it to generate structured commands. Searching Amazon With SerpApi This project relies on two SerpApi endpoints. The Amazon Search API is used to discover candidate products, while the Amazon Product API enriches them with much richer details. Search results provide ranking context and thumbnails, while product pages contain detailed information such as images, availability, and descriptions. Combining both produced much better product cards than using either endpoint alone. Modeling Product Data Amazon pages are messy. Some products have ratings but no availability. Some have images but no variants. Some fields simply don't exist. Missing data is normal, not an exception. My first instinct was to say, "Why not model most fields as optional?" That is true, but it is not the whole solution.The real issue was that SerpApi returns useful product data from several different places. Some fields are flat. Some fields are nested. Some fields have different names depending on whether they came from Amazon Search API or Amazon Product API. Some products should not be shown at all if they are missing the signals a seller actually needs. To make the data usable, I: Normalized both endpoints into a common Product model;Filtered out products with weak seller signals;Merged Product API details back into search results;Treated missing fields as expected rather than failures. The Product model still uses optional fields, because missing data is normal: But optional fields alone were not enough. I also filtered search results before choosing products for detail lookup: This was important for the Lark card. A cross-border seller does not want a table full of Rating: N/A and Reviews: N/A. Those rows make the card noisy and less actionable. The next issue was nested and inconsistent JSON. For price, SerpApi may return a string, a number-like value, or a nested object: For availability, the Product API does not always use one stable field. I had to check availability, stock, and sometimes delivery: Keeping the Agent Narrow The most interesting part of the project isn't Grogu. It's scope. The entry point only supports cross-border Amazon product research. If somebody asks: What is today's weather? the app simply rejects the request. Requests outside the project scope are rejected locally before calling OpenAI. For supported requests, OpenAI returns a strict command object: This keeps the agent predictable. It translates requests into commands instead of improvising actions. I deliberately avoided turning this into a general chatbot. The agent only knows one workflow: cross-border Amazon product research. That narrow scope makes the behavior easier to explain, test, and trust. It also keeps OpenAI API responsible for translation rather than improvisation. Generating Seller-Friendly Insights and Delivering Them to Lark Once product data has been collected, OpenAI generates seller-focused insights. I intentionally constrain the model to use only information returned by the APIs, avoiding hallucinated prices, ratings, or availability. Instead of producing generic summaries, the analysis focuses on demand signals, social proof, pricing, and obvious risks — information that is much more useful for sellers. The final result is delivered through a Lark Custom Bot webhook. Error Handling External APIs fail. That's normal. I treated each layer independently. If OpenAI fails, no command is generated. If SerpApi fails, the analysis stops with a clear message. If Lark delivery fails, the report can still be viewed locally. This separation keeps failures localized and prevents one component from bringing down the entire workflow. Conclusion The most interesting lesson from this project wasn’t the Grogu theme or the Lark card. It was learning where the boundaries should be. The agent works because it stays narrow. That narrowness makes the system easier to understand, debug, and trust. And if you’re building tools for cross-border e-commerce, SerpApi’s Amazon API provides a surprisingly rich source of product data. They made this entire workflow possible. If you’re working on product research, seller analytics, or marketplace intelligence, I’d recommend giving them a try. Check out the full SerpAPI article collection here.

By Magenta Qin

Scaling Teams, Scaling Systems: Unlocking Developer Productivity With Platform Engineering

Modern software delivery is complex. Developers are responsible not only for writing code that meets business requirements — both functional and non-functional — but also for navigating a long chain of supporting steps. From containerization, testing, configuration, security, deployment, and monitoring, each stage often relies on specialized tools and teams. When these processes aren’t standardized, every project risks reinventing the wheel. The result is inconsistency, delays, and frustration. For example, requesting a new test environment might require submitting detailed tickets to a DevOps team, slowing timelines and draining energy. As organizations scale, so does the complexity — and the pain of delivery. Platform engineering addresses these challenges by creating shared, reliable foundations. It provides self-service tools, standardized workflows i.e., golden paths, and built-in guardrails, enabling teams to focus on what matters most: writing code and shipping features. This article explores what platform engineering is, why it matters, and how it helps organizations move faster while reducing developer burnout. It also examines common challenges and how to avoid turning platforms into yet another layer of complexity. Platform Engineering Definition Platform engineering is a practice of building and maintaining an internal, self-service platform that makes it easy for development teams to build, deliver, and operate software. Key principles of platform engineering are: Self-service (with guardrails) → developers should be able to build, deploy, and operate services independently — without filing tickets for routine tasks. Also ensuring automated guardrails for compliance and cost control.Golden paths, not golden cages → provide opinionated, well-supported paths that make the right thing easy — without preventing teams from choosing alternatives when needed.Product mindset → treat the platform as a product. Define users (developers), gather feedback, measure adoption, and iterate based on value delivered.Reduce cognitive load → abstract away infrastructure and operational complexity that does not directly contribute to the developer’s core task, i.e., building and shipping business logic. It's imperative to note that Platform Engineering is not DevOps, but DevOps scaled through product thinking — treating developers as customers and the platform as the product. Adoption Journey Large organizations often face delivery challenges that rarely make it into executive summaries. Issues like developer friction, inconsistent and/or duplicate tooling, and fragmented workflows are deeply embedded in day‑to‑day operations. Their impact — delayed releases, inefficiencies, and frustration — may be visible, but the root causes often remain hidden or disconnected from leadership narratives. Thus, the first step is discovery and validation. Organizations must surface real pain points through design thinking workshops, targeted surveys, analysis of past initiatives, and continuous community/user feedback. These insights form the foundation for defining a clear and grounded Platform Mission Statement — one that aligns platform capabilities with genuine organizational needs. Once the mission is clear, enterprises move toward unified platforms that standardize common tools and processes. This consolidation reduces duplication and improves reliability. Guiding Principles should be maintained via ADRs as a standardized platform for uniformity. Also, it's recommended to have decentralized decision-making to avoid bottlenecks and maintain long-term sustainability. To achieve this, use a community-driven approach via various guilds. As maturity grows, self‑service enablement becomes the focus — developers can provision infrastructure, build & deploy applications, perform verification, and integrate monitoring with minimal friction. This can be achieved via Internal Developer Platforms (IDP) and Internal Developer Portals. Finally, mature organizations embrace continuous improvement. The platform evolves like a product — guided by developer feedback and metrics. The feedback loops should act to adapt the platform and be evolutionary in nature. Being preventive or proactive, rather than reactive, goes a long way in achieving a successful platform implementation. Internal Developer Platforms (IDP) vs. Internal Developer Portals A common source of confusion in platform engineering is the distinction between an internal developer platform (IDP) and an internal developer portal. While these concepts are related and often work in tandem, they serve distinct purposes and have different architectural and user experience implications. Internal Developer Platform (IDP) An IDP is the “engine room” of platform engineering. It is a cohesive set of tools, frameworks, and automation scripts that standardize and automate the provisioning, deployment, and management of infrastructure and services. Key components typically include: Self-service infrastructure provisioning → Developers can request and manage resources (VMs, databases, clusters) via APIs or CLI tools, eliminating ticket-based workflows.Unified deployment and orchestration → Standardized CI/CD pipelines, container orchestration (e.g., Kubernetes), and Infrastructure as Code ensure consistent, reliable releases.Centralized configuration and secrets management → Version-controlled settings, automated secret management, and policy enforcement across environments.Automated monitoring and observability → Integrated metrics, logs, and tracing provide real-time visibility into system health.Security and compliance automation → Policy-as-code frameworks (e.g., OPA) enforce security and compliance at every stage. IDPs are typically built and maintained by platform engineering teams and are consumed by developers and operations teams to accelerate software delivery and reduce operational risk. Internal Developer Portal An internal developer portal is the “front door” to the platform. It provides a user-friendly interface (often a web dashboard) that aggregates documentation, service catalogs, APIs, and organizational guidelines. Key features include: Service catalog → Centralized inventory of services, APIs, and infrastructure, supporting discoverability and ownership tracking.Integration ecosystem → Unified view of the development toolchain, integrating with version control, CI/CD, observability, and project management tools.Self-service workflows → Guided forms and wizards for routine operations (e.g., provisioning, deployments), with built-in approval workflows and RBAC.Onboarding and knowledge sharing → Centralized documentation, onboarding guides, and community Q&A features to accelerate ramp-up and collaboration.Metrics and scorecards → Dashboards tracking service health, maturity, and compliance, providing actionable insights for improvement. Portals are typically used by application developers, product teams, and managers to discover services, access documentation, and initiate self-service workflows. When to Use Each (or Both) Start with an IDP when the primary pain points are manual infrastructure provisioning, inconsistent environments, or the need for standardized automation.Start with a Portal when discoverability, onboarding, and knowledge sharing are the main challenges, or when existing tools are underutilized due to lack of visibility.Combine Both for maximum impact: the IDP provides the backend automation and guardrails, while the portal exposes these capabilities through an intuitive, developer-friendly interface Team Topologies and Platform Engineering The success of platform engineering is deeply influenced by organizational structure and team interactions. The “Team Topologies” framework provides a powerful lens for designing team structures that optimize for effective platform adoption Four Fundamental Team Types Stream-Aligned Teams → Aligned to a flow of work from a business domain (e.g., a product or service). They own the end-to-end delivery and operation of features.Platform Teams → Build and maintain internal platforms that provide reusable services and capabilities to stream-aligned teams, reducing their cognitive load.Enabling Teams → Help stream-aligned teams overcome obstacles, adopt new technologies, or fill skill gaps.Complicated Subsystem Teams → Own subsystems that require deep specialist knowledge (e.g., advanced algorithms, core infrastructure). Three Team Interaction Modes Collaboration → Teams work together for a defined period to discover new solutions.X-as-a-Service → One team provides a service that another team consumes with minimal interaction.Facilitation → One team helps another team acquire new skills or capabilities. Relevance to Platform Engineering Platform Teams as Product Teams → Platform engineering teams should operate as product teams, treating stream-aligned teams as customers, gathering feedback, and iterating on platform featuresReducing Cognitive Load → The primary goal of the platform team is to reduce the cognitive load on stream-aligned teams, enabling them to focus on delivering business value rather than infrastructure concernsClear Interfaces and Boundaries → Well-defined APIs, documentation, and support channels ensure that platform capabilities are discoverable and consumable as a service, minimizing dependencies and handoffsContinuous Adaptation → Team boundaries and responsibilities should evolve as business needs and technologies change, with feedback loops guiding organizational adjustments Conway’s Law and Its Applicability to Platform Engineering Conway’s Law, articulated by Melvin Conway in 1967, states that “organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations” In the context of platform engineering, this law has profound implications. How Conway’s Law Shapes Platform Design Organizational Silos Lead to Siloed Platforms → If engineering teams are organized in silos (e.g., by business unit or product line), the platforms they build will reflect this fragmentation, resulting in duplicated tools, inconsistent practices, and integration challengesCross-Functional Collaboration Enables Cohesive Platforms → Successful platform engineering requires cross-functional teams that span development, operations, security, and compliance, ensuring that the platform addresses the needs of all stakeholders and avoids becoming a new silo.Intentional Organizational Design → To achieve coherent, scalable platforms, organizations must deliberately design their communication structures and team interactions to support shared standards, rapid feedback, and continuous improvement. Real-World Example - Monolith to Microservices → Organizations transitioning from monolithic architectures to microservices often reorganize teams around domains or services. If team boundaries are not aligned with desired system boundaries, the resulting architecture may become fragmented or inconsistent, reflecting the underlying communication patterns rather than optimal technical design Platform Engineering as Both Solution and Symptom → Platform engineering often arises as a response to the silos and fragmentation created by previous organizational structures. However, if not implemented with a product mindset and cross-team alignment, platform engineering can inadvertently create new silos, perpetuating the very problems it seeks to solve Metrics and KPIs to Measure Platform Success Measuring the impact of platform engineering is essential for demonstrating value, securing buy-in, wider adoption, and guiding continuous improvement. Common Metrics DORA Metrics → Deployment frequency, lead time for changes, change failure rate, mean time to recovery (MTTR).SPACE Framework → Satisfaction and well-being, performance, activity, communication and collaboration, efficiency and flow.Adoption Rates → Percentage of teams and services using the platform.Time to Onboard → Time required for new developers to become productive.Operational Metrics → Incident rates, uptime, resource utilization, and cost savings.Developer Satisfaction: Surveys, Net Promoter Score (NPS), and qualitative feedback. Feedback Mechanisms Surveys and Office Hours → Regular check-ins with platform users to gather qualitative and quantitative feedback.Telemetry and Usage Analytics → Automated tracking of platform usage, feature adoption, and workflow bottlenecks. Limitations and Common Challenges of Platform Engineering Despite its benefits, platform engineering is not without challenges and limitations. Organizational Challenges Resistance to Change → Teams may be reluctant to adopt new tools or workflows, especially if they perceive a loss of autonomy or increased complexity.Alignment and Buy-In → Achieving consensus on standards, priorities, and platform direction can be difficult in large or distributed organizations.Skill Gaps → Building and maintaining platforms requires expertise in infrastructure automation, CI/CD, security, and developer experience, which may be lacking in existing teams. Technical Challenges Overengineering → Building overly complex or rigid platforms can lead to low adoption and maintenance burden.Integration Complexity → Aggregating data and workflows from diverse tools and systems requires careful planning and robust integrations.Legacy Systems and Technical Debt → Integrating with or migrating from legacy tools and architectures can be time-consuming and costly. Cultural Challenges Product Mindset → Treating the platform as a product, with continuous feedback and iteration, is essential but often overlooked.Avoiding the “Golden Cage” → Mandating platform adoption without addressing real developer needs can lead to resentment and shadow. Measurement and ROI Lack of Metrics → Many organizations fail to measure platform adoption, impact, or ROI, making it difficult to justify continued investment or guide improvements. Products Over Projects In platform engineering, the distinction between a project mindset and a product mindset is crucial. Project Mindset Focus → Deliverables, deadlines, and completion.Approach → Work is structured around a defined scope with a start and end date.Outcome → Once the project is “done” the team moves on, often with limited ongoing ownership.Risk → Platforms built this way may stagnate, as continuous improvement and user feedback loops are not prioritized. Product Mindset Focus → Long-term value, user experience, and continuous evolution.Approach → The platform is treated as a living product with ongoing investment, iteration, and support.Outcome → Teams own the platform end-to-end, ensuring it adapts to changing business and developer needs.Benefit → Encourages innovation and alignment with evolving enterprise strategy. In short, with a project mindset, tasks are done as “Deliver and finish,” while with a product mindset, it's “Deliver, own, and evolve.” Relationship Between Platform Engineering, DevOps, and SRE While platform engineering, DevOps, and site reliability engineering (SRE) share common goals, they address different layers of the software delivery lifecycle. DevOps Focus → Cultural and organizational transformation to break down silos between development and operations, emphasizing collaboration, automation, and continuous delivery.Practices → CI/CD, infrastructure as code, shared responsibility for quality and reliability. SRE Focus → Applying software engineering principles to operations, with a strong emphasis on reliability, scalability, and incident response.Practices → Service-level objectives (SLOs), error budgets, automated monitoring, and incident management. Platform Engineering Focus → Building and maintaining internal platforms that provide standardized, automated, and self-service capabilities for development teams.Practices → Product mindset, self-service, golden paths, policy-as-code, and platform-as-a-product. How They Interact Platform engineering provides the foundation on which DevOps and SRE practices can scale, standardizing workflows, embedding automation and compliance, and enabling self-service for developers and operations teams.DevOps and SRE teams collaborate with platform engineers to ensure that the platform supports reliability, scalability, and continuous improvement. Case Studies Netflix used their platform to solve developers’ challenges to manage multiple services and software, knowing which tools exist, and switching contexts between tools. Zalando leveraged their platform to unify the developer experience, promote compliance by default, and improve how the company operated over time. Carlsberg’s Gaia platform automated infrastructure provisioning, embedded compliance, and provided self-service capabilities, reducing manual work and accelerating project delivery. The platform’s success was attributed to cross-functional collaboration, a product mindset, and continuous feedback from developers eBay’s Velocity initiative (2020) boosted engineering productivity, cutting deployment times from 10 days to 1–2 and enabling same‑day mobile releases. Despite technical success, cultural resistance, outdated tech choices, and poor strategic execution prevented business growth. Conclusion Platform engineering represents a paradigm shift in how modern organizations build, deliver, and operate software at scale. By abstracting complexity, standardizing workflows, and empowering developers through self-service and automation, platform engineering accelerates delivery, improves reliability, and optimizes costs. However, its success depends on more than just technology — it requires intentional organizational design, a product mindset, continuous feedback, and a careful balance between standardization and flexibility. As the discipline matures, organizations that invest in platform engineering as a strategic capability will be best positioned to thrive in an increasingly complex and competitive digital landscape. The golden rule of platform engineering → Treat your platform as a product, your developers as customers, and adoption as a metric — not a mandate. Platform 2.0 — The AI‑Native Evolution Platform 2.0 represents the next stage of platform engineering, where artificial intelligence becomes a built‑in capability rather than an add‑on. It transforms platforms from static automation frameworks into adaptive, learning ecosystems that continuously optimize themselves. Core Principles Intelligence at every SDLC stage → AI augments design, development, testing, deployment, and operations with predictive and generative capabilities.Continuous learning → Feedback loops from telemetry and user behavior refine architecture and performance automatically.Autonomous optimization → Platforms self‑tune resources, detect anomalies, and evolve configurations without manual intervention.Human‑AI collaboration → Engineers focus on strategic design and governance while AI handles repetitive and analytical tasks. Platform 2.0 enables faster delivery, higher reliability, and smarter scalability, redefining platform engineering as an AI‑native discipline — intelligent, adaptive, and perpetually evolving. References and Further Reads Platform Engineering — WikipediaWhat is Platform EngineeringInfoQ trend report Platform Engineering as early adoptersThoughtWorks tech radar recommends adopting Platform Engineering as early as Apr/Oct 2021.Team TopologiesDZone survey where nearly half of respondents indicate using platform engineering.

By Ammar Husain

CORE

A Practical Guide to Temporal Workflow Design Patterns

Long-running, distributed business processes often require careful coordination, state management, and fault handling. Temporal offers a code-first approach to durable workflows: developers write ordinary code for orchestration, and the Temporal service persists state, retries failed tasks, and resumes execution after failures. This shifts focus from plumbing (queues, retries, timeouts) to domain logic, but it also encourages reuse of proven patterns. The Temporal community and documentation highlight several orchestration patterns — for example, sagas, state machines/actors, polling strategies, fan-out/fan-in, and versioning patterns — that solve recurring problems in workflow design. This article surveys these patterns, explaining when and how to use them, with concise code snippets to illustrate their implementation in Temporal. A classic pattern in distributed transactions is the Saga (compensating transaction). In a saga, a business process is broken into a sequence of steps, each with its own “undo” compensation. If any step fails, the saga executes compensations in reverse order to restore consistency. In Temporal, this maps naturally to a try/catch around activities or to the built-in Saga helper. For example, a vacation booking workflow might book a hotel, then a flight, then an excursion. Each step registers a compensation action before invoking the activity. If a failure occurs, the catch block calls saga.compensate() to run all registered compensations in reverse. The following Java-like snippet shows this approach: Java public void bookVacation(BookingInfo info) { Saga saga = new Saga(new Saga.Options.Builder().build()); try { saga.addCompensation(activities::cancelHotel, info.getClientId()); activities.bookHotel(info); saga.addCompensation(activities::cancelFlight, info.getClientId()); activities.bookFlight(info); saga.addCompensation(activities::cancelExcursion, info.getClientId()); activities.bookExcursion(info); // If all succeed, method returns normally. } catch (TemporalFailure e) { saga.compensate(); // undo previous steps throw e; // propagate failure } } If any book* activity throws an exception, the catch invokes saga.compensate(), which calls cancelExcursion, cancelFlight, and cancelHotel in reverse order. This pattern ensures that even if the workflow crashes after partial work, Temporal’s durable execution will eventually resume the compensation sequence. Because Temporal workflows are persistent, the saga logic itself is recoverable – the service records each step and its compensation in the history. In effect, workflows become distributed state machines where try/catch embodies the saga pattern. Polling and External Events Workflows often need to wait for external processes or inputs. In Temporal, there are two main polling strategies. Frequent polling (short interval) is implemented inside an activity loop: the activity repeatedly attempts a call, sleeps briefly, and heartbeats after each iteration. Because long-running activities must heartbeat to show liveness, the loop invokes Activity.getExecutionContext().heartbeat(null) each cycle. For example, a polling activity might look like this: Java @Override public String doPoll() { ActivityExecutionContext context = Activity.getExecutionContext(); while (true) { try { return service.getServiceResult(); } catch (TestServiceException e) { // Service not ready; will retry } // Heartbeat to prevent timeout, then sleep briefly context.heartbeat(null); sleep(POLL_DURATION_SECONDS); } } In this snippet, service.getServiceResult() is retried until it succeeds. Each loop iteration heartbeats and sleeps for a fixed interval. If the worker or service crashes, Temporal will resume the loop exactly where it left off. This pattern is ideal for rapid retries or waiting on resources that become available shortly. For infrequent polling, Temporal relies on activity retry options instead of a custom loop. A workflow can call an activity once, but configure its retry backoff so that failures trigger re-execution after longer delays. In practice, one sets a high initial retry interval and backoff coefficient in the ActivityOptions at workflow time. The workflow code itself is just a single activity call (no loop needed). If the activity throws an error, Temporal automatically retries it later, waiting longer each time. This approach leverages the built-in retry policy (e.g., exponential backoff) for occasional checks. To handle arbitrary external signals or time delays, Temporal workflows can also use Workflow.await(timeout, condition) or Workflow.newTimer(). For instance, a workflow might await a boolean flag that is set by a signal handler, or await a fixed timeout for human input. This avoids busy-wait loops at the workflow level. Signals themselves can come at any time; Temporal’s messaging system lets running workflows be interrupted by signals without polling. In short, Temporal workflows mix timers (Workflow.await) and external signals to wait efficiently. Frequent polling lives in an activity with heartbeats, whereas infrequent or one-off waits can use activity retry or workflow timers. Parallel and Batch Processing When processing large data sets or issuing many operations in parallel, Temporal’s fan-out/fan-in pattern is useful. A parent workflow can spawn multiple child workflows or activities concurrently and then wait for all to complete. This is commonly used for batch jobs, bulk queries, or any parallel computations. The following example shows a “page-by-page” batch processing workflow. For each batch of records, it spawns a child workflow per record and then uses Promise.allOf() to wait for all children. When a batch is done, it can optionally continue-as-new to process the next page without growing history indefinitely: Java @Override public int processBatch(int pageSize, int offset) { List<SingleRecord> records = recordLoader.getRecords(pageSize, offset); List<Promise<Void>> results = new ArrayList<>(); for (SingleRecord record : records) { String childId = Workflow.getInfo().getWorkflowId() + "/" + record.getId(); RecordProcessorWorkflow processor = Workflow.newChildWorkflowStub(RecordProcessorWorkflow.class, ChildWorkflowOptions.newBuilder().setWorkflowId(childId).build()); results.add(Async.procedure(processor::processRecord, record)); } // Wait for all child workflows to finish Promise.allOf(results).get(); // If no more records, return result and finish if (records.isEmpty()) { return offset; } // Otherwise continue as new for the next batch (to reset history) return nextRun.processBatch(pageSize, offset + records.size()); } In this code, each child workflow processes one record. The parent collects a list of Promise<Void> and calls Promise.allOf(...).get(), which blocks the parent until all child workflows complete. Using children allows highly parallel processing without overloading a single worker. After finishing a batch, the code checks if (records.isEmpty()) and returns; otherwise it calls a continueAsNew stub (nextRun) with an updated offset. This continueAsNew effectively starts a fresh workflow execution with a new history, avoiding unbounded history growth for long-running loops. As shown, Temporal’s Async and Promise primitives make parallel fan-out/fan-in straightforward. Beyond paging, fan-out can apply to any use case needing parallel work (bulk updates, scatter-gather queries, etc.). Conversely, gathering results into a list or aggregation is just collecting activity/child results into a shared variable, which Temporal safely persists in the history. Actor-Like Workflows and Event-Driven Patterns Temporal workflows are naturally stateful and can run indefinitely, making them suitable for actor or process-manager patterns. A workflow can “sleep” or wait for signals, maintain in-memory state, and react to external events. Clients can use signals (@SignalMethod) to send events into a running workflow and queries (@QueryMethod) to read its state without affecting it. This allows workflows to act like autonomous entities. For example, imagine a subscription service workflow. It starts with a customer on trial, waits for either trial expiration or a cancellation signal, then proceeds to billing periods. Signals like cancelSubscription() can interrupt the main flow. Meanwhile, queries like queryCustomerId() can retrieve the workflow’s state from outside. Temporal’s event system handles all this without polling: “a running workflow can receive external messages without polling, and clients can inspect workflow state at any time”. Internally, the workflow code can use Workflow.await(...) to pause until a signal sets a flag. Here’s a conceptual sketch (TypeScript/JavaScript style) of using signal and query definitions: TypeScript const abortSignal = defineSignal<[string]>('abort'); const updateSignal = defineSignal<[number]>('update'); const getStateQuery = defineQuery<State>('getState'); export async function statefulWorkflow(config: Config): Promise<Result> { let state: State = {...initial...}; let aborted = false; setHandler(abortSignal, (reason: string) => { aborted = true; }); setHandler(getStateQuery, () => state); // Main workflow logic: await condition(() => aborted, '1 minute'); if (aborted) { // cleanup or compensation return { status: 'aborted' }; } // ... continue normal processing return { status: 'completed' }; } In this pattern, external callers would workflow.signal(abortSignal, reason) or workflow.query(getStateQuery). Temporal’s signal-and-query features implement a process manager-style pattern: a workflow can behave like an event-driven state machine, reacting to signals in real time and allowing external inspection. This is more robust than polling, and since all state changes happen in the workflow code, consistency is guaranteed. (If a query is issued while the workflow is mid-activity, it will reflect the last completed state.) Note that newer Temporal releases also support Workflow Updates, which are like synchronous signals that can return values. In environments where Update is available, a workflow can reply to a message directly. Otherwise, a client can query state as a two-step “signal then query” process. Either way, this pattern empowers long-lived processes and human-in-the-loop steps. Versioning and Evolving Workflows Temporal requires workflow code to be deterministic, so changing logic in running workflows must be done carefully. The community and docs describe versioning strategies. For short-lived or rare workflows, one can deploy a new workflow definition (e.g. MyWorkflowV2) or use a new task queue for new versions. For long-lived workflows, Temporal’s Workflow.getVersion API lets the code branch on a version number recorded in the history. This is often called the “patch” strategy. For example: Java int version = Workflow.getVersion("checksumAdded", Workflow.DEFAULT_VERSION, 1); if (version == Workflow.DEFAULT_VERSION) { activities.upload(targetBucket, targetFilename, data); } else { long checksum = activities.calculateChecksum(data); activities.uploadWithChecksum(targetBucket, targetFilename, data, checksum); } Here, on first execution getVersion("checksumAdded", DEFAULT, 1) returns DEFAULT_VERSION and runs the original upload() call. When a new worker with updated code runs getVersion("checksumAdded", DEFAULT, 1) again, Temporal records version = 1 in the history. Future runs hit the else branch and use the new uploadWithChecksum() code. This ensures deterministic replay: workflows that started before the code change continue on the original branch, and newer executions use the new logic. After all old executions finish, the branching logic can often be removed. Overall, versioning patterns let developers evolve workflows without breaking running executions. Temporal offers multiple options — definition names, task queues, and the getVersion API — each with trade-offs. (Using separate definitions or queues isolates versions at the cost of more infrastructure, while getVersion keeps a single codebase but requires planned version markers.) Regardless, versioning is a key pattern to safely deploy workflow updates in production. Conclusion Temporal’s durable workflow engine incorporates many built-in aids for complex process patterns. By applying established designs — such as sagas for compensating transactions, retry and heartbeat loops for polling, fan-out/fan-in via child workflows, and event-driven actors with signals/queries — engineers can build robust systems without manual boilerplate. Each pattern leverages Temporal features: workflows and activities, promises, signals, queries, and continuations. The examples above show how little code is needed: a few method calls and standard control structures achieve what would otherwise be elaborate orchestration logic. In practice, adopting these patterns means that failures are handled gracefully and state is managed cleanly. For example, the saga code snippet illustrates reversing partial work on error, while the parallel batch example shows how to process unbounded data safely with continueAsNew. In summary, understanding Temporal’s idioms — as documented by the Temporal team and community — empowers developers to focus on business logic while the platform ensures reliability. Mastery of these workflow patterns leads to systems that are easier to reason about, easier to maintain, and resilient in production.

By Akhil Madineni

CORE

Cutting Data Pipeline Costs and Data Freshness Issues With Netflix Maestro and Apache Iceberg: A Practical Tutorial

Analytics pipelines tend to scale in both cost and the age of their data sources: costs increase with data volume growth, while data freshness decreases due to longer batch jobs. The common approach, scaling out the cluster, addresses the symptom rather than the architectural issue. In this tutorial, we will look at an alternative solution that addresses both problems at their root: using Netflix Maestro, a horizontally scalable workflow orchestrator open-sourced by Netflix in July 2024, along with Apache Iceberg, a standard table format for analytics on object storage. The former helps by shifting from time-based scheduling to event-driven, whereas the latter removes the overhead of listing files that slows down queries on large datasets and increases their costs. We will cover all aspects of creating a full-fledged pipeline, including code examples, explanations of why each component reduces costs, and real metrics showing what results to expect. What You'll Need ComponentPurposeNotesApache Iceberg + a catalogTable format and metadata managementREST catalog (Polaris, Nessie, Lakekeeper, Unity Catalog) recommended for new deployments; Glue/Hive also fineA compute engineReads and writes Iceberg tablesSpark 3.5+, Flink, Trino, or DuckDB via PyIcebergNetflix MaestroWorkflow orchestrationRequires Java 21, Docker, and Postgres or CockroachDB for stateCloud object storageData files and metadataS3, GCS, ADLS, or S3-compatible (MinIO works for local dev)Python 3.10+Lightweight tasks and ingestionPyIceberg 0.11+, PyArrow Terminology note: there are several products named "Maestro" in the data space. This guide is about Netflix's Maestro and is different from Maestro by Conductor, AWS Maestro, etc. Netflix's Maestro executes hundreds of thousands of workflows and up to 2 million jobs per day inside Netflix, so the scalability claim is valid — although some practitioners consider Maestro overengineered for small teams, so keep that in mind. The Problem Statement The standard stack on Hive tables stored in S3 has three structural inefficiencies: File listing dominates query planning. Listing operations on S3 are slow and rate-limited. For a query on a partitioned Hive table, listing might take more time than reading data itself.Small-file proliferation. Continuous or micro-batch writing produces thousands of Parquet files. Each query suffers from open-file overhead, and each list operation brings in additional results.Time-based scheduling wastes compute. Jobs are triggered based on a fixed schedule, not data availability. If upstream data is late, the job processes stale inputs. If the data is early, the job idles until the next scheduled run. Iceberg solves (1) and (2) in the storage tier. Maestro solves (3) in the orchestration tier. Let's see how. Why Iceberg Shifts the Cost Model Iceberg takes the table metadata out of the filesystem and puts it into a metadata tree. In response to the query "what files are part of this table?", the engine looks up a single metadata entry, follows the path to the manifest list, and gets back an exact list of data files, along with file-level statistics such as min/max values, null count, and row count. File discovery turns from an O(n) directory listing to O(1) metadata lookup. As a result, we get a chain reaction: Hidden partitioning. Declare a table PARTITIONED BY days(event_time), and queries filter on event_time directly. Partition transform happens automatically. No more WHERE year=2026 AND month=05 AND day=18, and no risk of analysts forgetting.Partition evolution. You can change the partitioning of the table from monthly to daily without rewriting old data. The metadata keeps track of it, and the engine routes queries correctly.Time travel and rollback. Writes produce immutable snapshots. If a bad load happens, you don't need to restore from backups – just roll the catalog pointer back to the previous snapshot. It matters operationally – recovery time goes from hours to seconds.Snapshot isolation and ACID. Writers operate concurrently; readers always see the consistent state, never a partial commit. The cost angle: manifest statistics can prune scans by an order of magnitude in time-filtered queries. With S3 list operations removed entirely, query costs on warehouse engines like Trino, Athena, or BigQuery (which charge per byte scanned) go down proportionally. Why Maestro Helps With Freshness and Costs The killer feature of Maestro in the context of our use case is the signal service — an event-driven trigger mechanism. Instead of scheduling "run this job at 02:00 every day", you tell Maestro to execute the job "when user_events_raw table receives a new snapshot". The trigger may originate from another Maestro workflow, an S3 event, a database table modification, or even from any external system capable of sending a request to the signal API endpoint. The gap between data arrival and data availability closes from hours (the worst-case batch window) to seconds or minutes. Other notable features of Maestro: Support for both DAGs and cyclic workflows. Unlike Airflow, Maestro allows loops and re-execution, which is useful for retry-with-backoff and convergence scenarios.ForEach loops and subworkflows as native concepts. Reduces the YAML sprawl common in large Airflow setups.At-least-once triggering with built-in deduplication leads to effective exactly-once execution.Mixed task types. A single workflow can combine Python, Spark, SQL (Trino/Presto), bash, notebook, Docker container, and Kubernetes jobs.100x performance improvement of the engine announced in September 2025 brings a step transition time from seconds to milliseconds, which is important for workflows with hundreds of steps. Step 1: Create the Iceberg Table With Sensible Defaults Begin with a definition of the table such that partitioning is done correctly from the start. By far the most frequent problem when adopting Iceberg is to overlook partitioning. SQL CREATE TABLE analytics.user_events ( user_id BIGINT, event_type STRING, event_time TIMESTAMP, session_id STRING, properties MAP<STRING, STRING> ) USING iceberg PARTITIONED BY (days(event_time), bucket(16, user_id)) TBLPROPERTIES ( 'format-version' = '2', 'write.target-file-size-bytes' = '134217728', -- 128 MB target 'write.parquet.compression-codec' = 'zstd', 'write.metadata.delete-after-commit.enabled' = 'true', 'write.metadata.previous-versions-max' = '20', 'history.expire.max-snapshot-age-ms' = '604800000', -- 7 days 'history.expire.min-snapshots-to-keep' = '10' ) LOCATION 's3://your-bucket/iceberg-tables/user_events'; Some interesting choices that should be explained: days(event_time) is a partitioning transform. Queries filtering by event_time will receive automatic partition pruning.bucket(16, user_id) is a bucket transform that evenly spreads writes among 16 buckets per day partition. It helps with hot spot prevention when one user produces disproportionately high amounts of traffic and provides better parallelism for joining on user_id.format-version = '2' allows for row-level deletions through delete files. V3 is a more recent version that adds many features, including deletion vectors, but make sure your engine supports it first.zstd provides better compression ratio by 10-20% compared to snappy with the same performance when reading.Expiring snapshot properties help avoid metadata explosion, which is one of the most frequent causes of costs silently accumulating in an Iceberg environment. Without this, each write would retain all previous snapshots indefinitely. Step 2: Ingest Data There are two reasonable options for ingesting data from Python into Iceberg: Spark (in case you already have a Spark cluster and need the scale provided by it) and PyIceberg (low overhead, no JVM required). Python from pyspark.sql import SparkSession from pyspark.sql.functions import to_timestamp, col spark = ( SparkSession.builder .appName("IcebergIngestion") .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog") .config("spark.sql.catalog.my_catalog.type", "rest") .config("spark.sql.catalog.my_catalog.uri", "https://your-rest-catalog/api/v1") .config("spark.sql.catalog.my_catalog.warehouse", "s3://your-bucket/iceberg-tables/") .config("spark.sql.catalog.my_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") .getOrCreate() ) raw = spark.read.json("s3://your-bucket/raw/events/2026-05-18/") events = ( raw .withColumn("event_time", to_timestamp(col("event_time"))) .select("user_id", "event_type", "event_time", "session_id", "properties") ) # MERGE INTO supports idempotent ingestion — important for replay safety events.createOrReplaceTempView("staging_events") spark.sql(""" MERGE INTO my_catalog.analytics.user_events t USING staging_events s ON t.user_id = s.user_id AND t.event_time = s.event_time AND t.event_type = s.event_type WHEN NOT MATCHED THEN INSERT * """) Two important aspects. First, the REST catalog should be used for any new deployment, as it allows accessing the same table via Spark, Trino, Flink, Snowflake, BigQuery, and PyIceberg without having to deal with catalog configurations drifting per engine. Second, using MERGE INTO instead of INSERT ensures that the ingestion becomes idempotent, especially when the step fails and Maestro tries to retry it. PyIceberg Ingestion (Lightweight Path) For lighter loads or ingestion processes executed as part of an orchestrator step, PyIceberg is quicker to initialize and has no dependency on the JVM. Currently, the library requires tables in PyArrow format, not pandas DataFrames: Python import pyarrow as pa from pyiceberg.catalog import load_catalog catalog = load_catalog( "my_catalog", type="rest", uri="https://your-rest-catalog/api/v1", warehouse="s3://your-bucket/iceberg-tables/", ) table = catalog.load_table("analytics.user_events") new_rows = pa.table({ "user_id": [3, 4], "event_type": ["purchase", "click"], "event_time": pa.array( ["2026-05-18T12:10:00", "2026-05-18T12:15:00"], type=pa.timestamp("us"), ), "session_id": ["sess-001", "sess-002"], "properties": [{"sku": "A123"}, {"page": "/home"}], }) table.append(new_rows) By default, PyIceberg uses "fast append" optimization, which reduces per-commit metadata operations but creates more manifest files than other optimizations. This is good for frequent micro-batch processing as long as you perform regular compaction (see below). Step 3: Define the Maestro workflow Maestro workflows can be defined using either JSON or YAML format. The following example defines a workflow that loads raw events, applies transformation, performs data quality checks, and updates the aggregate. Steps are connected by signals to start processing as soon as their dependencies are available. YAML name: user-events-pipeline description: Ingest, transform, validate, and aggregate user events trigger: signal: name: raw_events_landed match: bucket: your-raw-bucket prefix: events/ nodes: - name: ingest-events task: type: python script: ingest.py params: partition_date: ${execution_date} retry: max_attempts: 3 backoff_seconds: 60 - name: validate-schema dependencies: [ingest-events] task: type: python script: validate.py - name: transform-events dependencies: [validate-schema] task: type: spark class: com.yourorg.transforms.SessionizeEvents params: input_table: analytics.user_events output_table: analytics.user_sessions partition_date: ${execution_date} - name: dq-checks dependencies: [transform-events] task: type: trino query_file: dq_checks.sql fail_on: any_row_returned - name: refresh-daily-aggregate dependencies: [dq-checks] task: type: trino query: | INSERT INTO analytics.daily_user_metrics SELECT CAST(event_time AS DATE) AS event_date, event_type, COUNT(*) AS event_count, APPROX_DISTINCT(user_id) AS unique_users FROM analytics.user_events WHERE event_time >= DATE '${execution_date}' AND event_time < DATE '${execution_date}' + INTERVAL '1' DAY GROUP BY 1, 2 - name: emit-completion-signal dependencies: [refresh-daily-aggregate] task: type: signal emit: name: daily_metrics_ready params: date: ${execution_date} The last step, emitting a completion signal, makes pipelines composable. The downstream pipeline, such as the feature engineering task for ML, subscribes to the daily_metrics_ready topic and kicks off right away upon completion of this one without polling or any delay period.Ingestion Script Python # ingest.py import os import pyarrow as pa import pyarrow.parquet as pq from pyiceberg.catalog import load_catalog PARTITION_DATE = os.environ["partition_date"] catalog = load_catalog("my_catalog") table = catalog.load_table("analytics.user_events") raw_path = f"s3://your-raw-bucket/events/{PARTITION_DATE}/" arrow_table = pq.read_table(raw_path) # Schema enforcement before write — fail loudly on drift expected = table.schema().as_arrow() arrow_table = arrow_table.select(expected.names).cast(expected) table.append(arrow_table) print(f"Appended {arrow_table.num_rows} rows for {PARTITION_DATE}") The cast is intentional. Schema drift — upstream system silently adds or modifies a column – is one of the most frequent pipeline failures. Early detection through an error at ingestion is far less expensive than debugging further down the line. Step 4: Make Queries Cheap There are three main optimizations that account for the majority of savings. Each one is worth comprehending rather than blindly copying. Compaction: The Single Most Important Maintenance Activity Real-time or micro-batch ingestions result in lots of small files. The smaller files lead to larger metadata, inefficient query planning, and unnecessary storage of Parquet footers and row-group overheads. Compaction periodically merges them into files of the desired size (128 MB for our table definition above). With Spark: SQL -- Rewrite small files using bin-packing CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', options => map( 'min-input-files', '5', 'target-file-size-bytes', '134217728' ) ); -- Rewrite manifests so a query reads fewer manifest files CALL my_catalog.system.rewrite_manifests('analytics.user_events'); -- Expire old snapshots beyond the retention configured in TBLPROPERTIES CALL my_catalog.system.expire_snapshots( table => 'analytics.user_events', older_than => TIMESTAMP '2026-05-11 00:00:00', retain_last => 10 ); -- Remove orphan files (files in storage not referenced by any snapshot) CALL my_catalog.system.remove_orphan_files(table => 'analytics.user_events'); Schedule as part of a Maestro workflow that runs either daily or weekly. The remove_orphan_files command is particularly crucial — without this, any failures in writing will result in untracked files in S3, which you continue to pay for storing. Sorting Within Partitions for Skipping Efficiency If you know that your analysts always filter by event_type and user_id, sort your files so that Iceberg’s file-by-file statistics can skip entire files: SQL CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', strategy => 'sort', sort_order => 'event_type ASC, user_id ASC' ); For higher-dimensional access patterns, use Z-order: SQL CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', strategy => 'sort', sort_order => 'zorder(event_type, user_id, session_id)' ); Let Hidden Partitioning Do Its Job The query below requires no partition predicate — Iceberg derives the partition filter from event_time: SQL SELECT user_id, COUNT(*) AS event_count FROM analytics.user_events WHERE event_time >= TIMESTAMP '2026-05-17 00:00:00' AND event_time < TIMESTAMP '2026-05-18 00:00:00' AND event_type = 'purchase' GROUP BY user_id; In Hive, we would have to do AND year=2026 AND month=5 AND day=17 to enable pruning. In Iceberg, the transformation days(event_time) happen automatically, and the extra predicate event_type enables more pruning based on min/max statistics at the file level; files that don’t cover 'purchase' in their event_type range will not be opened. Step 5: Execute the Pipeline Execute the pipeline from the Maestro command-line interface: Shell # Trigger a manual run with parameters maestro start user-events-pipeline \ --param partition_date=2026-05-18 # Check workflow status and last N runs maestro status user-events-pipeline --last 10 # Inspect a specific run maestro instance describe user-events-pipeline <run_id> # Replay a failed run from a specific step maestro instance restart user-events-pipeline <run_id> \ --from-step transform-events Maestro exports metrics on queue depth, step latency, and failure rates via /metrics. Use this together with engine metrics (Spark UI, Trino query stats) to correlate any delays in orchestration with query performance. What Kind of Savings Should You Really Be Expecting? There is the old story about 90 percent savings when making such migrations that needs to be taken with a grain of salt. The real truth is highly dependent on your source. ScenarioRealistic savingsSource of savingsHive tables on S3 → Iceberg, same engine20–50% on query costsEliminated S3 listing, file pruning via stats, fewer small filesCron-scheduled batch → Maestro signalsVariable on compute, large on freshnessCompute drops only if jobs were over-running their window; freshness improves from hours to minutesProprietary warehouse → Iceberg + open engines40–80% on storage and licenseStorage decoupled from compute; engine competition on the same dataStreaming with no compaction → Iceberg + scheduled maintenance30–60% on query costsCompaction collapses small-file overhead The 90% number is realistic if the starting point is truly pathological, say a highly partitioned Hive table on S3 with no file size management that is being queried by a byte-scanned engine. Most organizations should budget for 30%-60% improvements and view anything higher as upside. Freshness improvements, by contrast, are reliably dramatic. Upgrading from a 4-hour cron job to an event-driven pipeline that fires within seconds of completion of its upstream is a structural win, not an incremental one. Comparing Maestro to Other Options Maestro is not the only option. The lay of the land as of 2026: Airflow has the broadest deployment and the most extensive provider ecosystem. Strengths: DAG construction; weaknesses: high-frequency triggering. Airflow's scheduler is traditionally been the bottleneck when operating at very high workflow volumes.Dagster has better data-aware abstractions (assets, partitions, software-defined assets) and integrates well with dbt and modern data tooling. The scale ceiling is lower than Maestro's.Prefect is native-Python and developer-friendly, offering good dynamic workflow capabilities. Still immature for very large scale.Temporal is the best general-purpose orchestrator for application workflows, less specialized for data pipelines.Maestro beats competitors on scale and on the signal/cyclic workflow paradigm. Cost factors: smaller community, steeper operational overhead, fewer out-of-the-box integrations. If you are already using Airflow and have fewer than a few thousand workflows per day, the migration costs to Maestro probably don't justify themselves through orchestration improvements alone — Iceberg adoption can be decoupled. However, if you are hitting Airflow scheduler limitations or have highly interdependent workflows across teams, Maestro's signal paradigm deserves a serious look. Common Mistakes Some recurring pitfalls in production: Deferment of catalog selection. Setting up Iceberg with a Hadoop or filesystem catalog "as a temporary solution" creates a future migration burden. Choose a REST catalog (Polaris, Nessie, Lakekeeper, or vendor-managed) from the start.No snapshot expiration policy. Snapshots persist indefinitely by default. High-volume tables generate gigabytes of metadata each month. Set expiration policies in table properties and run expire_snapshots periodically.No orphan file removal. Failing writes leave behind Parquet files not referenced by any snapshot. Remove orphan files weekly.Over-partitioning. Partitioning by the hour on a low-volume table results in more partitions than rows. Partition by the resolution of your query filters and target file sizes, not finer.Using signals as a free pass on idempotency. Workflow execution triggered by signals can be replayed or backfilled. Make every step idempotent — use MERGE INTO for writes, de-dupe on natural keys, and never make assumptions about "this only runs once."Skipping compaction. Streaming pipelines without compaction gradually degrade query performance until someone notices that the queries are 10x slower than at launch time. Conclusion Iceberg and Maestro solve two aspects of the same problem. Iceberg makes the data layer cheap to query by converting filesystem state into metadata state. Maestro makes the orchestration layer responsive by substituting signals for clocks. Adopting either technology creates tangible value, while adoption of both yields a pipeline that is inherently cheaper to operate and inherently fresher than a cron-based/Hive setup. If your current challenge is query cost or small file issues, start with Iceberg. If you are plagued with data staleness or unreliable scheduling, start with Maestro (or any other modern orchestrator). But eventually aim to adopt both if your goal is a data platform that scales without scaling your cloud bill. Where to learn more: Netflix Maestro: github.com/Netflix/maestroApache Iceberg: iceberg.apache.orgPyIceberg: py.iceberg.apache.orgApache Polaris (Iceberg REST catalog): polaris.apache.org

By Intiaz Shaik

WebSocket Debugging Without a Proxy — A Browser-First Workflow

WebSocket debugging is one of those things that sounds simple until you actually have to do it. The connection looks fine in DevTools, but messages are malformed, timing is off, or the server is behaving unexpectedly — and you have no easy way to inspect what's happening at the frame level without setting up a proxy or installing something heavy. Here's a practical workflow that requires nothing beyond a browser, illustrated with a real debugging scenario. The Problem With WebSocket Debugging HTTP requests are easy to inspect. DevTools shows you the full request and response, you can replay them with curl, mock them with interceptors, and diff payloads in seconds. WebSocket connections are different. Once the handshake completes, it's a persistent bidirectional channel, and most tooling treats frames as an afterthought. The Chrome DevTools WebSocket panel shows you raw frames, but it doesn't let you filter, transform, or replay them. You can see that a frame was sent with a 400-byte payload — but you can't easily extract it, modify it, and resend it to see how the server responds. The common workarounds all have friction: console.log on both sides – requires access to server code, adds noise, and still doesn't let you test edge cases without changing the clientCharles Proxy or mitmproxy – heavyweight, requires SSL certificate setup, and adds a network hop that can change timing behaviorCustom proxy server – takes time to build and maintain, and is overkill for a one-off debugging session None of these is fast when you just need to understand what's happening right now. A Real Scenario: Debugging a Real-Time Chat Feature To make this concrete, here's a situation that comes up often in practice. You're building a chat feature on top of a WebSocket backend. The UI looks fine in testing, but in production, some users report that messages occasionally appear out of order or that a specific type of system message causes the client to crash. You can't reproduce it reliably in your local environment, and you don't have direct access to the production server's logs. The questions you need to answer: What does the actual message payload look like when the crash happens?Is the issue in the message structure (missing field, unexpected type), or is it a timing problem (two messages arriving within milliseconds of each other)?How does the server respond if you send a deliberately malformed message? This is exactly the kind of debugging that browser-only tooling handles well — if you have the right tools. Step 1: Validate the Endpoint With an Online Tester Before anything else, confirm that the WebSocket endpoint is reachable and responding correctly. The tests.ws WebSocket tester is a browser-based tool that lets you connect to any ws:// or wss:// server, send arbitrary messages, and see server responses in real time. No install, no configuration, no account. For the chat scenario: connect directly to your production WebSocket endpoint, send a message that matches the format your client normally sends, and verify the server acknowledges it correctly. If this works as expected, the issue is likely in how the client processes incoming messages, not in the connection itself. The site also provides a free public echo server at wss://echo.tests.ws. Anything you send comes back immediately. This is useful for validating your client-side message serialization — connect to the echo server, send your payload, and confirm what comes back matches what you sent. If there's a mismatch, you've found a serialization bug before you even involve a real server. For the real-time testing step, the interface also shows frame-level details: message direction, payload size, timestamp, and raw content. This is enough to identify structural issues in isolation. Step 2: Intercept Live Traffic With the Chrome Extension Once you've validated the endpoint in isolation, the next step is observing what actually happens in your running application. The tests.ws Chrome extension adds a WebSocket proxy layer directly into Chrome DevTools, without modifying your application code or network configuration. Install the extension, open your application, and open DevTools. A new panel appears that logs every WebSocket frame — direction (sent/received), timestamp, payload size, and raw content — for all connections on the page simultaneously. Unlike the built-in DevTools WebSocket view, you can filter frames by content, copy payloads, and see a cleaner timeline. For the chat scenario, reproduce the conditions where messages go out of order. In the extension panel, you can see the exact sequence of frames with millisecond timestamps. If two messages are arriving 3ms apart and your client processes them synchronously, you'll see the problem immediately in the frame log — even if your application-level logging shows them in the wrong order. Step 3: Modify Outgoing Messages to Test Edge Cases This is where the extension's real value shows up. The extension lets you write JavaScript transform rules that intercept outgoing frames and modify them before they're transmitted to the server. For the crash scenario: you suspect the crash happens when a system message arrives with a missing userId field. Instead of waiting for it to happen in production, you write a transform rule: JavaScript if (message.type === 'system') { delete message.userId; } The extension applies this rule to matching outgoing frames. The server receives the malformed payload, you observe its response in the frame log, and you can immediately see whether it sends back an error, silently drops the message, or sends something that would cause the client to crash. This replaces a workflow that would otherwise require: modifying client code, building a new bundle, deploying to a test environment, and hoping you can reproduce the right conditions. With the extension, the iteration loop is: write a rule, trigger the action in the UI, observe the server response. No code changes, no deployment. Step 4: Test Protocol Edge Cases Beyond the immediate crash scenario, the transform approach is useful for systematic protocol testing: Missing required fields – remove fields one at a time to see which ones the server validatesType mismatches – send a string where the server expects an integer, or an array where it expects an objectOversized payloads – test the server's behavior when message size exceeds expected limitsRapid sequences – send the same message 10 times in quick succession to test for race conditions server-sideMalformed JSON – send a syntactically invalid payload to verify error handling Each of these can be tested in minutes, directly against a running server, without writing test harnesses or modifying application code. When This Approach Has Limits Browser-based WebSocket debugging works well for: Front-end debugging when you don't have server accessQA validation of message formats and server behaviorSecurity testing and input validation checksLearning how a third-party service's WebSocket protocol works It doesn't replace load testing tools. If you need to simulate 10,000 concurrent connections or measure throughput under sustained load, you need something like k6 or Artillery running outside the browser. Similarly, for server-side issues — memory leaks, connection pool exhaustion, handler bugs — you need server-side observability tools. But for the class of problems that are most common during development and integration — "why is the client behaving unexpectedly when it receives this specific message?" — the browser-only workflow gets you to an answer faster than any other approach. Summary The debugging workflow for the chat scenario above: Validate the endpoint – use the online WebSocket tester at tests.ws to confirm the server responds correctly to well-formed messagesObserve live traffic – install the Chrome extension, open the application, and capture the actual frame sequence that leads to the problemReproduce and test – write a transform rule that simulates the malformed message, trigger it in the UI, observe the server's response Total time to go from "users are reporting a crash" to "here's the exact server response that causes it": under 15 minutes, with no infrastructure changes, no deployments, and no server access required. WebSocket tooling has historically lagged behind HTTP tooling. The gap is smaller than it used to be.

By Dan Pan

Workflows vs AI Agents vs Multi-Agent Systems: A Practical Guide for Developers

When I first started building AI applications, I kept hearing the same words everywhere: workflows, agents, and multi-agent systems. At first, they all sounded like different labels for the same thing. After all, in every case, you are still calling an LLM, sending some context, and getting something back. That assumption turns out to be one of the easiest ways to design the wrong system. Once you start building real projects, the difference becomes very obvious. Some systems need strict control. Some need flexibility. Some need multiple specialized roles. If you choose the wrong model, you usually pay for it in cost, reliability, debugging pain, or unnecessary complexity. This is the explanation I wish I had when I started. I want to keep it beginner-friendly, but also useful enough that you can apply it in real projects without walking away with the usual “everything is an agent” confusion. Workflow vs Agent vs Multi-Agent System The simplest way to understand the whole topic is this: A workflow is when you decide the steps in advance. An agent is a model that decides what to do next. A multi-agent system is one in which multiple agents, usually with different roles, coordinate to solve a larger problem. That core distinction aligns closely with external references: workflows follow predefined code paths, while agents dynamically direct their own tool usage and execution flow. That sounds simple, but it becomes much clearer with a relatable example. Imagine you are ordering pizza. In a workflow, the restaurant follows a script. They ask for size, toppings, crust, and address in a fixed sequence. It is fast, reliable, and predictable. In an agent-style system, you might say, “I’m hungry, and I want something good for movie night,” and the system figures out whether you usually order vegetarian, whether you want something quick, whether it should ask a follow-up question, and what option best fits your past behavior. In a multi-agent setup, one specialist handles the order, another checks ingredient availability, and another optimizes delivery timing. Each one does a narrower job, but together they solve a broader problem. That is the real difference. The question is not whether all three use AI. The question is who is controlling the process. What a Workflow Really Is A workflow is the most structured option. You define the steps, the order, and often the failure points. The model may still do useful work inside the system, but the system itself is not making open-ended decisions about how to proceed. Think of it like a recipe. Step one happens first. Step two happens second. If something goes wrong, you usually know where it happened. A simple example is a blog post generator that deliberately separates outline generation, introduction writing, body drafting, and final assembly. TypeScript import Anthropic from '@anthropic-ai/sdk'; const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); async function generateBlogPost(topic: string) { const outlineResponse = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 1024, messages: [ { role: 'user', content: `Create a blog post outline about: ${topic}` } ] }); const outline = outlineResponse.content[0].text; console.log('Step 1: Outline created'); const introResponse = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 1024, messages: [ { role: 'user', content: `Based on this outline, write an introduction:\n\n${outline}` } ] }); const intro = introResponse.content[0].text; console.log('Step 2: Introduction written'); const bodyResponse = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 2048, messages: [ { role: 'user', content: `Based on this outline, write the body:\n\n${outline}` } ] }); const body = bodyResponse.content[0].text; console.log('Step 3: Body written'); return `${intro}\n\n${body}`; } The reason workflows dominate production is not that teams lack ambition. It is that predefined orchestration is easier to reason about. Predictable systems are easier to test, monitor, certify, and price. That is exactly why guidance around production AI systems keeps steering builders toward workflows first, especially for reliability-critical environments. The referenced material also repeatedly points out that workflows are the better fit when requirements are stable, boundaries are clear, and reliability matters more than open-ended autonomy. That makes workflows a very strong fit for document processing, onboarding, report generation, fixed moderation pipelines, approval chains, and regulated systems. What an Agent Really Is An agent changes one important thing. Instead of hardcoding the order of operations, you give the model a goal, a set of tools, and enough context to decide what should happen next. That is where the flexibility comes from. The model can inspect the task, choose a tool, look at the result, decide whether another tool is needed, and continue until it reaches a stopping point. That pattern is what makes an agent feel more like a smart assistant than a pipeline. The external guides describe this clearly as dynamic decision-making, autonomous tool selection, reasoning, and self-directed task execution. A simple research assistant is a good example for beginners. TypeScript import Anthropic from '@anthropic-ai/sdk'; const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); const tools = [ { name: 'search_web', description: 'Search the web for information about a topic', input_schema: { type: 'object', properties: { query: { type: 'string' } }, required: ['query'] } }, { name: 'save_notes', description: 'Save research notes to a file', input_schema: { type: 'object', properties: { notes: { type: 'string' } }, required: ['notes'] } } ]; async function searchWeb(query: string): Promise<string> { return `Results for ${query}`; } async function saveNotes(notes: string): Promise<void> { console.log(`Saved notes: ${notes.slice(0, 80)}...`); } async function researchAgent(topic: string) { const messages: any[] = [ { role: 'user', content: `Research ${topic} and save comprehensive notes.` } ]; let done = false; while (!done) { const response = await client.messages.create({ model: 'claude-3-5-sonnet-20241022', max_tokens: 4096, tools, messages }); if (response.stop_reason === 'tool_use') { const toolUse = response.content.find( (block: any) => block.type === 'tool_use' ); if (toolUse.name === 'search_web') { const results = await searchWeb(toolUse.input.query); messages.push({ role: 'assistant', content: response.content }); messages.push({ role: 'user', content: [ { type: 'tool_result', tool_use_id: toolUse.id, content: results } ] }); } if (toolUse.name === 'save_notes') { await saveNotes(toolUse.input.notes); done = true; } } else { done = true; } } } What matters here is not the SDK syntax. What matters is that you did not hardcode “search first, summarize second, save last.” The agent decides that. It may search once. It may search five times. It may decide it has enough information early. That is precisely why agents are useful for research, support, exploratory planning, and other tasks where you cannot fully predict the required path ahead of time. The trade-off is that you lose some of the certainty that workflows give you. The number of tool calls can vary. The runtime can vary. The cost can vary. If something behaves strangely, you often need stronger logs and better observability to understand why. Seeing the Difference Side by Side One of the best parts of your attached draft was the side-by-side review analysis example, because it shows the difference without abstract theory. That absolutely deserves to stay. Suppose the task is to analyze a customer review and generate a response. The workflow version might look like this. TypeScript async function analyzeReviewWorkflow(review: string) { const sentiment = await callLLM( `Analyze sentiment of this review as positive, negative, or neutral: ${review}` ); const topics = await callLLM( `Extract the main topics from this review: ${review}` ); const response = await callLLM( `Generate a customer support response for a ${sentiment} review about ${topics}` ); return { sentiment, topics, response }; } This is clean and efficient. It makes the same three calls every time. The cost is predictable. The behavior is stable. It is also rigid. A weird review gets handled through the same path as a normal one. Now compare that with an agent version. TypeScript async function analyzeReviewAgent(review: string) { return await runAgent({ task: `Analyze this review and generate a support response: ${review}`, tools: [ 'check_sentiment', 'extract_topics', 'search_knowledge_base', 'generate_response' ] }); } Now the system can decide whether a highly emotional complaint requires a knowledge base lookup before responding, while a simple positive review may only require sentiment classification and a thank-you response. That flexibility is exactly what makes agents attractive. It is also what makes them less predictable. This is one of the most important beginner lessons in the whole topic. A workflow handles every case with the same planned path. An agent adapts its path to the case. When Workflows Are the Better Choice This is where most of the production reality sits. If you know the exact steps, a workflow is almost always the first thing you should build. If predictability matters, a workflow is usually safer. If cost matters, workflows are easier to manage because you know roughly how many model calls happen per run. For debugging, workflows are easier because every state transition is explicit. That is also why modern workflow-oriented systems emphasize type safety, checkpointing, durable execution, human-approval steps, and clear routing. Those capabilities are not flashy, but they are exactly what real teams need when a system runs in production for weeks or months. A customer onboarding pipeline is a simple example. TypeScript async function onboardCustomer(email: string) { await sendWelcomeEmail(email); await createAccount(email); await setupDefaultPreferences(email); await sendTutorial(email); } A document processing pipeline is another. TypeScript async function processDocument(pdfPath: string) { const text = await extractText(pdfPath); const summary = await summarize(text); const keywords = await extractKeywords(text); await saveToDatabase({ text, summary, keywords }); await notifyUser(); } A content moderation flow is another good fit. TypeScript async function moderatePost(post: string) { const isSpam = await checkSpam(post); const isToxic = await checkToxicity(post); return isSpam || isToxic ? 'reject' : 'approve'; } None of these tasks benefits much from letting the model invent the control flow on the fly. They benefit from clean orchestration. When Agents Are the Better Choice Agents make more sense when the task is open-ended, when the path cannot be fully predefined, or when adaptability matters more than deterministic execution. Customer support is a classic example because every issue arrives in a different way. Research is another reason because you do not know in advance which leads will be useful. Trip planning is another challenge because different users, constraints, budgets, dates, and preferences change the best route through the task. A travel helper captures this nicely. TypeScript async function travelAgent(request: string) { return await runAgent({ task: `Help the user with this travel request: ${request}`, tools: [ 'search_flights', 'search_hotels', 'get_weather', 'suggest_itinerary', 'ask_followup_question' ] }); } The system may begin by asking a clarifying question. It may check the weather before hotels. It may avoid hotel search entirely if the user says they are staying with friends. This is exactly the sort of context-dependent behavior that agents are designed for. The guides also specifically call out use cases like deep research, agentic RAG, customer support, virtual assistants, and coding assistants as agent-friendly territory. What Multi-Agent Systems Add Multi-agent systems take the idea one step further. Instead of having one agent handle everything, you split the work among multiple specialists. This matters when specialization actually improves the result. One agent might research. Another might write. Another might review or validate. The Inkeep article makes an important distinction: true multi-agent systems are not just a sequential workflow with different names for each step. The key idea is autonomous coordination between specialized agents, often through direct communication or delegated responsibilities. A simple content team example makes this concrete. TypeScript async function researchAgent(topic: string) { return callLLM(`Research ${topic}. Return key facts, trends, and context.`); } async function writerAgent(research: string, topic: string) { return callLLM(`Using this research, write an article about ${topic}:\n${research}`); } async function editorAgent(article: string) { return callLLM(`Edit this article for clarity, accuracy, and flow:\n${article}`); } async function contentCreationTeam(topic: string) { const research = await researchAgent(topic); const draft = await writerAgent(research, topic); const final = await editorAgent(draft); return final; } This is still a simple coordinator-led version, but it shows the value of specialization. A more advanced system might allow the editor to request a revision from the writer, or the writer to request more supporting evidence from the researcher. That is where multi-agent systems start to feel like collaborative problem-solving rather than a chain of prompts. The caution here is important. Multi-agent systems are not “the next level” you should jump to just because they sound advanced. They introduce more moving parts, more coordination overhead, more debugging complexity, and higher cost. They are useful when the problem actually needs multiple kinds of expertise, not when you are just trying to make a simple app look more impressive. The Practical Decision Model A good beginner question is not “which one is the smartest?” It is “how much uncertainty does this task have, and who should own the decision-making?” If the task is well-defined and stable, start with a workflow. If the task is open-ended and the system needs to choose how to proceed, consider an agent. If the task genuinely benefits from multiple specialists with separate responsibilities, consider multiple agents. That decision model lines up closely with the source material as well. Use workflows when requirements are clear, control is important, cost matters, and debugging stays simple. Use agents when tasks are exploratory, human-like reasoning is valuable, and adaptability matters more than fixed control flow. Use multi-agent systems when a single reasoning unit is no longer sufficient to capture the problem's diversity. The Beginner Mistakes That Cost Time and Money The first mistake is using agents for simple tasks that should be handled by normal code or a fixed workflow. If you want to add two numbers, do not build an agent. If you want to categorize simple support tickets with a stable schema, start with a workflow. Not every AI problem needs autonomy. TypeScript function addNumbers(a: number, b: number) { return a + b; } The second mistake is forcing a workflow onto a task that clearly needs adaptation. Creative writing, research, and support escalation often branch in ways that are hard to encode cleanly in advance. If you keep adding if-statements and exception paths to rescue a rigid workflow, that is often a sign the task wants agent behavior. The third mistake is building multi-agent systems too early. Three agents for a simple email writer is usually just an expensive ceremony. You should earn that complexity by hitting a real need first. These mistakes sound obvious when written down, but they are very common because the AI space rewards novelty in demos more than maintainability in products. The Cost Conversation Matters More Than People Admit A workflow-based newsletter creator might always make three model calls, one for the intro, one for the main copy, and one for the closing section. That means the cost per run is fairly easy to estimate. TypeScript async function createNewsletter(topics: string[]) { const intro = await generateIntro(topics); const articles = await generateArticles(topics); const outro = await generateOutro(); return { intro, articles, outro }; } An agent-based newsletter creator might decide it needs extra research, then rewrite one section twice, then call another tool to validate tone. Sometimes that flexibility is useful, but it also means cost and latency can move around more than you expect. TypeScript async function newsletterAgent(topics: string[]) { return runAgent({ task: `Create a newsletter about these topics: ${topics.join(', ')}`, tools: ['research_topic', 'draft_section', 'revise_section', 'validate_tone'] }); } That does not automatically make agents bad. It just means the operational model is different. The broader production guidance on workflows versus agents keeps coming back to exactly this point: deterministic systems are easier to budget for, observe, and control. The Hybrid Model Is Usually the Best Answer This is probably the most useful real-world takeaway in the entire topic. You do not have to choose one pattern forever. Many successful systems use workflows to structure the outer system and agents only where flexibility is genuinely needed. The Prompt Engineering Guide explicitly recommends hybrid approaches, such as using workflows for structure and agents for open-ended subtasks. That pattern looks like this. TypeScript async function smartCustomerSupport(message: string) { const category = await categorize(message); if (category === 'simple_faq') { return faqWorkflow(message); } if (category === 'complex_issue') { return supportAgent(message); } return escalateToHuman(message); } This is a very practical architecture. The workflow gives you control, routing, and predictability. The agent only appears where variability is too high for rigid orchestration. That means you keep the system understandable while still benefiting from adaptive behavior. If you are building beginner-to-intermediate AI products, this is one of the best mental models to adopt early. A Cleaner Way to Think About Real Projects A document processor usually wants a workflow because the same stages repeat every time. A support assistant may want an agent because issues differ, and tool selection depends on context. A software delivery assistant might eventually become a multi-agent system if planning, implementation, testing, and review are separate responsibilities that benefit from specialization. Here is a simplified example of that last case. TypeScript async function developFeature(requirement: string) { const specs = await productManagerAgent(requirement); const code = await developerAgent(specs); const testResults = await qaAgent(code); if (!testResults.passed) { return developerAgent(`Fix these issues:\n${testResults.issues}`); } return code; } This kind of setup can make sense, but only if the complexity is real. It should come from the nature of the work, not from the desire to use more agents. Conclusion If you are just starting, build a workflow first. That advice is not anti-agent. It is pro-clarity. Workflows teach you how to decompose tasks, define boundaries, measure outcomes, and understand where AI actually adds value. Once you understand the stable parts of your system, it becomes much easier to identify the unstable parts that may benefit from an agent. Once you understand where one agent becomes overloaded, it becomes much easier to justify multiple specialized agents. That progression is healthier than starting with maximum autonomy and then trying to reverse-engineer stability later. So my practical rule is simple. If the task can be described as a sequence of reliable steps, use a workflow. If the system needs to decide the steps as it goes, use an agent. If the problem truly needs multiple specialized minds working together, then and only then reach for a multi-agent design. The best AI systems are not the ones with the most autonomy. They are the ones that stay understandable when something goes wrong.

By Raju Dandigam

A Deep Dive into Tracing Agentic Workflows (Part 2)

Part 1 dived into what to trace in an agentic system and why. How the traditional tracing and metrics, such as latency, scale, cost, uptime, and throughput, need to be redefined. And how to define the new metrics that are at the core of an agentics system, such as response quality, accuracy, and task completion. This part is about the mechanics: how a trace is structured, how context propagates across agent boundaries, and how to make sense of it all. And then, how to automate it all with an observability agent. Anatomy of a Trace Every event in the system that's triggered when processing a member's request — from the initial user request to receiving a response back (and any subsequent tracking and orchestration) — belongs to a single trace. The structure of a trace is hierarchical, built from three key identifiers. These three fields travel in the headers of every internal request and response - traceId: Generated once per request, traceId persists throughout and connects all the operations in the entire execution together.spanId: Generated for each "unit of work", e.g., agent invocation, LLM call, memory persistence, response chunk streaming. SpanId is what uniquely identifies an operation in the trace.parentSpanId: When one operation triggers another, the child span stores the parent’s spanId as its parentSpanId. ParentSpanId is what helps define the relationship between operations, which can then be used to create the execution tree. This is how these IDs get logged in a trace: XML traceId: 8f3a... [A1] Orchestrator Plan traceId = 8f3a... spanId = A1 parentSpanId = null //root span [B1] Fetch Profiles traceId = 8f3a... spanId = B1 parentSpanId = A1 [C1] Profile Ranking traceId = 8f3a... spanId = C1 parentSpanId = B1 [D1] LLM Synthesis traceId = 8f3a... spanId = D1 parentSpanId = A1 Every service, agent, and skill that receives a request reads these headers, generates a new spanId for its own work, and passes its spanId as the parentSpanId in any downstream calls it makes. The result is a complete, traversable execution tree where every node knows its parent. Along with these three fields, a trace should include all important artifacts that need to be tracked for a particular span, such as timestamp, token usage, API, or model version, etc. When collated into a DAG as explained below, whatever can be collected in a span can be optimized in turn. And that is pretty much how tracing works. From a Single Trace to a System DAG A single trace captures one execution. That alone enables replay — debugging and validating system behavior on a per-request basis. For example, it can help confirm whether the intended tool was triggered, whether graceful degradations kicked in as expected, whether the system took the intended path, or where the workflow faltered to produce an inaccurate or low-quality response. Many traces sampled together — across users, queries, surfaces, and over time — produce something much more useful: an empirical Directed Acyclic Graph (DAG) of how the system is actually running in production. This aggregated graph shows what is happening, and can be sliced by invocation frequency, timestamp, and cost. A few critical things become visible only at this level: Critical Paths A critical path is a sequence of operations that consistently dominates the end-to-end latency of a system. Tracing surfaces insights such as - which spans sit on the critical path versus which run in parallel without affecting wall-clock time, which calls block downstream work, and where serialization could be replaced with concurrent fan-out. This is where latency optimization actually pays off. Speeding up off-critical-path operations changes nothing for the user. Resource and Cost-Intensive Workflows In agentic systems, token consumption is the dominant cost driver, and it can be orchestrated to show up in aggregated traces. Tracing makes spending attributable by agent, by query type, by model variant, by use case, or by workflow. Common patterns it exposes: An agent quietly pulls oversized context on every invocation.A frontier model used for trivial classification or routing where a smaller, cheaper one would suffice.Retry behavior or speculative calls that multiply token cost without measurably improving response quality.Tools whose outputs are fetched but never end up in the final prompt. System Inefficiencies Aggregated tracing surfaces some common pitfalls in the agentic workflows Dead branches – skills or agents that exist on paper but rarely fire, or only fire on one narrow query type, signaling candidates for removal or consolidation.Outdated paths – code paths intended to be deprecated that still receive traffic from a handful of callers, blocking cleanup.Unexpected fan-out – an orchestrator that, under certain prompts, triggers far more downstream calls than its design suggests, inflating both latency and cost.Loops and near-loops – repeated transitions between the same agents that indicate stalled reasoning rather than forward progress, often invisible until you look at transition counts across many traces. And to reiterate, it can be measured, it can be improved. The system DAG helps highlight specific, addressable engineering gaps — backed by data rather than intuition. Tracing Overhead Tracing doesn’t come free. Naive instrumentation adds real latency and ships sensitive payloads through the observability stack. A few defaults worth setting early to minimize these overheads: Sample judiciously in production so as not to oversample without losing critical signals. For example, depending on the traffic, one can choose to sample 100% of traces with errors and slow response time, and 5% of the rest.Be wary of what’s actually being logged to avoid misuse — redact PII at the span boundary rather than at query time,Emit spans asynchronously, as a blocking exporter on the request path defeats the purpose. Closing the Loop: Agents Observing Agents The structural primitives that render traces queryable for humans similarly make them consumable by machine intelligence. When spans are emitted in a consistent, structured format, an observer agent can ingest the stream to compute rolling metrics and detect drifts, such as rising p95 on a specific tool, token consumption creeping up on an untouched agent, or transition counts that signal an emerging loop. This observer doesn't have to stop at detection and alerting. A higher-order agent can subscribe to these signals to automate corrective actions: replacing a failing node, shifting traffic to faster model variants, tightening retry limits, disabling misbehaving skills, or falling back to cached responses. By attaching offending trace IDs to an auto-generated ticket, the trace becomes the input for the system's next decision: detect → diagnose → mitigate, automated end-to-end. The tractability of this approach lies in the contract between layers: traceId, spanId, parentSpanId, and a payload. Observer agents require no bespoke integrations or complex plumbing; they read the same spans humans monitor in dashboards. Ultimately, the agentic system observes and corrects itself using the very substrate it produces. Conclusion Done well, the same trace stream that engineers query for debugging becomes the input layer for observer agents that monitor, diagnose, and remediate the system in flight, turning observability from a human dashboard into a closed loop that the system runs on itself.

By VIVEK KATARYA

Orchestrating Zero-Downtime Deployments With Temporal

Zero-downtime deployment is often described as a rollout strategy, but in production, it is more accurately a coordination problem. Traffic must remain on healthy instances while new ones warm up, controllers must wait for readiness before shifting load, and promotion must stop cleanly when metrics degrade. Kubernetes rolling updates already replace Pods incrementally and wait for new instances to start before removing old ones, while readiness probes determine when a Pod should receive traffic. Progressive delivery systems such as Argo Rollouts add weighted traffic shifts, pauses, and analysis gates. The difficult part is not the individual primitive, but the stateful control flow around all of them when retries, human approvals, controller restarts, and rollback decisions intersect. Stateful Release Logic Temporal fits this problem because a Workflow Execution is a durable, reliable, and scalable function execution that persists state and resumes from the latest recorded event after failure. A workflow can wait on timers, external messages, or child workflows without turning those waits into a fragile in-memory state. Temporal also persists durable timers, so a canary soak period or a maintenance window survives worker restarts and infrastructure interruptions instead of being tied to the lifetime of a CI runner or a shell script. That property changes the nature of deployment logic. Instead of treating a release as a short-lived pipeline job, the release can be modeled as a long-running control loop with explicit state such as requested version, current traffic weight, observed health, approval status, and rollback reason. Temporal also guarantees that at most one open Workflow Execution can exist for a given Workflow ID, which makes a fixed ID such as payments-prod a practical concurrency control mechanism for serializing production rollouts and preventing overlapping deploys to the same environment. A Long-Lived Environment Workflow A particularly effective pattern is a long-lived environment workflow that receives release requests by Signal, exposes current status by Query, and periodically uses Continue-As-New to keep its event history fresh. Temporal message handlers operate on workflow state, Signals can be sent from clients or other workflows, and Continue-As-New starts a fresh run in the same chain with the same Workflow ID when history grows. That combination turns a deployment lane into a durable queue and a durable mutex at the same time. If the lane is not already running, Signal-With-Start can start it and enqueue the first release in a single atomic client call. Java @WorkflowInterface public interface EnvironmentDeploymentWorkflow { @WorkflowMethod void run(String service, String environment); @SignalMethod void enqueue(ReleaseCandidate release); @SignalMethod void approve(String releaseId); @QueryMethod DeploymentView current(); } private final Deque<ReleaseCandidate> queue = new ArrayDeque<>(); private boolean approved; @Override public void run(String service, String environment) { while (true) { Workflow.await(() -> !queue.isEmpty()); ReleaseCandidate release = queue.removeFirst(); approved = false; deployRelease(release); if (Workflow.getInfo().isContinueAsNewSuggested()) { Workflow.continueAsNew(service, environment); } } } This pattern keeps rollout ownership inside the workflow rather than in an external scheduler. Approval is a state transition, not a webhook race. Waiting is explicit through Workflow.await, not an ad hoc sleep in a pipeline stage. The workflow can remain open for months, continue across runs when suggested, and still preserve a single logical identity for the service and environment being managed. Activities Encode the Real Work The workflow should not talk directly to Kubernetes, Argo Rollouts, load balancers, or telemetry backends. Temporal workflow code must remain deterministic, and direct I/O belongs in Activities. Activity executions can be retried with explicit retry options, and Temporal recommends designing activities to be idempotent because they may be retried if failures happen before completion is recorded. That requirement has an immediate impact on deployment APIs: methods such as setCanaryWeight(10) or applyManifest(version) are far safer than imperative operations such as increaseTrafficBy(10) or deployAgain(), because retries converge on a desired state instead of amplifying side effects. Java private final RolloutActivities rollout = Workflow.newActivityStub( RolloutActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(5)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(2)) .setMaximumAttempts(5) .build()) .build()); private void deployRelease(ReleaseCandidate release) { rollout.applyManifest(release.service(), release.version()); rollout.waitForAvailable(release.service(), release.version()); rollout.setCanaryWeight(release.service(), 10); Workflow.sleep(Duration.ofMinutes(5)); HealthSnapshot health = rollout.measureHealth(release.service(), release.version()); if (health.errorRate() > 0.01 || health.p95LatencyMs() > 250) { rollout.rollback(release.service(), release.previousVersion()); return; } Workflow.await(() -> approved); rollout.setCanaryWeight(release.service(), 100); rollout.waitForStable(release.service(), release.version()); } The snippet is intentionally narrow: the workflow owns orchestration, while the activity layer owns interaction with external systems. waitForAvailable usually maps to deployment status checks and readiness conditions. In Kubernetes, readiness probes determine when a Pod is ready to accept traffic, Pods that are not Ready are removed from Service endpoints, and a stalled rollout surfaces through progress conditions such as ProgressDeadlineExceeded. If Argo Rollouts is the execution layer, the activity boundary often maps cleanly to its setWeight, pause, and inline analysis steps. One additional design constraint matters here: activity inputs and results are recorded in workflow history, so deployment activities should return compact state, such as health verdicts or revision identifiers, rather than whole manifests or large telemetry payloads. Parallel Waves Without Fragile Fan-Out Many deployments are not single-cluster events. Regional waves, cluster cohorts, and dependency checks often need to run in parallel but still report into one release decision. Temporal child workflows are a natural fit because they are started from a parent workflow, they have their own histories, and they can be invoked asynchronously. This keeps failure domains separate and prevents one large release workflow from becoming an unbounded event log. Java RegionDeploymentWorkflow east = Workflow.newChildWorkflowStub( RegionDeploymentWorkflow.class, ChildWorkflowOptions.newBuilder() .setWorkflowId("payments-prod-" + release.version() + "-us-east") .build()); RegionDeploymentWorkflow west = Workflow.newChildWorkflowStub( RegionDeploymentWorkflow.class, ChildWorkflowOptions.newBuilder() .setWorkflowId("payments-prod-" + release.version() + "-eu-west") .build()); Promise<Void> p1 = Async.procedure(east::deploy, release); Promise<Void> p2 = Async.procedure(west::deploy, release); Promise.allOf(p1, p2).get(); Abort handling also becomes more disciplined in this model. Temporal distinguishes cancel from terminate, and cancel is usually the safer operator action because the workflow receives a cancellation request and can still execute cleanup logic, such as traffic restoration or stable version re-pinning. Terminate stops execution immediately and gives the workflow no chance to run rollback code, which makes it the right tool only for genuinely stuck executions. For deployment orchestration, graceful cancellation aligns with operational reality because rollback is part of the business logic, not an afterthought. The Deployer Must Remain Deployable There is a second deployment problem hidden inside the first one: release workflows often stay open while Temporal workers themselves are being upgraded. Temporal addresses that are directly related to workflow versioning. In the Java SDK, Patching allows a workflow definition to branch safely so that existing executions remain compatible, while newer executions use updated logic. Temporal’s production guidance now recommends Worker Versioning as the default approach for most teams, because worker deployments can be tagged into versions so that old workers continue running old code paths and new workers take new paths, enabling gradual traffic ramps and fast rollback for workflow code itself. Java int v = Workflow.getVersion("post-canary-health-v2", Workflow.DEFAULT_VERSION, 1); boolean accepted = v == Workflow.DEFAULT_VERSION ? health.errorRate() < 0.02 : health.errorRate() < 0.01 && health.p95LatencyMs() < 250; That capability matters because deployment orchestration is rarely static. Health thresholds change, additional gates appear, and new regions get introduced. Without safe workflow versioning, the deployment controller eventually becomes the source of deployment risk. Temporal’s own pre-production guidance is aligned with that concern: deliberately killing all workers and restarting them validates at-least-once semantics, idempotent activities, and clean replay behavior. A zero-downtime deployer should therefore be tested under the same failure patterns it is supposed to absorb on behalf of the application being released. Conclusion Zero-downtime deployment is not achieved by replacing Pods slowly or by adding a canary percentage alone. It is achieved when the full release process can survive restarts, wait safely for readiness and analysis, accept approvals without race conditions, and roll back deterministically when health degrades. Kubernetes and progressive delivery controllers provide the runtime primitives for availability, but Temporal provides the durable control plane that turns those primitives into a reliable deployment application. With stable workflow identities, idempotent activities, durable timers, child workflows for regional waves, and safe versioning for the orchestrator itself, deployment logic stops behaving like a fragile CI episode and starts behaving like production software.

By Akhil Madineni

CORE

Observability for Agents and Workflows: Tracing Prompts, Tool Calls, and Business Outcomes End-to-End

AI agents have come a long way. They aren’t just answering simple questions, but they’re handling order checks, summarizing support tickets, updating records, routing incidents, approving requests, and even calling internal tools. As these agents slip deeper into real business workflows, just peeking at model logs isn’t enough. Teams need to see everything: what the agent did, why it did it, which systems it poked, and whether the end result actually helped the business. Agent Observability That’s where agent observability comes in. Traditional observability lets teams watch over their apps, APIs, databases, and infrastructure. Agent observability goes a step further. It shines a light on the whole AI workflow: it connects the dots from the user’s request to the agent’s decisions, the tools it touches, the systems it interacts with, and all the way to the final outcome. Let’s see a customer support example. Say a customer messages, “My subscription renewal failed, but I got charged twice.” A human rep checks the account, payment history, billing rules, refund policy, and ticket history before answering. Now, an AI agent might do that job automatically. It’ll spot the billing problem, look up the customer record, call the billing system, check for duplicate payments, and either resolve the issue or escalate it if things get too messy. On the surface, this whole thing just looks like a simple chat. However, under the hood, it’s a full-on workflow. If you want good observability, you need that behind-the-scenes view: Why bother? Because the final response doesn’t tell you the whole story. If the customer comes back unhappy, you need to nail down whether the agent checked the right account, used the right billing tool, hit an error, misread the request, or escalated when it couldn’t help. Don’t just watch the answer: Follow the whole journey When you break down agent interactions, a few basic layers show the full picture. First, track the user request. What did the user ask? Was it urgent, fuzzy, sensitive, or bound to a customer contract? Second, watch the agent’s action. Did it answer straight away, ask a follow-up question, search a knowledge base, use a tool, or hand off to a human? Third, note the context. What sort of information did it use? Did it pull a help article, customer details, invoice, ticket, policy, or product data? Fourth, log tool usage. Did the agent call billing APIs, CRM systems, databases, incident tools, or an approval workflow? Did those calls work, or did they fail? Lastly, look at the result. Did the agent fix the customer’s problem? Was the ticket reopened? Did a human have to clean up after the agent? Without these layers, you’ll know when something was slow or incorrect, but not why. Maybe the context was off, a tool call failed, it lacked permissions, the prompt changed, or something further downstream broke. Use a Single ID to Track Everything One of the easiest fixes is to tag the whole workflow with a tracking ID. Let that ID travel with the request, from the interface through the agent, tools, APIs, and your business systems. Now, if a support ticket gets botched, the team can retrace every step: what the customer asked, what the agent understood, which account it checked, what the billing system said back, and why the agent chose to close or escalate. It’s not just for support. Maybe your SRE team uses an AI agent to help dig into a production alert. The agent scans logs, checks recent deployments, reviews database metrics, and suggests the likely cause. That same tracking ID means you’ll know exactly which systems the agent checked and whether it missed anything crucial. Don’t ignore tool calls; they’re real actions Here’s where things get serious. When an agent calls a tool, it’s taking action. Looking up customers, updating records, approving requests, creating tickets, and kicking off workflows need to be watched closely. For each tool call, capture details like tool name, how long it took, success or failure, retries, permission results, error messages, and what actually happened. Take a finance workflow. Say the agent reviews vendor invoices by extracting details, matching with a purchase order, checking taxes, and routing exceptions to finance. If an invoice gets approved by mistake, did the agent misread the invoice? Match it with the wrong purchase order? Miss a policy update? Or did the finance system return incomplete info? That’s why tracking tool calls is critical. A wrong answer in chat is one thing, but a wrong move in your business system can lead to trouble such as money lost, operations disrupted, and even compliance issues. Understand Agent Decisions, But Protect Privacy Teams need to understand what the agent did, but you don’t want to log every single “thought” it had; it’s just unnecessary noise. Instead, record decision details in a structured way. Example: Intent: billing disputeConfidence: mediumTool: billing lookupReason: account verification neededPolicy result: escalateFinal action: handoff to human Now you have enough to debug the workflow and for reporting, without exposing raw thought streams. You can spot how often agents escalate from low confidence, where tools fail, or if policy rules stop an action. Connect Observability to Business Outcomes Don’t just track the tech stuff; what really matters is whether the agent gets the job done. Watch business metrics like: Resolution timeEscalation rateWorkflow completion rateTool failuresCost per workflowSLA hits or missesReworkHow often humans step in If you’ve got an e-commerce agent helping buyers pick products, check inventory, apply discounts, and guide checkout, you want to know: did the customer actually buy the item? If checkout drops after you tweak a prompt, find out why. Did the agent push out-of-stock items? Apply discounts wrong? Use the wrong tool? Lose customers with confusing answers? Observability at this level helps both engineering and business teams get answers, fast. Build Dashboards for Different Audiences Everyone’s got different needs. SREs care about latency, failed tools, retries, issues with dependencies, and expensive cost spikes. Security teams focus on policy denials, suspicious tool actions, sensitive data flags, or prompt injection attempts. Product owners want completion rates, escalations, customer satisfaction, and abandoned workflows. Engineers need to see how agent behavior shifts after you change the model, prompt, workflow, or deployment. Business folks need throughput, SLAs, cost savings, and improvements to customer experience. Take security operations. Say an agent checks suspicious logins, identity logs, privilege changes, and endpoint activity. Security needs to know: did the agent just review info, or did it try to lock an account? If it got blocked, you want that visible, too. Alert on AI-Specific Failures AI agents fail in new ways. Teams need alerts for things like sudden spikes in tool denials, fallback responses, unexpected tool usage, cost blowups, prompt injection attempts, completion drops, or escalating cases. If an agent suddenly goes wild with refund actions, it could mean a prompt is off, a policy is weak, or something’s getting abused. If fallback responses shoot up, maybe the knowledge base is broken. Costs spike? Maybe the agent is stuck looping, retrying, or making unnecessary expensive calls. Tie alerts to deployments, too. Agents change behavior after you update a prompt, switch models, change schema, adjust policies, or edit a workflow. Teams should compare how the agent behaved before and after. A Simple Way to Grow Observability Observability matures in steps. Basic logs: prompts, responses, errors, timestampsTool visibility: what got used, if it worked, how long it tookEnd-to-end traces: follow the user request through the agent, tools, APIs, systemsBusiness-level result tracking: resolution, escalation, completion, rework, cost, SLAAutomated alerts: regressions after updates, anomalies, unusual patterns Observability is more about making sense of the whole workflow and visibility. Teams need to know what users wanted, what the agent decided, which info it used, which tools it grabbed, which systems it touched, and whether business value was delivered. As AI agents settle into production, observability has to cover more than just servers and app logs. The teams that win will be the ones who trace agent behavior end to end, spot failures early, explain what happened, and keep improving safely.

By Srinivas Chippagiri

CORE

Team Management

DZone's Featured Team Management Resources

Monthly Top Team Management Experts

The Latest Team Management Topics