Team Management

For application users, waiting for an upload to finish entering a web portal is familiar enough. Waiting again while the application analyzes that upload, however, can leave a user wondering whether anything is happening at all. The application can reassure the user by confirming the upload was received before analysis finishes, but it still needs a way to prevent the uploaded content from being published or used downstream until it's been evaluated against content policies and approved. We can accomplish this for a user-facing upload application by simply separating the submission process from the evaluation process. That allows the application to save the upload in private staging and queue it for background analysis before returning information to the frontend. With this approach, the uploaded content remains unavailable to downstream workflows until after the application records a content approval decision. Let's consider, for example, an editorial portal where photographers can submit their images for some publication. By rule, the publication categorically rejects AI-generated content, so it keeps each upload in private staging while its AI-image detector evaluates photo content in the background. When images meet the portal's policy, they can proceed to the publication workflow; when they don't, they must remain on hold for an editor to carefully review. In this article, we’ll explore building this workflow in C# around an actual AI-generated content detection service, and we'll use the above "editorial portal" concept as a concrete implementation example. We’ll include private staging, a queue, a background worker calling our external image-analysis service, and a recorded decision. The upload endpoint will ultimately return a submission ID to the frontend, which will then use that ID behind the scenes to retrieve the current upload status and display updates beside the uploaded image. Users will then be able to see whether their submission is processing, approved, or awaiting review without ever handling the ID themselves. With this implementation, both the user and the application get exactly what they need. Separating Submission From Evaluation We’ll create a clear divide in our processing workflow between an upload endpoint and a background worker. The endpoint will save the image and queue its submission ID; the worker will retrieve that ID, open the saved image, and request an assessment. This type of separation means the upload response can be returned before detection actually finishes. If we were to await detection inside the endpoint, we would still leave the client waiting for that result. ASP.NET Core provides BackgroundService for implementing the worker, so we’ll take advantage of that in our code. As we’ve described, the frontend will retain the returned ID and use it to retrieve status automatically. The actual user who submitted the image will be able to see “Processing” next to their upload without handling the response ID manually. Just note that the .NET 10 excerpts we’ll use assume an existing authenticated application with some form of upload validation in place. We’ll specifically demonstrate the processing components and their integration points. Creating a Shared Submission Record Right off the bat, we know the endpoint and worker need a shared record that connects an image to its current status. We’ll create this relationship with an immutable record and a concurrent dictionary for our single-instance demonstration: C# using System.Collections.Concurrent; public enum SubmissionStatus { Pending, Processing, Approved, AwaitingReview, Unverified } public sealed record DetectionResult( bool? CleanResult, double? AiGeneratedRiskScore, string? AiSource); public sealed record Submission( Guid Id, string OwnerId, string FilePath, SubmissionStatus Status, DateTimeOffset UpdatedAt, DetectionResult? Detection = null); public sealed class SubmissionStore { public ConcurrentDictionary<Guid, Submission> Items { get; } = new(); } Here, the endpoint creates a Pending record, and the worker replaces it with updated copies as evaluation progresses. This lets status requests read complete snapshots. The AwaitingReview status identifies flagged content, while Unverified identifies an unusable or failed assessment. Approved means the image passed this image-origin policy (not every possible publication or security check; we’ll assume there are several others). Creating the Queue Now we’ll create the actual connection between the endpoint and the worker. A bounded Channel<Guid> holds submission IDs until the worker reads them: C# using System.Threading.Channels; public sealed class ReviewQueue { private readonly Channel<Guid> channel = Channel.CreateBounded<Guid>(new BoundedChannelOptions(100) { SingleReader = true, SingleWriter = false, FullMode = BoundedChannelFullMode.Wait }); public bool TryEnqueue(Guid id) => channel.Writer.TryWrite(id); public IAsyncEnumerable<Guid> ReadAllAsync(CancellationToken ct) => channel.Reader.ReadAllAsync(ct); } Note that in our example, we’ve allowed 100 waiting jobs. Wait mode prevents existing jobs from being discarded, while TryWrite immediately returns false when capacity is completely exhausted. That allows our endpoint to reject the submission instead of sitting around waiting for it indefinitely. Saving and Scheduling the Upload Within our existing upload handler, we’ll first enforce file-size and supported-format checks (this rejects invalid or potentially unsafe files before they're stored or processed). To prevent users from guessing filenames or accessing uploads directly, it’s important that we save the image under a server-generated filename in a private directory. After validation, the below code saves image, an IFormFile. The configured staging directory is privateDirectory, and ownerId comes from the authenticated identity: C# Directory.CreateDirectory(privateDirectory); var id = Guid.NewGuid(); var path = Path.Combine(privateDirectory, id.ToString("N")); var fileCreated = false; try { await using var output = new FileStream( path, FileMode.CreateNew, FileAccess.Write); fileCreated = true; await image.CopyToAsync(output, cancellationToken); } catch { if (fileCreated) File.Delete(path); throw; } store.Items[id] = new Submission( id, ownerId, path, SubmissionStatus.Pending, DateTimeOffset.UtcNow); if (!queue.TryEnqueue(id)) { store.Items.TryRemove(id, out _); File.Delete(path); return Results.StatusCode( StatusCodes.Status503ServiceUnavailable); } return Results.Accepted( $"/submissions/{id}", new { Id = id, Status = "Pending" }); Note that this excerpt uses injected SubmissionStore store and ReviewQueue queue instances; privateDirectory must be an absolute path outside publicly served directories. We’ve designed this so the file is closed before enqueueing, which allows the worker to reopen it. If scheduling fails for some reason, we simply remove the unscheduled record and file. The 202 Accepted response is what confirms acceptance for processing; its location points the frontend to the status endpoint we’ll implement below. Implementing the Detection Call We’ll now put the remote request in a dedicated Detector class. This gives the worker exactly one operation to invoke — DetectAsync(path, cancellationToken). We could theoretically use a hosted classifier for the same role, with our team managing its model and infrastructure. In this case, the implementation calls our AI image-detection endpoint using multipart form data. C# using System.Net.Http.Json; public sealed class Detector( IHttpClientFactory clients, IConfiguration configuration) { public async Task<DetectionResult?> DetectAsync( string path, CancellationToken ct) { var key = configuration["Cloudmersive:ApiKey"]; if (string.IsNullOrWhiteSpace(key)) throw new InvalidOperationException( "Configure the Cloudmersive API key."); using var client = clients.CreateClient("Cloudmersive"); using var file = File.OpenRead(path); using var form = new MultipartFormDataContent(); form.Add( new StreamContent(file), "imageFile", "submission"); using var request = new HttpRequestMessage( HttpMethod.Post, "image/ai-detection/file"); request.Headers.Add("Apikey", key); request.Content = form; using var response = await client.SendAsync(request, ct); response.EnsureSuccessStatusCode(); return await response.Content .ReadFromJsonAsync<DetectionResult>( cancellationToken: ct); } } Note that the API key here comes from server-side configuration. The method just opens the staged image, submits its bytes, and then deserializes the response on the other end. Any HTTP failures cause an exception for the worker to handle. If you happened to read my previous article on AI detection, you’ll know that CleanResult supplies the summarized classification for this service, and AiGeneratedRiskScore provides the risk signal we’re looking for. AiSource then adds optional source context (this is information about the exact AI model that created the image; it’s not always available, even when images are deemed high-probability AI). The key thing to note is that we’re working with probabilistic signals rather than definitive proof of authorship. Turning the Assessment into a Decision We’ll now implement the decision portion separately so the worker can apply the same rule to every response: C# public static class ReviewPolicy { public static SubmissionStatus Evaluate(DetectionResult? result) { if (result?.AiGeneratedRiskScore is not double score || !double.IsFinite(score) || score < 0 || score > 1 || result.CleanResult is null) { return SubmissionStatus.Unverified; } // Illustrative application threshold. const double reviewThreshold = 0.5; return result.CleanResult == false || score > reviewThreshold ? SubmissionStatus.AwaitingReview : SubmissionStatus.Approved; } } In this example, we approve usable results with a clean classification and an AI-generated-probability score no higher than 0.5. Processing Jobs and Saving Outcomes At this point, we can connect all the pieces. Our worker will read an ID, load its record, mark it as “processing,” and call Detector. After that, it'll replace the record with the response and policy decision: C# public sealed class ReviewWorker( ReviewQueue queue, SubmissionStore store, Detector detector, ILogger<ReviewWorker> logger) : BackgroundService { protected override async Task ExecuteAsync(CancellationToken ct) { try { await foreach (var id in queue.ReadAllAsync(ct)) { if (!store.Items.TryGetValue(id, out var item)) continue; store.Items[id] = item with { Status = SubmissionStatus.Processing, UpdatedAt = DateTimeOffset.UtcNow }; try { var result = await detector.DetectAsync( item.FilePath, ct); store.Items[id] = item with { Status = ReviewPolicy.Evaluate(result), Detection = result, UpdatedAt = DateTimeOffset.UtcNow }; } catch (Exception ex) { store.Items[id] = item with { Status = SubmissionStatus.Unverified, UpdatedAt = DateTimeOffset.UtcNow }; if (ct.IsCancellationRequested) break; logger.LogError( ex, "Evaluation failed for {Id}", id); } } } catch (OperationCanceledException) when (ct.IsCancellationRequested) { // Application shutdown while waiting for work. } } } We’ve handled each failure inside the loop such that unsuccessful scans don’t prevent subsequent jobs from happening. Note that in this compact version, we only perform one attempt. If we were to add retries, that would require bounded attempts, along with transient-error classification and respect for Retry-After. Before builder.Build(), we register the shared components and configure the HTTP client: C# builder.Services.AddSingleton<SubmissionStore>(); builder.Services.AddSingleton<ReviewQueue>(); builder.Services.AddSingleton<Detector>(); builder.Services.AddHostedService<ReviewWorker>(); builder.Services.AddHttpClient("Cloudmersive", client => { client.BaseAddress = new Uri("https://api.cloudmersive.com/"); client.Timeout = TimeSpan.FromSeconds(60); }); Requests are connected with the worker via the singleton store and the queue. The timeout bounds each remote call (its value is an application choice). Returning Status and Enforcing Approval The frontend can now poll a status route. This handler checks the authenticated owner before returning a limited view of the record. We’ll place both routes after builder.Build() and before app.Run(), with the using directive at the top of Program.cs: C# using System.Security.Claims; app.MapGet("/submissions/{id:guid}", (Guid id, ClaimsPrincipal user, SubmissionStore store) => { var ownerId = user.FindFirstValue(ClaimTypes.NameIdentifier); if (string.IsNullOrWhiteSpace(ownerId) || !store.Items.TryGetValue(id, out var item) || item.OwnerId != ownerId) { return Results.NotFound(); } return Results.Ok(new { item.Id, Status = item.Status.ToString(), item.UpdatedAt }); }).RequireAuthorization(); The host needs to validate authentication and provide the same stable owner claim used during the upload process. The frontend will use the returned status to update its interface automatically. That same ownership check is performed by the download route. This additional condition is then applied before the file is returned: C# app.MapGet("/submissions/{id:guid}/file", (Guid id, ClaimsPrincipal user, SubmissionStore store) => { var ownerId = user.FindFirstValue(ClaimTypes.NameIdentifier); if (string.IsNullOrWhiteSpace(ownerId) || !store.Items.TryGetValue(id, out var item) || item.OwnerId != ownerId) { return Results.NotFound(); } if (item.Status != SubmissionStatus.Approved) { return Results.Conflict(new { Error = "Submission is not approved." }); } return Results.File( item.FilePath, "application/octet-stream", fileDownloadName: "submission"); }).RequireAuthorization(); Note that private storage will prevent direct public access from bypassing this check, so human reviewers need a separate authorized access path. Testing and Production Considerations To verify things like approval, review, and missing-result handling, we can substitute controlled detector responses. In the event we get simulated request failures, they should produce Unverified, while downloads should remain blocked during processing. Critically, another user’s request should NOT reveal the submission. It’s important to bear in mind that this demo code also loses its queue and records on restart. That’s obviously not production-ready, since production usually requires durable jobs and records on top of coordinated enqueuing and duplicate-job handling. Additionally, we should probably retain things like creation timestamps and attempt history in our production code. Conclusion In this article, we learned how to connect private portal uploads to a background AI-content detector through a shared record and queue. Our worker records an explicit outcome, our frontend retrieves it automatically, and our download route enforces content approval. With this workflow in place, our application can acknowledge submissions promptly while keeping unfinished and unsuccessful content evaluations out of normal publication workflows.

By Brian O'Neill CORE

A few months ago, I watched a senior engineer spend forty-five minutes reviewing a single pull request — a PR that an AI assistant had generated in under two minutes. The code looked clean. The tests passed. But she kept cross-referencing an incident postmortem from eight months earlier, muttering something about retry amplification. She caught a real production risk. The AI reviewer had flagged nothing. That moment stuck with me. We've spent years optimizing how fast we can write code. But we haven't seriously reckoned with what happens when review can't keep up. The Bottleneck Has Shifted A single engineer with AI assistance can now produce hundreds of lines of code, large refactors, infrastructure changes, and test suites — all within minutes. Review complexity, however, grows exponentially with change size and system interdependency. The core problem is no longer "Can AI write code?" It's "Can humans reliably validate what AI wrote?" Code generation speed increases. Human cognitive review capacity stays flat. That imbalance is quietly accumulating risk in engineering organizations everywhere. Why Current AI Reviewers Fall Short Most AI PR review systems today operate on static diffs, syntax-level reasoning, and shallow best-practice detection. They produce comments like: "Potential null pointer.""Consider renaming this variable.""Possible optimization opportunity." Occasionally useful. Rarely sufficient for production-critical systems. The structural problem is that these tools treat PR review as a language problem instead of a systems reasoning problem. They assume software correctness is inferable from local code semantics alone. In reality, production safety emerges from interactions between architecture, runtime behavior, operational history, and organizational context. The Shallow Review Problem in Practice Here's a concrete example. An AI assistant generates this database query optimization: Python # AI-optimized version def get_user_orders(user_id): return db.query(""" SELECT o.*, p.*, i.* FROM orders o JOIN payments p ON o.id = p.order_id JOIN items i ON o.id = i.order_id WHERE o.user_id = ? """, user_id) Typical AI reviewer comment: "Query optimized with JOIN to reduce round trips." What a senior engineer sees: "This will cause a Cartesian explosion. The orders table has 50M rows, items averages 8 per order. This returns 400M+ rows for power users. We had a nearly identical incident (INC-287) that took down the read replica. Needs pagination and selective columns." The difference isn't token count or model size. It's operational memory and causal reasoning. The Real Challenge Is Not Context Windows Many people assume the fix is larger context windows. Feed the model the whole repo, and it'll review like a senior engineer. But experienced engineers don't review code by loading entire systems into working memory. They use abstraction, selective attention, and compressed mental models. A senior engineer reviewing a Kafka retry change doesn't reread the entire messaging subsystem — they remember prior incidents, retry amplification risks, and historical outages. That's cognitive compression, not token recall. Modern LLMs are exceptional at syntax fluency, pattern completion, and probabilistic association — what you might call token intelligence. But effective PR review requires something deeper: causal reasoning, architectural abstraction, operational memory, risk forecasting. Call it cognitive intelligence — persistent contextual reasoning grounded in operational history and causality. The distinction matters because it changes what we need to build. What a Cognitive Review Architecture Looks Like Instead of: Plain Text Large Prompt + Large LLM → Review We need: Plain Text Structured Memory + Semantic Retrieval + Runtime Context + Specialized Review Agents + Reasoning Layer + LLM → Review The LLM should not be the memory. It should be the reasoning interface over structured engineering knowledge. Intent Reconstruction Before reviewing code, the system needs to understand why the change exists. Business intent, bug root cause, architectural motivation. Inputs include Jira tickets, PR descriptions, ADRs, incident reports, and commit timelines. Without intent, review quality stays shallow regardless of model size. Engineering Knowledge Graphs Human reviewers carry organizational memory: fragile services, latency-sensitive paths, scaling bottlenecks, previous outages, dangerous dependencies. AI reviewers need persistent semantic memory systems encoding the same — service relationships, API contracts, operational metadata, incident history, ownership boundaries. This creates an engineering cognition layer far richer than raw repository context. Multi-Agent Review Systems A single reviewer model is insufficient. Future systems will consist of specialized agents working together: Architecture Reviewer – dependency boundaries, coupling risk, architectural driftReliability Reviewer – retries, backpressure, idempotency, failover behaviorSecurity Reviewer – injection risks, auth issues, secret exposurePerformance Reviewer – memory growth, query amplification, scaling regressionsHistorical Regression Reviewer – correlation with past outages, postmortems, incident fingerprints This begins to approximate how experienced engineering organizations actually review software. Runtime-Aware Review Static analysis alone misses emergent runtime behavior. Future cognitive review systems will integrate observability telemetry, tracing data, production metrics, and traffic patterns. Compare these two responses to a retry configuration change: Traditional AI reviewer: "Code follows retry best practices." Cognitive AI reviewer with operational memory: "HIGH RISK: Similar retry configuration caused incident on 2023-09-15. This service processes 2M messages/hour at peak. 10 retries with exponential backoff = up to 17 minutes per message. Previous incident resulted in 8M message consumer lag and cascading downstream failures. Recommend: max 3 retries, circuit breaker, dead letter queue, idempotency check before db.save(). See ADR-089." That is a fundamentally different class of intelligence — and a fundamentally different class of safety. Engineering Memory Is the Missing Piece One of the biggest gaps in current AI systems is durable operational memory. Experienced engineers develop intuition through outages, failed deployments, debugging sessions, and production emergencies. These experiences become compressed heuristics: "This retry increase feels dangerous" — not because of syntax, but because of remembered causal relationships. Replicating this requires episodic memory systems, incident-aware reasoning, and causal knowledge graphs. Much of this mirrors practices long established in Site Reliability Engineering, where institutional learning from incidents is treated as critical infrastructure. Incident postmortems aren't just documentation — they're organizational immune system responses. Getting AI systems to genuinely learn from incidents rather than just pattern-match against them remains one of the harder open problems in this space. What Teams Can Do Today Fully cognitive review systems don't exist yet. But organizations can meaningfully improve AI-assisted review quality right now: Capture architectural knowledge in machine-readable form. Service boundaries, retry policies, timeout configurations, scaling assumptions — not just in wikis, but in structured formats AI systems can query.Link PRs explicitly to incident history. Build connections between code changes and the incidents they caused or prevented. This is organizational memory that AI systems can leverage today.Tag services with operational metadata. Criticality tier, traffic patterns, known failure modes, blast radius. Treat repositories as systems, not just files.Integrate observability into review pipelines. Connect production metrics and tracing data to code review. Runtime context dramatically improves review quality.Prioritize high-signal AI feedback. Review fatigue from noisy, low-signal comments is a real trust problem. Focus AI comments on incident-correlated patterns, architectural violations, and operational risks. The Trust Calibration Problem One concern I keep coming back to: bad AI reviewers are dangerous not because they miss things, but because they sound confident while missing things. They reduce human vigilance through automation bias. They generate fatigue through noise. They normalize shallow approval. Future cognitive review systems need to be not just more accurate, but properly calibrated — knowing when they lack sufficient context and escalating accordingly. An AI reviewer should be able to say: "I may not have enough confidence to validate this safely." That self-awareness may matter more than raw capability. The Road Ahead The next era of AI software engineering will not be defined by who generates the most code. It will be defined by trust, reasoning quality, and operational awareness. The future belongs to systems capable of understanding not just what changed — but why it changed, what it affects, and whether it's safe. That's the difference between code generation and engineering intelligence. And honestly, solving it seems harder and more interesting than anything we've built so far. Key Takeaways The bottleneck has shifted from code generation to code review and validation.Larger context windows alone won't bridge token intelligence and cognitive intelligence.Human-like review requires structured memory, causal reasoning, and operational awareness.Multi-agent architectures with specialized reviewers mirror how engineering teams actually work.Runtime-aware systems integrating production telemetry represent the next frontier.Engineering memory — learning from incidents — is critical for trust and safety.Teams can start today by capturing architectural knowledge and linking incidents to code changes. References Vaswani, A., et al. (2017). "Attention Is All You Need." NeurIPS.Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.Lewis, P., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS.Shinn, N., et al. (2023). "Reflexion: Language Agents with Verbal Reinforcement Learning." arXiv.Beyer, B., et al. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media.Allspaw, J. (2012). "Blameless PostMortems and a Just Culture." Etsy Engineering.

A mobile request can fail without the server-side work failing. An iOS app may time out, lose the response after a POST has reached the service, or retry after connectivity changes while the original execution is still progressing. Apple explicitly distinguishes safe retry behavior by HTTP method and notes that URLSession can retry requests in some connection-loss cases, waitsForConnectivity can also cause the system to continue a request when connectivity returns. The dangerous state is therefore not “request failed,” but “completion is unknown.” If that request starts an agent that charges an account, reserves inventory, sends a message, or invokes an MCP tool, a second submission can become a second side effect. The Retry Boundary Is the Real Transaction Boundary “Exactly once” is too strong for a workflow crossing an iPhone, HTTP, an agent runtime, an MCP server, Kafka, a database, and an external API. Kafka can provide exactly-once guarantees within defined Kafka processing boundaries, but those guarantees do not atomically include arbitrary remote tool effects. The practical target is effectively-once behavior, and retries are expected, but every effect is guarded by a stable operation identity and converges on one committed outcome. Kafka’s idempotent producer suppresses duplicate records caused by producer retries, while transactional producers can atomically publish across Kafka partitions; the producer documentation also limits idempotence guarantees to a producer session and requires read_committed consumers for end-to-end transactional visibility. The operation identity must exist before the first network attempt. An iOS client can create an operationId when an action becomes durable local intent, persist it, and reuse it across transport retries. Transport material such as a server challenge may change, but the business ID must not. The server treats (subjectId, operationId) as a uniqueness boundary and stores a canonical payload hash with it. PostgreSQL unique constraints enforce row uniqueness, while INSERT ... ON CONFLICT provides an atomic conflict path under concurrency. SQL INSERT INTO agent_operation(subject_id, operation_id, payload_hash, status) VALUES (:subject, :operationId, :payloadHash, 'ACCEPTED') ON CONFLICT (subject_id, operation_id) DO NOTHING; A conflict with the same payload hash returns the existing operation; a different hash rejects key reuse. The record should exist before agent execution starts, and the accepted response should expose the durable operation identity. Let LangGraph Resume Without Repeating Effects LangGraph persistence is useful precisely because durable execution can replay code. With a checkpointer, LangGraph saves state at super-step boundaries; if execution resumes after a failure, an affected node can run again from the beginning. Official guidance consequently requires idempotent node logic, and task results can be checkpointed so completed task work can be reused during resume instead of recomputed. Replaying from an earlier checkpoint can also re-trigger later LLM calls and API requests. A stable business operation should therefore map to a stable LangGraph thread, while every effectful tool boundary receives the same operation ID. Python config = {"configurable": {"thread_id": operation_id} result = graph.invoke( {"operation_id": operation_id, "command": command}, config ) Checkpointing reduces recomputation but does not replace downstream idempotency. A reservation can succeed before its task result is durably checkpointed. LangGraph’s functional API therefore recommends idempotent tasks because an incomplete task can execute again during resume. Python @task def reserve_inventory(operation_id, sku, quantity): return mcp.call_tool("reserve_inventory", { "operationId": operation_id, "sku": sku, "quantity": quantity }) The significant property in this snippet is not the decorator. The important part is that the business identity crosses the graph boundary and reaches the tool implementation. A downstream inventory service can then use that identity to return a previously committed reservation rather than creating another one. MCP Tasks Are Durable Handles, Not Deduplication Keys The current MCP Tasks design is especially relevant to long-running agent tools. In the July 28, 2026 protocol revision, Tasks moved into the io.modelcontextprotocol/tasks extension. A server can return a durable task handle, and the client can poll with tasks/get, provide input with tasks/update, or request cancellation with tasks/cancel. The task is durably created before its handle is returned, which allows polling after a disconnect. That durability solves result retrieval after task creation, but it does not by itself deduplicate the request that creates the task. The task ID is server-generated. If the server creates task A, the response disappears, and the original tools/call is sent again, a naïve implementation can create task B. Therefore, the business operationId must be part of the tool arguments or equivalent application metadata, and task creation must first look up an existing operation. This follows directly from MCP’s server-generated task-ID model combined with retry ambiguity at the HTTP boundary. The MCP server can return an existing task handle for the same authenticated subject, operation ID, and payload hash, and later return the stored terminal result. Cancellation should also be idempotent because MCP defines it as cooperative rather than a guarantee that underlying work stops immediately. Keep Kafka Guarantees Inside Kafka Kafka is most valuable after the operation has been claimed. A database transaction can persist operation state with an outbox row carrying the same ID. Kafka producer idempotence protects against duplicates caused by producer retries, while consumers can still use the operation ID for application-level deduplication. Kafka transactions can atomically cover Kafka writes, but they do not extend over an MCP server or payment API. The event contract should preserve causality rather than inventing a new identity at each hop. JSON { "operationId": "8E7B6D9E-...", "type": "AgentToolCompleted", "tool": "reserve_inventory", "status": "SUCCEEDED" } A consumer can enforce uniqueness on (consumerName, operationId, eventType) or make the state transition conditional. Kafka delivery guarantees and application idempotency then reinforce each other instead of being treated as interchangeable. Bind Retry Identity to App Attest Without Blocking Legitimate Retries App Attest addresses a different failure mode: whether a request comes from a legitimate app instance and whether signed request material has been replayed or altered. Apple’s current guidance uses a server-provided challenge for assertions and requires the server to validate a strictly increasing assertion counter; that counter is specifically an anti-replay signal. Assertions are generated locally on the device after key attestation. The App Attest assertion must not become the business idempotency token. A legitimate retry should obtain fresh challenge material and generate a fresh assertion while retaining the original operation ID. The data hashed for the assertion can bind the server challenge, operation ID, and canonical payload hash together. Swift let payloadHash = SHA256.hash(data: body) let clientData = challenge + operationID.data + Data(payloadHash) let clientDataHash = Data(SHA256.hash(data: clientData)) let assertion = try await service.generateAssertion( keyID, clientDataHash: clientDataHash ) Apple recommends server-controlled challenges, server-side validation, and assertion-counter tracking as assertions are generated on demand without a round trip to Apple’s servers. The server verifies App Attest, checks that the challenge binds the operation ID and payload, then performs the idempotency lookup. A fresh assertion can retry the same operation; a replayed assertion fails anti-replay validation; an altered payload fails the hash check. Effectively-Once Behavior Is a Composition Property Reliable agent execution does not come from asking iOS to retry less often or from labeling a Kafka pipeline “exactly once.” It comes from carrying one durable business identity across every retry and every boundary, claiming that identity atomically before execution, making LangGraph effects idempotent under resume, using MCP Tasks as durable result handles rather than creation-time deduplication keys, restricting Kafka’s exactly-once guarantees to Kafka’s transactional domain, and using App Attest to prove request integrity without confusing anti-replay state with business deduplication. When those boundaries align, a lost mobile response can cause another HTTP attempt, another graph invocation, or another poll, but it does not cause another business effect. That is the operational meaning of effectively once.

By Uthej Mopathi CORE

A Temporal Workflow that appears stuck is rarely “stuck” in the conventional process sense. Temporal persists Workflow state through Event History and resumes execution through replay, so an open execution can remain healthy while waiting for a timer, Signal, Activity, or external condition. The operational problem is therefore not simply lack of completion; it is lack of expected progress. Effective diagnosis starts by establishing what event should have happened next, why it did not happen, and whether remediation can preserve the Workflow’s business invariants. Temporal’s history model makes that analysis unusually tractable because commands, task transitions, Activity attempts, failures, timers, and external interactions are durably represented as Events. Progress Is Visible in the Event History The first diagnostic artifact should be the execution description and raw history, not application logs. temporal workflow describe exposes current execution information and pending Activity state, while temporal workflow show --output json returns Event History in a form suitable for programmatic replay or analysis. A Workflow Query can additionally expose application-defined state without mutating the execution. Shell temporal workflow describe --workflow-id order-7814 temporal workflow show \ --workflow-id order-7814 \ --output json History should be read as a state-transition trace. A WorkflowTaskScheduled event with no corresponding start suggests that work is waiting for a Worker. A started Workflow Task that repeatedly times out can indicate blocked Workflow code, Worker instability, or excessive work inside a task. Repeated WorkflowTaskFailed events can indicate replay or deterministic-compatibility failures after code deployment. Workflow Task failures are retried by Temporal rather than governed by an Activity-style Retry Policy, so a Workflow can remain open while repeatedly failing to make application-level progress. Activity sequences reveal a different failure surface. ActivityTaskScheduled without ActivityTaskStarted points toward dispatch capacity, missing pollers, queue mismatch, or backlog. Temporal persists Workflow and Activity Tasks in Task Queues, and worker-health guidance identifies Schedule-to-Start latency and approximate backlog count as key signals when tasks wait for Workers. ActivityTaskStarted without completion requires inspection of Start-to-Close and Heartbeat behavior because Temporal relies on Start-to-Close timeout to detect a Worker crash after an Activity has started. Not every long pause is pathological. A timer that has not fired, a Workflow waiting for a Signal, or an Activity still inside a valid timeout window can represent correct durable waiting. Conversely, very large histories can become an operational risk. Temporal warns after 10,240 events or 10 MB and enforces a limit of 51,200 events or 50 MB; Continue-As-New creates a new run with a fresh history while carrying forward relevant state. Triage Works Best as Deterministic Evidence Before Model Judgment LangGraph is useful for automating this analysis, but the safest design keeps Temporal facts deterministic and uses an LLM only for classification, hypothesis ranking, and explanation. LangGraph explicitly supports graphs that mix deterministic nodes with model-driven nodes, while structured output can constrain routing decisions into a defined schema rather than free-form text. A compact analyzer can first reduce raw history into evidence that is difficult to hallucinate: the last completed Workflow Task, consecutive Workflow Task failures, pending Activity IDs, the latest Activity attempt, the timeout type, the last Signal, the last timer, the history size, the task queue, and deployment/version metadata. The model then receives that normalized evidence instead of thousands of raw events. Python def extract_facts(state): events = state["events"] return { "facts": temporal_fact_extractor(events), "tail": events[-60:], } def classify(state): result = triage_model.with_structured_output(TriageResult).invoke({ "facts": state["facts"], "tail": state["tail"], "allowed_causes": [ "worker_unavailable", "activity_retrying", "workflow_task_failure", "intentional_wait", "history_pressure", "unknown", ], }) return {"triage": result} That separation matters operationally. Event parsing can enforce hard rules such as “scheduled but never started,” while the model can correlate several weak signals and produce an explanation. Conditional edges can then route low-risk cases to observation, ambiguous cases to deeper diagnostics, and recovery candidates to an approval gate. LangGraph’s graph API supports conditional routing, and persistence stores checkpoints so triage state survives interruptions or process failures. Recovery Must Preserve Temporal and Business Semantics Diagnosis and remediation should remain separate graph stages. A model-generated recommendation must not directly issue cancellation, reset, or termination. LangGraph interrupts provide a natural control boundary because execution can pause with persisted state and resume only after external approval. Python def approval_gate(state): decision = interrupt({ "workflow_id": state["workflow_id"], "cause": state["triage"].cause, "action": state["triage"].recommended_action, "evidence": state["triage"].evidence, }) return {"approved": decision == "approve"} The remediation choice depends on the failure mode. A transient Worker outage usually requires restoring Worker capacity rather than mutating Workflow state because queued tasks persist until Workers can process them. An Activity repeatedly failing on a recoverable dependency can often be left to its Retry Policy, while permanent errors should be made non-retryable in application design to avoid pointless retries. Activity side effects should be idempotent because Activity attempts may execute more than once under retry and recovery behavior. Cancellation is the preferred stop mechanism when Workflow cleanup logic must run. Temporal records a cancellation request and schedules a Workflow Task so Workflow code can react. Termination is forceful: Workflow code does not receive a chance to clean up, and the terminated event closes the history. That makes termination an escalation path for executions that cannot process cancellation normally. Reset is more powerful and more dangerous. Temporal terminates the current execution and creates a new execution that copies history through a selected reset point, then replays forward using current Workflow code. Progress after the reset point is discarded. Reset is therefore appropriate only after the underlying cause has been corrected and after downstream side effects are reviewed for possible re-execution beyond the reset boundary. Shell temporal workflow reset \ --workflow-id order-7814 \ --event-id 42 \ --reason "Recovered after deterministic-compatibility fix" For history pressure rather than a fault, Continue-As-New is generally the safer lifecycle mechanism because it preserves logical continuity under the same Workflow ID while starting a fresh Event History with a new Run ID. It should be designed into long-lived or high-volume Workflow logic instead of used as an improvised emergency action. Safe Automation Requires an Explicit Remediation Envelope A production triage graph should treat remediation as a constrained transaction. The evidence snapshot, selected run ID, candidate reset event, intended action, reason, approval identity, and execution result should all be persisted before any mutation. The action node should re-read the Workflow immediately before execution and reject the operation if the run has changed or the observed condition no longer matches the diagnosis. This is an engineering safeguard rather than a Temporal requirement, but it reduces time-of-check/time-of-use errors when active Workflows continue progressing during investigation. LangGraph’s checkpoint model supports durable approval state, but resumed graph nodes can re-execute from checkpoint boundaries. Its documentation therefore recommends isolating side effects and designing them to be idempotent. A remediation executor should consequently use an operation ID, record completion externally, and refuse duplicate destructive actions. Recovery Without Guesswork Reliable recovery of a stuck Temporal Workflow is fundamentally an event-history problem, not a process-restart problem. The strongest diagnostic path reconstructs expected progress from Workflow Tasks, Activity attempts, timers, Signals, queue state, timeouts, and history growth before considering mutation. LangGraph can turn that evidence into a durable triage pipeline by combining deterministic extraction, constrained model reasoning, conditional routing, and interrupt-based approval. Safe remediation then follows Temporal semantics: restore Workers when dispatch is the issue, allow bounded retries for transient Activities, cancel when cleanup matters, terminate only as a last resort, reset only after the root cause is fixed, and use Continue-As-New to control long-running history growth. The result is automation that accelerates incident response without allowing probabilistic diagnosis to become an unchecked control plane.

By Akhil Madineni CORE

Convolutional neural network workloads rarely fail because the forward pass is mathematically difficult. They fail because modern training and inference pipelines are distributed systems: datasets arrive late, GPU workers disappear, validation jobs stall, model registration breaks halfway through, and long-running executions need to resume without corrupting state. Temporal is designed for exactly that class of problem. A Temporal Workflow Execution is durable, reliable, and scalable, and Temporal defines durable execution as the ability of a workflow to maintain state and progress through crashes or outages. That makes it a strong fit for CNN pipelines whose control plane must survive for hours, days, or even longer while the actual tensor computation runs elsewhere. Why Temporal Fits CNN Pipelines A useful way to think about Temporal in ML systems is as a durable control plane rather than a replacement for PyTorch, CUDA, or a model server. Temporal Workflows hold orchestration state, issue commands, wait on results, and recover by replaying event history. The workflow code itself must stay deterministic, while failure-prone and non-deterministic work belongs in Activities. Temporal’s own documentation is explicit on that boundary: API calls, file I/O, database access, and other external interactions belong in Activities, while workflows should remain replay-safe. That separation aligns naturally with CNN systems, where data staging, training job submission, checkpoint writes, validation, and registry updates interact with external systems constantly. That design matters because a CNN training run is not one step. Even a modest image-classification job usually has a training phase, a validation phase, best-model selection, a checkpointing loop, and a final artifact publication step. The PyTorch transfer-learning tutorial illustrates that pattern directly by alternating train and validation phases, persisting the best state_dict, and loading the best weights at the end; the quickstart tutorial likewise uses model.eval() and torch.no_grad() before prediction. Temporal does not change the math of those stages. It makes the sequence durable, observable, and restartable. A concise workflow can therefore stay almost entirely orchestration-focused: Python @workflow.defn class CnnTrainingWorkflow: @workflow.run async def run(self, req: TrainRequest) -> ModelArtifact: dataset = await workflow.execute_activity( prepare_dataset, req.dataset_ref, start_to_close_timeout=timedelta(minutes=20), ) trained = await workflow.execute_activity( train_cnn, TrainJob(dataset_uri=dataset.uri, config=req.config), start_to_close_timeout=timedelta(hours=8), heartbeat_timeout=timedelta(minutes=1), retry_policy=RetryPolicy(maximum_attempts=3), ) metrics = await workflow.execute_activity( evaluate_cnn, EvalJob(model_uri=trained.best_model_uri, dataset_uri=dataset.val_uri), start_to_close_timeout=timedelta(minutes=30), ) return await workflow.execute_activity( register_model, RegisterRequest(trained.best_model_uri, metrics), start_to_close_timeout=timedelta(minutes=5), ) The important detail in this snippet is not syntax but placement. The workflow issues durable commands, while every side effect lives inside an activity. That matches Temporal’s execution model, where workflows await activity results, activities carry retry policies, and activity timeouts define how long a unit of external work is allowed to run. For long GPU jobs, heartbeat_timeout is especially important because an activity heartbeat tells Temporal the worker is still alive and making progress. Making Long Training Jobs Resumable The natural temptation with CNN training is to keep a Python process alive for hours and hope that the host, container runtime, and storage path all behave. Temporal offers a more robust approach. Activities can be retried automatically after transient failure, and Temporal recommends Start-To-Close timeouts for activity executions. If heartbeats stop arriving inside the heartbeat timeout, the activity can be considered failed and retried according to policy. For training jobs that run on flaky GPU nodes or preemptible infrastructure, that is a meaningful improvement over ad hoc retry shells. The training activity itself should then follow framework-native checkpoint discipline instead of trying to serialize the full training loop into workflow state. PyTorch’s guidance is centered on saving and loading model state with state_dict, and the transfer-learning tutorial shows a production-relevant pattern: save the best model parameters during validation, then reload them at the end. Temporal’s own ML Ops example follows the same philosophy by highlighting checkpoint-aware fine-tuning and resumable inference, with deterministic orchestration in workflows and non-deterministic ML work in activities. A minimal activity therefore looks more like this: Python @activity.defn def train_cnn(job: TrainJob) -> TrainingResult: state = restore_checkpoint(job.resume_uri) model = build_model(job.config, state) best_acc = state.best_acc if state else 0.0 for epoch in range(state.next_epoch if state else 0, job.epochs): train_one_epoch(model, job.train_loader) val_acc = validate(model, job.val_loader) save_checkpoint(job.resume_uri, model, epoch, val_acc, best_acc) if val_acc >= best_acc: save_best_weights(job.best_model_uri, model) best_acc = val_acc activity.heartbeat({"epoch": epoch, "best_acc": best_acc}) return TrainingResult(best_model_uri=job.best_model_uri, best_acc=best_acc) This pattern is deliberately boring, which is a strength. The epoch loop stays in the activity, checkpoints remain in object storage or a shared filesystem, and each heartbeat advertises progress to Temporal. If a retry occurs, the activity can resume from the latest checkpoint rather than restarting from epoch zero. That is the same operational idea highlighted in Temporal’s ML sample repository, and it matches PyTorch’s recommendation to persist model parameters through state_dict-based saves and reloads. When training runs are submitted to an external batch scheduler instead of executing directly inside the worker process, Temporal’s asynchronous activity completion becomes especially useful. An activity can submit the job, capture the task token, and return without marking the activity complete. The external system can later heartbeat and complete the activity through a Temporal client. Temporal documents this explicitly and notes that asynchronous completion is preferable when the external process needs heartbeats or cancellation. Python @activity.defn async def submit_training(job: TrainJob): token = activity.info().task_token launch_gpu_job(job, token) activity.raise_complete_async() Treating Inference as a Workflow Only When It Is One Temporal is not a substitute for a low-latency online model server. A single image classification request that must return in milliseconds usually belongs in the serving layer. Temporal becomes valuable when inference is part of a larger durable process, such as nightly batch scoring, asynchronous document-image analysis, model fallback, approval gates, or multi-stage post-processing. That recommendation follows directly from Temporal’s model of workflows as durable, stateful executions that communicate through activities, child workflows, signals, queries, and schedules. Inside the inference activity, the framework rules remain unchanged. PyTorch examples set the model to evaluation mode and disable gradient tracking during inference. That is essential for CNNs that use dropout- or batchnorm-sensitive behavior and for avoiding unnecessary autograd overhead. Python @activity.defn def score_batch(req: BatchScoreRequest) -> BatchScoreResult: model = load_model(req.model_uri) model.eval() with torch.no_grad(): return predict_batch(model, req.input_uri) Temporal’s message-passing model then makes the surrounding orchestration easier to operate. The Python SDK documents that a workflow can act like a stateful web service receiving queries, signals, and updates. For a batch scoring workflow, a query can expose current shard progress, while a signal can switch the canary model version for the remaining work without restarting the execution. Recurring inference or retraining can be started through Temporal Schedules, which the documentation describes as a more flexible and user-friendly approach than cron jobs. Python @workflow.query def status(self) -> dict: return {"phase": self.phase, "completed": self.completed, "model": self.model_uri} @workflow.signal def switch_model(self, model_uri: str) -> None: self.model_uri = model_uri Keeping ML Workflows Operable as They Grow CNN platforms do not stay small for long. A single training run becomes a hyperparameter sweep, then a retraining program, then a fleet of region-specific models. Temporal scales that expansion through composition. A parent workflow can start child workflows for each experiment, fold, or dataset shard, and the child workflow APIs guarantee that the child has actually started before the call resolves. That makes fan-out training and batch inference easier to reason about than out-of-band job launchers with partial status tracking. Long-running ML control planes also run into two operational realities: event history growth and code evolution. Temporal addresses the first with Continue-As-New, which closes the current execution successfully and starts a new run with the same workflow ID and a fresh event history. It addresses the second with versioning support and, in production, Worker Versioning. Temporal recommends Worker Versioning as the default way to deploy changes safely, and pinned workflows can stay on the worker deployment version where they started. That is especially relevant for training or evaluation flows that may stay active across multiple application releases. Conclusion Temporal brings discipline to CNN systems by separating durable orchestration from non-deterministic GPU work. The workflow owns state, retries, waiting, composition, and observability. Activities own data movement, training submission, checkpointing, evaluation, and inference execution. PyTorch continues to provide the familiar mechanics of state_dict checkpoints, evaluation mode, and gradient-free inference, while Temporal turns those stages into a resilient end-to-end process that can survive infrastructure faults, external scheduler delays, and repeated deployment cycles. For teams building CNN platforms that have outgrown shell scripts and brittle job glue, that combination is not merely convenient. It is often the boundary between a model pipeline that occasionally works and one that can be operated confidently in production.

By Akhil Madineni CORE

A few weeks ago, I participated in a 24-hour AI hackathon where we built a product using AI. Necessary tools were provided, a large number of engineers participated eagerly, and a few business folks also joined to bring their ideas into a real-world product using AI. During brainstorming, people drafted end-to-end process flow diagrams and started working on development, using all the recent available models and platforms to build their product. When the development time window ended, it was time for presentations. As I watched each team present their results, I observed that they couldn’t automate the end-to-end process flow. What they had planned during brainstorming didn’t turn out to be a complete, end-to-end product. Most of the solutions followed the same pattern: they did something in one product and took the output to another product, and the output of that product went somewhere else to finish the loop. Since the transition was handled by a human, it was considered “human in the loop.” No doubt this speeds up the process, but it left a gap — between an AI system that helps finish certain tasks in a process flow and an AI system that finishes the process end to end. Which one is the better model to build? Let’s Try to Define Assist and Complete An assisting system sits inside a workflow. AI takes some kind of input and produces an output. A human then takes the output, reviews it, and decides what to do with it — moving to the next system with or without AI, and continuing the process flow. If we removed the AI from this process flow, it could still run, and the work would still get done. It would just take more time. For example, in a real-world tech process, a business team uses AI to document a requirement, which is converted into a JIRA story using the Confluence AI tool, and a developer reads it and uses Copilot or another AI assistant to develop the code. A complete system owns the workflow’s outcome. Here, the system takes the input, processes it, makes the decision, triggers an action, and writes a log documenting the actions it has performed. If we remove any component from this process flow, it won’t just slow down — it will completely stop. As a real-world example, there is a solution that takes all the reviews for a company, analyzes them, categorizes them, summarizes them, and then raises a ticket with that summary and assigns it to the respective team. As an evolution of this process, teams often look only at their current work and how they can leverage AI in it, rather than looking at the end-to-end process flow integration. This is one of the reasons why AI investment doesn’t show up in the metrics that matter. It was architected to help, for sure, but not to finish. The Four-Part Test After watching this pattern across a number of different workflows, I have formed a set of four questions to address whether a process is structurally ready for a complete system, or whether it’s just going to support an assisting one. Is the trigger unambiguous? Something has to start the cycle — a batch starts on a schedule, a microservice is triggered by a button in the UI, a sensor crosses a threshold. If the start of the workflow is itself a judgment call that varies by who is looking at it, then we don’t have a clear trigger, and the downstream process isn’t ready to be fully automated. Can you define the decision logic? “Define” isn’t simple, but the system’s decision logic can be made explicit, even if the underlying model is complex. In a reliability process flow system, you can define what should be considered an error, what the threshold is, and when to act. Contrast that with deciding whether a customer complaint is about to escalate into a PR problem — the reasoning experienced staff uses to answer that is often genuinely tacit, inconsistent between two employees, and resistant to being written down.Is it executable in a real system? A complete system needs more than just an AI model. It might need an API, a write permission, a button it can press — not a dashboard a human reads and acts on separately based on what they see in it. This sounds obvious, doesn’t it? But teams often build excellent reasoning and never build the hooks to act on it. Can the outcome be verified? This is the most important question of all. After the action completes, can the system check and confirm whether the right action was taken and that it worked? Did the email send, did the case close correctly, did the transaction process without a downstream error? Verifiability is the closing step of a fully automated system, and it’s what makes the system safe. It’s what catches a bad decision before it compounds tenfold. When a workflow answers all four of these questions well, it is a strong candidate for a complete system. When even one question fails, that’s a signal that the workflow should stay an assisting one, regardless of how sophisticated the underlying model is. Why Most Systems Assist But Don’t Fully Complete A simple, straightforward question is why so many production systems stop at assist. In my experience, there are three structural reasons: Narrow decision logic: If a system has narrow decision logic in the process flow — binary answers like yes or no, or a defined set of cases — it can often be handled within the process flow regardless of AI involvement. But people try to introduce AI here anyway, and when it picks an answer that wasn’t pre-defined, that’s treated as a failure.Evidence validation failure: Teams give AI the middle of the workflow because it’s recoverable — a human validates the output at the end and catches problems before real damage occurs. Handing over the full cycle means trusting the outcome, but most organizations haven’t built the measurement discipline to know when that trust is justified. They deploy the AI and measure in time savings, but they never run the harder experiment: tracking what the system’s success rate would actually be if it were allowed to complete the cycle unsupervised. Unresolved accountability: This one is more cultural than technical, but it’s a real blocker. When a human closes the loop, it’s clear who is responsible. But when a system closes it, that question gets harder — especially in any environment where decisions face external scrutiny, like an auditor, a regulator, or a customer dispute. Most organizations keep a human in the closing position by default, regardless of whether that person is actually adding judgment in the majority of cases. The teams that get real value out of AI in the next few years won’t be the ones who bolt AI onto every step they can find. They will be the ones who took the time to evaluate, workflow by workflow, whether they were building something that helps or something that finishes.

Large language models have evolved from simple chat interfaces into autonomous systems capable of planning, reasoning, and interacting with external tools. The next stage of this evolution is multi-agent software engineering, where specialized AI agents collaborate to solve complex business workflows instead of relying on a single monolithic model. A planner may decompose work, researcher agents retrieve enterprise knowledge, coding agents generate implementations, reviewer agents validate outputs, and execution agents perform approved actions. Although this architecture appears attractive, production deployments reveal that coordinating multiple agents resembles building a distributed system far more than writing prompt chains. The primary challenge is not model intelligence but system reliability. Every additional agent introduces another opportunity for hallucinations, context loss, latency, retries, and cascading failures. A workflow containing five agents with individually high accuracy can still produce inconsistent outcomes because each handoff becomes another source of uncertainty. The engineering challenge therefore shifts from prompt engineering toward orchestration, state management, resilience, and observability. Most successful enterprise implementations begin with a planner-worker architecture. Instead of allowing every agent to communicate freely, a planner receives the business objective, decomposes it into smaller tasks, distributes work to specialized agents, and aggregates the responses into a final result. This pattern simplifies coordination, enables centralized policy enforcement, and provides a single location for monitoring execution. Java AgentPlan plan = planner.createPlan(request); List<CompletableFuture<AgentResult>> workers = plan.tasks().stream() .map(task -> CompletableFuture.supplyAsync( () -> worker.execute(task))) .toList(); List<AgentResult> results = workers.stream() .map(CompletableFuture::join) .toList(); return aggregator.combine(results); Bottlenecks Arise As the number of agents increases, direct synchronous communication quickly becomes a bottleneck. Event-driven messaging provides better scalability by allowing each agent to publish completed work while downstream agents subscribe only to events they understand. Kafka is particularly effective because partitions naturally distribute workloads across worker instances while preserving message ordering for individual workflows. The orchestration layer no longer manages worker availability directly and instead publishes work to topics, allowing consumer groups to handle scaling and recovery. A durable workflow engine becomes equally important. Stateless orchestration fails whenever a process crashes, a deployment occurs, or an agent exceeds execution time. Platforms such as Temporal persist workflow history so execution resumes from the last successful checkpoint rather than restarting an expensive reasoning process. This separation between orchestration and agent execution prevents duplicated work while making long-running AI workflows operationally reliable. Addressing Context Management Context management presents another significant engineering problem. Passing the complete conversation between every agent rapidly increases token consumption while reducing response quality. Instead, enterprise systems maintain workflow state separately from prompts. Business context is stored in persistent databases, semantic knowledge resides in vector stores, and external capabilities are exposed through Model Context Protocol (MCP) servers. Each agent retrieves only the information required for its current task instead of inheriting the entire execution history. Java workflowRepository.save( WorkflowState.builder() .workflowId(id) .currentAgent("SecurityReviewer") .status(Status.RUNNING) .context(serializedContext) .build() ); Standardizing communication between agents also improves maintainability. Rather than exchanging natural language, production systems often define structured contracts that include workflow identifiers, task types, priorities, and correlation identifiers. JSON { "workflowId": "WF-2041", "source": "Planner", "target": "CodeReviewer", "task": "Validate generated API", "traceId": "9bdc-421" } Structured messaging enables retries, auditing, replay, and interoperability across heterogeneous agents developed by different teams. It also aligns naturally with emerging protocols designed for agent interoperability. Reliability patterns from distributed systems remain equally valuable in AI applications. Agent failures should never stall an entire workflow. Timeouts, retries, circuit breakers, and dead-letter queues prevent individual components from consuming unlimited resources while protecting downstream services from cascading failures. Java try { AgentResponse response = future.get(20, TimeUnit.SECONDS); } catch (TimeoutException ex) { retryQueue.publish(task); circuitBreaker.recordFailure(); } Additional Issues to Consider Unlike conventional microservices, however, AI systems introduce another category of failure called reasoning loops. An agent may repeatedly invoke different tools while attempting to improve its answer without ever reaching completion. Runtime safeguards therefore extend beyond traditional retry limits to include maximum reasoning depth, token budgets, and execution deadlines. These controls prevent runaway costs while ensuring workflows terminate predictably. Production systems require complete visibility into every agent interaction. Traditional application logs reveal infrastructure failures but rarely explain why an AI workflow produced an incorrect decision. Distributed tracing with OpenTelemetry allows each planner, worker, and tool invocation to emit correlated telemetry containing workflow identifiers, agent names, execution latency, token usage, and tool calls. A single trace can reconstruct the entire reasoning path, making failures reproducible instead of mysterious. Java Span span = tracer.spanBuilder("agent-execution").startSpan(); span.setAttribute("workflow.id", workflowId); span.setAttribute("agent.name", "SecurityReviewer"); span.setAttribute("tokens.input", 1350); span.setAttribute("tokens.output", 512); worker.execute(task); span.end(); Observability should extend beyond infrastructure metrics. Enterprises benefit from tracking reasoning iterations, tool invocation frequency, retrieval latency, hallucination rates, retry counts, and token consumption. These operational metrics quickly reveal inefficient prompts, unreliable tools, or expensive reasoning loops before they impact production workloads. Testing also changes significantly. Traditional unit tests validate deterministic functions, whereas AI agents produce probabilistic outputs. Instead of asserting exact responses, enterprise pipelines evaluate workflows against acceptance criteria such as schema validation, factual correctness, safety policies, latency budgets, and execution cost. Regression suites should replay representative business workflows after every prompt, model, or orchestration change to ensure quality remains stable despite model updates. Security becomes increasingly important as agents gain permission to execute external actions. Every tool invocation should follow least-privilege principles, while generated code executes only inside isolated containers or sandboxes. Human approval remains essential for high-impact operations such as financial transactions, infrastructure changes, or customer-facing decisions. Durable workflow engines make this straightforward by pausing execution until approval arrives rather than blocking application threads. Standardizing Integrations The emergence of Model Context Protocol (MCP) further standardizes enterprise integrations. Instead of creating custom connectors for every application, MCP exposes databases, repositories, APIs, and enterprise tools through a consistent interface that any compliant agent can consume. Combined with Kafka-based messaging and workflow engines such as Temporal, MCP enables independently developed agents to cooperate without tightly coupling business logic to individual AI models. Despite growing enthusiasm, multi-agent architectures should not become the default solution. Many business problems remain better served by a single agent with carefully selected tools. Every additional agent increases latency, infrastructure complexity, operational cost, and potential failure points. Multi-agent systems become valuable only when tasks naturally decompose into specialized responsibilities requiring parallel execution, independent security boundaries, or domain-specific reasoning. Successful production deployments therefore resemble distributed systems more than prompt engineering experiments. Planner-worker orchestration, durable workflow persistence, event-driven communication, standardized protocols, resilient execution, comprehensive observability, and continuous evaluation collectively determine whether an AI system scales beyond demonstrations. A Final Word Multi-agent software engineering represents an important architectural evolution rather than simply a larger collection of language models. Organizations that approach agent collaboration with the same engineering discipline applied to microservices, distributed messaging, and cloud-native platforms will build systems capable of remaining reliable under production workloads. Those that treat agent orchestration as little more than chained prompts will likely encounter escalating costs, inconsistent behavior, and operational instability long before realizing the expected productivity gains.

By Uthej Mopathi CORE

Why Most Platforms Fail to Become Products Many companies are heavily investing in internal developer platforms (IDPs) with the expectation that they will speed up delivery and governance, and increase developer productivity. Despite significant investment in Kubernetes, CI/CD, observability, security tooling, and cloud infrastructure, many platforms struggle to gain adoption. The reason is simple: they are built and operated like infrastructure projects, not products. Infrastructure teams are often very focused on technical excellence: automation, scalability, reliability, and compliance. Developers, on the other hand, are interested in a different goal — getting their applications into production quickly and safely without having to go through so much complexity. IDP is successful when developers choose it voluntarily because it makes their lives easier. That shift requires platform architects to think less like infrastructure engineers and more like product managers. Building an IDP is like operating an airport. Nobody travels because they love airports. They travel because they want to reach a destination efficiently. Similarly, developers do not care about Kubernetes clusters, pipelines, secrets management, or observability stacks. They care about shipping features to customers. The platform's job is to make the journey smooth, fast, and safe. This article explores the core practices that differentiate successful product-centric platforms from infrastructure-centric ones. Practice 1: Start With Developer Journeys, Not Technology Choices Imagine constructing a shopping mall by selecting elevators, security systems, and air-conditioning units before understanding customer traffic patterns. The result is often technically impressive but operationally frustrating. The same happens with developer platforms. Architects should first map the customer journey (developer journey) before designing platform capabilities. Many platform initiatives begin with questions like: Which Kubernetes distribution should we use?Which GitOps framework is best?Which CI/CD tool should be standardized? These are important questions, but they should not be the starting point. Successful platform architects begin by understanding developer workflows: How does a new service get created?How long does environment provisioning take?Where do deployment delays occur?What causes support tickets?Which activities are repetitive and manual? The goal is to identify friction and eliminate it. Organizations using platforms based on technologies like Red Hat OpenShift, IBM Cloud Kubernetes Service, or other cloud-native platforms have found that developers adopt only when the platform team focuses on reducing the friction in workflow rather than adding more infrastructure features to the platform. Practice 2: Create Golden Paths, Not Golden Handcuffs A highway encourages drivers to use the fastest route while still allowing exits when necessary. Successful IDPs behave like highways. Developers naturally choose the Golden Path because it is easier and safer than building everything from scratch. One of the most powerful concepts in modern platform engineering is the Golden Path. A Golden Path provides: Recommended architecturesStandard deployment patternsPre-approved security controlsBuilt-in observabilityAutomated CI/CD workflows Developers should be able to move fast along a paved road while retaining flexibility for unique requirements. Platform teams that leverage services from cloud provider environments often realize that standardized self-service templates drive significantly higher adoption than restrictive governance models. Practice 3: Make Self-Service the Primary Interface Every banking transaction once required a visit to a physical branch. Today, customers expect to do everything from a mobile app. Developers hope for the same experience from inside their own software. Nothing kills developer productivity faster than dependency queues. Consider a common case of dependency queues. Open a ticket for infrastructure.Wait for approval.Wait for provisioning.Request secrets.Request monitoring.Request deployment access. Weeks can pass before development even begins. Modern platforms must provide self-service experiences where developers can do the following without opening tickets. Create environmentsProvision databasesConfigure pipelinesAccess observability dashboardsRequest infrastructure resources An IDP should function like a digital banking application—secure, streamlined, and available on demand. Below is the Product-Centric IDP reference architecture. Developers consume platform capabilities through self-service experiences, while the platform embeds security, observability, governance, and delivery capabilities and exposes them through Golden Paths. Practice 4: Treat Platform APIs as Products A power drill might have sophisticated engineering in it. Users judge it by a very simple standard: “Can I drill a hole fast and reliably?" Many platform teams are focused on infrastructure automation and not developer experience. Each API, template, workflow, and portal interaction is a product interface. Questions worth asking include: Is the API predictable?Is documentation clear?Are error messages actionable?Is onboarding intuitive?Can developers discover capabilities easily? Developers evaluate IDPs the same way. They are not interested in the complexity underneath. They care about usability. This principle is especially important when integrating observability services, cloud provisioning layers, or deployment automation platforms. For example, IBM Cloud's managed services can significantly simplify operational complexity, but value is realized only when developers experience that simplicity through intuitive platform workflows. Practice 5: Build Observability into the Platform, Not Around It Imagine when you are driving a car without any speedometer, fuel gauge or warning indicators. You may still reach your destination but the risk increases dramatically. Observability is the dashboard for software systems. Observability is often treated as an afterthought. A team deploys an application and later attempts to add: MetricsLogsTracesDashboardsAlerting This approach creates inconsistency and operational blind spots. Platform teams should embed observability from day one. Every service created through the platform should automatically include: Logging standardsDistributed tracingMetrics collectionHealth monitoringService dashboards Whether organizations use IBM Cloud Observability, Instana, OpenTelemetry, Prometheus, Grafana, or other solutions, the platform should make observability automatic rather than optional. Practice 6: Make Security Invisible but Ubiquitous When entering a modern office building, people rarely think about security. Access badges, surveillance, and emergency controls are built into the environment — the building is secure without requiring employees to become security experts. The same principle applies to IDPs. In immature environments, security is seen as a series of checkpoints, review meetings, manual compliance approvals, vulnerability assessments, and audit evidence collection. Developers find it as friction because it arrives late in the delivery lifecycle. Traditional security models operate as gates. Platform-centric security operates as guardrails. The objective is not fewer security controls — it is fewer manual interactions. Build Secure-by-Default Golden Paths Every new service created through the platform should automatically inherit: Secure CI/CD pipelines with dependency and container image scanningSecret detection and policy enforcementAccess control standards and audit loggingEncryption best practices Automate Policy Enforcement Manual compliance verification is one of the biggest sources of deployment delays. Platform teams should adopt policy-as-code (PaC) approaches that automatically validate deployment configurations, infrastructure standards, and regulatory controls. Instead of asking, "Did someone review this configuration?" the platform asks, "Does this configuration satisfy our policies?" Reduce Security Cognitive Load Developers should not need deep expertise in every security domain. The platform should abstract identity management, secrets management, certificate management, and vulnerability remediation workflows—particularly in hybrid and multi-cloud environments where security complexity grows rapidly. A useful measure of progress: the percentage of security controls inherited from the platform versus manually implemented by application teams. The higher the inheritance rate, the lower the cognitive load. Practice 7: Measure Platform Success Like a Product A gym owner does not measure success by counting treadmills—they measure it by member outcomes. Platform teams should apply the same logic. Traditional infrastructure metrics like cluster utilization, pipeline counts, and resource consumption tell you whether the platform is running. They do not tell you whether it is working for developers. Product-oriented platform teams focus on: Developer satisfactionPlatform adoptionTime to first deploymentDeployment frequencyLead time for changes If developers still circumvent the platform, no amount of technical sophistication matters. The Developer Experience Scorecard Measuring developer experience requires balancing sentiment, effort, and adoption. High-performing platform teams track four key measures: Metric What It Measures How to Collect Developer Satisfaction Score (DSS) Overall platform sentiment Quarterly survey, 1–10 scale Platform NPS Willingness to recommend the platform "How likely are you to recommend this platform?" scored 0–10 Ease-of-Use Score How intuitive common workflows feel Per-task rating, 1–5 scale Developer Effort Score How much work is required to achieve an outcome Survey question on effort per task Together, these reveal not just whether developers are using the platform but whether they genuinely value it. Satisfaction Is a Leading Indicator Most delivery metrics lag behind—deployment frequency (e.g., lead time, incident count) and other metrics. Developer satisfaction is a leading indicator. Developers discover friction long before it is observable from the data. A declining DSS today will result in a decline in productivity and adoption tomorrow. Listening early allows platform teams to respond before problems grow into organizational challenges. The real measure of success is not how many developers use the platform—it is how they feel while using it. The IDP Health Dashboard High-performing platform teams monitor a balanced set of metrics across four categories: Category Metrics Sentiment DSS, Platform NPS, Ease-of-Use ratings Adoption Golden Path adoption, self-service usage, onboarding rates Friction Support ticket volume, documentation search failures, manual approval requests Productivity Time to First Deployment (TTFD), environment provisioning time, lead time for changes A platform succeeds not when developers are forced to use it, but when they prefer to use it. Practice 8: Reduce Cognitive Load Relentlessly The automotive industry spent decades simplifying the driving experience so drivers could focus on reaching their destination rather than understanding the mechanics of their vehicles. IDPs should do the same. As organizations evolve into cloud-native architectures, developers are expected to navigate containers, Kubernetes, CI/CD, IaC, security policies, service meshes, observability tools, and compliance requirements all at once. Each one solves a very important problem individually. As a whole, they overwhelm developers and take focus away from developing business capabilities. A successful platform is not one that exposes every infrastructure capability. It is one that hides unnecessary complexity while providing simple, intuitive paths to outcomes. The goal of platform engineering is not to eliminate complexity. It is to absorb complexity so developers don't have to. Common indicators of excessive cognitive load: Developers struggling to find documentationFrequent support requests for routine tasksLong onboarding times for new servicesMultiple handoffs between teamsTool sprawl across the engineering ecosystem Reduce Tool Sprawl Every tool a developer must learn introduces new interfaces, terminology, documentation, and configuration models. Platform teams should create a unified experience through a developer portal, service catalog, or platform API, that minimizes the number of decisions and interfaces developers encounter. Minimize Context Switching Every transition between tools, teams, or approval processes introduces cognitive overhead. Platform teams should ask: Can this be automated? Can these steps be consolidated? Can approvals be replaced with automated guardrails? The goal is fewer interruptions between code creation and deployment. Platform Teams Are Complexity Brokers Complexity never disappears — it moves. Organizations can either push complexity onto every development team, or centralize and manage it within the platform. High-performing platform teams choose the latter, absorbing operational, security, infrastructure, and compliance complexity so application teams can focus on features. Practice 9: Obsess Over Time to First Deployment The first experience developers have with a platform often determines whether they embrace it or avoid it. Imagine a shopping mall where opening a new store requires twelve forms, multiple approval queues, and manual setup of every utility. Store owners would go elsewhere. The best malls provide ready-made spaces where businesses can start operating almost immediately. Developer platforms should do the same. High-performing platform teams focus relentlessly on Time to First Deployment (TTFD) — the time between creating a service and successfully deploying it. The Biggest Contributors to Poor TTFD Bottleneck Root Cause Fix Manual infrastructure provisioning Ticket-driven approval chains Self-service IaC, service catalogs, platform portals CI/CD pipelines built from scratch No standard templates Pre-built, reusable pipeline templates Security reviews at the end Late-stage compliance gates Shift left — embed scans and policy checks in Golden Paths Observability setup delays Manual metrics/dashboard configuration Auto-provision logging, tracing, and health checks by default Too many decisions Choice overload at onboarding Provide Golden Paths with sensible defaults Measure Every Stage Stage Target Service creation < 5 mins Repository creation Automated Pipeline creation Automated Infrastructure provisioning < 10 mins First build < 5 mins First deployment < 15 mins Observability enablement Automatic TTFD = Provisioning Time + Setup Time + Approval Time + Deployment Time Many organizations discover that approval time is larger than all technical activities combined. The fastest platforms replace approvals with automated guardrails. Practice 10: Build a Platform Community, Not Just a Platform Team Cities flourish when residents contribute feedback and shape growth. Cities planned entirely from a central authority often struggle to meet citizen needs. IDPs are no different. The best platforms evolve through continuous collaboration. Platform teams should create feedback loops through office hours, community forums, developer councils, internal documentation reviews, and experience surveys. Developers become co-creators rather than consumers. Community Health Metrics Running community mechanisms is not enough — each one needs a way to know whether it is working. Track these six indicators to measure community health: Metric What It Measures Healthy Signal Monthly Active Community Members Developers engaging in forums, channels, or office hours Steady growth quarter over quarter Developer-to-Developer Answer Rate % of forum questions answered by non-platform-team members Above 40% indicates a self-sustaining community External Contributions per Quarter Pull requests or documentation edits from application teams Increasing trend Roadmap Items from Community Input % of platform backlog items originating from developer feedback Above 50% signals product-centric culture Office Hours Repeat Attendance Rate % of attendees who return across multiple sessions Above 60% indicates ongoing value Support Ticket Deflection Rate % of issues resolved via community before a ticket is opened Rising deflection reduces platform team toil The ultimate sign of a mature platform community is a change in how developers talk about the platform—from something that happens to them to something they help shape. Practice 11: Think in Products, Roadmaps, and Customer Value Smartphones succeeded because manufacturers continuously improved user experience. Customers did not buy phones because of processor specifications. They bought outcomes—better communication, productivity, and convenience. Developers adopt platforms for the same reason. The strongest indicator that a platform is becoming a product is a change in language. Instead of asking: What infrastructure should we standardize? Platform teams begin asking: What developer problems should we solve next? Which user journeys create the most friction?Which capabilities deliver the highest value?What does our product roadmap look like? Features matter only when they improve the developer experience. Practice 12: Design for Platform Reliability, not Just Application Reliability Imagine a city that invests heavily in building roads, bridges, and public transport for its citizens, but has no maintenance crew, no traffic monitoring, and no plan for when a bridge closes. The infrastructure exists, but without reliability commitments, citizens cannot depend on it. Internal developer platforms face exactly the same risk. Most platform engineering conversations focus on the reliability of applications running on the platform — uptime, error rates, latency SLOs for customer-facing services. What is rarely discussed is the reliability of the platform itself. Yet the platform is load-bearing infrastructure for every engineering team in the organisation. When the CI/CD pipeline degrades, every team's delivery stops. When the service catalog is unavailable, no new services can be provisioned. The platform's reliability is a multiplier — a single failure can simultaneously impact dozens of teams. Define Platform SLOs Before Developers Define Them for You Platform teams that do not define their own Service Level Objectives will find that developers define them informally — through frustration, workarounds, and loss of trust. Effective platform SLOs cover the experiences developers depend on most: Pipeline availability — what percentage of CI/CD pipeline executions succeed without infrastructure-related failures?Provisioning latency — how long does environment or resource provisioning take at the 95th percentile?Portal availability — is the developer portal and service catalog accessible during working hours?Golden Path build time — how long does a standard pipeline template take to complete? These are the experience metrics developers encounter every day. A platform team that publishes and tracks these SLOs operates as a reliable internal service provider. A team that does not is invisible until something breaks. IDP Maturity Model Stage Characteristics Infrastructure Platform Standardized infrastructure, clusters, CI/CD tooling Self-Service Platform Service catalogs, automation, infrastructure on demand Developer Platform Golden Paths, integrated observability and security, DevEx focus Platform Product Platform roadmaps, adoption metrics, developer satisfaction measurement Adaptive Platform Continuous feedback loops, AI-assisted operations, continuous platform evolution Most organizations do not start with a Platform Product. They evolve toward it. The goal of the maturity model is not to reach the highest stage overnight, but to identify the next set of capabilities that will improve developer experience and platform adoption. High-performing platform teams treat platform maturity as a journey rather than a destination. Assessing Your Current Stage To identify where your platform currently sits, ask three diagnostic questions: How do developers access platform capabilities today? If the answer is "by opening a ticket," the platform is at the infrastructure stage. If developers provision resources on demand without human approval, they are at the self-service stage or beyond.Do developers choose the platform voluntarily or use it because they must? Voluntary adoption driven by speed and simplicity signals a developer platform or platform product. Mandatory usage with frequent workarounds signals an earlier stage.Does the platform team maintain a product roadmap prioritized by developer feedback? A yes here is the clearest indicator of a platform product. The absence of a roadmap almost always reflects an infrastructure or self-service mindset. Moving to the Next Stage Each stage has a single dominant unlock that drives progression: Infrastructure → Self-Service: Replace ticket-driven provisioning with self-service automation and a service catalog.Self-Service → Developer Platform: Introduce Golden Paths that embed security, observability, and CI/CD by default.Developer Platform → Platform Product: Establish a formal platform roadmap, measure developer satisfaction (DSS, NPS), and treat developer feedback as a product backlog.Platform Product → Adaptive Platform: Build continuous feedback loops, introduce AI-assisted operations, and invest in platform telemetry that proactively surfaces friction before developers report it. The most common mistake is attempting to skip stages. Teams that build Golden Paths before self-service exists create well-designed paths nobody can access independently. Teams that adopt satisfaction metrics before Golden Paths exist measure friction without the tools to address it. Progress through the stages in order. The IDP Architect's Checklist Before launching any new platform capability, ask: ✅ Does this feature remove friction from a developer workflow? ✅ Can developers access it through self-service? ✅ Is it aligned with a Golden Path? ✅ Is observability included by default? ✅ Is security built into the platform? ✅ Is governance automated rather than manual? ✅ Can success be measured through developer outcomes? ✅ Does it reduce cognitive load? ✅ Does it improve Time to First Deployment? ✅ Would developers choose this platform if they had alternatives? If the answer to several of these questions is "no," the capability is probably infrastructure-focused rather than product-focused. Final Thoughts The future of platform engineering is not about building more infrastructure. It is about delivering better developer experiences. The most successful IDPs combine the discipline of site reliability engineering (SRE), the automation of cloud-native technologies, and the mindset of product management. Whether your foundation runs on IBM Cloud, OpenShift, hyperscaler cloud services, or a hybrid environment, the winning formula remains the same: Treat developers as customers. Treat the platform as a product. Treat developer productivity as the ultimate business metric. When platform architects embrace this mindset, platforms stop being collections of tools and start becoming accelerators of innovation—and that's when platforms truly become products.

By Josephine Eskaline Joyce, Ph.D CORE

Cross-border e-commerce sellers often spend hours comparing the same products across different Amazon marketplaces. Prices, reviews, and seller signals vary by country, but the process is still largely manual. I wanted to see how far I could automate it with a small AI agent built using Codex, SerpApi, and Lark. Suppose I ask in Codex: Compare Grogu products in the US and Japan The workflow is straightforward: Search Amazon marketplacesFetch product detailsSummarize seller signalsSend a report to Lark I kept the stack intentionally simple: Python 3.12uvPydanticSerpApi Python SDKOpenAI structured outputsLark custom bot webhookCodex for both development and as the conversational interface Running the Agent From Codex Once everything was wired together, I could simply ask Codex: Compare Grogu products in the US and Japan Codex understood the request, triggered the CLI workflow, collected marketplace data via SerpApi, and delivered a structured report to Lark. Here’s the entire flow in action: The Workflow The overall flow looks like this: Starting with a natural-language request in Codex, the agent first validates that the question is related to cross-border Amazon research. OpenAI then translates the request into a structured command. SerpApi handles both product discovery and detailed product retrieval, while OpenAI extracts seller-focused insights from the collected data. Finally, the results are packaged into a Lark card and delivered via a Custom Bot webhook. I intentionally did not start with FastAPI, background jobs, or a database. For the MVP, the important question was simpler: Can I go from a natural-language product question to a useful cross-border product card? Translating Natural Language into Commands The natural-language entry point uses OpenAI to translate user requests into structured commands. For example: Compare Grogu products in the US and Japan becomes: YAML query="Grogu" marketplaces=["us","jp"] output_mode="send_lark" Using a strict schema keeps the pipeline predictable and much easier to debug. Instead of letting the model orchestrate everything, I only ask it to generate structured commands. Searching Amazon With SerpApi This project relies on two SerpApi endpoints. The Amazon Search API is used to discover candidate products, while the Amazon Product API enriches them with much richer details. Search results provide ranking context and thumbnails, while product pages contain detailed information such as images, availability, and descriptions. Combining both produced much better product cards than using either endpoint alone. Modeling Product Data Amazon pages are messy. Some products have ratings but no availability. Some have images but no variants. Some fields simply don't exist. Missing data is normal, not an exception. My first instinct was to say, "Why not model most fields as optional?" That is true, but it is not the whole solution.The real issue was that SerpApi returns useful product data from several different places. Some fields are flat. Some fields are nested. Some fields have different names depending on whether they came from Amazon Search API or Amazon Product API. Some products should not be shown at all if they are missing the signals a seller actually needs. To make the data usable, I: Normalized both endpoints into a common Product model;Filtered out products with weak seller signals;Merged Product API details back into search results;Treated missing fields as expected rather than failures. The Product model still uses optional fields, because missing data is normal: But optional fields alone were not enough. I also filtered search results before choosing products for detail lookup: This was important for the Lark card. A cross-border seller does not want a table full of Rating: N/A and Reviews: N/A. Those rows make the card noisy and less actionable. The next issue was nested and inconsistent JSON. For price, SerpApi may return a string, a number-like value, or a nested object: For availability, the Product API does not always use one stable field. I had to check availability, stock, and sometimes delivery: Keeping the Agent Narrow The most interesting part of the project isn't Grogu. It's scope. The entry point only supports cross-border Amazon product research. If somebody asks: What is today's weather? the app simply rejects the request. Requests outside the project scope are rejected locally before calling OpenAI. For supported requests, OpenAI returns a strict command object: This keeps the agent predictable. It translates requests into commands instead of improvising actions. I deliberately avoided turning this into a general chatbot. The agent only knows one workflow: cross-border Amazon product research. That narrow scope makes the behavior easier to explain, test, and trust. It also keeps OpenAI API responsible for translation rather than improvisation. Generating Seller-Friendly Insights and Delivering Them to Lark Once product data has been collected, OpenAI generates seller-focused insights. I intentionally constrain the model to use only information returned by the APIs, avoiding hallucinated prices, ratings, or availability. Instead of producing generic summaries, the analysis focuses on demand signals, social proof, pricing, and obvious risks — information that is much more useful for sellers. The final result is delivered through a Lark Custom Bot webhook. Error Handling External APIs fail. That's normal. I treated each layer independently. If OpenAI fails, no command is generated. If SerpApi fails, the analysis stops with a clear message. If Lark delivery fails, the report can still be viewed locally. This separation keeps failures localized and prevents one component from bringing down the entire workflow. Conclusion The most interesting lesson from this project wasn’t the Grogu theme or the Lark card. It was learning where the boundaries should be. The agent works because it stays narrow. That narrowness makes the system easier to understand, debug, and trust. And if you’re building tools for cross-border e-commerce, SerpApi’s Amazon API provides a surprisingly rich source of product data. They made this entire workflow possible. If you’re working on product research, seller analytics, or marketplace intelligence, I’d recommend giving them a try. Check out the full SerpAPI article collection here.

Modern software delivery is complex. Developers are responsible not only for writing code that meets business requirements — both functional and non-functional — but also for navigating a long chain of supporting steps. From containerization, testing, configuration, security, deployment, and monitoring, each stage often relies on specialized tools and teams. When these processes aren’t standardized, every project risks reinventing the wheel. The result is inconsistency, delays, and frustration. For example, requesting a new test environment might require submitting detailed tickets to a DevOps team, slowing timelines and draining energy. As organizations scale, so does the complexity — and the pain of delivery. Platform engineering addresses these challenges by creating shared, reliable foundations. It provides self-service tools, standardized workflows i.e., golden paths, and built-in guardrails, enabling teams to focus on what matters most: writing code and shipping features. This article explores what platform engineering is, why it matters, and how it helps organizations move faster while reducing developer burnout. It also examines common challenges and how to avoid turning platforms into yet another layer of complexity. Platform Engineering Definition Platform engineering is a practice of building and maintaining an internal, self-service platform that makes it easy for development teams to build, deliver, and operate software. Key principles of platform engineering are: Self-service (with guardrails) → developers should be able to build, deploy, and operate services independently — without filing tickets for routine tasks. Also ensuring automated guardrails for compliance and cost control.Golden paths, not golden cages → provide opinionated, well-supported paths that make the right thing easy — without preventing teams from choosing alternatives when needed.Product mindset → treat the platform as a product. Define users (developers), gather feedback, measure adoption, and iterate based on value delivered.Reduce cognitive load → abstract away infrastructure and operational complexity that does not directly contribute to the developer’s core task, i.e., building and shipping business logic. It's imperative to note that Platform Engineering is not DevOps, but DevOps scaled through product thinking — treating developers as customers and the platform as the product. Adoption Journey Large organizations often face delivery challenges that rarely make it into executive summaries. Issues like developer friction, inconsistent and/or duplicate tooling, and fragmented workflows are deeply embedded in day‑to‑day operations. Their impact — delayed releases, inefficiencies, and frustration — may be visible, but the root causes often remain hidden or disconnected from leadership narratives. Thus, the first step is discovery and validation. Organizations must surface real pain points through design thinking workshops, targeted surveys, analysis of past initiatives, and continuous community/user feedback. These insights form the foundation for defining a clear and grounded Platform Mission Statement — one that aligns platform capabilities with genuine organizational needs. Once the mission is clear, enterprises move toward unified platforms that standardize common tools and processes. This consolidation reduces duplication and improves reliability. Guiding Principles should be maintained via ADRs as a standardized platform for uniformity. Also, it's recommended to have decentralized decision-making to avoid bottlenecks and maintain long-term sustainability. To achieve this, use a community-driven approach via various guilds. As maturity grows, self‑service enablement becomes the focus — developers can provision infrastructure, build & deploy applications, perform verification, and integrate monitoring with minimal friction. This can be achieved via Internal Developer Platforms (IDP) and Internal Developer Portals. Finally, mature organizations embrace continuous improvement. The platform evolves like a product — guided by developer feedback and metrics. The feedback loops should act to adapt the platform and be evolutionary in nature. Being preventive or proactive, rather than reactive, goes a long way in achieving a successful platform implementation. Internal Developer Platforms (IDP) vs. Internal Developer Portals A common source of confusion in platform engineering is the distinction between an internal developer platform (IDP) and an internal developer portal. While these concepts are related and often work in tandem, they serve distinct purposes and have different architectural and user experience implications. Internal Developer Platform (IDP) An IDP is the “engine room” of platform engineering. It is a cohesive set of tools, frameworks, and automation scripts that standardize and automate the provisioning, deployment, and management of infrastructure and services. Key components typically include: Self-service infrastructure provisioning → Developers can request and manage resources (VMs, databases, clusters) via APIs or CLI tools, eliminating ticket-based workflows.Unified deployment and orchestration → Standardized CI/CD pipelines, container orchestration (e.g., Kubernetes), and Infrastructure as Code ensure consistent, reliable releases.Centralized configuration and secrets management → Version-controlled settings, automated secret management, and policy enforcement across environments.Automated monitoring and observability → Integrated metrics, logs, and tracing provide real-time visibility into system health.Security and compliance automation → Policy-as-code frameworks (e.g., OPA) enforce security and compliance at every stage. IDPs are typically built and maintained by platform engineering teams and are consumed by developers and operations teams to accelerate software delivery and reduce operational risk. Internal Developer Portal An internal developer portal is the “front door” to the platform. It provides a user-friendly interface (often a web dashboard) that aggregates documentation, service catalogs, APIs, and organizational guidelines. Key features include: Service catalog → Centralized inventory of services, APIs, and infrastructure, supporting discoverability and ownership tracking.Integration ecosystem → Unified view of the development toolchain, integrating with version control, CI/CD, observability, and project management tools.Self-service workflows → Guided forms and wizards for routine operations (e.g., provisioning, deployments), with built-in approval workflows and RBAC.Onboarding and knowledge sharing → Centralized documentation, onboarding guides, and community Q&A features to accelerate ramp-up and collaboration.Metrics and scorecards → Dashboards tracking service health, maturity, and compliance, providing actionable insights for improvement. Portals are typically used by application developers, product teams, and managers to discover services, access documentation, and initiate self-service workflows. When to Use Each (or Both) Start with an IDP when the primary pain points are manual infrastructure provisioning, inconsistent environments, or the need for standardized automation.Start with a Portal when discoverability, onboarding, and knowledge sharing are the main challenges, or when existing tools are underutilized due to lack of visibility.Combine Both for maximum impact: the IDP provides the backend automation and guardrails, while the portal exposes these capabilities through an intuitive, developer-friendly interface Team Topologies and Platform Engineering The success of platform engineering is deeply influenced by organizational structure and team interactions. The “Team Topologies” framework provides a powerful lens for designing team structures that optimize for effective platform adoption Four Fundamental Team Types Stream-Aligned Teams → Aligned to a flow of work from a business domain (e.g., a product or service). They own the end-to-end delivery and operation of features.Platform Teams → Build and maintain internal platforms that provide reusable services and capabilities to stream-aligned teams, reducing their cognitive load.Enabling Teams → Help stream-aligned teams overcome obstacles, adopt new technologies, or fill skill gaps.Complicated Subsystem Teams → Own subsystems that require deep specialist knowledge (e.g., advanced algorithms, core infrastructure). Three Team Interaction Modes Collaboration → Teams work together for a defined period to discover new solutions.X-as-a-Service → One team provides a service that another team consumes with minimal interaction.Facilitation → One team helps another team acquire new skills or capabilities. Relevance to Platform Engineering Platform Teams as Product Teams → Platform engineering teams should operate as product teams, treating stream-aligned teams as customers, gathering feedback, and iterating on platform featuresReducing Cognitive Load → The primary goal of the platform team is to reduce the cognitive load on stream-aligned teams, enabling them to focus on delivering business value rather than infrastructure concernsClear Interfaces and Boundaries → Well-defined APIs, documentation, and support channels ensure that platform capabilities are discoverable and consumable as a service, minimizing dependencies and handoffsContinuous Adaptation → Team boundaries and responsibilities should evolve as business needs and technologies change, with feedback loops guiding organizational adjustments Conway’s Law and Its Applicability to Platform Engineering Conway’s Law, articulated by Melvin Conway in 1967, states that “organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations” In the context of platform engineering, this law has profound implications. How Conway’s Law Shapes Platform Design Organizational Silos Lead to Siloed Platforms → If engineering teams are organized in silos (e.g., by business unit or product line), the platforms they build will reflect this fragmentation, resulting in duplicated tools, inconsistent practices, and integration challengesCross-Functional Collaboration Enables Cohesive Platforms → Successful platform engineering requires cross-functional teams that span development, operations, security, and compliance, ensuring that the platform addresses the needs of all stakeholders and avoids becoming a new silo.Intentional Organizational Design → To achieve coherent, scalable platforms, organizations must deliberately design their communication structures and team interactions to support shared standards, rapid feedback, and continuous improvement. Real-World Example - Monolith to Microservices → Organizations transitioning from monolithic architectures to microservices often reorganize teams around domains or services. If team boundaries are not aligned with desired system boundaries, the resulting architecture may become fragmented or inconsistent, reflecting the underlying communication patterns rather than optimal technical design Platform Engineering as Both Solution and Symptom → Platform engineering often arises as a response to the silos and fragmentation created by previous organizational structures. However, if not implemented with a product mindset and cross-team alignment, platform engineering can inadvertently create new silos, perpetuating the very problems it seeks to solve Metrics and KPIs to Measure Platform Success Measuring the impact of platform engineering is essential for demonstrating value, securing buy-in, wider adoption, and guiding continuous improvement. Common Metrics DORA Metrics → Deployment frequency, lead time for changes, change failure rate, mean time to recovery (MTTR).SPACE Framework → Satisfaction and well-being, performance, activity, communication and collaboration, efficiency and flow.Adoption Rates → Percentage of teams and services using the platform.Time to Onboard → Time required for new developers to become productive.Operational Metrics → Incident rates, uptime, resource utilization, and cost savings.Developer Satisfaction: Surveys, Net Promoter Score (NPS), and qualitative feedback. Feedback Mechanisms Surveys and Office Hours → Regular check-ins with platform users to gather qualitative and quantitative feedback.Telemetry and Usage Analytics → Automated tracking of platform usage, feature adoption, and workflow bottlenecks. Limitations and Common Challenges of Platform Engineering Despite its benefits, platform engineering is not without challenges and limitations. Organizational Challenges Resistance to Change → Teams may be reluctant to adopt new tools or workflows, especially if they perceive a loss of autonomy or increased complexity.Alignment and Buy-In → Achieving consensus on standards, priorities, and platform direction can be difficult in large or distributed organizations.Skill Gaps → Building and maintaining platforms requires expertise in infrastructure automation, CI/CD, security, and developer experience, which may be lacking in existing teams. Technical Challenges Overengineering → Building overly complex or rigid platforms can lead to low adoption and maintenance burden.Integration Complexity → Aggregating data and workflows from diverse tools and systems requires careful planning and robust integrations.Legacy Systems and Technical Debt → Integrating with or migrating from legacy tools and architectures can be time-consuming and costly. Cultural Challenges Product Mindset → Treating the platform as a product, with continuous feedback and iteration, is essential but often overlooked.Avoiding the “Golden Cage” → Mandating platform adoption without addressing real developer needs can lead to resentment and shadow. Measurement and ROI Lack of Metrics → Many organizations fail to measure platform adoption, impact, or ROI, making it difficult to justify continued investment or guide improvements. Products Over Projects In platform engineering, the distinction between a project mindset and a product mindset is crucial. Project Mindset Focus → Deliverables, deadlines, and completion.Approach → Work is structured around a defined scope with a start and end date.Outcome → Once the project is “done” the team moves on, often with limited ongoing ownership.Risk → Platforms built this way may stagnate, as continuous improvement and user feedback loops are not prioritized. Product Mindset Focus → Long-term value, user experience, and continuous evolution.Approach → The platform is treated as a living product with ongoing investment, iteration, and support.Outcome → Teams own the platform end-to-end, ensuring it adapts to changing business and developer needs.Benefit → Encourages innovation and alignment with evolving enterprise strategy. In short, with a project mindset, tasks are done as “Deliver and finish,” while with a product mindset, it's “Deliver, own, and evolve.” Relationship Between Platform Engineering, DevOps, and SRE While platform engineering, DevOps, and site reliability engineering (SRE) share common goals, they address different layers of the software delivery lifecycle. DevOps Focus → Cultural and organizational transformation to break down silos between development and operations, emphasizing collaboration, automation, and continuous delivery.Practices → CI/CD, infrastructure as code, shared responsibility for quality and reliability. SRE Focus → Applying software engineering principles to operations, with a strong emphasis on reliability, scalability, and incident response.Practices → Service-level objectives (SLOs), error budgets, automated monitoring, and incident management. Platform Engineering Focus → Building and maintaining internal platforms that provide standardized, automated, and self-service capabilities for development teams.Practices → Product mindset, self-service, golden paths, policy-as-code, and platform-as-a-product. How They Interact Platform engineering provides the foundation on which DevOps and SRE practices can scale, standardizing workflows, embedding automation and compliance, and enabling self-service for developers and operations teams.DevOps and SRE teams collaborate with platform engineers to ensure that the platform supports reliability, scalability, and continuous improvement. Case Studies Netflix used their platform to solve developers’ challenges to manage multiple services and software, knowing which tools exist, and switching contexts between tools. Zalando leveraged their platform to unify the developer experience, promote compliance by default, and improve how the company operated over time. Carlsberg’s Gaia platform automated infrastructure provisioning, embedded compliance, and provided self-service capabilities, reducing manual work and accelerating project delivery. The platform’s success was attributed to cross-functional collaboration, a product mindset, and continuous feedback from developers eBay’s Velocity initiative (2020) boosted engineering productivity, cutting deployment times from 10 days to 1–2 and enabling same‑day mobile releases. Despite technical success, cultural resistance, outdated tech choices, and poor strategic execution prevented business growth. Conclusion Platform engineering represents a paradigm shift in how modern organizations build, deliver, and operate software at scale. By abstracting complexity, standardizing workflows, and empowering developers through self-service and automation, platform engineering accelerates delivery, improves reliability, and optimizes costs. However, its success depends on more than just technology — it requires intentional organizational design, a product mindset, continuous feedback, and a careful balance between standardization and flexibility. As the discipline matures, organizations that invest in platform engineering as a strategic capability will be best positioned to thrive in an increasingly complex and competitive digital landscape. The golden rule of platform engineering → Treat your platform as a product, your developers as customers, and adoption as a metric — not a mandate. Platform 2.0 — The AI‑Native Evolution Platform 2.0 represents the next stage of platform engineering, where artificial intelligence becomes a built‑in capability rather than an add‑on. It transforms platforms from static automation frameworks into adaptive, learning ecosystems that continuously optimize themselves. Core Principles Intelligence at every SDLC stage → AI augments design, development, testing, deployment, and operations with predictive and generative capabilities.Continuous learning → Feedback loops from telemetry and user behavior refine architecture and performance automatically.Autonomous optimization → Platforms self‑tune resources, detect anomalies, and evolve configurations without manual intervention.Human‑AI collaboration → Engineers focus on strategic design and governance while AI handles repetitive and analytical tasks. Platform 2.0 enables faster delivery, higher reliability, and smarter scalability, redefining platform engineering as an AI‑native discipline — intelligent, adaptive, and perpetually evolving. References and Further Reads Platform Engineering — WikipediaWhat is Platform EngineeringInfoQ trend report Platform Engineering as early adoptersThoughtWorks tech radar recommends adopting Platform Engineering as early as Apr/Oct 2021.Team TopologiesDZone survey where nearly half of respondents indicate using platform engineering.

By Ammar Husain CORE