DZone Spotlight

Wednesday, November 11 View All Articles »

Building a Feature Store With Neo4j and Feast

By Akmal Chaudhri

CORE

Feature stores solve a problem that every machine learning (ML) team eventually hits: the features you compute during training are different from the ones you retrieve at serving time. Feast is an open-source feature store that bridges that gap, handling the registry, materialization, and online retrieval so a model sees the same data in production as it did during training. Often, Feast tutorials pair it with a relational database or a cloud data warehouse. In this article, we'll do something different. We'll use Neo4j Desktop as the feature computation engine. Five features are ordinary aggregations that Cypher handles as easily as SQL. The sixth, a customer community_id, comes from graph algorithms: customers with similar purchase mixes are found with Graph Data Science (GDS) node similarity and then grouped with Louvain community detection. We'll hand all six features to Feast, which materializes them into a SQLite online store, serves them at inference time, and builds point-in-time correct training data. We'll also test, on held-out orders, whether the graph feature tells a model anything that a plain favorite_category column doesn't. The full source code is available on GitHub. Architecture Figure 1. Architecture As shown in Figure 1, the architecture has three layers. Neo4j Desktop handles feature computation: standard aggregations via Cypher and a graph-native community feature via the Graph Data Science (GDS) library. The results, computed at several points in time, land in a Parquet file, which acts as Feast's offline store. Feast then materializes the latest values from Parquet into SQLite for low-latency online serving. At inference time, a single get_online_features() call retrieves the full feature vector, regardless of whether a feature came from a Cypher query or a GDS algorithm. For training data, get_historical_features() reads the same offline store. The key insight is that Feast doesn't know or care how the Parquet file was produced. What We're Building We'll work with a synthetic e-commerce dataset of 100 customers, 3,000 orders, and 5 product categories (Electronics, Clothing, Home, Books, and Sports) spread over the last 90 days. Each customer belongs to a hidden segment defined by a primary and a secondary category. For 55% of their orders the category is the segment's primary one, for 30% it is the secondary one, and for the remaining 15% it is drawn uniformly from all five categories. There are five segments: Electronics (primary) and Home (secondary)Electronics and SportsClothing and BooksClothing and HomeBooks and Sports The design is deliberate. Two segments share Electronics as their primary category and two share Clothing, so a customer's single most-bought category can't tell those segments apart. Only the combination of categories can. To make the churn example meaningful, 18 customers are also made to go quiet: 8 have not ordered for more than 60 days and 10 for more than 30 days. The segment is stored on each Customer node, but only so we can check our results later. It is never used to compute any feature. We'll compute six features per customer, as of a given date: total_orders – Number of orders placed up to the as-of datetotal_spent – Total order valueavg_order_value – Mean order valuedays_since_last_order – Recency, measured from the as-of datefavorite_category – The category with the most orderscommunity_id – Graph-native feature from node similarity and Louvain community detection We compute them at three as-of dates (60 days ago, 30 days ago, and today), so that Feast can later reconstruct what was known at any point in time. Setup A virtual environment is highly recommended. For example: Shell python3 -m venv ~/feast-env source ~/feast-env/bin/activate The notebook was run against Neo4j 2026.07.0 (Enterprise) and GDS 2026.07.0. We need Neo4j Desktop running with the Graph Data Science (GDS) plugin installed. In Neo4j Desktop, go to Plugins and install the Graph Data Science Library. Credentials go in environment variables before running the Jupyter notebook: Shell export NEO4J_URI="bolt://localhost:7687" export NEO4J_USERNAME="neo4j" export NEO4J_PASSWORD="your-password" export NEO4J_DATABASE="your-database" Warning: The notebook removes all data, indexes, and constraints from the database named in NEO4J_DATABASE before loading its sample data. Use a database you are happy to wipe. We'll connect to Neo4j with the standard driver: Python driver = GraphDatabase.driver( NEO4J_URI, auth = (NEO4J_USERNAME, NEO4J_PASSWORD) ) Feature Definitions Feast needs two files: a project config and a feature view declaration. The config points to SQLite for the online store: YAML project: ecommerce_features registry: feature_repo/registry.db provider: local online_store: type: sqlite path: feature_repo/online_store.db The feature definitions declare the customer_id entity and a feature view with all six features, including community_id as an Int64: Python from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int64, String, ValueType from datetime import timedelta customer = Entity( name = "customer_id", description = "Customer identifier", value_type = ValueType.STRING ) customer_features = FeatureView( name = "customer_features", entities = [customer], ttl = timedelta(days = 90), schema = [ Field(name = "total_orders", dtype = Int64), Field(name = "total_spent", dtype = Float32), Field(name = "avg_order_value", dtype = Float32), Field(name = "days_since_last_order", dtype = Int64), Field(name = "favorite_category", dtype = String), Field(name = "community_id", dtype = Int64), ], online = True, source = FileSource( path = "data/customer_features.parquet", timestamp_field = "event_timestamp" ), ) From Feast's perspective, community_id is just another integer column. It doesn't know it took graph algorithms to produce it. Loading Data into Neo4j Rather than rows in a table, we'll model the data as a graph. Figure 2 shows the structure. Figure 2. Graph Data Model Customer and Order nodes connect via PLACED relationships. Each Order connects to a Category node via IN_CATEGORY. We don't store a direct customer-to-category relationship. Instead, the GDS projection derives one at query time, so that it can be limited to orders placed before a given date. All timestamps are timezone-aware (UTC). Cypher UNWIND $orders AS o MERGE (c:Customer {id: o.customer_id}) ON CREATE SET c.segment = o.segment MERGE (cat:Category {name: o.product_category}) CREATE (ord:Order { id: o.order_id, value: o.order_value, timestamp: datetime(o.order_timestamp), quantity: o.quantity }) MERGE (c)-[:PLACED]->(ord) MERGE (ord)-[:IN_CATEGORY]->(cat) Standard Features via Cypher The first five features come from two Cypher queries, both limited to orders placed on or before the as-of date $as_of. The main aggregation computes counts, totals, averages, and recency in a single pass: Cypher MATCH (c:Customer)-[:PLACED]->(o:Order) WHERE o.timestamp <= datetime($as_of) RETURN c.id AS customer_id, count(o) AS total_orders, sum(o.value) AS total_spent, avg(o.value) AS avg_order_value, duration.inDays(max(o.timestamp), datetime($as_of)).days AS days_since_last_order Favorite category requires a separate query, since we need to find the top category per customer and collect() the first result after ordering. Ties are broken alphabetically so that the result is deterministic: Cypher MATCH (c:Customer)-[:PLACED]->(o:Order)-[:IN_CATEGORY]->(cat:Category) WHERE o.timestamp <= datetime($as_of) WITH c.id AS customer_id, cat.name AS category, count(*) AS cnt ORDER BY customer_id, cnt DESC, category WITH customer_id, collect(category)[0] AS favorite_category RETURN customer_id, favorite_category The event_timestamp that Feast uses is the as-of date itself. In other words, the moment the features describe, rather than the time of the customer's last order. Graph-Native Feature: Community ID The sixth feature groups customers whose purchase mixes are similar. It takes four steps, all run inside Neo4j. Step 1: Project customers and categories. We use the current Cypher projection, which builds an in-memory graph with the gds.graph.project aggregation function and replaces the deprecated gds.graph.project.cypher procedure. Each customer-category relationship is weighted by the number of orders placed up to the as-of date: Cypher MATCH (c:Customer)-[:PLACED]->(o:Order)-[:IN_CATEGORY]->(cat:Category) WHERE o.timestamp <= datetime($as_of) WITH c, cat, count(o) AS order_count WITH gds.graph.project( $name, c, cat, { sourceNodeLabels: labels(c), targetNodeLabels: labels(cat), relationshipType: 'PURCHASED_IN', relationshipProperties: {order_count: order_count} } ) AS g RETURN g.graphName AS graph, g.nodeCount AS nodes, g.relationshipCount AS relationships Step 2: Find similar customers. GDS node similarity with the cosine metric, weighted by order count, scores two customers close to 1 when their category mixes point in the same direction. We keep each customer's 15 most similar customers above a cutoff of 0.8: Cypher CALL gds.nodeSimilarity.stream($name, $config) YIELD node1, node2, similarity RETURN gds.util.asNode(node1).id AS src, gds.util.asNode(node2).id AS dst, similarity Python config = { "similarityMetric": "COSINE", "relationshipWeightProperty": "order_count", "topK": SIMILARITY_TOP_K, "similarityCutoff": SIMILARITY_CUTOFF, } Step 3: Project a customer-to-customer graph from those pairs, weighted by similarity: Cypher UNWIND $pairs AS p MATCH (a:Customer {id: p.src}), (b:Customer {id: p.dst}) WITH gds.graph.project( $name, a, b, { sourceNodeLabels: labels(a), targetNodeLabels: labels(b), relationshipType: 'SIMILAR', relationshipProperties: {similarity: p.similarity} }, {undirectedRelationshipTypes: ['*']} ) AS g RETURN g.graphName AS graph, g.nodeCount AS nodes, g.relationshipCount AS relationships Step 4: Run Louvain on that graph, using the similarity scores as relationship weights. consecutiveIds gives us small, tidy community IDs: Cypher CALL gds.louvain.stream($name, $config) YIELD nodeId, communityId WITH gds.util.asNode(nodeId) AS n, communityId WHERE n:Customer RETURN n.id AS customer_id, communityId AS community_id A customer with no sufficiently similar neighbor isn't in the similarity graph, so we give each such customer a community of their own. The two settings, SIMILARITY_TOP_K (15) and SIMILARITY_CUTOFF (0.8), are the only knobs, and we didn't tune them. Why not run Louvain directly on the customer-category graph and skip the similarity step? We tried, and the next section shows why we didn't keep it. Which Grouping Recovers the Hidden Segments? We compared four ways of grouping the same customers against the hidden segments, using the Adjusted Rand Index (ARI), where 1.0 means identical groupings and values near 0 mean no better than chance: Plain Text Grouping Groups ARI vs hidden segments ---------------------------------------------------- -------- ------------------------ 1. favorite_category 5 0.52 2. Louvain, customer-category graph, unweighted 2 0.08 3. Louvain, customer-category graph, weighted 3 0.41 4. Louvain, customer-similarity graph (community_id) 5 0.98 A plain favorite_category scores 0.52. It can't separate segments that share a primary category and the cross-tabulation shows exactly that: Electronics lumps together 21 customers from one segment and 22 from another and Clothing does the same with 20 and 15. Plain Text favorite_category vs hidden segment (latest snapshot): segment Books + Sports Clothing + Books Clothing + Home Electronics + Home Electronics + Sports favorite_category Books 14 2 0 0 0 Clothing 0 20 15 0 0 Electronics 0 0 0 21 22 Home 0 0 0 2 0 Sports 2 0 0 0 2 Louvain run directly on the customer-category graph does worse than that, with an ARI of 0.41 when weighted by order count and 0.08 when unweighted. We didn't investigate why, but the likely cause is that two segments sharing a primary category both attach to the same category node, which pulls them together. Without weights, every customer is linked to every category equally, and there is little left to separate them. The similarity-graph method scores 0.98 and finds five communities that line up with the five segments: Plain Text community_id vs hidden segment (latest snapshot): segment Books + Sports Clothing + Books Clothing + Home Electronics + Home Electronics + Sports community_id 0 0 0 0 23 0 1 0 1 15 0 0 2 0 21 0 0 0 3 0 0 0 0 24 4 16 0 0 0 0 Figure 3 shows the five communities, each labeled with the segment that makes up most of it. Figure 3. Communities Found by Node Similarity and Louvain. Only one customer lands in the wrong community. The algorithm was never given the segments, so it recovered them from purchase behavior alone. Does the Community Feature Add Predictive Signal? Recovering hidden segments is reassuring, but a feature earns its place by helping predictions. We test that without training a model, using only the features as they were 30 days ago. For each group of customers, we estimate a mix of product categories from orders up to that date, then score how well that mix predicts the category of every order placed afterward. The score is log-loss, the average of the negative log of the probability given to the category actually bought, so lower is better. There is no leakage: both favorite_category and community_id were computed from orders placed before the cutoff. Python def heldout_losses(orders, snapshot, cutoff_ts, group_col, alpha = 1.0): """How well does a grouping of customers predict the category of FUTURE orders? For each group, estimate the category mix from orders up to `cutoff_ts` (with add-`alpha` smoothing). Then score each later order by the negative log of the probability that mix gave its category. Lower is better. Returns one loss per held-out order.""" groups = snapshot[["customer_id", group_col]].rename(columns = {group_col: "group"}) train = orders[orders["ts"] <= cutoff_ts].merge(groups, on = "customer_id") test = orders[orders["ts"] > cutoff_ts].merge(groups, on = "customer_id") counts = (train.groupby(["group", "category"]).size().unstack(fill_value = 0) .reindex(columns = CATEGORIES, fill_value = 0) + alpha) probs = counts.div(counts.sum(axis = 1), axis = 0) idx = pd.MultiIndex.from_arrays([test["group"], test["category"]]) p = probs.stack().reindex(idx).fillna(1 / len(CATEGORIES)).values return -np.log(p) We compare no grouping at all, favorite_category, community_id and the hidden segment as an upper bound that no real feature could beat. A paired bootstrap gives a confidence interval for the improvement of community_id over favorite_category: Python diff = losses["favorite_category"] - losses["community_id"] rng = np.random.default_rng(SEED) boot = np.array([rng.choice(diff, size = len(diff), replace = True).mean() for _ in range(2000)]) low, high = np.percentile(boot, [2.5, 97.5]) Plain Text Held-out orders (placed after the 30-day snapshot): 827 Grouping Groups Held-out log-loss (lower is better) ------------------------------------- -------- ------------------------------------- No grouping (everyone shares one mix) 1 1.565 favorite_category 5 1.171 community_id 5 1.025 Hidden segment (oracle upper bound) 5 1.009 Improvement of community_id over favorite_category: 0.146 (95% CI 0.110 to 0.180) Grouping customers by favorite_category cuts the loss from 1.565 to 1.171. Grouping by community_id cuts it to 1.025, close to the 1.009 of the hidden segments. The improvement over favorite_category is 0.146, with a 95% confidence interval of 0.110 to 0.180. The reason is intuitive: knowing a customer's secondary category tells you what they are likely to buy after their favorite, and the community captures that while a single favorite category can't. Some caveats apply. The data are synthetic and designed so the combination of categories matters, so the size of the gain reflects that design, not what you would see in real orders. The predictor is a simple count-based one rather than a trained model. The bootstrap resamples individual orders and ignores that orders from one customer are related, so the interval is somewhat optimistic. And this is one dataset from one random seed. It is also worth saying what happened before we changed the data. In an earlier version of this example, every customer had a single preferred category. There, community_id and favorite_category agreed exactly (ARI 1.00), so the graph feature added nothing. The gain here appears when group membership depends on a combination of categories, which is the situation where a single aggregate column falls short. Snapshots and Stable Community IDs To support point-in-time retrieval, we compute all six features at three as-of dates: 60 days ago, 30 days ago, and today, and stack the results. Louvain runs independently for each date, and that creates a problem. Community IDs are arbitrary labels, and a label such as 3 in one run need not mean the same group in the next. A model trained on one snapshot would then be served IDs with a different meaning. We solve this by relabeling each snapshot to match the previous one, assigning each new community the old ID it overlaps with most: Python def align_communities(current, reference): """Relabel `current` community IDs to match `reference` by greatest member overlap.""" merged = current[["customer_id", "community_id"]].merge( reference[["customer_id", "community_id"]], on = "customer_id", suffixes = ("_cur", "_ref") ) overlap = (merged.groupby(["community_id_cur", "community_id_ref"]) .size().reset_index(name = "n") .sort_values("n", ascending = False)) mapping, used = {}, set() for _, r in overlap.iterrows(): cur, ref = int(r["community_id_cur"]), int(r["community_id_ref"]) if cur not in mapping and ref not in used: mapping[cur] = ref used.add(ref) # Any community with no counterpart in the reference gets a fresh ID next_id = int(max(reference["community_id"].max(), max(mapping.values(), default = -1))) + 1 for cur in current["community_id"].unique(): if int(cur) not in mapping: mapping[int(cur)] = next_id next_id += 1 aligned = current.copy() aligned["community_id"] = aligned["community_id"].map(lambda x: mapping[int(x)]).astype("int64") return aligned Python for days in SNAPSHOT_OFFSETS_DAYS: as_of = now - timedelta(days = days) df, stats = compute_features_as_of(as_of) if reference is not None: df = align_communities(df, reference) reference = df[["customer_id", "community_id"]] frames.append(df) features_df = pd.concat(frames, ignore_index = True) The notebook prints how well each snapshot recovers the segments and how much the labels would have disagreed without alignment: Plain Text As of 2026-08-27 (60 days ago): 100 customers, 844 similar pairs, 5 communities, ARI vs hidden segments 0.95 As of 2026-09-26 (30 days ago): 100 customers, 847 similar pairs, 5 communities, ARI vs hidden segments 0.95 label agreement with previous snapshot: raw 65% -> aligned 96% As of 2026-10-26 ( 0 days ago): 100 customers, 856 similar pairs, 5 communities, ARI vs hidden segments 0.98 label agreement with previous snapshot: raw 63% -> aligned 99% Each snapshot finds five communities and recovers the hidden segments well (ARI 0.95, 0.95 and 0.98), even the earliest one, which has far fewer orders per customer. Without alignment, only 65% and 63% of customers kept their raw label from one run to the next. After alignment, 96% and 99% did. The remainder are customers whose community changed as more orders arrived. Because community_id is an arbitrary label, a model should treat it as a categorical value (one-hot or embedding), never as an ordered number. Merging Features and Writing to Parquet The three snapshots are written to a single Parquet file, one row per customer per as-of date: Python features_df.to_parquet("feature_repo/data/customer_features.parquet", index = False) Plain Text Wrote 300 rows to Parquet. event_timestamp 2026-08-27 18:02:57.920521+00:00 100 2026-09-26 18:02:57.920521+00:00 100 2026-10-26 18:02:57.920521+00:00 100 From here, Feast takes over. It doesn't know that community_id came from graph algorithms and it's just another column in the Parquet file. Feast Apply and Materialize We'll apply the feature definitions and materialize into the SQLite online store: Python result = subprocess.run( ["feast", "-c", "feature_repo", "apply"], capture_output = True, text = True ) end_date = datetime.now(timezone.utc) start_date = end_date - timedelta(days = 90) result = subprocess.run( ["feast", "-c", "feature_repo", "materialize", start_date.isoformat(), end_date.isoformat()], capture_output = True, text = True ) feast apply registers the entity and feature view. feast materialize reads the Parquet file and loads the most recent value for each customer into SQLite, ready for low-latency retrieval. SQLite and Parquet are convenient for a local demo; in production you would swap in stores built for scale and concurrency. Feast keeps that largely a configuration change rather than a change to the feature code. Online Feature Retrieval We'll reinitialize the feature store and retrieve features for three customers, one active, one cooling, and one churned: Python fs = FeatureStore(repo_path = "feature_repo") feature_vector = fs.get_online_features( features = [ "customer_features:total_orders", "customer_features:total_spent", "customer_features:avg_order_value", "customer_features:days_since_last_order", "customer_features:favorite_category", "customer_features:community_id", ], entity_rows = [{"customer_id": cid} for cid in test_customers] ) The retrieved features look like this: Plain Text customer_id favorite_category total_spent total_orders days_since_last_order community_id avg_order_value ------------- ------------------- ------------- -------------- ----------------------- -------------- ----------------- CUST_0081 Books 7765.10 28 5 4 277.33 CUST_0007 Electronics 7000.73 26 35 0 269.26 CUST_0032 Clothing 6580.14 26 66 2 253.08 community_id appears alongside the standard aggregations, served identically through the same Feast interface. Using Features for Prediction A simple rule-based churn risk function shows how the features feed into inference. Recency drives the risk tier, and community_id is returned with the result so that downstream systems can segment or report by community: Python def predict_customer_churn(customer_id): """Use features for real-time churn prediction.""" features = fs.get_online_features( features = [ "customer_features:total_orders", "customer_features:total_spent", "customer_features:days_since_last_order", "customer_features:community_id", ], entity_rows = [{"customer_id": customer_id}] ).to_df() days_since = features["days_since_last_order"].iloc[0] community = features["community_id"].iloc[0] if days_since > 60: risk, action = "HIGH", "Send 20% discount" elif days_since > 30: risk, action = "MEDIUM", "Send personalized email" else: risk, action = "LOW", "Continue engagement" return { "customer_id": customer_id, "days_since_last_order": int(days_since), "churn_risk": risk, "action": action, "community_id": int(community), } predictions = [predict_customer_churn(c) for c in test_customers] print(tabulate(predictions, headers = "keys", tablefmt = "simple")) Plain Text customer_id days_since_last_order churn_risk action community_id ------------- ----------------------- ------------ ----------------------- -------------- CUST_0081 5 LOW Continue engagement 4 CUST_0007 35 MEDIUM Send personalized email 0 CUST_0032 66 HIGH Send 20% discount 2 The rule itself doesn't use community_id; the point is that serving is identical whichever way a feature was computed. One caveat applies to days_since_last_order: it was calculated when we materialized, so served a week later without re-materializing it would be a week out of date. In production, re-run materialization on a schedule, or store the last-order timestamp and compute recency at request time, for example with a Feast on-demand feature view. Point-in-Time Historical Features The other half of a feature store's job is building training data in which each row sees only what was known at that row's timestamp. We ask Feast for one customer's features at three timestamps: Python entity_df = pd.DataFrame({ "customer_id": [target] * 3, "event_timestamp": [ pd.Timestamp(now - timedelta(days = 45)), pd.Timestamp(now - timedelta(days = 20)), pd.Timestamp(datetime.now(timezone.utc)), ], }) historical_df = fs.get_historical_features( entity_df = entity_df, features = [ "customer_features:total_orders", "customer_features:total_spent", "customer_features:avg_order_value", "customer_features:days_since_last_order", "customer_features:favorite_category", "customer_features:community_id", ], ).to_df() Plain Text customer_id event_timestamp total_orders total_spent avg_order_value days_since_last_order favorite_category community_id ------------- -------------------------------- -------------- ------------- ----------------- ----------------------- ------------------- -------------- CUST_0081 2026-09-11 18:02:57.920521+00:00 12 3609.80 300.82 2 Books 4 CUST_0081 2026-10-06 18:02:57.920521+00:00 20 6086.00 304.30 6 Books 4 CUST_0081 2026-10-26 18:03:04.843967+00:00 28 7765.10 277.33 5 Books 4 Feast joins each timestamp to the most recent snapshot at or before it, so the customer's order count grows from 12 to 20 to 28 as time moves forward. Every snapshot, including its community_id, was computed using only orders placed up to its own date, so none of these rows contain information from the future. The event_timestamp shown is the one we asked about, not the snapshot's own date. Gotchas and Lessons Learned A few things worth knowing before you run the notebook. Dependency conflicts. To avoid incompatibilities between Feast and the graphdatascience Python client, the solution is to drop graphdatascience entirely and call GDS via raw Cypher through the Neo4j driver. Every GDS procedure used here (gds.graph.project, gds.nodeSimilarity.stream, gds.louvain.stream, gds.graph.drop) is callable this way and the result is a cleaner install. GDS notifications. GDS warns that the schema field returned by gds.graph.drop is deprecated. The warning is harmless, but it repeats every time the procedure is called. Rather than silencing all server notifications on the driver, we yield only the field we need: CALL gds.graph.drop($name, false) YIELD graphName RETURN graphName. With that change no deprecation notifications appeared in our run. Choose the graph that matches the question. Louvain on the customer-category graph groups customers by which categories they are attached to, and struggles when groups share a category. Comparing whole category mixes with node similarity first, then clustering that customer-to-customer graph, is what separated the overlapping segments. Community IDs are labels, not values. Independent Louvain runs relabel the same groups, as the 65% and 63% agreement above shows. Align IDs across snapshots, and treat them as categorical in your model. Check your environment variables. The notebook reads its connection details from NEO4J_* environment variables. Stale variables left over from another project can point the notebook at a different server, which can show up as confusing errors that have nothing to do with your code. Pandas version. Feast 0.66.0 requires pandas<3 and >=1.4.3. Summary Neo4j works well as a feature computation engine alongside Feast. The Cypher queries for standard aggregations are comparable in complexity to their SQL equivalents, and GDS lets you add graph-algorithm output to the same feature set with a few projections and procedure calls. In our example, node similarity followed by Louvain recovered the hidden customer segments with an ARI of 0.98, where a plain favorite-category column managed 0.52, and Feast served the result like any other column. On held-out orders, grouping customers by community_id predicted category mix better than grouping by favorite_category, by 0.146 log-loss (95% interval 0.110 to 0.180). Those numbers come from synthetic data built so that combinations of categories matter, and from a simple count-based predictor, so treat them as a demonstration of the pipeline and the method rather than a benchmark. When we first used single-category preferences, the graph feature was redundant with favorite_category, which is a useful reminder to test whether a graph feature adds something before building it into a pipeline. The pattern generalizes beyond community detection. PageRank scores and node embeddings (for example from FastRP, node2vec or GraphSAGE) can be written back to node properties and flow through the same pipeline, with embeddings stored as array-valued features. Two cautions apply. Some graph outputs, such as shortest-path distances, are defined between pairs of nodes, so they need a fixed reference node before they can become a per-entity feature. And graph features are computed in batch, so a brand-new customer has none until the next run. The offline store (Parquet) and the serving layer (Feast) stay the same regardless of how the features were computed. The full source code is available on GitHub. More

Build Production-Ready AI Phone Agents: Infrastructure, Models, and Telephony

By Xavier Portilla Edo

CORE

Introduction: From Voice AI Prototype to Production Phone Agent A voice AI demo is a model, a microphone, and a speaker. A phone agent is a distributed system. Audio arrives from a carrier over a SIP trunk, callers interrupt on bad lines, backends time out, model providers have bad minutes, and callers want a person the moment something goes wrong. This article shows you how to build an AI receptionist for a dental clinic. It answers calls to the clinic’s number, looks up patients by caller ID, looks up and books appointments through a backend API, and transfers calls to the front desk when a person is needed. It runs on LiveKit Agents, the open-source framework for Python and Node.js. The logs highlight which features require LiveKit Cloud. Every snippet comes from one working project, and every output shows what that project printed. The architecture has six parts: A telephony provider that owns the number.SIP connectivity that turns a call into a session participant.Real-time audio infrastructure between the caller and the agent.The AI models: speech-to-text, a language model and text-to-speech.Application logic: conversation state, tools, and escalation.Monitoring: latency, errors, and cost for every call. Step 1: Choose Your Real-Time Voice Infrastructure A phone call is not a request. Audio flows both ways at once, callers talk over the agent, and a pause more than a second feels broken. You need a transport that moves audio frames in real time and an agent that reacts as they arrive: SIP is the phone network's signaling protocol. It sets up, transfers, and ends calls, while the audio travels over RTP.WebRTC carries low-latency audio between participants and absorbs packet loss and jitter.A media server bridges the two and turns the call into a participant in a WebRTC room. In LiveKit, the telephony media layer consists of the LiveKit server plus the LiveKit SIP service, both open source. A call becomes a SIP participant in a room, and LiveKit dispatches a job to an agent server. Each job runs in its own process — in this setup, one per call. The agent server connects out over a WebSocket for job dispatch and uses WebRTC for media, so it needs no public inbound ports. Run it all yourself or on LiveKit Cloud, or connect self-hosted agents to Cloud. LiveKit Inference, Agent insights, LiveKit Phone Numbers, and the participant move behind this warm-transfer workflow are Cloud-only. Cloud also provides enhanced noise cancellation, while basic WebRTC cancellation and separately licensed alternatives are available outside Cloud. There are other routes. Twilio (the carrier) can also stream audio calls over WebSockets (Media Streams), which LiveKit Cloud accepts through its Twilio Connector. At the fully managed end, Voiceflow runs the whole stack: you design the agent, import a Twilio or Vonage number, and assign it. Calls, recordings, and transcripts just work. The SDK route trades that convenience for control over every model, timeout, and transfer rule. Step 2: Select and Configure Your AI Models You need a cascaded pipeline that chains speech-to-text (STT), a large language model (LLM) that decides what to say and which tools to call, and text-to-speech (TTS). A real-time speech-to-speech model — such as OpenAI Realtime or Gemini Live — does all three in one. Cascaded STT, LLM, TTSRealtime speech-to-speechLatencyModerate; streaming overlaps the stagesLowest; one model, no hand-offsChanging modelsAny stage, independentlyOne provider for the whole pathTool callingMature, text-basedLess matureDebugging and auditText at every stageTranscripts arrive lateExact phrasing with say()YesNot the same wayCostThree meters, each tunableOne meter A receptionist must read appointment data exactly and keep an auditable transcript, so the agent uses the cascaded pipeline — LiveKit's recommended production default. Keep the models in configuration, since prices, voices, and availability change. In LiveKit Agents, the pipeline is an AgentSession with one parameter per stage. In LiveKit Cloud, the inference module provides hosted models with server-side fallbacks and no provider keys: Python from livekit.agents import AgentSession, TurnHandlingOptions, inference session = AgentSession( stt=inference.STT(model="deepgram/nova-3", language="en", fallback=[{"model": "assemblyai/universal-3-6-pro"}]), llm=inference.LLM(model="google/gemma-4-31b-it"), tts=inference.TTS(model="cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc", fallback=[{"model": "fishaudio/s2.1-pro", "voice": "fa4c9eb3dccc4806b382b40d61c6b10a"}]), turn_handling=TurnHandlingOptions( turn_detection=inference.TurnDetector(), endpointing={"min_delay": 0.5, "max_delay": 3.0}, interruption={"min_duration": 0.5}, ), ) When self-hosted, the same session takes provider plugins and your own keys. Nothing else changes. The receptionist picks one session with a MODEL_PROVIDER variable, and the calls in this guide run on the plugin version: Python from livekit.plugins import openai, silero session = AgentSession( stt=openai.STT(model="gpt-4o-mini-transcribe", use_realtime=True), # streaming transcripts llm=openai.LLM(model="gpt-4.1-mini"), tts=openai.TTS(model="gpt-4o-mini-tts", voice="coral"), vad=silero.VAD.load(), turn_handling=TurnHandlingOptions(turn_detection="vad", endpointing={"min_delay": 0.5, "max_delay": 3.0}), ) Step 3: Connect Your AI Agent to the Telephone Network The PSTN is the public phone network. The SIP trunk connects this to your system: the provider owns the number and forwards each call as a SIP INVITE. LiveKit mirrors the trunk with an inbound trunk (which claims the number) and a dispatch rule (which picks the room and the agent). Get a number. LiveKit Cloud sells US numbers through LiveKit Phone Numbers, which need only a dispatch rule. With Twilio, create an Elastic SIP trunk, point its origination URI at your LiveKit SIP endpoint (your project ID without p_, followed by .sip.livekit.cloud), and attach the number: Shell twilio api trunking v1 trunks create --friendly-name "Harbor Dental" --domain-name "harbor-dental.pstn.twilio.com" twilio api trunking v1 trunks origination-urls create --trunk-sid <TRUNK_SID> --friendly-name "LiveKit SIP" \ --sip-url "sip:<subdomain>.sip.livekit.cloud;transport=tcp" --weight 1 --priority 1 --enabled twilio api trunking v1 trunks phone-numbers create --trunk-sid <TRUNK_SID> --phone-number-sid <NUMBER_SID> Create the inbound trunk and the dispatch rule. The rule puts each caller in a fresh room and dispatches the agent named receptionist: JSON { "trunk": { "name": "Harbor Dental inbound", "numbers": ["+15105550100"] } } JSON { "dispatch_rule": { "name": "Harbor Dental receptionist", "trunk_ids": ["ST_TgirZ2Wz75YG"], "rule": { "dispatch_rule_individual": { "room_prefix": "call-" } }, "room_config": { "agents": [{ "agent_name": "receptionist", "metadata": "{\"clinic\": \"harbor-dental\"}" }] } } } Shell lk sip inbound create inbound-trunk.json lk sip dispatch create dispatch-rule.json lk sip dispatch list Plain Text ┌───────────────────┬────────────────────────────┬─────────────────┬─────────────────────┬─────────────────────────┬──────────────┐ │ SipDispatchRuleID │ Name │ SipTrunks │ Type │ RoomName │ Agents │ ├───────────────────┼────────────────────────────┼─────────────────┼─────────────────────┼─────────────────────────┼──────────────┤ │ SDR_w4oMyPKcc2qT │ Harbor Dental receptionist │ ST_TgirZ2Wz75YG │ Individual (Caller) │ call-_<caller>_<random> │ receptionist │ └───────────────────┴────────────────────────────┴─────────────────┴─────────────────────┴─────────────────────────┴──────────────┘ Create both once and reuse them for every call. Pin the rule to your trunk with trunk_ids, since a rule without it matches every trunk. The metadata reaches the agent as ctx.job.metadata, so one agent can serve several clinics. Answer the call. LiveKit SIP matches the trunk and the rule, creates the room with the caller in it, and dispatches a job to an agent server registered under the same agent_name. The caller's number arrives as a participant attribute, so the receptionist agent finds the patient before saying a word: Python server = AgentServer() @server.rtc_session(agent_name="receptionist", on_session_end=on_session_end) async def receptionist(ctx: JobContext) -> None: await ctx.connect() participant = await ctx.wait_for_participant() customer = None if participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP: phone = participant.attributes.get("sip.phoneNumber", "") logger.info("inbound call from %s to %s (call %s)", phone, participant.attributes.get("sip.trunkPhoneNumber"), participant.attributes.get("sip.callID")) try: customer = await clinic_api("GET", "/customers", params={"phone": phone}) except ToolError: customer = None # greet anonymously; the agent can identify the caller later session = build_session() observe(session) await session.start(room=ctx.room, agent=Receptionist(customer), room_options=room_options()) # A scripted greeting goes straight to TTS: no LLM round trip before the caller hears a voice. who = f"Hello {customer['name'].split()[0]}, " if customer else "Hello, " await session.say(f"{who}this is Ava from Harbor Dental Clinic, how can I help you today?") Check it with a call. A number that rings without an answer may indicate that no agent is registered under the configured name. A call that connects to silence may indicate the agent is not publishing audio. A healthy call looks like this in the agent log: Plain Text INFO livekit.agents - registered worker {"agent_name": "receptionist", "id": "AW_MMW9r2DMkYYP", "url": "ws://localhost:7880"} INFO livekit.agents - received job request {"job_id": "AJ_8pRxKpmvNjmf", "room": "call-_+15550002_BFK5wQ3zB25J", "agent_name": "receptionist"} INFO receptionist - inbound call from +15550002 to +15105550100 (call SCL_LeFbnPRqGmDK) lk room list shows the call- room while the call is up, and lk room participants get prints the SIP attributes, including sip.twilio.callSid, for finding the call in the Twilio console. Outbound calls and existing phone systems. Outbound calls and the warm transfer in Step 5 need an outbound trunk configured with the provider’s termination address, credentials if required, and the caller ID to present. Your code dials with CreateSIPParticipant. A SIP-capable PBX or contact center connects the same way, using an inbound trunk, an outbound trunk, or both, depending on the call direction. JSON { "trunk": { "name": "Harbor Dental outbound", "address": "harbor-dental.pstn.twilio.com", "numbers": ["+15105550100"], "auth_username": "<user>", "auth_password": "<password>" } } Step 4: Implement Conversations, Interruptions, and Backend Integrations Conversation state. AgentSession keeps the conversation history, including turns, tool calls, and results. The active agent’s chat context is sent to the LLM each turn. The receptionist stores the identified patient on the agent and includes that information in its instructions. Without that explicit context, the model may ask known callers for their number again: Python class Receptionist(Agent): def __init__(self, customer: dict | None = None) -> None: end_call = EndCallTool( extra_description="Only end the call after the caller confirms they need nothing else.", end_instructions="Thank the caller for calling Harbor Dental Clinic and say goodbye.", ) instructions = INSTRUCTIONS if customer: # matched by caller ID before the session started instructions += (f"\nThe caller ID matches the patient {customer['name']} (id {customer['id']}), " "so they are already identified: do not ask for their phone number.") super().__init__(instructions=instructions, tools=end_call.tools) self.customer = customer Turns and interruptions. Voice activity detection (VAD) finds speech, while turn detection estimates when the caller has finished talking. LiveKit’s default audio turn detector weighs both what the caller says and how they say it. With interruptions enabled, LiveKit stops the agent’s speech and trims the conversation history to reflect what was played before the interruption. This supports corrections such as “actually, make it Thursday” without custom interruption-handling code, though updating the appointment still requires application logic. The sessions wait 0.5 to 3 seconds before closing a turn, and LiveKit Cloud ignores interruptions under half a second, which filters coughs. LiveKit Cloud's adaptive interruption model also tells "mm-hm" from a real interruption; without Cloud credentials, the agent falls back to VAD: Plain Text INFO livekit.agents - adaptive interruption disabled due to unrecoverable error, falling back to VAD-based interruption Tools. A method decorated with @function_tool becomes a tool, and its docstring and type hints define the tool description and parameter schema the LLM sees. The receptionist calls a clinic API through the job’s shared HTTP session with a four-second timeout and turns handled failures into messages the model can act on: Python async def clinic_api(method: str, path: str, **kwargs: Any) -> Any: http = utils.http_context.http_session() try: async with http.request(method, f"{BACKEND_URL}{path}", timeout=aiohttp.ClientTimeout(total=4), **kwargs) as resp: if resp.status == 404: return None data = await resp.json() if resp.status >= 400: raise ToolError(data.get("error", f"clinic system error {resp.status}")) return data except (aiohttp.ClientError, asyncio.TimeoutError) as exc: raise ToolError("The clinic system is not responding. Apologize, and offer a transfer or a callback.") from exc class Receptionist(Agent): @function_tool() async def get_appointments(self, ctx: RunContext) -> list[dict]: """List the identified caller's upcoming appointments.""" if self.customer is None: raise ToolError("The caller is not identified yet. Ask for the phone number on file.") return await clinic_api("GET", f"/customers/{self.customer['id']}/appointments") @function_tool() async def book_appointment(self, ctx: RunContext, day: str, time: str, service: str) -> dict: """Book an appointment for the identified caller. Call it only after they said yes. Args: day: The date in YYYY-MM-DD format. time: The start time in 24 hour HH:MM format. service: The service, for example "dental cleaning". """ ctx.disallow_interruptions() # a write must finish even if the caller keeps talking if self.customer is None: raise ToolError("Identify the caller before booking.") return await clinic_api("POST", "/appointments", json={"customer_id": self.customer["id"], "service": service, "start": f"{day}T{time}"}) Three rules help keep tools safe on a phone line. Lookups may be interrupted, but completed lookup results stay in history. Writes call ctx.disallow_interruptions() first, because interrupting a booking does not automatically roll it back. A ToolError returns an actionable error to the LLM so the agent can explain the failure to the caller. The agent should confirm an appointment only after the backend reports success. The receptionist can book but not cancel, and booking requires spoken confirmation from the caller. Step 5: Add Human Call Transfers and Failure Handling An agent with no way out of the conversation is a dead end. The receptionist hands the call over when the caller asks for a person—for billing, insurance, complaints, or medical questions—or after two failed attempts to complete the caller’s request. A cold transfer forwards the call using SIP REFER: the carrier connects the caller to the new number, and the agent drops out once the transfer succeeds. A warm transfer puts the caller on hold, calls the human into a private room, briefs them, and merges the calls, reducing the need for the caller to explain the problem twice. The receptionist uses warm transfer for billing, insurance, and complaints. It uses cold transfer for other requests to speak with a person. Cold transfer. TransferSIPParticipant sends the REFER. The caller's participant identity is not the phone number, so sip_caller() finds the caller by participant type: Python @function_tool() async def transfer_to_human(self, ctx: RunContext, reason: str) -> str | None: """Cold transfer to the front desk. Call it only after the caller agreed to it.""" ctx.disallow_interruptions() line = sip_caller() # (job, SIP participant), or None outside a phone call if line is None: return "There is no phone line to transfer. Offer a callback instead." job, caller = line await ctx.session.say("Transferring you to the front desk now.", allow_interruptions=False) try: await job.api.sip.transfer_sip_participant(api.TransferSIPParticipantRequest( room_name=job.room.name, participant_identity=caller.identity, transfer_to=FRONT_DESK_URI, # "tel:+1..." on Twilio, "sip:+1...@host" on most PBXs play_dialtone=True, )) except api.SipCallError as exc: # the front desk rejected the call or did not answer logger.warning("transfer failed: SIP %s %s", exc.sip_status_code, exc.sip_status) return "The front desk did not answer. Apologize and offer to take a message or arrange a callback." except api.ServerError: return "The transfer could not be started. Apologize and offer a callback." return None # the caller has left the room The trunk must allow transfers. On Twilio, run twilio api trunking v1 trunks update --sid <SID> --transfer-mode enable-all. The target format depends on the carrier. Twilio takes tel:+15105550199, while a PBX such as Asterisk answers a tel: target with 400 Bad Request and needs sip:+15105550199@host. A front desk that rejects the call or doesn't answer within ringing_timeout (30 seconds by default) raises a SipCallError, and the caller is still on the line to be offered a callback. Warm transfer. The prebuilt WarmTransferTask (Python’s beta.workflows module) plays hold music, dials the front desk through the outbound trunk, and uses a second agent to brief the front desk based on the chat history. Once the front desk accepts the transfer, the task moves their SIP participant into the caller’s room: Python @function_tool() async def transfer_with_handoff(self, ctx: RunContext, summary: str) -> str | None: """Warm transfer: put the caller on hold, brief the front desk, then connect them.""" ctx.disallow_interruptions() if sip_caller() is None: return "There is no phone line to transfer. Offer a callback instead." if not ON_LIVEKIT_CLOUD: # the merge uses MoveParticipant, a LiveKit Cloud API return "Warm transfers are not available here. Use transfer_to_human; the caller already agreed." await ctx.session.say("Please hold while I brief the front desk.", allow_interruptions=False) try: result = await WarmTransferTask( sip_call_to=FRONT_DESK_NUMBER, sip_trunk_id=os.environ["LIVEKIT_SIP_OUTBOUND_TRUNK"], chat_ctx=self.chat_ctx, extra_instructions=f"Context for the front desk: {summary}", ringing_timeout=30.0, ) except ToolError: # no answer, voicemail, or the front desk declined return "The front desk could not take the call. Apologize, then offer a cold transfer or a callback." await ctx.session.say("You're now connected with the front desk. Goodbye!", allow_interruptions=False) ctx.session.shutdown() # the agent leaves; the caller and the front desk stay in the room return None Three details matter. First, merging the calls requires MoveParticipant, which is available only in LiveKit Cloud. On a self-hosted deployment, the transfer workflow can dial and brief the front desk, but the merge request returns not implemented. Second, the caller’s room must remain open after the agent session ends. Setting delete_room_on_close=False prevents session shutdown from deleting it. Third, a ringing timeout, detected voicemail, or an explicit decline ends the transfer task with a ToolError, allowing the agent to resume the caller conversation. Any fallback to a cold transfer must be handled separately by the application. The following log from the agent and LiveKit SIP service shows a billing-call transfer on a self-hosted deployment: Plain Text # agent (local time) 13:17:50 DEBUG livekit.agents - executing tool {"function": "transfer_with_handoff", ...} 13:17:51 DEBUG livekit.agents - executing tool {"function": "transfer_to_human", ...} 13:17:51 INFO receptionist - transferring sip_+15550002 to sip:+15105550199@asterisk (The patient Maria Lopez called reporting being charged twice on their last bill and needs assistance with billing.) # LiveKit SIP service (UTC) 12:17:55.918Z INFO sip transferring SIP call {"callID": "SCL_Gfe6warmSi7b", "transferTo": "<sip:+15105550199@asterisk>"} 12:17:56.938Z INFO sip inbound call transferred {"callID": "SCL_Gfe6warmSi7b"} Other failures. Catch backend timeouts and raise a ToolError with a useful recovery message, then instruct the agent to explain the problem to the caller. Model outages switch providers through fallbacklists on LiveKit Cloud. You can also configure stt.FallbackAdapter, llm.FallbackAdapter, and tts.FallbackAdapter in the agent process, whether Cloud-hosted or self-hosted. The session’s error event exposes ev.error.recoverable to guide error handling. A caller hang-up closes the session with the reason participant_disconnected. Adding the prebuilt EndCallTool lets the agent end the call cleanly when the conversation is over. Step 6: Deploy, Monitor, and Scale Your AI Phone Agent In production, the agent is a container running python agent.py start, which registers with LiveKit and waits for jobs. LiveKit's Dockerfile template builds in two stages and runs as a non-root user: Dockerfile ARG PYTHON_VERSION=3.12 FROM python:${PYTHON_VERSION}-slim AS base ENV PYTHONUNBUFFERED=1 ENV PIP_DISABLE_PIP_VERSION_CHECK=1 FROM base AS build RUN apt-get update && apt-get install -y gcc g++ python3-dev && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt ./ RUN python -m venv .venv ENV PATH="/app/.venv/bin:$PATH" RUN pip install --no-cache-dir -r requirements.txt COPY . . FROM base ARG UID=10001 RUN adduser --disabled-password --gecos "" --home "/app" --shell "/sbin/nologin" --uid "${UID}" appuser WORKDIR /app COPY --from=build --chown=appuser:appuser /app /app ENV PATH="/app/.venv/bin:$PATH" USER appuser CMD ["python", "agent.py", "start"] The image is 1.01 GB. In start mode, the agent answers a health check on port 8081 and writes JSON logs, ready for a log drain. On LiveKit Cloud, lk agent create builds and deploys it, lk agent deploy ships new versions, lk agent logs tails them, and --region keeps the agent in the same area as your callers. Capacity. Each call is a job in its own process. LiveKit recommends using 4 cores and 8 GB per agent server to start, for 10 to 25 concurrent calls — depending on the models you run locally. A server stops taking jobs above load_threshold (CPU, 0.7 by default), so scale out earlier, around 0.5. Give servers a grace period of ten minutes or more, so calls finish before shutdown. Latency. Every message in the history logs carries a metrics report. Log it to take note of the time spent for every turn: Python @session.on("conversation_item_added") def _on_item(ev: ConversationItemAddedEvent) -> None: if not isinstance(ev.item, ChatMessage): return m = ev.item.metrics if ev.item.role == "user" and m.get("end_of_turn_delay") is not None: logger.info("user turn: end_of_turn=%.2fs transcription=%.2fs", m["end_of_turn_delay"], m.get("transcription_delay") or 0.0) elif ev.item.role == "assistant" and m.get("e2e_latency") is not None: logger.info("agent turn: e2e=%.2fs llm_ttft=%.2fs tts_ttfb=%.2fs", m["e2e_latency"], m.get("llm_node_ttft") or 0.0, m.get("tts_node_ttfb") or 0.0) Plain Text 13:18:26,334 INFO receptionist - user turn: end_of_turn=1.21s transcription=1.21s 13:18:26,877 DEBUG livekit.agents - executing tool {"function": "get_appointments", "lk.pii.arguments": "{}"} 13:18:38,520 INFO receptionist - agent turn: e2e=3.45s llm_ttft=0.50s tts_ttfb=1.00s Maria's appointment question took 3.45 seconds from her last word to the agent's first, and tamesu (an open-source testing framework) measured 3.2 seconds at the end of the line (Step 7) — over three times the one-second allotment. The log shows where it went: 1.21 seconds to close the turn while the final transcript arrived, half a second for the LLM to call get_appointments, another half second to start the answer, and a full second to the first TTS audio. Each number points to a fix: an STT with faster finals or its own endpointing, the turn detector model instead of plain VAD, a short filler before slow tools, and models served near your agent servers. What to watch. The following metrics will tell you whether the agent is healthy. The SDK has the data for all of them: MetricWhere it comes fromCall connection and completionSIP participant attributes and the session close_reasonResponse latencye2e_latency per turn, as p50 and p95STT, LLM and TTS errorsthe error event and the report's errors countersTool failuresToolError raises in your logsTransfer successthe outcome each transfer tool logsCost per callsession.usage per model, plus the carrier's per-minute rate After a test call, the session report from ctx.make_session_report() answers most questions in one file. The following is an excerpt from an appointment call: JSON { "room": "call-_+15550002_BFK5wQ3zB25J", "duration_ms": 47621, "user_turns": 2, "close_reason": "CR_USER_INITIATED", "errors": { "llm": 0, "stt": 0, "tts": 0, "realtime": 0, "interruption": 0 }, "usage": [ { "model": "gpt-4.1-mini", "input_tokens": 3193, "output_tokens": 73 }, { "model": "gpt-4o-mini-tts", "input_tokens": 69, "output_tokens": 561, "audio_duration": 21.77 }, { "model": "gpt-4o-mini-transcribe", "input_tokens": 90, "output_tokens": 30, "audio_duration": 8.94 } ] } On LiveKit Cloud, session data appears in Agent Insights, with recordings and traces when agent observability is enabled. For Cloud-deployed agents, log drains forward runtime logs to Datadog, CloudWatch, Sentry, or New Relic. For fully self-hosted deployments, store the session report yourself, and export traces with OpenTelemetry. Step 7: Test the Complete Calling Workflow Before Launch Local audio tests confirm that the pipeline works under local test conditions. They don’t confirm that calls to the number reach the agent, that caller ID arrives, that telephone audio is clear enough for STT, or that a transfer reaches the front desk. Test in three layers. Unit tests on every commit. session.run() runs one conversation turn using text input and returns the results. You get assertions for messages and tool calls, an LLM judge to check whether a response matches the expected intent, and mock_tools to simulate tool failures: Python def called(result, name: str) -> bool: try: result.expect.contains_function_call(name=name) return True except AssertionError: return False @pytest.mark.asyncio async def test_billing_gets_a_warm_transfer(clinic_backend): # fixture: the mock clinic API async with openai.LLM(model="gpt-4.1-mini") as judge, AgentSession(llm=judge) as session: await session.start(Receptionist(customer=MARIA)) first = await session.run(user_input="I was charged twice on my last bill. Can I talk to someone?") second = await session.run(user_input="Yes please, transfer me.") assert any(called(r, "transfer_with_handoff") for r in (first, second)), "no warm transfer" await second.expect[-1].is_message(role="assistant").judge( judge, intent="Explains the transfer is not possible right now and offers a callback or to take a message.") Plain Text $ pytest -q 7 passed in 22.13s The seven tests cover the greeting, the lookup, confirmation before booking, both transfers, and a backend outage. Write judge intents without ambiguity: “offers a transfer or a callback” may be interpreted as requiring both options. If either is acceptable, write “offers at least one of these options: a transfer or a callback; either option alone is sufficient.” Simulated callers on every pull request. Agent Simulations, a LiveKit Cloud feature, runs an LLM-driven caller with a goal against your real agent and grades the whole conversation. Run each scenario several times, since the caller and the judge are both models. Real phone calls before launch. tamesu is an open-source testing framework whose siptarget calls your agent through a LiveKit SIP service and a trunk you provide, like a customer would. A simulated caller with a persona and a goal speaks through TTS and listens through VAD and STT. It can interrupt or add background noise, while deterministic checks and LLM judges evaluate the call, regardless of the framework used to build the agent. SIP testing requires the full build from the releases page; the lite build does not include SIP support. YAML suite: receptionist-phone targets: clinic: type: sip direction: outbound to: "+15105550100" # the clinic from: "+15550002" # Maria's number, so the agent should recognize her ring_timeout: 30s max_call_duration: 3m tests: - name: a patient asks for her next appointment target: clinic persona: { name: Maria, style: "polite", language: en, voice: en-US-female } scenario: script: - wait_for: "Harbor Dental Clinic" # a real caller waits for the greeting goal: "You are Maria Lopez. Find out when your next appointment is, then say goodbye." max_turns: 4 stop_when: "the agent has said the date and time of the next appointment" assert: - judge: "The agent greets Maria by name and says her next appointment is a dental cleaning on November 12 at 10 in the morning" - metric: { name: greeting_ms, lt: 3000 } # call answered to the agent's first word - name: a billing question reaches the front desk target: clinic persona: { name: Maria, style: "firm", language: en, voice: en-US-female } scenario: script: - wait_for: "Harbor Dental Clinic" goal: "You were charged twice on your last bill and want to talk to a person. Say yes to a transfer. When someone from the front desk answers, say thanks and goodbye." max_turns: 5 stop_when: "someone from the front desk has answered the call" assert: - judge: "The agent offers to transfer the billing question, and then a person from the front desk answers the call" - call_ended_by: [agent, caller] Without the wait_for step, the simulated caller starts talking the moment the call connects, right over the greeting. tamesu sip setup creates the outbound trunk on your LiveKit SIP service, then tamesu doctor checks the secrets, the models, and the SIP service. Meanwhile, tamesu run places the calls and keeps a transcript, a stereo recording, and the call timeline of each run: Plain Text $ tamesu run tamesu run: 2 tests, output in .tamesu/runs/2026-11-09T14-16-40 PASS receptionist-phone / a billing question reaches the front desk 47.6s PASS receptionist-phone / a patient asks for her next appointment 48.2s 2 passed, 0 failed, 0 errors in 1m36s · 14 model calls, 3892 tokens, $0.0087 · llm cache: 0 hits, 6 misses Pre-Launch Checklist The number rings the agent from mobiles, landlines, and withheld caller IDs.Known patients are identified by caller ID, and unknown callers still get help.Every tool has a timeout and a ToolError path, and the agent never logs data it did not receive.Writes need a spoken "Yes" and cannot be interrupted, and the agent has no tool it should not have.Transfers work on a real call, and a failed transfer brings the caller back to the agent.Hang-ups, silence, and network drops leave no rooms open or jobs left behind.Latency, errors, transfer outcomes, and usage are logged and charted per call.Simulated callers at peak load plus a margin stay under load_threshold.The hard cases pass: speakerphone in a car, accents, interruptions, long silences, requests the agent has no tool for, a backend or model provider down, and a front desk that doesn't answer.Caller ID alone never unlocks sensitive actions. Callers hear a recording notice where consent is required, recordings and transcripts expire, and keys live in a secret store.Every model or prompt change reruns all of the above. Conclusion: Building Reliable AI Phone Agents An AI phone agent is a complete application, not a language model wired to a microphone. What makes it production-ready is invisible in a demo: a trunk and a dispatch rule that reliably reach the agent, swappable models, turn handling tuned for phone lines, tools with timeouts and limits, an escalation path that works when nobody picks up, and metrics for every call. Keep those parts modular, with narrow interfaces between them. Then you can test each one alone, replace a model when a better one ships, and scale agent servers without touching the trunk. Resources and Documentation LiveKit Agents, the voice AI quickstart and pipeline typesTurns and interruptions and function toolsTelephony: inbound trunks, dispatch rules, Twilio and testing your setupCall transfers, WarmTransferTask, and the warm transfer exampleDeployment, self-hosted deployments, and metrics and data hooksTesting: unit tests and Agent Simulationstamesu: the SIP target and SIP setupVoiceflow phone numbers and the LiveKit Twilio Connector Versions used: LiveKit Agents 1.8.10 on Python 3.12, LiveKit CLI 2.19.3, a self-hosted livekit-server 1.13.11, and LiveKit SIP service 1.10 with Asterisk as the carrier, tamesu 1.20.1, and GPT-4.1 mini for the agent and the judges. More

Beyond State Consistency: Preventing Stale Decisions in AI Agent Systems

By Anish Yennapusa

You Do Not Need a PhD to Become an AI Engineer: The Essential Skill Stack

By Faisal Feroz

Two AI Agents, One Database: Solving Concurrent Updates With Optimistic Locking

By Tanushree Das

Time to First Token: How Long Does It Take for a Large Model to Start Responding?

By Jake Tao

Prompt Caching for LLMs, Part 1: Exact-Match Caching in Production

Why Prompt Caching Matters for LLM Applications? Modern LLM apps often get the same, or almost the same, prompt more than once. This happens with form submissions, common FAQ questions, agent tool calls, or a user asking the same thing in different words. Without caching, each of these repeat requests pays full token cost, full latency, and adds unwanted pressure on your rate limits. There are four caching methods used in production today. This article, Part 1 of a 3-part series, covers the first and most basic one: exact-match caching. Part 2 covers semantic caching, for requests that mean the same thing but are worded differently. Part 3 covers what happens at the model-serving layer itself: self-hosted KV cache reuse and provider prefix caching, plus the production details every team ends up needing: watching your metrics, working out your costs, and a full checklist. Gateway or application exact-match caching. A hash lookup on a byte-identical or normalized request. Zero cost on a hit. Covered here.Semantic caching. Embedding plus a vector similarity search. Zero model cost on a hit for reworded queries. Covered in Part 2.Inference-engine KV cache reuse, for teams running their own models (vLLM, SGLang). Covered in Part 3.Provider prompt or prefix caching (Anthropic, OpenAI, Bedrock). Covered in Part 3. These layers stack. Check exact match first, then check semantic, then let the request reach the model, where either your own inference engine or the API provider reuses whatever part of the prompt stayed the same. The Full Picture Exact match sits right at the front of the stack. It is the cheapest check to run, so it should catch as many requests as it can before anything more costly runs. Layers 1 and 2 sit in front of the model and can skip the model call entirely. Layers 3 and 4, covered in Part 3, sit inside the model call itself and never skip it. They just make it much cheaper when a part of the prompt repeats. 1. Exact Query Caching (Baseline) Treat the prompt, or a cleaned-up version of it, as a key in a fast key-value store: Redis, Memcached, or an in-process LRU cache. Key Design Hash the full context that matters: system prompt, user message(s), tool definitions, and any settings (like temperature) that change the output.Clean up whitespace, and sort tool lists if their order does not change how the model behaves.Store the full response, or structured output, plus details like timestamp, model version, and TTL.For agents, the key needs more than just the user message. Include the full tool schema (name, description, parameters), any tool-call results you fold back into the context if they are always the same, and leave out anything you don't want changing the key, such as a request ID or trace ID that changes every call but doesn't affect the output. Two common ways a cache silently breaks here: tool definitions listed in a different order between calls, and JSON keys written in a different order by different parts of the same codebase. Sort both before hashing. Don't count on the order matching by chance. Where Should the Cache Live? The store you pick changes how it can fail, more than most teams expect at first: In-process LRU (a local dict, or a library like cachetools) is the fastest choice, under a millisecond, with no network call. It only sees traffic hitting that one process, so on a deployment with many instances, each instance builds its own cache on its own, and the hit rate stays low until every instance has warmed up. This is fine for small deployments, or for caching things that are cheap to redo per instance anyway.A shared store (Redis, Memcached) gives every instance the same view, so a hit on instance A also counts as a hit for a request landing on instance B. This is the right default once you go beyond a single instance, at the cost of a network round trip, usually a few milliseconds on the same network.A two-level setup (a small in-process LRU in front of shared Redis) gives you the best of both: the busiest keys never leave the process, and everything else falls through to the shared store. Worth the extra work once a small number of keys account for most of your traffic, which is common on FAQ-style workloads. Handling Stale Data A fixed TTL, for example, 60 seconds, for time-sensitive data such as travel prices or stock levels.LRU eviction once the cache fills up.Clear the cache on purpose when something changes: a price update, an outage, a policy change. Python example (Redis): Python import hashlib import json import redis from typing import Optional r = redis.Redis(host="localhost", port=6379, db=0) def make_key(prompt: str, model: str, temperature: float = 0.0) -> str: payload = json.dumps({"prompt": prompt, "model": model, "temp": temperature}, sort_keys=True) return "llm:exact:" + hashlib.sha256(payload.encode()).hexdigest() def get_exact(prompt: str, model: str, temperature: float = 0.0) -> Optional[str]: key = make_key(prompt, model, temperature) try: return r.get(key) except redis.RedisError as exc: # Cache being down should never break the request. # Log it and fall through to the model. logger.warning("exact cache read failed: %s", exc) return None def set_exact(prompt: str, response: str, model: str, temperature: float = 0.0, ttl: int = 300): key = make_key(prompt, model, temperature) try: r.setex(key, ttl, response) except redis.RedisError as exc: logger.warning("exact cache write failed: %s", exc) Treat the cache as a nice-to-have, not something the app depends on. If the cache goes down, fall back to full model calls; don't fail the request. Should You Cache Errors? Most production systems say no, or only in some cases, and it is easy to skip thinking about this until something goes wrong. A few good defaults: Never cache a 429 (rate-limited) or 5xx error from the model provider as if it were a real answer. Retry with backoff instead of remembering the failure.If your own guardrail or content filter rejects a response, that is a judgment call. Some teams cache the rejection briefly (a few seconds to a minute) just to stop a broken or hostile client from triggering the same expensive rejection path again and again. Others never cache rejections at all, so they don't end up serving a stale "rejected" result after the input has been fixed.If you do cache rejections, use a clearly different TTL and a different key prefix (llm:reject: instead of llm:exact:) so you can clear them, or leave them out of your hit-rate numbers, separately from real answers. Warming the Cache A cache that just went live is empty, and the first few minutes after any deploy that clears the cache see a drop in hit rate. If your logs keep old prompts and responses, a simple warm-up job that loads the most common prompts from the last few days into the new cache, before it takes real traffic, avoids that drop. This matters most for FAQ-style traffic where a small number of questions make up most of the volume. Request Coalescing: Solving the Thundering Herd Exact-match caching has one failure mode that only shows up under real load: if fifty identical requests arrive at almost the same time, before the first one has filled the cache, all fifty can miss, and all fifty can call the model on their own. This is called a cache stampede, and it is not rare at real traffic levels. A viral post, a batch job starting up, or a retry storm from another system can all set it off. How to Fix It Request coalescing, also called single-flight. The first request for a key takes a short lock (or sets an in-flight marker), and the other requests wait on that same result instead of calling the model on their own. Redis's SET NX pattern, or a library like singleflight in Go, handles this cleanly.Cache the "in progress" state too, not just finished responses, so other requests can see "someone is already working on this" instead of racing to do it themselves.Put a fair timeout on the wait, with a fallback to a normal model call if the in-flight request does not finish in time, so one stuck request does not hold up every other request waiting on it. Without this fix, exact-match caching can actually make traffic spikes worse, not better. Every process learns about a busy key at almost the same moment and hits the model all at once, right when the model can least handle it. When It Shines Templated queries, form-driven flows, test suites, batch jobs — anywhere the same string keeps coming back. Hit rates can be very high in narrow domains and close to zero in open, free-form chat, which is exactly the gap semantic caching, covered in Part 2, is built to close. What's Next Exact-match caching only helps when the input matches exactly, which is common for templated and form-driven traffic but rare in open conversation. Part 2 covers semantic caching: matching reworded questions with embeddings, doing it safely with a verification step, and the invalidation, multi-turn, and security issues that come with trusting a similarity score instead of an exact match. Further Reading Redis documentation, SET command options (including NX for locking patterns)Redis documentation, key eviction policies (LRU, LFU, TTL-based)

By Ram Ghadiyaram

CORE

Agent Harness, Memory Indexing, and AI Governance in Test Automation

Software teams are adopting AI agents to write, run, and repair tests faster than ever. The breakthrough is usually not the model alone. ChatGPT and Cursor can share the same underlying intelligence, yet one feels like a copy-and-paste assistant, while the other feels like a teammate that can edit files, run commands, and iterate until the job is done. The difference is the agent harness: the orchestration layer that wraps the model with instructions, tools, context, and guardrails so raw reasoning becomes reliable action. In test automation, that gap matters even more. A failing scenario can produce thousands of lines of Playwright output, span page objects, step definitions, test data, and multiple environments. Dump all of that into a single long chat, and the model slows down, costs rise, and decisions get noisier. A well-designed harness treats the model as a planner, not a dump truck. It indexes the codebase so the agent can ask structural questions instead of reading entire files. It delegates test execution and file generation to smaller, isolated workers that return compact summaries. It validates credentials, environment profiles, and reference fixtures before a run starts, so failures are caught early instead of mislabeled as “broken locators.” And when something does break, it routes the failure through a governed path: diagnose, fix, audit the patch, re-run only what changed, report results in a form humans can act on. This article walks through that architecture in practice: what an agent harness is, how memory and code indexing reduce token waste, why tiered model routing balances cost and quality, and how governance gates keep AI-assisted automation predictable rather than magical. Whether you are building on Cursor, Claude Code, or your own SDK integration, the pattern is the same. The model is the engine. The harness is what makes it drivable at scale. Agent Harness An agent harness is the software infrastructure that wraps around an AI model (like Claude, GPT-4) to turn it into an autonomous working agent. Harness acts more like an operating system, providing the model with memory, tools, execution environments, and guardrails to complete multi-step tasks in the real world What the Harness Does Harness handles three things: Instructions – system prompts, rules (like the CLAUDE.md), skills, and subagent definitions that shape behavior.Tools – file read/write, terminal, search, MCP servers, browser automation, etc.The loop – the model proposes an action → the harness executes it → results go back to the model → repeat until the task is done Beyond that, a good harness also manages: Context – which files, docs, and tool outputs the model sees (and when to trim them)Orchestration – routing tasks to specialized subagents (planner vs. runner vs. reviewer)Safety boundaries – sandboxing, permissions, hooksTermination – knowing when the job is finished vs. stuck What can a typical agent harness look like: Agent harness How Agent Harness Can Be Achieved Agent harness cannot be achieved by a single library — it is a layered orchestration system built from Claude Code config files under .claude/, and CLAUDE.md as the control plane. The codebase needs to add rules for who does what, when to stop, and how information flows back. Control pane – CLAUDE.md + .claude/settings.jsonSkills – Orchestrated WorkflowSubagents – constrained workersInfrastructure – MCP tools + hooks Control Pane Considering we are using Anthropic's ecosystem, the main session is always Opus (planning, judgment, MCP). Heavy but repetitive work goes to Haiku subagents in isolated contexts. CLAUDE.md defines the routing table and hard rules — e.g., never run Playwright directly in Opus; always delegate to test-runner: Markdown ## Tiered model routing (Opus parent + Haiku runner) This project pins the main Claude Code session to **Opus** (`.claude/settings.json` → `model: claude-opus-4-7`) and delegates **test execution only** to a Haiku subagent defined in `.claude/agents/test-runner.md`. ... 1. When the user says "run", "execute", "verify", or "re-run" a scenario id / tag / feature file → invoke the `test-runner` subagent. Do not run `npx playwright test` directly in the main session. ... 4. If `failure_category` is `auth`, `config`, `network`, or `other` → do **not** self-heal. Surface the JSON report and stop. Skills: Orchestrated Workflows Skills are markdown playbooks the Opus session follows. They encode multi-step pipelines with explicit gates. /selfheal — failure → fix → verify: Markdown # Self-Heal Test 0. **Preflight.** Invoke `/preflight` ... 0.5. **Fixture check** ... 1. Run the failing scenario(s) via the `test-runner` subagent. 2. Inspect failures via MCP Playwright browser snapshot ... ... 5. **Invoke the `heal-edit-reviewer` subagent** ... 7. Identify **all affected scenarios** via `get_impact_radius_tool` ... 8. **Re-run in BOTH env1 and env2 — in parallel.** /codegen — CSV → BDD with planner/renderer split: Markdown 0. **Preflight.** ... 2. **Plan reuse via graph.** ... `semantic_search_nodes_tool` ... 6. **Invoke `codegen-writer`** with the BuildSpec. 7. **Audit the `.feature` file** via `feature-reviewer` ... 8. **Audit the emitted `.ts` files** via `heal-edit-reviewer` ... 10. **Smoke-run one scenario.** ... via `test-runner` ... Skills are the harness’s state machine — they define order, stop conditions, and which actor handles each step. Subagents: Constrained Haiku Workers Each subagent in .claude/agents/*.md is a specialized worker with a narrow tool set, fixed output schema, and hard prohibitions. Example — test-runner: YAML --- name: test-runner description: ... Does NOT self-heal, edit code, or modify any files. model: claude-haiku-4-5 tools: Bash, Read, Glob --- It must return strict JSON (~200 bytes) instead of flooding Opus with Playwright stdout: JSON { "status": "pass" | "fail", "exit_code": <number>, ... "failure_category": "locator-timeout" | ... | null, "stdout_tail": "<last ~30 lines of stdout/stderr>" } So the different subagents that the codebase harness are: Subagent harness MCP servers extend what the harness can touch: Playwright browser – live UI inspection for locator discovery and debuggingcode-review-graph – structural + semantic code index (avoid reading full files) Hooks in .claude/settings.json run automatically outside the agent loop: Hooks used in the Claude ecosystem Memory and Indexing: Two Complementary Systems “Memory indexing” here is not one thing. The project uses structural code indexing (knowledge graph) and persistent session memory (conventions/feedback). They solve different problems. Code Knowledge Graph (Structural Indexing) Plugin: code-review-graph v1.8.x Update cadence: Incremental after every Edit/Write via PostToolUse hooks (~2s typical)What gets indexed (Tree-sitter AST parsing): AST parsing output Each node stores: qualified name, file path, line range, parameters, return type, SHA-256 hash (for change detection). Semantic layer: After embed_graph_tool, all ~611 code nodes get 384-dim vectors (all-MiniLM-L6-v2). This powers meaning-based search — e.g., CSV step "Validate that the username field is present" matches Username.verifyUsernameFieldVisible() even when names differ. Primary MCP tools: Graph-first rule (from CLAUDE.md): Query the graph before reading files. Read narrowly with offset/limit only when you need exact insertion points. Session Memory (Governance & Conventions) Location: .claude/memory/ (initialized from .claude/memory-template/) Purpose: Persist decisions and feedback across sessions so the agent doesn’t re-learn conventions every time. Documented memory categories include: Session memory is always loaded alongside CLAUDE.md. It governs how the agent behaves; the knowledge graph governs what exists in the codebase. How Indexing Is Used in Practice During/codegen (CSV → BDD): Opus parses CSV → scenario titles and stepsFor each step: semantic_search_nodes_tool("<step text>") → reuse existing step or mark reuse: nullGraph checks existing page object methods, fixtures, scenarios before emitting anythingHaiku codegen-writer renders files from BuildSpec only — no judgment, no MCPReviewers gate output before lint/smoke run During/selfheal (locator failure): test-runner returns compact JSON (not raw stdout in Opus context)Opus scopes via graph: callers_of(brokenMethod) → read only affected stepsMCP browser diagnoses live UIAfter fix: get_impact_radius_tool → re-run only affected scenariosParallel test-runner on redama + fedramp During reviews: /code-review-graph:review-delta — changes since last commit, target <800 tokens context/code-review-graph:review-pr — full PR with blast-radius and test-coverage gaps Tiered Model Routing (Cost + Quality Governance) The harness explicitly separates judgment from execution: Tiered routing architecture Haiku costs ~1/15th of Opus per token — so even when total system tokens rise slightly, the dollar bill drops. Clean Governance of AI Usage Preflight Gates (Fail Fast, Don't Self-Heal the Unhealable) Governance here means predictable behavior, auditable outputs, and hard stops on bad patterns — not just “use AI less.” Separation of Concerns (No Guessing) If spec is ambiguous → error JSON, not a guess Hygiene Rules Enforced By Reviewers Context Budget Rules (9 Rules in CLAUDE.md) Graph-first queriesNarrow reads (offset/limit)Narrow MCP snapshotsDelegate bulk I/O to subagentsTrim subagent payloads (no stdout on pass)Reuse over rewrite (semantic search before authoring)Impact-radius re-runs (not whole suite)No echo-back of raw artifactsNo cat/head/tail on source Automatic Enforcement via Hooks From .claude/settings.json: Hooks run silently — governance without Opus token spend. Output Contracts (Structured, Not Prose) Every subagent returns strict JSON. The parent renders tables — never pastes raw JSON or full traces to the user. This is both UX governance and context governance. End-to-end: One failure, fully governed Conclusion The value of AI in test automation is not unlocked by choosing a larger model. It is unlocked by building a harness that knows what to delegate, what to validate, and what to keep out of the planner’s context. At a high level, four principles hold across any stack: Separate judgment from execution. One session plans, inspects, and decides. Specialized workers run tests, emit files, and audit changes — each with a narrow mandate and a structured response format. That boundary is what prevents a single conversation from drowning in logs and generated code. Validate before you execute. Credential sessions, environment profiles, and reference fixtures should be checked at the door. Failures that originate in setup or data drift must not enter a self-heal loop designed for code and locator problems. Fail fast, report clearly, and reserve automated repair for repairable categories. Index before you read. A maintained structural and semantic index turns broad codebase questions into small, precise queries. That reduces cost, speeds up triage, and keeps the planner focused on the lines that actually need to change. Govern every exit path. Delegation rules, audit gates, bounded heal attempts, parallel multi-environment re-verification, and human confirmation on data substitution are not bureaucracy — they are the difference between a demo and a production workflow. Taken together, these standards describe an agent harness that teams can adopt incrementally: start with an execution worker and structured JSON results, add preflight gates, introduce a code index, then layer in render and review workers as codegen volume grows. The model remains the reasoning engine throughout. The harness is what makes that reasoning trustworthy enough to run unattended, cheap enough to run often, and clear enough that engineers still know what happened when something goes wrong.

By Soumyajit Basu

Using the Anthropic Evaluation Tool to Check the Claude Code Plugin Efficiency

A Claude Code skill is a Markdown document that the assistant loads when a task matches its description. A collection of skills, agents, and hooks can be packaged as a plugin and distributed through a marketplace. Since the summer of 2026, my development environment has relied on such a plugin for Java work, consisting of a set of Spring Boot, Quarkus, Apache Camel, and Jakarta EE agents and skills. This collection started as a small set of artifacts but, little by little, I extended it such that, after several months, it contained 43 files and 17,338 lines. This raised a precise question: were the stack conventions provided by this plugin really contributing anything measurable to the final outcome, or could the model, by its intrinsic general knowledge, infer the same result with much fewer tokens? This article describes how the Anthropic evaluation tool answered the question, and what the measurements revealed about the skills and the evaluation method itself. A Criterion for Useful Skill Content A skill contributes when it changes what the model produces. Four kinds of content satisfy that criterion: Personal or team conventions, which the model has no means to infer.Facts more recent than the model's training data, such as new versions and renamed artifacts.Non-obvious pitfalls and the decisions that avoid them.Corrections to guidance that is wrong elsewhere, including in upstream plugins. Textbook material, such as the definition of an Enterprise Integration Pattern or the meaning of an HTTP status code, belongs to what the model already knows. Trying to provide this same knowledge in skills consumes between 5,000 and 8,000 tokens each time a skill loads. Additionally, the given skill might be outdated. For example, my files still referenced Spring Boot 3.2.1 months after the move to Spring Boot 4. Restructuring Before Measuring The first step was to collect all the loose folders in ~/.claude, defining agents, skills, and hooks, to remove the duplicates and to consolidate them in stand-alone plugins. The result is a marketplace named simplex-software that lists five plugins: PluginActivationspring-boot-templateper projectquarkus-templateper projectjava-extrasuser scope, every sessioncamelper projectjakarta-eeper project The stack plugins take precedence on every conflict. java-extras holds personal rules, cross-stack decisions, operational notes, and a UserPromptSubmit hook that reminds the model, on every prompt, which skills apply to the detected stack. Activation per project keeps stack material out of sessions that don't use it. The Measurement Method Claude Code provides claude plugin eval, an open-source evaluation tool allowing you to test the pertinence of your skills, agents, and plugins. For example, you might want to evaluate the efficiency of a given Claude Code plugin. To do that, you need to define "test cases," execution scenarios that specify an input prompt, the required context or tools, and the logic or criteria for determining success. A case is a directory with a prompt and a set of graders. The harness runs each case with and without the plugin, several times per arm, and reports the score difference. A case whose score difference stays near zero identifies a plugin that contributes little to the content, since the model can produce the same result without it. Such a plugin must be ablated because its skills add only tokens. Instead of creating new files in the project, each case asked Claude to provide the requested code and configuration directly in its answer. Therefore, they don't require a prepared sample project, write permission, etc. The graders examined the answer text. Each case combined three kinds of graders: Regular expressions for the specific facts the skill claims. These cost nothing.A judge model (Claude Haiku by default) with three votes for overall correctness.A tool_used grader that checks whether Claude invoked the skill through the Skill tool. It shows whether the plugin arm actually loaded the skill. It is excluded from the score because the arm without the plugin has no skill to invoke and would fail it in every run, which would inflate the difference. A case for the Camel plugin illustrates the format. The prompt file: Markdown --- name: quarkus-route-test tags: [camel, quarkus] max_turns: 8 allowed_tools: [Skill, Read, Glob, Grep] --- Quarkus 3 project using Camel Quarkus. A RouteBuilder defines route id `process-route` from `direct:process` to `kafka:processed-orders`. Write a test that verifies one message reaches the Kafka endpoint without a real Kafka broker being used for that endpoint. Include the Maven test dependencies. Its regular expression grader, graders/test-artifact.md: Markdown --- type: regex pattern: "camel-quarkus-junit(?!5)" match: contains --- Uses camel-quarkus-junit, the successor of the deprecated camel-quarkus-junit5. Its judge grader, graders/judge.md, whose body is the criterion the judge model applies to the answer: Markdown --- type: llm --- Pass only if ALL hold: - test dependency is org.apache.camel.quarkus:camel-quarkus-junit (not camel-quarkus-junit5) - test extends CamelQuarkusTestSupport, overrides isUseAdviceWith() to return true, advises route "process-route" (mockEndpoints / mockEndpointsAndSkip) and then starts the context - does not use @MockEndpoints (not available in Camel Quarkus) - no Thread.sleep Its skill indicator, graders/skill-fired.md, which passes when Claude loads a skill whose name contains camel: Markdown --- type: tool_used tool: Skill input_match: "camel" --- Indicator only: the plugin skill was loaded. The command runs every case of a plugin, three runs per arm, with a cost ceiling: Shell claude plugin eval . -j 3 --no-publish --max-cost-usd 5 First Results Ten cases covered the three own plugins. The table reports the mean score with the plugin, without it, and the difference. Two scores are corrected for grader defects described in the next section. PluginCaseWithWithoutDifferencecamelCamel Quarkus route test1.000.33+0.67camelSpring Boot route unit test1.000.33+0.67camelSpring Boot Kafka integration test1.000.92+0.08jakarta-eeintegration test on WildFly0.890.33+0.56jakarta-eepom setupequalequal0jakarta-eeresource, service and injectionequalequal0java-extrasCircleCI with Testcontainers1.000.33+0.67java-extrasgraceful shutdown on Kubernetes1.000.50+0.50java-extrasSpring Boot to Quarkus migration1.000.89+0.11java-extrasReactor to Mutiny translation0.560.67about 0 The cases with a large difference show that the plugin had a consistent contribution to the final result. In the case of Camel, for example, without the plugin the model estimated the Maven artifact version as 4.18.0, whereas the skill states 4.23.0, built on Spring Boot 4.1.1. It also selected the test artifacts camel-test-spring-junit6 instead of camel-quarkus-junit with the skill. For CircleCI it chose a plain Docker executor, which offers no Docker daemon to Testcontainers. For graceful shutdown, it omitted the arithmetic that relates the preStop delay, the application shutdown timeout and terminationGracePeriodSeconds. The cases with a difference near zero show the opposite pattern. The Reactor to Mutiny cheat sheet, the Jakarta EE pom and the CDI constructor injection rules describe knowledge the model already applies correctly. Reading the answers confirmed it: both arms produced the same code. Trimming, and the Regression the Evaluation Caught The trimming pass removed content with near-zero differences and reduced the rest to facts, decisions, and pitfalls. The Reactor to Mutiny section disappeared. The Spring Boot to Quarkus table kept only the rows where a mistake is likely; the Jakarta EE setup became a short list of facts and The Camel Kafka test skeleton became one sentence. The second run, after trimming, confirmed that every case with a large difference scored 1.00 with the plugin. Only the migration from Spring Boot to Quarkus case declined from 1.00 to 0.78, but part of the decline came from judge noise. However, a part was genuine: the shortened table row stated that the Quarkus @Transactional annotation has no readOnly attribute, which led the model to mistakenly remove the read-only behavior of the original method. Without the skill, the model preserved it through a read-only Hibernate session. The corrected rule prescribes that the intended behavior should be achieved via a read-only Session with FlushMode.MANUAL, or a org.hibernate.readOnly query hint. A third run of that case, judged by Claude Sonnet with a criterion that checks the read-only behavior, passed six runs out of six. The episode is the strongest argument for the evaluation: a detail that appeared harmless changed the generated code, and only the measurement detected it. Final State and Cost MeasureBeforeAfterAgent and skill files436Lines of agent and skill content17,338638Tokens added to every sessionabout 2,100about 520, plus about 150 for each optional plugin enabled in the projectEvaluation casesnone6, each with a measured difference The complete exercise, including the exploratory run, both full runs and the verification run, cost about 20 US dollars in model usage. A full run of the remaining suite costs about 6 dollars and takes about five minutes with three parallel runs. Lessons The useful content of a skill is narrow: versions, renamed artifacts, corrections and non-obvious conventions. Explanations of well-known frameworks contribute nothing measurable.An evaluation with and without the plugin is inexpensive and decisive. It replaces a debate about relevance with a number per case.Graders require the same scrutiny as the skills, and a small judge model is inconsistent on long answers.A reduction of content is a change like any other and deserves a regression test. The readOnly episode demonstrates it. Limitations Each arm ran three times, which indicates a direction rather than a statistically established effect. The results describe one model at one date. Here, our Camel plugin pinned version 4.23, which a later model may know without assistance. This is precisely why the suite should run again after each model change. The cases reflect one developer's projects, and the personal rules in `house-rules` were deliberately left out of the evaluation, since the model has no means of guessing them.

By Nicolas Duminil

CORE

Three Measured Outcomes of Manually Scaling Fleet in Prod

Every GitOps setup meets the same moment: something is on fire, someone runs kubectl scale on prod, and the fix works. What happens next depends on two settings that rarely get checked before an incident: self-healing on the GitRepo, and pause on the cluster. I ran all three combinations in a lab on Rancher 2.15.1 with Fleet (biased, I work at SUSE). Three EKS clusters: dev and staging on Kubernetes 1.36, prod on 1.35. Cases 1 and 2 used the guestbook app from the public rancher/fleet-examples repo, with the frontend Deployment at 3 replicas. Case 3 used the podinfo Helm chart from my own repo. Every manual change went through the scale button in the Rancher UI, which sends a PATCH to the Deployment's scale subresource, the same request kubectl scale makes. The Two Settings Self-healing is spec.correctDrift.enabled on the GitRepo. It is off by default. In the Rancher UI it sits in step 4 of the GitRepo wizard, Advanced, Resource Handling, "Enable self-healing". To change it on an existing GitRepo: Continuous Delivery, App Bundles, the GitRepo, the three-dot button top right, Edit Config, then Next until step 4. Pause lives on the cluster: Continuous Delivery, Clusters, three-dot menu, Pause. Fleet stops deploying to that cluster until you unpause it. 1. Self-Healing On Turning self-healing on for a GitRepo that already had drift fixed the drift in about 11 seconds. Then I scaled the frontend from 3 to 1 again. Fleet had it back at 3, with 3 pods ready, in under 7 seconds. The GitRepo flickered NotReady for about 3 seconds and went back to Active. Anyone watching the UI would likely never see the drift. That is what you want from GitOps on a normal day. During an incident it works against you: your fix is gone before you can tell whether it helped. 2. Self-Healing On, Cluster Paused I paused the dev cluster and scaled the frontend to 5. It held, 5 of 5 ready, for the twelve minutes the cluster stayed paused. While the cluster was paused, the GitRepo read Ready, 2/2 clusters, with no Modified state anywhere. The app looked healthy in Continuous Delivery while one cluster ran something Git did not say. If you review drift by looking at the GitRepo, a paused cluster drops out of the review. When I unpaused, the frontend went from 5 back to 3 in about 9 seconds. The GitRepo showed Modified for a moment and then Active. The fix lasted exactly as long as the pause. 3. Self-Healing Off With self-healing off, the manual scale stays, and Fleet reports it precisely. I scaled prod from 2 to 4 and the GitRepo went to Modified, 2/3, and stayed there. The Ready condition read: Modified(1) [Cluster fleet-default/c-kqgtk]; deployment.apps shop/shop-podinfo modified {"spec":{"replicas":2} That is the best drift report in the product. Two details to know when you read one: it names the cluster by its ID rather than its display name, and the number it shows is the one Git wants. The value currently on the cluster is missing. The surprise came with the next commit. The repo deploys one app in two lanes: two cluster groups, non-prod and prod, and one fleet.yaml with prod pinned a chart version behind. Promotion is a one-line commit that bumps the prod target. I promoted podinfo 6.14.0 to 6.15.0 while the manual scale was still in place. The GitRepo sat in WaitApplied, 2/3, and prod stayed on the old version with 4 replicas. The Helm release on the prod cluster recorded the reason: Upgrade "shop" failed: conflict occurred while applying object shop/shop-podinfo apps/v1, Kind=Deployment: Apply failed with 1 conflict: conflict with "Mozilla" with subresource "scale" using apps/v1: .spec.replicas This is server-side apply doing its job. The manual scale made the browser's client the field manager of spec.replicas. "Mozilla" is the user agent the browser sent, so managedFields records the client that made the call. The person who clicked appears nowhere in the object. From a terminal the manager would be kubectl. When Fleet's upgrade tried to set spec.replicas back to 2, the API server refused to take the field from another manager without being forced. None of that showed in Continuous Delivery. The GitRepo showed WaitApplied and the drift message. The conflict text existed only in the failed Helm release on the downstream cluster. The fix needed no force. I scaled prod back to 2, the value Git asks for, and the promotion landed about 40 seconds later: new chart version, 2 of 2 ready. The Kubernetes docs explain why that works. When two managers set a field to the same value, they share ownership and there is no conflict. What to Check on Your Own Clusters Read the setting before the incident. Check spec.correctDrift.enabled on each GitRepo that targets prod, so you know which of the three cases you are in.If you pause a cluster, keep a list of what you paused. Continuous Delivery will not show drift on it until you unpause.If a promotion sits in WaitApplied after an incident, look on the downstream cluster. Run helm history on the release and read the description of the failed revision.Inspect ownership with kubectl get deployment <name> -n <namespace> -o yaml --show-managed-fields. A manager like "Mozilla" or kubectl next to a field that Git also sets is a manual change waiting to block a deploy. Limits of This Test One run, on Rancher 2.15.1 with Fleet and EKS 1.35 and 1.36. Every timing is a single measurement, and the GitRepos polled Git every 60 seconds. I scaled from the UI, not from kubectl in a terminal. I did not test correctDrift.force, which the Fleet docs say can recreate resources, or Helm's force option for this case. After the fix, the browser client may still co-own spec.replicas. Per the server-side apply docs, a later commit that changes replicas could conflict again. I did not test that. What I'd Take From It Pause is a working break-glass. Use it, and keep in mind that while a cluster is paused, Continuous Delivery tells you nothing about it.Put the fix in Git before you unpause, or you lose it.With self-healing off, an emergency change that never makes it back to Git blocks your next deploy. If a promotion sits in WaitApplied after an incident, look for a field someone changed by hand. This is a lab finding, not official guidance.

By Victor Estival

AI-Assisted Refinement Preparation

TL; DR: The Product Backlog Refinement Agent and the Risk When a Suggestion Becomes a Decision Preparing a refinement session is tedious, and AI is good at the tedious part: reading every Product Backlog item, every repeated comment column, and the team’s own records on the afternoon before. The risk sits elsewhere. A refinement agent asked to “help with refinement” can take the conversation away from the team without writing a single acceptance criterion: “suggesting” to the Product Owner or Product Manager which items to close, merge, split, or create is enough, and the questions the team was supposed to work out arrive in the brief already answered. The remedy is a boundary decided before the first run: the agent marks evidence and asks questions, the Product Owner checks the brief and sets the agenda, and the team refines. Writing that boundary into an A3 Handoff Canvas, together with what the agent reads and who checks its output, turns a good intention into a rule the agent follows and the Product Owner can verify. Thesis: This article explains how Product Owners and Product Managers can use AI to prepare Product Backlog refinement without pre-empting the team’s conversation: four evidence checks, a four-part test separating suggestions from decisions, written boundaries, and human verification of the agent’s brief. Disclaimer: I read Charniak/McDermott’s book on “Artificial Intelligence” decades ago; of course, I use AI for research, translations, proofreading, challenging story arcs and article structures, creating artifacts such as graphics, and summarization. It is a production tool, not a substitute for thinking. The Danger of a Preparation Turning Into a Decision The familiar worry about AI in Product Backlog refinement is that a model writes the Product Backlog items, the acceptance criteria, and the estimates, and a tired team nods them through on a Tuesday afternoon. The worry is justified; however, it describes the easy case, because a drafted acceptance criterion is visible, and a team would usually notice that someone else did its homework. The more challenging case looks like help. Consider a hypothetical brief by a refinement agent that contains no descriptions, no acceptance criteria, and no new order, yet recommends closing one item as a duplicate, splitting another, creating a third, and calling four items “ready candidates” for the next Sprint. Every line reads as friendly advice to the Product Owner, and every line settles a question before the team has discussed it. A recommendation decides nothing until someone accepts it and acts on it. Once it sits on the agenda, however, the team is likely to discuss the recommendation instead of the question behind it, and that is how preparation quietly turns into a decision. It sneaks in through the back door, feeding on the human desire to avoid unnecessary work when organizational pressure is already high. What Refinement Is For, and What Preparation Should Leave Alone The Scrum Guide describes refinement as “an ongoing activity to add details, such as a description, order, and size,” which captures the mechanics and leaves out the purpose. I see refinement as the final part of product discovery, once hypothesis validation has been established: the team asks whether an item is really the best use of its time or whether something else would create more value for customers. A Product Backlog item is a token for that discussion, an invitation to align, and a good session ends with shared understanding sufficient to make the next investment decision, including the decision not to build something. Many teams hold a regular session for this, and often those sessions turn into a hunt for problems someone could have spotted the day before: an item a stakeholder insists on moving to the top, a backlog item without acceptance criteria that everyone expected someone else to write, an item nobody has touched for 6 months. Preparation exists to spare the team that kind of administrative work, and reading a Jira export and the team’s notes on the afternoon before is exactly the kind of thorough, tedious work an AI agent does well. The question is where preparation ends. In September, I described ten anti-patterns that get worse when teams add AI to a dysfunctional Product Backlog process, “Refinement Without the Team” among them, and an assistant that hands the Product Owner ready-made resolutions is a polite version of that anti-pattern: the team still meets, yet the decisions it was supposed to reach already arrive pre-made. In my view, the most critical consequence of dysfunctional refinement is a team that loses its ability to have productive discourse. Typically, building the wrong thing follows; think of lack of return on investment and opportunity costs. So my rule for such a refinement agent is short: mark, suggest, never solve. The agent marks what the team needs to talk about and may suggest what to discuss, and solving stays the team’s job. Four Things a Product Backlog Refinement Agent Can Check, and What It Cannot Know Of the 27 Product Backlog and refinement anti-patterns I have described, many are behavioral and leave no trace in a Jira export. Four do leave traces, provided the agent reads the team’s own records next to the export: the Product Goal, the product roadmap with its change log, the anti-product backlog, the notes from previous Sprint Reviews, and the actions from the last few Retrospectives. Readiness The agent looks at items early in the current order and reports what the records contain: whether an item states a problem or an intended outcome, whether it lists open questions, and whether a Product Backlog item waits for an open spike for technical clarification. Whether that is enough for a useful conversation remains the Product Owner’s judgment. A missing acceptance criterion at the start of refinement is no defect; it is an agenda question, because the Product Owner will have ideas about how an item should meet the desired outcome, and the team negotiates the criteria together by the end of refinement. Being early in the order does not preselect an item for the next Sprint, either: the order reflects the team’s best current judgment of how to use its time, and in Sprint Planning, the team first agrees on the Sprint Goal, and then the Developers select Product Backlog items they deem necessary to accomplish the goal. Goal Fit and Prioritization by Proxy The agent compares the Product Backlog with the Product Goal, the roadmap, and the anti-product backlog, which lists the ideas and requests the team decided not to build, either not now or not in this product, with the reason for each decision. It marks items that serve no part of the Product Goal and items that revive an idea the team already rejected. It also marks prioritization by proxy, which an export reveals only indirectly, because no outsider has write access to the Product Backlog: a stakeholder can neither create an item nor change its position. What the export does show is a member of the Scrum Team, usually the Product Owner, creating or moving an item on a stakeholder’s behalf: a comment such as “per the sales director’s request,” a label such as “mgmt-request,” a creation date right after an escalation, and no discovery decision and no link to the Product Goal behind the item. An item that skipped discovery because someone with political weight asked for it was never put to the validation test. However, the agent cannot know what the conversation behind the item looked like, or whether a rejected idea deserves a second look; that takes the people involved. Outdated Items Items nobody has touched for months, or that still name a release that shipped long ago, deserve a look, because a Product Backlog that turns into a storage for ideas hides the work the Product Goal needs. Age is a signal for inspection, nothing more: an untouched item can be the very work the Product Goal depends on, and a refinement agent that simply sorts by the “Updated” column will happily declare it dead without inspection. Missing Work The Scrum Guide calls the Product Backlog “the single source of work undertaken by the Scrum Team.” The agent can hold that claim against the team’s records: open items from the Sprint Review, Retrospective actions, and even work done outside the Sprint Backlog. A discrepancy is a finding. Whether it leads to a new item is for the Product Owner and the team to decide, because the work may sit in another team’s Product Backlog, stay outside it on purpose, or already be resolved. This check is also why the agent needs more than the export, since an export cannot show what is missing from it. For any of the four checks, “no supported finding” and “insufficient evidence” are valid results. However, the teams that would gain most from this preparation often keep the fewest records: no anti-product backlog, no product roadmap change log, Sprint Review notes that say little more than “stakeholders loved what we built,” and the reasons behind past decisions stored in the heads of the Product Owner and a few senior engineers, or buried in Slack threads. Such a team will get a brief full of “insufficient evidence.” That brief is still a finding because it shows where the team has no memory of its own decisions. Also, it helps only when people read it as a statement about their records rather than a failed run. Otherwise, the team has added AI on top of a dysfunctional system. A small start is enough to make the checks useful: the Product Goal, the last Sprint Review notes, and the last Retrospective actions all cover readiness and missing work, and a short list of recent “no” decisions, one line of reason each, gives the goal-fit check something to work with. In the agent’s instruction, name only the records the team actually keeps, so the brief does not report the same missing roadmap change log on every run. Where a Suggestion Turns Into a Decision A hypothetical example shows where this red line runs; the item keys, people, and dates are invented. A Jira export taken on Monday shows EXP-187, “Export for enterprise accounts,” created last Tuesday by the Product Owner, in the second position of the order, with the Priority field set to “Highest,” a one-line description, and the Product Owner’s own comment “Per the sales director’s request, moved to the top; a big customer is waiting.” The roadmap’s change log keeps the export feature in “Next,” no discovery decision mentions the request, and EXP-112, a CSV export item from June, may cover part of the same need. A brief that does its job marks this as prioritization by proxy and cites its sources: the comment, the creation date, the position in the order, the roadmap entry, and the missing discovery decision. It adds what it cannot know (which customer, and what problem that customer has), and it may suggest that the Product Owner revisit the request with the sales director before refinement. A brief that crosses the line writes “Close EXP-187 as a duplicate of EXP-112,” or “Move EXP-187 back down the order,” or “Ready candidates for the next Sprint: EXP-187.” Each of these settles a decision that belongs to the Product Owner, however helpful it sounds. What follows is the Product Owner’s work, and most of it happens outside refinement. The sales director’s request is exactly what my discovery article describes as a passive source: “Product requirements from internal stakeholders, which arrive unrequested and usually with political weight attached.” It goes through the funnel I described in AI-Assisted Product Discovery, where only what serves the general product vision and the current product goal passes the neck and earns a place in the work: it may become a hypothesis to validate with an experiment, an entry in the anti-product backlog with its reason, or an item the Product Owner places where it belongs. Until then, EXP-187 has no claim on the second position, until the Product Owner reconsiders the order of the Product Backlog. A sales director who can push an item to the top through the Product Owner points to an impediment for the Product Owner and the Scrum Master. Also, it is not an agenda item for the Developers, as refinement time stays reserved for work that has legitimately entered the Product Backlog. Questions that belong in refinement need the same discipline. Take a story that waits for a spike that is still open: “What do we need to learn from the spike before we can negotiate the story’s acceptance criteria?” leaves the conversation to the team. “Should we split the story by permission level?” ends with a question mark, yet it carries a solution, and it leaves the team to say yes or no to an answer the agent chose. A valid refinement suggestion by an AI agent has to pass four tests: It rests on the supplied records,It makes the uncertainty visible,It leaves materially different answers open, andIt does not assume the disputed resolution. Why be this strict about a polite suggestion? A ready-made answer shapes the judgment of the person who reads it. In two preregistered experiments with 2,582 participants, Sterling Williams-Ceci, Mor Naaman, and colleagues found that people writing about societal issues with a biased AI writing assistant shifted their attitudes toward its position, most of them unaware of the bias and its influence, and warnings before or after the task did not mitigate the shift (Biased AI Writing Assistants Shift Users’ Attitudes on Societal Issues, Science Advances, 2026). The study tested autocomplete suggestions during writing, which influenced people more than similar arguments shown as a static list. An AI-assisted refinement brief is a static list, so the study is a reason for caution and does not prove that a brief anchors a Product Owner. Writing the Line Down Before the First Run Writing the red line down before a refinement agent runs for the first time turns it from an intention into a rule. The A3 Handoff Canvas is the place for it: it records what the agent reads, what its brief looks like, who checks it, what the agent must never do, and the last part goes into the agent’s instruction word for word. For an AI-assisted refinement agent, the stop rules read: Never write acceptance criteria,Never estimate,Never split, reorder, or remove an item, and never move one to the anti-product backlog,Never advise closing, merging, splitting, creating, or editing an item; ask the team a question insteadNever propose what goes into the next Sprint or what the Sprint Goal should beNever invent a source, a reason, or a figure. (If a record needed for a check was not supplied, cannot be located in the supplied material, or cannot be read, state that limitation under “What I could not check.” Do not infer that the record does not exist. Do not treat an experiment, proposal, or unverified note as a completed decision or shipped release; state the exact status recorded in the source.) Three of these rules deserve a word. Acceptance criteria stay out because the team negotiates them by the end of refinement, and an agent’s draft would preempt that negotiation; a different workflow can still help with criteria after the team has talked. Estimates stay out because their value lies in the Developers’ discussion that reveals diverging ideas of the effort before work starts, as I argued in Estimates Are Useful, Just Ditch the Numbers; the number was never the point. And the last rule addresses a quieter error: a model that reads a planned experiment as a commitment, or a hypothesis as a result, turns the team’s open question into a fact nobody decided. In terms of the A3 Framework, this preparation is Assist: “AI expands options. You own the outcome.” The refinement agent works from a read-only export, so it cannot change the Product Backlog even if a rule fails. And if you already run an agent for product discovery, it needs its own Handoff Canvas, because it reads different inputs, delivers different output, and stops at a different line; I consider that AI transparency at the operational level. The Product Owner Routes Every Finding A brief that sounds thorough invites a dangerous shortcut: forwarding it to the team unread. Polished AI artifacts quickly fool people into believing that the AI is doing the real job. So the Product Owner opens the item or record behind every finding that could change the agenda and then looks at a few things the brief did not flag, for example, the items nearest the top that it never mentions, because a brief cannot report its own omissions. Then the Product Owner decides where each finding goes, and only some of them belong in refinement: Settled by the Product Owner: Housekeeping, such as an obvious duplicate created by mistake or a release field nobody updated, and restoring the order of the Product Backlog.Sent into discovery: Requests that skipped the funnel, such as the sales director’s export, and rejected ideas whose revisit condition may have occurred.On the refinement agenda: Questions that need the team’s knowledge, such as what a Product Backlog item needs before the team can negotiate its acceptance criteria.Raised as an impediment: Systemic causes, such as stakeholder pressure that bypasses discovery or records the team does not keep, for the Product Owner and the Scrum Master, or for the next Retrospective. Sharing the agenda with the Developers and the Scrum Master before the session gives them the chance to add the questions the brief missed. If a brief proposes resolutions anyway, correct the instruction and run it again; if it keeps failing, prepare the agenda by hand, the way the Product Owner hopefully did before the agent existed. The remaining failure mode relates to the Product Owner: Treating the brief as sufficient grounds to settle a question that needs the team’s knowledge brings back the very problem the stop rules were meant to prevent, only with a human signature. The brief must not become a substitute for a discussion that has not happened. Conclusion: Try It Before Three Refinements, Then Decide Treat the first runs as a local experiment. Note how long your usual preparation takes, then run the refinement agent on a sanitized export, in an AI account your company has approved, before three or four refinements. Record the total time, including checking the findings that proved useful, the findings the records did not support, and the omissions you noticed, knowing you won’t catch all of them. If checking and correcting the brief keeps taking longer than preparing the agenda by hand, stop, because the agent then costs more than it saves. Otherwise, ask the team whether refinement gained time for the critical discussion, and compare the results with your usual preparation; a few sessions give you the basis to decide whether to continue, adjust, or stop. If you would rather practice first, the AI4Agile Online Course v4, which launches on November 27, 2026, includes an exercise in which you build this refinement preparation agent: you fill its A3 Handoff Canvas, run it on a 40-item Product Backlog export and the team records of MegaBrain.io, the course’s fictional startup, check its brief, and set the Product Owner’s agenda before comparing your decisions with my worked solution. When the next brief arrives on your desk the afternoon before refinement, which of its suggestions already contains an answer your team has not given yet? Key Questions This Article on the Product Backlog Refinement Agent Answers How Can AI Help a Product Owner Prepare Product Backlog Refinement? A refinement agent can read the Product Backlog export and the team’s records on the afternoon before refinement and mark Product Backlog anti-patterns: items not ready for a conversation, items that serve no part of the Product Goal or entered at a stakeholder’s request without a discovery decision, outdated items, and work the Product Backlog does not show. The Product Owner checks the brief and routes each finding; the team refines. What Can an AI Agent Check in a Product Backlog Before Refinement? Four checks leave traces in an export and a team’s records: readiness, goal fit, outdated items, and missing work. Prioritization by proxy shows only indirectly, because no outsider can write to the Product Backlog: an item the Product Owner created or moved up at a stakeholder’s request, visible in comments, labels, and a missing discovery decision. None of the checks can establish what the team understands or still wants; the people involved need to do that. How Do You Tell an Open Question From a Disguised Recommendation? A question passes four tests when it rests on the supplied records, makes uncertainty visible, leaves materially different answers open, and does not assume the disputed resolution. “Should we split the story by permission level?” fails despite its question mark, because it carries a solution. “What do we need to learn from the spike before we can negotiate the acceptance criteria?” leaves the answer to the team. Should a Refinement Agent Write Acceptance Criteria or Estimates Before the Team Refinement? Not in a preparation brief. The Product Owner brings ideas about how an item should meet the desired outcome, and the team negotiates the acceptance criteria together by the end of refinement; a refinement agent’s draft would preempt that negotiation. The value of an estimate lies in the Developers’ discussion of diverging ideas of the effort; the Scrum Guide assigns sizing to “the Developers who will be doing the work.”

By Stefan Wolpers

CORE

Tracing Dangerous Python Code Paths With rce-path

Finding an eval() call in a Python repository is easy. Understanding where its argument comes from takes more work. The same problem appears with shell commands. A scanner can point to subprocess.run(), but a reviewer still needs to trace the command through assignments, string formatting, and branches. Then comes the harder question: can someone outside the intended trust boundary control that input? I built rce-path to make that investigation more structured. It scans a local Python checkout, identifies modeled execution sinks, and records the input flow it can establish from the source. Each finding includes locations, ordered evidence, and the assumptions that still need review. The package also supports optional OpenAI and Ollama annotations. Static scanning works independently, so you can inspect the evidence before deciding whether to involve a model. The Problem I Wanted to Address A suspicious function call is a starting point for security review. It leaves several questions unanswered: Where did the argument originate?Which assignments or expressions carried it to the execution call?Does the path depend on a branch or an unknown value?Is the source request data, a function parameter, or local input?Which parts of the flow remain unresolved? These details matter because a function parameter is not necessarily remotely controlled. A request-derived value may pass through authentication or validation outside the scanner's view. A shell invocation using fixed literals needs a different investigation from one assembled from request data. My goal with rce-path is to give the reviewer a traceable path and make the remaining uncertainty visible. The Core Idea The current release, version 0.1.0, uses Python's abstract syntax tree and per-scope control-flow analysis to track modeled inputs toward execution sinks. A source is an input origin the analyzer recognizes. A sink is an operation that can execute Python code or a shell command. Between them, the analyzer follows supported assignments and expressions, including concatenation and f-strings. It also handles branches, early returns, and bounded loop analysis. The report preserves the steps connecting these operations. If a call's return value cannot be resolved, the scanner records that uncertainty rather than filling the gap with an assumed data flow. Targets are read as source data. A normal scan does not import the target package, install its dependencies, invoke its build hooks, or execute its code. What the First Release Covers The initial sink rules cover: eval() and exec(), including supported built-in aliases.os.system() and os.popen().Shell execution through subprocess.run(), call(), check_call(), check_output(), and Popen().subprocess.getoutput() and getstatusoutput(), which imply shell execution. Modeled sources include function parameters and Flask request inputs such as request.args, request.form, request.data, and request.get_data(). Environment variables, command-line input, stdin, and file reads are categorized as local inputs. The subprocess rules account for shell behavior. An unknown shell value introduces an explicit condition. Cases using shell=False have coverage limitations; they should not be interpreted as a general safety guarantee. This is a focused first release. It lets researchers and Python developers investigate supported execution paths without requiring an AI backend. A Small Example Consider this deliberately unsafe example. Save it as demo.py, but do not run it as an application: Python from flask import request import subprocess def lookup(): host = request.args.get("host", "") command = f"nslookup {host}" subprocess.run(command, shell=True) The security question is whether request data reaches a shell command. Here, the analyzer can model the request source, its assignment to host, its inclusion in the f-string, and the assignment to command before the shell invocation. There is no route registration in this snippet. It illustrates a data-flow pattern; it does not establish that the function is deployed or reachable over a network. To try rce-path, use Python 3.11 or newer and install the package from PyPI: PowerShell pip install rce-path Save the example as demo.py, then generate a Markdown report: PowerShell rce-path scan demo.py --format markdown --output report.md For this example, the report identifies the subprocess.shell rule at line 7, with static_path evidence and a network_request source. Its reachability field is network_source_modeled. The evidence follows the request read on line 5, assignment to host, f-string construction on line 6, assignment to command, and the execution sink on line 7. Those fields need careful interpretation. network_source_modeled means the analyzer recognized a network-input pattern. Authentication, deployment exposure, and actual exploit feasibility remain questions for human review. Using the Python API The command is named rce-path; the Python import is rce_path. The same scan is available through the public API: Python from rce_path import ScanConfig, scan if __name__ == "__main__": report = scan( "demo.py", config=ScanConfig(ai_enabled=False), ) print(report.summary) for finding in report.findings: print( finding.rule_id, finding.location.file, finding.location.line, ) for step in finding.steps: print( step.kind, step.location.line, step.description, ) The report and findings are typed Pydantic records. You can export them with report.to_json() or report.to_markdown() and use the structured fields in your own review workflow. Each finding carries a fingerprint, source category, evidence tier, source locations, and review state. Conditions, assumptions, and unresolved calls are available alongside the evidence. The report also records analyzed and skipped files, source content hashes, and coverage diagnostics. Fingerprints exclude the absolute checkout root and AI annotations, which helps compare the same finding across equivalent checkouts. They are location-based, so moving code can change a fingerprint. Where AI Review Fits An LLM can explain a finding, identify follow-up questions, or assess whether the available evidence supports an interpretation. In rce-path, that review is an optional annotation layer. Both provider implementations use a common evidence-and-review contract. Responses are validated, including references to evidence IDs. AI annotations do not rewrite the static finding or change its fingerprint. For a local Ollama review, install the extra: PowerShell pip install "rce-path[ollama]" Start your Ollama server and choose a model you already have installed: PowerShell rce-path scan demo.py --ai ollama --model YOUR_MODEL The default endpoint is http://127.0.0.1:11434. The package does not automatically download a model or fall back to a cloud provider. For OpenAI, install the corresponding extra and set OPENAI_API_KEY in your environment: PowerShell pip install "rce-path[openai]" rce-path scan demo.py --ai openai --model YOUR_MODEL --allow-remote-code YOUR_MODEL is a placeholder for an operator-selected model that supports the provider's required structured output. OpenAI and non-loopback Ollama endpoints require explicit permission to transmit code. The package sends bounded evidence slices and applies best-effort redaction of known secret patterns. Redaction can miss sensitive content, so review what you permit it to transmit. OpenAI requests use store=False, which is not a promise of zero retention. Review failures stay visible. A refused, unavailable, invalid, or budget-exceeded review is reported as unsuccessful rather than being presented as a completed assessment. Running It in a Review Pipeline For machine-readable output, scan a local checkout and save JSON: PowerShell rce-path scan ./checkout --format json --output findings.json You can also choose a findings policy: PowerShell rce-path scan ./checkout --fail-on static-path This fails the selected policy when the report contains a modeled static path. It does not mean the package has confirmed an exploitable vulnerability. The CLI uses three exit codes: the scan completed and the selected policy passed.the findings policy failed.an operational error, incomplete scan, or failed requested AI review occurred. The default policy is --fail-on none. Completion status takes precedence over the findings policy, so an incomplete scan cannot be treated as a successful clean scan. A repository-level rce-path.toml can configure rules, exclusions, and analysis budgets. Repository configuration cannot enable AI or grant permission to transmit code; those decisions require runtime opt-in. Current Limits rce-path uses bounded static analysis. Cross-function and cross-file flow, dynamic dispatch, closures, complex heap mutation, comprehensions, and precise exception handling are outside its current coverage. Branches are joined conservatively. A reported path can include conditions that are not jointly feasible at runtime. An unknown function is not automatically treated as a sanitizer, and a guard does not by itself prove an argument safe. For that reason, static_path means the analyzer established a path within its models. A sink_only finding identifies a sink whose input flow remains unresolved. Neither establishes remote code execution, and the package assigns no automatic CVSS score or exploitability probability. Read the coverage diagnostics alongside the findings. An empty completed report means "No modeled findings," not that the repository has been proved secure. What Comes Next The roadmap includes bounded local call summaries, additional framework and deserialization models, SARIF output, and separate human-approved validation workflows. These capabilities are planned for later phases and are not implemented in the current release. For now, I want rce-path to make a specific part of vulnerability review easier to inspect: how an input reaches an execution operation, what evidence supports that path, and what remains unknown. Start with a small file whose behavior you understand. Read the evidence steps, check the assumptions, and then apply the same workflow to a local repository. That is the most useful way to judge whether the package fits your research or code-review process. Project Links PyPIGitHubUsage and limitations

By Sai Teja Erukude

Why Successful Tech Transformation Starts With Clarity

Every generation of technology arrives with the same expectation: this time, organizations will transform successfully, absorb the new technology, and reap the benefits. And sometimes they do. Yet after decades of technological progress, one pattern remains consistent. Organizations continue to spend valuable resources and dollars on technology initiatives that fail to achieve their intended outcomes. They fail because organizations attempt to automate what they don’t understand. They fail because the strategic purpose is not defined, change-readiness is not assessed, cultural barriers are not explored, and the outcomes expected are not clarified. Projects overrun. Adoption stalls. Strategic priorities drift. Employees become cynical. Competitive advantage slips away. A Pattern That Repeats With Every New Technology The history of business technology is a narrative of change. Mainframes gave way to personal computers. Enterprise applications consolidated isolated systems. The Internet created a global network. Cloud computing redefined infrastructure. Agile promised a better way to build software. Today, artificial intelligence promises another transformation. Technology is tangible. It can be purchased, implemented, configured, and deployed. Organizational clarity cannot. It requires conversation, alignment, judgment, and difficult choices. Those choices are often postponed because technology appears to offer a shortcut. Technology does not resolve organizational ambiguity. It exposes it. What Technology Can’t Decide For You An ERP system cannot decide how purchasing should work. A CRM system cannot define what constitutes a customer. Analytics cannot settle disagreements about which metrics matter. Artificial intelligence cannot compensate for unclear objectives or conflicting priorities. Every technology faithfully reflects the decisions that came before it. When those decisions are thoughtful, technology amplifies them. When they are confused, technology scales the confusion. This lesson has resurfaced in every technological era. The industry has repeatedly searched for technical solutions to organizational problems. We debated Waterfall versus Agile. On-premises versus cloud. Build versus buy. Low-code versus custom development. Today we debate copilots, agents, and autonomous systems. These are important decisions. None of them substitutes for clarity. Methodologies improve execution. Platforms expand capability. Artificial intelligence accelerates delivery. But no methodology, platform, or model can determine why an organization is changing, what success should look like, who owns the decisions, or how people will ultimately work. Clarity Before Technology This is why our engagement philosophy begins with clarity before technology. Clarity comes from ensuring that before the conversation turns to technology, software, infrastructure, or AI agents, leaders should have clear answers in the following dimensions: Purpose: Why are we making this change?Outcomes: What benefits will we gain when this effort succeeds?Ownership: Who has the authority to make decisions?Process: How does the work actually happen now? What will it look like after the initiative succeeds?Priorities: What matters most when tradeoffs arise?Tradeoffs: What are we willing to sacrifice to achieve something valuable?Governance: What constraints, risks, or obligations must shape our decisions?Culture: How much change can our teams absorb? How will we prepare them to succeed in the new environment? The Cost of Skipping This Step The rush to technology doesn’t answer these questions. It amplifies the lack of answers. The consequences of ignoring them extend far beyond an unsuccessful implementation. Organizations lose momentum. Investments fail to produce strategic advantage. Executive attention is consumed by avoidable challenges. Employees stop believing the next initiative will be different. Competitors, often with fewer resources, move ahead because their direction is clearer. The cost of ambiguity isn’t measured by a project plan. We’ve learned the lesson repeatedly that projects can be on time, on budget, and in scope, yet still deliver no business benefit. Lack of clarity in these dimensions is measured in lost opportunity and squandered advantage. Why AI Raises the Stakes And now, artificial intelligence has made the clarity discipline even more important. For the first time, organizations can prototype capabilities in days that once required months of engineering effort. Building has become dramatically easier. Experimentation has become far cheaper. This is an extraordinary opportunity, but it can also pose immense risk. The easier technology becomes to build, the easier it becomes to build the wrong thing. The faster an organization can act, the more certain it must be that it is acting on the right objectives, with the right governance, and in service of the right strategy. How Much Clarity Is Enough? Beginning by establishing clarity before turning to technology manages these potential pitfalls. This doesn’t have to mean that every initiative must begin with lengthy planning exercises. Because in truth, not every initiative requires months of analysis. Some decisions require only a few hours of focused alignment. But others deserve deeper examination. The discipline should always match the complexity and risk of the work. Navor’s engagement philosophy is to help organizations reach the level of understanding necessary to make the right investment decisions, execute them with confidence, and adopt change that lasts. Technology is one of the most powerful forces available to modern organizations. The potential of AI is monumental. The decisions firms make now require clear objectives, clear metrics, clear expectations. Successful transformation begins with clarity.

By Rick Freedman

Member Spotlight: Srinivas Chippagiri

Have you considered what the defining infrastructure challenges of the next few years will be? If not, my interview with Srinivas Chippagiri may have you thinking more about how AI could change the way you manage infrastructure and what that means for building reliable, secure systems. But before we get into all of that, I always like to start by learning a little more about the person behind the expertise. So, let's kick it off with where Srinivas's journey began! Tell me what initially drew you into the field of technology. My journey began in software engineering, drawn to what I still think is the hardest question in the field: how to build software systems reliably at scale? I didn’t want to just build systems. I wanted to understand and document why they behave the way they do, which is what led me into research. Over the years, I’ve published peer-reviewed papers and book chapters on cloud security, Kubernetes performance, distributed scheduling, and applied AI. For me, engineering and research feed each other. Building at scale gives me the hard problems, and writing about them forces the clarity that makes the next system better. You mentioned that engineering and research feed into each other, which, for some developers, can make it a bit harder to navigate. When you're faced with a difficult technical problem, how do you typically approach finding a solution? I start by quantifying the problem before I touch a design. Who depends on this, what breaks if it fails, and what “good” actually looks like in numbers. Then I design for reuse rather than for the immediate fix. A lot of my research work comes directly out of that discipline. My work on Kubernetes network performance and on distributed task scheduling and load balancing using nature-inspired optimization began as real engineering problems that I then studied rigorously and published. That loop is my process: observe a hard problem in production, model it, measure it, and generalize it into something reusable. I trust measurement over intuition, and I try to leave behind a pattern others can apply, not just a one-off solution. I like that you focus on creating solutions others can reuse rather than just addressing the immediate problem. With that in mind, what's something you wish more people understood about cloud-native development? That “moving to the cloud” and “being cloud-native” are not the same thing. Many teams lift-and-shift a monolith onto managed infrastructure, watch the bill climb, and conclude cloud-native was oversold. The real win isn’t the platform. It’s designing systems as reusable, observable, governed building blocks, with security and compliance treated as engineering problems from day one rather than checklists bolted on after an incident. I’ve argued this in my research too. Several of my papers and book chapters look at cloud security and the organizational side of cloud transformation, because the hardest part of cloud-native isn’t the technology. It’s the discipline and architecture around it. You brought up the importance of building systems with security and compliance in mind from the start. Looking ahead, what emerging technology or trend has your attention right now, and why? Applied AI in infrastructure, meaning using AI not just as the workload but to operate and self-manage the systems underneath it. It sits right at the intersection of my two research threads: distributed systems and applied AI. My published work has gone deep on both sides. In a recent journal article, my co-authors and I designed an IoT-based AI framework for real-time health monitoring that pairs edge and fog processing with federated learning and differential privacy, so intelligence runs close to the data while privacy is preserved. On the systems side, I’ve studied scheduling and optimization. What fascinates me now is the reliability question. As we let intelligent systems make operational decisions, how do we keep them observable, safe, and accountable? That’s the same reliability-first instinct that has driven my entire career, applied to a far more autonomous stack, and I think it’s the defining infrastructure challenge of the next few years. It's interesting that reliability has remained a priority throughout your career, even as the technology evolves. What's one lesson you've learned along the way that's had a lasting impact on how you approach your work? That your highest-impact work is the work other people can build on. That reframed how I work. I now deliberately design and document for reuse, whether that’s a system architecture or a paper. I optimize for the multiplier, not just the deliverable, and the recognition I value most isn’t finishing something. It’s seeing my approach become a pattern others adopt. And now for my final question, which is usually my favorite: When you’re not working or thinking about technology, how do you like to spend your free time? A lot of my off time still points back toward the community. I’m active in the professional cloud computing community, and I mentor younger engineers and researchers, which I genuinely enjoy. Outside of tech, I like unhurried time with family and getting away from screens entirely. Travel and being outdoors help me return to hard problems with a clearer head. I’ve found the best architectural ideas rarely arrive at a keyboard. Check out more of his content here.

By Dominique Pugh

Engineering Data Observability: Detecting Schema Drift, Data Drift, and Silent Pipeline Failures

Modern data platforms are increasingly responsible for powering critical decisions, applications, analytics systems, and AI workloads. As organizations move more business processes onto data-driven architectures, the reliability of data pipelines becomes as important as the reliability of application services. A pipeline that fails loudly is usually easy to diagnose. A failed job, broken dependency, or unavailable service typically generates an alert and directs engineers toward a specific issue. The more difficult problems are the ones that do not interrupt execution at all. A pipeline can complete successfully while producing incorrect, incomplete, or misleading data. This is where data observability becomes essential. Traditional monitoring focuses primarily on whether systems are running. Data observability focuses on whether the data flowing through those systems remains trustworthy. It examines changes in structure, volume, freshness, distribution, and behavior to identify failures that traditional operational monitoring cannot detect. The Problem With Silent Data Failures Modern data ecosystems are rarely static. Enterprises often operate hundreds or thousands of pipelines connecting transactional systems, applications, event streams, warehouses, and analytical platforms. These pipelines evolve continuously as teams add new features, modify applications, and introduce new data sources. A database team may add a new column to a production table. An application team may change the format of an event payload. A third-party provider may alter the frequency or structure of delivered data. Each change may be intentional and technically valid, but it can create unexpected consequences downstream. Consider a customer analytics pipeline that receives transaction events from multiple systems. The pipeline may continue processing records after an upstream application changes a field format from a numeric value to a string. The ingestion process may not fail. The transformation jobs may still complete. Dashboards may continue refreshing. However, the business logic depending on that field may now produce incorrect results. The challenge is not detecting that a system is unavailable. The challenge is detecting that a system is operating while producing unreliable output. Detecting Schema Drift Before It Creates Downstream Issues Schema drift occurs when the structure of data changes over time. This can include adding or removing fields, changing data types, renaming attributes, or modifying relationships between datasets. Many organizations still rely on static schemas and manual documentation to manage these changes. While these approaches provide value, they often cannot keep pace with highly distributed environments. Data observability introduces automated schema monitoring that continuously compares incoming data against expected patterns. Instead of only checking whether a pipeline completed, observability systems examine whether the data structure remains consistent. For example, an observability layer can identify: A previously required field becoming consistently emptyA new field appearing unexpectedly in event streamsA data type changing from integer to textA significant reduction in the number of columns arriving from a source system The goal is not to prevent every schema change. Modern systems need to evolve. The goal is to identify changes quickly, understand their impact, and determine whether they represent expected evolution or an unintended failure. Understanding Data Drift Through Behavioral Changes Schema validation only examines structure. Many data quality issues happen without any structural change. Data drift occurs when data characteristics change over time. A dataset can maintain the same columns while its underlying patterns shift significantly. Examples include: A sudden increase in transactions from a specific regionA sharp decrease in average order valuesUnexpected changes in customer behavior patternsA growing percentage of missing valuesAbnormal distributions in machine learning features These changes are particularly important for AI and analytics systems. Machine learning models often assume that incoming data resembles the data used during development. When production data changes significantly, model performance can degrade without any visible infrastructure failure. Data observability helps teams establish baselines for normal behavior. Instead of defining every possible failure condition manually, systems can analyze historical patterns and identify statistically unusual changes. Combining Deterministic Rules With Intelligent Detection A common misconception is that data observability requires replacing traditional validation with machine learning. In practice, reliable data platforms need both deterministic checks and intelligent anomaly detection. Deterministic validation remains valuable because some conditions are clear. A customer identifier should not be null. A timestamp should follow an expected format. A transaction amount should not be negative. These rules provide strong guarantees for known requirements. However, not every failure can be captured through predefined rules. Unknown issues require systems that can recognize unusual behavior. Statistical and machine learning approaches can analyze: Record volume changesDistribution shiftsUnusual null patternsProcessing delaysUnexpected relationships between datasetsChanges in key business metrics The combination creates a layered defense. Rules catch known problems. Intelligent detection identifies emerging problems that engineers did not anticipate. Data Lineage: Understanding Where Problems Begin Detecting an anomaly is only the first step. The harder question is determining where the issue originated. Large organizations often have complex dependency chains. A single dashboard may depend on dozens of upstream tables, transformations, and source systems. Without visibility into these relationships, engineers may spend hours investigating symptoms rather than causes. Data lineage provides a map of how information moves through the organization. When an anomaly appears, lineage allows teams to answer important questions: Which pipelines depend on this dataset?Which upstream source introduced the change?Which downstream applications may be affected?When did the behavior begin changing? This turns incident response from manual investigation into a structured troubleshooting process. Building Effective Alerting and Root Cause Analysis Poor alerting can create another operational challenge. If every unusual data point generates an alert, engineering teams quickly experience alert fatigue. If alerts are too limited, important failures remain unnoticed. Effective data observability requires context-aware alerting. A sudden increase in records may not indicate a failure if it aligns with a known business event. A small change in a critical dataset may be more significant than a large change in a less important pipeline. Alerts should incorporate factors such as: Business importance of the affected dataSeverity of the anomalyHistorical behaviorDownstream dependenciesConfidence level of detection The objective is not simply generating more notifications. It helps engineers prioritize issues that require action. Designing Reliable Data Platforms for the Future As organizations adopt real-time analytics, AI applications, and increasingly complex data architectures, data reliability becomes an engineering discipline of its own. Reliable pipelines require more than successful job execution. They require confidence that the information moving through those pipelines remains accurate, complete, and useful. A mature data observability strategy combines multiple capabilities: Automated schema monitoringData quality validationStatistical anomaly detectionFreshness trackingPipeline performance monitoringData lineageIntelligent alertingRoot cause analysis The future of data engineering will not be defined only by how much data organizations can process. It will be defined by how confidently they can trust that data. Data observability provides the foundation for building that trust. By detecting silent failures before they become business problems, engineering teams can create data platforms that are not only scalable but also dependable.

By RISHI SHRIVASTAVA

Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check

Everyone is rushing to build AI agents right now, but GenAI integration really just puts a stronger spotlight on solid system design and backend reliability. An AI agent relies entirely on the APIs and data pipelines underneath it. If your backend drops events or double-processes payments during a network blip, even the best agent will just execute a broken workflow faster. To build resilient systems, you need to see how they break. Reading Kafka documentation on fault tolerance helps, but intentionally introducing network chaos shows how the underlying mechanics actually behave. In this post, we will set up a local testing environment using Spring Boot, Kafka, and Toxiproxy. We'll configure producer properties for topic-level idempotence and walk through common network simulation pitfalls that lead to duplicate data. 1. Setting Up the Chaos Test Architecture To test network issues accurately, stopping the Kafka broker isn't enough. We need to simulate realistic network conditions like latency spikes, packet drops, and bandwidth throttling. Toxiproxy acts as a proxy between our producer and broker to simulate these conditions. The Network Architecture Creating a Local/Dev Kafka Cluster We start by spinning up the Kafka cluster and Kafka UI using Docker. YAML # docker-compose-kafka.yml version: "3.8" #avoid this services: kafka: image: apache/kafka:latest container_name: kafka ports: - "9092:9092" environment: CLUSTER_ID: "4L62xdwCQKWvZs96A39vGA" KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL" # Configure distinct internal/external routing ports KAFKA_LISTENERS: "INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" # We will bookmark this line, dear reader! KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:19092" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: "/var/lib/kafka/data" volumes: - kafka_data:/var/lib/kafka/data kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "9080:8080" depends_on: - kafka environment: DYNAMIC_CONFIG_ENABLED: "true" KAFKA_CLUSTERS_0_NAME: "local-cluster" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092" volumes: kafka_data: To spin it up, we use Docker Compose: docker compose -f "docker-compose -kafka.yml" up Both your local Kafka cluster and the kafka-ui should spin up. Note that KAFKA_ADVERTISED_LISTENERS includes the external port localhost:19092; we'll see why. A Local Proxy to Sit Between Kafka Broker and Producer We use Docker for this, too! YAML # docker-compose-toxiproxy.yml version: "3.8" # Don’t use this version - it’s deprecated services: toxiproxy: image: shopify/toxiproxy:latest container_name: toxiproxy ports: - "8474:8474" # Toxiproxy Management API - "8000:8000" # Example port exposed for a proxied application service - "19092:19092" networks: - toxiproxy-net toxiproxy-ui: image: buckle/toxiproxy-frontend:latest container_name: toxiproxy-ui ports: - "8080:8080" # We don’t use UI in this article environment: - TOXIPROXY_URL=http://toxiproxy:8474 depends_on: - toxiproxy networks: - toxiproxy-net networks: toxiproxy-net: driver: bridge docker compose -f docker-compose-toxiproxy.yml up Once running, create a proxy tunnel via the Toxiproxy CLI to listen on port 19092 and forward traffic to Kafka on port 9092: ./toxiproxy-cli create -l 0.0.0.0:19092 -u host.docker.internal:9092 kafka-proxy Note: We use the above CLI inside the toxiproxy container. So, you must use docker exec. In some versions, the CLI tool might not be included in the $PATH, and you might need to switch the current directory: Shell /go/bin # ls toxiproxy toxiproxy-cli If you're using this toxiproxy tool for the first time, you can refer to the official Toxiproxy documentation or this starter guide we created. 2. The Kafka Client Producer Code We create a basic application using Spring Boot 4.1 (though we do not have any version restriction until now) and use the following dependency: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-kafka</artifactId> </dependency> We'll keep the Spring Boot service simple. We define a basic payload configuration and direct messages to topic T3 via KafkaConfig.java. Config: Java // KafkaConfig.java package com.soham.selfteaching.kafka.kafkadeepdive.config; import lombok.Data; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @Configuration @Data public class KafkaConfig { @Value("${app.topic.name:T3}") private String topic; } Our application.yml is minimal, accepting defaults provided by the Kafka library. Properties files # application.yml server: port: 8082 spring: application: name: kafka-deep-dive kafka: bootstrap-servers: "localhost:19092" producer: acks: "1" # Required to bypass idempotence shield key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: request.timeout.ms: 3000 delivery.timeout.ms: 120000 enable.idempotence: false # We disable it first before we explore producer level idempotence The domain event is defined as a Java record. Java // EventDto.java package com.soham.selfteaching.kafka.kafkadeepdive.dto; public record EventDto(String id,String payload) { } Producer Code The core logic sits in ProducerService.java. We use CompletableFuture to handle delivery timeouts asynchronously without blocking thread execution. If the broker is unreachable initially, max.block.ms handles the immediate block on the main thread while background retries handle transient issues. Java // ProducerService.java package com.soham.selfteaching.kafka.kafkadeepdive.service; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.kafka.support.SendResult; import org.springframework.stereotype.Service; import java.util.concurrent.CompletableFuture; @Service @Slf4j @RequiredArgsConstructor public class ProducerService { private final KafkaTemplate<String, Object> kafkaTemplate; public void sendMessage(String topic, String key, Object payload) { log.info("Attempting to send message to topic: {} with key: {}", topic, key); CompletableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, key, payload); future.whenComplete((result, exception) -> { if (exception == null) { log.info("Message successfully delivered. Topic: {}, Partition: {}, Offset: {}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } else { log.error("Fatal delivery failure for key {}. Exhausted delivery.timeout.ms.", key, exception); } }); } } 3. Configuring Topic-Level Idempotence During a network timeout, the producer cannot verify whether the message failed to reach the broker or if the acknowledgment (ACK) was lost on the return path. By default, the producer retries the request, which can produce duplicate records without proper safeguards. Exactly-Once Semantics (EOS) and topic-level idempotence address this by assigning a producer ID (PID) and sequence number to each message batch. If the broker receives a retried batch with a sequence number it has already processed, it ignores the duplicate write. Disabling Idempotence to Force Duplicates To demonstrate retry behavior, we can explicitly disable these safety settings in application.yml. Since Kafka 3.0, enable.idempotence defaults to true. To allow duplicates during testing, set acks: "1" and explicitly set enable.idempotence: false. Setting request.timeout.ms to 3000 (3 seconds) creates a tight window to trigger network retries during latency testing. 4. Running the Kafka Producer and Verify We run the Spring Boot application: Java @SpringBootApplication public class KafkaDeepDiveApplication { public static void main(String[] args) { SpringApplication.run(KafkaDeepDiveApplication.class, args); } } And hit a REST API (Hold on! It's a simple HTTP API which may NOT quite follow REST paradigms). curl -X GET http://localhost:8082/api/send Java @RestController @RequestMapping("/api") @Slf4j @RequiredArgsConstructor public class ProducerController { private final ProducerService service; private final KafkaConfig kafkaConfig; @GetMapping("/send") public String send() { long ts = System.currentTimeMillis(); Object payload = new EventDto("p1.t1." + UUID.randomUUID().toString(), ("Hello message " + i)); service.sendMessage(kafkaConfig.getTopic(), "k1", payload); return "Process took :%s ms".formatted(System.currentTimeMillis() - ts); } } In the case when idempotence is disabled, it's fair to expect that duplicate messages will arrive on a topic if a producer does not receive timely ACK from the broker leaders in alignment with the ack flag. But we do not yet see any duplicate messages at all. This is fairly explainable, since you're all set up locally, and there's usually no lost acknowledgment. Introduce Chaos to the Local Network Add 5000ms latency to the proxy. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy Send a message to the endpoint. After an API hit, wait for 2–3 seconds and remove the toxic Remove the Toxic While the Retry Is in Flight Instantly run this command to remove the toxic while the retry is in flight: ./toxiproxy-cli toxic remove -n false_positive kafka-proxy While this should have produced duplicates due to retries by the producer following a timed-out broker ACK, we find NO DUPLICATE MESSAGES even after repeating the process above many times. 5. Let Us Debug Our Disconnect Common Pitfalls During Network Injection Simply starting the proxy and adding latency doesn't always trigger duplicates as expected. Below are two common issues related to how Kafka clients handle metadata and connections. Issue 1: Advertised Listeners Bypass If Toxiproxy is running on port 19092 and a 5000ms latency is emulated, messages sent through the proxy might still complete in milliseconds. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy This occurs because of how Kafka advertises broker addresses during metadata discovery. If you look closely at the internal Kafka NetworkClient logs during startup, you will find out why: Plain Text brokers=[MetadataResponseBroker(nodeId=1, host='localhost', port=9092)] When the producer connects to port 19092 for initial cluster metadata, the broker returns its configured advertised listener (e.g., localhost:9092). The producer client caches this response and routes subsequent produce requests directly to port 9092, bypassing the proxy entirely. We must ensure that our KAFKA_ADVERTISED_LISTENERS explicitly broadcasts EXTERNAL://localhost:19092 so the producer continues routing traffic through Toxiproxy. If updated, clear existing Docker volumes (docker-compose down -v) to avoid using cached listener configurations. Issue 2: Initial Metadata Fetch Blocking If high latency (e.g., 15,000ms) is applied before sending any messages, the application may time out after 60 seconds with a TimeoutException: Topic T3 not present in metadata, rather than failing on delivery.timeout.ms. The delivery.timeout.ms timer only applies once a record enters the producer's accumulator buffer. If the client hasn't fetched topic metadata yet, the call blocks on max.block.ms (defaulting to 60 seconds) while attempting to retrieve metadata through the degraded network path. We apply the fix: Send an initial message without active toxics to populate the local metadata cache before injecting network latency. Finally With latency removed, the retried request reaches the broker successfully. Because acks=1 and enable.idempotence=false, the broker writes the payload again without verifying duplicate sequence numbers. Checking Kafka UI confirms two identical records are present in the topic partition. Simulating network edge cases directly demonstrates why configurations like advertised listeners, metadata timeouts, and idempotence defaults exist in modern Kafka clients. Now we get duplicate messages across a topic partition. Duplicate Messages We can now enable back idempotence flag to see that no duplicate messages are created repeating the above steps with enable.idempotence:true, ensuring that idempotence is preserved enable.idempotence= true only ensures idempotence across a partition only and not the entire topic. We shall explore it in a next article.

By Soham Sengupta