DZone Spotlight

We all saw the rise and fall of GraphQL. The technology was hip at the time, and then we discovered it was slow, very complex, and it was easy to shoot yourself in the foot on security. REST won that fight. One major factor that went in favor of REST was that every language speaks it, every developer understands it, and you don’t need to run a special server just to serve a GraphQL API. But does this still stand true in the age of AI? Let us try to unpack this question and see if this time it could be different for GraphQL? There’s a New API Consumer, and It Doesn’t Think Like Humans For years, APIs had two audiences: first, the services (predictable, hard-coded integrations) like APIs talking to APIs, and humans using apps (who don’t mind a bit of extra data; nobody notices 40 fields traveling across the wire while the screen only renders 10 fields). AI agents are a third audience, and they behave nothing like the first two. Think of it like this: a human browsing a shopping site doesn’t care if the product page quietly loads size charts, reviews, and shipping data if the human is not interested in those. An AI agent, though, has to read every field it’s handed, and every one of those fields sits in its memory, costing money and crowding out the things it actually needs to think about. It’s less like browsing and more like being handed the whole filing cabinet when you asked for one folder. Over-Fetching Isn’t Just Wasteful for Agents; It’s Expensive in a Different and Costly Currency Let us assume an agent asks “who manages this account?” A typical REST endpoint hands back the entire user record, the email, address, and ten other fields. This is because building a trimmed-down endpoint for every possible question is a lot of upfront engineering work. A human skims past the noise. An agent has to carry it around for the rest of the conversation, like packing your whole closet for a weekend trip because folding a smaller bag felt like too much effort. GraphQL flips that: the agent asks for exactly “manager name and email,” and that’s all that comes back. The N+1 Problem, Agent Edition Anyone who’s worked with databases knows the pain: you fetch a list of 10 orders, then make 10 more calls to get customer details for each one. REST APIs often have the same shape. For an agent, every one of those round trips is another context-window hit and another few seconds of latency, like sending ten separate texts instead of one paragraph. GraphQL lets the agent ask for orders and their customers in a single request. A Schema the Agent Can Actually Read REST documentation is a promise: “this is what the API looks like, we hope, as of whenever someone last updated the docs.” When it drifts out of date, an agent’s fallback is basically the same as a stressed junior developer’s: search the web, then go read the source code. GraphQL bakes the documentation into the API itself. The agent can ask the server, at runtime, “What exists, what does it need, what’s deprecated?” It’s the difference between asking a new coworker to guess your team’s tools from an outdated wiki page, versus just asking the tool itself how it works. Security That Matches How Agents Actually Work Most REST permission systems are coarse, calendar.read, repos.write and so on. Fine for a human logging into one app with one role. But an agent might handle customer support in one breath and billing cleanup in the next, and you don't want it holding a master key for both. GraphQL checks access field-by-field, not just endpoint-by-endpoint. That means you can grant an agent “read the customer’s name” without also granting “read their payment history”, even if both live on the same object. It’s the difference between giving someone a key to the building versus a key to one specific drawer. Errors an Agent Can Actually Act On REST failure: “400 Bad Request.” Sometimes JSON, sometimes an HTML page; format varies by provider and sometimes even within the same provider. GraphQL failure, “the field user.team.name failed, no read access on team 7." That's something an agent can act on directly; it can even retry a different query, ask for permission, or explain the problem to a person instead of burning another model call just to figure out what went wrong. Where REST Still Wins, and Probably Always Will This isn’t “GraphQL beats REST.” Caching is nearly free with REST; every CDN on earth understands it natively. GraphQL caching is a genuine engineering project. Uploading a file or streaming video over REST is simple; doing it over GraphQL is awkward. And running a GraphQL server is real operational overhead REST doesn’t have. So what’s the actual comeback? Not GraphQL replacing REST for humans and services. More like this shape, Human or agent → MCP server / CLI tool → GraphQL → your actual backend Today, most people bolt an MCP server onto REST, then hand-build the exact “shape” of every response, field by field, tool by tool, basically reinventing what GraphQL already does natively. Put GraphQL underneath instead, and the MCP layer can just pass the agent’s query straight through, precise fields, typed schema, field-level permissions, structured errors, all included. I’m not saying rip out your REST APIs. I’m saying the layer sitting between AI agents and your systems might quietly end up looking a lot like GraphQL, and if you’re building tools for agents right now, this is worth an experiment.

Citizen development was supposed to free up IT teams, not give them a new category of risk to manage. Yet that is precisely what has happened in many organizations running Microsoft Power Platform at scale. Business users build apps, automate workflows, and connect data sources at a pace that traditional governance models were never designed to keep up with. Each new app or flow is a small decision about data access, and when hundreds of these decisions are made independently across departments, the result is a security posture nobody fully understands. The instinct to lock everything down defeats the purpose of low-code platforms in the first place. The real objective is to enable rapid development while keeping data, connections, and environments under deliberate control. Microsoft has built a substantial set of security and governance capabilities directly into Power Platform for exactly this reason, but they only work when an organization actually configures and enforces them. Left on default settings, the platform favors flexibility over restriction, and that gap is where most enterprise security gaps quietly form. In this blog, I will discuss the core security controls within Power Platform and the governance practices that make enterprise-grade security achievable without slowing down development. Core Security Controls Within Microsoft Power Platform Power Platform's security model is built around environments, data policies, and connector restrictions, working together to contain what any single app or flow can reach. Understanding how these controls interact is the starting point for any serious governance effort. Environment strategy and segmentation: Environments are the primary security boundary in Power Platform, and a flat, single-environment setup is one of the most common governance failures organizations make. Separating development, testing, and production environments prevents experimental apps from touching live business data. Environments can also be scoped by department or business function, so that a Dataverse database in one environment is not implicitly reachable from apps built elsewhere. Assigning environment-level roles through Microsoft Entra ID security groups, rather than individual user accounts, keeps access manageable as teams grow and change.Data Loss Prevention policies for connectors: DLP policies classify connectors into business, non-business, and blocked groups, controlling which data sources can be combined within a single app or flow. Without this control, a maker could unintentionally connect a corporate SharePoint site to a personal Gmail account in the same flow, creating an unmanaged path for sensitive data to leave the organization. Tenant-level DLP policies provide a baseline, while environment-level policies allow tighter restrictions for sensitive business units such as finance or HR. Reviewing connector classifications quarterly matters, since Microsoft regularly adds new connectors that need to be triaged before makers discover them first.Dataverse security roles and field-level protection: For apps built on Dataverse, security roles define exactly what a user can view, create, edit, or delete, down to the level of individual tables and records. Business units within Dataverse allow record-level ownership to mirror organizational structure, so a regional sales record is only visible to the relevant team. Column-level security adds another layer by restricting access to specific sensitive fields, such as compensation data, within a table that is otherwise broadly accessible. Combining these controls properly takes more upfront design work than a flat permission model, but it pays for itself the first time an app needs to scale beyond a single team. Building a Sustainable Governance Framework Technical controls only hold up if there is a governance structure behind them that defines who is responsible for what, and how the platform is monitored as usage grows. This is where many citizen development programs lose control after an initially strong start. Establishing a Center of Excellence: Microsoft's Center of Excellence Starter Kit gives organizations a working inventory of every app, flow, and environment across the tenant, which is often the first time leadership sees the platform's actual footprint. The kit automates the discovery of unmanaged apps, flags orphaned flows left behind by departed employees, and tracks adoption trends over time. A CoE does not need to be a large standing team. In most organizations, it is two or three people who own governance policy, review DLP exceptions, and provide a support path for makers building anything beyond a basic app.Application lifecycle management for critical apps: Not every app needs the same level of rigor, and treating a quick departmental tool the same as a finance-critical application wastes governance effort where it matters least. For apps that genuinely matter to the business, solutions should move through managed pipelines using Power Platform's native ALM tooling, with version control and a defined approval process before production deployment. Tiering apps by business impact lets governance teams apply heavier scrutiny only where the consequences of a security gap would actually be significant. This tiered approach is also what makes governance sustainable as the number of apps grows into the hundreds.Bringing in experienced guidance for complex rollouts: Designing a governance model that balances security with developer velocity is harder than it looks, particularly for organizations managing multiple business units with different compliance requirements. Engaging Power Platform consulting expertise early in the rollout helps organizations avoid the common mistake of retrofitting security after dozens of apps are already in production. An experienced partner brings tested environment architectures, DLP policy templates, and CoE configurations that would otherwise take months of trial and error to develop internally. That head start matters most for organizations under regulatory pressure, where security gaps are not just an operational risk but a compliance one. Final Words Enterprise security in Power Platform is not a single setting to enable, but the outcome of deliberate environment design, enforced DLP policies, granular Dataverse permissions, and a governance team with the authority to maintain all of it as usage grows. Organizations that treat governance as a one-time setup task tend to find their security posture eroding within a year, as new makers, apps, and connectors outpace the original controls. Those that build governance as an ongoing discipline get the best of both outcomes: fast development cycles for the business and a security model that holds up under scrutiny.

Every performance guide starts the same way. "Add an index." And yes, indexes matter. But I've spent years fixing production databases, and here's the truth: indexing is the easy 20%. The hard 80% is everything nobody writes blog posts about. I once spent three days chasing a query that had a perfect index. The index wasn't the problem. The problem was that the database's own statistics were lying to it. This article is about that other 80%. Why This Problem Keeps Coming Back Most teams treat database performance as a one-time task. Add indexes during launch week. Move on. But databases are not static. Data grows. Traffic patterns shift. Your "small lookup table" from six months ago now has four million rows. The query that ran in 2ms during testing can quietly become a 4-second query in production. Nobody notices until users complain. Here's the uncomfortable part: indexing advice assumes your query planner always makes good decisions. It doesn't. Query planners are guessing machines. They guess based on statistics, and statistics go stale. Why Developers Struggle With This Most backend engineers learn SQL as a language, not as an execution engine. You write SELECT * FROM orders WHERE customer_id = 123, it returns rows, and that feels like magic. But behind that query is a planner making dozens of decisions: Should it use an index or scan the whole table?Should it join tables in this order or that order?Should it use a hash join or a nested loop? Developers rarely see this decision-making. So when performance drops, the first (and often only) fix is "add an index." Sometimes that helps. Often it doesn't touch the real issue. The Real Problem: Stale Statistics Most relational databases (Postgres, MySQL, SQL Server) use cost-based optimizers. These optimizers don't know your data. They estimate it using statistics — sampled snapshots of your table's shape. If those statistics are outdated, the optimizer makes bad guesses. It might think a column has 10 distinct values when it actually has 10 million. Here's a real example from a Postgres system I worked on: SQL -- Table: events (48 million rows) EXPLAIN ANALYZE SELECT * FROM events WHERE event_type = 'checkout_completed' AND created_at > NOW() - INTERVAL '7 days'; The plan showed a sequential scan, even though we had an index on event_type. Why? The table statistics thought checkout_completed made up 40% of rows. In reality, it was 0.3%. The fix wasn't a new index. It was this: SQL ANALYZE events; One command. Query time dropped from 6.2 seconds to 90 milliseconds. Lesson: An index is only useful if the planner trusts it's worth using. Common Mistakes Developers Make Let's go through the mistakes I see over and over, across different companies and different stacks. 1. Trusting SELECT * Pulling every column, even ones you don't need, forces the database to read more data pages than necessary. On wide tables, this alone can double query time. 2. Ignoring the N+1 Query Pattern This one is everywhere in ORM-heavy codebases. Python # Bad: 1 query for orders + N queries for customers orders = Order.objects.all() for order in orders: print(order.customer.name) # triggers a new query each time Python # Good: 1 query total orders = Order.objects.select_related("customer").all() for order in orders: print(order.customer.name) If you have 500 orders, the bad version runs 501 queries. The good version runs 1. 3. Deep Pagination With OFFSET SQL -- Gets slower as the offset grows SELECT * FROM products ORDER BY id LIMIT 20 OFFSET 100000; The database still has to scan and discard 100,000 rows before returning your 20. On a page 5,000 request, this crawls. Better approach — keyset pagination: SQL SELECT * FROM products WHERE id > 100000 ORDER BY id LIMIT 20; This uses the index directly. No wasted scanning. Pagination MethodPerformance at Page 10Performance at Page 5000ComplexityOFFSET/LIMITFastVery slowLowKeyset (cursor-based)FastFastMediumPrecomputed pagesFastFastHigh (needs caching) 4. Doing Math on Indexed Columns SQL -- Index on created_at is useless here SELECT * FROM orders WHERE DATE(created_at) = '2026-07-20'; Wrapping a column in a function usually breaks the database's ability to use its index. SQL -- This keeps the index usable SELECT * FROM orders WHERE created_at >= '2026-07-20' AND created_at < '2026-07-21'; Small rewrite. Big difference. How Modern Systems Actually Solve This Real production systems don't rely on a single trick. They layer several defenses. Plain Text Client Request │ ▼ API Layer │ ▼ Query Cache (Redis) ├── cache hit? return here ▼ Connection Pool (PgBouncer) │ ▼ Read Replica (for reads) ──── Primary DB (for writes) │ ▼ Query Planner + Statistics │ ▼ Storage Engine Each layer exists to reduce pressure on the layer below it. Miss the cache, and you hit the pool. Miss the primary's write load, and reads go to a replica. Connection Pooling Matters More Than People Think Opening a raw database connection is expensive. It involves a TCP handshake, authentication, and memory allocation on the database side. Without pooling, a burst of traffic can create hundreds of connections in seconds. Postgres, for example, starts choking well before 500 connections. Plain Text # pgbouncer.ini [databases] mydb = host=127.0.0.1 port=5432 dbname=mydb [pgbouncer] pool_mode = transaction max_client_conn = 1000 default_pool_size = 25 With transaction pooling mode, PgBouncer hands out a real database connection only for the duration of a transaction, then returns it to the pool. This lets 1,000 app connections share just 25 real ones. Lock Contention: The Silent Killer This is the bottleneck that almost nobody talks about, because it doesn't show up in slow query logs the same obvious way. Here's what happened to us. A "quick" query started timing out during peak hours: SQL UPDATE inventory SET stock = stock - 1 WHERE product_id = 42; Individually, this query was fast. But during a flash sale, hundreds of these updates hit the same row at the same time. Each transaction had to wait for the previous one to release its row lock. The queries weren't slow. They were queued. Plain Text Time Transaction A Transaction B Transaction C 0ms LOCK row 42 waiting... waiting... 5ms UPDATE + COMMIT LOCK row 42 waiting... 6ms UPDATE + COMMIT LOCK row 42 7ms UPDATE + COMMIT How we fixed it: Moved to an eventual-consistency model for stock counts (queue-based decrement)Used SELECT ... FOR UPDATE SKIP LOCKED for job-queue-style tablesBatched decrements instead of doing them one row at a time SQL -- Instead of 100 individual UPDATE statements UPDATE inventory SET stock = stock - sub.qty FROM ( VALUES (42, 3), (43, 1), (44, 7) ) AS sub(product_id, qty) WHERE inventory.product_id = sub.product_id; One batched statement instead of a hundred lock acquisitions. Isolation Levels: A Trade-off, Not a Setting You Ignore Most engineers leave the isolation level at whatever the database defaults to. That's usually fine — until it isn't. Isolation LevelPreventsPerformance CostCommon Use CaseRead UncommittedNothing muchLowestRarely used, riskyRead CommittedDirty readsLowDefault in Postgres, most web appsRepeatable ReadNon-repeatable readsMediumFinancial reports, reconciliationSerializablePhantom readsHighestBanking transactions, inventory locks Higher isolation means more correctness guarantees. It also means more locking, more retries, and lower throughput. Don't default to Serializable "to be safe." You'll pay for it in throughput, and most apps don't need it. Query Plan Reading: A Skill Most Engineers Skip If you only remember one thing from this article, remember this: learn to read EXPLAIN ANALYZE output. It tells you the truth. Everything else is a guess. SQL EXPLAIN ANALYZE SELECT o.id, c.name FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.status = 'pending'; Sample output to watch for: SQL Hash Join (cost=120.50..3400.22 rows=850 width=64) (actual time=12.100..340.556 rows=42000 loops=1) Hash Cond: (o.customer_id = c.id) -> Seq Scan on orders o (cost=0.00..2900.00 rows=850) (actual time=0.020..300.100 rows=42000 loops=1) Notice the gap: the planner estimated 850 rows. The actual count was 42,000. That's a 49x miss. When estimated and actual rows differ by a wide margin, that's your signal. Stale statistics, bad indexes, or a query shape the planner can't reason about well. Denormalization: Sometimes the Right Move Normalization is taught as the "correct" way to design schemas. In practice, strict normalization can hurt performance on read-heavy systems. We had a dashboard query joining six tables to compute one number: total revenue per region. SQL SELECT r.name, SUM(o.total) FROM orders o JOIN customers c ON o.customer_id = c.id JOIN regions r ON c.region_id = r.id JOIN order_items oi ON oi.order_id = o.id JOIN products p ON oi.product_id = p.id JOIN categories cat ON p.category_id = cat.id GROUP BY r.name; This ran in 4 seconds. Dashboard needed sub-second response. We added a summary table, updated by a nightly job: SQL CREATE TABLE revenue_by_region ( region_name TEXT PRIMARY KEY, total_revenue NUMERIC, updated_at TIMESTAMP ); Dashboard query became: SQL SELECT region_name, total_revenue FROM revenue_by_region; From 4 seconds to 8 milliseconds. The trade-off: data is now up to 24 hours stale. This only works if your business can tolerate staleness. For real-time fraud detection, this approach would be wrong. Know your consistency requirements before you denormalize. Performance Considerations Checklist Before shipping a query to production, run through this: ✔ Did you check EXPLAIN ANALYZE, not just EXPLAIN? ✔ Are your table statistics current (ANALYZE run recently)? ✔ Does the query avoid functions wrapped around indexed columns? ✔ Are you selecting only the columns you need? ✔ Is pagination using keyset instead of large OFFSET values? ✔ Are batch writes used instead of row-by-row loops? ✔ Is the isolation level appropriate for the use case, not just the default? ✔ Have you tested this query against production-sized data, not a dev sample? Security Considerations Performance work sometimes creates security gaps. Watch for these: Dynamic query building for "flexible filters" often leads to string concatenation, which opens SQL injection risk. Use parameterized queries even for performance-tuned raw SQL.Read replicas used for reporting sometimes get looser access controls because "it's just a read replica." That's still your data.Caching layers (Redis, Memcached) can leak sensitive data if you cache full row objects without checking what's in them. Scaling Challenges As systems grow, new problems appear that indexing can't fix: Plain Text Single DB Instance │ ▼ Growing write load │ ▼ Read Replicas (helps reads, not writes) │ ▼ Still hitting write limits │ ▼ Sharding (splits writes across nodes) │ ▼ Cross-shard joins become painful Sharding solves write throughput but creates a new problem: joins across shards don't work the way they used to. You end up doing joins in application code, which is slower and more error-prone than letting the database do it. This is why teams delay sharding as long as possible. It's a last resort, not a first optimization. What We Learned A few honest lessons from years of doing this: Statistics decay silently. Schedule ANALYZE (or your database's equivalent) as a routine job, not an afterthought.The slowest part of a query is often not the query itself. It's lock waiting, connection exhaustion, or network round trips.ORMs hide problems well. They also hide the N+1 pattern extremely well. Turn on query logging in staging and actually read it.Caching isn't free. Cache invalidation bugs have cost us more debugging time than the queries we were trying to avoid.Nobody reads execution plans until something breaks. Read them earlier. It's a habit, not a rescue tool. When Not to Use These Techniques Not every optimization belongs in every system. Don't denormalize a table that changes every second the sync job will never catch up.Don't add read replicas if your write load, not read load, is the actual bottleneck.Don't reach for sharding if a bigger instance and better indexing would solve it for the next two years.Don't tune isolation levels down for "performance" on a system handling money movement. Optimization without a clear bottleneck measurement is just guessing with extra steps. Final Thoughts Indexing is the first lesson in database performance, not the last one. The real bottlenecks stale statistics, lock contention, bad pagination, and isolation level mismatches don't show up in a "10 SQL Tips" listicle. They show up at 2 AM, during a traffic spike, when your on-call phone rings. The next challenge for most teams isn't learning these techniques. It's building the habit of checking for them before a query becomes a production incident. That habit reading EXPLAIN ANALYZE, tracking replication lag, watching lock wait times matters more than any single trick in this article.

The Problem: Our p99 Was 3-5 Seconds Our PyFlink pipeline was missing its latency SLO by seconds. The pipeline itself was straightforward: consume events from Kafka, transform them, serialize them as Protobuf, and write the results to downstream systems. Yet under production load, p99 end-to-end latency was consistently in the 3-5 second range. Profiling pointed us to an unexpected bottleneck: we were deserializing Protobuf messages in Python, even though the Flink runtime processing our stream was JVM-based. Every record that entered the Python path had to cross the JVM-to-Python process boundary, get parsed by a Python UDF, and then cross back. The business logic wasn't the problem. The doorway was. We moved Protobuf deserialization to Flink's JVM-side Protobuf format and kept Python for orchestration and SQL. In our environment, p99 dropped to approximately 500 milliseconds, with less code and a pipeline that is easier to reason about. Verified on AWS Managed Service for Apache Flink (formerly Kinesis Data Analytics). Why Python-Side Deserialization Is So Expensive The naive PyFlink architecture looks like this: A Kafka source table declared with a generic format (raw, json, or a SimpleStringSchema), so every record arrives as opaque bytes or a string.A Python map() or UDF that imports generated _pb2.py classes and calls ParseFromString() on every message.Downstream transforms and sinks. Two costs hide in step 2, and they compound at high throughput. The process boundary. PyFlink is not Python running inside Flink; it is a JVM runtime coordinating with a separate Python execution environment. Every record that enters the Python execution path incurs overhead associated with moving data between the JVM and Python, and depending on the operator and execution mode, that can involve serialization and inter-process communication in both directions. For a per-record deserialization UDF on a latency-sensitive pipeline, that overhead is paid before the actual business transformation begins. Per-record parse cost. Even when Python's Protobuf implementation uses its native backend, parsing in a Python UDF still requires the record to enter the Python execution path. When the workload is latency-sensitive and high-throughput, the combination of serialization, inter-process communication, Python execution, and parsing overhead can become significant. In our case, profiling showed that this path was a major contributor to our latency. In our pipeline, these two costs together accounted for the bulk of the gap between a 3–5 second p99 and the ~500ms target we needed, before the enrichment logic even began executing. The Key Realization: PyFlink Already Runs on the JVM Here's the insight that changes the architecture: if Protobuf is declared at the table DDL level, Flink's Kafka connector deserializes it with its native, optimized JVM-based Protobuf format before any data reaches the Python side. Your columns simply arrive typed and ready. Python's role shrinks to what it's genuinely good at in this stack: orchestration and SQL. No rewrite to Java. No change to how jobs are deployed. Just a different declaration of intent. The trade is that Flink's native Protobuf format needs compiled Java message classes on the classpath; it does not consume .proto files or Python _pb2 modules directly. That means adding a small build step to your workflow, which we'll cover below. Implementation The pipeline splits into two declarative jobs. Job 1: JSON In, Protobuf Out The source table reads the raw JSON topic; the sink table declares format = 'protobuf' and points at the compiled Java class. The JVM handles typed-row-to-Protobuf encoding. SQL -- SOURCE: raw JSON payload as STRING plus Kafka record timestamp CREATE TABLE source_events_json ( event_data STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = '${INPUT_JSON_TOPIC}', 'properties.bootstrap.servers' = '${KAFKA_BOOTSTRAP_SERVERS}', 'scan.startup.mode' = 'latest-offset', 'format' = 'raw' ); -- SINK: Protobuf out to Kafka (JVM handles typed row to Protobuf) CREATE TABLE sink_events_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent' ); -- TRANSFORM: pure SQL, no Python UDFs INSERT INTO sink_events_pb SELECT JSON_VALUE(event_data, '$.id') AS id, JSON_VALUE(event_data, '$.organization_id') AS organization_id, ROW( UNIX_TIMESTAMP(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '')), CAST(EXTRACT(NANOSECOND FROM CAST(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '') AS TIMESTAMP_LTZ(9))) AS INT) ) AS event_ts, CAST(JSON_VALUE(event_data, '$.is_active') AS BOOLEAN) AS is_active, JSON_VALUE(event_data, '$.event_type') AS event_type FROM source_events_json; Note what's absent: no ParseFromString(), no _pb2.py imports, no Python deserialization loop. The Python program registers DDL and runs SQL. Job 2: Protobuf In, OpenSearch Out Downstream, the sanitized Protobuf topic becomes a typed source, using the same protobuf.message-class-name property, plus ignore-parse-errors so a malformed record can't poison the pipeline. SQL -- SOURCE: Protobuf from the sanitized Kafka topic CREATE TABLE kafka_source_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'scan.startup.mode' = 'latest-offset', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent', 'protobuf.ignore-parse-errors' = 'true' ); -- SINK: OpenSearch (JSON) CREATE TABLE opensearch_sink ( id STRING, organization_id STRING, event_ts TIMESTAMP_LTZ(3), is_active BOOLEAN, event_type STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'opensearch-2', 'hosts' = '${OPENSEARCH_ENDPOINT}:443', 'index' = 'acme-events-v1', 'format' = 'json' ); INSERT INTO opensearch_sink SELECT id, organization_id, TO_TIMESTAMP_LTZ(event_ts.seconds * 1000, 3), is_active, event_type FROM kafka_source_pb; The Build Step: Getting Java Classes Onto Flink's Classpath The one genuinely new piece of workflow is compiling your .proto definitions to Java and packaging them into the job's fat JAR. The essential Maven pieces: XML <dependencies> <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> <version>3.25.5</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-protobuf</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>${flink.connector.kafka.version}</version> </dependency> <!-- plus your sink connectors, e.g. flink-connector-opensearch2 --> </dependencies> Two practices that made this maintainable for us: Version-control the generated Java sources (or generate them in CI from a single canonical .proto repo) and pull them in with build-helper-maven-plugin's add-source, rather than compiling .proto files in every consuming project. One schema source of truth, many consumers.Shade everything into one JAR with maven-shade-plugin, excluding signature files (META-INF/*.SF, *.DSA, *.RSA). On AWS Managed Flink, pass it via the job's JAR configuration; on self-managed Flink, drop it in lib/ or use --classpath. The full workflow: define the .proto, compile it to Java with protoc, package the fat JAR, put it on Flink's classpath, author the PyFlink job with the DDL above, then deploy and watch end-to-end p99. How We Measured the Improvement We measured end-to-end p99 latency as the time from a record landing on the source Kafka topic to the corresponding OpenSearch write being acknowledged MetricBeforeAfterp99 latency3-5s~500msSustained throughput~5,000 events/sec~5,000 events/secFlink parallelism128Python UDF parsingYesNoJVM/Python boundary on hot pathYesNoProtobuf decodingPythonJVM Results End-to-end p99 latency around 500 milliseconds in our environment at production load, down from a 3-5 second baseline, by eliminating per-record JVM-to-Python crossings and Python-side parsing on the hot pathLess code. The deserialization UDFs, the _pb2 imports, and their error handling all disappeared. What remains is DDL plus SQLSimpler and easier to operate. The pipeline now relies on Flink's Kafka connector and Protobuf format for serialization and parsing, with built-in parse-error handling, instead of hand-rolled Python parsing When This Optimization Won't Help Moving Protobuf decoding to the JVM won't automatically solve every latency problem. If your pipeline's critical path is dominated by sink backpressure, network latency, external API calls, state access, or checkpointing overhead rather than deserialization, changing the serialization path may have little effect on end-to-end latency. This optimization is most valuable when profiling specifically shows that Python execution and JVM/Python data movement are significant contributors to the critical path, which is why we'd recommend profiling first rather than applying this as a default change. When You Should Still Use Python UDFs This pattern is not "never write Python UDFs." It's "keep them off the per-record deserialization path." Python remains the right tool when: The transformation genuinely needs Python libraries (ML feature computation, model inference, specialized parsing that has no SQL equivalent).Throughput is modest and developer velocity matters more than the last hundred milliseconds.You're prototyping. Even then, declare the format natively from day one anyway; it costs nothing and you won't have to migrate later. If a UDF is unavoidable on a hot path, at least let the JVM do the deserialization first so the UDF receives typed columns rather than raw bytes. Gotchas Worth Knowing Before You Ship Property syntax varies by Flink version. Some versions use format = 'protobuf'; newer key/value descriptors prefer value.format = 'protobuf'. Check your version's docs.Enums: surface them as STRING if you need ergonomic SQL manipulation, or keep them numeric with a lookup table.Schema evolution: favor backward-compatible, additive changes with defaults. Because the compiled Java classes are baked into the JAR, a schema change means a rebuild and redeploy, so make that a deliberate, versioned step in CI rather than an afterthought. ignore-parse-errors is your safety net during rollout windows, but monitor the drop counter so it doesn't silently eat data.Benchmark end-to-end, not just the UDF: source lag, operator latency, and sink acknowledgments under production load patterns. Deserialization wins can be masked, or dwarfed, by sink backpressure.Security: lock down OpenSearch credentials and TLS; pin Kafka client versions compatible with your Flink release. Closing Thoughts We didn't rewrite the pipeline in Java. We removed an unnecessary per-record JVM-to-Python boundary from the hot path and let Flink's JVM-native Protobuf format do the work it was designed to do. If your PyFlink job parses Protobuf messages in Python today, check whether Flink's native format support can move that work into the JVM-side execution path. For latency-sensitive pipelines, eliminating unnecessary Python boundaries may be one of the highest-leverage optimizations to investigate, especially when profiling shows that serialization and Python execution are on the critical path.

Building autonomous AI agents with large language models (LLMs) is easy when writing single-turn demo scripts. However, moving multi-agent loops into production introduces serious architectural challenges. Agents hallucinate, loop infinitely without reaching convergence, require human approval for high-risk operations, and need standard tool-calling integrations alongside clear operational governance. Historically, Java developers faced a tough choice: either rely on heavyweight, external workflow clusters (like Temporal or Camunda) that add operational overhead, or hand-craft fragile while loops and custom state machines inside their services. Quarkus Flow bridges this gap. Built on the Cloud Native Computing Foundation (CNCF) Serverless Workflow specification, Quarkus Flow brings light-footprint, specification-compliant workflow orchestration directly into your Quarkus application. When combined with LangChain4j, Model Context Protocol (MCP) tool connections, and AGENTS.md context governance, Java developers can construct deterministic, observable, and resilient agentic AI workflows using idiomatic CDI and a fluent Java DSL. The Modern Agentic Stack: Quarkus Flow, MCP, and AGENTS.md To run production AI agents, you need three distinct layers: orchestration, standardized tool connectivity, and behavioral governance. Orchestration (Quarkus Flow): Manages state transitions, retries, conditional loops, max-iteration caps, and Human-in-the-Loop (HITL) gates inside the JVM.Tool standardization (MCP): Connects agents to enterprise data, databases, and APIs using the Model Context Protocol (MCP) without writing custom API adapters for every LLM host.Behavioral governance (AGENTS.md): A project-level markdown specification that defines system boundaries, agent roles, required output formats, and safety rules that agents read at runtime. Markdown ┌────────────────────────────────────────────────────────────────────────┐ │ `AGENTS.md` Governance │ │ (Runtime System Prompts, Rules & Security Boundaries) │ └───────────────────────────────────┬────────────────────────────────────┘ │ Loaded via GovernanceLoader ▼ ┌────────────────────────────────────────────────────────────────────────┐ │ ArticlePublisherWorkflow (Quarkus Flow) │ │ │ │ 1. generateDraft ──> 2. evaluateDraft ──> 3. reviewCheck │ │ (Writer) (Critic) │ │ │ ▲ │ [approved || >=3] │ │ │ ├───> 5. publishArticle │ │ │ 4. reviseDraft <────────────┤ │ │ └─────────────────┘ [needs revision] │ │ └──────────────────────────┬─────────────────────────────────────────────┘ │ │ Tool Invocation via McpToolProvider ▼ ┌────────────────────────────────────────────────────────────────────────┐ │ Stateless MCP Servers │ │ (External Data, Database Tools, & APIs) │ └────────────────────────────────────────────────────────────────────────┘ Defining Governance With AGENTS.md Instead of hardcoding prompt strings deep inside Java classes, place an AGENTS.md file in your src/main/resources. This allows developers and prompt engineers to adjust system instructions and security boundaries without re-compiling the application. Here is the src/main/resources/AGENTS.md file based on the reference repository: GitHub Flavored Markdown # Content Reviewer Agent Governance & Rules ## Writer Agent Rules - You are an expert Java and Quarkus developer. - Draft concise, technically accurate blog posts based on requested topics. - Query available MCP tools when database context or tool parameters are required. ## Critic Agent Rules - You are a strict editor reviewing for clarity, security, and technical accuracy. - Return ONLY a valid JSON object matching this schema: {"approved": boolean, "feedback": "string"} ## Security Boundaries - Do not output shell commands or execute arbitrary code. - Always enforce character limits and avoid hallucinated imports. Practical Example: Multi-Agent Workflow With MCP and AGENTS.md Let's build a production-grade Content Publisher Agent Workflow matching the exact structure from quarkus-flow-mcp-agents. The workflow reads system instructions from AGENTS.md, uses a Writer Agent that fetches real data via an MCP Server, submits the draft to a Critic Agent, and loops until approved or max iterations are reached. Note: You can find the complete reference implementation repository at https://github.com/danieloh30/quarkus-flow-mcp-agents.git. 1. pom.xml Dependencies XML ... <properties> <compiler-plugin.version>3.15.0</compiler-plugin.version> <maven.compiler.release>25</maven.compiler.release> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id> <quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id> <quarkus.platform.version>3.38.0</quarkus.platform.version> <skipITs>true</skipITs> <surefire-plugin.version>3.5.6</surefire-plugin.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>${quarkus.platform.group-id}</groupId> <artifactId>${quarkus.platform.artifact-id}</artifactId> <version>${quarkus.platform.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>${quarkus.platform.group-id}</groupId> <artifactId>quarkus-langchain4j-bom</artifactId> <version>${quarkus.platform.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>${quarkus.platform.group-id}</groupId> <artifactId>quarkus-flow-bom</artifactId> <version>${quarkus.platform.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> ... <dependency> <groupId>io.quarkiverse.langchain4j</groupId> <artifactId>quarkus-langchain4j-openai</artifactId> </dependency> <dependency> <groupId>io.quarkiverse.langchain4j</groupId> <artifactId>quarkus-langchain4j-mcp</artifactId> </dependency> <dependency> <groupId>io.quarkiverse.flow</groupId> <artifactId>quarkus-flow-langchain4j</artifactId> </dependency> ... </dependencies> ... 2. Application Configuration: src/main/resources/application.properties Properties files # Enable OpenAI quarkus.langchain4j.openai.api-key=${OPENAI_API_KEY} quarkus.langchain4j.openai.chat-model.model-name=gpt-4o-mini quarkus.langchain4j.openai.log-requests=true quarkus.langchain4j.openai.log-responses=true 3. Orchestrating the Write-Review Loop With @LoopAgent ArticlePublisher is the orchestrator that wires the multi-agent loop together using Quarkus Flow's declarative API. Here's what each annotation does: @LoopAgent – runs WriterAgent then CriticAgent repeatedly (up to 3 iterations). At build time, Quarkus Flow compiles this into a CNCF Serverless Workflow definition — no separate workflow engine at runtime.@ExitCondition – a static method (isApproved) that checks if the critic's review starts with "APPROVED". It runs after each loop iteration (testExitAtLoopEnd = true). If true, the loop breaks early.@Output – a static method (extractArticle) that extracts the final result. It pulls the draft from the shared agent scope and returns it as the workflow output.The flow: Writer drafts → Critic reviews → if not approved, Writer revises using feedback → repeat until approved or 3 iterations hit → return the final draft. Java public interface ArticlePublisher { @LoopAgent( subAgents = { WriterAgent.class, CriticAgent.class }, maxIterations = 3) String publishArticle(String topic); @ExitCondition(testExitAtLoopEnd = true, description = "Exit when the critic approves the draft") static boolean isApproved(String review) { return review != null && review.toUpperCase().startsWith("APPROVED"); } @Output static String extractArticle(String draft) { return draft; } } 4. WriterAgent — Drafting With MCP-Powered Research WriterAgent is a declarative LLM agent that researches a topic via Brave Search and drafts a technical blog post. @Agent – marks the method as an agent entry point. outputKey = "draft" stores the result in the shared scope so other agents (like CriticAgent) can access it.@ToolBox(WebSearchTool.class) – gives the LLM access to the webSearch tool. The LLM decides when to call it based on the prompt — it's not forced. This is how MCP tools connect to declarative agents.@SystemMessage – instructs the LLM to research before writing, produce accurate content, and revise based on prior feedback. That last part is critical for the loop — on iteration 2+, the LLM sees the critic's feedback in the chat memory and adjusts the draft accordingly. The interface has no implementation — Quarkus generates it at build time. Java public interface WriterAgent { @Agent(outputKey = "draft", description = "Drafts or revises a technical article based on the topic") @ToolBox(WebSearchTool.class) @SystemMessage(""" You are an expert Java and Quarkus developer. Use the webSearch tool to research the topic before writing. Write concise, technically accurate blog drafts based on your research. Never generate raw shell commands or suggest unsafe practices. If the reviewer has given you feedback in a previous turn, revise the draft to address it. """) @UserMessage("Write a short technical blog post about: {topic}") String writeDraft(String topic); } 5. CriticAgent — Reviewing for Accuracy and Clarity CriticAgent is the quality gate in the loop. It reviews the draft and either approves or rejects it with feedback. @Agent – outputKey = "review" stores the review in the shared scope. The @ExitCondition in ArticlePublisher reads this key to decide whether to exit the loop.@UserMessage – injects the {draft} variable from the shared scope, so the critic always reviews the latest version of the article.@SystemMessage – enforces a strict contract: if the draft is acceptable, the response must start with "APPROVED:". This is what makes the @ExitCondition work — it's a simple string check, not another LLM call. No tools are attached — the critic relies solely on the LLM's reasoning to evaluate the draft. Java public interface CriticAgent { @Agent(outputKey = "review", description = "Reviews the draft for technical accuracy and clarity") @SystemMessage(""" You are a strict editor checking for technical accuracy and clarity. If the draft is acceptable, your response MUST start with "APPROVED:" followed by a brief note. If the draft needs improvement, provide constructive feedback. """) @UserMessage(""" Review this draft: {draft} """) String reviewDraft(String draft); } 5. WebSearchTool — Bridging MCP and Declarative Agents WebSearchTool is a CDI bean that connects the Brave Search MCP server to the agent workflow. Why it exists – @McpToolBox only works with @RegisterAiService, not with @Agent. This class bridges that gap by creating an MCP client programmatically and exposing it as a @Tool.MCP client setup – the constructor creates a DefaultMcpClient with stdio transport, spawning npx -y @brave/brave-search-mcp-server as a subprocess. The BRAVE_API_KEY is passed via environment variables.@Tool – the webSearch method builds a ToolExecutionRequest targeting the brave_web_search tool on the MCP server, executes it, and returns the results. The LLM sees this as a regular function it can call.@PreDestroy – cleans up the MCP client (and the subprocess) when the CDI context shuts down. This pattern — wrapping an MCP client in a @Tool CDI bean and attaching it via @ToolBox — is reusable for any MCP server you want to connect to a declarative @Agent. Java public class WebSearchTool { private final McpClient mcpClient; WebSearchTool(@ConfigProperty(name = "brave.api.key", defaultValue = "${BRAVE_API_KEY:}") String braveApiKey) { mcpClient = new DefaultMcpClient.Builder() .transport(new StdioMcpTransport.Builder() .command(List.of("npx", "-y", "@brave/brave-search-mcp-server")) .environment(Map.of("BRAVE_API_KEY", braveApiKey)) .logEvents(true) .build()) .build(); } @Tool("Search the web for up-to-date information about a given query using Brave Search") public String webSearch(String query) { var request = ToolExecutionRequest.builder() .name("brave_web_search") .arguments("{\"query\": \"" + query + "\"}") .build(); return mcpClient.executeTool(request).resultText(); } @PreDestroy void close() { try { mcpClient.close(); } catch (Exception ignored) { } } } Production Guardrails and Enterprise Readiness Deploying agentic AI systems into enterprise cloud environments requires strict governance, tracing, and high performance: Standardized tools via MCP: By consuming external systems through stateless Model Context Protocol endpoints, tool definitions are decoupled from LLM host code.Context control with AGENTS.md: Business analysts and security leads can audit or update prompt guidelines without re-deploying code artifacts.Human-in-the-loop (HITL): Use Quarkus Flow event filters or pause states to suspend execution until a human administrator approves sensitive tool actions.OpenTelemetry and distributed tracing: Quarkus Flow and quarkus-opentelemetry pass W3C trace contexts across every workflow transition, LLM call, and MCP request.GraalVM native images: Compile the entire stack — Quarkus Flow engine, LangChain4j, MCP connections, and REST interface — into an ultra-fast, native binary with sub-10ms startup times and minimal memory footprint. By combining Quarkus Flow, LangChain4j, MCP, and AGENTS.md, Java developers can replace unmaintainable AI scripts with clean, specification-compliant, and enterprise-ready agentic architectures.

By Daniel Oh CORE

Why Most Platforms Fail to Become Products Many companies are heavily investing in internal developer platforms (IDPs) with the expectation that they will speed up delivery and governance, and increase developer productivity. Despite significant investment in Kubernetes, CI/CD, observability, security tooling, and cloud infrastructure, many platforms struggle to gain adoption. The reason is simple: they are built and operated like infrastructure projects, not products. Infrastructure teams are often very focused on technical excellence: automation, scalability, reliability, and compliance. Developers, on the other hand, are interested in a different goal — getting their applications into production quickly and safely without having to go through so much complexity. IDP is successful when developers choose it voluntarily because it makes their lives easier. That shift requires platform architects to think less like infrastructure engineers and more like product managers. Building an IDP is like operating an airport. Nobody travels because they love airports. They travel because they want to reach a destination efficiently. Similarly, developers do not care about Kubernetes clusters, pipelines, secrets management, or observability stacks. They care about shipping features to customers. The platform's job is to make the journey smooth, fast, and safe. This article explores the core practices that differentiate successful product-centric platforms from infrastructure-centric ones. Practice 1: Start With Developer Journeys, Not Technology Choices Imagine constructing a shopping mall by selecting elevators, security systems, and air-conditioning units before understanding customer traffic patterns. The result is often technically impressive but operationally frustrating. The same happens with developer platforms. Architects should first map the customer journey (developer journey) before designing platform capabilities. Many platform initiatives begin with questions like: Which Kubernetes distribution should we use?Which GitOps framework is best?Which CI/CD tool should be standardized? These are important questions, but they should not be the starting point. Successful platform architects begin by understanding developer workflows: How does a new service get created?How long does environment provisioning take?Where do deployment delays occur?What causes support tickets?Which activities are repetitive and manual? The goal is to identify friction and eliminate it. Organizations using platforms based on technologies like Red Hat OpenShift, IBM Cloud Kubernetes Service, or other cloud-native platforms have found that developers adopt only when the platform team focuses on reducing the friction in workflow rather than adding more infrastructure features to the platform. Practice 2: Create Golden Paths, Not Golden Handcuffs A highway encourages drivers to use the fastest route while still allowing exits when necessary. Successful IDPs behave like highways. Developers naturally choose the Golden Path because it is easier and safer than building everything from scratch. One of the most powerful concepts in modern platform engineering is the Golden Path. A Golden Path provides: Recommended architecturesStandard deployment patternsPre-approved security controlsBuilt-in observabilityAutomated CI/CD workflows Developers should be able to move fast along a paved road while retaining flexibility for unique requirements. Platform teams that leverage services from cloud provider environments often realize that standardized self-service templates drive significantly higher adoption than restrictive governance models. Practice 3: Make Self-Service the Primary Interface Every banking transaction once required a visit to a physical branch. Today, customers expect to do everything from a mobile app. Developers hope for the same experience from inside their own software. Nothing kills developer productivity faster than dependency queues. Consider a common case of dependency queues. Open a ticket for infrastructure.Wait for approval.Wait for provisioning.Request secrets.Request monitoring.Request deployment access. Weeks can pass before development even begins. Modern platforms must provide self-service experiences where developers can do the following without opening tickets. Create environmentsProvision databasesConfigure pipelinesAccess observability dashboardsRequest infrastructure resources An IDP should function like a digital banking application—secure, streamlined, and available on demand. Below is the Product-Centric IDP reference architecture. Developers consume platform capabilities through self-service experiences, while the platform embeds security, observability, governance, and delivery capabilities and exposes them through Golden Paths. Practice 4: Treat Platform APIs as Products A power drill might have sophisticated engineering in it. Users judge it by a very simple standard: “Can I drill a hole fast and reliably?" Many platform teams are focused on infrastructure automation and not developer experience. Each API, template, workflow, and portal interaction is a product interface. Questions worth asking include: Is the API predictable?Is documentation clear?Are error messages actionable?Is onboarding intuitive?Can developers discover capabilities easily? Developers evaluate IDPs the same way. They are not interested in the complexity underneath. They care about usability. This principle is especially important when integrating observability services, cloud provisioning layers, or deployment automation platforms. For example, IBM Cloud's managed services can significantly simplify operational complexity, but value is realized only when developers experience that simplicity through intuitive platform workflows. Practice 5: Build Observability into the Platform, Not Around It Imagine when you are driving a car without any speedometer, fuel gauge or warning indicators. You may still reach your destination but the risk increases dramatically. Observability is the dashboard for software systems. Observability is often treated as an afterthought. A team deploys an application and later attempts to add: MetricsLogsTracesDashboardsAlerting This approach creates inconsistency and operational blind spots. Platform teams should embed observability from day one. Every service created through the platform should automatically include: Logging standardsDistributed tracingMetrics collectionHealth monitoringService dashboards Whether organizations use IBM Cloud Observability, Instana, OpenTelemetry, Prometheus, Grafana, or other solutions, the platform should make observability automatic rather than optional. Practice 6: Make Security Invisible but Ubiquitous When entering a modern office building, people rarely think about security. Access badges, surveillance, and emergency controls are built into the environment — the building is secure without requiring employees to become security experts. The same principle applies to IDPs. In immature environments, security is seen as a series of checkpoints, review meetings, manual compliance approvals, vulnerability assessments, and audit evidence collection. Developers find it as friction because it arrives late in the delivery lifecycle. Traditional security models operate as gates. Platform-centric security operates as guardrails. The objective is not fewer security controls — it is fewer manual interactions. Build Secure-by-Default Golden Paths Every new service created through the platform should automatically inherit: Secure CI/CD pipelines with dependency and container image scanningSecret detection and policy enforcementAccess control standards and audit loggingEncryption best practices Automate Policy Enforcement Manual compliance verification is one of the biggest sources of deployment delays. Platform teams should adopt policy-as-code (PaC) approaches that automatically validate deployment configurations, infrastructure standards, and regulatory controls. Instead of asking, "Did someone review this configuration?" the platform asks, "Does this configuration satisfy our policies?" Reduce Security Cognitive Load Developers should not need deep expertise in every security domain. The platform should abstract identity management, secrets management, certificate management, and vulnerability remediation workflows—particularly in hybrid and multi-cloud environments where security complexity grows rapidly. A useful measure of progress: the percentage of security controls inherited from the platform versus manually implemented by application teams. The higher the inheritance rate, the lower the cognitive load. Practice 7: Measure Platform Success Like a Product A gym owner does not measure success by counting treadmills—they measure it by member outcomes. Platform teams should apply the same logic. Traditional infrastructure metrics like cluster utilization, pipeline counts, and resource consumption tell you whether the platform is running. They do not tell you whether it is working for developers. Product-oriented platform teams focus on: Developer satisfactionPlatform adoptionTime to first deploymentDeployment frequencyLead time for changes If developers still circumvent the platform, no amount of technical sophistication matters. The Developer Experience Scorecard Measuring developer experience requires balancing sentiment, effort, and adoption. High-performing platform teams track four key measures: Metric What It Measures How to Collect Developer Satisfaction Score (DSS) Overall platform sentiment Quarterly survey, 1–10 scale Platform NPS Willingness to recommend the platform "How likely are you to recommend this platform?" scored 0–10 Ease-of-Use Score How intuitive common workflows feel Per-task rating, 1–5 scale Developer Effort Score How much work is required to achieve an outcome Survey question on effort per task Together, these reveal not just whether developers are using the platform but whether they genuinely value it. Satisfaction Is a Leading Indicator Most delivery metrics lag behind—deployment frequency (e.g., lead time, incident count) and other metrics. Developer satisfaction is a leading indicator. Developers discover friction long before it is observable from the data. A declining DSS today will result in a decline in productivity and adoption tomorrow. Listening early allows platform teams to respond before problems grow into organizational challenges. The real measure of success is not how many developers use the platform—it is how they feel while using it. The IDP Health Dashboard High-performing platform teams monitor a balanced set of metrics across four categories: Category Metrics Sentiment DSS, Platform NPS, Ease-of-Use ratings Adoption Golden Path adoption, self-service usage, onboarding rates Friction Support ticket volume, documentation search failures, manual approval requests Productivity Time to First Deployment (TTFD), environment provisioning time, lead time for changes A platform succeeds not when developers are forced to use it, but when they prefer to use it. Practice 8: Reduce Cognitive Load Relentlessly The automotive industry spent decades simplifying the driving experience so drivers could focus on reaching their destination rather than understanding the mechanics of their vehicles. IDPs should do the same. As organizations evolve into cloud-native architectures, developers are expected to navigate containers, Kubernetes, CI/CD, IaC, security policies, service meshes, observability tools, and compliance requirements all at once. Each one solves a very important problem individually. As a whole, they overwhelm developers and take focus away from developing business capabilities. A successful platform is not one that exposes every infrastructure capability. It is one that hides unnecessary complexity while providing simple, intuitive paths to outcomes. The goal of platform engineering is not to eliminate complexity. It is to absorb complexity so developers don't have to. Common indicators of excessive cognitive load: Developers struggling to find documentationFrequent support requests for routine tasksLong onboarding times for new servicesMultiple handoffs between teamsTool sprawl across the engineering ecosystem Reduce Tool Sprawl Every tool a developer must learn introduces new interfaces, terminology, documentation, and configuration models. Platform teams should create a unified experience through a developer portal, service catalog, or platform API, that minimizes the number of decisions and interfaces developers encounter. Minimize Context Switching Every transition between tools, teams, or approval processes introduces cognitive overhead. Platform teams should ask: Can this be automated? Can these steps be consolidated? Can approvals be replaced with automated guardrails? The goal is fewer interruptions between code creation and deployment. Platform Teams Are Complexity Brokers Complexity never disappears — it moves. Organizations can either push complexity onto every development team, or centralize and manage it within the platform. High-performing platform teams choose the latter, absorbing operational, security, infrastructure, and compliance complexity so application teams can focus on features. Practice 9: Obsess Over Time to First Deployment The first experience developers have with a platform often determines whether they embrace it or avoid it. Imagine a shopping mall where opening a new store requires twelve forms, multiple approval queues, and manual setup of every utility. Store owners would go elsewhere. The best malls provide ready-made spaces where businesses can start operating almost immediately. Developer platforms should do the same. High-performing platform teams focus relentlessly on Time to First Deployment (TTFD) — the time between creating a service and successfully deploying it. The Biggest Contributors to Poor TTFD Bottleneck Root Cause Fix Manual infrastructure provisioning Ticket-driven approval chains Self-service IaC, service catalogs, platform portals CI/CD pipelines built from scratch No standard templates Pre-built, reusable pipeline templates Security reviews at the end Late-stage compliance gates Shift left — embed scans and policy checks in Golden Paths Observability setup delays Manual metrics/dashboard configuration Auto-provision logging, tracing, and health checks by default Too many decisions Choice overload at onboarding Provide Golden Paths with sensible defaults Measure Every Stage Stage Target Service creation < 5 mins Repository creation Automated Pipeline creation Automated Infrastructure provisioning < 10 mins First build < 5 mins First deployment < 15 mins Observability enablement Automatic TTFD = Provisioning Time + Setup Time + Approval Time + Deployment Time Many organizations discover that approval time is larger than all technical activities combined. The fastest platforms replace approvals with automated guardrails. Practice 10: Build a Platform Community, Not Just a Platform Team Cities flourish when residents contribute feedback and shape growth. Cities planned entirely from a central authority often struggle to meet citizen needs. IDPs are no different. The best platforms evolve through continuous collaboration. Platform teams should create feedback loops through office hours, community forums, developer councils, internal documentation reviews, and experience surveys. Developers become co-creators rather than consumers. Community Health Metrics Running community mechanisms is not enough — each one needs a way to know whether it is working. Track these six indicators to measure community health: Metric What It Measures Healthy Signal Monthly Active Community Members Developers engaging in forums, channels, or office hours Steady growth quarter over quarter Developer-to-Developer Answer Rate % of forum questions answered by non-platform-team members Above 40% indicates a self-sustaining community External Contributions per Quarter Pull requests or documentation edits from application teams Increasing trend Roadmap Items from Community Input % of platform backlog items originating from developer feedback Above 50% signals product-centric culture Office Hours Repeat Attendance Rate % of attendees who return across multiple sessions Above 60% indicates ongoing value Support Ticket Deflection Rate % of issues resolved via community before a ticket is opened Rising deflection reduces platform team toil The ultimate sign of a mature platform community is a change in how developers talk about the platform—from something that happens to them to something they help shape. Practice 11: Think in Products, Roadmaps, and Customer Value Smartphones succeeded because manufacturers continuously improved user experience. Customers did not buy phones because of processor specifications. They bought outcomes—better communication, productivity, and convenience. Developers adopt platforms for the same reason. The strongest indicator that a platform is becoming a product is a change in language. Instead of asking: What infrastructure should we standardize? Platform teams begin asking: What developer problems should we solve next? Which user journeys create the most friction?Which capabilities deliver the highest value?What does our product roadmap look like? Features matter only when they improve the developer experience. Practice 12: Design for Platform Reliability, not Just Application Reliability Imagine a city that invests heavily in building roads, bridges, and public transport for its citizens, but has no maintenance crew, no traffic monitoring, and no plan for when a bridge closes. The infrastructure exists, but without reliability commitments, citizens cannot depend on it. Internal developer platforms face exactly the same risk. Most platform engineering conversations focus on the reliability of applications running on the platform — uptime, error rates, latency SLOs for customer-facing services. What is rarely discussed is the reliability of the platform itself. Yet the platform is load-bearing infrastructure for every engineering team in the organisation. When the CI/CD pipeline degrades, every team's delivery stops. When the service catalog is unavailable, no new services can be provisioned. The platform's reliability is a multiplier — a single failure can simultaneously impact dozens of teams. Define Platform SLOs Before Developers Define Them for You Platform teams that do not define their own Service Level Objectives will find that developers define them informally — through frustration, workarounds, and loss of trust. Effective platform SLOs cover the experiences developers depend on most: Pipeline availability — what percentage of CI/CD pipeline executions succeed without infrastructure-related failures?Provisioning latency — how long does environment or resource provisioning take at the 95th percentile?Portal availability — is the developer portal and service catalog accessible during working hours?Golden Path build time — how long does a standard pipeline template take to complete? These are the experience metrics developers encounter every day. A platform team that publishes and tracks these SLOs operates as a reliable internal service provider. A team that does not is invisible until something breaks. IDP Maturity Model Stage Characteristics Infrastructure Platform Standardized infrastructure, clusters, CI/CD tooling Self-Service Platform Service catalogs, automation, infrastructure on demand Developer Platform Golden Paths, integrated observability and security, DevEx focus Platform Product Platform roadmaps, adoption metrics, developer satisfaction measurement Adaptive Platform Continuous feedback loops, AI-assisted operations, continuous platform evolution Most organizations do not start with a Platform Product. They evolve toward it. The goal of the maturity model is not to reach the highest stage overnight, but to identify the next set of capabilities that will improve developer experience and platform adoption. High-performing platform teams treat platform maturity as a journey rather than a destination. Assessing Your Current Stage To identify where your platform currently sits, ask three diagnostic questions: How do developers access platform capabilities today? If the answer is "by opening a ticket," the platform is at the infrastructure stage. If developers provision resources on demand without human approval, they are at the self-service stage or beyond.Do developers choose the platform voluntarily or use it because they must? Voluntary adoption driven by speed and simplicity signals a developer platform or platform product. Mandatory usage with frequent workarounds signals an earlier stage.Does the platform team maintain a product roadmap prioritized by developer feedback? A yes here is the clearest indicator of a platform product. The absence of a roadmap almost always reflects an infrastructure or self-service mindset. Moving to the Next Stage Each stage has a single dominant unlock that drives progression: Infrastructure → Self-Service: Replace ticket-driven provisioning with self-service automation and a service catalog.Self-Service → Developer Platform: Introduce Golden Paths that embed security, observability, and CI/CD by default.Developer Platform → Platform Product: Establish a formal platform roadmap, measure developer satisfaction (DSS, NPS), and treat developer feedback as a product backlog.Platform Product → Adaptive Platform: Build continuous feedback loops, introduce AI-assisted operations, and invest in platform telemetry that proactively surfaces friction before developers report it. The most common mistake is attempting to skip stages. Teams that build Golden Paths before self-service exists create well-designed paths nobody can access independently. Teams that adopt satisfaction metrics before Golden Paths exist measure friction without the tools to address it. Progress through the stages in order. The IDP Architect's Checklist Before launching any new platform capability, ask: ✅ Does this feature remove friction from a developer workflow? ✅ Can developers access it through self-service? ✅ Is it aligned with a Golden Path? ✅ Is observability included by default? ✅ Is security built into the platform? ✅ Is governance automated rather than manual? ✅ Can success be measured through developer outcomes? ✅ Does it reduce cognitive load? ✅ Does it improve Time to First Deployment? ✅ Would developers choose this platform if they had alternatives? If the answer to several of these questions is "no," the capability is probably infrastructure-focused rather than product-focused. Final Thoughts The future of platform engineering is not about building more infrastructure. It is about delivering better developer experiences. The most successful IDPs combine the discipline of site reliability engineering (SRE), the automation of cloud-native technologies, and the mindset of product management. Whether your foundation runs on IBM Cloud, OpenShift, hyperscaler cloud services, or a hybrid environment, the winning formula remains the same: Treat developers as customers. Treat the platform as a product. Treat developer productivity as the ultimate business metric. When platform architects embrace this mindset, platforms stop being collections of tools and start becoming accelerators of innovation—and that's when platforms truly become products.

By Josephine Eskaline Joyce CORE

Cloud migration projects almost always treat security as a downstream concern something to bolt on after workloads have already moved, once the “real” migration work is done. Across dozens of enterprise migrations spanning finance, healthcare, and manufacturing workloads, that ordering is consistently the source of the costliest rework: reopened firewall rules, retrofitted identity models, and access reviews that should have happened before a single virtual machine was provisioned. The pattern holds regardless of which cloud provider is on the receiving end. What follows is a framework provider-agnostic by design for embedding zero-trust principles into the migration process itself, rather than applying them after the fact. Why Bolt-On Security Fails Traditional migration playbooks are organized around workload movement: discover, assess, re-platform, cut over, optimize. Security tasks are usually inserted late, as a checklist item before go-live. Three consequences follow reliably: Implicit trust survives the move. Implicit trust survives the move. On-premises networks often rely on perimeter trust: anything inside the firewall is assumed safe. When that assumption is lifted-and-shifted into the cloud without redesign, the perimeter simply becomes larger and harder to defend.Identity sprawl compounds. Identity sprawl compounds. Migrations frequently multiply service accounts, temporary roles, and cross-environment credentials used to bridge on-prem and cloud during cutover. Few of these get cleaned up.Retrofitting is expensive. Retrofitting is expensive. Segmenting a network or re-scoping IAM roles after hundreds of workloads are already live requires downtime windows and change approvals that could have been avoided by designing correctly the first time. The Framework: 4 Pillars, Applied in Migration Order The framework below organizes zero-trust adoption into four pillars, sequenced to match the natural phases of a migration rather than treated as a parallel workstream. 1. Identity as the New Perimeter Before any workload assessment begins, establish the identity model the migrated environment will use, not the one the source environment happens to have. Define role-based access aligned to job function, not to legacy group membership inherited from the source directory.Require multi-factor authentication for every administrative path into the target environment before migration tooling is granted access, not after.Treat every migration-tooling service account as temporary by default, with an explicit expiration and re-certification date. 2. Segment Before You Migrate, Not After Network segmentation decisions made during the assessment phase are cheap. The same decisions made post-migration require change windows and stakeholder sign-off. Group workloads into trust tiers during discovery (e.g., internet-facing, internal-only, regulated-data) rather than assuming a flat network topology will be corrected later.Design micro-segmentation boundaries around workload tiers before the first server moves, so that day-one network policy already reflects least-privilege communication paths.Validate east-west traffic rules against actual application dependency maps, not assumed ones; dependency mapping tools exist for this precisely because assumptions are usually wrong. 3. Encrypt and Verify at Every Hop, Not Just at Rest Most cloud providers make encryption at rest close to a default setting. The gap is almost always in transit and in verification. Require mutual TLS or equivalent between service-to-service calls introduced during migration, especially temporary bridging connections between source and target environments.Treat data classification as a migration input, not a post-migration audit finding. Classify before you move, so encryption and access policy can be applied by tier from day one.Build verification checkpoints into the cutover plan itself: an environment isn't “migrated” until its access logs confirm no implicit-trust paths remain from the legacy network. 4. Assume Breach, Instrument Accordingly The final pillar is operational rather than architectural: build the assumption of compromise into monitoring from the start of the migration, not after an incident. Instrument logging and alerting for the target environment before cutover, so that abnormal access patterns are visible from hour one rather than backfilled weeks later.Run tabletop exercises against the migrated architecture; specifically, lessons from the legacy environment's incident response plan rarely transfer cleanly.Track a small set of leading indicators (privileged session anomalies, unexpected cross-tier traffic, credential reuse across environments) rather than waiting for a full SIEM rollout to catch up. Lessons From Enterprise Deployments A few patterns show up consistently across large, regulated deployments: Sequencing beats scope. Organizations that tried to implement all four pillars simultaneously across an entire estate stalled. The deployments that succeeded phased identity and segmentation first, then layered encryption verification and monitoring in as workloads landed.Legacy exceptions need sunset dates. Legacy exceptions need sunset dates. Every migration produces temporary trust exceptions to keep the business running during cutover. Without a hard expiration date attached at creation, these exceptions become permanent attack surface.Cross-functional ownership matters more than tooling. Cross-functional ownership matters more than tooling. The deployments with the fewest post-migration security incidents were the ones where network, identity, and application teams jointly signed off on the trust model before migration started, not the ones with the most sophisticated tooling. Common Pitfalls Treating zero trust as a product purchase rather than an architectural discipline applied throughout the migration lifecycle.Migrating identity and network configuration as-is with the intention to “harden it later” rarely comes without an incident forcing it.Measuring migration success purely on workload count and timeline, with security posture reviewed only at the end. Closing Thought Zero trust and cloud migration are often treated as separate initiatives running on separate timelines. The organizations that get the best outcomes fewer post-migration incidents and faster time-to-secure-operations are the ones that treat zero trust as a design constraint on the migration itself, sequenced into discovery, assessment, and cutover rather than appended afterward. The framework above is intentionally provider-agnostic because the discipline it describes identity first, segmentation before movement, verification at every hop, and instrumentation from day one holds regardless of which cloud the workloads land on.

Almost every backend eventually needs to run code on a schedule. Send the invoice at midnight. Retry the failed payment in five minutes. Generate the weekly report every Monday at 7 AM. Clean up expired sessions every hour. On one server, this is easy. You write a cron line and move on. The trouble starts when one server becomes ten. Now the same cron line lives on every box, so the invoice job fires ten times instead of once. Move the cron to a single “scheduler” box, and that box becomes a single point of failure. Every time you deploy new code, that process restarts, and if it crashes or the host dies, there is no second node to cover for it. Any job due during that downtime window silently never fires. A distributed job scheduler solves this. It runs jobs reliably across a fleet of machines, fires each job once even when nodes crash, and keeps working when parts of the system fail. This post walks through how to design one, the trade-offs at each step, and the mistakes that bite teams in production. What the Scheduler Has to Do Before drawing boxes, it helps to pin down the requirements. They split into two groups. Functional requirements: Run a job once at a specific time (a one-time job).Run a job on a repeating schedule, usually defined with cron (a recurring job).Support job dependencies, where job B runs only after job A succeeds.Retry a job automatically when it fails.Respect priority, so urgent jobs run before bulk jobs.Cancel or pause a job that is scheduled or already running. Non-functional requirements: Durability. Once the system accepts a job, it must not lose it, even if a node dies one second later.At-least-once execution. Every due job runs at least one time.Scale. The design should handle millions of jobs per day across many workers.Fault tolerance. A crashed worker must not block other jobs, and its work should be picked up by someone else. One requirement is worth calling out early. People often ask for “exactly-once” execution. In a distributed system, you cannot truly get it. What you can build is at-least-once delivery plus idempotent jobs, which together behave like exactly-once from the outside. More on that later. The Core Architecture The single most important idea in this design is to separate deciding when a job runs from actually running it. These are two different problems with different scaling needs, so they become two different components. A clean design has four parts: A scheduler that watches the clock and decides which jobs are due.A queue that holds ready-to-run jobs and hands them out.A pool of stateless workers that pull jobs and execute them.A datastore that holds job definitions and execution history, and acts as the source of truth. Why decouple the queue from the workers at all? Because load is bursty. At midnight, a thousand daily jobs may become due at the same second. If the scheduler called workers directly, that spike would hit them all at once. The queue absorbs the spike and lets workers drain it at a steady rate. It also lets you scale workers up and down without touching the scheduler. This is the same reason queues show up across system design, which I covered in detail in Role of Queues in System Design. Modeling Jobs in the Database The datastore is the source of truth, so the schema matters. A common approach uses two tables. One holds the recurring definition, the other holds individual runs. SQL CREATE TABLE jobs ( id BIGINT PRIMARY KEY, name TEXT NOT NULL, cron TEXT, -- null for one-time jobs payload JSONB, next_run_at TIMESTAMPTZ, -- when this job is next due enabled BOOLEAN DEFAULT TRUE ); CREATE TABLE job_runs ( id BIGINT PRIMARY KEY, -- unique id per run job_id BIGINT REFERENCES jobs(id), status TEXT NOT NULL, -- PENDING, RUNNING, SUCCEEDED, FAILED, DEAD attempt INT NOT NULL DEFAULT 1, scheduled_at TIMESTAMPTZ, started_at TIMESTAMPTZ, lease_until TIMESTAMPTZ ); CREATE INDEX idx_jobs_due ON jobs (next_run_at) WHERE enabled = TRUE; The partial index on next_run_at is the workhorse. The scheduler asks “which jobs are due now” many times per second, and this index keeps that query fast even with millions of rows. Each run moves through a small set of states. Drawing the state machine makes the retry and failure logic obvious. Defining Schedules With Cron Recurring jobs need a way to express “every day at 2:30 AM” or “every 15 minutes.” Cron is still the standard. A classic cron expression has five fields: Plain Text minute hour day-of-month month day-of-week 30 2 * * * -> 2:30 AM every day The Java world often uses Quartz cron, which adds a seconds field at the front and a year field at the end, giving six or seven fields. The two formats look similar but are not interchangeable, and mixing them up is a frequent source of jobs that never fire. The scheduler stores the cron string and computes a concrete next_run_at timestamp from it. After a run is enqueued, it computes the next one. This raises a real question: what happens if the scheduler was down for an hour and three runs were missed? This is the misfire problem. You generally pick one of two policies: Catch up. Run every missed occurrence in order. Correct for billing, expensive for everything else.Skip. Run only the next future occurrence and forget the missed ones. Right for jobs like cache refreshes where stale runs add no value. Make this an explicit setting per job. Teams that leave it implicit get surprised after the first outage. Picking Which Jobs to Run The scheduler needs to find due jobs and hand them off. There are three common ways to find them. Polling. Every second, query the database for jobs where next_run_at <= now(). Simple and reliable. The partial index keeps it cheap. The cost is a small delay, up to your poll interval.Timer wheel. Keep upcoming jobs in an in-memory structure sorted by time. Very precise and great for short delays, but you have to rebuild it from the database after a restart.Push. An external timing service fires an event when a job is due. Real-time, but now you depend on another moving part. For most systems, polling with a one-second interval is the right default. It is boring, and boring is good for a component you are trusting with billing runs. The harder problem is concurrency. If you run several scheduler instances for availability, they will all poll the same table at the same time. Without care, two of them pick the same job, and it runs twice. The clean fix in PostgreSQL is row locking with SKIP LOCKED: SQL SELECT id FROM jobs WHERE enabled = TRUE AND next_run_at <= now() ORDER BY next_run_at LIMIT 100 FOR UPDATE SKIP LOCKED; FOR UPDATE locks the rows this instance selects. SKIP LOCKED tells other instances to ignore locked rows and grab the next free ones instead. Many schedulers can now poll in parallel, each claiming a different batch, with no coordination service and no duplicate pickups. Airflow uses exactly this approach instead of a heavier consensus protocol, which is a good reminder that the simplest mechanism that meets the requirement usually wins. Why Exactly-Once Is a Myth Here is the scenario that breaks naive designs. A worker pulls a job, runs it successfully, and then crashes before it can tell the system “done.” The system still thinks the job is running. The lease expires, another worker picks it up, and the job runs a second time. You charged the card twice. You cannot delete this scenario. Networks drop messages and processes die at the worst moment. So you stop chasing exactly-once delivery and instead make the work safe to repeat. That means two things working together: At-least-once delivery. The system guarantees a due job runs at least one time, accepting that it may occasionally run more than once.Idempotent jobs. Running the same job twice has the same effect as running it once. The standard trick is an idempotency key built from stable identifiers, for example {job_id, run_id, attempt}, or a key tied to the business action like invoice_2026_06_charge. The worker records that key before committing side effects. If the same key shows up again, the worker sees the work is already done and acknowledges without repeating it. This is why each run gets its own unique id. A time-ordered id such as a Snowflake id or a ULID works well, because it is unique across the whole fleet without coordination and it sorts by creation time, which keeps the job_runs table naturally ordered. I explained the structure of these ids in How Snowflake IDs Work, and the deduplication pattern itself in Idempotent Receiver Pattern. There is one more subtle gap. The worker has to update the database and publish to the queue, and those are two systems. If it writes to the database and then dies before publishing, the job is lost. The transactional outbox pattern closes this gap by writing the job and an outbox row in one local transaction, then publishing from the outbox separately. I covered that in The Transactional Outbox Pattern. Coordinating at Scale A single scheduler instance has a throughput ceiling. Past a certain number of jobs per second, one process polling one database cannot keep up. There are two ways to grow. The first is leader election. You run several scheduler instances, but only one is active at a time. The others stand by and take over if the leader dies. A coordination service like etcd or ZooKeeper holds the leadership lock. This is simple to reason about, but the single active leader is still a throughput bottleneck. The second is sharding. You split the job space across many active schedulers. A simple scheme hashes the job id into one of N partitions, and each scheduler owns a set of partitions. Every job has exactly one owner, so there are no duplicate pickups, and throughput grows by adding schedulers. Consistent hashing makes it cheaper to add or remove schedulers without reshuffling everything. Sharding has one sharp edge. During a handover, while leases for a partition are changing hands, two schedulers can briefly believe they own the same partition. This is split brain. You do not try to make it impossible, because that is expensive. Instead, you let the worker-side idempotency check be the final safety net. If both schedulers enqueue the same run, the idempotency key means it still executes once. Google’s cron service takes a stricter route for its most sensitive launches. It writes the launch record to a quorum using Paxos before the job actually starts, so a failover cannot lose or double-fire it. For most teams, leases plus idempotency are enough, and full consensus is overkill. Detecting Failures and Recovering Workers crash. The scheduler has to notice and reassign their work, without stealing jobs from workers that are simply slow. The mechanism is a lease with a heartbeat. When a worker claims a run, it sets lease_until to a short time in the future, say 30 seconds. While the job runs, the worker periodically extends the lease. If the worker dies, it stops extending, the lease expires, and a recovery sweep moves the run back to PENDING so another worker can take it. SQL -- recovery sweep: reclaim runs whose lease has expired UPDATE job_runs SET status = 'PENDING' WHERE status = 'RUNNING' AND lease_until < now(); Two details make this robust. First, the lease timeout must be comfortably longer than a normal heartbeat interval, or a brief pause will cause a healthy job to be wrongly reclaimed. Second, you need protection against a zombie worker, one that froze on a long garbage collection pause, lost its lease, and then woke up and tried to finish writing results. A fencing token solves this. The reclaimed run gets a higher token, and the datastore rejects any write carrying an older token. I went deeper on time-bound ownership and fencing in The Lease Pattern in Distributed Systems. Retries Done Right A failed job should usually be retried, but retrying badly makes outages worse. If a downstream service is struggling and every failed job retries immediately, you pile on more load at the exact moment it can least handle it. The fix is exponential backoff with jitter. Each retry waits longer than the last, and a random jitter spreads the retries out so they do not all fire at the same instant. Plain Text attempt 1 fails -> wait ~1s attempt 2 fails -> wait ~2s attempt 3 fails -> wait ~4s attempt 4 fails -> wait ~8s (each wait randomized by +/- a few hundred ms) After a fixed number of attempts, stop. A job that keeps failing should not retry forever. Move it to a dead letter queue, a separate place for runs that exhausted their retries, and alert a human. The dead letter queue keeps a poisoned job from clogging the pipeline while preserving it for investigation. Operating the Thing A scheduler is infrastructure other teams depend on, so it has to be observable and controllable. For observability, track the metrics that tell you the system is healthy: Queue depth. A queue that keeps growing means workers cannot keep up.Scheduling lag, the gap between when a job was due and when it actually started.Run outcomes per minute, split by succeeded, failed, and dead.Lease reclaims, which spike when workers are crashing. For control, give operators real knobs. They should be able to pause a queue, drain a worker before a deploy so it finishes current jobs and takes no new ones, and replay a dead-lettered job after fixing the cause. Building these in from the start saves a lot of pain during the first incident. How Real Systems Approach This None of this is theoretical. The same building blocks show up across well-known tools, each making a different trade-off. Quartz. A mature Java scheduler. Multiple instances coordinate through a shared database using row locks, the same idea as the SKIP LOCKED approach above.Airflow. Orchestrates dependency graphs of tasks. Its scheduler uses database locks rather than a consensus protocol, favoring operational simplicity.Temporal. Models workflows as code and replays an append-only event history to recover state after a crash, which sidesteps a whole class of mid-task failure bugs.Celery. A popular task queue in Python, with a beat component that handles periodic scheduling.Kubernetes CronJobs. Run containerized jobs on a cron schedule inside a cluster, with configurable policies for missed runs and concurrency. See the Kubernetes CronJob docs.Google distributed cron. Writes launch state to a Paxos quorum before launching, so a leader failover never loses or doubles a run. The pattern across all of them is consistent. Decouple scheduling from execution, lean on the database or a quorum for coordination, accept at-least-once and make jobs idempotent, and design for failure as the normal case. Takeaways If you remember five things from this, make it these. Separate the decision of when a job runs from the work of running it. They scale differently.Do not chase exactly-once. Build at-least-once delivery and make every job idempotent.Use the database as a coordination primitive. SELECT ... FOR UPDATE SKIP LOCKED lets many schedulers poll safely.Use leases with heartbeats and fencing tokens to detect dead workers and reclaim their runs without double execution.Retry with exponential backoff and jitter, cap the attempts, and send the rest to a dead letter queue. A good scheduler is not clever. It is careful. It assumes nodes will die, messages will duplicate, and clocks will drift, and it keeps running anyway.

Retrieval-augmented generation (RAG) reduces hallucinations in generative AI chatbots by grounding each response in retrieved source data instead of relying only on what the model learned during training. Before the model writes a reply, the system fetches relevant passages from a trusted knowledge store and passes them in as context. The model then answers from that evidence, which shrinks the room it has to invent facts. This article looks at why hallucinations happen at the token level, how a RAG pipeline counters them, and the engineering choices that decide whether grounding actually holds up in production. Why Generative AI Chatbots Hallucinate A large language model predicts the next token from statistical patterns, not from a fact store it can look up. Ask it about something outside its training data or about a recent change, and it still returns fluent, confident text, sometimes wrong. That confident-but-wrong output is a hallucination. Three causes show up most often in conversational AI systems: Knowledge gaps. The training corpus has a cutoff, so newer facts are missing.Ambiguous prompts. Vague input pushes the model to guess.Pattern completion. The decoder prefers plausible phrasing over accurate phrasing when both fit. For a customer-facing bot, the cost is concrete: invented pricing, fictional policies, or wrong API behavior, all delivered in the same tone as a correct answer. What Retrieval-Augmented Generation Actually Does RAG connects the model to an external knowledge base at query time. Rather than answering from parameters alone, the chatbot searches a document store first, pulls the closest matches, and injects them into the prompt. The pipeline runs in three stages: Retrieve: embed the user query and run a similarity search against a vector index.Augment: place the top passages into the prompt as grounding context.Generate: the model composes an answer constrained by that context. Because the output is tied to retrieved text, the system can also return citations pointing at the exact source. How RAG Reduces Hallucinations RAG targets the root cause: missing or stale context. Supplying current, relevant evidence narrows the space where the model has to improvise. Grounding in approved sources The model reads from your documents, so answers reflect your data rather than internet averages. A well-built pipeline also instructs the model to reply "not found" when retrieval returns nothing useful, instead of filling the gap with a guess. Fresh data without retraining You update the index, not the weights. New policies or product details become answerable the moment they are ingested, which removes a major source of dated, wrong replies. Traceable answers Each response can carry a reference back to its source passage. For regulated domains, that audit trail is often the difference between a system people use and one nobody trusts. A Minimal RAG Loop The core retrieval-then-generate step looks like this in pseudocode: Python def answer(query, index, llm): q_vec = embed(query) passages = index.search(q_vec, top_k=5) if not passages: return "I don't have that information." context = "\n".join(p.text for p in passages) prompt = f"Answer using only this context:\n{context}\n\nQ: {query}" return llm.generate(prompt) The top_k cutoff, the "only this context" instruction, and the empty-result fallback are small details that carry most of the anti-hallucination weight. Where RAG Pipelines Break Retrieval quality, not model size, is where most accuracy is won or lost. Common failure points: Bad chunking. Segments too large dilute relevance; too small and they lose meaning.Weak embeddings. A mismatched embedding model returns passages that look related but aren't.No reranking. Top-k by cosine similarity alone often buries the best passage below near-duplicates.Silent context overflow. When retrieved text exceeds the window, passages get truncated, and the model fills the gaps on its own. 2026 Patterns Worth Knowing A few shifts are changing how teams build these systems this year. Agentic RAG. Instead of one lookup, the chatbot plans multi-step retrieval, calling tools and querying several sources before answering. This handles compound questions a single search cannot. GraphRAG. Pairing a knowledge graph with vector search captures relationships between entities, which improves answers over connected or multi-hop data. Continuous evaluation. Automated grounding checks score every answer for faithfulness to its sources, catching regressions before users report them. As enterprise adoption grows, this kind of automated eval is moving from nice-to-have to default. Decision Factors Before You Build If you are weighing RAG for a production bot, the factors that matter most: Data freshness and cleanliness beat any single model choice.Chunking and overlap shape retrieval accuracy more than people expect.Guardrails: confidence thresholds and fallback responses so the bot declines rather than fabricates.An eval pipeline that measures grounding rate, not just fluency.Latency budget: retrieval adds round trips, so cache common queries. FAQ Does RAG remove hallucinations completely? No. It reduces them sharply, but noisy data or poor retrieval can still produce errors, which is why evaluation and guardrails stay necessary. Is RAG better than fine-tuning? For fresh, factual answers, RAG usually wins because you update data without retraining. Fine-tuning suits tone and format. Many systems use both. What data does a RAG chatbot need? A curated knowledge base: documentation, FAQs, policies, or product data, cleaned and chunked for retrieval. A Final Word Hallucination is the line between a chatbot demo and a system a team can put in front of real users. RAG addresses it directly by grounding generative AI chatbots in current evidence rather than hoping the weights remember. The hard part lives in retrieval quality and evaluation, not in the model alone.

Then the search form grows, filters multiply, and nested criteria appear. Since using GET means placing the query inside the URI, a length limit problem emerges. Worse, placing sensitive query values in the URI increases the chance of exposure through access logs, browser history, proxies, and monitoring systems. Because the HTTP protocol does not forbid it, sending a body with GET may look like a way out, but building your design on behavior the standards leave undefined is not a recommended practice. Elasticsearch's GET-with-body search API is a well-known example, and Elastic's own documentation openly acknowledges the problem: "As a result, some HTTP servers allow it, and some—especially caching proxies—don't. [...] However, because GET with a request body is not universally supported, the search API also accepts POST requests." HTTP POST, on the other hand, carries the query in the request payload rather than the URI, which overcomes both the length limit and the data leakage problems. But POST is neither safe nor idempotent, since the protocol allows every invocation to change state on the server, and its response is not cached unless it carries explicit freshness information. This nature of POST also imposes a performance cost: results are recomputed and retransferred on every call, and a timed-out request cannot be safely retried. What is missing is clear: a method that is safe and idempotent like GET but carries content like POST. Until June 2026, HTTP did not have such a method in standardized form. The QUERY Method To address this need, the IETF introduced the QUERY method in RFC 10008. QUERY is the first new HTTP method since RFC 5789 was standardized in 2010. The core idea can be summarized as follows: a QUERY request asks the target resource to process the enclosed content in a safe and idempotent manner and to respond with the result. Everything else the RFC introduces either follows from this definition or builds practical machinery around it. Let's look at the key concepts one by one: Safe and Idempotent A QUERY is defined as a safe operation: it does not request a state change on the target resource. It can be retried, repeated, or restarted automatically without concern for partial side effects. This is the contract that separates it from POST. Meaning Comes From Content-Type RFC 10008 deliberately does not define a query language. The same endpoint may accept a JSON filter document, a form-encoded string, or any other query language defined by a media type; the media type of the request content defines how the server should interpret it. Servers are required to reject requests whose Content-Type is missing or inconsistent with the content. The RFC goes as far as forbidding content sniffing: a server is not allowed to infer a media type from the request content and use it to repair a missing or erroneous Content-Type. Explicitly Cacheable Unlike POST, QUERY introduces cacheability for body-carrying requests, with one crucial twist: the cache key must include the request content in addition to the URI, since two QUERY requests to the same URI with different bodies are different queries. Discovery via Accept-Query A server can advertise QUERY support with the Accept-Query response header, which lists the media types it accepts as query content. The Equivalent Resource A QUERY response may include a Location header pointing to a URI that represents the same query. A client can later re-fetch the result with a plain GET, no body required. The spec also gives 303 See Other a natural role for redirecting a query to a retrievable resource. The RFC's Security Considerations add one caveat here: when the query contains sensitive information that must not be logged, the URI assigned to such a resource should not include any sensitive portions of the original query content; otherwise, the exposure problem QUERY avoids would simply reappear one response later. Familiar Error Semantics The RFC recommends specific status codes for the failure cases: 400 when media type information is missing, 415 when the media type is not supported by the resource, and 422 when the content is well-formed but the query cannot be processed. A Decade in the Making The RFC had a long journey. The idea traces back to WebDAV's SEARCH method (RFC 5323, 2008), which demonstrated the demand for body-driven queries but remained confined to the XML-based WebDAV ecosystem. In 2021, the HTTP Working Group adopted the effort as a working group item, moving it from an individual proposal into the IETF standardization process. The method was later renamed from SEARCH to QUERY to avoid confusion with the existing WebDAV SEARCH method and to better reflect its purpose. The document was published as RFC 10008 in June 2026. Eleven years from the first draft to Proposed Standard is a useful reminder that even a seemingly simple addition to HTTP touches an enormous installed base and therefore receives extensive scrutiny. Where Ecosystem Support Stands Today As of July 2026, HTTP QUERY has completed the standardization phase with RFC 10008, but ecosystem adoption remains in its early stage. Many HTTP servers and proxies can forward QUERY requests without protocol changes, but native support across frameworks, browser APIs, caches, WAFs, and API tooling is still emerging. The primary barrier is no longer the protocol itself, but the large installed base of software that assumes a fixed set of HTTP methods. The Java ecosystem offers a useful snapshot of adoption in progress: Apache Tomcat A pull request adding QUERY support was merged on July 1, 2026 (apache/tomcat#1026). Support is available only in Tomcat 12 because it required Servlet API changes. Eclipse Jetty Eclipse Jetty has an open pull request (jetty/jetty.project#15316) implementing the core RFC 10008 semantics: method registration as safe and idempotent, the Accept-Query header, redirect behavior, and integration with compression and buffering handlers. It was initially aimed at Jetty 12.1 but has been retargeted to Jetty 13, aligning with a possible Jakarta Servlet 6.2 timeline. Jakarta Servlet There is an open issue (jakartaee/servlet#1068) proposing the addition of QUERY to the specification itself, so that HttpServlet gains first-class support and QUERY requests receive the same form parameter processing model currently defined for POST. This is arguably the most significant milestone for the broader Jakarta EE ecosystem, because it moves QUERY from container-specific support into the platform specification itself. Once Servlet defines QUERY, application servers such as WildFly, Payara, and Open Liberty can inherit support through their servlet containers as they move to the new specification level. As of this writing, none of them has shipped QUERY support ahead of the specification. What About Spring? Spring deserves its own section because of how request mapping is modeled. Spring MVC and WebFlux expose their annotation-based request mapping model through the RequestMethod enum, and that enum currently contains GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, and TRACE. There is no RequestMethod.QUERY, which means you cannot declaratively map a QUERY request through Spring's annotation-based programming model today. The available workarounds are awkward and bypass Spring's normal request-mapping model: declare a generic mapping and inspect request.getMethod() manually, or implement a custom RequestMappingHandlerMapping. Unlike the Servlet case, this is not primarily a container problem; it is primarily a framework API and abstraction problem. The Spring team is aware. A community pull request adding QUERY support (spring-projects/spring-framework#34993) has been open since before RFC 10008 was published. It supersedes a feature request that had remained open for nearly two years, and maintainers have indicated an intention to target Spring Framework 7.1, currently expected in November 2026. There is even a naming collision to solve first: the obvious convenience annotation @QueryMapping is already used by Spring for GraphQL. Why Quarkus Can Do It Today This is where an underappreciated property of HTTP pays off: the request method is simply a token defined by the HTTP grammar. A server does not need to have built-in knowledge of every method to parse it. Quarkus builds its HTTP layer on Netty and Vert.x, and neither requires the method to be one of a predefined set; the request can reach the routing layer without requiring special handling for QUERY. On top of that, Jakarta REST has had a standard extension point for custom methods since JAX-RS 1.0: the @HttpMethod meta-annotation, the same mechanism that has enabled JAX-RS applications to expose WebDAV methods like PROPFIND for years. Put the two together and RFC 10008-compatible QUERY endpoints in Quarkus require no framework changes; they can be enabled through a single Jakarta REST extension point: Java @HttpMethod("QUERY") @Documented @Target(ElementType.METHOD) @Retention(RetentionPolicy.RUNTIME) public @interface QUERY { } The remaining work is implementing RFC 10008 semantics at the application layer, which is precisely what the example project demonstrates. The Example: A Product Catalog You Can QUERY The demo repository is available on GitHub: hakdogan/http-query-method. It is a small Quarkus application exposing a product catalog at /products, deliberately compact, with only a handful of classes, but each RFC 10008 concept has a concrete counterpart in the code. One Query, Two Media Types The resource accepts the same logical filter in two representations, demonstrating that the query semantics are determined by the Content-Type, not the URI: Java @QUERY @Consumes(MediaType.APPLICATION_JSON) public Response query(ProductFilter filter) { ... } @QUERY @Consumes(MediaType.APPLICATION_FORM_URLENCODED) public Response queryForm(String body) { ... } So both of these work, and mean the same thing: Shell curl -i -X QUERY http://localhost:8080/products \ -H 'Content-Type: application/json' \ -d '{"category":"laptop","maxPrice":2000}' curl -X QUERY http://localhost:8080/products \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'category=laptop&maxPrice=2000' A request with an unsupported media type is rejected with 415, and a filter that is well-formed but self-contradictory, such as minPrice greater than maxPrice, returns 422. The second part is a design choice rather than an RFC requirement: Section 2.1 says 422 can be used when the content matches its media type, but the query cannot be processed due to its actual contents, and returning an empty result with 200 would be an equally valid reading. The demo treats the contradiction as a client error because an empty 200 response would be indistinguishable from a legitimately empty match, silently hiding what is almost certainly a bug in the caller. The Response Tells the Whole Story A successful QUERY comes back like this: Shell HTTP/1.1 200 OK Content-Type: application/json Accept-Query: application/json, application/x-www-form-urlencoded Location: http://localhost:8080/products?category=laptop&maxPrice=2000 Cache-Control: no-transform, max-age=60 ETag: "f675e29b" [{"category":"laptop","id":2,"name":"ThinkPad X1 Carbon","price":1899.00}, ...] Three headers carry the RFC's ideas: Accept-Query advertises which media types the resource accepts as query content. In the demo, it is added by a small response filter.Location points to the equivalent resource from Section 2.2 of the RFC: the same query expressed through the request URI. Fetch it with a plain GET, and you get the identical result, no body needed. One of the tests does exactly that round trip.Cache-Control and ETag make the cacheability promise concrete. The ETag is derived from the result, so repeating the query with If-None-Match returns 304 Not Modified without resending the result: Shell HTTP/1.1 304 Not Modified ETag: "f675e29b" This is the answer to "why not just POST": QUERY was designed to provide query semantics without giving up the cache-friendly properties associated with safe methods. Discovery Without Prior Knowledge How does a client discover that a resource supports QUERY? One OPTIONS request: Shell curl -i -X OPTIONS http://localhost:8080/products The response answers with two headers, one listing the methods the resource accepts and one listing the media types it accepts as query content: Shell HTTP/1.1 200 OK Allow: HEAD, QUERY, GET, OPTIONS Accept-Query: application/json, application/x-www-form-urlencoded In this case, Quarkus generated the Allow header automatically, including QUERY, simply because a resource method is bound to it. Proving Idempotency The demo's test suite covers the filtering logic, the media type handling, the error codes, the equivalent-resource round trip, the conditional request flow, and, fittingly for a method whose defining feature is repeatability, a test that repeats the same QUERY several times and verifies the operation remains safe and produces a consistent response. The key lesson from this example is not how QUERY was implemented, but why it was possible: the HTTP extension point already existed, and the framework did not need to invent a new abstraction. Conclusion QUERY is not a revolution; it is the standardization of a pattern that many systems have implemented through POST-based query endpoints for years. That is exactly why it matters. The gap between "works" and "works with the guarantees the protocol gives you" is where caching, idempotent retries, and better tooling become possible. Adoption is arriving unevenly: first in protocol implementations and servers, then in frameworks, gateways, and CDNs. But as the example shows, on a stack like Quarkus that treats the method as an extensible value rather than a hardcoded list, you do not have to wait to start experimenting. The protocol was ready for extension; the interesting question was whether the layers above it preserved that flexibility. The complete example, including all tests, is available on GitHub: hakdogan/http-query-method. References RFC 10008, The HTTP QUERY Method: https://www.rfc-editor.org/info/rfc10008/IETF Datatracker, document history: https://datatracker.ietf.org/doc/rfc10008/RFC 9110, HTTP Semantics: https://www.rfc-editor.org/info/rfc9110/RFC 4918, WebDAV: https://www.rfc-editor.org/info/rfc4918/RFC 5323, WebDAV SEARCH: https://www.rfc-editor.org/info/rfc5323/RFC 5789, PATCH: https://www.rfc-editor.org/info/rfc5789/

By Hüseyin Akdoğan CORE