Performance

What Makes a Good Build Server? In modern cloud-native application development, Continuous Integration, with automated building and testing of software on every commit, has become a standard best practice. This typically involves maintaining a farm of build nodes, which can be physical devices, virtual machines, or containers, that can be provisioned on demand and retired once build tasks are completed. This guide aims to help you configure the ultimate build server for Ampere's Arm-based architecture. We will explore various configuration options (or “knobs and switches”) to optimize a Linux build server’s performance, detailing the performance improvement with each adjustment. This tuning guide will focus on building LLVM-MinGW, a toolchain for creating Windows binaries using the LLVM project. This will be conducted on a Fedora 40 server running on an Ampere® Altra® 128-core server and compared with results from AmpereOne® 192-core servers, highlighting the advancements in that CPU. What Build Server Workloads Look Like Running software builds and testing workloads on a modern build server are inherently dynamic. They typically feature short bursts of CPU-intensive tasks, interspersed with other tasks that aren't as CPU-intensive. Figure 1 illustrates this CPU utilization behavior over time, which we will discuss later. While many build tasks, such as compiling source files, can execute in parallel across numerous CPU cores, other steps are inherently serial, including initial build configuration and linking. Therefore, optimizing complex builds means effectively managing these concurrent processes around unavoidable serial choke points To build the best possible build server, we aim to ensure that our compilation processes remain uninterrupted, that we avoid saturating the system's disk I/O, and that we minimize memory thrashing. This specifically includes minimizing memory page allocation and TLB misses. Furthermore, when builds can be parallelized, it is crucial to keep all available cores busy. Using All Your Cores Modern build servers derive most of their performance from parallel compilation. If your build system is not explicitly configured to execute tasks concurrently, large multi-core systems will be underutilized and build times will scale very poorly. For GNU Make, parallelism is explicit, where -j<N> represents the number of parallel jobs to be started, typically set to the number of available CPU cores (for example, up to -j128 on Ampere Altra or up to -j192 on AmpereOne). Shell make -j<N> Most modern build systems provide similar mechanisms: CMake: Parallelism is controlled by the underlying build tool: Shell cmake --build . --parallel <N> Or by passing -j<N> to make or ninja. Ninja: Parallel by default; concurrency can be limited with: Shell ninja -j<N> Maven: Supports parallel module and project builds: Shell mvn -T <N> or -T <N>C (cores) Gradle: Enables parallel task execution: Shell gradle build --parallel Or via org.gradle.workers.max= <N>. SCons: Similar to Make and Ninja, uses –j: Shell scons -j<N> Building LLVM-MinGW We describe building the LLVM-MinGW project to highlight the steps we used to optimize an Ampere-based build server. LLVM-MinGW is a toolchain to build Windows binaries using the LLVM project that supports the i686, x86_64, armv7, and arm64 architectures. The LLVM-mingw repository provides detailed documentation about the project. After cloning the git repo, we set up a 50 GB RAM disk to run the build on to improve performance. We use the build-all.sh script to run the build for 5 different architectures specified in the TOOLCHAIN_ARCHS variable. Shell git clone https://github.com/mstorsjo/llvm-mingw.git ode Shell # Create RAM disk mkdir llvm-mingw-tmpfs sudo mount | grep llvm-mingw-tmpfs mount -t tmpfs -o size=50G,mode=1777 tmpfs ./llvm-mingw-tmpfs Shell # run build from RAM disk LOG=build-llvm-mingw.log cd llvm-mingw-tmpfs && rm -rf * && cp -r ../llvm-mingw/* . && TOOLCHAIN_ARCHS="i686 x86_64 armv7 aarch64 arm64ec" LLVM_CMAKEFLAGS="-DLLVM_ENABLE_LIBXML2=OFF -DLLDB_ENABLE_PYTHON=OFF -DLLVM_USER_LINKER=lld -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++" /usr/bin/time -f '%U, %S, %e, %P' ./build-all.sh --disable-lldb-mi $(pwd)/install/llvm-mingw >& ${LOG} RAM Disk: Using DRAM to Avoid Disk I/O A RAM disk is a filesystem backed by system memory rather than persistent storage. On Linux, this is typically implemented using tmpfs, which allows a directory to behave like a regular filesystem while storing its contents in RAM. Reads and writes to a RAM disk occur at memory speed and are much faster than reading from disks or SSDs. On a build server, a RAM disk is commonly used to store temporary build artifacts such as object files, dependency caches, and intermediate outputs. The build system can then be configured to place its build directory, temporary files, or compiler cache under this path (for example, using an out-of-tree build directory). In our example, the entire build is done on the RAM disk. RAM disks can improve performance because large parallel builds can generate and consume tens of thousands of small files, especially during compilation and linking. Even on fast NVMe SSDs, metadata operations and concurrent I/O can become a bottleneck when hundreds of compiler processes run in parallel. A RAM disk eliminates these I/O constraints by keeping all intermediate data in memory, reducing latency and contention. On high-core-count servers running large parallel builds, this can significantly improve throughput by ensuring that compiler processes are not stalled waiting for disk access. The benefit is most visible when the build is otherwise CPU-bound and sufficient memory is available to avoid swapping. In this configuration, a RAM disk helps the system sustain full CPU utilization across all cores, shortening end-to-end build times. It is always recommended to measure the performance of any configuration changes to understand their impact and to verify that performance has improved. In this case, we tested building using a RAM disk vs. using the system SSD. The build took 1069.1 s running on the SSD and 1062.7 s using the RAM disk, for a ~0.6% speedup. The mpstat utility, part of the sysstat package, was used to measure CPU utilization during the build, with the results plotted in Figure 1. This data shows that the overall CPU utilization is very low for the build, except for the region from ~100 to ~450 seconds and for brief spikes of high CPU utilization. Overall, the average build total CPU utilization was just 53.4% of the Ampere ® Altra ® 128-core server. This raised the question: Why is utilization so poor? Fig 1: Server CPU utilization vs. time for initial LLVM-MinGW build, illustrating poor overall server utilization except for the region approximately 100 – 450 seconds. Our initial studies focused on running the build using the perf record performance profiler to measure performance. Linux’s perf profiler is a very powerful tool that allows you to see what’s running on the CPU at the application level and drill down to shared libraries, functions, individual source code lines and assembly instructions. However, sometimes a focus on low-level metrics prevents understanding global issues. This proved to be one such example. For this project, understanding the true bottleneck required higher-level performance data to grasp why the build was not utilizing the system more efficiently. Instrumenting Build Scripts The LLVM-MinGW build is complex, involving many sub-projects built with different configurations and architectures. To gain detailed insights, we implemented a powerful yet simple instrumentation method using bash’s PS4 variable. This approach inserted timestamps into the build log file before every executed command. By parsing the output log file, we could accurately determine the duration of each command. While analyzing and understanding the various, often nested, phases of the build took time, the instrumentation via bash’s PS4 variable was trivial to implement. Shell # Script to add timestamps to all scripts PS4='DEBUG $0 Line ${LINENO}, time since epoch, $(date "+ %s"): ' SCRIPTS=$(ls *sh) for SCRIPT in ${SCRIPTS}; do cp ${SCRIPT} ${SCRIPT}.orig sed -i '1d' ${SCRIPT} # remove #!/bin/sh line # converts existing scripts to bash scripts # with timestamps using bash’s PS4 cat ../convert_sh_to_bash_debug.sh ${SCRIPT} > bash_${SCRIPT} cp bash_${SCRIPT} ${SCRIPT} done chmod +x *.sh Figure 2 shows an example instrumented output. This output can be parsed to calculate the execution time of individual commands, such as this example of running cmake –G Ninja. Fig 2: Example output after adding timestamps to the build. After analyzing the build output with the timestamps we added, we identified significant time spent in processes that utilized only a single CPU core: running configure and cmake -G Ninja configuration phases, and pulling sources via git. Now that we understood the significant serial phases of the build, we were ready to optimize the server configuration to improve the build performance. Figure 3: Breakdown of build, based on adding instrumentation to measure its phases. This shows a large fraction of the build is due to running configuration scripts (cmake -G Ninja and configure), which are serialized and limit scaling. CPU Performance Governor The CPU frequency governor in Linux is the kernel mechanism that controls a range of performance and power management-related features for each CPU core. It can be used to tell CPU cores to run at maximum frequency, maximum power efficiency, or to dynamically scale on demand. You can see what the current setting is for all CPU cores with the command: Shell cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor The possible values for Linux are: powersave: Prioritizes energy efficiency, but modern CPUs can still ramp frequency under load. A good option for power-limited workloads and laptopsondemand/conservative: Ramps up frequency under load, and down when idle. conservative ramps up and down more slowly than ondemandperformance: Ensures the CPU cores are always running at maximum frequencyschedutil: A common default for modern Linux distributions, uses scheduler utilization metrics to adjust frequency If your CPU cores are not running at their maximum frequency, or if they need to ramp up to maximum frequency when you start a build, this can have a huge impact on build times. By explicitly tuning all CPU cores to be in performance mode, we can maximize build throughput for CPU-bound workloads like software compilation. You can manually set the governor to performance mode using either the cpupower utility or by changing the setting that we viewed earlier on the command line: Shell echo performance | sudo tee \ /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor These changes are runtime only, however. If you want this change to persist across reboots, you can use the tuned service and the tuned-adm utility on Linux to configure how aggressive the power management functions of your operating system behave when your system is idle, and to ensure that the option persists across restarts. To use tuned to configure your server for high performance mode, run: Shell sudo systemctl enable --now tuned sudo tuned-adm profile performance This setting should make a huge difference for high-core, CPU-bound build servers. Software compilation is a workload that benefits from cores running at maximum frequency. Dynamic governors (powersave, ondemand, conservative, schedutil) may let cores scale up only after work arrives, but this will introduce latency before full CPU power is available. For builds that spawn hundreds of parallel compilation jobs, this delay can add up across many cores and many files. Explicitly setting all cores to performance, the governor ensures that every core is immediately available at peak frequency, maximizing throughput for CPU-bound workloads. Performance Governor Impact: 17% Speedup After configuring the server to use the performance governor, we saw a 17% speedup in build time. The build running with a RAM disk improved from 1,063 to 909 seconds. Figure 4: Server CPU utilization vs. time for initial LLVM-MinGW after setting the CPU performance governor, we observed a 17% speedup in build time. CONFIG_HZ: The Heartbeat of the Linux Kernel Have you ever tried to work on multiple tasks at the same time — say, composing an email while preparing a presentation, and attempting to characterize and fix a software bug? This is inefficient for humans because we need time to stop working on one task, gather enough context for the new task, and regain full efficiency — a process we call context switching. CPU cores experience a similar overhead when changing the task they are executing. They must load the instructions and data for the new task into L1 and L2 caches, evicting the old context in the process, and refilling the CPU pipeline with new instructions. This whole process can cost hundreds to a few thousand CPU cycles on modern CPUs. With cores operating at 3 GHz (three billion CPU cycles per second), that means each context switch can incur an overhead of around one microsecond. Inside the kernel process scheduler, there is a separate heartbeat, as the Linux kernel determines which process will get access to each CPU core, based on its priority, how long it has been waiting for a time slice, and a few other criteria. The kernel compile-time parameter CONFIG_HZ determines the frequency that the scheduler makes these decisions, potentially triggering a context switch as it evicts one process and gives another process a time slice. By default, on most Linux distributions, the kernel is compiled with an idle-tickless configuration (the kernel build option CONFIG_NO_HZ=y) and a timer frequency of 250 Hz (the compile-time option CONFIG_HZ_250=y, which sets CONFIG_HZ=250). That means that on busy CPU cores, the process scheduler triggers 250 times a second, or once every 4 milliseconds, but on idle CPU cores, the tick is suppressed, reducing unnecessary interrupts. When the scheduler is triggered, the kernel checks whether the current process running on the core should be preempted and replaced with another process that is waiting for CPU time. On a build server, we typically do not want compilation processes to be preempted. These tasks are relatively short-lived for most source code files, often completing in under a second (and frequently under 100 ms) from C source to object file. As a result, if a compiler process is preempted by the kernel, context switches can represent a significant percentage of the execution time for compiling that source file. There are four possible values for CONFIG_HZ, with 100 Hz, 250 Hz, and 1000 Hz being most common. While higher CONFIG_HZ values are often chosen for workloads demanding very low latency and high responsiveness, for throughput-focused CPU-bound tasks like compilation, a lower CONFIG_HZ is generally beneficial. A lower frequency means the kernel's scheduler checks less often, reducing the likelihood of preemption for short-lived compiler processes and decreasing overall scheduler overhead. For our build server optimization, we will test the effect of setting CONFIG_HZ to 100, which sets the length of a jiffy (the time between clock ticks) to 10ms." Running with a 100 Hz kernel improved the build time to 889.2 vs. 909.31 seconds, for an additional 2% improvement. Building With AmpereOne Lastly, we ran the build on the AmpereOne 192-core processor and compared its performance against the Ampere Altra 128-core processor. The build completed in 603.6 s. The graphs (Figure 5) demonstrate that the multicore-intensive build phase was effectively halved, improving from over 300 seconds to approximately 150 seconds. Despite this significant improvement, we measured that approximately 75% of build time utilized one CPU core, with occasional spikes in demand. To further improve the server CPU utilization, one could run multiple builds and/or CI testing in parallel. Figure 5: Server CPU utilization over time for LLVM-MinGW on Ampere Altra Max ( top) vs. AmpereOne A192-32X Processor (bottom), showing a significant speedup Summary and Recommendations This guide explored our approach to configuring Linux application build servers for optimal performance on Ampere's Arm-based architecture. We detailed how various build systems enable parallel execution and demonstrated effective methods for measuring server utilization. Utilizing an innovative bash shell PS4 variable method, we instrumented the LLVM-MinGW build process, which involves numerous sub-projects across five different architectures. Our analysis revealed that a significant fraction of the build time was consumed by serial configuration phases. As these are inherently serial, they severely limit the overall CPU utilization of the server, even with powerful multi-core processors. Through our targeted optimization efforts — including using RAM disks, setting the CPU performance governor, and tuning kernel parameters such as CONFIG_HZ — we significantly improved build throughput for this specific project. Our findings also highlighted that for complex projects with persistent serial bottlenecks, the ultimate performance gains on high-core systems often come from parallelizing multiple independent builds or CI testing workflows. Recommendations We encourage you to apply these optimization strategies to your own Ampere-based build servers. To efficiently utilize the resources on a multi-core dedicated build server, we recommend that you implement the following recommendations: Implement parallel build flags – this enables the critical compute-intensive phase of builds to completely saturate available cores.Configure the performance CPU governor – ensure that your build server cores are not reducing performance by aggressively entering power-saving mode.Consider CONFIG_HZ adjustments to unleash the full potential of your hardware – context switches can have a significant impact, and slowing the heartbeat of a system minimizes them.To improve server throughput, explore offsetting build jobs or running testing jobs simultaneously. We have seen that configuration and linking stages do not scale as efficiently to multiple cores. If you start builds offset by time, or run jobs with different compute demands, you can execute these single-threaded parts of the build process in parallel with the intensive build stages of other jobs. How to optimally schedule build jobs to maximize throughput is left as an exercise to the reader. For further discussion, share your experiences or seek additional guidance by joining the Ampere developer community forum at https://community.amperecomputing.com. Check out the full Ampere article collection here.

Managing high-volume message traffic in distributed architectures is crucial. Efficient use of database and CPU resources is also very important. There are structures that allow us to receive messages in batches. The default Spring Kafka "BatchMessageListener" structure addresses this need. However, the processing of these messages often goes through a sequential bottleneck. This article will discuss the structure and usage of Kotlin Coroutines in detail. We will examine how to maximize Kafka message processing performance using Structured Concurrency principles and Resource Throttling techniques. Architectural Bottleneck: Sequential I/O Blocking On the current Kafka listener: Database or external service calls made for each message directly increase total processing times. If the processing speed of a message lags behind the message arrival speed and the max-poll-interval-ms time is exceeded, the consumer is removed from the consumer group. Rebalancing is triggered, and the partitions of that consumer are redistributed to other consumers in the group. Kotlin @KafkaListener(topics = ["usage-pool-topic"]) fun usagePoolListener(records: List<ConsumerRecord<String, String>>) { records.forEach { record -> processRecord(record) // Network latency + DB I/O blocking } } Solution 1. Batch-Fetch and In-Memory Map Structure Before any concurrent code is entered, data is retrieved collectively from all necessary entities. Multiple separate queries are converted into a batch query before data processing begins. The N+1 query problem is solved at the application layer. All data is cached once before being broken down into concurrent operations. Having the data cached significantly reduces our reliance on the database. Using the associateBy function, we transform the data into a map structure with X access times. This allows us to read the data safely from the maps instead of reading each concurrent operation from the database. Kotlin val messages = records.map { objectMapper.readValue(it.value(), UsagePoolRecord::class.java) } val usagePoolEntities = usagePoolRepository .findByIds(messages.map { it.usagePoolId.toBigInteger() }) .associateBy { it.usagePoolId } val lockEntities = lockRepository .findByUserIds(messages.map { it.userId }) .associateBy { it.userId } 2. Structured Concurrency Memory Management With Chunking The chunk structure serves two purposes. It prevents the creation of coroutines simultaneously. This prevents unnecessary memory usage. Each chunk writes to the database after all coroutines have completed their operations. Unnecessary connection pool consumption is avoided. Kotlin messages.chunked(150).forEach { chunk -> // Each chunk of 150 records is processed concurrently } Resource Isolation With limitedParallelism Why limitedParallelism? If the database connection pool has, for example, X connections, keeping the parallelism limit below X prevents "Connection Timeout" errors. Kotlin messages.chunked(150).forEach { chunk -> val deferredResults = chunk.map { record -> CoroutineScope(Dispatchers.IO.limitedParallelism(15)).async { try { processRecord(record, usagePoolEntities, lockEntities) } catch (e: Exception) { log.error("Operation error: ${record.key()}", e) buildErrorRecord(record, e) } } } val results = deferredResults.awaitAll() // Structural waiting collectAndAggregate(results) } The Dispatchers.IO.limitedParallelism(X) command limits the number of concurrent coroutines to X, preventing the DB connection pool from being exhausted.Each coroutine returns a result with the async command. The awaitAll() command waits for all coroutines in the chunk to finish before proceeding to the next step. runBlocking This function blocks callers until all concurrent operations are complete. This is the correct approach here because: It ensures that the Kafka consumer remains blocked to maintain its offset commit structure until all records in the batch are processed. We still benefit from concurrent operation parallelism within the runBlocking block. 3. Thread-Safe Result Structure After the awaitAll() operation, all results are collected in thread-safe queues. Then a single batch write operation takes place. Using MutableList structures to combine results returned from parallel processed coroutines can lead to data loss. At this point, lock-free data structures should be preferred. ConcurrentLinkedQueue uses CAS (Compare-And-Swap) algorithms instead of synchronized blocks. This provides superior performance in high-content write operations. Why Shouldn't We Use ConcurrentLinkedQueue? Concurrent operations (concurrent functions) perform simultaneous write operations to a shared collection of results. Using MutableList leads to race conditions. It performs well in secure and concurrent write operations. Kotlin data class AggregatedRecords( val processedSave: ConcurrentLinkedQueue<ProcessedEntity> = ConcurrentLinkedQueue(), val toDelete: ConcurrentLinkedQueue<UsagePoolEntity> = ConcurrentLinkedQueue(), val retryQueue: ConcurrentLinkedQueue<RetryEntity> = ConcurrentLinkedQueue() ) The DataIntegrityViolationException return is important. When two consumer instances are processing the same record, one of them falls into a unique constraint violation. Instead of making the entire batch fail, record-by-record deletion is performed. Kotlin AggregatedRecords.processedSave .chunked(150) .forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { batch.forEach { record -> try { processedRepository.save(record) } catch (e: DataIntegrityViolationException) {} } } } 4. Error Tolerance in Write Operations Batch write (saveAll) operations are performant. However, a "Unique Constraint" error in a single record can cause the entire batch to fail. The following structure is critical to meet Optimistic Locking or Idempotency requirements. Kotlin aggregatedRecords.processedSave.chunked(150).forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { // Fallback: Try one by one if batch fails batch.forEach { record -> try { processedRepository.save(record) } catch (innerException: DataIntegrityViolationException) { log.warn("Duplicate record skipped: ${record.id}") } } } } 5. Data Flow Diagram Ingress: The Kafka batch is caught with runBlocking.Preparation: All necessary context data is retrieved bulk from the DB.Execution: Coroutines are started asynchronously in chunks.Synchronization: The completion of all coroutines is awaited as a barrier point with awaitAll().Egress: Collected results are made permanent with saveAll. Performance Analysis and Results Conclusion Processing Kafka messages in Spring Boot with Kotlin Coroutines not only increases speed but also improves code readability and makes resource management deterministic (predictable). The use of runBlocking allows us to build a bridge between the blocking Kafka consumer thread and the suspended world without disrupting Kafka's offset management mechanism. Dependencies XML <dependency> <groupId>org.jetbrains.kotlinx</groupId> <artifactId>kotlinx-coroutines-core</artifactId> <version>1.7.3</version> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency>

Why Long Chats Need Session-Level Guardrails (CRA) Who this is for: Anyone building chat features, support bots, internal Q&A, coaching tools, RAG assistants. The Usual Setup (and What It Misses) A typical flow: User sends a message.You run moderation, rules, or a small model on that message (sometimes the reply too).If it passes, the big model answers. That is per message. It does not really “remember” the story of the chat. In a long chat: Message 5 looks normal.Message 12 still passes your keyword list.By message 20, something is wrong only if you compare it to how the chat started. So you can pass every single check and still end up with a bad session. That gap is what we call CRA: risk that adds up across turns, not in one obvious line. Figure 1: Each turn can look “green” while the overall thread is not. CRA in Plain English CRA = Conversational Risk Accumulation Idea: Each turn might look okay on its own, but together they break the purpose of the chat or what your company is okay with. What to build: Keep a little session memory (not the full transcript in logs — think IDs, hashes, and scores). After each assistant reply, update a few numbers that describe “how this session feels right now.” Those numbers are hints for dashboards, alerts, and gentle UI — not a courtroom verdict. Three Simple Scores + One Total (Example) We use a small, fixed set of scores and one combined score. Version tag in code: cra_telemetry_v1. Figure 2: Three inputs, one combined CRA score. ScorePlain meaningHow you might compute it (conceptually)S1Topic driftCompare the user’s recent text to how the chat started (or a stated goal). If they wander far from that, S1 goes up.S2Sensitive-looking repliesThe assistant’s answer looks like it contains patterns you care about (fake email shapes, “API key” wording, etc.). This means “flag for review,” not “we proved a leak.”S3Refusal tone shiftingTrack refusal-style phrases in the assistant’s answers over time. If refusals seem to soften late in the thread, S3 captures that shape.CRAOverall session riskA weighted sum of S1, S2, and S3, plus a small extra bump if the user or assistant text looks like prompt injection playbooks. Example weights we used: 35% S1, 45% S2, 20% S3. Rule of thumb: If you cannot explain a score in one short sentence to a product manager, do not use it to auto-block users. Hard Guardrails = Simple, Fast, “No” Hard guardrails are rules, not vibes. They should be cheap and run before you waste tokens. Examples: Max request size – reject giant payloads (HTTP 413).Rate limits – cap requests per IP so one client cannot drain your budget (429).Known-bad phrases – block obvious “ignore all previous instructions” junk (400).“Don’t paste secrets” – block prompts that look like “here is my SSN” (400) with a clear error.Lock down outputs – if your product only allows certain actions, check model output and tool calls against an allowlist before anything runs. These are not CRA. They are basics. CRA sits beside them. Figure 3: Hard = block or validate. Soft = warn, log, nudge. Soft Guardrails = CRA-Friendly, “Heads Up” Soft means: warn, log, maybe show a banner — not silent blocking. After a response, the API can add fields such as: cra_soft_notices – short text for humans (“high drift”, “sensitive-looking wording”, …).cra_signals – numbers for debugging: S1, S2, S3, CRA, turn count. Why start soft: Rules and heuristics misfire. A user might ask for fake email examples for a demo; S2 might spike on purpose. That is why the score is a signal, not proof. Bonus: Cache Duplicate Questions (Save Money) If someone double-clicks Send or retries the same text, do not call the model twice. Cache key idea: Python normalize(question) + mode + endpoint Cache the JSON answer for a few minutes. Mark responses with something like cached: true so the UI can say “from cache.” Browser Tip: Don’t Mix Up “New Chat” and Old Intent If S1 uses “first message of this session” as the anchor, browser storage can fool you: a new tab can look like a new thread while an old “first message” is still stored. Fixes: Store the anchor per session_id, not one global value.Expire or rotate the browser session after idle time so deploys and stale tabs do not reuse the wrong anchor. Telemetry vs. Guardrails (Two Different Jobs) TelemetryGuardrailJobMeasure and learnBlock or change behaviorWhen it hurts youToo many logs, privacyFalse positives, angry usersCRAGood fitUse soft first; hard only after review In logs, avoid raw secrets. Prefer hashes, lengths, and labels (channel, product area). Three Lines for Your Security Reviewer CRA is about conversation behavior over time, not a replacement for database security or tool-permission design.Labels for “bad session” are rare in the real world — use CRA to prioritize review, not as automatic guilt.If weights are public, people might game them — keep basic hard rules and spot checks anyway. Rollout Order (Keep It Boring) Ship hard limits (size, rate, obvious injection, output checks).Add session logging with safe IDs.Show soft notices only inside internal tools first.Tune thresholds on real traffic.Only then add hard session actions (pause tools, re-auth, etc.). Takeaway One-message checks are not enough for long chats. CRA gives you a simple story and a small set of session scores. Hard rules stop obvious abuse; soft CRA helps you see drift before it becomes an incident. Start with telemetry. Add blocking only when you understand the false positives. About the author: Sanjay Mishra is author of two books, The SQL Universe and Oracle Database Performance Tuning: A Checklist Approach. His research spans RAG architectures, NL2SQL, LLM safety, and enterprise AI governance, with work published in IEEE Access, Springer LNNS, and SSRN. He speaks regularly at universities and industry events on applied AI and data engineering. Tags / topics: #LLM #Security #Guardrails #Observability #OpenAI #Architecture #Chatbots

This post has a lot to cover. Before we get to any of it, I want to take on the uncomfortable subject first: quality. Two incidents from the past two weeks deserve a public explanation: one was a bug that fits into our normal iteration loop, and one was a serious mistake on my part. Both deserve the kind of explanation I would want if I were on the other side of the import. What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com. How We Think About Quality Codename One is a small open-source company. We are not a 200-engineer platform team with a dedicated SRE rotation and a separate QA org. We move fast, fast enough that we ship meaningful new code every week, and we put a lot of effort into making sure that speed does not come at the cost of breaking your apps. "A lot of effort" is doing some work in that sentence, so here is what it actually looks like: LayerCoverageUnit and integration tests710 Java test files exercised on every PR.Screenshot tests45 tests producing 190+ golden PNGs that are diffed across the iOS simulator, Android emulator, JavaSE, and headless Chrome. Both the OpenGL and Metal backends are diffed in parallel.ParparVM bytecode-translation suiteA separate, deeper test pass that exercises VM functionality (bytecode translation, garbage collector, native interop) beyond what the framework-level tests can reach.Cross-platform build matrix24 GitHub workflows build every PR against iOS, Android, JavaSE, and JavaScript.JDK matrixJDK 8 build, JDK 11 through 25 runtime. That is a meaningful amount of automated coverage, and it catches a lot before code ever lands. What it does not catch is brand new behavior, because there is nothing yet to compare a brand new feature against. The first golden rule of a new test is also the bug, until somebody actually runs the feature and tells us so. With that in mind, let's talk about the two specific incidents from the past two weeks. Sticky Headers Were Half-Baked, and That Was By Design Last week's post introduced StickyHeaderContainer with an animated transition between section headers. Within a couple of days the issue tracker had #4849, the NONE and FADE transitions were not behaving correctly and the swap had visible jitter. We turned the fix around within hours of the report. That round trip, ship, hear from a real user, fix, is the deal we make with the community. Our pixel-diff harness is excellent at catching regressions in code that already exists. It is structurally bad at catching the first version of a brand-new component because there is nothing yet to compare against. We could have sat on StickyHeaderContainer for another two weeks polishing it in private and we would have shipped a worse component, because we would not have had Thomas's eyes on it. Iterating in public, with a tight feedback loop, is how a team our size keeps moving. The SIMD Bug, Which Was My Mistake PR #4842 is a different story. The SIMD code on iOS uses alloca to put working buffers on the stack for speed. That is the right call for small buffers and the wrong call for large ones, past a certain size the stack request fails outright and the process crashes. The image API uses these buffers, and on large images the buffer crossed the threshold. Result: image API crash in production. This was not a new feature. It was a change to existing, mature code. We took the precautions we always take when changing code with a long history, ran the existing tests, every existing test passed, and the patch shipped. The bug got through anyway because the test coverage validated the SIMD path on small images and never on the large ones that actually triggered the failure. That hole in the tests was mine to spot, and I did not spot it. Once it was reported, the fix turned around in well under 24 hours. The patched version detects the over-large case and falls back to a heap allocation. Small SIMD ops keep the fast alloca path. Large ones no longer crash. New tests cover the threshold case, so this specific shape of bug cannot regress. This should not have happened, but realistically, it will happen again, not this exact bug, but one like it. Tests are not perfect; mine certainly are not. So the take-home is the part I want to lean on: Codename One is the first line of defense between bugs and your end users. We are not the last line. Test your application before you release. If your app supports it, use a beta channel (TestFlight on iOS, Play Console internal or closed tracks on Android) so a bug like this hits you before it hits the people who paid for your app. That tiny extra step is the most reliable protection your users have. We are also actively brainstorming the next generation of crash protection inside the framework. The current crash protection sits at the EDT and catches RuntimeExceptions that user code throws. The next generation needs to extend further, into native crashes, into earlier startup, and into a more useful diagnostic payload that comes back to the developer instead of just the device log. There is no PR yet, we are still working out the shape, but it is the major framework-level investment we are making to give the community a stronger floor underneath their apps. With the quality conversation out of the way, the rest of this post is about the things that actually shipped. Metal Is Here PR #4799 is the largest single change we have landed in months: a complete Metal rendering backend for iOS. It sits next to the existing OpenGL ES 2 path, behind a single build hint, with its own CI job and its own pixel-diff goldens. Metal is Apple's modern graphics API. OpenGL on iOS was deprecated by Apple back in iOS 12; it still runs today, and we kept it running for years, but "deprecated" on Apple is a slow countdown that ends with the platform pulling support. Moving to Metal now is how we get ahead of that, and it brings real benefits to your apps: Better rendering performance. Lower draw-call overhead, modern command-encoder batching, and pipeline state caching add up to smoother scrolling and faster transitions on the same hardware.Less battery use. Metal's reduced CPU overhead per frame means the GPU spends less time idling and the CPU spends less time bookkeeping. Long-running, graphics-heavy apps benefit the most.Crisper text. Glyphs go through a CoreText atlas, which produces noticeably sharper rendering at the same size, with proper kerning and correct handling of complex scripts.Pure-GPU gradients. Linear and radial gradients render entirely on the GPU instead of round-tripping through a CGContext bitmap.Access to modern Apple graphics features. New iOS rendering features (variable-rate shading, mesh shaders, ray tracing on Apple silicon) are Metal-only. Sticking with GL means watching that train leave without us. To enable Metal in your project, set the build hint: Plain Text ios.metal=true Everything else stays the same. The Java surface is unchanged; your existing code keeps working. We plan to flip Metal to be the default within two weeks, assuming no major issues surface. The ios.metal hint will stay around (set it to false to opt back into GL), but new projects and the build server's default behavior will move over. If you ship an iOS app, please set the hint now and put your real flows through it. We want regressions to surface against your real screens, not the day after the default changes. The most user-visible improvement from the Metal port is text. Here is the ShowcaseTheme capture from the Metal screenshot suite: And the SpanLabelTheme capture, which is the real test for body-copy rendering, multiple lines, variable widths, the kind of paragraphs that show up in real apps: The Metal Dialog capture is also worth showing because the translucent surface composites correctly against the textured backdrop: The End of the Skin Downloader PR #4758 ships the Skin Designer as a JavaScript bundle, embedded into the website at /skindesigner/, the same way the Playground and Initializr are embedded. You can build a skin in the browser, save it, and use it in your simulator without installing anything. This is bigger than a website convenience. It is how we get out of the skin business. For the entire history of Codename One, "no skin for the iPhone 16 Pro Max" or "no skin for the iPad mini 7" has been a recurring complaint, and we have published skins as fast as we could. That model never scaled. Apple ships new device sizes faster than any of us want to maintain a parallel skin catalog, and Android has effectively infinite device shapes. Today, we are deprecating the skin downloader and moving to a generic browser-based authoring tool. To be clear about what is changing: Existing skins are not going anywhere. Every skin that ships today will continue to work, will continue to load in the simulator, and will continue to be supported. We are not removing them. If your team has a workflow built around an existing skin, that workflow keeps working.We will stop issuing new skins. When the next iPhone or iPad ships, we will not publish an official skin for it. Anyone can build one in the new designer in minutes, and that "anyone" includes us, of course, but it also includes you. The "no skin for X" problem is solved generically. If you are running a niche enterprise app on a less-common Android device, you no longer have to wait on us to produce a skin for it. Build it once, drop it into your team's shared assets, done. How the Wizard Works The Skin Designer turns a device specification (resolution, PPI, fonts, safe-area insets, cutouts) into a .skin file that the JavaSE simulator can load. It runs in your browser. There is nothing to install. The wizard is intentionally opinionated. It ships with a curated device catalog, generates the device frame procedurally, and writes a skin layout that matches the iPhoneTheme.res, iOS7Theme.res, and android_holo_light.res themes shipped with Codename One. If you only want a skin and don't care how it is built, pick a device, accept the defaults, click Finish, then Download skin. The file is ready to load via Add in the simulator's Skins menu. Stage 1, pick a device. The first step shows a card per device from the bundled catalog. The search box filters by name (it matches both the model and the brand), and the chips below narrow by form factor: All / Phones / Tablets / Foldables. Picking a device pulls in its resolution, PPI, screen size, default safe-area insets, and the iOS or Android system font names from the catalog, then seeds a sensible starting frame: notch, island, or hole presets are applied automatically based on the device's hardware. The catalog is large, the grid is capped to the most recent matches by default, and type into the search field to find older or less-common devices. Stage 2, pick a starting source. There are three ways to seed the skin's body image: Pick a shape generates the device frame procedurally from a small preset library (rounded rect, notch, dynamic island, punch-hole, corner hole, classic home button). The frame is rendered as a dark gradient with the screen rect (and any cutouts) carved into it. Best when you want a generic-looking iPhone or Android frame and don't care about exact hardware fidelity.Upload an image opens an image picker. The wizard scales the image into the device's resolution, then carves the screen rect and cutouts on top. Use this when you have a marketing render of the specific device you are targeting.Blank rectangle collapses the bezel and corner radius to almost nothing, drops every cutout, and turns the home indicator off. The screen fills the entire skin. Useful for desktop or web simulators where the device frame would just be visual noise. Stage 3, the editor. The editor is split into two panes: a live preview on the left that paints the device frame, screen tint, cutouts, and home indicator, and a sidebar on the right with three tabs. The Shape tab shows a preset grid (Rounded rect, Notch, Dynamic Island, Punch-hole, Corner hole, Classic home) and dimension fields for corner radius, bezel thickness, and a toggle for the bottom home indicator. iPhones from X onward and most modern Androids should leave the indicator on; classic devices with a hardware home button should turn it off. The Cutouts tab lists every cutout currently on the skin. Tap a row to expand its width, height, and offset fields. The three add buttons at the bottom seed a sensible default of each type. Notch (180 x 30 viewbox px) is a physical hardware cutout drawn in the device frame above the screen rect, mirroring iPhone X / 11 / 12 / 13 hardware. Island (120 x 35) is a Dynamic Island, software-reserved space rendered as an opaque pill inside the screen rect, floating on top of the iOS status bar. Hole (28 x 28) is an Android punch-hole camera, rendered like the island. When the wizard generates the .skin, it automatically extends safePortraitTop to cover any in-screen cutouts so app content lands below the floating shape. The Info tab is mostly read-only and shows what is about to be written into skin.properties: name, width, height, PPI, pixels-per-millimeter, and the user-editable safe-area insets. The wizard intentionally does not write smallFontSize, mediumFontSize, or largeFontSize, when those are absent the simulator auto-derives them from pixelMilliRatio, which is what you want on high-PPI screens. Stage 4, finish and download. Clicking Finish renders the portrait skin image at the device's actual resolution with rounded corners, transparent screen, opaque cutouts, and a home indicator if enabled. It synthesises the landscape skin by 90-degree rotation, writes the skin_map.png overlays that mark the screen rectangle for the simulator's screen-position detection, bundles the appropriate native theme inside the skin zip, and writes skin.properties with the platform metadata, safe-area, PPI, and display rect. Clicking Download skin hands the file to the browser's download dialog. After the file is on disk, drop it into your simulator's skins folder (or use the Add command in the simulator's Skins menu) and your new device should appear in the picker. A generated .skin is just a renamed zip: Plain Text Apple-iPhone-16-Pro.skin/ skin.png # portrait body (device frame + transparent screen + cutouts) skin_l.png # 90-degree rotated portrait skin_map.png # black rect = screen, white = frame, used for hit-testing skin_map_l.png # rotated map iOS7Theme.res # bundled native theme (or android_holo_light.res / winTheme.res) The full developer-guide chapter at Skin-Designer.asciidoc walks through every stage with annotated screenshots and documents the skin.properties keys the wizard writes (roundScreen, displayX/Y/Width/Height, safePortrait*, safeLandscape*, overrideNames, system font families, PPI, and pixel ratio). Eating Our Own Dog Food While we're talking about the Skin Designer, this is the right moment to point out something I think is genuinely worth highlighting. The Initializr, the Playground, and the Skin Designer are all open source Codename One apps. They are written in Java using the same Codename One UI framework you use to build your iOS and Android apps, and they are deployed to the browser through our JavaScript port. Every interaction you have with these tools, the device picker grid, the live preview rendering the device frame and cutouts, the form-driven editor with its tabbed sidebar, the file generation that bundles a .skin zip in your browser tab, is the same Codename One code that ships in your apps. The Container, Form, BoxLayout, theming, and event-handling code is identical to what you would write for a phone build. The JavaScript port translates it into something a browser can run. These three tools are the most direct demonstration we can give of what Codename One is capable of: real, non-trivial UIs, with state, file I/O, image generation, and complex layouts, running smoothly inside a browser tab. If you have ever wondered whether the JavaScript port is production-grade enough for a real application, the Initializr, Playground, and Skin Designer are your answer. They are also the answer to "Can Codename One build apps that go beyond mobile?" Same codebase, deployed to a fourth target, with no rewrite. The source for all three lives in the same CodenameOne repository, the framework itself does. If you want to see how a non-trivial Codename One app is structured, those are three good places to start reading. iOS multi-line TextArea: Return as Done PR #4859, driven by issue #4854, gives multi-line TextArea an opt-in flag that makes the iOS keyboard's Return key act as Done. It closes the editor and fires the Done listener instead of inserting a newline. This is the iOS Reminders app behavior: a growing, multi-line task-title field where Return finishes the entry. The reason it has to be a flag is that real iOS does not expose this as a built-in primitive. Reminders implements it on a UITextView whose delegate intercepts \n in shouldChangeTextInRange:. We replicate that exactly, gated behind a client property so existing layouts are untouched: Java TextArea ta = new TextArea("", 3, 30); ta.putClientProperty("iosReturnExitsEditing", Boolean.TRUE); While the flag is set, the keyboard's Return key is relabelled to Done (UIReturnKeyDone). Default behavior is unchanged: the flag defaults to off, only takes effect on multi-line TextAreas, and only intercepts an exact "\n" replacement so pasted multi-line text is unaffected. Diagnostics for Status-Bar Tap Scroll-to-Top PR #4868, driven by issue #3589, adds three complementary diagnostics for the iOS status-bar tap path. We shipped a fix earlier (#4857), and the reporter still saw no scroll on the device. Rather than another sweep in the dark, we built tools to make the path observable. Simulator menu, Simulate > iOS Status Bar Tap. Synthesises the same (displayWidth/2, 0) tap that scrollViewShouldScrollToTop: dispatches, pops a dialog reporting the responder UIID, the build-hint state, and an OK / PROBLEM verdict, then actually fires pointerPressed and pointerReleased so any wired-up scroll-to-top is observable.Device-side properties. Display.getProperty("cn1.iosStatusBarTap.count"), cn1.iosStatusBarTap.lastEpochMillis, cn1.iosStatusBarTap.lastX/Y, and cn1.iosStatusBarTap.proxyInstalled let you inspect the path on a real iPhone. Run your app on the device, tap the status bar, and read the property. That distinguishes "iOS never delivered the message" from "iOS delivered it but a CodenameOne component intercepted the tap".Regression coverage. StatusBarTapDiagnosticScreenshotTest exercises the exact same code path through a 2x3 frame grid, with the visible counter rising and the scroll position alternating, so future regressions surface in CI. Simulator: Dark/Light Mode Toggle PR #4871 adds a Dark/Light Mode submenu under the simulator's Simulate menu with three options: Dark Mode, Light Mode, and Unsupported (the default). Selecting an option flips Display.isDarkMode() (Boolean.TRUE / Boolean.FALSE / null) and calls refreshSkin(...) so themes that branch on @darkModeBool re-render immediately. The choice is persisted under the cn1.simulator.darkMode Preference so the simulator restarts in the mode you left it. Combined with the Native Theme menu we shipped two weeks ago, you can now sit on a single skin and flip between iOS Modern, Material 3, iOS 7, and Holo Light, in light, dark, and unsupported, in seconds. The everyday win is being able to verify your own theme looks right in dark mode without restarting the simulator. Heads-Up: Weekend Backend Maintenance This weekend, we will be doing some maintenance on our build backend servers. The work is mostly invisible from the outside, but it touches enough of the infrastructure that you might see intermittent build issues during the window: slower-than-usual builds, the occasional retry, possibly a short period where new builds are queued. We are doing it because the underlying backend needs to move forward, and the cost of putting that work off keeps compounding. We will keep the disruption as short as we can. If you have a hard release deadline that lands this weekend, please plan around it. Otherwise, the impact should be small, and you can build through it normally. Warning: Android 16 Will Effectively Disallow Locking Orientation Thanks to Durank for flagging #4879. The Android 16 behavior changes include a meaningful change to how Android handles orientation, in short, on large-screen devices the platform will ignore an app's request to lock orientation. If your app calls Display.lockOrientation(...) or sets a fixed orientation in the Android manifest, that lock will be honored on phones but effectively ignored on tablets and foldables once the device targets Android 16. There is not much we can do about this on the framework side. It is a platform-level decision, and there is no public opt-out for general apps. The realistic path forward is to design layouts that work in both orientations, and to test your app against both portrait and landscape on a tablet before Android 16 reaches your users. We will keep watching for any opt-in path Google publishes, but for the moment, please plan accordingly. Why the Version Jumped to 7.0.242 A small note on versioning: the current release is 7.0.242, not 7.0.238 as you might expect from the cadence. The gap is real and worth explaining. We made a fix to the Maven archetype that brings over the features we added in the Codename One Initializr to projects created from the command line. The change itself is straightforward, but it interacted badly with our release build automation, and we had to delete several releases along the way to get the pipeline back on its feet. The version numbers we burned in the process are the visible scar. The bright side is that command-line mvn archetype:generate now produces projects that line up with what the Initializr generates, which is what we wanted all along. Wrapping Up We closed 24 issues in the past week, a meaningful share of them direct beneficiaries of the Metal port. Old GL-only rasterization diffs, font sizing on retina, polygon drawing artifacts, perspective transform issues, things that the Metal pipeline simply renders correctly out of the box. Migrating the rendering layer turned out to be the cleanest way to retire a long tail of small bugs at once. With the new Skin Designer landing in the same week, two long-running structural problems went from "we should fix this someday" to "this is fixed and shipping". If you ship an iOS app, please flip ios.metal=true this week and run your real app through it. We want to find any remaining issues now, not the day we flip the default. The issue tracker is here, the Playground is the easiest place to poke at the new themes, and the Skin Designer is live on the site. A specific thank-you this week to Thomas (@ThomasH99) for the sticky-header transition report and the Picker centering follow-up, Francesco Galgani (@jsfan3) for the iOS Reminders-style Return RFE, and the reporter on #3589 for sticking with us through a multi-PR diagnosis on the status-bar tap. The "tests cannot catch everything" section above is also a "and that is why we need you" section. It works because you keep filing.

By Shai Almog CORE

Here is what a production cascade looks like when nobody did anything wrong. An alert fires on a microservice showing elevated latency. The signal is accurate. The automated remediation agent picks it up immediately and does exactly what it was built to do: restart the affected service and reroute traffic. The action is within scope, the credentials are valid, and three seconds later, the platform reports a successful remediation. Then, four dependent services go dark. The postmortem will call it a cascade. The dashboard will show a clean execution on the first incident and a second incident opening 90 seconds later. Nobody will find an error log on the remediation itself because there was none. The agent was not wrong. The action was technically correct. What nobody had built was the ability to ask: given what the system is carrying right now, is this the moment to add more disruption to it? That is not a monitoring gap. Monitoring told everyone exactly what was broken. It is an observability architecture gap — the difference between knowing what is failing and knowing whether the system can safely absorb what you are about to do to fix it. Figure 1: The alert was correct. The instrumentation gap was not in detection — it was in the question asked before acting. The Failure Pattern Is More Consistent Than Teams Expect I ran into this structurally while doing chaos engineering on enterprise SD-WAN infrastructure at Cisco. We were running experiments against production-grade environments across large financial services and telecom customers, and standard chaos tooling kept finding the wrong failures. It was injecting faults into systems whose state had already shifted past the parameters we had set at the start of the experiment. The faults that caused real damage were the ones that chained with conditions already present in the environment — elevated resource utilization, two services over, a background process that had been running for 45 minutes, consuming memory that a restarted service needed, a connection pool sitting at 89 percent because of an unrelated batch job. None of those conditions was hidden. Everyone was instrumented. The problem was that nobody was reading them together as a composite signal before deciding how hard to push the system. We were answering the wrong question. We built a methodology to fix it. Instead of setting static experiment parameters, the engine reads live telemetry before each iteration, derives from that telemetry the system's current capacity to absorb perturbation, and calibrates the intervention intensity accordingly. A feedback loop between the actual impact and the intended impact across successive iterations finds the behavioral boundary without disabling the environment. That methodology became USPTO Patent No. US12242370B2. Patent: https://patents.google.com/patent/US12242370B2/en What we built for SD-WAN infrastructure is the same thing agentic AI deployments need now. The underlying problem is identical: an automated actor is making decisions about whether and how to intervene in a live system, using a signal that accurately describes what is broken but says nothing about what the system can safely absorb in the moment the decision is made. Why AWS FIS and Gremlin Will Not Find This for You Infrastructure fault injection is good at what it does. AWS FIS, Gremlin, and Chaos Toolkit test whether your Lambda survives throttling, whether the event pipeline recovers from a queue outage, and whether the hosting environment holds up under resource pressure. These are legitimate questions, and the tooling answers them well. They just do not test the failure mode that is generating the most expensive incidents as agentic AI deployments scale. An agent's worst production failure is not a cold start timeout or a concurrency breach. It is a clean, successful invocation that executes the wrong sequence — because the combination of inputs, tool call results, and current system state put the agent at the edge of its operational envelope, and nobody built a test that ever got it there. Air Canada's chatbot did not crash. It executed correctly in a scenario the designers never tested. No infrastructure fault injection exercise would have found that boundary because the boundary was not in the infrastructure. The same structure shows up in autonomous remediation. The agent reads a real signal, takes a valid action within its authorized scope, and produces an outcome nobody intended because the action was correct in isolation but wrong given the composite state around it. Standard tooling reports a clean execution. The cascade shows up in the next incident ticket. Finding the behavioral boundary requires a test methodology that reads live system state before calibrating experiment intensity — not one that applies static parameters to a system whose state has already shifted. Static parameters applied to dynamic systems find the failure modes you designed the test to find. They miss the ones that actually hurt. Three Instrumentation Gaps to Close Before Your Agents Hold Production Credentials These did not come from a research paper. They came from postmortems — at Cisco across financial services and telecom customers, and at Splunk across thousands of enterprise observability deployments. The same three gaps show up every time. 1. Concurrent workload state across the dependency graph, not just the service under incident. A service restart that is safe in isolation is frequently dangerous when adjacent services are already running above their normal resource ceilings. The absorb capacity question is a system-level question, not a component-level one. Most runbooks do not include a pre-action resource check across the dependency graph of the service being touched. Automated agents have no reason to be different. What to build: a pre-action query that checks whether any first-degree dependency of the service being remediated is above 80 percent of its 24-hour baseline utilization. One data point. It exists in most observability stacks already. It is almost never surfaced in an incident context. 2. Pending operations competing for the same recovery resources. A recovering service needs I/O headroom during the 60 to 90 seconds after restart while it rebuilds its in-memory state. A background index rebuild consuming 30 percent of available I/O is invisible to the incident response flow because it is not itself failing. It does not show up in any alert. It shows up in the postmortem as a contributing factor. What to build: a pre-action inventory query against active background and scheduled operations on the same infrastructure tier as the remediation target. Not continuous monitoring — just one read before acting. 3. Intervention intensity matched to current system state, not last month's playbook. The remediation that worked last Tuesday was calibrated to last Tuesday's system state. Applying it at the same intensity to a system currently carrying three extra loads is not a reliable practice — it is reusing a number that made sense in a context that no longer exists. Every automated remediation action should answer one question before executing: Is the system in the same absorb capacity range as when this intervention was validated? If it is not, stage the action, reduce intensity, or escalate. This is not complicated engineering. It is a check that almost nobody has built. The automation is not the problem. The automation acting without a pre-action absorb capacity check is the problem. Building that check is a day's work. Not building it is how you get cascades that look like they came from nowhere. "We were validating system health, not output integrity. That experience changed how we define resilience; it is no longer just about systems staying up but about systems staying correct under stress." — John Russo, VP Healthcare Technology Solutions, OSP Labs Which Automated Actions Need This Check and How Urgently Not every intervention carries the same absorb capacity risk. Here is a working classification based on what I have watched produce incidents. The cluster restart and downstream workflow rows are where most of the expensive postmortems come from. Intervention Absorb Risk Minimum Pre-Action Check Automate or Escalate Read-only diagnostics (health checks, metric queries, log pulls) Very Low None Fully automatable, no check needed Traffic rerouting (LB weight shifts, circuit breaker trips) Low to Medium Downstream service vs. 24hr baseline Automate with dependency check; escalate if downstream >75% baseline Single service restart (pod recycle, instance restart) Medium I/O headroom + active background ops on same tier Automate if headroom clear; escalate if background ops active Cluster-level restart (rolling or full, multiple instances) High Full dependency graph resource state + pending ops inventory Stage the restart; never run under pre-existing cross-service stress Config or schema change (feature flags, parameter updates) High to Very High All checks + rollback path validated Human review required outside the nominal absorb capacity range Agent-initiated downstream workflow (external API calls, cross-service triggers) Very High (often irreversible) Intent-execution separation + full pre-action assessment Human authorization unless the action is fully reversible Table 1: The cluster restart and downstream workflow tiers are where most production cascades originate. The check is cheap. The postmortem is not. How to Build the Absorb Capacity Layer Adding absorb capacity as a first-class observable does not mean replacing what you have. Your existing metrics, traces, and logs are doing their job. The gap is not in those signals — it is in the layer that reads them together and produces a single pre-action number before any automated intervention fires. The architecture has three parts. First, a live absorb capacity index: a rolling calculation across the dependency graph of each critical service, reading resource utilization deltas against the 24-hour baseline, shared connection pool saturation, active background operation inventory, and concurrent workload state. Output is a single number per service cluster — current absorb capacity as a percentage of the validated intervention tolerance.Second, an intervention intensity governor that reads that number before any automated remediation executes. If the index is within range, the action proceeds. If not, the governor selects a reduced-intensity variant, stages the action, or sends it to human review. It does not touch the remediation logic. It gates execution.Third, a behavioral boundary testing loop adapted from the intent-based chaos engineering methodology in Patent US12242370B2. Periodic pre-production tests read live telemetry, derive calibrated adversarial pressure from the current absorb capacity model, and use an actual-versus-intended impact feedback loop to keep the model current. Without this loop, the pre-action check is comparing today's system state against a capacity model that was valid when you built it six months ago. Figure 2: The absorb capacity layer sits between existing observability and the autonomous agent. The behavioral testing loop (Patent US12242370B2) keeps the capacity model current as the system evolves over time. The Check That Almost Nobody Has Built Most teams I have worked with have good observability. The signals are there. The alerting is tuned. The dashboards show what is failing in real time. What they have not built is the layer that reads all of it together and answers a different question: not what is broken, but whether the system is in a state that can take what you are about to do to it. Autonomous remediation agents and agentic AI systems make that question urgent in a way it was not when the decision-maker was a human engineer with pattern recognition built over years. The human hesitated. They glanced at adjacent services. They asked the on-call SRE if anything else was running before they pushed the big red button. The agent does not hesitate. It reads the signal, acts within scope, and files the result as success. RL-calibrated infrastructure failures are recoverable. A cluster goes down, the runbook fires, the service comes back. Behavioral failures in systems with real external side effects — agents that trigger downstream workflows, confirm transactions, modify records across services — are not always recoverable in the same way. The damage lands in external systems before any alert fires. Adding absorb capacity as a first-class observable is not a large infrastructure project. The signals you need are already in your stack. The composite read, the pre-action check, the governor that gates execution — none of this requires new technology. It requires deciding to ask the right question before the agent acts, and building the thin layer that makes that question answerable in real time. The observability you have is telling the truth. It is just not telling the whole truth yet.

AI agents have come a long way. They aren’t just answering simple questions, but they’re handling order checks, summarizing support tickets, updating records, routing incidents, approving requests, and even calling internal tools. As these agents slip deeper into real business workflows, just peeking at model logs isn’t enough. Teams need to see everything: what the agent did, why it did it, which systems it poked, and whether the end result actually helped the business. Agent Observability That’s where agent observability comes in. Traditional observability lets teams watch over their apps, APIs, databases, and infrastructure. Agent observability goes a step further. It shines a light on the whole AI workflow: it connects the dots from the user’s request to the agent’s decisions, the tools it touches, the systems it interacts with, and all the way to the final outcome. Let’s see a customer support example. Say a customer messages, “My subscription renewal failed, but I got charged twice.” A human rep checks the account, payment history, billing rules, refund policy, and ticket history before answering. Now, an AI agent might do that job automatically. It’ll spot the billing problem, look up the customer record, call the billing system, check for duplicate payments, and either resolve the issue or escalate it if things get too messy. On the surface, this whole thing just looks like a simple chat. However, under the hood, it’s a full-on workflow. If you want good observability, you need that behind-the-scenes view: Why bother? Because the final response doesn’t tell you the whole story. If the customer comes back unhappy, you need to nail down whether the agent checked the right account, used the right billing tool, hit an error, misread the request, or escalated when it couldn’t help. Don’t just watch the answer: Follow the whole journey When you break down agent interactions, a few basic layers show the full picture. First, track the user request. What did the user ask? Was it urgent, fuzzy, sensitive, or bound to a customer contract? Second, watch the agent’s action. Did it answer straight away, ask a follow-up question, search a knowledge base, use a tool, or hand off to a human? Third, note the context. What sort of information did it use? Did it pull a help article, customer details, invoice, ticket, policy, or product data? Fourth, log tool usage. Did the agent call billing APIs, CRM systems, databases, incident tools, or an approval workflow? Did those calls work, or did they fail? Lastly, look at the result. Did the agent fix the customer’s problem? Was the ticket reopened? Did a human have to clean up after the agent? Without these layers, you’ll know when something was slow or incorrect, but not why. Maybe the context was off, a tool call failed, it lacked permissions, the prompt changed, or something further downstream broke. Use a Single ID to Track Everything One of the easiest fixes is to tag the whole workflow with a tracking ID. Let that ID travel with the request, from the interface through the agent, tools, APIs, and your business systems. Now, if a support ticket gets botched, the team can retrace every step: what the customer asked, what the agent understood, which account it checked, what the billing system said back, and why the agent chose to close or escalate. It’s not just for support. Maybe your SRE team uses an AI agent to help dig into a production alert. The agent scans logs, checks recent deployments, reviews database metrics, and suggests the likely cause. That same tracking ID means you’ll know exactly which systems the agent checked and whether it missed anything crucial. Don’t ignore tool calls; they’re real actions Here’s where things get serious. When an agent calls a tool, it’s taking action. Looking up customers, updating records, approving requests, creating tickets, and kicking off workflows need to be watched closely. For each tool call, capture details like tool name, how long it took, success or failure, retries, permission results, error messages, and what actually happened. Take a finance workflow. Say the agent reviews vendor invoices by extracting details, matching with a purchase order, checking taxes, and routing exceptions to finance. If an invoice gets approved by mistake, did the agent misread the invoice? Match it with the wrong purchase order? Miss a policy update? Or did the finance system return incomplete info? That’s why tracking tool calls is critical. A wrong answer in chat is one thing, but a wrong move in your business system can lead to trouble such as money lost, operations disrupted, and even compliance issues. Understand Agent Decisions, But Protect Privacy Teams need to understand what the agent did, but you don’t want to log every single “thought” it had; it’s just unnecessary noise. Instead, record decision details in a structured way. Example: Intent: billing disputeConfidence: mediumTool: billing lookupReason: account verification neededPolicy result: escalateFinal action: handoff to human Now you have enough to debug the workflow and for reporting, without exposing raw thought streams. You can spot how often agents escalate from low confidence, where tools fail, or if policy rules stop an action. Connect Observability to Business Outcomes Don’t just track the tech stuff; what really matters is whether the agent gets the job done. Watch business metrics like: Resolution timeEscalation rateWorkflow completion rateTool failuresCost per workflowSLA hits or missesReworkHow often humans step in If you’ve got an e-commerce agent helping buyers pick products, check inventory, apply discounts, and guide checkout, you want to know: did the customer actually buy the item? If checkout drops after you tweak a prompt, find out why. Did the agent push out-of-stock items? Apply discounts wrong? Use the wrong tool? Lose customers with confusing answers? Observability at this level helps both engineering and business teams get answers, fast. Build Dashboards for Different Audiences Everyone’s got different needs. SREs care about latency, failed tools, retries, issues with dependencies, and expensive cost spikes. Security teams focus on policy denials, suspicious tool actions, sensitive data flags, or prompt injection attempts. Product owners want completion rates, escalations, customer satisfaction, and abandoned workflows. Engineers need to see how agent behavior shifts after you change the model, prompt, workflow, or deployment. Business folks need throughput, SLAs, cost savings, and improvements to customer experience. Take security operations. Say an agent checks suspicious logins, identity logs, privilege changes, and endpoint activity. Security needs to know: did the agent just review info, or did it try to lock an account? If it got blocked, you want that visible, too. Alert on AI-Specific Failures AI agents fail in new ways. Teams need alerts for things like sudden spikes in tool denials, fallback responses, unexpected tool usage, cost blowups, prompt injection attempts, completion drops, or escalating cases. If an agent suddenly goes wild with refund actions, it could mean a prompt is off, a policy is weak, or something’s getting abused. If fallback responses shoot up, maybe the knowledge base is broken. Costs spike? Maybe the agent is stuck looping, retrying, or making unnecessary expensive calls. Tie alerts to deployments, too. Agents change behavior after you update a prompt, switch models, change schema, adjust policies, or edit a workflow. Teams should compare how the agent behaved before and after. A Simple Way to Grow Observability Observability matures in steps. Basic logs: prompts, responses, errors, timestampsTool visibility: what got used, if it worked, how long it tookEnd-to-end traces: follow the user request through the agent, tools, APIs, systemsBusiness-level result tracking: resolution, escalation, completion, rework, cost, SLAAutomated alerts: regressions after updates, anomalies, unusual patterns Observability is more about making sense of the whole workflow and visibility. Teams need to know what users wanted, what the agent decided, which info it used, which tools it grabbed, which systems it touched, and whether business value was delivered. As AI agents settle into production, observability has to cover more than just servers and app logs. The teams that win will be the ones who trace agent behavior end to end, spot failures early, explain what happened, and keep improving safely.

By Srinivas Chippagiri CORE

If you've ever run a data streaming service that handles more than one type of workload, you've probably hit a wall that no amount of round-robin tuning can fix. This is a common failure mode in production streaming environments. This post is about the specific ways traditional load-balancing strategies break down when your traffic isn't uniform. I'll focus on CPU utilization as the primary example throughout, since it's the most common bottleneck in compute-heavy streaming workloads, but the same principles apply to memory, network bandwidth, and other system resources. What Makes Streaming Services Different A typical data streaming service ingests messages from an upstream log or message bus, partitions them into logical units of work, and processes each partition on one or more compute instances. An orchestrator may assign partitions to compute instances and attempt to keep the load even. On paper, this sounds like any other load-balancing problem. In practice, it's trickier. The load balancer operates on observable proxy metrics - bytes per second, messages per second, number of assigned partitions - while the actual bottleneck can often be the resource consumption per partition. In a perfectly homogeneous world, these proxies correlate tightly with actual resource consumption. But the real world is not homogeneous, and the gap between proxy and reality is where all the problems live. Not All Messages Are Created Equal The most insidious source of imbalance is heterogeneous processing cost per message. I think of this as the "hidden weight" problem. Consider a streaming service that processes two classes of workloads. The first class is lightweight messages: each triggers a simple, low-cost operation, and processing cost tracks proportionally with message count or byte volume. The second class is compute-intensive messages: a single message may trigger operations that are 10-100x more expensive than a lightweight one. The cost depends on domain-specific factors - algorithm complexity, data structure sizes, model inference, and so on. A load balancer that distributes partitions based on throughput sees these two classes as equivalent if their throughput numbers are similar. Instance A gets six lightweight partitions; Instance B gets five lightweight partitions and one compute-intensive partition. The balancer reports equal throughput on both instances. But Instance B's CPU is at 80% while Instance A sits at 30%. The core problem is that throughput is a necessary but not sufficient signal for load balancing when workloads are heterogeneous. Messages that trigger expensive computation look identical to simple messages from a throughput counter's perspective, but they consume fundamentally different amounts of system resources. Heterogeneous Hardware Makes It Worse Even if all workloads had identical per-message processing costs, fleet hardware heterogeneity can still defeat throughput-based balancing. Whether you're running on-premise or in the cloud, your fleet may contain multiple generations and configurations of compute instances. One instance might have a modern high-core-count processor; another might be running on older hardware with fewer cores or different per-core performance characteristics. If the load balancer distributes an equal number of messages to both, the older machine hits 80% CPU while the newer one idles at 20%. The classic answer is weighted round-robin with weights proportional to compute instance capacity. But this only works when: You can accurately quantify relative compute instance capacity for your specific workloadThat capacity ratio remains stable across workload mixesThe mapping from capacity units to actual throughput is linear In streaming services with heterogeneous traffic, none of these assumptions reliably holds. A compute instance's effective capacity depends on which partitions are assigned to it, and that changes continuously as partitions are rebalanced. In production environments, throughput distribution often appears uniform while CPU utilization spans 2-3x across hardware generations. Uniform throughput does not mean uniform load. The Proxy Metric Trap When the direct signal — per-partition resource usage — isn't available, engineers naturally reach for proxy metrics. The most common ones are: Messages per second – assuming each message costs roughly the same to processBytes per second – assuming larger messages are more expensiveNumber of partitions – assuming each partition represents equal work Each of these breaks in predictable ways: Messages per second fails with heterogeneous workloadsBytes per second fails when cost comes from computation, not I/ONumber of partitions fails when partition throughput and complexity vary The temptation is to try increasingly sophisticated combinations of these proxies. I ran several experiments along these lines for my streaming use case, and the results were consistent: balancing on messages per second alone gets you part of the way but leaves a 2-3x spread in CPU utilization, and removing secondary metrics to focus on the single best proxy helps only marginally. The lesson I took from this: no amount of algebraic creativity with throughput-derived proxies can substitute for measuring the thing you actually care about. The Outlier Problem Load imbalance doesn't manifest as a smooth gradient. It concentrates on a small number of outlier instances, and those outliers are where the real damage happens. In a fleet of 100 instances, a typical distribution might look like this: The difference between the median and the worst case is 3×, but the number of truly overloaded instances might be just one or two. p50 CPU utilization: ~20%p95 CPU utilization: ~35%MAX CPU utilization: ~65%+ Those one or two instances are the ones that cause tail latency spikes, processing lag, OOM kills, and cascading failures. Your effective capacity is governed not by your average utilization, but by your worst-case instances. A fleet running at 20% average CPU sounds healthy until you realize the hottest instances are already at risk of overload. Partitioning Constraints Add Friction Data streaming services may have an additional constraint that stateless web services typically don't: partition affinity. Each partition represents a subset of the input data, and reassigning a partition to a different instance may involve state transfer, warm-up time, or temporary processing gaps. When this constraint applies, the load balancer can't freely shuffle work around the way a web load balancer directs HTTP requests. Rebalancing has a cost, and frequent rebalancing creates its own instability. The balancer must find the sweet spot between: Reacting quickly enough to prevent overload on individual instancesNot thrashing partitions so aggressively that the system never stabilizes What Actually Works: Measuring Real Resource Consumption After systematic experimentation with proxy metrics, the answer turned out to be straightforward in concept: measure actual per-partition resource consumption and use it as the primary load-balancing signal. In practice, this means three things: 1. Instrumenting the service to report per-partition resource usage — CPU time, memory, or a normalized compute metric: Python class PartitionCPUTracker: # Accumulated CPU time per partition (in seconds) _cpu_time: dict[str, float] = field(default_factory=lambda: defaultdict(float)) # Timestamp of last report _last_report_time: float = field(default_factory=time.monotonic) # Reporting interval (seconds) report_interval: float = 300 # 5 minutes def measure_partition_work(self, partition_id: str, process_fn, message): """Wrap message processing to measure CPU time per partition.""" cpu_start = time.process_time() try: result = process_fn(message) return result finally: cpu_elapsed = time.process_time() - cpu_start self._cpu_time[partition_id] += cpu_elapsed def get_cpu_usage_report(self) -> dict[str, float]: """Return per-partition CPU usage as a fraction of total capacity. Called periodically by the orchestrator to make placement decisions. """ now = time.monotonic() wall_elapsed = now - self._last_report_time if wall_elapsed <= 0: return {} num_cpus = os.cpu_count() or 1 total_capacity = wall_elapsed * num_cpus # max possible CPU-seconds report = {} for partition_id, cpu_seconds in self._cpu_time.items(): # Fraction of total CPU capacity consumed by this partition report[partition_id] = cpu_seconds / total_capacity # Reset counters for next interval self._cpu_time = defaultdict(float) self._last_report_time = now return report 2. Reporting these metrics to the orchestrator as the primary resource to balance: Python def report_to_orchestrator(self, orchestrator_client): """Send per-partition CPU usage to the orchestrator.""" report = self.get_cpu_usage_report() for partition_id, cpu_fraction in report.items(): orchestrator_client.report_resource( partition_id=partition_id, resource_type="partition_cpu_usage", utilization=cpu_fraction, ) 3. Letting the orchestrator make placement decisions based on actual resource consumption rather than throughput proxies: JSON # Load-balancer (orchestrator) config ... number_of_instances = 100 max_partitions_per_instance = 50 resources_to_balance = { # "partition_messages_per_second": {"deviation_pct": 10} # <-- old metric "partition_cpu_usage": {"deviation_pct": 5} # <-- new metric } ... When I implemented this approach, the results were significant: Load distribution became much more uniform. The difference between p10 and p95 CPU utilization tightened dramatically — from roughly 50% down to less than 10%, with p50 settling around 30%.Maximum fleet capacity increased. Because peak CPU utilization on the hottest instances dropped from ~65% to ~40%, the fleet could absorb substantially more traffic before any single instance became a bottleneck. The more uniform distribution also made capacity planning much easier to reason about. When load correlates with a directly measured resource, you can look at aggregate resource usage, compare it to fleet capacity, and make defensible decisions about provisioning. But the Numbers Weren't the Most Important Part: Implementation Considerations Measuring per-partition resource usage is harder than it sounds, and there are several practical challenges worth calling out. Attribution accuracy. In a multi-threaded service processing multiple partitions, correctly attributing resource consumption to individual partitions requires careful instrumentation. Approaches include per-partition timing around processing loops, or proportional attribution based on known cost proxies within the measurement framework.Hardware normalization. Resource consumption on different hardware must be normalized. The same workload will report different absolute numbers on different processor generations. Establishing a common unit of compute across your fleet is essential but non-trivial.Reporting granularity. The orchestrator needs per-partition resource reports at a granularity that captures steady-state behavior without being too noisy. In my particular case, reporting intervals of 5-10 minutes worked better than 1-minute intervals, which tended to be too reactive.Cold start. When a partition is first assigned to an instance, there's no resource usage history. The balancer must rely on throughput-based estimates until enough data accumulates. Key Takeaways If you're running a data streaming service with heterogeneous traffic, here's what I've found worth keeping in mind: Throughput-based load balancing fails silently when per-message processing costs vary across workloads. Round-robin distribution can result in 2-3x CPU spread between median and worst-case hosts.Measuring actual per-partition resource consumption is the only reliable load signal for heterogeneous streaming workloads. No combination of proxy metrics substitutes for direct measurement.Hardware heterogeneity compounds the proxy metric problem. Uniform throughput across a mixed hardware fleet does not mean uniform resource utilization.Load imbalance concentrates in a small number of outlier hosts. Capacity planning is governed by peak utilization, not average utilization.Per-partition instrumentation requires careful attention to attribution accuracy, hardware normalization, reporting granularity, and cold-start behavior. The challenges described here are common across any system that processes heterogeneous streaming data at scale — from real-time ML feature pipelines and search index builders to event-driven microservices and change data capture (CDC) processors. The specifics vary, but the fundamental tension between throughput-based proxies and actual resource consumption is universal.

(Note: A list of links for all articles in this series can be found at the conclusion of this article.) The Scalability Wall In previous posts of this COMPASS series, we demonstrated how OSCAL enables compliance-as-code from Catalogs through Component Definitions, to System Security Plans (Part 3), how Compliance Policy Administration Centers bridge compliance to policy enforcement (Parts 4–7), and how these patterns scale to complex environments (Part 9). Yet organizations still hit a fundamental bottleneck: the relentless proliferation of regulatory frameworks. Consider a financial services firm operating globally. They must simultaneously satisfy DORA in the EU, PCI DSS for payment processing, SOC 2 for SaaS customers, ISO 27001 for international contracts, NIST 800-53 for federal clients, and state-specific privacy regulations. Each framework brings hundreds of controls requiring documentation, implementation, testing, and evidence collection. The traditional approach treats each as an independent program with separate teams, spreadsheets, and evidence repositories. When a new regulation emerges — NIST AI RMF or ISO 42001 — the organization stands up yet another parallel program. This doesn’t scale. Operational overhead grows quadratically while security posture improves marginally. The OSCAL v1.2.1 Mapping Model, introduced in March 2026, provides the architectural solution. By enabling systematic, machine-readable mappings between control frameworks, OSCAL transforms multi-framework compliance from an O(N²) problem to O(N). The result: evidence reuse, automated gap analysis, and genuinely scalable continuous compliance. Why Spreadsheet Mappings Failed The compliance industry has always understood that frameworks overlap. Multi-factor authentication appears in NIST 800-53 as IA-2, PCI DSS as Requirement 8.3, ISO 27001 as A.9.4.2, and SOC 2 as CC6.1. Implementing MFA once should count toward all four frameworks. The challenge has been formalizing these relationships for compliance teams, auditors, and automation tooling. The predominant approach — manual crosswalk spreadsheets from consulting firms — suffers from fundamental flaws. They lack semantic precision. Does “maps to” mean controls are equivalent, or that one subsumes the other, or merely that they’re related? They exist as static documents disconnected from actual OSCAL artifacts. When frameworks update — PCI DSS v3.2.1 to v4.0, NIST 800-53 Rev 4 to Rev 5 — spreadsheets become instantly stale. Most critically, they’re human artifacts, not machine-readable structures that automation can reason over. OSCAL Mapping (see Figure 1) addresses each limitation. Mappings use formal set theory relationships: equal (identical requirements), subset (A’s requirements fully contained in B), superset (A broader than B), intersects (partial overlap requiring delta analysis), and not-applicable (no relationship, important for gap identification). Mappings are OSCAL documents following the same schema and validation rules as other artifacts. They version control alongside catalogs and SSPs, compose through standard reference mechanisms, and integrate into GitOps workflows. Mappings are bidirectional and composable, supporting Framework A to B, B to A, and transitive chains like EU AI Act to ISO 42001 to NIST 800-53. When mappings are machine-readable, tooling automatically identifies which controls in a new framework are satisfied by existing implementations, which require deltas, and which are entirely new obligations. When mapping version control alongside frameworks, updates trigger automated validation rather than silent staleness. Figure 1: Open Safety Controls Assessment Language Models, with red highlight on the newly released Mapping Model 4 Architectural Patterns OSCAL community experience across government agencies, enterprise IT, and emerging regulatory domains has converged on four patterns. Pattern 1: Version-to-Version Mapping Version-to-version mapping addresses regulation evolution within the same framework. When PCI DSS transitions from v3.2.1 to v4.0 or NIST 800-53 updates from Revision 4 to Revision 5, organizations face expensive manual analysis determining which existing implementations remain compliant and which need updates. Version mappings create explicit relationships between control versions using the same semantic types. For requirements with equal relationships — where fundamental obligations remain unchanged despite text clarifications — existing implementations automatically satisfy the new version with no work. For subset relationships — where the new version narrows the scope — implementations may now exceed requirements. For superset relationships — where a new version adds requirements — the mapping identifies exactly which delta implementations are needed. For intersects relationships — where a control splits or merges between versions — the mapping documents the partial overlap and flags affected implementations for review. NIST published official mappings from 800-53 Rev 4 to Rev 5, enabling organizations to query the OSCAL Mapping document and determine precisely which controls need reassessment. This automated impact analysis, traditionally requiring weeks of manual document comparison, now completes in hours and produces actionable gap reports showing new, modified, and unchanged requirements. Pattern 2: Direct Framework Mapping Direct mapping applies when an organization has established compliance with one framework and must demonstrate compliance with a second. A company with mature NIST 800-53 compliance that needs PCI DSS certification creates an OSCAL mapping artifact documenting relationships between NIST and PCI controls using semantic relationship types. For controls with equal or superset relationships — where NIST fully satisfies PCI — the mapping provides machine-readable evidence that existing implementations already meet new requirements. For intersects relationships — where NIST partially addresses PCI — the mapping identifies exactly which delta requirements need attention. For PCI requirements with no NIST mapping, analysis immediately surfaces genuinely new obligations. This pattern typically yields 40-60% coverage through equal or superset relationships. The remaining 40-60% splits between partial coverage requiring delta implementations and no coverage requiring new implementations. Implementation time drops from 12-18 months for greenfield programs to 4-6 months for delta implementation. Pattern 3: Enterprise Baseline Mapping Enterprise baseline mapping addresses organizations maintaining proprietary internal control frameworks that must map multiple external regulations to that baseline. IBM’s IT Security Standard (ITSS), Google’s Control Framework, and similar enterprise-specific sets represent distillations of institutional practices refined over decades. External regulations map to the baseline, not to technical implementations. Technical implementations documented as Component Definitions map to baseline controls. Assessment results aggregate to the baseline. When auditors request framework-specific evidence, the organization computes it through the two-hop relationship: external framework to baseline to implementation evidence. Adding a new framework requires mapping it to the baseline — an O(N) operation. Critically, it doesn’t require touching Component Definitions describing technical implementations or evidence collection infrastructure. The organization escapes O(N²) complexity, where each new framework forces a review of all existing mappings. The baseline provides stability and abstraction that direct external-framework-to-implementation mappings cannot offer. Pattern 4: Harmonized Framework Construction Harmonized framework approach addresses organizations subject to multiple regulations simultaneously, where maintaining separate programs creates unacceptable overhead. The organization constructs a single unified catalog representing the superset of all applicable requirements. All Component Definitions implement controls from this harmonized catalog. Framework-specific compliance reports are computed by mapping the harmonized catalog back to each source regulation. Construction begins with a foundational framework providing broad coverage, typically NIST 800-53. The organization maps the second framework to NIST, identifying requirements already covered, those requiring catalog extensions for deltas, and those representing entirely new obligations. The process repeats for each additional framework, mapping to the growing harmonized catalog. Component Definitions implement harmonized controls once, automatically satisfying multiple frameworks simultaneously through mapping relationships. A single implementation for multi-factor authentication satisfies the harmonized control, which, through mappings, simultaneously satisfies NIST IA-2, PCI 8.3, ISO A.9.4.2, and any other mapped requirement. Real-World Impact IBM’s internal IT compliance manages compliance across more than 40 frameworks globally. The shift to OSCAL-based compliance with formal mappings uses ITSS as the enterprise baseline (Pattern 3). When the EU’s Digital Operational Resilience Act introduced new requirements, automated gap analysis through DORA-to-ITSS mapping identified that 65% of DORA requirements have equal or superset relationships to existing ITSS controls, existing implementations fully satisfy these with no new work. Another 25% have intersects relationships requiring delta implementations. The remaining 10% represent genuinely new obligations. This gap analysis, requiring months of manual review pre-OSCAL, now completes in hours. IBM reduced average time-to-compliance from 12-18 months to 4-6 months, achieved 70% reduction in duplicate documentation, and cut ongoing assessment effort by 3x through evidence reuse. Integration With Continuous Compliance OSCAL mappings integrate seamlessly into the compliance-to-policy workflows detailed in earlier parts of this series. Without mappings, separate Component Definitions would document pod security policies against NIST controls and again against PCI controls, duplicating validation logic. With mappings, a harmonized catalog contains a control for pod security policies with documented mappings to both NIST 800-53 SC-7 and PCI DSS Requirement 2.2. A single Component Definition implements this harmonized control by referencing OPA policies validating pod security. When OPA validation runs and collects evidence, that evidence is recorded against the harmonized control. Assessment result computation applies the mappings: evidence satisfying the harmonized control automatically counts toward both NIST SC-7 and PCI 2.2. Validation logic is written once, maintained once, executed once, and produces evidence once — but that single evidence artifact satisfies multiple regulatory obligations simultaneously through the mapping layer. The mapping layer enables unified compliance posture dashboards showing which technical implementations satisfy which framework requirements through which mapping relationships. The Path Forward A critical challenge is mapping quality and consensus. The relationship between controls can be interpreted differently by different domain experts. The OSCAL Foundation addresses this through collaborative mapping development in public repositories. Mappings are authored in Git using Trestle-based workflows. Pull requests propose mappings or modifications. Domain experts from multiple organizations review and discuss each mapping, documenting disagreements and consensus rationale. Approved mappings merge into canonical repositories with clear provenance and confidence scores. NIST has published official mappings between NIST 800-53 Revision 5 and NIST Cybersecurity Framework 2.0. FedRAMP is developing mappings from FedRAMP baselines to broader NIST 800-53 controls. The OSCAL Foundation community is creating mappings between major frameworks like ISO 27001, PCI DSS, SOC 2, and NIST 800-53. These mappings version control with clear change tracking, update when frameworks evolve, and accumulate community feedback, improving accuracy. The next frontier involves extending mappings to emerging regulatory domains. The AI safety landscape — NIST AI RMF, EU AI Act, ISO 42001 — represents a new compliance frontier where mapping infrastructure can prevent fragmentation. By establishing OSCAL mappings between AI regulations from the outset, the community can enable organizations to implement AI safety controls once and automatically satisfy multiple frameworks through mapping relationships. The next article in this series will demonstrate how these layers combine to address AI safety and governance, showing how Component Definitions for AI technical stack components (KServe, LangChain, MLflow) map to AI-specific regulations (NIST AI RMF, EU AI Act, ISO 42001) through OSCAL mapping mechanisms, enabling continuous compliance at scale for AI systems. References OSCAL Mapping Resources: OSCAL v1.2.1 MappingOSCAL Mapping Best Practices White Paper: Community review at LF AI & Data Security & Compliance WG OSCAL Foundation Community Presentations: NIST OSCAL Workshop (July 2025): “Collaboratively Maturing the OSCAL Control Mapping Model”SunStone OSCAL PlugFest (May 2025): “OSCAL Mapping Scenarios and Process” Tools: Compliance-TrestleNIST OSCAL The authors welcome collaboration through the OSCAL Foundation and LF AI & Data Security & Compliance Working Group. Below are the links to other articles in this series: Compliance Automated Standard Solution (COMPASS), Part 1: Personas and RolesCompliance Automated Standard Solution (COMPASS), Part 2: Trestle SDKCompliance Automated Standard Solution (COMPASS), Part 3: Artifacts and PersonasCompliance Automated Standard Solution (COMPASS), Part 4: Topologies of Compliance Policy Administration CentersCompliance Automated Standard Solution (COMPASS), Part 5: A Lack of Network Boundaries Invites a Lack of ComplianceCompliance Automated Standard Solution (COMPASS), Part 6: Compliance to Policy for Multiple Kubernetes ClustersCompliance Automated Standard Solution (COMPASS), Part 7: Compliance-to-Policy for IT Operation Policies Using AuditreeCompliance Automated Standard Solution (COMPASS), Part 8: Agentic AI Policy as Code for Compliance Automation With Prompt Declaration LanguageCompliance Automated Standard Solution (COMPASS), Part 9: Taking OSCAL-Compass to Industry Complexity Level

The Pipeline Did Not Fail Cleanly Most pipeline failures don't look like "the job failed." Consider a common scenario. A Glue job reads overnight event files, applies business rules, and writes to an Iceberg curated table. The job runs at its scheduled time and errors out partway through. The control table shows SUCCESS for the previous batch and FAILED for the current one, which is what you'd expect. The problem is what happened between those two states: the job wrote nine of the day's twelve partitions to the staging table before failing. A downstream report ran on its own schedule, picked up the partial data, and the discrepancy didn't surface until a downstream consumer noticed records were missing. By the time someone looks at the failure, the question is no longer "Why did the job fail?" It's "Is it safe to rerun, and what's already corrupted downstream?" That's where debugging gets messy. CloudWatch logs, Glue run metadata, the source S3 path, record counts, data quality results, target table state, and Iceberg snapshots. An experienced engineer can connect those signals, but it takes time, and a less experienced engineer often misses one. In a busy production environment that delay leads to blind reruns, duplicate records, overwritten partitions, or worse. The frustrating part is that the evidence existed. The pipeline just had no structured way to explain itself. That's the gap a triage layer can fill. Not by fixing the pipeline. Not by changing schemas. Not by restarting jobs. By observing the evidence already produced, classifying the failure, explaining what likely happened, and recommending what to do next. What Agentic Observability Means The word "agentic" gets misused a lot right now, especially in data engineering. It's worth being precise. An agentic observability layer is not an LLM with permission to control production. It's a controlled workflow that collects pipeline evidence, builds incident context, classifies the failure against known categories, and produces a structured recommendation. The loop is observe, classify, explain, recommend, and that's where it stops. Everything past "recommend" stays with engineers, deterministic rules, or approval workflows. The difference from normal alerting is the depth of the output. A normal alert says "Glue job daily_customer_interactions failed." An agentic observability layer should produce something closer to: "The job failed because the input contains a new column not present in the curated schema. The staging write started before the failure, so a blind retry will create duplicate records. Quarantine the batch, review the schema contract, and rerun with the same batch_id after validation." That difference is what saves time during an incident. The goal isn't replacing engineers. It's reducing the manual triage work needed before someone can make a real decision. Reference Architecture This does not need to start as a new platform. The triage layer can sit beside existing Glue pipelines and consume signals that already exist. Figure 1. Agentic observability flow for AWS Glue pipelines. Pipeline evidence is collected, converted into structured context, analyzed by an LLM triage layer, and returned as a structured incident output. The component that matters most here is the incident context builder. The LLM should never receive a raw dump of ten thousand log lines. That produces noisy, low-confidence output and burns tokens. The collector should pull a curated set of signals: Glue job name and run ID, status and duration, batch ID, source path, target table, the last fifty error log lines, data quality results, record counts, attempt count, recent deployment version, table snapshot or commit ID, and control table status. That's enough context to analyze the failure without guessing from disconnected log lines. Where This Fits Before going further, one thing worth being honest about: this pattern depends on the platform already having its house in order. The agent can only work with the observability that the platform already has. It is not a substitute for basic pipeline hygiene. It works when the platform tracks batch IDs, clear source paths, data quality results, structured logs, table commits, deployment versions, and ownership mapping. Without those signals, the agent has very little to reason over. If a pipeline doesn't track batch IDs, the agent can't reliably tell whether a run is a retry or a new batch. If quality results aren't stored, it can't reason about input validity. If table commits aren't tracked, it can't tell whether the failure happened before or after a write. LLMs don't create observability. They summarize and reason over the observability that already exists. The teams that get the most out of this pattern are the ones with disciplined data engineering underneath. Failure Categories Manual debugging takes time, partly because every failure looks unique at first glance. Most don't stay unique once you classify them. A small fixed set of categories makes the output easier to review, compare, and route. Failure categoryCommon signalsRecommended actionSchema driftNew column, missing column, cast failure, contract mismatchQuarantine the batch and review the schema contractData skewLong-running tasks, shuffle spill, uneven partitionsRepartition or isolate skewed keysSmall file pressureHigh file count, slow planning, frequent commitsCompact affected partitionsSource delayMissing input path, low record count, late file arrivalWait, retry later, or mark the batch delayedCode regressionRecent deployment plus transformation errorRoll back or compare with the previous runPermission issueAccess denied, catalog failure, IAM or Lake Formation errorFix access policy before retryingPartial write riskFailure after write startedCheck staging and control tables before rerunUnknownWeak or conflicting evidenceEscalate to an engineer with summarized context The category list isn't only documentation. It's part of the system contract. The agent picks from this list rather than inventing categories on each run, which makes downstream routing tractable. Schema drift can go to the data contract owner. Permission issues route to the platform team. Source delays go to the ingestion owner. Partial write risk triggers a manual review workflow rather than auto-retry. This is what makes the system more useful than a chatbot that summarizes logs. Structured Incident Output The output should also be structured. Free-form summaries help humans skim, but they're hard to store, compare, or evaluate over time. JSON works better because it can be written to an incident table and consumed by Slack, Teams, Jira, or ServiceNow without parsing prose. JSON { "pipeline_name": "daily_customer_interactions", "job_run_id": "jr_2026_05_02_001", "status": "FAILED", "failure_category": "SCHEMA_DRIFT", "likely_root_cause": "Input file contains a new column named device_type that is not defined in the curated table schema.", "affected_source_path": "s3://raw/events/date=2026-05-02/", "affected_table": "curated.customer_interactions", "safe_to_retry": false, "recommended_action": "Quarantine the batch, update the schema contract, and rerun with the same batch_id after validation.", "confidence": 0.87 } A structured output gives engineers a quick summary, and it gives downstream tools something reliable to use. If safe_to_retry is false, the orchestrator blocks automatic retry. If failure_category is PERMISSION_ERROR, the issue routes to the platform queue. If confidence is low, the system asks for human review. If the same failure category recurs across runs, dashboards can track it over time. One important framing point: the LLM is not the system of record. The control table, logs, table metadata, and quality checks remain the source of truth. The agent is a reasoning layer that produces structured evidence on top of that. Implementation Sketch A simple implementation starts with assembling the incident context. The example below is intentionally simplified. In production, the LLM call should use structured outputs or schema-validated responses rather than free-form text parsing. Python def build_incident_context(job_run, control_record, dq_results, recent_logs): return { "job_name": job_run["JobName"], "job_run_id": job_run["Id"], "status": job_run["JobRunState"], "started_on": str(job_run["StartedOn"]), "completed_on": str(job_run.get("CompletedOn")), "batch_id": control_record.get("batch_id"), "source_path": control_record.get("source_path"), "target_table": control_record.get("target_table"), "attempt_count": control_record.get("attempt_count"), "control_status": control_record.get("status"), "data_quality_results": dq_results, "recent_error_logs": recent_logs[-50:] } The classifier receives a fixed category list and explicit rules about what it shouldn't recommend. Python def classify_failure(llm_client, incident_context): prompt = f""" You are analyzing a failed data pipeline run. Classify the failure into one of these categories: SCHEMA_DRIFT, DATA_SKEW, SOURCE_DELAY, PERMISSION_ERROR, CODE_REGRESSION, PARTIAL_WRITE_RISK, SMALL_FILE_PRESSURE, UNKNOWN. Return only valid JSON with: failure_category, likely_root_cause, safe_to_retry, recommended_action, confidence. Rules: - Do not recommend a retry if there is partial write risk. - Do not recommend schema changes without human review. - Do not recommend permission changes without platform approval. - Use UNKNOWN when evidence is weak or conflicting. Incident context: {incident_context} """ return llm_client.invoke(prompt) In a real implementation, this prompt should be paired with a strict response schema (failure_category as an enum, likely_root_cause as a string, safe_to_retry as a boolean, recommended_action as a string, confidence as a float between 0 and 1), and the system should reject any output that doesn't match. In production, structured outputs are the better choice when the API supports them. The free-form prompt above is illustrative. The result gets stored, not acted on: Python def store_incident_summary(summary, incident_table): incident_table.put_item( Item={ "pipeline_name": summary["pipeline_name"], "job_run_id": summary["job_run_id"], "failure_category": summary["failure_category"], "safe_to_retry": summary["safe_to_retry"], "recommended_action": summary["recommended_action"], "confidence": summary["confidence"], "created_at": current_timestamp() } ) The agent writes an explanation. Other systems decide what to do with it. What the Agent Should Never Decide This boundary is the most important design choice in the whole pattern, and it's worth being explicit about. An observability agent helps engineers understand a failure. It does not control production data systems. Even at high confidence, certain actions stay out of scope: Changing table schemasGranting IAM or Lake Formation permissionsDeleting dataMarking a partially written batch as successfulOverriding data quality failuresPromoting quarantined dataRewriting production tablesTriggering cross-pipeline backfillsCompacting or expiring table snapshots without approval These actions move from observability into production control, and that line should stay clear. In regulated or business-critical environments, the safest design lets the agent produce structured evidence and recommendations while deterministic rules, approval workflows, or engineers decide whether anything actually executes. An agent saying "this looks like schema drift, the batch is not safe to retry" is useful. The same agent updating the curated table schema on its own is not. It's a future incident waiting to happen. Same with permissions: the agent flagging an IAM issue is useful; the agent granting itself access is a security violation. The trade-off here is real. Letting the agent take action would reduce the mean time to recovery. But the cost of a confident wrong action (silently corrupted data, an unauthorized permission grant, a dropped partition) is much higher than the cost of a few extra minutes of human review. In a regulated data environment, that trade-off is usually easy to justify. This matters as teams move toward self-healing pipelines. Before a pipeline can safely fix itself, it has to first explain itself reliably, at scale, with measurable accuracy. That bar isn't met yet in most production environments. Evaluating the Triage Layer A triage layer should be evaluated like any other production component. "The summary looks good" is not an evaluation. To check whether the pattern behaves reasonably, a small synthetic evaluation can be assembled across common Glue failure modes. Each scenario includes a short set of log lines, control-table state, data quality results, and table metadata, and the agent is scored on two things: whether it picks the correct failure category, and whether the safe_to_retry decision is appropriate. This is a starter evaluation, not a benchmark. Ten synthetic scenarios are enough to sanity-check the design. A real production rollout needs hundreds of labeled historical incidents, edge cases, and human-reviewed outcomes. Anything less should be treated as an early prototype, not production validation. ScenarioExpected categoryAgent categorySafe-to-retry decisionMissing source pathSOURCE_DELAYSOURCE_DELAYCorrectNew column in inputSCHEMA_DRIFTSCHEMA_DRIFTCorrectAccess denied on catalog tablePERMISSION_ERRORPERMISSION_ERRORCorrectShuffle spill and one long taskDATA_SKEWDATA_SKEWCorrectFailure after staging writePARTIAL_WRITE_RISKPARTIAL_WRITE_RISKCorrectToo many small filesSMALL_FILE_PRESSURESMALL_FILE_PRESSURECorrectRecent code deployment plus null pointerCODE_REGRESSIONCODE_REGRESSIONCorrectLow record count, no hard errorSOURCE_DELAYUNKNOWNConservative escalationCast failure due to bad input valueSCHEMA_DRIFTSCHEMA_DRIFTWrong, recommended retryConflicting log signalsUNKNOWNUNKNOWNCorrect escalation In a small evaluation like this one, a well-designed classifier should pick the expected category in most scenarios and, more importantly, get the safe-to-retry decision right in nearly all of them. The illustrative results above show eight correct retry decisions, one conservative escalation (the agent returns UNKNOWN rather than guessing), and one wrong call. That wrong call is the most instructive. On the cast failure, the agent classifies the issue correctly as schema drift but recommends cleanup-and-retry instead of quarantine-and-contract-review. A wrong root cause is inconvenient. A wrong retry recommendation can corrupt data. Safe-retry precision should be weighted higher than classification accuracy when evaluating this kind of system, and that weighting should be reflected in the prompt rules and in the validation rubric. The metrics worth tracking in production: MetricWhy it mattersClassification accuracyWhether the agent identifies the right failure typeSafe-retry precisionWhether retry recommendations are actually safeFalse confidence rateConfident-but-wrong recommendationsMean triage timeReduction in manual debugging timeHuman override rateHow often engineers reject the recommendationCost per incidentLLM and log-processing cost per failed run False confidence rate deserves attention. A low-confidence wrong answer is manageable because engineers know to scrutinize it. A high-confidence wrong answer is dangerous because teams stop scrutinizing. Confidence belongs in the output, but it should never be treated as truth. It's one signal among several in the routing decision. Closing Glue job failures aren't hard because the logs are long. They're hard because the evidence is scattered across logs, run metadata, data quality results, control tables, and table commits, and an engineer has to assemble it before deciding what to do next. An agentic observability layer turns that scattered evidence into a structured incident summary. The safest version of this pattern is controlled triage, not autonomous repair: observe, classify, explain, recommend, and stop there. Deterministic rules, approval workflows, and engineers decide what happens next. Before pipelines can fix themselves, they need to explain themselves. That's the work worth doing first.

Intro: When Good Models Go Wrong A few years ago, I spent months working on a microservices-based customer intake processing system for our application. The code was good, the tests were passing, and we had load-tested it with crazy high TPS. Yet, on one particular Tuesday afternoon, a small change to the response schema from an upstream service, where the date field changed from ISO 8601 to epoch milliseconds, cascaded through four downstream services and corrupted a day’s transactions without anyone realizing it until it was too late. We fixed it in a few hours, but the lesson has stayed with me, and it’s affected every integration I’ve worked on since then. Crashes are easy to see. Silent data corruption is not. I see the exact same thing happening with AI and machine learning pipelines today. Except now, the consequences are larger, and the feedback cycles are slower. A model will not throw an exception if the input schema changes slightly. It will, however, make worse predictions. Quietly. Confidently. For weeks. In this article, I’d like to propose a solution that brings together two worlds in which I’ve spent my entire professional life: software engineering’s resilience patterns and data governance. What I’d like to argue is the need to combine the concept of data contracts with the Circuit Breaker pattern to build a proactive defense against silent data quality failures that affect AI reliability. The Real Reason AI Models Fail in Production There’s a general understanding that if a model is not performing well, it’s a problem with the model itself, its architecture, its hyperparameters, the training process, etc. Sometimes this is true, but far more solvable. The upstream data changed. Nobody told the model. “Poor data quality is a silent killer of any AI projects.” This is consistent with many production environments. The data engineers did their job, the modelers did their job, and nobody owned the contract between the two. This might manifest in a number of ways: Schema drift: A column that has always been a float type now starts arriving as a string type. The feature engineering process, quietly and behind the scenes, attempts to convert it and introduces some error in the process.Semantic drift: An attribute named "account_status" that previously only had values such as "ACTIVE", "CLOSED", and "DELINQUENT" now starts having a new value, "UNDER_REVIEW", that the model has never seen before. The model maps it to the category that it is closest to in the embedding space, which could be completely incorrect.Distribution shift: The data source the model uses changes how it samples data or alters other aspects of the data. The model is now seeing a different data distribution than it was previously trained on, and the schema looks exactly the same. So, nothing appears to have changed.Cadence changes: A data source that is a batch source and has historically refreshed data every day now starts refreshing data every hour, or vice versa. A 2023 Gartner study found that the primary cause of AI project failure is poor data quality, and more than 60% of organizations reported that data issues, rather than model issues, were the primary cause of most of their production incidents. The diagram below shows the silent changes in the data and how they propagate through the machine learning pipeline. Data changes upstream will flow through the pipeline without error, producing confidently wrong model outputs that will go undetected for weeks. The fundamental issue here is that there is no contract between data producers and data consumers. In a microservices world, we solved this problem a decade ago through API contracts and schema registries. In a data world — and this might sting a little — we're still operating on trust and hope. This is a data governance and data quality issue. And the good news is that the data management community has a conceptual toolkit to solve this problem. We just need to integrate it into the AI pipeline. What Are Data Contracts? Most teams believe they have data contracts. In reality, they have documentation with good intentions. A wiki page with "this field is supposed to be a float" or a Slack channel where someone will ask, "Hey, did the schema change here?" A data contract is not documentation. A data contract is enforceable, but documentation is not. A real data contract is an enforceable agreement between two parties: the producer (the system or team that produces the data) and the consumer (the system or team that consumes the data). A real data contract is an agreement that includes: Schema: The exact structure, data types, and allowed values for every single field.Semantics: What each field means, including business definitions and edge cases.Quality: Minimum quality thresholds for completeness, freshness, accuracy, and uniqueness.SLAs: Service level agreements for delivery cadence, latency, and availability. Versioning: A definition for schema changes, including deprecation schedules for backward-incompatible changes. It’s like thinking of it as a data API specification. Just as OpenAPI (Swagger) has standardized how we specify a REST API, data contracts have standardized how we specify a data interface. It’s a concept that’s been getting a lot of traction among the DataOps community. Andrew Jones has been a prominent influencer in formalizing data contract specifications, and tools like Soda and Great Expectations provide frameworks for data quality expectations, which are part of a data contract. The importance of AI is unparalleled, as every ML model relies on a set of data assumptions that are not only unspecified but also unenforced. When those assumptions are violated, the model starts to deteriorate. A data contract spells out those assumptions, making it testable and enforceable — bringing the level of rigor that data stewardship teams have been advocating for, into the ML pipeline. The Circuit Breaker Pattern: A Primer You already know what a circuit breaker is; there is one in your house. It works by tripping and shutting off the electricity if the load gets too high. You simply flip it back on to restore service. Simple, elegant, and has saved many houses from burning to the ground. The concept of circuit breakers has been around for a long time in software development, popularized by Michael Nygard in his book “Release It!” It has been a standard pattern for building resilient distributed systems. I have been using this concept for a long time. We use Spring Cloud Circuit Breaker based on Resilience4j to handle circuit breakers for our microservices-based application to prevent cascading failures in downstream services, which are very critical to business. The circuit breaker works as follows: Closed state – this is the normal operating state. All requests go through to the downstream service. The circuit breaker is monitoring the failure rate.Open state – this is where the circuit breaker has detected a failure rate above a certain threshold. It has “tripped” and will stop sending requests to the downstream service. Instead, it will immediately send a fallback response or error.Half-open state [recovery probe] – after a cooldown period, the breaker allows a limited number of test requests to pass. If they are successful, the breaker closes; otherwise, it stays in the open position. State machine for circuit breaker here^; the circuit breaker changes states based on failure rates and recovery probes. This pattern has become accessible to every Java developer with the introduction of frameworks such as Spring Cloud Circuit Breaker and Netflix Hystrix. The pattern is simple but very useful. It’s all about failing fast. We have been using this pattern for service-to-service communication for more than a decade. We have 100s of our services with a circuit breaker pattern implemented on our platform. If our XXX critical service goes down, we simply trip the circuit breaker and fail gracefully. But if our upstream data source changes schema silently and starts corrupting our ML features? Nothing. No circuit breaker. No fallback. Just a degradation of our features for weeks. The failure mode is the same: a degraded upstream service silently corrupts a downstream service. But we didn’t have a similar pattern implemented for our data pipelines until we did. Applying the Circuit Breaker to Data Pipelines The basic idea is not as complex as it sounds: we propose that every data input to an AI model is a dependency that can cause a circuit breaker to trip. If we do this with HTTP calls to other microservices, we can do this with data going into a model. While a traditional microservice circuit breaker monitors HTTP request error rate and latency, a data circuit breaker monitors data quality metrics defined in the data contract: Circuit Breaker State Trigger Condition Action Closed (healthy) All contract quality thresholds met Data flows normally into the model pipeline Open (tripped) Quality metrics breach contract thresholds (e.g., null rate > 5%, freshness > 2 hours stale, schema mismatch detected) Data flow is halted; model receives no new input; fallback strategy activates Half-Open (probing) After cooldown, a sample batch is validated against the contract If the sample passes, the breaker closes; if it fails, the breaker stays open The fallback options when the breaker trips can be: Stale but safe – using the last known good data snapshot. The model will continue to run, just on slightly outdated, but still good, data.Graceful degradation – the model will continue to run, but flag its output as "low confidence" and send it to a human for review.Full halt – for high-stakes applications like fraud detection or compliance, the model will simply stop running until the data quality is resolved. This is a fundamental shift from "we'll detect the problem when it happens and send an alert" to "we'll prevent the problem from happening in the first place." Architecture: Data Contracts + Circuit Breakers in Practice Let me walk through a concrete data architecture that ties these patterns together. This is heavily inspired by how we operate this on our lending platform, but adapted for the data to model case: The Data Contract Registry A centralized service responsible for storing all active data contracts. Each data contract is versioned and associated with a data source and a consumer. The service provides APIs for: Registering a data contractValidating data against a data contractPublishing a data contract violation event The Quality Gate A lightweight service (or a 'sidecar' pattern, if you will) that sits in between the data source and the model pipeline. For every data batch or stream event received, the quality gate: Fetches the relevant data contract from the registryValidates data against schema, semantics, and quality rulesReports metrics to the circuit breaker The Circuit Breaker Controller A stateful component that: Aggregates quality metrics from the quality gate over a specified window sizeManages the breaker state (closed, open, half-open)Publishes state change events to a Kafka topic for downstream consumptionExecutes fallback strategies when the breaker is opened The Flow The architecture is an end-to-end solution that includes data contracts, quality gates, and circuit breakers. The circuit breaker is located between the quality gates and the model pipeline, automatically routing to fallbacks if the quality of the data worsens. If you are using AWS, which we are, then this architecture fits nicely with existing AWS services. For example, the quality gate can be performed by a Lambda function or ECS task, the contract registry can be on DynamoDB or other AWS-native datastores, the circuit breaker state can be maintained by ElastiCache (Redis), and the event bus can be on Kafka (or MSK, the AWS variant). We already make significant use of all these tools for our financial platform microservices, so the marginal cost for using them with the data pipeline is negligible. If you are using Kubernetes, then the quality gate can also function nicely as a sidecar container to your model serving pods. The key architectural concept is the separation of concerns. The data producer is responsible for the data contract, the quality gate is responsible for the quality, and the circuit breaker is responsible for the fail-fast. There is no need for a single team to “own” the entire process. From Chaos Engineering to Data Resilience The last time I intentionally broke my data pipeline and saw what happened was? On our system, we do disaster recovery drills regularly — an orchestrated set of exercises on 100+ components, including APIs, batch jobs, and streaming apps. The team is very good at infrastructure chaos engineering. However, when I asked, “What happens if the credit bureau feed starts sending garbage schema for two hours?” nobody answered because nobody had ever really tested this scenario. Most organizations practice chaos engineering on infrastructure, but very few practice data chaos engineering — intentionally introducing data quality errors to see if their systems correctly detect and respond to those errors. Data Chaos Engineering in Practice Schema injection: Apply a schema modification temporarily, for example, by adding a column or changing a data type. Validate that the quality gate detects this modification and the circuit breaker is triggered.Null injection: Increase the proportion of null values for a critical feature beyond the contract value. Validate that the breaker is triggered.Staleness simulation: Apply a delay in the data delivery beyond the SLA value. Validate that the staleness check is triggered.Distribution poisoning: Apply a small perturbation to the distribution of a critical feature. Validate the detection. The data chaos engineering cycle. Here, faults are injected to ensure that the contracts and breakers are functioning correctly. The missing pieces are fed back into the contract and breaker development. I have seen that by running these experiments every month, taking the same level of discipline that we already take in running our existing DR drills for our services, instills enormous confidence in the system's ability to look after itself. It also reveals missing pieces in your data contracts that you might never find by just reviewing your documentation. If you introduce a fault and nothing catches it, that means your contract is incomplete. We learned that we had three missing contract clauses just by running data chaos experiments for the first month. The principles of chaos engineering are applicable in this case. You are not testing if your system works under perfect conditions; you are testing if your system fails safely under realistic, degraded conditions. Real-World Scenario: Stopping a Bad Prediction Before It Ships For example, a financial services company might use ML models to predict customer behavior for risk analysis. The ML model might use various data sources as features, such as an external third-party data provider for customer risk indicators. The scenario: A third-party vendor changes their API and doesn't notify anyone. A critical field in the data set now returns numeric data instead of categories. The field previously returned HIGH_RISK, MEDIUM_RISK, LOW_RISK, and MINIMAL_RISK categories, but now it returns numeric data between 1 and 100. The ETL process doesn't fail but defaults to a mapping of the data, which essentially flattens all the risk into a single category across all customers. Without a data contract and circuit breaker: The model runs for weeks with corrupted features. Predictions are no longer accurate, but the gradual change is mistaken for market conditions or seasonality. By the time the actual cause is determined, thousands of decisions are made based on incorrect predictions. The process to address the problem involves several teams working in war rooms over the course of days, analyzing logs and assessing the damage, a considerable engineering and possibly business waste. With a data contract and circuit breaker: The data contract is very specific in that it requires the risk indicator field to contain one of four string values. If the vendor changes the format of the API, the quality gate immediately recognizes that the data is not passing the semantic validation. The circuit breaker is triggered within minutes. The system defaults to the last verified snapshot of the data and flags all predictions as "Degraded Confidence." An alert is sent to the data engineering team. The schema is fixed within hours, and zero corrupted predictions are ever made. The speed is a secondary benefit, the actual value is in the prevention of damage (as a preventative control rather than a detective). The circuit breaker prevented the bad data from entering the model before the corrupted prediction was ever made. FAQs What is the difference between a data contract and a schema registry, e.g., Confluent Schema Registry? A schema registry will verify structure, e.g., field names, data types, and nesting. A data contract extends that with semantic rules, e.g., allowed values, definitions, quality rules, e.g., nulls, freshness, and SLAs, e.g., delivery cadence, availability. In other words, the schema registry is just part of the data contract. Won't triggering circuit breakers cause the model to stop working too often? This is not a fundamental flaw; it's just a calibration issue. People often underestimate the amount of variation that is normal in their data. We did. Start with large values, then adjust them once you know your data's normal behavior. The half-open state helps with recovery. In practice, circuit breakers will not often fail, and when they do, it's likely due to real issues. Does this apply to real-time streaming data, or is it limited to batch data? Both. For streaming, the quality gate checks every event or micro-batch. The circuit breaker aggregates metrics over a time window. For batch, the quality gate checks at the batch level, prior to writing to the feature store. This pattern is unaware of the delivery mechanism. What about unstructured data, like text and images? For unstructured data, like text and images, the data contracts are concerned with other quality aspects, like encoding, language, document size, and metadata. The Circuit Breaker still applies, just to other metrics. For example, in an image processing pipeline, if 90% of the images received are 90% smaller than the average, it could be a sign of corrupted images or thumbnail images only. How do I get data producers to adopt contracts? Start with the highest value, highest risk data sources. Present it in the context of reducing their support load. The producer team is interrupted every time a consumer reports a bug because of the change in the data. I have been in enough cross-team incident reviews to know that these interruptions are not popular. Contracts remove the need for these interruptions. Once one producing team has adopted contracts and seen the reduction in downstream incidents, the rest tend to spread naturally. We began with a data feed and now have contracts in place for our most critical internal data sources. Conclusion The data engineering community has spent years developing ever-more sophisticated monitoring, alerting, and observability tools. That's all been good work. But let's be honest: monitoring is fundamentally reactive. Monitoring just lets you know something's gone wrong... after the damage is done. You want monitoring and prevention, but only prevention will stop the damage before it happens. Data contracts and circuit breakers are a fundamental shift in data resiliency: Contracts make the expectations explicit. Circuit breakers make those expectations active, in real time, before the bad data ever gets to the models and agents that rely on it. When building AI systems that make critical decisions... and increasingly, all of us are doing this... You simply cannot operate on implicit trust between data producers and data consumers. The chasm between "the data exists" and "the data is fit for purpose" is where model reliability goes to die. The data governance and data quality practices that this community has advocated for over the years are precisely what you need. And now, taking them to the AI layer is what's next. Bridge the gap. Write the contract. Wire the breaker. Start with one data source, the one that has burned you before. You know the one. Your models will thank you. Key Takeaways The cause of AI system failure is data, not code. The most common cause of production AI system failure is a change in data schema or semantics, which degrades model predictions silently.Data contracts make data producer and consumer expectations around schema, semantics, data quality thresholds, and SLAs explicit, making implicit assumptions explicit and testable.The Circuit Breaker pattern stops bad data from being fed to a model by automatically stopping data flow when data quality thresholds are violated, allowing for fallbacks to be implemented.Data chaos engineering makes you confident that your data contracts and circuit breakers will work when your data quality actually fails by intentionally inducing data quality failures.Target high-value, high-risk data sources first. Success in one area can generate enough organizational momentum for wider application.