DZone Spotlight

What Makes a Good Build Server? In modern cloud-native application development, Continuous Integration, with automated building and testing of software on every commit, has become a standard best practice. This typically involves maintaining a farm of build nodes, which can be physical devices, virtual machines, or containers, that can be provisioned on demand and retired once build tasks are completed. This guide aims to help you configure the ultimate build server for Ampere's Arm-based architecture. We will explore various configuration options (or “knobs and switches”) to optimize a Linux build server’s performance, detailing the performance improvement with each adjustment. This tuning guide will focus on building LLVM-MinGW, a toolchain for creating Windows binaries using the LLVM project. This will be conducted on a Fedora 40 server running on an Ampere® Altra® 128-core server and compared with results from AmpereOne® 192-core servers, highlighting the advancements in that CPU. What Build Server Workloads Look Like Running software builds and testing workloads on a modern build server are inherently dynamic. They typically feature short bursts of CPU-intensive tasks, interspersed with other tasks that aren't as CPU-intensive. Figure 1 illustrates this CPU utilization behavior over time, which we will discuss later. While many build tasks, such as compiling source files, can execute in parallel across numerous CPU cores, other steps are inherently serial, including initial build configuration and linking. Therefore, optimizing complex builds means effectively managing these concurrent processes around unavoidable serial choke points To build the best possible build server, we aim to ensure that our compilation processes remain uninterrupted, that we avoid saturating the system's disk I/O, and that we minimize memory thrashing. This specifically includes minimizing memory page allocation and TLB misses. Furthermore, when builds can be parallelized, it is crucial to keep all available cores busy. Using All Your Cores Modern build servers derive most of their performance from parallel compilation. If your build system is not explicitly configured to execute tasks concurrently, large multi-core systems will be underutilized and build times will scale very poorly. For GNU Make, parallelism is explicit, where -j<N> represents the number of parallel jobs to be started, typically set to the number of available CPU cores (for example, up to -j128 on Ampere Altra or up to -j192 on AmpereOne). Shell make -j<N> Most modern build systems provide similar mechanisms: CMake: Parallelism is controlled by the underlying build tool: Shell cmake --build . --parallel <N> Or by passing -j<N> to make or ninja. Ninja: Parallel by default; concurrency can be limited with: Shell ninja -j<N> Maven: Supports parallel module and project builds: Shell mvn -T <N> or -T <N>C (cores) Gradle: Enables parallel task execution: Shell gradle build --parallel Or via org.gradle.workers.max= <N>. SCons: Similar to Make and Ninja, uses –j: Shell scons -j<N> Building LLVM-MinGW We describe building the LLVM-MinGW project to highlight the steps we used to optimize an Ampere-based build server. LLVM-MinGW is a toolchain to build Windows binaries using the LLVM project that supports the i686, x86_64, armv7, and arm64 architectures. The LLVM-mingw repository provides detailed documentation about the project. After cloning the git repo, we set up a 50 GB RAM disk to run the build on to improve performance. We use the build-all.sh script to run the build for 5 different architectures specified in the TOOLCHAIN_ARCHS variable. Shell git clone https://github.com/mstorsjo/llvm-mingw.git ode Shell # Create RAM disk mkdir llvm-mingw-tmpfs sudo mount | grep llvm-mingw-tmpfs mount -t tmpfs -o size=50G,mode=1777 tmpfs ./llvm-mingw-tmpfs Shell # run build from RAM disk LOG=build-llvm-mingw.log cd llvm-mingw-tmpfs && rm -rf * && cp -r ../llvm-mingw/* . && TOOLCHAIN_ARCHS="i686 x86_64 armv7 aarch64 arm64ec" LLVM_CMAKEFLAGS="-DLLVM_ENABLE_LIBXML2=OFF -DLLDB_ENABLE_PYTHON=OFF -DLLVM_USER_LINKER=lld -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++" /usr/bin/time -f '%U, %S, %e, %P' ./build-all.sh --disable-lldb-mi $(pwd)/install/llvm-mingw >& ${LOG} RAM Disk: Using DRAM to Avoid Disk I/O A RAM disk is a filesystem backed by system memory rather than persistent storage. On Linux, this is typically implemented using tmpfs, which allows a directory to behave like a regular filesystem while storing its contents in RAM. Reads and writes to a RAM disk occur at memory speed and are much faster than reading from disks or SSDs. On a build server, a RAM disk is commonly used to store temporary build artifacts such as object files, dependency caches, and intermediate outputs. The build system can then be configured to place its build directory, temporary files, or compiler cache under this path (for example, using an out-of-tree build directory). In our example, the entire build is done on the RAM disk. RAM disks can improve performance because large parallel builds can generate and consume tens of thousands of small files, especially during compilation and linking. Even on fast NVMe SSDs, metadata operations and concurrent I/O can become a bottleneck when hundreds of compiler processes run in parallel. A RAM disk eliminates these I/O constraints by keeping all intermediate data in memory, reducing latency and contention. On high-core-count servers running large parallel builds, this can significantly improve throughput by ensuring that compiler processes are not stalled waiting for disk access. The benefit is most visible when the build is otherwise CPU-bound and sufficient memory is available to avoid swapping. In this configuration, a RAM disk helps the system sustain full CPU utilization across all cores, shortening end-to-end build times. It is always recommended to measure the performance of any configuration changes to understand their impact and to verify that performance has improved. In this case, we tested building using a RAM disk vs. using the system SSD. The build took 1069.1 s running on the SSD and 1062.7 s using the RAM disk, for a ~0.6% speedup. The mpstat utility, part of the sysstat package, was used to measure CPU utilization during the build, with the results plotted in Figure 1. This data shows that the overall CPU utilization is very low for the build, except for the region from ~100 to ~450 seconds and for brief spikes of high CPU utilization. Overall, the average build total CPU utilization was just 53.4% of the Ampere ® Altra ® 128-core server. This raised the question: Why is utilization so poor? Fig 1: Server CPU utilization vs. time for initial LLVM-MinGW build, illustrating poor overall server utilization except for the region approximately 100 – 450 seconds. Our initial studies focused on running the build using the perf record performance profiler to measure performance. Linux’s perf profiler is a very powerful tool that allows you to see what’s running on the CPU at the application level and drill down to shared libraries, functions, individual source code lines and assembly instructions. However, sometimes a focus on low-level metrics prevents understanding global issues. This proved to be one such example. For this project, understanding the true bottleneck required higher-level performance data to grasp why the build was not utilizing the system more efficiently. Instrumenting Build Scripts The LLVM-MinGW build is complex, involving many sub-projects built with different configurations and architectures. To gain detailed insights, we implemented a powerful yet simple instrumentation method using bash’s PS4 variable. This approach inserted timestamps into the build log file before every executed command. By parsing the output log file, we could accurately determine the duration of each command. While analyzing and understanding the various, often nested, phases of the build took time, the instrumentation via bash’s PS4 variable was trivial to implement. Shell # Script to add timestamps to all scripts PS4='DEBUG $0 Line ${LINENO}, time since epoch, $(date "+ %s"): ' SCRIPTS=$(ls *sh) for SCRIPT in ${SCRIPTS}; do cp ${SCRIPT} ${SCRIPT}.orig sed -i '1d' ${SCRIPT} # remove #!/bin/sh line # converts existing scripts to bash scripts # with timestamps using bash’s PS4 cat ../convert_sh_to_bash_debug.sh ${SCRIPT} > bash_${SCRIPT} cp bash_${SCRIPT} ${SCRIPT} done chmod +x *.sh Figure 2 shows an example instrumented output. This output can be parsed to calculate the execution time of individual commands, such as this example of running cmake –G Ninja. Fig 2: Example output after adding timestamps to the build. After analyzing the build output with the timestamps we added, we identified significant time spent in processes that utilized only a single CPU core: running configure and cmake -G Ninja configuration phases, and pulling sources via git. Now that we understood the significant serial phases of the build, we were ready to optimize the server configuration to improve the build performance. Figure 3: Breakdown of build, based on adding instrumentation to measure its phases. This shows a large fraction of the build is due to running configuration scripts (cmake -G Ninja and configure), which are serialized and limit scaling. CPU Performance Governor The CPU frequency governor in Linux is the kernel mechanism that controls a range of performance and power management-related features for each CPU core. It can be used to tell CPU cores to run at maximum frequency, maximum power efficiency, or to dynamically scale on demand. You can see what the current setting is for all CPU cores with the command: Shell cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor The possible values for Linux are: powersave: Prioritizes energy efficiency, but modern CPUs can still ramp frequency under load. A good option for power-limited workloads and laptopsondemand/conservative: Ramps up frequency under load, and down when idle. conservative ramps up and down more slowly than ondemandperformance: Ensures the CPU cores are always running at maximum frequencyschedutil: A common default for modern Linux distributions, uses scheduler utilization metrics to adjust frequency If your CPU cores are not running at their maximum frequency, or if they need to ramp up to maximum frequency when you start a build, this can have a huge impact on build times. By explicitly tuning all CPU cores to be in performance mode, we can maximize build throughput for CPU-bound workloads like software compilation. You can manually set the governor to performance mode using either the cpupower utility or by changing the setting that we viewed earlier on the command line: Shell echo performance | sudo tee \ /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor These changes are runtime only, however. If you want this change to persist across reboots, you can use the tuned service and the tuned-adm utility on Linux to configure how aggressive the power management functions of your operating system behave when your system is idle, and to ensure that the option persists across restarts. To use tuned to configure your server for high performance mode, run: Shell sudo systemctl enable --now tuned sudo tuned-adm profile performance This setting should make a huge difference for high-core, CPU-bound build servers. Software compilation is a workload that benefits from cores running at maximum frequency. Dynamic governors (powersave, ondemand, conservative, schedutil) may let cores scale up only after work arrives, but this will introduce latency before full CPU power is available. For builds that spawn hundreds of parallel compilation jobs, this delay can add up across many cores and many files. Explicitly setting all cores to performance, the governor ensures that every core is immediately available at peak frequency, maximizing throughput for CPU-bound workloads. Performance Governor Impact: 17% Speedup After configuring the server to use the performance governor, we saw a 17% speedup in build time. The build running with a RAM disk improved from 1,063 to 909 seconds. Figure 4: Server CPU utilization vs. time for initial LLVM-MinGW after setting the CPU performance governor, we observed a 17% speedup in build time. CONFIG_HZ: The Heartbeat of the Linux Kernel Have you ever tried to work on multiple tasks at the same time — say, composing an email while preparing a presentation, and attempting to characterize and fix a software bug? This is inefficient for humans because we need time to stop working on one task, gather enough context for the new task, and regain full efficiency — a process we call context switching. CPU cores experience a similar overhead when changing the task they are executing. They must load the instructions and data for the new task into L1 and L2 caches, evicting the old context in the process, and refilling the CPU pipeline with new instructions. This whole process can cost hundreds to a few thousand CPU cycles on modern CPUs. With cores operating at 3 GHz (three billion CPU cycles per second), that means each context switch can incur an overhead of around one microsecond. Inside the kernel process scheduler, there is a separate heartbeat, as the Linux kernel determines which process will get access to each CPU core, based on its priority, how long it has been waiting for a time slice, and a few other criteria. The kernel compile-time parameter CONFIG_HZ determines the frequency that the scheduler makes these decisions, potentially triggering a context switch as it evicts one process and gives another process a time slice. By default, on most Linux distributions, the kernel is compiled with an idle-tickless configuration (the kernel build option CONFIG_NO_HZ=y) and a timer frequency of 250 Hz (the compile-time option CONFIG_HZ_250=y, which sets CONFIG_HZ=250). That means that on busy CPU cores, the process scheduler triggers 250 times a second, or once every 4 milliseconds, but on idle CPU cores, the tick is suppressed, reducing unnecessary interrupts. When the scheduler is triggered, the kernel checks whether the current process running on the core should be preempted and replaced with another process that is waiting for CPU time. On a build server, we typically do not want compilation processes to be preempted. These tasks are relatively short-lived for most source code files, often completing in under a second (and frequently under 100 ms) from C source to object file. As a result, if a compiler process is preempted by the kernel, context switches can represent a significant percentage of the execution time for compiling that source file. There are four possible values for CONFIG_HZ, with 100 Hz, 250 Hz, and 1000 Hz being most common. While higher CONFIG_HZ values are often chosen for workloads demanding very low latency and high responsiveness, for throughput-focused CPU-bound tasks like compilation, a lower CONFIG_HZ is generally beneficial. A lower frequency means the kernel's scheduler checks less often, reducing the likelihood of preemption for short-lived compiler processes and decreasing overall scheduler overhead. For our build server optimization, we will test the effect of setting CONFIG_HZ to 100, which sets the length of a jiffy (the time between clock ticks) to 10ms." Running with a 100 Hz kernel improved the build time to 889.2 vs. 909.31 seconds, for an additional 2% improvement. Building With AmpereOne Lastly, we ran the build on the AmpereOne 192-core processor and compared its performance against the Ampere Altra 128-core processor. The build completed in 603.6 s. The graphs (Figure 5) demonstrate that the multicore-intensive build phase was effectively halved, improving from over 300 seconds to approximately 150 seconds. Despite this significant improvement, we measured that approximately 75% of build time utilized one CPU core, with occasional spikes in demand. To further improve the server CPU utilization, one could run multiple builds and/or CI testing in parallel. Figure 5: Server CPU utilization over time for LLVM-MinGW on Ampere Altra Max ( top) vs. AmpereOne A192-32X Processor (bottom), showing a significant speedup Summary and Recommendations This guide explored our approach to configuring Linux application build servers for optimal performance on Ampere's Arm-based architecture. We detailed how various build systems enable parallel execution and demonstrated effective methods for measuring server utilization. Utilizing an innovative bash shell PS4 variable method, we instrumented the LLVM-MinGW build process, which involves numerous sub-projects across five different architectures. Our analysis revealed that a significant fraction of the build time was consumed by serial configuration phases. As these are inherently serial, they severely limit the overall CPU utilization of the server, even with powerful multi-core processors. Through our targeted optimization efforts — including using RAM disks, setting the CPU performance governor, and tuning kernel parameters such as CONFIG_HZ — we significantly improved build throughput for this specific project. Our findings also highlighted that for complex projects with persistent serial bottlenecks, the ultimate performance gains on high-core systems often come from parallelizing multiple independent builds or CI testing workflows. Recommendations We encourage you to apply these optimization strategies to your own Ampere-based build servers. To efficiently utilize the resources on a multi-core dedicated build server, we recommend that you implement the following recommendations: Implement parallel build flags – this enables the critical compute-intensive phase of builds to completely saturate available cores.Configure the performance CPU governor – ensure that your build server cores are not reducing performance by aggressively entering power-saving mode.Consider CONFIG_HZ adjustments to unleash the full potential of your hardware – context switches can have a significant impact, and slowing the heartbeat of a system minimizes them.To improve server throughput, explore offsetting build jobs or running testing jobs simultaneously. We have seen that configuration and linking stages do not scale as efficiently to multiple cores. If you start builds offset by time, or run jobs with different compute demands, you can execute these single-threaded parts of the build process in parallel with the intensive build stages of other jobs. How to optimally schedule build jobs to maximize throughput is left as an exercise to the reader. For further discussion, share your experiences or seek additional guidance by joining the Ampere developer community forum at https://community.amperecomputing.com. Check out the full Ampere article collection here.

In recent years, building modern applications has changed from what has been seen historically. Usually, in the past, systems were developed with a single, large block of code (referred to as a monolithic design) and would operate fairly well for smaller applications, but with time, as they got larger and more complex, the method of writing software became more of a hindrance to the applications as they required more users and increased speed. Now, companies need their applications to be able to grow quickly, adapt to changes quickly, and be able to support millions of users without any impact on performance, and that is where microservice architecture is so relevant. Microservice architecture has become the way to design scalable applications because applications can be broken into smaller, individual services that can work independently from each other. The trend towards microservice architecture in developing applications that can scale indicates to me that there is a shift in value towards being flexible, quick, and resilient in the highly competitive digital environment we live in today. What Is Microservices Architecture? Microservice architecture is a method of designing an application as a set of distinct parts that operate independently and perform specific tasks. Each microservice communicates with the others via APIs. With a microservice architecture, as opposed to a traditional monolithic system where all of the application’s components are dependent upon one another, developers can modify/update/deploy/scale a single microservice without impacting any of the other microservices in the application. In an e-commerce application, the components include user authentication, product catalog, payment processing, and order processing (each of these services exists as a microservice). Why Microservices Are Gaining Popularity Microservices are more than just a trend; they are the answer to increased demands for scalable, flexible, and high-performing applications. As digital-first business models grow, traditional architectures simply can't keep up, driving a preference for microservices. 1. Scalability Requirements Modern applications often deal with unpredictable user traffic, especially during peak times such as high-volume sales, new product launches, or virally driven surges in user traffic. In a monolithic architecture, scaling means replicating your entire application on expensive resources over a long period, which is inefficient. 2. Quick Development Cycles With the rapid pace of change in the marketplace, speed is key to success in competitive industries today. The use of a microservices architecture enables development teams to develop different services simultaneously without affecting one another’s progress. 3. Technology Flexibility The flexibility of technology is one of the greatest benefits of microservices architecture. Unlike Monolithic systems that typically use only one tech stack, each microservice can be built using the best programming language, framework, or database. For example, a data-intensive microservice can use a high-performance programming language as its primary language, while the UI microservice can use a more flexible front-end framework. 4. Enhanced Fault Containment Failure is a fact of life for big programs. What you do when it happens can make a difference. In a monolithic program, a single bug or failure can shut down an entire application. Microservices provide better fault containment by isolating faults to independent services. When an individual service fails, the failure won't automatically affect the rest of the program. This results in higher overall system availability and an improved user experience. 5. Agreement With DevOps Microservices architecture aligns well with DevOps practices, which focus on automation, collaboration, and continuous delivery. With microservices, teams can develop CI/CD pipelines for each of their services so that they can deploy frequently and reliably. Automated testing, monitoring, and deployment allow them to release updates efficiently with minimal risk. The Benefits of Microservice Architecture The rise in popularity of microservices aligns with current trends in the enterprise landscape; however, many organizations are beginning to realize significant value in microservice architecture for application development and performance. Through the use of microservices, an organization can break down large, complex systems into smaller parts (components). By creating applications using smaller components or microservices, organizations can develop highly scalable, resilient, and efficient systems. 1. Services Can Be Deployed Independently Deployment of one or more services can occur independently using a microservices-based architecture. In traditional applications, deploying even a small change would require deploying the entire application (which could take a long time and add significant risk). 2. Improved Scalability Since microservices inherently have scalability as a key design feature, software development companies can concentrate just on scaling those parts of their applications that require more resources rather than scaling an entire application as was done with Monolith-type applications. 3. Greater Agility Agility is extremely important in today’s digital market that changes rapidly. By allowing multiple teams that consist of members from different functional areas to independently develop their own services using microservices, microservices allow us to increase development speed and decision-making speed. 4. Easier to Manage Codebase It is common for large codebases to become challenging to manage over time. One of the advantages of using a microservices architecture is the ability to create smaller codebases that can be easily managed. 5. Increased Reliability Reliability is one of the most important aspects of any system, especially those with a large number of users. Microservices can help improve reliability by isolating faults between services. Conclusion The increase in the use of microservice architectures within scalable applications has led to a change of focus to properly design systems that are flexible, durable, and can grow with an organization's business needs. By breaking down large, complex applications into smaller independent services, organizations can take advantage of better speed of development, increased scalability, and greater reliability of their systems. Although there are some challenges associated with implementing microservices, the long-term benefits will more than justify any upfront investment required to adopt this architectural style in a modern enterprise. Businesses that have a plan, the right tools, and the right people can quickly realize the full benefits of the microservice architecture while providing high-quality digital experiences to their customers.

WebSocket debugging is one of those things that sounds simple until you actually have to do it. The connection looks fine in DevTools, but messages are malformed, timing is off, or the server is behaving unexpectedly — and you have no easy way to inspect what's happening at the frame level without setting up a proxy or installing something heavy. Here's a practical workflow that requires nothing beyond a browser, illustrated with a real debugging scenario. The Problem With WebSocket Debugging HTTP requests are easy to inspect. DevTools shows you the full request and response, you can replay them with curl, mock them with interceptors, and diff payloads in seconds. WebSocket connections are different. Once the handshake completes, it's a persistent bidirectional channel, and most tooling treats frames as an afterthought. The Chrome DevTools WebSocket panel shows you raw frames, but it doesn't let you filter, transform, or replay them. You can see that a frame was sent with a 400-byte payload — but you can't easily extract it, modify it, and resend it to see how the server responds. The common workarounds all have friction: console.log on both sides – requires access to server code, adds noise, and still doesn't let you test edge cases without changing the clientCharles Proxy or mitmproxy – heavyweight, requires SSL certificate setup, and adds a network hop that can change timing behaviorCustom proxy server – takes time to build and maintain, and is overkill for a one-off debugging session None of these is fast when you just need to understand what's happening right now. A Real Scenario: Debugging a Real-Time Chat Feature To make this concrete, here's a situation that comes up often in practice. You're building a chat feature on top of a WebSocket backend. The UI looks fine in testing, but in production, some users report that messages occasionally appear out of order or that a specific type of system message causes the client to crash. You can't reproduce it reliably in your local environment, and you don't have direct access to the production server's logs. The questions you need to answer: What does the actual message payload look like when the crash happens?Is the issue in the message structure (missing field, unexpected type), or is it a timing problem (two messages arriving within milliseconds of each other)?How does the server respond if you send a deliberately malformed message? This is exactly the kind of debugging that browser-only tooling handles well — if you have the right tools. Step 1: Validate the Endpoint With an Online Tester Before anything else, confirm that the WebSocket endpoint is reachable and responding correctly. The tests.ws WebSocket tester is a browser-based tool that lets you connect to any ws:// or wss:// server, send arbitrary messages, and see server responses in real time. No install, no configuration, no account. For the chat scenario: connect directly to your production WebSocket endpoint, send a message that matches the format your client normally sends, and verify the server acknowledges it correctly. If this works as expected, the issue is likely in how the client processes incoming messages, not in the connection itself. The site also provides a free public echo server at wss://echo.tests.ws. Anything you send comes back immediately. This is useful for validating your client-side message serialization — connect to the echo server, send your payload, and confirm what comes back matches what you sent. If there's a mismatch, you've found a serialization bug before you even involve a real server. For the real-time testing step, the interface also shows frame-level details: message direction, payload size, timestamp, and raw content. This is enough to identify structural issues in isolation. Step 2: Intercept Live Traffic With the Chrome Extension Once you've validated the endpoint in isolation, the next step is observing what actually happens in your running application. The tests.ws Chrome extension adds a WebSocket proxy layer directly into Chrome DevTools, without modifying your application code or network configuration. Install the extension, open your application, and open DevTools. A new panel appears that logs every WebSocket frame — direction (sent/received), timestamp, payload size, and raw content — for all connections on the page simultaneously. Unlike the built-in DevTools WebSocket view, you can filter frames by content, copy payloads, and see a cleaner timeline. For the chat scenario, reproduce the conditions where messages go out of order. In the extension panel, you can see the exact sequence of frames with millisecond timestamps. If two messages are arriving 3ms apart and your client processes them synchronously, you'll see the problem immediately in the frame log — even if your application-level logging shows them in the wrong order. Step 3: Modify Outgoing Messages to Test Edge Cases This is where the extension's real value shows up. The extension lets you write JavaScript transform rules that intercept outgoing frames and modify them before they're transmitted to the server. For the crash scenario: you suspect the crash happens when a system message arrives with a missing userId field. Instead of waiting for it to happen in production, you write a transform rule: JavaScript if (message.type === 'system') { delete message.userId; } The extension applies this rule to matching outgoing frames. The server receives the malformed payload, you observe its response in the frame log, and you can immediately see whether it sends back an error, silently drops the message, or sends something that would cause the client to crash. This replaces a workflow that would otherwise require: modifying client code, building a new bundle, deploying to a test environment, and hoping you can reproduce the right conditions. With the extension, the iteration loop is: write a rule, trigger the action in the UI, observe the server response. No code changes, no deployment. Step 4: Test Protocol Edge Cases Beyond the immediate crash scenario, the transform approach is useful for systematic protocol testing: Missing required fields – remove fields one at a time to see which ones the server validatesType mismatches – send a string where the server expects an integer, or an array where it expects an objectOversized payloads – test the server's behavior when message size exceeds expected limitsRapid sequences – send the same message 10 times in quick succession to test for race conditions server-sideMalformed JSON – send a syntactically invalid payload to verify error handling Each of these can be tested in minutes, directly against a running server, without writing test harnesses or modifying application code. When This Approach Has Limits Browser-based WebSocket debugging works well for: Front-end debugging when you don't have server accessQA validation of message formats and server behaviorSecurity testing and input validation checksLearning how a third-party service's WebSocket protocol works It doesn't replace load testing tools. If you need to simulate 10,000 concurrent connections or measure throughput under sustained load, you need something like k6 or Artillery running outside the browser. Similarly, for server-side issues — memory leaks, connection pool exhaustion, handler bugs — you need server-side observability tools. But for the class of problems that are most common during development and integration — "why is the client behaving unexpectedly when it receives this specific message?" — the browser-only workflow gets you to an answer faster than any other approach. Summary The debugging workflow for the chat scenario above: Validate the endpoint – use the online WebSocket tester at tests.ws to confirm the server responds correctly to well-formed messagesObserve live traffic – install the Chrome extension, open the application, and capture the actual frame sequence that leads to the problemReproduce and test – write a transform rule that simulates the malformed message, trigger it in the UI, observe the server's response Total time to go from "users are reporting a crash" to "here's the exact server response that causes it": under 15 minutes, with no infrastructure changes, no deployments, and no server access required. WebSocket tooling has historically lagged behind HTTP tooling. The gap is smaller than it used to be.

One of the most common statements we hear in the software industry is: "The job of a tester is to find bugs." While bug detection is undoubtedly an important part of testing, reducing testing to only finding bugs is one of the biggest misconceptions about the profession. Testing is a systematic process of evaluating software to understand its quality, identify risks, validate requirements, and provide confidence for release decisions. Bugs are simply one outcome of that process. If testing were only about finding bugs, then the tester who reported the highest number of defects would automatically be considered the best tester. However, most experienced engineering teams know that this is far from reality. The Bug Count Trap Many organizations unknowingly create a culture where testing success is measured by the number of defects found. This often leads testers to focus primarily on breaking the system. Breaking the system is important. Exploratory testing, negative testing, boundary testing, and resilience testing all have their place. However, there is a danger when breaking the system becomes the primary objective. A tester may discover multiple corner-case crashes and still miss a much larger problem: the product does not meet customer expectations. In such situations, the system may survive unusual failure scenarios while failing to deliver value in everyday usage. The customer rarely cares about how many defects were found during testing. The customer cares whether the product solves their problem correctly and reliably. Testing Is About Understanding Quality Quality is much broader than defect detection. A tester should continuously ask questions such as: Does the product meet the stated requirements?Does it satisfy the implicit expectations of the customer?Is it usable?Is it reliable?Is it secure?What are the biggest risks before release?What could go wrong in production?What assumptions are we making? These questions provide significantly more value than simply asking, "Can I make this crash?" The best testers are often the people who understand the product, business domain, customer workflows, and operational risks — not necessarily the people who report the most defects. The Missing Piece: Customer Expectations One area where testing frequently falls short is understanding customer expectations. Requirements documents describe what the system should do. However, customers often expect much more than what is explicitly written. For example: A login page may satisfy every documented requirement, but customers still expect: Fast response timesClear error messagesSecure handling of credentialsConsistent behavior across browsers and devices These expectations are rarely documented in detail because they are considered obvious. A tester who focuses only on written requirements may miss these areas completely. A tester who understands customer expectations will naturally test for them. This is where true testing begins. Process Alone Does Not Create Good Testers The industry often swings between two extremes. The first extreme is believing that testing is simply about finding bugs. The second extreme is believing that following a process automatically guarantees quality. Neither is true. Many organizations have adopted Agile practices, ceremonies, templates, and checklists. While these practices provide structure, they cannot replace critical thinking. A tester can execute every step in a process and still miss major quality risks. Good testing requires judgment. It requires curiosity. It requires asking uncomfortable questions. Most importantly, it requires understanding the product and the customer. Testing Is More About Mindset Than Techniques Testing techniques are valuable. Boundary value analysis, equivalence partitioning, decision tables, pairwise testing, state transition testing, and exploratory testing all help improve coverage. The good news is that techniques can be learned from books, courses, mentors, and increasingly from AI. What is much harder to teach is mindset. A strong testing mindset involves: CuriosityCritical thinkingRisk awarenessCustomer empathyProduct understandingThe ability to challenge assumptions In fact, if you observe experienced testers, you will often notice them applying testing techniques naturally without consciously referring to their textbook names. The mindset drives the technique — not the other way around. The AI Shift Changes the Game The rise of AI is forcing the testing profession to re-evaluate where its real value lies. Today, AI can already: Generate test casesSuggest edge casesCreate automation scriptsAnalyze requirementsReview code changes As these capabilities continue to improve, the differentiating factor for testers will not be their ability to produce more test cases. The differentiating factor will be their understanding of: The productThe customerBusiness risksReal-world usage patternsHidden assumptions AI can generate tests. It cannot fully understand organizational context, customer frustrations, business priorities, or the subtle quality concerns that experienced testers identify through years of product exposure. The testers who develop these skills will become significantly more valuable. The testers who rely solely on mechanical execution may find themselves competing directly with increasingly capable AI systems. Building the Right Testing Culture Creating effective testers is not only an individual responsibility. Organizations, managers, and technical leaders all have a role to play. Instead of measuring success solely through defect counts, teams should encourage: Product understandingCustomer empathyRisk analysisExploratory thinkingCross-functional collaborationContinuous learning The goal should be to develop testers who understand why they are testing, not just how they are testing. Final Thoughts Testing is not about finding bugs. Finding bugs is important, but it is only one outcome of effective testing. The real purpose of testing is to provide information about quality, uncover risks, validate expectations, and help teams make informed release decisions. A tester with the right mindset may not always report the highest number of defects. However, they will consistently help the team build better products, reduce risk, and deliver greater value to customers. And ultimately, that is what testing is really about. "Bugs are the byproduct of testing. Confidence in quality is the goal."

Managing high-volume message traffic in distributed architectures is crucial. Efficient use of database and CPU resources is also very important. There are structures that allow us to receive messages in batches. The default Spring Kafka "BatchMessageListener" structure addresses this need. However, the processing of these messages often goes through a sequential bottleneck. This article will discuss the structure and usage of Kotlin Coroutines in detail. We will examine how to maximize Kafka message processing performance using Structured Concurrency principles and Resource Throttling techniques. Architectural Bottleneck: Sequential I/O Blocking On the current Kafka listener: Database or external service calls made for each message directly increase total processing times. If the processing speed of a message lags behind the message arrival speed and the max-poll-interval-ms time is exceeded, the consumer is removed from the consumer group. Rebalancing is triggered, and the partitions of that consumer are redistributed to other consumers in the group. Kotlin @KafkaListener(topics = ["usage-pool-topic"]) fun usagePoolListener(records: List<ConsumerRecord<String, String>>) { records.forEach { record -> processRecord(record) // Network latency + DB I/O blocking } } Solution 1. Batch-Fetch and In-Memory Map Structure Before any concurrent code is entered, data is retrieved collectively from all necessary entities. Multiple separate queries are converted into a batch query before data processing begins. The N+1 query problem is solved at the application layer. All data is cached once before being broken down into concurrent operations. Having the data cached significantly reduces our reliance on the database. Using the associateBy function, we transform the data into a map structure with X access times. This allows us to read the data safely from the maps instead of reading each concurrent operation from the database. Kotlin val messages = records.map { objectMapper.readValue(it.value(), UsagePoolRecord::class.java) } val usagePoolEntities = usagePoolRepository .findByIds(messages.map { it.usagePoolId.toBigInteger() }) .associateBy { it.usagePoolId } val lockEntities = lockRepository .findByUserIds(messages.map { it.userId }) .associateBy { it.userId } 2. Structured Concurrency Memory Management With Chunking The chunk structure serves two purposes. It prevents the creation of coroutines simultaneously. This prevents unnecessary memory usage. Each chunk writes to the database after all coroutines have completed their operations. Unnecessary connection pool consumption is avoided. Kotlin messages.chunked(150).forEach { chunk -> // Each chunk of 150 records is processed concurrently } Resource Isolation With limitedParallelism Why limitedParallelism? If the database connection pool has, for example, X connections, keeping the parallelism limit below X prevents "Connection Timeout" errors. Kotlin messages.chunked(150).forEach { chunk -> val deferredResults = chunk.map { record -> CoroutineScope(Dispatchers.IO.limitedParallelism(15)).async { try { processRecord(record, usagePoolEntities, lockEntities) } catch (e: Exception) { log.error("Operation error: ${record.key()}", e) buildErrorRecord(record, e) } } } val results = deferredResults.awaitAll() // Structural waiting collectAndAggregate(results) } The Dispatchers.IO.limitedParallelism(X) command limits the number of concurrent coroutines to X, preventing the DB connection pool from being exhausted.Each coroutine returns a result with the async command. The awaitAll() command waits for all coroutines in the chunk to finish before proceeding to the next step. runBlocking This function blocks callers until all concurrent operations are complete. This is the correct approach here because: It ensures that the Kafka consumer remains blocked to maintain its offset commit structure until all records in the batch are processed. We still benefit from concurrent operation parallelism within the runBlocking block. 3. Thread-Safe Result Structure After the awaitAll() operation, all results are collected in thread-safe queues. Then a single batch write operation takes place. Using MutableList structures to combine results returned from parallel processed coroutines can lead to data loss. At this point, lock-free data structures should be preferred. ConcurrentLinkedQueue uses CAS (Compare-And-Swap) algorithms instead of synchronized blocks. This provides superior performance in high-content write operations. Why Shouldn't We Use ConcurrentLinkedQueue? Concurrent operations (concurrent functions) perform simultaneous write operations to a shared collection of results. Using MutableList leads to race conditions. It performs well in secure and concurrent write operations. Kotlin data class AggregatedRecords( val processedSave: ConcurrentLinkedQueue<ProcessedEntity> = ConcurrentLinkedQueue(), val toDelete: ConcurrentLinkedQueue<UsagePoolEntity> = ConcurrentLinkedQueue(), val retryQueue: ConcurrentLinkedQueue<RetryEntity> = ConcurrentLinkedQueue() ) The DataIntegrityViolationException return is important. When two consumer instances are processing the same record, one of them falls into a unique constraint violation. Instead of making the entire batch fail, record-by-record deletion is performed. Kotlin AggregatedRecords.processedSave .chunked(150) .forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { batch.forEach { record -> try { processedRepository.save(record) } catch (e: DataIntegrityViolationException) {} } } } 4. Error Tolerance in Write Operations Batch write (saveAll) operations are performant. However, a "Unique Constraint" error in a single record can cause the entire batch to fail. The following structure is critical to meet Optimistic Locking or Idempotency requirements. Kotlin aggregatedRecords.processedSave.chunked(150).forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { // Fallback: Try one by one if batch fails batch.forEach { record -> try { processedRepository.save(record) } catch (innerException: DataIntegrityViolationException) { log.warn("Duplicate record skipped: ${record.id}") } } } } 5. Data Flow Diagram Ingress: The Kafka batch is caught with runBlocking.Preparation: All necessary context data is retrieved bulk from the DB.Execution: Coroutines are started asynchronously in chunks.Synchronization: The completion of all coroutines is awaited as a barrier point with awaitAll().Egress: Collected results are made permanent with saveAll. Performance Analysis and Results Conclusion Processing Kafka messages in Spring Boot with Kotlin Coroutines not only increases speed but also improves code readability and makes resource management deterministic (predictable). The use of runBlocking allows us to build a bridge between the blocking Kafka consumer thread and the suspended world without disrupting Kafka's offset management mechanism. Dependencies XML <dependency> <groupId>org.jetbrains.kotlinx</groupId> <artifactId>kotlinx-coroutines-core</artifactId> <version>1.7.3</version> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency>

Every time you open your banking app, send a private message, or log into your company's systems, a math problem is standing between your data and the rest of the world. A very specific kind of math problem, one that takes thousands of years to solve, even for the fastest computers we have today. Here is the uncomfortable truth: quantum computers are coming. And when they arrive, that math problem gets solved in hours. The lock breaks. Everything behind it becomes readable. The good news? There is already a replacement. It is called lattice cryptography, it is already available, and the window to start adopting it is open right now. Whether you act on that window is the real question. Why "Hard Math" Is the Entire Foundation of Internet Security Most encryption used today, including the RSA algorithm that secures the majority of HTTPS connections, e-commerce transactions, and enterprise systems, rests on a single idea: it is very easy to multiply two large prime numbers together, but extraordinarily hard to reverse that process. Multiply 7 and 3, and you get 21 instantly. But hand someone the number 21 and ask them to find the two prime factors without any hints, and the process gets harder. Scale that problem up to a 600-digit number, and even the most powerful supercomputers on Earth would need thousands of years to crack it. That difficulty is what makes your data safe. For now. A quantum computer with 4000 stable qubits could run Shor's algorithm to factor large integers, breaking RSA-2048 in a matter of hours. - NIST IR 8105, "Report on Post-Quantum Cryptography," 2016 Shor's algorithm, discovered in 1994, is essentially a quantum shortcut through that math. Once quantum hardware catches up to the algorithm's requirements, RSA and similar schemes collapse. Not weaken. Collapse. The Threat You Cannot See Yet: Harvest Now, Decrypt Later Here is what makes this threat different from most others in cybersecurity: you do not need to wait for quantum computers to exist before they can hurt you. Sophisticated adversaries, including nation-state actors, are already collecting encrypted data today. They store it. They wait. When sufficiently powerful quantum systems arrive, they decrypt everything in that archive. Health records, financial data, intellectual property, classified communications from years ago - all of it becomes accessible retroactively. Adversaries may be stealing encrypted data now with the intent to decrypt it later when quantum computing capabilities mature. This 'harvest now, decrypt later' strategy is a real and present danger. — CISA, NSA, NIST Joint Advisory: "Quantum-Readiness: Migration to Post-Quantum Cryptography" 2023 If your systems handle data that must remain confidential for more than five to ten years, that window is already a concern. Medical records. Legal documents. Financial histories. Long-term contracts. Any of these could be sitting in an adversary's archive right now, waiting. Lattice Cryptography: A Math Problem Even Quantum Computers Cannot Shortcut The replacement is built on a completely different class of hard math problem. One where quantum computers have no known shortcut. Picture a chess knight on an infinite board. In standard chess, a knight moves in a fixed pattern: two squares in one direction, one in another. If you know the move pattern, you can easily reach any target square by combining moves. That is basic, predictable cryptography. Now imagine the board has a thousand dimensions instead of two. The target point does not land exactly on any reachable square. You can only get close, never exact. And every attempt to get closer involves navigating a space so vast that trying every possible combination of moves would take longer than the age of the universe. That is the core idea behind lattice cryptography, and more specifically, a problem called Learning With Errors (LWE). The Learning With Errors problem asks to find a secret vector given a set of approximate linear equations over a finite field. The hardness of LWE is based on the worst-case hardness of standard lattice problems, which are believed to be resistant to quantum attacks. - Oded Regev, "On Lattices, Learning with Errors, Random Linear Codes, and Cryptography," Journal of the ACM, 2009 The "noise" Regev introduces into the problem is the key. Without it, solving the system of equations would be straightforward. With it, even a quantum computer exploring multiple solution paths simultaneously hits a wall. There is no elegant shortcut. Just brute force, across a space too large to brute force. NIST Has Already Done the Hard Work The U.S. National Institute of Standards and Technology ran an open global competition for nearly a decade, inviting cryptographers worldwide to submit quantum-resistant algorithms. In 2024, three algorithms were standardized. NIST has finalized its principal set of encryption algorithms designed to withstand cyberattacks from a quantum computer. These post-quantum cryptography (PQC) standards are ready for immediate use. - NIST, "NIST Releases First 3 Finalized Post-Quantum Encryption Standards," August 2024 The three standards are CRYSTALS-Kyber (now called ML-KEM) for key encapsulation, CRYSTALS-Dilithium (ML-DSA) for digital signatures, and SPHINCS+. All three are publicly available, open source, and deployable today on existing hardware. You do not need quantum computers to run quantum-safe encryption. That is a critical point. The algorithms run on the same servers and devices you already have. What "Crypto Agility" Actually Means in Practice For software architects and engineering leaders, the challenge is not just adopting new algorithms. It is building systems that can swap algorithms without a full architectural overhaul. The concept is called crypto agility. Think of it as designing your cryptographic layer the same way you would design a database abstraction layer: the rest of your system should not care which specific algorithm is running underneath. When a vulnerability surfaces, or when standards evolve, you should be able to change the algorithm with minimal blast radius. Getting there requires a structured approach. It starts with discovery: building a complete inventory, sometimes called a Cryptographic Bill of Materials (CBOM), of every place in your environment where cryptography is in use. That includes custom implementations, third-party libraries, hardware security modules, APIs, certificates, and protocols. Many organizations discover they have hundreds of instances they were not tracking. From that inventory, you triage by sensitivity. Data with long confidentiality requirements gets migrated first. Then you remediate, test, and build the feedback loop that lets you keep the CBOM current as your systems evolve. Organizations that do not understand their current cryptographic deployments will be unable to prioritize or execute a successful migration to post-quantum cryptography. - NIST SP 1800-38B, "Migration to Post-Quantum Cryptography," 2023 (Draft) This is not a one-time project. It is an ongoing capability. The organizations that will handle the next generation of cryptographic transitions well are the ones building that capability now, not the ones scrambling to respond when a deadline arrives. The Clock Is Running, But the Path Is Clear Estimates on when quantum computers will be capable enough to break RSA at production scale vary. Some researchers say a decade. Some say sooner. Nobody says never. We assess that a cryptographically relevant quantum computer could be built within the next decade, with nation-state actors most likely to be first. - Global Risk Institute, "2023 Quantum Threat Timeline Report," Michele Mosca and Marco Piani What is not in dispute is that the migration itself takes time. Updating cryptographic infrastructure across large organizations, particularly those running complex legacy systems or regulated environments, is measured in years, not weeks. The organizations that start now will be ready when the capability arrives. The ones that wait will be in the worst possible position: racing to retrofit under pressure. The math has already changed. The only remaining variable is whether your architecture changes with it.

Most SaaS breaches do not happen through failure. They happen through valid authentication being trusted too far, for too long, across systems that were never designed to question each other. That distinction is worth sitting with. Because if authentication failed, you'd know. You'd see it in the logs. The SIEM would fire. The investigation would start in an obvious place. When authentication succeeds — and authorization is simply absent, or context has shifted since the token was issued — the system looks healthy right up until it isn't. The logs show normal traffic. The requests look legitimate. The damage accumulates silently. This is the actual threat model for modern SaaS, and it is not adequately reflected in how most teams design, audit, or respond to their systems. The Cloudflare Case Is the Template In February 2024, Cloudflare published one of the more technically honest post-mortems the industry has seen. Their internal Atlassian environment — 14,099 Confluence wiki pages, 2 million Jira tickets, 11,904 Bitbucket repositories — had been accessed by a suspected nation-state actor. The intrusion ran for nine days before detection. The entry point was not an exploit. During the Okta breach on October 18, 2023, attackers stole one service token and three service account credentials belonging to Cloudflare. These credentials were not rotated because, mistakenly, they were believed to be unused. That is the full story of the breach. Credentials issued during one incident. Not rotated. Still valid. Still honored by Cloudflare's systems months later. A JWT created for the Moveworks Gateway was forwarding authenticated HTTP requests directly to the private, self-hosted Atlassian server. Incoming HTTP requests that attached the JWT were forwarded without further challenge. The token was valid at issuance. The system never re-evaluated whether the holder still had legitimate standing to use it. Most SaaS breaches are not authentication failures — they are trust relationships that were never designed to expire. That line is not a platitude. It is a precise description of how Cloudflare, Microsoft, BeyondTrust, and dozens of less-publicized organizations were breached in the past eighteen months — not because their authentication systems failed, but because token validity was treated as a continuous proxy for authorization correctness. It is not. What Stacking Trust Layers Actually Produces Modern SaaS architectures are composites. A single user action might pass through an API gateway, traverse a microservice boundary, call an identity provider, issue a token validated by a third-party integration, and write to a data layer with its own access model. Each component was built by different teams, under different threat models, in different years. Each layer assumes the previous one enforced constraints correctly. This assumption is not verified at runtime. It is inherited from the original design — which means it degrades silently as the design evolves. JWTs remove central control points, which also removes real-time revocation visibility. OAuth delegation enables fast integration, which also means trust propagates across service boundaries that nobody charted when the original token was issued. API gateways handle routing and coarse-grained access control, which services downstream interpret as authorization clearance they did not themselves perform. The result is not insecurity in any one component. It is trust drift across the composite — a gradual divergence between what the system was designed to permit and what it actually permits, with no mechanism to detect the gap until something external forces the question. IAM Drift: The Slow Accumulation Nobody Audits By the time a breach is discovered in a SaaS environment, the permissions that made it possible have typically been accumulating for months. Sometimes years. Through entirely routine, well-intentioned decisions. A role gets created for a project and is never sunset. A contractor is provisioned at an elevated scope to expedite an integration, then forgotten during offboarding. An OAuth application receives administrative permissions during testing, and nobody downgrades it before the production cutover. A CISA warning from early 2024 highlighted how Russian-affiliated APT29 was targeting dormant cloud accounts belonging to former employees of government agencies — accounts with standing permissions that outlasted the people they were created for. Dormant accounts with live permissions are not an edge case. They are a near-universal condition in organizations running SaaS stacks for more than three years. Russian attackers known as Midnight Blizzard gained access to Microsoft's internal systems, exploiting compromised credentials through a legacy OAuth application, which enabled the exfiltration of senior executives' emails. The phrase "legacy OAuth application" deserves more attention than it usually gets in the incident coverage. Legacy here does not mean ancient. It means provisioned before the current access model, never audited for scope creep, and still fully honored by every downstream service that inherited trust from the original identity provider. In modern SaaS, trust is not broken — it is inherited too broadly, and then never re-examined. Organizations that treat IAM as a provisioning function rather than a continuous enforcement function will produce permission surfaces that nobody at the organization can fully account for. That surface is exactly what sophisticated attackers map before they move. The Authorization Gap Nobody Wants to Instrument Authentication got the industry's attention first because it is legible. Failed authentication produces clear signals. Broken authorization, by contrast, is architecturally subtle and operationally expensive to detect — which is why it remains the more reliable attack surface. The production pattern looks like this: a user authenticates correctly, receiving a valid, properly signed token from a trusted provider. They make an API call. The gateway routes it because authentication passed. The downstream service validates the token signature and executes the operation — without independently evaluating whether the scope in that token is appropriate for this specific operation, or whether the tenant context in the request header was derived server-side from verified identity, or provided by the client. In August 2025, threat actor UNC6395 used stolen OAuth tokens from Drift's Salesforce integration to access customer environments across more than 700 organizations. The attacker needed no exploit and no phishing. The activity looked legitimate because it came from a trusted SaaS connection rather than a compromised user account. 700 customer environments. No exploit. No phishing. Just a token accepted by systems built to honor tokens — with no service in the chain asking whether this token should be trusted to make these calls on behalf of those customers. The authorization logic that would have caught it was simply not there. One integration became a doorway into everything connected to it. That is not an accident of implementation. It is the predictable consequence of treating third-party integrations as trusted extensions of the platform rather than as external parties with scoped, audited, time-limited access. Multi-Tenant Isolation: Where the Shortcut Becomes the Attack Vector Multi-tenant isolation is architecturally expensive. The pressure to shortcut it is real, and I say that without judgment — I have talked to enough platform engineers to understand the sprint calculus. The common shortcut is this: tenant context flows as a client-supplied parameter — a header, a query field, a value in the request body — which the server accepts and processes as valid context. The reasoning is that only authenticated clients can reach the endpoint, so the tenant ID they provide can be treated as ground truth. This reasoning holds until a token is stolen, a scope is broader than intended, or authorization checks are inconsistent across services. At that point, tenant boundary enforcement becomes entirely dependent on client honesty — and attackers are not honest. When tenant identity is client-provided rather than server-derived from verified credentials, cross-tenant data exposure is not a vulnerability. It is a design property. The only questions are timing and who finds it first. SaaS breaches surged 300% in 2024, with attackers able to compromise core systems in as little as nine minutes. Nine minutes is not reconnaissance time. It is the execution time of someone who already understood the gap, because architectural gaps are consistent and therefore mappable in advance. What Secure Systems Actually Do The teams I have observed building more durable SaaS security postures are not necessarily running more tools. They are enforcing different constraints at the design layer. Authorization is evaluated independently at every layer. Not "the gateway checked, so the service trusts." The service evaluates the request. The data layer enforces row-level policies. Each component performs its own authorization decision in context, at request time. This is operationally expensive. It is also the only architecture that fails safely when one layer is compromised. Identity is bound to the runtime context, not the login state. A token issued at login does not carry indefinite authorization for sensitive operations. Context — session recency, request origin, device posture — is re-evaluated at privilege boundaries. Escalation patterns trigger reauthentication. The cached token is not sufficient. Tenant isolation is a server-side invariant, not a client-side convention. Tenant ID is derived from verified identity. It is never accepted as input. Non-human identity receives the same lifecycle discipline as human identity. In December 2024, BeyondTrust identified a security incident in which a BeyondTrust infrastructure API key for Remote Support SaaS had been compromised and used to enable access to certain Remote Support SaaS instances by resetting local application passwords. API keys, service account tokens, and integration credentials are identity. They accumulate permissions. They outlast the contexts that justified them. Organizations that audit human identities quarterly and review machine credentials annually will find that the gap between those schedules is exactly where attackers operate. The Real Gap Is Not Knowledge There is a version of this analysis that ends with a list of OWASP API Security Top 10 items and a recommendation to evaluate SSPM vendors. That version is accurate. It is also not the reason any of this keeps happening. The issue is not just credentials or misconfigurations; it is the lack of visibility, real-time threat detection, and the inability to block threats before damage occurs. But even that framing undersells the structural problem. Engineers know what broken object-level authorization looks like. Security architects understand token scope. Post-mortems from Okta, Cloudflare, and Microsoft have been widely read. The gap is enforcement under velocity pressure. Authorization models do not get updated when features ship. Integrations get added without full accounting of the trust they inherit. Scopes get provisioned broadly because narrow provisioning takes time that the sprint cannot absorb. The system keeps working — correctly, from its own perspective — until someone external points out what it has been silently permitting. Brian Soby, CTO of AppOmni, framed the organizational consequence clearly: "In 2024, business was disrupted by costly SaaS 'bypass' breaches that circumvented IAM and zero-trust controls. 2025 will bring awareness to end-to-end controls needed for SaaS, with tight interdependencies between zero trust, identity, SaaS posture, and detection and response capabilities." End-to-end. Not perimeter. Not gateway. Not identity provider in isolation. Every integration point. Every inherited trust relationship. The threat model has to be continuous, or the gaps accumulate exactly where the coverage stops. The Question That Catches the Failure Verizon's 2025 Data Breach Investigations Report examined more than 22,000 security incidents; 30% originated from a third party, including SaaS applications and software vulnerabilities. Third-party integrations are now a primary attack surface — not because they are inherently insecure, but because they are the points at which one system extends trust to another system it did not design, does not control, and often does not monitor. The engineers who consistently build more defensible systems are not necessarily the ones with the most security certifications. They are the ones who read an architecture diagram and ask the productive question before anything ships: what does this component assume the other layer is enforcing — and what happens when that assumption is wrong? That question, applied systematically, catches most of the failure modes described above. Not all of them. Systems are complex, and attackers are patient. But it catches the predictable ones — the inherited trust that was never re-examined, the token that outlived its context, the tenant boundary that depended on client honesty. The question your systems need to be able to answer is not whether they are secure at the edge. It is whether your trust relationships are still valid after they were first created — and whether you have any mechanism to know if they are not. Most production systems do not. They will continue operating correctly — until correctness is no longer the same thing as safety. The author covers cybersecurity architecture, DevSecOps, and identity systems engineering. Pushback, corrections, and firsthand incident accounts are welcome.

By Igboanugo David Ugochukwu CORE

Analytics pipelines tend to scale in both cost and the age of their data sources: costs increase with data volume growth, while data freshness decreases due to longer batch jobs. The common approach, scaling out the cluster, addresses the symptom rather than the architectural issue. In this tutorial, we will look at an alternative solution that addresses both problems at their root: using Netflix Maestro, a horizontally scalable workflow orchestrator open-sourced by Netflix in July 2024, along with Apache Iceberg, a standard table format for analytics on object storage. The former helps by shifting from time-based scheduling to event-driven, whereas the latter removes the overhead of listing files that slows down queries on large datasets and increases their costs. We will cover all aspects of creating a full-fledged pipeline, including code examples, explanations of why each component reduces costs, and real metrics showing what results to expect. What You'll Need ComponentPurposeNotesApache Iceberg + a catalogTable format and metadata managementREST catalog (Polaris, Nessie, Lakekeeper, Unity Catalog) recommended for new deployments; Glue/Hive also fineA compute engineReads and writes Iceberg tablesSpark 3.5+, Flink, Trino, or DuckDB via PyIcebergNetflix MaestroWorkflow orchestrationRequires Java 21, Docker, and Postgres or CockroachDB for stateCloud object storageData files and metadataS3, GCS, ADLS, or S3-compatible (MinIO works for local dev)Python 3.10+Lightweight tasks and ingestionPyIceberg 0.11+, PyArrow Terminology note: there are several products named "Maestro" in the data space. This guide is about Netflix's Maestro and is different from Maestro by Conductor, AWS Maestro, etc. Netflix's Maestro executes hundreds of thousands of workflows and up to 2 million jobs per day inside Netflix, so the scalability claim is valid — although some practitioners consider Maestro overengineered for small teams, so keep that in mind. The Problem Statement The standard stack on Hive tables stored in S3 has three structural inefficiencies: File listing dominates query planning. Listing operations on S3 are slow and rate-limited. For a query on a partitioned Hive table, listing might take more time than reading data itself.Small-file proliferation. Continuous or micro-batch writing produces thousands of Parquet files. Each query suffers from open-file overhead, and each list operation brings in additional results.Time-based scheduling wastes compute. Jobs are triggered based on a fixed schedule, not data availability. If upstream data is late, the job processes stale inputs. If the data is early, the job idles until the next scheduled run. Iceberg solves (1) and (2) in the storage tier. Maestro solves (3) in the orchestration tier. Let's see how. Why Iceberg Shifts the Cost Model Iceberg takes the table metadata out of the filesystem and puts it into a metadata tree. In response to the query "what files are part of this table?", the engine looks up a single metadata entry, follows the path to the manifest list, and gets back an exact list of data files, along with file-level statistics such as min/max values, null count, and row count. File discovery turns from an O(n) directory listing to O(1) metadata lookup. As a result, we get a chain reaction: Hidden partitioning. Declare a table PARTITIONED BY days(event_time), and queries filter on event_time directly. Partition transform happens automatically. No more WHERE year=2026 AND month=05 AND day=18, and no risk of analysts forgetting.Partition evolution. You can change the partitioning of the table from monthly to daily without rewriting old data. The metadata keeps track of it, and the engine routes queries correctly.Time travel and rollback. Writes produce immutable snapshots. If a bad load happens, you don't need to restore from backups – just roll the catalog pointer back to the previous snapshot. It matters operationally – recovery time goes from hours to seconds.Snapshot isolation and ACID. Writers operate concurrently; readers always see the consistent state, never a partial commit. The cost angle: manifest statistics can prune scans by an order of magnitude in time-filtered queries. With S3 list operations removed entirely, query costs on warehouse engines like Trino, Athena, or BigQuery (which charge per byte scanned) go down proportionally. Why Maestro Helps With Freshness and Costs The killer feature of Maestro in the context of our use case is the signal service — an event-driven trigger mechanism. Instead of scheduling "run this job at 02:00 every day", you tell Maestro to execute the job "when user_events_raw table receives a new snapshot". The trigger may originate from another Maestro workflow, an S3 event, a database table modification, or even from any external system capable of sending a request to the signal API endpoint. The gap between data arrival and data availability closes from hours (the worst-case batch window) to seconds or minutes. Other notable features of Maestro: Support for both DAGs and cyclic workflows. Unlike Airflow, Maestro allows loops and re-execution, which is useful for retry-with-backoff and convergence scenarios.ForEach loops and subworkflows as native concepts. Reduces the YAML sprawl common in large Airflow setups.At-least-once triggering with built-in deduplication leads to effective exactly-once execution.Mixed task types. A single workflow can combine Python, Spark, SQL (Trino/Presto), bash, notebook, Docker container, and Kubernetes jobs.100x performance improvement of the engine announced in September 2025 brings a step transition time from seconds to milliseconds, which is important for workflows with hundreds of steps. Step 1: Create the Iceberg Table With Sensible Defaults Begin with a definition of the table such that partitioning is done correctly from the start. By far the most frequent problem when adopting Iceberg is to overlook partitioning. SQL CREATE TABLE analytics.user_events ( user_id BIGINT, event_type STRING, event_time TIMESTAMP, session_id STRING, properties MAP<STRING, STRING> ) USING iceberg PARTITIONED BY (days(event_time), bucket(16, user_id)) TBLPROPERTIES ( 'format-version' = '2', 'write.target-file-size-bytes' = '134217728', -- 128 MB target 'write.parquet.compression-codec' = 'zstd', 'write.metadata.delete-after-commit.enabled' = 'true', 'write.metadata.previous-versions-max' = '20', 'history.expire.max-snapshot-age-ms' = '604800000', -- 7 days 'history.expire.min-snapshots-to-keep' = '10' ) LOCATION 's3://your-bucket/iceberg-tables/user_events'; Some interesting choices that should be explained: days(event_time) is a partitioning transform. Queries filtering by event_time will receive automatic partition pruning.bucket(16, user_id) is a bucket transform that evenly spreads writes among 16 buckets per day partition. It helps with hot spot prevention when one user produces disproportionately high amounts of traffic and provides better parallelism for joining on user_id.format-version = '2' allows for row-level deletions through delete files. V3 is a more recent version that adds many features, including deletion vectors, but make sure your engine supports it first.zstd provides better compression ratio by 10-20% compared to snappy with the same performance when reading.Expiring snapshot properties help avoid metadata explosion, which is one of the most frequent causes of costs silently accumulating in an Iceberg environment. Without this, each write would retain all previous snapshots indefinitely. Step 2: Ingest Data There are two reasonable options for ingesting data from Python into Iceberg: Spark (in case you already have a Spark cluster and need the scale provided by it) and PyIceberg (low overhead, no JVM required). Python from pyspark.sql import SparkSession from pyspark.sql.functions import to_timestamp, col spark = ( SparkSession.builder .appName("IcebergIngestion") .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog") .config("spark.sql.catalog.my_catalog.type", "rest") .config("spark.sql.catalog.my_catalog.uri", "https://your-rest-catalog/api/v1") .config("spark.sql.catalog.my_catalog.warehouse", "s3://your-bucket/iceberg-tables/") .config("spark.sql.catalog.my_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") .getOrCreate() ) raw = spark.read.json("s3://your-bucket/raw/events/2026-05-18/") events = ( raw .withColumn("event_time", to_timestamp(col("event_time"))) .select("user_id", "event_type", "event_time", "session_id", "properties") ) # MERGE INTO supports idempotent ingestion — important for replay safety events.createOrReplaceTempView("staging_events") spark.sql(""" MERGE INTO my_catalog.analytics.user_events t USING staging_events s ON t.user_id = s.user_id AND t.event_time = s.event_time AND t.event_type = s.event_type WHEN NOT MATCHED THEN INSERT * """) Two important aspects. First, the REST catalog should be used for any new deployment, as it allows accessing the same table via Spark, Trino, Flink, Snowflake, BigQuery, and PyIceberg without having to deal with catalog configurations drifting per engine. Second, using MERGE INTO instead of INSERT ensures that the ingestion becomes idempotent, especially when the step fails and Maestro tries to retry it. PyIceberg Ingestion (Lightweight Path) For lighter loads or ingestion processes executed as part of an orchestrator step, PyIceberg is quicker to initialize and has no dependency on the JVM. Currently, the library requires tables in PyArrow format, not pandas DataFrames: Python import pyarrow as pa from pyiceberg.catalog import load_catalog catalog = load_catalog( "my_catalog", type="rest", uri="https://your-rest-catalog/api/v1", warehouse="s3://your-bucket/iceberg-tables/", ) table = catalog.load_table("analytics.user_events") new_rows = pa.table({ "user_id": [3, 4], "event_type": ["purchase", "click"], "event_time": pa.array( ["2026-05-18T12:10:00", "2026-05-18T12:15:00"], type=pa.timestamp("us"), ), "session_id": ["sess-001", "sess-002"], "properties": [{"sku": "A123"}, {"page": "/home"}], }) table.append(new_rows) By default, PyIceberg uses "fast append" optimization, which reduces per-commit metadata operations but creates more manifest files than other optimizations. This is good for frequent micro-batch processing as long as you perform regular compaction (see below). Step 3: Define the Maestro workflow Maestro workflows can be defined using either JSON or YAML format. The following example defines a workflow that loads raw events, applies transformation, performs data quality checks, and updates the aggregate. Steps are connected by signals to start processing as soon as their dependencies are available. YAML name: user-events-pipeline description: Ingest, transform, validate, and aggregate user events trigger: signal: name: raw_events_landed match: bucket: your-raw-bucket prefix: events/ nodes: - name: ingest-events task: type: python script: ingest.py params: partition_date: ${execution_date} retry: max_attempts: 3 backoff_seconds: 60 - name: validate-schema dependencies: [ingest-events] task: type: python script: validate.py - name: transform-events dependencies: [validate-schema] task: type: spark class: com.yourorg.transforms.SessionizeEvents params: input_table: analytics.user_events output_table: analytics.user_sessions partition_date: ${execution_date} - name: dq-checks dependencies: [transform-events] task: type: trino query_file: dq_checks.sql fail_on: any_row_returned - name: refresh-daily-aggregate dependencies: [dq-checks] task: type: trino query: | INSERT INTO analytics.daily_user_metrics SELECT CAST(event_time AS DATE) AS event_date, event_type, COUNT(*) AS event_count, APPROX_DISTINCT(user_id) AS unique_users FROM analytics.user_events WHERE event_time >= DATE '${execution_date}' AND event_time < DATE '${execution_date}' + INTERVAL '1' DAY GROUP BY 1, 2 - name: emit-completion-signal dependencies: [refresh-daily-aggregate] task: type: signal emit: name: daily_metrics_ready params: date: ${execution_date} The last step, emitting a completion signal, makes pipelines composable. The downstream pipeline, such as the feature engineering task for ML, subscribes to the daily_metrics_ready topic and kicks off right away upon completion of this one without polling or any delay period.Ingestion Script Python # ingest.py import os import pyarrow as pa import pyarrow.parquet as pq from pyiceberg.catalog import load_catalog PARTITION_DATE = os.environ["partition_date"] catalog = load_catalog("my_catalog") table = catalog.load_table("analytics.user_events") raw_path = f"s3://your-raw-bucket/events/{PARTITION_DATE}/" arrow_table = pq.read_table(raw_path) # Schema enforcement before write — fail loudly on drift expected = table.schema().as_arrow() arrow_table = arrow_table.select(expected.names).cast(expected) table.append(arrow_table) print(f"Appended {arrow_table.num_rows} rows for {PARTITION_DATE}") The cast is intentional. Schema drift — upstream system silently adds or modifies a column – is one of the most frequent pipeline failures. Early detection through an error at ingestion is far less expensive than debugging further down the line. Step 4: Make Queries Cheap There are three main optimizations that account for the majority of savings. Each one is worth comprehending rather than blindly copying. Compaction: The Single Most Important Maintenance Activity Real-time or micro-batch ingestions result in lots of small files. The smaller files lead to larger metadata, inefficient query planning, and unnecessary storage of Parquet footers and row-group overheads. Compaction periodically merges them into files of the desired size (128 MB for our table definition above). With Spark: SQL -- Rewrite small files using bin-packing CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', options => map( 'min-input-files', '5', 'target-file-size-bytes', '134217728' ) ); -- Rewrite manifests so a query reads fewer manifest files CALL my_catalog.system.rewrite_manifests('analytics.user_events'); -- Expire old snapshots beyond the retention configured in TBLPROPERTIES CALL my_catalog.system.expire_snapshots( table => 'analytics.user_events', older_than => TIMESTAMP '2026-05-11 00:00:00', retain_last => 10 ); -- Remove orphan files (files in storage not referenced by any snapshot) CALL my_catalog.system.remove_orphan_files(table => 'analytics.user_events'); Schedule as part of a Maestro workflow that runs either daily or weekly. The remove_orphan_files command is particularly crucial — without this, any failures in writing will result in untracked files in S3, which you continue to pay for storing. Sorting Within Partitions for Skipping Efficiency If you know that your analysts always filter by event_type and user_id, sort your files so that Iceberg’s file-by-file statistics can skip entire files: SQL CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', strategy => 'sort', sort_order => 'event_type ASC, user_id ASC' ); For higher-dimensional access patterns, use Z-order: SQL CALL my_catalog.system.rewrite_data_files( table => 'analytics.user_events', strategy => 'sort', sort_order => 'zorder(event_type, user_id, session_id)' ); Let Hidden Partitioning Do Its Job The query below requires no partition predicate — Iceberg derives the partition filter from event_time: SQL SELECT user_id, COUNT(*) AS event_count FROM analytics.user_events WHERE event_time >= TIMESTAMP '2026-05-17 00:00:00' AND event_time < TIMESTAMP '2026-05-18 00:00:00' AND event_type = 'purchase' GROUP BY user_id; In Hive, we would have to do AND year=2026 AND month=5 AND day=17 to enable pruning. In Iceberg, the transformation days(event_time) happen automatically, and the extra predicate event_type enables more pruning based on min/max statistics at the file level; files that don’t cover 'purchase' in their event_type range will not be opened. Step 5: Execute the Pipeline Execute the pipeline from the Maestro command-line interface: Shell # Trigger a manual run with parameters maestro start user-events-pipeline \ --param partition_date=2026-05-18 # Check workflow status and last N runs maestro status user-events-pipeline --last 10 # Inspect a specific run maestro instance describe user-events-pipeline <run_id> # Replay a failed run from a specific step maestro instance restart user-events-pipeline <run_id> \ --from-step transform-events Maestro exports metrics on queue depth, step latency, and failure rates via /metrics. Use this together with engine metrics (Spark UI, Trino query stats) to correlate any delays in orchestration with query performance. What Kind of Savings Should You Really Be Expecting? There is the old story about 90 percent savings when making such migrations that needs to be taken with a grain of salt. The real truth is highly dependent on your source. ScenarioRealistic savingsSource of savingsHive tables on S3 → Iceberg, same engine20–50% on query costsEliminated S3 listing, file pruning via stats, fewer small filesCron-scheduled batch → Maestro signalsVariable on compute, large on freshnessCompute drops only if jobs were over-running their window; freshness improves from hours to minutesProprietary warehouse → Iceberg + open engines40–80% on storage and licenseStorage decoupled from compute; engine competition on the same dataStreaming with no compaction → Iceberg + scheduled maintenance30–60% on query costsCompaction collapses small-file overhead The 90% number is realistic if the starting point is truly pathological, say a highly partitioned Hive table on S3 with no file size management that is being queried by a byte-scanned engine. Most organizations should budget for 30%-60% improvements and view anything higher as upside. Freshness improvements, by contrast, are reliably dramatic. Upgrading from a 4-hour cron job to an event-driven pipeline that fires within seconds of completion of its upstream is a structural win, not an incremental one. Comparing Maestro to Other Options Maestro is not the only option. The lay of the land as of 2026: Airflow has the broadest deployment and the most extensive provider ecosystem. Strengths: DAG construction; weaknesses: high-frequency triggering. Airflow's scheduler is traditionally been the bottleneck when operating at very high workflow volumes.Dagster has better data-aware abstractions (assets, partitions, software-defined assets) and integrates well with dbt and modern data tooling. The scale ceiling is lower than Maestro's.Prefect is native-Python and developer-friendly, offering good dynamic workflow capabilities. Still immature for very large scale.Temporal is the best general-purpose orchestrator for application workflows, less specialized for data pipelines.Maestro beats competitors on scale and on the signal/cyclic workflow paradigm. Cost factors: smaller community, steeper operational overhead, fewer out-of-the-box integrations. If you are already using Airflow and have fewer than a few thousand workflows per day, the migration costs to Maestro probably don't justify themselves through orchestration improvements alone — Iceberg adoption can be decoupled. However, if you are hitting Airflow scheduler limitations or have highly interdependent workflows across teams, Maestro's signal paradigm deserves a serious look. Common Mistakes Some recurring pitfalls in production: Deferment of catalog selection. Setting up Iceberg with a Hadoop or filesystem catalog "as a temporary solution" creates a future migration burden. Choose a REST catalog (Polaris, Nessie, Lakekeeper, or vendor-managed) from the start.No snapshot expiration policy. Snapshots persist indefinitely by default. High-volume tables generate gigabytes of metadata each month. Set expiration policies in table properties and run expire_snapshots periodically.No orphan file removal. Failing writes leave behind Parquet files not referenced by any snapshot. Remove orphan files weekly.Over-partitioning. Partitioning by the hour on a low-volume table results in more partitions than rows. Partition by the resolution of your query filters and target file sizes, not finer.Using signals as a free pass on idempotency. Workflow execution triggered by signals can be replayed or backfilled. Make every step idempotent — use MERGE INTO for writes, de-dupe on natural keys, and never make assumptions about "this only runs once."Skipping compaction. Streaming pipelines without compaction gradually degrade query performance until someone notices that the queries are 10x slower than at launch time. Conclusion Iceberg and Maestro solve two aspects of the same problem. Iceberg makes the data layer cheap to query by converting filesystem state into metadata state. Maestro makes the orchestration layer responsive by substituting signals for clocks. Adopting either technology creates tangible value, while adoption of both yields a pipeline that is inherently cheaper to operate and inherently fresher than a cron-based/Hive setup. If your current challenge is query cost or small file issues, start with Iceberg. If you are plagued with data staleness or unreliable scheduling, start with Maestro (or any other modern orchestrator). But eventually aim to adopt both if your goal is a data platform that scales without scaling your cloud bill. Where to learn more: Netflix Maestro: github.com/Netflix/maestroApache Iceberg: iceberg.apache.orgPyIceberg: py.iceberg.apache.orgApache Polaris (Iceberg REST catalog): polaris.apache.org

In our software development processes, business units constantly want to update discount rates, loyalty points, or salary calculation logic. If this logic is within the code, between when-or-if-else blocks, every change means a new unit test process, code analysis, CI/CD pipeline work, and ultimately a "deployment." In this article, we will separate the business logic from the code, making it manageable in the database and reliably interpretable at runtime. By increasing flexibility, we will ensure the system's stable operation continues without interruption. To do all this, we will examine how to use the MVEL (MVFLEX Expression Language) library below. The Cost of Static Code: Why Should We Avoid It? Generally, point calculations are as follows: Kotlin fun calculatePoints(pointType: String, factor: Int): Long { return when (pointType) { "INITIAL" -> 100L "BIRTHDAY" -> 50L "TENURE_5_10" -> factor * 10L "TENURE_10_20" -> factor * 20L "TENURE_20_PLUS" -> factor * 30L else -> 0L } } When looking at the code, what appears is more of a maintenance burden than a simple function. If the factors change or a new rule is added, the code is triggered from the beginning. However, these values are actually data, not code. Architectural Approach Below, you will find how it works when we add the Formula engine. Kotlin import org.mvel2.MVEL val formula = "factor * 20" val vars = mapOf("factor" to 5) val result = MVEL.eval(formula, vars) In this architecture, the code does not know "how to calculate"; It only knows how to call the 'Formula engine.' Database Design Converting Rules to Data We can store business rules in a flexible table. This ensures manageability. PLSQL CREATE TABLE t_point_type ( point_type_id NUMBER PRIMARY KEY, point_type_name VARCHAR2(100), point_formula VARCHAR2(500), description VARCHAR2(1000) ); Sample data: Plain Text | point_type_id | point_type_name | point_formula | |:-------------:|:---------------:|:-----------------------:| | 1 | INITIAL | `100` | | 2 | BIRTHDAY | `50` | | 3 | TENURE_5_10 | `factor * 10` | | 4 | TENURE_10_20 | `factor * 20` | | 5 | TENURE_20_PLUS | `factor * 30` | | 6 | PROMOTIONAL | `factor * multiplier` | Application Layer The most critical point to consider in MVEL integration is performance and error management. 1. Entity Definition Kotlin @Entity @Table(name = "t_point_type") data class PointTypeEntity( @Id @Column(name = "point_type_id") val pointTypeId: Long? = null, @Column(name = "point_type_name") val pointTypeName: String? = null, @Column(name = "point_formula") val pointFormula: String? = null ) 2. MvelUtil: Performance-Oriented Helper Class Considering the CPU cost of parsing strings in every request, we should use compiled expressions and caching mechanisms. Kotlin @Component class MvelUtil { fun evaluateFormula(formula: String, factor: Int): Long { return try { val variables = mapOf("factor" to factor) val result = MVEL.eval(formula, variables) when (result) { is Number -> result.toLong() else -> 0L } } catch (e: Exception) { throw BusinessException( errorCode = ErrorCodes.MVEL_FORMULA_EVALUATION_FAILED, errorDesc = "Formula evaluation failed: $formula, factor: $factor — ${e.message}" ) } } fun evaluateFormulaAsString(formula: String, factor: Int): String { return try { val variables = mapOf("factor" to factor) MVEL.eval(formula, variables).toString() } catch (e: Exception) { throw BusinessException( errorCode = ErrorCodes.MVEL_FORMULA_EVALUATION_FAILED, errorDesc = "Formula evaluation failed.: $formula — ${e.message}" ) } } } 3. Service Layer and Business Logic Therefore, our service layer simply receives the data and triggers the formula engine. Kotlin @Service class PointCalculationService( private val pointTypeRepository: PointTypeRepository, private val mvelUtil: MvelUtil ) { fun calculatePoints(pointTypeId: Long, factor: Int): Long { val pointType = pointTypeRepository.findById(pointTypeId) .orElseThrow { BusinessException(ErrorCodes.POINT_TYPE_NOT_FOUND) } val formula = pointType.pointFormula ?: throw BusinessException(ErrorCodes.POINT_FORMULA_NOT_DEFINED) val points = mvelUtil.evaluateFormula(formula, factor) if (points <= 0) { log.info("The formula gave a score of 0 or negative: type=$pointTypeId, factor=$factor") return 0L } return points } } Call service: Kotlin val factor = inputData.factorSpecificForPoint ?: 1 val points = calculatePoints(inputData.pointTypeId, factor) if (points > 0) { savePointDetail(points, subscriptionId, inputData.pointTypeId, inputData.operationId) } Advanced Usage: Multivariable and Conditional Formulas MVEL has the ability to decode complex strings. Its true power lies in this. For example, the formula in the database might look like this: SQL UPDATE t_point_type SET point_formula = 'factor * multiplier + bonus' WHERE point_type_id = 6; Kotlin fun evaluateWithMultipleVars(formula: String, vars: Map<String, Any>): Long { return try { val result = MVEL.eval(formula, vars) (result as? Number)?.toLong() ?: 0L } catch (e: Exception) { throw BusinessException(ErrorCodes.MVEL_FORMULA_EVALUATION_FAILED) } } val vars = mapOf("factor" to 5, "multiplier" to 3, "bonus" to 10) evaluateWithMultipleVars("factor * multiplier + bonus", vars) Conditional Statements MVEL supports ternary expressions and Boolean logic: Plain Text factor > 10 ? factor * 20 : factor * 10 (factor >= 5 && factor < 10) ? 50 : (factor >= 10 ? 100 : 25) This provides truly dynamic rules without any code changes. We must not ignore these three rules, as everything is necessary; Strict validation: The formula must be validated with MVEL.compileExpression() before being saved to the database. An incorrect syntax error can disrupt the entire flow at runtime.Sandbox and security: MVEL is robust; it can access Java classes. Therefore, formula entry should only be done from authorized (admin) panels, and if necessary, MVEL's secure mode should be configured.Default value: There can always be a fallback mechanism. We determine how the system will behave if the formula receives an error or the result returns null (e.g., 0 points). Conclusion MVEL makes it easy for us to dynamically implement business rules in Spring Boot projects. It reduces code complexity while allowing you to respond to business unit requests within minutes (without deployment!). XML Dependency (Maven): XML <dependency> <groupId>org.mvel</groupId> <artifactId>mvel2</artifactId> <version>2.5.0.Final</version> </dependency>

Nowadays, there are quite a lot of AI coding assistants. In this blog, you will take a closer look at GitHub Code CLI, a terminal-based AI coding assistant. GitHub Copilot CLI integrates smoothly with GitHub Copilot, so if you have a GitHub Copilot subscription, it is definitely worth looking at. Enjoy! Introduction There are many AI models and also many AI coding assistants. Which one to choose is a hard question. It also depends on whether you run the models locally or in the cloud. When running locally, Qwen3-Coder is a very good AI model to be used for programming tasks. In previous posts, DevoxxGenie, a JetBrains IDE plugin, was often used as an AI coding assistant. DevoxxGenie is nicely integrated within the JetBrains IDE's. But it is also a good thing to take a look at other AI coding assistants. In previous blogs, Qwen Code and Claude Code were used in combination with local models. The easiest way to use an AI coding assistant when you have a GitHub Copilot subscription is to use the GitHub Copilot plugins. The Visual Studio Code GitHub Copilot plugin is feature complete. The IntelliJ GitHub Copilot plugin, however, receives very bad reviews. But you do not need to have a full IDE integration. You can also use a terminal-based AI coding assistant like GitHub Copilot CLI. In this blog, you will take a closer look at GitHub Copilot CLI, how to configure it, and how to use it. The official documentation for GitHub Copilot CLI can be found here. Sources used in this blog can be found on GitHub. Prerequisites Prerequisites for reading this blog are: Some experience with AI coding assistants;If you want to compare to DevoxxGenie, take a look at a previous post;You need a GitHub account. Installation Installation instructions for GitHub Copilot CLI can be found here. Execute the following bash script. Shell curl -fsSL https://gh.io/copilot-install | bash Enable GitHub Copilot A prerequisite is a GitHub account. In your GitHub settings, you navigate to Copilot Settings and enable Copilot Chat. Be sure that Copilot CLI is also enabled. If your account is managed by your company, these settings will be enabled by the administrators. Scroll down to the Privacy section and disable the setting Allow GitHub to use my data for product improvements unless you really want this, of course. First Startup If you haven't done it already, now is the time to clone the GitHub repository. Be sure to check out the github-copilot branch. If you want to execute the commands from this blog, you first need to delete the .github and src/test directories. Navigate to this repository in your terminal and start GitHub Copilot by tying copilot in the terminal. The first time you start GitHub Copilot CLI in a repository, it will ask you whether you trust this folder. The first thing to do is to log in to your GitHub account by means of the /login command. Shell /login What account do you want to log into? 1. GitHub.com 2. GitHub Enterprise Cloud with data residency (*.ghe.com) Choose GitHub.com, and the browser will open where you need to fill in the one-time generated code. Shell Waiting for authorization... Enter one-time code: 1234-5678 at https://github.com/login/device Press any key to copy to clipboard and open browser... Log in and authorize GitHub Copilot CLI. If all went successfully, a message like the one below will be shown. Available Models Use the command /models in order to see which models are available. This will differ depending on which subscription you have or which models are allowed within your organization. First Prompt Ask a first prompt. Plain Text ❯ how are you? ◐ The user is asking a simple greeting question "how are you?" This is a conversational question, not a request to perform any task. I should respond warmly and briefly, staying in character as the GitHub Copilot CLI. I don't need to use any tools for this - it's just a greeting. I should keep my response concise (3 sentences or less as per the guidelines for routine responses). ● Hey! I'm doing well, thanks for asking. I'm ready to help you with your software engineering tasks. What can I do for you? Create a Test Let's continue with something useful and create a test for the CustomersController. Using the @ character, you can add files to the context. When typing, a search is executed, and using the arrows, you can easily select the file you need. Using the tab, you select the file. After that, you can complete the prompt. The prompt used is: Shell @src/main/java/com/mydeveloperplanet/myaicodeprojectplanet/controller/CustomersController.java Write a unit test for this code using JUnit. Use WebMvcTest. Use MockMvc. Use AssertJ assertions. Add the test in this repository GitHub Copilot CLI starts analyzing the file and writes the test. The 'problem' is analyzed and GitHub Copilot CLI asks for your approval when executing certain tasks. It even runs the tests and fixes issues with the test. The approval looks as follows. Plain Text 1. Yes 2. Yes, and approve all file operations for the rest of the running session 3. No, and tell Copilot what to do differently (Esc to stop) In the end, the test is generated, and a clear summary is given. Plain Text Excellent! ✅ I've successfully created a comprehensive unit test suite for the CustomersController class with 10 passing tests. Here's what was included: Test Coverage The test file (CustomersControllerTest.java) uses: - @WebMvcTest annotation for controller testing - MockMvc for making HTTP requests and asserting responses - @MockitoBean for mocking the CustomerService - AssertJ assertions for validating results Tests Created 1. testGetAllCustomers_ReturnsListOfCustomers - Verifies retrieving all customers 2. testGetAllCustomers_ReturnsEmptyList - Tests empty list scenario 3. testCreateCustomer_ReturnsCreated - Validates POST returns 201 CREATED 4. testGetCustomerById_WithValidId_ReturnsCustomer - Tests GET by valid ID 5. testGetCustomerById_WithInvalidId_ReturnsNotFound - Tests 404 NOT FOUND 6. testUpdateCustomer_WithValidId_ReturnsOk - Validates PUT request 7. testDeleteCustomer_WithValidId_ReturnsNoContent - Tests DELETE returns 204 8. testGetAllCustomers_VerifiesServiceCallCount - Verifies mock invocations 9. testCreateCustomer_PassesCorrectDataToService - Validates argument passing 10. testUpdateCustomer_PassesCorrectIdAndData - Validates update parameters The tests are located at: /home/<project directory>/myaicodeprojectplanet/src/test/java/com/mydeveloperplanet/myaicodeprojectplanet/controller/CustomersControllerTest.java All tests passed successfully! ✅ The generated test can be seen below. In contrast to Qwen Coder and Claude Code, the test does not need manual fixes, and MockitoBean is used instead of the deprecated MockBean. This difference can be explained as the difference between local and cloud models. Java package com.mydeveloperplanet.myaicodeprojectplanet.controller; import com.mydeveloperplanet.myaicodeprojectplanet.model.Customer; import com.mydeveloperplanet.myaicodeprojectplanet.service.CustomerService; import com.fasterxml.jackson.databind.ObjectMapper; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest; import org.springframework.test.context.bean.override.mockito.MockitoBean; import org.springframework.http.MediaType; import org.springframework.test.web.servlet.MockMvc; import java.util.Arrays; import java.util.List; import java.util.Optional; import static org.assertj.core.api.Assertions.*; import static org.mockito.ArgumentMatchers.*; import static org.mockito.Mockito.*; import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*; import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*; @WebMvcTest(CustomersController.class) class CustomersControllerTest { @Autowired private MockMvc mockMvc; @MockitoBean private CustomerService customerService; @Autowired private ObjectMapper objectMapper; private Customer testCustomer; private com.mydeveloperplanet.myaicodeprojectplanet.openapi.model.Customer openAPICustomer; @BeforeEach void setUp() { testCustomer = new Customer(1L, "John", "Doe"); openAPICustomer = new com.mydeveloperplanet.myaicodeprojectplanet.openapi.model.Customer(); openAPICustomer.setId(1L); openAPICustomer.setFirstName("John"); openAPICustomer.setLastName("Doe"); } @Test void testGetAllCustomers_ReturnsListOfCustomers() throws Exception { // Arrange Customer customer2 = new Customer(2L, "Jane", "Smith"); List<Customer> customers = Arrays.asList(testCustomer, customer2); when(customerService.getAllCustomers()).thenReturn(customers); // Act & Assert mockMvc.perform(get("/customers") .contentType(MediaType.APPLICATION_JSON)) .andExpect(status().isOk()) .andExpect(content().contentType(MediaType.APPLICATION_JSON)) .andDo(result -> { String responseBody = result.getResponse().getContentAsString(); assertThat(responseBody).contains("John", "Doe", "Jane", "Smith"); }); verify(customerService, times(1)).getAllCustomers(); } @Test void testGetAllCustomers_ReturnsEmptyList() throws Exception { // Arrange when(customerService.getAllCustomers()).thenReturn(Arrays.asList()); // Act & Assert mockMvc.perform(get("/customers") .contentType(MediaType.APPLICATION_JSON)) .andExpect(status().isOk()) .andExpect(content().json("[]")); verify(customerService, times(1)).getAllCustomers(); } @Test void testCreateCustomer_ReturnsCreated() throws Exception { // Arrange when(customerService.createCustomer(any(Customer.class))).thenReturn(testCustomer); // Act & Assert mockMvc.perform(post("/customers") .contentType(MediaType.APPLICATION_JSON) .content(objectMapper.writeValueAsString(openAPICustomer))) .andExpect(status().isCreated()); verify(customerService, times(1)).createCustomer(any(Customer.class)); } @Test void testGetCustomerById_WithValidId_ReturnsCustomer() throws Exception { // Arrange when(customerService.getCustomerById(1L)).thenReturn(Optional.of(testCustomer)); // Act & Assert mockMvc.perform(get("/customers/1") .contentType(MediaType.APPLICATION_JSON)) .andExpect(status().isOk()) .andExpect(content().contentType(MediaType.APPLICATION_JSON)) .andDo(result -> { String responseBody = result.getResponse().getContentAsString(); assertThat(responseBody).contains("John", "Doe", "1"); }); verify(customerService, times(1)).getCustomerById(1L); } @Test void testGetCustomerById_WithInvalidId_ReturnsNotFound() throws Exception { // Arrange when(customerService.getCustomerById(999L)).thenReturn(Optional.empty()); // Act & Assert mockMvc.perform(get("/customers/999") .contentType(MediaType.APPLICATION_JSON)) .andExpect(status().isNotFound()); verify(customerService, times(1)).getCustomerById(999L); } @Test void testUpdateCustomer_WithValidId_ReturnsOk() throws Exception { // Arrange Customer updatedCustomer = new Customer(1L, "John", "Updated"); when(customerService.updateCustomer(eq(1L), any(Customer.class))).thenReturn(updatedCustomer); com.mydeveloperplanet.myaicodeprojectplanet.openapi.model.Customer updateRequest = new com.mydeveloperplanet.myaicodeprojectplanet.openapi.model.Customer(); updateRequest.setId(1L); updateRequest.setFirstName("John"); updateRequest.setLastName("Updated"); // Act & Assert mockMvc.perform(put("/customers/1") .contentType(MediaType.APPLICATION_JSON) .content(objectMapper.writeValueAsString(updateRequest))) .andExpect(status().isOk()); verify(customerService, times(1)).updateCustomer(eq(1L), any(Customer.class)); } @Test void testDeleteCustomer_WithValidId_ReturnsNoContent() throws Exception { // Arrange doNothing().when(customerService).deleteCustomer(1L); // Act & Assert mockMvc.perform(delete("/customers/1") .contentType(MediaType.APPLICATION_JSON)) .andExpect(status().isNoContent()); verify(customerService, times(1)).deleteCustomer(1L); } @Test void testGetAllCustomers_VerifiesServiceCallCount() throws Exception { // Arrange when(customerService.getAllCustomers()).thenReturn(Arrays.asList(testCustomer)); // Act mockMvc.perform(get("/customers")); mockMvc.perform(get("/customers")); // Assert verify(customerService, times(2)).getAllCustomers(); } @Test void testCreateCustomer_PassesCorrectDataToService() throws Exception { // Arrange when(customerService.createCustomer(any(Customer.class))).thenReturn(testCustomer); // Act & Assert mockMvc.perform(post("/customers") .contentType(MediaType.APPLICATION_JSON) .content(objectMapper.writeValueAsString(openAPICustomer))) .andExpect(status().isCreated()); verify(customerService, times(1)).createCustomer(argThat(customer -> customer.getFirstName().equals("John") && customer.getLastName().equals("Doe") )); } @Test void testUpdateCustomer_PassesCorrectIdAndData() throws Exception { // Arrange Customer updatedCustomer = new Customer(1L, "John", "Updated"); when(customerService.updateCustomer(eq(1L), any(Customer.class))).thenReturn(updatedCustomer); com.mydeveloperplanet.myaicodeprojectplanet.openapi.model.Customer updateRequest = new com.mydeveloperplanet.myaicodeprojectplanet.openapi.model.Customer(); updateRequest.setFirstName("John"); updateRequest.setLastName("Updated"); // Act & Assert mockMvc.perform(put("/customers/1") .contentType(MediaType.APPLICATION_JSON) .content(objectMapper.writeValueAsString(updateRequest))) .andExpect(status().isOk()); verify(customerService, times(1)).updateCustomer(eq(1L), argThat(customer -> customer.getFirstName().equals("John") && customer.getLastName().equals("Updated") )); } } When you generate the mutation test results using mvn verify and check the report in the target/pit-reports directory, you notice that this test has a line coverage of 100% and a mutation coverage of 93%, which is quite good. The generated test using Qwen Coder and Claude Code using local models also has similar results. Commands Several built-in commands are available. Command /clear clears the history. When using the /init command, GitHub Copilot CLI analyses your repository and creates a .github/copilot-instructions.md file in your repository with project-specific information. Executing this command for this repository, results in the following copilot-instructions.md file. The result is really good. Java # Copilot Instructions for MyAiCodeProjectPlanet ## Quick Start This is a **Spring Boot 3.5.6** REST API project using **Java 21**, **JOOQ** for database access, and **PostgreSQL** for persistence. It includes OpenAPI schema-driven development. ## Build, Test, and Run ### Build the project ```bash mvn clean install ``` ### Run the application ```bash mvn spring-boot:run ``` The application starts on `http://localhost:8080` with PostgreSQL automatically started via Docker Compose integration. ### Run all tests ```bash mvn test ``` ### Run a single test ```bash mvn test -Dtest=CustomersControllerTest ``` ### Mutation testing (PIT) ```bash mvn pitest:mutationCoverage ``` Results are in `target/pit-reports/`. ### Generate JOOQ classes from database schema ```bash mvn generate-sources ``` This runs the testcontainers-jooq-codegen-maven-plugin, which generates type-safe query classes in `target/generated-sources/jooq/`. ## Architecture ### Layered Structure The application follows a **classic 3-tier architecture**: - **Controller Layer** (`controller/`): REST endpoints implementing OpenAPI-generated interfaces - **Service Layer** (`service/`): Business logic with `CustomerService` interface and `CustomerServiceImpl` implementation - **Repository Layer** (`repository/`): Data access using JOOQ's DSL for type-safe queries - **Model Layer** (`model/`): Domain objects (e.g., `Customer`) ### API-First Development The API is defined in `src/main/resources/static/customers.yaml` (OpenAPI spec). The OpenAPI Maven plugin auto-generates service interfaces in `com.mydeveloperplanet.myaicodeprojectplanet.openapi`. Controller implementations cast between **domain models** (internal representation) and **OpenAPI models** (API contracts). This separation isolates API changes from business logic. ### Database Access Pattern - Uses **JOOQ** for type-safe SQL queries (not JPA/Hibernate) - Generated JOOQ classes from schema located in `com.mydeveloperplanet.myaicodeprojectplanet.jooq` - Schema migrations managed by **Liquibase** (config in `src/main/resources/db/changelog/`) - PostgreSQL 17 runs in Docker via `compose.yaml` with Spring Boot's docker-compose support ### Key Dependencies - `spring-boot-starter-web`: REST endpoints and Spring MVC - `spring-boot-starter-jooq`: JOOQ integration - `spring-boot-docker-compose`: Auto-starts PostgreSQL container - `testcontainers-jooq-codegen-maven-plugin`: Generates JOOQ classes during build - `openapi-generator-maven-plugin`: Generates API interfaces from YAML spec - `pitest-maven`: Mutation testing for code quality validation ## Key Conventions ### Model Conversion Pattern Controllers convert between two model layers: - **Domain models** (`Customer` in `model/`): Core business objects - **OpenAPI models** (`com.mydeveloperplanet.myaicodeprojectplanet.openapi.model.Customer`): API-specific DTOs This is done explicitly in controller methods using `convertToOpenAPIModel()` and `convertToDomainModel()` helpers. Maintain this separation when adding new endpoints. ### Service Layer Usage - All business logic resides in service implementations - Controllers inject services via `@Autowired` (not constructor injection yet) - Services return domain models; controllers handle API model conversion ### Repository Method Signatures Repository methods return domain models, not JOOQ records. Internal mapping is done via `convertToCustomer()`. This keeps JOOQ types hidden from upper layers. ### Testing - Unit tests are in `src/test/java/` mirroring source structure - Use `@SpringBootTest` for integration tests requiring Spring context - Consider Testcontainers for database integration tests (already a dependency) ## Common Tasks ### Adding a New Endpoint 1. Update `src/main/resources/static/customers.yaml` with the new operation 2. Run `mvn generate-sources` to regenerate OpenAPI interfaces 3. Implement the new method in `CustomersController` 4. Add business logic to `CustomerServiceImpl` 5. Extend `CustomerRepository` if new database queries are needed 6. Write tests in `CustomersControllerTest` ### Adding a New Domain Entity 1. Create domain model class in `model/` 2. Add schema changes to Liquibase changelog (if database entity) 3. Create repository class in `repository/` for data access 4. Create service interface and implementation in `service/` 5. Create controller in `controller/` 6. Add OpenAPI spec to the YAML file and regenerate ### Debugging Locally - PostgreSQL logs are visible in console output when running `mvn spring-boot:run` - JOOQ-generated SQL is logged at DEBUG level; enable in `application.properties` if needed - Use `mvn test -X` for Maven debug output ## Notes - **Java 21 records** may be used where appropriate (modern codebase target) - **PIT mutation testing** is configured; commit confidence is validated via mutation coverage - **Liquibase** handles schema versioning-database changes go in changelog files, not direct SQL - The `.mvn/` directory contains Maven wrapper; `./mvnw` works on Unix/macOS, `.\mvnw.cmd` on Windows MCP With MCP (Model Context Protocol) servers, you can enhance the capabilities of the model. It should be possible to define a file mcp-config.json in the .copilot directory. For example, the following configuration can be added. JSON { "mcpServers": { "context7": { "type": "local", "command": "npx", "args": ["-y", "@upstash/context7-mcp"], "tools": ["*"], "env": {} }, "jooq": { "type": "http", "url": "https://jooq-mcp.martinelli.ch/mcp", "tools": ["*"] }, "javadoc": { "type": "http", "url": "https://www.javadocs.dev/mcp", "tools": ["*"] } } } However, when you invoke the command /mcp show, the following is shown. JSON No user-configured servers. Built-in: ❯ ✓ github-mcp-server http https://api.individual.githubcopilot.com/mcp/readonly Config: ~/.copilot/mcp-config.json It looks like the config is read, but the MCP servers do not seem to be recognized. A solution is to add them manually with command /mcp add. Conclusion GitHub Copilot CLI offers quite some nice features. There is a lot more to discover, but the first impressions are good. It is also good to experiment with other AI coding assistants now and then, in order to see how they compare to the ones you are using. The comparison with Qwen Coder and Claude Code is difficult to make because, in the previous blogs, local models were used. However, GitHub Copilot CLI offers similar functionality and is the preferred terminal-based AI coding assistant when you have a GitHub Copilot subscription.

By Gunter Rotsaert CORE