Software Design and Architecture

The rise of autonomous AI agents within business software demands a fresh approach to security. Unlike earlier chatbot tools, modern agents act with real privileges, such as updating databases, calling microservices, composing and even executing code, or triggering workflows on their own. This shift expands the blast radius of any flaw or compromise. As one Microsoft analysis observes, today’s AI agents “can update database records, trigger enterprise workflows, access sensitive data, and interact with production systems all autonomously.” In practice, that means a mistake or exploit can have immediate operational impact instead of just a reputational cost. With agents in the loop, input manipulation becomes especially dangerous. Prompt-injection attacks let adversaries commandeer an AI by feeding it malicious instructions in user inputs or hidden in external data. A carefully crafted prompt or document can cause an agent to reveal secrets or perform harmful actions. These manipulations can be direct (an attacker’s text overriding the agent’s instructions) or indirect (for example, hidden commands embedded in HTML or metadata that the agent ingests). By definition, even inputs imperceptible to humans can subvert the model, forcing it to break safety rules. In effect, prompt injection can trick an AI into disclosing internal prompts, executing arbitrary commands, or making unauthorized changes. Protect AI Agents With Layered Security Controls Defending against prompt injection requires layered controls. It is not enough to trust the LLM’s built-in safeguards. The application must sanitize and constrain every input. For example, the OWASP GenAI guidelines recommend semantic filtering of user inputs and strict output validation. In practice, this often means cleaning or escaping suspicious tokens in the prompt, enforcing clear response schemas, and even tagging or quarantining untrusted data before it reaches the model. Developers should also build resilience into AI calls, for example by wrapping each agent invocation in a circuit-breaker or retry mechanism so that anomalous behavior triggers a safe fallback rather than a cascade of errors. Java @CircuitBreaker(name="agentService", fallbackMethod="fallbackAgent") @Retry(name="agentService", maxAttempts=3, backoff=@Backoff(delay=200)) public String executeAgentTask(String taskId, String input) { String safeInput = inputFilter.sanitize(input); return agentClient.postForObject("/tasks/" + taskId, safeInput, String.class); } private String fallbackAgent(String taskId, String input, Exception ex) { log.error("Agent {} failed: {}", taskId, ex.getMessage()); return "error"; } In this example, inputFilter.sanitize strips any suspicious content from the prompt, and the circuit-breaker ensures repeated failures lead to a controlled fallback. The fallbackAgent method logs the failure and returns a safe default response, preventing a hijacked prompt from causing uncontrolled retries or side effects. Embedding such patterns helps contain injected instructions and makes anomalies visible for audit. Agents also expand supply-chain and data-poisoning attack surfaces. AI applications often depend on third-party models, libraries, or datasets, each of which could harbor backdoors. In a real incident, attackers compromised an open-source Python package used in a model’s pipeline, effectively inserting malicious logic into every system that imported it. To guard against this, organizations must treat AI dependencies as critically as any library or service. Models and data should come from verifiable, signed sources, and teams should maintain an AI-focused Software Bill of Materials (SBOM) tracking each model and dataset. Regular scans of model files and packages (for example, by verifying cryptographic hashes or digital signatures) can detect tampering before models reach production. Another insidious vector is agent memory poisoning. Unlike stateless microservices, AI agents may accumulate knowledge across sessions or tasks. If an adversary can insert malicious “memories” or biased information into that knowledge base, the agent may repeat or amplify harmful logic over time. Researchers have shown that injecting only a few hundred carefully crafted documents into a training or retrieval database can reliably hijack a model’s outputs in specific domains. In an enterprise, this might translate to a support chatbot that starts rejecting valid requests or approving fraudulent transactions because its knowledge was skewed. Mitigations include thoroughly vetting any external data fed to the agent, cross-checking facts against trusted sources, and periodically resetting or auditing the agent’s internal state. For example, a system could clear an agent’s “short-term memory” after each sensitive transaction, or require digital signatures on any new knowledge items. Protect AI Agents With Layered Security Controls Identity and access control for agents is equally critical. Agents act as non-human service identities, so if an attacker steals an agent’s credentials, they essentially hijack its privileges. Recorded Future warns that “compromised credentials, SSO platforms, or agent identities could enable large-scale... data exfiltration”. In practice, a stolen token could let an attacker quietly siphon data or trigger commands anywhere the agent has access. To counter this, enterprises should issue each agent a unique short-lived token and restrict its scope strictly. Java String token = credentialService.issueShortLivedToken(agentId); apiClient.setAuthToken(token); apiClient.callExternalService(requestPayload); Here, each API call by the agent uses a fresh, scoped token. If the token is leaked or abused, its very short life and limited permissions contain the damage. In practice, agent tokens should be rotated frequently, and every action should be logged under the agent’s identity. If an agent suddenly tries to access an unexpected endpoint, automated policies should block or flag the request. In essence, treat agents like privileged users with their own IAM lifecycle by implementing least-privilege roles, multi-factor approvals for high-value operations, and full auditing of their activities. Multi-agent workflows introduce additional complexity. Agents often invoke other tools or orchestrate chains of sub-agents. In such pipelines, a compromise anywhere can cascade. For example, if Agent A trusts a data input or command from Agent B, and B has been misled or maliciously tampered with, A may unknowingly act on bad instructions. To mitigate this, every handoff between agents or tools should be authenticated and checked. Enforce endpoint authentication and message signing on each channel between agents, and apply authorization checks at every step. Segmentation and strong encryption on inter-agent communications can prevent a breach in one component from jumping to others. Monitor AI Agents for Anomalies and Unauthorized Actions At runtime, anomaly detection and monitoring provide a final safety net. Agents in production should exhibit well-defined baselines of behavior. An agent that usually looks up customer records, for instance, should not suddenly be streaming large volumes of payroll data. Security telemetry that logs every prompt, response, and tool invocation lets defenders spot when an agent deviates from its norm. Modern SIEM and AIOps platforms can ingest these logs and flag unusual patterns (for example, spikes in outbound data or unexpected API calls). By correlating agent activity with traditional logs and threat intelligence, teams can detect and contain a misbehaving agent before it causes systemic damage. In summary, securing enterprise applications in the agent era means integrating AI-specific defenses throughout the stack. Zero-Trust principles apply fully where we treat each agent call as untrusted until verified, grant agents only minimal permissions, and require human approval for any high-impact decision. Defense-in-depth remains essential as it sanitizes every input, isolates AI subsystems from sensitive resources, and monitors all outputs continuously. Industry standards are beginning to catch up. For example, NIST’s new AI Risk Management Framework and the Cloud Security Alliance’s guidelines explicitly recommend continuous threat modeling, red teaming of AI, and traceability for data and models. Ultimately, the agentic AI era raises the security stakes from theory into daily practice. Organizations that build AI-aware threat modeling, least-privilege IAM, prompt filtering, and anomaly monitoring into their DevSecOps pipelines will be best equipped to embrace AI agents safely. By doing the hard work now by integrating model security into the software lifecycle, enterprises can unlock the productivity of agents while keeping adversaries at bay.

“...premature optimization is the root of all evil…” Donald Ervin Knuth Introduction "Premature optimization is the root of all evil." Most software engineers know this, attributed to Donald Knuth, author of The Art of Computer Programming and one of the most influential figures in computer science. Many have also picked up the practical conclusion that followed: "let's make it work first, fix performance later." After all, it's easier to add another EC2 instance than to find the root cause. But here is what Knuth actually wrote: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." A little different, isn't it? The second sentence is almost never quoted — and that is convenient, because it turns a careful statement into a simple excuse. Sometimes for laziness. Sometimes because people assume that optimization means sacrificing readability: cryptic bit manipulation, obscure tricks, code that only the author understands at 2 am. I believe Knuth was indeed warning against that kind of optimization. But that assumption is wrong more often than people think. Good, clean code is frequently efficient code too — not by accident, but because choosing the right tool for the job tends to be both clearer and faster. The examples in this article are proof of that. Scope This article focuses on simple, cheap, and foolproof tips that can be applied universally — regardless of your architecture, framework, or domain. In my experience, they carry virtually no risk of making things worse. Architecture, design, networking, database connectivity, threading — these are deliberately out of scope. Not because they are unimportant, but because they are context-dependent. The right answer depends on your specific system, and each of these topics deserves its own article. Examples String Operations We are all familiar with built-in JDK string utilities like: equals(), startsWith(), endsWith(), contains(): Java s1.equals(s2); s1.startsWith(s2); s1.endsWith(s2); s1.contains(s2); Unfortunately, JDK provides only one function for case-insensitive comparison: Java s1.equalsIgnoreCase(s2) There are no functions for case-insensitive startsWith(), endsWith(), contains(). So, often we combine toLowerCase() or toUppserCase() with startsWith(), endsWith(), contains(): Java s1.toLowerCase().startsWith(s2.toLowerCase()); s1.toLowerCase().endsWith(s2.toLowerCase()); s1.toLowerCase().contains(s2.toLowerCase()); A little verbose and null-prone, but just fine if not on the critical path. However, this technique might cause some performance problems. Do not forget that String is an immutable class, so instead of just a char-to-char comparison between two strings, we create two additional strings that then must be garbage-collected. Considering that String is a wrapper over a char array, the memory allocation may become expensive. The solution is to use case-insensitive utilities provided by different libraries, e.g., Apache Lang3: Java startsWithIgnoreCase(s1, s2); endsWithIgnoreCase(s1, s2); containsIgnoreCase(s1, s2); Or, starting from version 3.18.0: Java Strings.CI.startsWith(s1, s2); Strings.CS.startsWith(s1, s2); Where CI exposes case-insensitive and CS — case-sensitive utilities. Many people like regular expressions and use java.util.Pattern class sometimes, not where it is really necessary. For example: Java Pattern.compile("^prefix.+suffix$").matcher(s).find() Instead of: Java s.startsWith("prefix") && s.endsWith("suffix") Or even: Java Pattern.compile("^prefix").matcher(s).find() instead of s.startsWith("prefix") Pattern.compile("suffix$").matcher(s).find() instead of s.endsWith("suffix") Pattern matching is significantly slower than trivial substring matching. The following table shows evaluation time for 1 million operations: Operation * 1 million times Time, ms s.equals("hello") 7 s.startsWith("hello") 6 s.endsWith("hello") 11 s.contains("hello") 24 s.toUpperCase().startsWith("HELLO") 65 s.equalsIgnoreCase("hello") 5 Pattern.compile("hello").matcher(s).find() 238 pattern.matcher(s).find() 31 What can we see from this table? Performance of equals() and startsWith() is similarendsWith() is 2 times more expensivecontains() is 4 times more expensive than equalsChanging case followed by startsWith() is 10 times (!) more expensiveCase-insensitive comparison functions do not have any performance penaltiesSearching for a substring using a precompiled pattern is about 20% more expensive than using a plain contains() method. Compiling the pattern and using it is almost 10 times more expensive than the plain contains() method. So next time you reach for Pattern.compile(), it is worth pausing for a second: is regex actually needed here, or is a plain string method both simpler and faster? If you really need a pattern, at least compile it in advance — better yet, declare it as a private static final class member. Collections Let’s assume that we want to know whether a given list contains the specific element: Java list.contains("red"); In fact, this call invokes code like this: Java int n = list.size(); for (int i = 0; i < n; i++) { if ("red".equals(list.get(i))) { return true; } } Starting from Java 8, we have a streaming API that just hides from us the same gory details: Java list.stream().anyMatch("red"::equals); This is perfectly fine when the list is short, changes frequently, or is searched only occasionally. But if the list is large, stable, and searched repeatedly, a HashSet is the right tool — offering average O(1) lookup instead of O(n). If you cannot change the original data structure, converting it once at initialization time and searching the Set from that point forward is almost always worth it. If both the guaranteed element order and the fast lookup are needed, we can either hold duplicated data structures — a list for ordering and a set for search or just use LinkedHashSet, which solves both problems. Another common case is case-insensitive search. We already saw above that the combination of toLowerCase() or toUpperCase() with comparison significantly reduces the performance. This can be solved by using TreeSet with custom comparator, e.g. String.CASE_INSENSITIVE_ORDER: Java Set<String> set = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); This gives you a sorted, case-insensitive set with no extra allocations - and the same approach works for TreeMap when your data is key-value pairs. Enum Lookups Everyone knows that an enum entry can be found by its name using a built-in method valueOf(s). However, what to do if the given string is lowercase while enum entries following the naming convention are called using capital letters? Some people use a combination of toUpperCase() and valueOf() that work just fine but have the penalty we discussed above. However, very often people prefer to create a special field representing a “custom” name, so the simple enum like: Java enum Color { RED, GREEN, BLUE } Turns into: Java enum Color { RED("red"), GREEN("green"), BLUE("blue"), … } Let’s mention that this design has at least two disadvantages: Duplicate data: The custom name is the same as a built-in but in a different case, which can be solved much more easily. This allows using really custom names that, according to my experience, in most cases are not needed and just create so-called “edge cases” that, in turn, in most cases are just a signal of bad design and might cause a lot of “stupid” bugs. However, let’s continue. How do people often use this custom name? Java public static Color ofColor(String color) { return Arrays.stream(values()) .filter(c -> c.color.equals(color)) .findFirst() .orElseThrow(() -> new IllegalArgumentException("No enum constant %s.%s".formatted(Color.class.getName(), color))); } The implementation looks pretty nice, but this approach means that each call of ofColor() iterates over the list. Yes, in most cases enums are not huge, so the list is short, but anyway, why do this if we can just create a map from the custom name to the enum entry once during initialization and then use it with O(1) complexity? The following example solves both problems at once: it uses a case-insensitive map where the key is the standard name() of the enum entry during initialization: Java private static final Map<String, Color> colors = Arrays.stream(values()).collect(toMap(Enum::name, e -> e, (existing, replacement) -> replacement, () -> new TreeMap<>(CASE_INSENSITIVE_ORDER))); So, now the method ofColor() becomes trivial: Java public static Color ofColor(String color) { return Optional.ofNullable(colors.get(color)) .orElseThrow(() -> new IllegalArgumentException("No enum constant for " + color)); } One can argue that a map-based implementation is not always possible because sometimes the lookup criteria are too complex to be reduced to a simple key. Although I agree in general, I can say in turn that in many (if not in most) cases this is still possible. So far, the lookup key was a simple string. But what if the search criteria is a range rather than an exact value? Consider a more physically accurate model of colors as ranges of electromagnetic waves. Java public enum Color { BLUE(450, 495), GREEN(495, 570), RED(620, 750); …} How to implement the method ofWaveLength(int waveLength)? The straight-forward way is to iterate over the values of the enum and compare the given wave length with the range for each entry, i.e. implement O(n) search. But we can do better using NavigableMap, which is designed exactly for this kind of range query: Java private static final NavigableMap<Integer, Color> wavelengthMap = Arrays.stream(values()) .collect(Collectors.toMap( color -> color.minNm, color -> color, (existing, replacement) -> existing, TreeMap::new )); Unfortunately, the search method is not as trivial as in the previous example, but still very simple and fast: Java public static Color ofWaveLength(int nm) { return Optional.ofNullable(wavelengthMap.floorEntry(nm)) .map(Entry::getValue) .filter(value -> nm <= value.maxNm) .orElseThrow(() -> new IllegalArgumentException("No enum constant for wavelength: " + nm + " nm")); } Now, let’s compare the performance. Operation * 1 million times Time, ms valueOf(s) 34 valueOf(toUpperCase(s)) 78 Iteration with equals() 40 Color.ofColor() iteration 166 Color.ofColor() map 20 Color.ofWaveLength() map 32 The table shows that: As expected, toUpperCase() reduces performance twiceIteration with call of equals is a little bit more expensive than valueOf() although the enum has only three members and will grow linearly as the enum grows. The more members enum has, the more time iteration takes. Map-based implementation is even faster than one based on the built-in valueOf(). Stream-based iteration (ofColor() iteration) is surprisingly slow. Stream setup overhead (boxing, lambda dispatch, spliterator initialization) is non-trivial for tiny collections Pre-Intitialization The principle here is: do not do something several times if you can do it once. The most trivial example is string or numeric constants: Java private static final String FILE_NAME = "config.json"; private static final int MAX_VALUE = 10_000; However, the same principle applies to heavier objects — and that is where it really matters. Let’s take a look at logging. Most people are used to writing the following “magic” line at the beginning of each class (unless we use Lombok’s @Slf4j annotation): Java private static final Logger logger = LoggerFactory.getLogger(MyClass.class); Are all these modifiers (private static final) really needed? Some people try to save typing time: Java private final Logger logger = LoggerFactory.getLogger(MyClass.class); Moreover, if the logger is not static, we can do even more: Java private final Logger logger = LoggerFactory.getLogger(getClass()); This line looks better because it is error-proof: the class here is not hard-coded, so this line can be copied as-is from one class to another or inherited from the base class. So, what’s the problem? The problem is that retrieving the correct logger is potentially expensive due to synchronized registry lookups. Doing this on every instantiation adds up. A friend of mine told me that once in the company where he worked, this change in some critical path improved performance so much that they managed to reduce the AWS cluster by about one hundred large EC2 machines. The same rule applies to pattern compilation. As the benchmark table showed, compiling a pattern on every method call is nearly ten times slower than reusing a precompiled one. The result of Pattern.compile() should always be stored in a static final field. The only exception is the case when the regular expression is generated dynamically, but we should do our best to avoid such a design. Very often we have to format or parse dates. Traditionally I used SimpleDateFormat. What can be more obvious than this: Java private static final String FORMAT = "yyyy-MM-dd HH:mm:ss"; private static final DateFormat format = new SimpleDateFormat(FORMAT); Frankly speaking, I did this many times following the principle I stated above: there is no reason to create the instance every time we need it if we can create it only once. The problem is that SimpleDateFormat is not thread-safe, so sharing the same instance among different threads can cause the problem. Even worse: we can live with this bug for years without knowing about it, since it only happens under high load and in some cases can just produce slightly wrong results that can be lost in an ocean of valid data. So, should we create instances of SimpleDateFormat every time we need it and cause CPU and GC to work hard? Fortunately, starting from Java 8, we can use DateTimeFormatter instead: Java private static final DateTimeFormatter formatter = DateTimeFormatter.ofPattern(DATE_FORMAT); This class is thread-safe, so we can share its instance among different threads and get consistent results. Conclusion We started with a quote that is almost always cited incomplete. Knuth never said ignore performance — he said don't sacrifice clarity for speculative gains, while reminding us not to pass up opportunities in that critical 3%. The examples in this article live in that 3%. None of the performance issues described here should ever appear in production code. They are not hard to avoid — they require no profiler, no benchmarking framework, no architectural discussion. Just the habit of reaching for the right tool. And that habit pays off. Choosing equalsIgnoreCase() over toLowerCase().equals() is cleaner and faster. A static final logger is simpler and cheaper. A pre-built enum map is more readable and O(1). Good code and efficient code are not in conflict here — they are the same code. The only thing required is the habit of pausing for a second and asking: am I doing this n times when once would do? All code examples from this article are available on Gist.

When putting their model into production, every team or organization encounters the same issue. Failures go unnoticed for days at first because there is no monitoring. As teams begin to fix the issues, they identify areas where production results deviate from the training data, create dashboards for every metric, and set alerts for every threshold. This results in engineers being paged at two in the morning for a bug that fixes itself within an hour, and when an important alert arises, it goes unanswered due to alert fatigue, creating a pipeline that silently feeds garbage into the model. When a team learns to disregard 95% of the issues, they are very likely to disregard the remaining 5% that are actually important, and the solution to this isn’t less monitoring. The good solution to this problem is monitoring, which is tiered, routed, and pruned differently from the infrastructure monitoring that most teams already know. The Problem With Applying Old Monitoring Rules To AI Traditionally, application monitoring used to be binary, which is whether the application or service is up or down, latency is high or low, etc. But AI models don’t fail with these signs; they usually degrade over time. For instance, a recommendation model does not show exceptions when the user behavior shifts; it just silently gets worse at what it was supposed to do. A classifier model does not throw an error when its input distribution changes; it just returns answers confidently with increasingly wrong predictions. An AI application does not crash when it hallucinates; instead, it returns a normal HTTP 200 response with incorrect content. This creates two problems: When AI models fail, the reason for failure is invisible to classical infrastructure monitoring, which causes teams to bolt on multiple checks like data quality checks, drift detectors, and output scorers, each introducing a new source of noise. AI models are statistical in behavior and not deterministic, so setting threshold alerts on them leads to them firing constantly, and training teams have to tune the model. As a result, thorough AI monitoring does not make the application safer; beyond a certain point, it only makes things worse. What to Actually Monitor Monitoring issues that no one will ever take action on is often the first step towards alert fatigue. It is useful to consider it in four layers, each with its own owner and mode of failure. Infrastructure and service: Metrics like inference latency, throughput, Graphics Processing Unit (GPU)/Central Processing Unit (CPU) utilization, error rates, and cost per request and token consumption for anything calling a hosted large language model (LLM) API are classic operational metrics and can usually be monitored with the existing Application Performance Monitoring (APM) tools. Data quality: This is another important thing to keep an eye on because it can cause broken feature pipelines, upstream schema changes, input formats being changed without getting noticed, and null-rate spikes. These are usually the worst failures because you can't see them unless you're looking for them, and the model keeps making predictions based on bad data. Model quality: This can be tracked by looking at changes in the Confidence Score or how much the prediction distribution has changed from what was seen during training. This can be used instead of measuring accuracy because it's hard to tell right away how measures like accuracy are calibrating, because to measure accuracy, you would have to compare the predicted result to the actual correct answer, which doesn't always exist at the time of prediction. Generative artificial intelligence/large language model quality: Metrics like hallucination rate, coherence, factual grounding, toxicity, and susceptibility to prompt injection need different types of tooling to identify them because they are not like traditional metrics and would require human-in-the-loop sampling or an LLM as a judge for identifying them. The mistake many teams make is that they apply the same alerting techniques to all four layers, which is the infrastructure one, as that is the traditional way of setting up monitoring for applications, but issues related to data quality and model quality require a trend-based review. How to Alert Without the Noise Replace static thresholds with adaptive baselines. When systems learn a baseline from historical behavior and trigger alerts on deviations from it, like “alert if latency exceeds 200ms,” this ignores the daily and weekly traffic patterns, and the same is valid for data volume and null rates, which leads to a large number of false alarms being raised. So, teams that have made this switch from static thresholds to adaptive baselines have reportedly reduced noisy alerts by 60–90%. Introduce real severity tiers. When an alert is critical and poses an instant business risk, it is sent to an on-call engineer so that the problem can be fixed right away. Warnings about poor performance that are not critical are sent to a Teams chat channel during business hours, and signals about long-term trends land on the dashboard to be looked at from time to time. This helps to make sure that the notification's urgency matches its real urgency. Correlate and deduplicate before notifying. One change to the schema upstream can cause a dozen problems downstream. Sending a dozen alerts for one root cause either makes the team too busy or forces them to mentally group alerts together, which your tools should be doing for you. Route alerts to whoever can act on them. Misrouting is a common cause of tiredness. If the central platform team doesn't know about the business, they might ignore a spike they can't understand, and the domain team that would be able to understand it would never see the alert. Both problems are solved by linking alerts to the right person by domain, based on where the problem starts. Prioritize by business impact. A system that looks for unusual events handles all alerts the same way because it doesn't know which parts of your system are important to the business. When you think about how important each problem is before choosing how loud to alert, you get a lot fewer alerts overall, and a lot more of them are ones that you should actually act on. Conclusion It's important to understand that all of the ideas we've talked about work together; none of them can be used on their own. For example, adaptive thresholds only give out fewer alerts that aren't differentiated by severity. Without proper routing, severity tiers send the wrong messages about how important something is to the incorrect individuals. To avoid alert fatigue, teams need to take comprehensive actions, which include proper alert designs and organizational practices. They should also ensure that every alert can be acted on, which is better than monitoring everything, because AI monitoring only scales, and not having anyone see a model fail could have serious consequences. Good monitoring means building a system that sends alerts only when it matters, so when it does, people actually act on it.

Key Takeaways In regulated industries, cloud migration success is determined less by technology selection and more by how deliberately you decouple risk vectors — compliance risk, organizational hesitation, user adoption gaps, and integration changes — so no single failure can derail the whole program.You can successfully migrate an application to AWS while keeping data on-premises by routing through a REST API abstraction (e.g., IBM’s DB2 REST API layer) paired with dedicated AWS security groups controlling cloud-to-on-prem traffic, allowing the data migration to proceed on its own compliance and trust-building timeline.The most dangerous compliance gap in regulated applications isn’t declared sensitive fields — it’s free-form text fields where users may inadvertently type SSNs, credit cards, or other regulated identifiers; proactive tokenization in the application’s write path closes this gap before any audit finds it.Long-tenured business users carry a decade of UX muscle memory that QA testing cannot replicate; allocating real production validation time (such as a 15-day dark deployment cohort) is essential when migrating systems users have relied on daily for 10+ years.Before starting a regulated cloud migration, ask which risk vector each architectural decision is decoupling and whether your team is aligned on why — this single question reframes "cloud migration" from a technology project into a coordinated risk-management exercise. Introduction Most published writing on legacy-to-cloud migration treats it as a technical exercise: pick the stack, plan the cutover, flip the switch. In regulated industries, that framing fails — and the failure mode isn’t a missed deployment window. It’s a stalled program, a failed compliance audit, or a client who pulls back from the cloud strategy entirely. A cloud migration in healthcare insurance is as much about regulatory risk management, organizational trust-building, and user adoption as it is about microservices and Fargate. Get the technology right and miss the risk choreography, and the project doesn’t ship. I led the first WebSphere-to-AWS migration in the health division of a Fortune 50 insurer — a multi-year program touching PHI data, long-tenured business partners, and downstream services concurrently migrating to the cloud. Over that program, six architectural patterns emerged as decisive. Not for the technology they enabled, but for the risks they made manageable. None are individually novel. What’s distinctive is how they work together — as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. Pattern 1: Strangler Fig With Dark Deployment When migrating critical production systems to the cloud, the temptation is a hard cutover — flip the switch at 2 AM on a Sunday and hope for the best. We chose a different path: a 15-day dark deployment on AWS production, accessible only to a designated cohort of business partners. Three factors drove this decision. 1. First-mover risk in the department. This was the first WAS-to-AWS migration in this Fortune 50 insurer’s health division. There was no internal precedent to draw from — no playbook, no lessons learned from a prior AWS rollout. A "big bang" cutover would have exposed our full user base to whatever unknowns we hadn’t anticipated. Dark deployment let us pioneer the path with limited blast radius. 2. Regulatory exposure on PHI data. The application processes Protected Health Information. Any data integrity issue — a missed field, a misformatted record, a sync gap — could have triggered regulatory scrutiny. By exposing the new AWS environment to a small group of business partners first, we could validate end-to-end data flow in real production conditions without putting the full user base or compliance posture at risk. 3. UX learning curve. We had explicitly rejected a lift-and-shift approach. The new application wasn’t just re-hosted — the UI had been redesigned, the APIs restructured, and user workflows updated. Even excellent technical execution couldn’t eliminate the learning curve our users would face. Dark deployment gave us 15 days of real-world UX observation: where do users hesitate, what do they misunderstand, which workflows feel awkward? By the time we cut over publicly, we had already addressed the rough edges. The result: When we replaced the WAS production URL with the AWS production URL, end users perceived the change as a routine UI update, not a foundational technology migration. Pattern 2: Decouple Application Migration From Data Migration The default assumption in cloud migration is that application and data should move together. We made the opposite choice: migrate the application to AWS while keeping the underlying DB2 data on-premises. Three factors made this the right call. 1. PHI/HIPAA compliance complexity. The application processes Protected Health Information governed by HIPAA. Moving regulated healthcare data to a new environment raises a long list of compliance questions — encryption-at-rest configurations, audit logging, access control policies, business associate agreements with the cloud provider, breach notification readiness. None of these are insurmountable, but they take months of compliance review. Treating data migration as a separate workstream with its own compliance approval cycle was significantly less risky than bundling it into the application cutover. 2. Client comfort and trust-building. Cloud migration is as much a psychological transition for the client as a technical one. Moving an application to AWS is one decision; moving sensitive data off the client’s own infrastructure is a much larger one — it changes their security perimeter, their incident response posture, and in some cases their regulatory filings. Insisting on moving both at once would have either delayed the program waiting for full executive comfort, or risked a "no" on the entire initiative. Application-first let us demonstrate the new architecture working successfully before the data migration conversation began. 3. Parallel team enablement. Decoupling created room for a separate analytics team to independently assess which data could move to the cloud, on what timeline, and under what compliance framework. The application architecture was designed from day one to support a hybrid future — partial data on AWS, other data on-prem — so the analytics team’s work didn’t block application progress. How the technical decoupling works. The natural temptation when keeping data on-prem is to expose a direct database connection from the AWS application back to the on-prem DB2 instance. We rejected that — opening database ports across the cloud-to-on-prem boundary is a security liability, a latency problem, and a fragile dependency. Instead, we used IBM’s DB2 REST API layer to expose data access through authenticated HTTPS-based service calls. The AWS application talks to data through an API, not a database connection. This abstraction also positions the application to seamlessly switch to AWS-resident data later, without any application code change — only the API endpoint moves. Network-layer security follows the same decoupling principle. We provisioned dedicated AWS security groups on the Fargate side specifically for the IMS and DB2 connections back to the on-premises environment — only requests from those approved security groups can traverse the firewall to the on-prem data tier. Combined with the REST API abstraction, this gives us both application-layer (authenticated HTTPS) and network-layer (security-group-controlled) protection across the cloud-to-on-prem boundary. The result: A successful cloud migration with regulatory exposure isolated to a single workstream, and a forward path that doesn’t force the client into uncomfortable decisions before they’re ready. Pattern 3: EJB Monolith → Containerized Microservices on Fargate The original application was a Java EJB monolith running on WebSphere. The "lift-and-shift" temptation would have been to containerize the existing EJB code as-is into AWS Fargate — preserving the architecture, just moving the deployment substrate. We rejected that and instead decomposed the monolith into bounded REST microservices. Three reasons drove this decision. 1. Downstream services were also migrating. The application integrated with 5–7 SOAP-based services owned by adjacent teams — agreement service, customer service, sensitive data masking, and others. Those teams were simultaneously migrating their own services from WAS to AWS, which meant interface contracts, protocols, and endpoints would inevitably change. Inside an EJB monolith, every downstream integration change forces a recompile-redeploy-retest cycle of the entire application. Inside microservices, only the integration adapter for the affected service needs to change. With multiple active migration interfaces, the flexibility difference compounds quickly. 2. EJB development velocity is structurally slow. Even routine changes to EJB code require a full WAR/EAR build, redeployment to the WAS instance, and a heavy test cycle. The technology wasn’t designed for the iteration speed we needed to support a multi-year migration alongside actively changing downstream dependencies. Microservices on Fargate gave us a development model — fast container builds, independent deployments, isolated test environments — that matched the pace of the work. 3. Future data migration optionality. As noted in Pattern 2, the underlying data was kept on-premises for now, but a phased data migration to AWS was planned. By isolating database calls and IMS calls into dedicated microservices, the change required when the data eventually moves is localized — swap one service’s data access logic rather than reworking the monolith. The architecture is positioned for the data move whenever the client is ready. How we sized the decomposition. The boundaries followed natural integration points: each external SOAP integration became its own bounded microservice with a thin REST API. Data access calls (DB2 via REST, IMS) were isolated into dedicated services. The frontend talks to a coordination layer that orchestrates calls across these services. The result was a clean set of containerized microservices on AWS Fargate — each independently deployable, scalable, and testable. The result: A modernization that didn’t just relocate the code, but restructured it to absorb the inevitable changes coming from adjacent migrations across the organization — without recompile-redeploy-retest pain. Pattern 4: Frontend Decoupling via S3 + CloudFront The original WAS application followed the classic tightly-coupled pattern: JSP pages rendered server-side, deployed alongside the backend, scaling and updating as one unit. We made an architectural break in the migration — the frontend became a fully independent single-page React application hosted on Amazon S3 and served via CloudFront. Three factors made this the right call. 1. Independent deployment cadence. Frontend and backend evolve at different speeds. UI tweaks — copy changes, validation logic, visual updates — are frequent and low-risk. Backend API changes are slower and require careful coordination with downstream service migrations. Decoupling them means UI changes can be deployed instantly through a separate UI pipeline (different Git repository, different infrastructure, different release cadence) without touching the backend microservices. A small label change no longer requires a full backend deployment. 2. Adopting an accessibility-first enterprise UI library. Alongside our migration, an internal innovation track was building a shared component library to unify UX patterns across the organization’s applications — consistent typography, controls, brand elements, and critically, accessibility as a first-class concern: full screen reader support, keyboard navigation, sufficient color contrast, and ARIA-compliant semantics. JSP-based legacy pages couldn’t meaningfully integrate this kind of library. By rebuilding the frontend as a React single-page application, we adopted the library fully — and incorporated rigorous accessibility testing into every release cycle. Users who rely on assistive technologies (screen readers, alternative input devices, magnification) get full application access. For an application processing PHI in a regulated industry, this proactive accessibility-first approach is itself a substantial improvement over the legacy app. 3. Global performance through edge caching. S3 alone would have served the static assets, but we layered CloudFront on top to push content to edge locations closer to users. Business partners access the application from different geographic regions; CloudFront cuts load times by serving cached assets from the nearest edge, not the S3 origin in a single AWS region. This is a substantial UX improvement that simply wasn’t possible with WAS-hosted JSPs. How the architecture flows. User requests hit CloudFront, which serves cached React bundles, HTML shells, and static assets from the nearest edge. The React application then makes authenticated REST API calls back to the backend microservices on AWS Fargate. The frontend has no awareness of which microservice serves any particular request — it talks to a coordination API layer that handles orchestration. The result: A UI architecture that’s faster (edge-cached), cheaper (no application servers for the frontend), easier to update (independent pipeline), more inclusive (accessibility-first), and aligned with the broader enterprise UX modernization effort. Pattern 5: Business Partner Real-Production Validation Cohort Pattern 1 described the deployment mechanism — a 15-day dark deployment exposing AWS production to a limited cohort. Pattern 5 is about who was in that cohort and why we deliberately chose real business partners over our QA team for production validation. Two factors shaped this decision. 1. Decades of muscle memory in the existing UX. Our business partners — long-tenured users of the application — had been using the legacy UI for 10–15 years. They knew every workflow, every shortcut, every quirk. The new React application introduced not just a new visual style but new patterns from the organization’s modern component library. Even with rigorous accessibility and usability testing in QA, a brand-new UI in front of users with a decade of habits guaranteed friction. The 15-day validation cycle gave those users time to acclimate to the new patterns and surface UX issues that only show up at the speed of real daily work — keyboard shortcuts they used unconsciously, screens they navigated to multiple times an hour, validation logic that affected their flow. QA testers, by definition, don’t have that muscle memory. 2. First-of-its-kind migration with concurrent change. This was the first WAS-to-AWS migration in the health division, and we’d simultaneously re-architected the UI, the API layer, and incorporated changes from downstream services that were also mid-migration. With that many concurrent changes, even thorough QA can’t realistically simulate the full combinatorial space of real production usage — real customer data, real edge cases, real integration timing, real load patterns. Putting real business partners on the actual AWS production environment for 15 days was our safety net: anything QA missed, the cohort would surface, and we could fix it before broad cutover. Beyond the cohort: maturing the delivery pipeline. A secondary benefit of running an extended validation window was that it gave the engineering team time to mature the CI/CD pipeline alongside the application. By the second application in the migration program, we’d evolved the cohort approach into a full blue/green deployment model on AWS — building organizational learning alongside the application portfolio. The validation pattern isn’t static; it strengthens with each subsequent migration. The result: a validation approach that combined deep domain familiarity (real business partners) with controlled exposure (limited cohort, real production) — catching the issues QA can’t, well before public cutover. Pattern 6: Defensive Tokenization for Sensitive Data in Free-Form Fields In regulated industries, the obvious sensitive data — SSN fields, credit card fields, account number fields — gets protected automatically. The dangerous category is the unstructured data: a free-form text field where a user can type anything. In our application, users entered "health notes" — narrative text describing customer interactions. The risk: nothing in the application schema prevents a user from typing an SSN, a credit card number, a driver’s license, or other regulated identifiers directly into that note. Once stored, that PHI/PII data is sitting in a free-text column with no encryption-at-rest tailored to it, no masking on display, no controlled access — and our compliance posture changes accordingly. We addressed this proactively by integrating an internal sensitive-data-masking service into the application’s write path. Before any free-form text reaches the data layer, the masking service scans the input, identifies regulated identifiers (SSN-pattern strings, credit card numbers via Luhn check, driver’s license formats), and applies tokenization — replacing the identifier with a non-reversible token or masked representation. The original value never lands in the database in plaintext. Three things made this a deliberate architectural pattern, not an afterthought: 1. It was incorporated before the formal risk assessment, not in response to it. Risk assessment was a new exercise for the team — none of us had been through one for AWS-hosted PHI before. Rather than wait for the assessment to flag the free-form field as a finding, we performed our own data classification first, identified the free-form notes as a regulated-data risk vector, and integrated the masking service pre-emptively. When the formal risk assessment ran, this control was already in place. 2. We reused an existing internal service, not built a new one. The masking service already existed in another WAS-hosted application within the broader life/health portfolio. Instead of re-implementing tokenization logic, we adopted the existing service — saving development time and inheriting the existing security review and operational maturity of that service. Migrations are a good moment to identify reusable internal capabilities rather than reinvent them. 3. It addresses a class of risk most compliance reviews don’t anticipate. Compliance checklists focus on declared sensitive fields ("the SSN field," "the account number field"). They rarely interrogate free-form text fields, because those fields aren’t supposed to hold sensitive data. But in practice, users type whatever they need to type — and what they type is what your application stores. Proactive defensive tokenization closes that gap. The result: free-form notes that look normal to users, but whose backend storage is sanitized of any regulated identifiers the user may inadvertently include. The application’s compliance posture is robust to user behavior, not just to user intent. Conclusion: The Through-Line Is Decoupling Looking back across the six patterns, the through-line isn’t any specific technology — it’s a posture: deliberate decoupling of risk vectors so that no single failure, regulatory finding, organizational hesitation, or user adoption gap can derail the whole migration. Pattern 1 (Strangler Fig with Dark Deployment) decouples cutover risk from broader rollout.Pattern 2 (Decouple App from Data) decouples application migration from the data-and-compliance timeline.Pattern 3 (EJB → Microservices) decouples downstream integration changes from our own deployment cadence.Pattern 4 (Frontend on S3/CloudFront) decouples UI release cadence from backend release cadence.Pattern 5 (Business Partner Validation Cohort) decouples real-world UX surprises from public rollout.Pattern 6 (Defensive Tokenization) decouples user behavior risk from data-layer compliance posture. None of these patterns are individually novel. What’s distinctive is choosing them together, as a coordinated set of risk-decoupling decisions in a first-of-its-kind regulated cloud migration. The result was a migration that didn’t surprise our compliance team, didn’t surprise our users, and didn’t surprise our auditors — which, in a regulated industry, is the kind of unsexy outcome that defines success. If you’re starting a similar program, the question isn’t which of these patterns to adopt. It’s: which risk vector are you decoupling, and is your team aligned on why?

If you let users publish something, such as a page, prototype, or dashboard, sooner or later you want an "embed this" button so they can drop it into a blog, a portfolio, or docs, the way a CodePen result embeds. Then you ship the iframe, and it renders a blank box: refused to connect. The reflex is to blame the iframe. It's almost never the iframe. It's a response header. The Two Headers That Decide Whether You Can Be Framed There are two mechanisms, and they are not equivalent: X-Frame-Options is the legacy control. It has three meaningful states: DENY, SAMEORIGIN, and the deprecated, widely-ignored ALLOW-FROM. Crucially, there is no value that means "allow any origin" or "allow this list of origins." It is deny / same-origin / nothing-useful. If your edge returns X-Frame-Options: SAMEORIGIN, a third-party site can never frame you, full stop.CSP frame-ancestors is the modern replacement. It is part of Content-Security-Policy and takes a real source list: frame-ancestors 'none', 'self', https://example.com, or *. It is granular where X-Frame-Options is binary. The catch that trips people up: if you send both, X-Frame-Options is still honored by many browsers and will block framing regardless of how permissive your frame-ancestors is. So to actually be embeddable by third parties, you have to remove X-Frame-Options, not just add a permissive frame-ancestors next to it. The Footgun: One Global Security-Headers Middleware Here is the trap. The application that rendered our published sites already made the right call in code: it disabled frameguard and emitted a permissive frame-ancestors. And yet every embed was blank. The header was not coming from the app. It was re-added at the edge. A single shared "secure-headers" middleware, the kind every reverse proxy ships and every security checklist tells you to apply globally - included X-Frame-Options: SAMEORIGIN in its response headers. The proxy ran that middleware on the router that served published user sites, stamping SAMEORIGIN on top of the app's deliberate "please frame me" headers. The edge won. State it plainly: applying one blanket security-headers policy to every route is a footgun the moment one of those routes is supposed to serve embeddable content. That middleware is correct for your API and your authenticated app. It is wrong for the one route whose entire job is to be put inside someone else's <iframe>. The Fix: Scope Headers Per Trust Zone The fix is not "turn off security headers." It is to stop treating every route as one trust zone: Authenticated and sensitive routes (/api, realtime/WebSocket, the editor app) keep the full secure-headers set, including X-Frame-Options: SAMEORIGIN. Those should never be framed; clickjacking protection stays.The route that serves published, public, client-only user pages gets a near-identical header set - same X-Content-Type-Options, Referrer-Policy, Strict-Transport-Security - but without X-Frame-Options. Whether such a page can be framed is then governed by the frame-ancestors the page itself serves. In practice, that is a second middleware that is a copy of the first minus one header, pointed only at the published-pages router. Surgical. Nothing else loses protection. YAML secure-headers: # sensitive routes - keeps clickjacking protection headers: customResponseHeaders: X-Frame-Options: "SAMEORIGIN" contentTypeNosniff: true referrerPolicy: "strict-origin-when-cross-origin" stsSeconds: 31536000 pages-headers: # same set, minus X-Frame-Options - embeddable pages only headers: contentTypeNosniff: true referrerPolicy: "strict-origin-when-cross-origin" stsSeconds: 31536000 Then the page that is meant to be embeddable expresses its own policy: YAML Content-Security-Policy: frame-ancestors *; (or a specific allowlist, if only certain hosts should embed it). Embedding User-Generated Content Safely "Make it embeddable" and "make it safe" have to hold at the same time, because you are putting code you did not write into a frame. A few rules that travel well: Isolate every project on its own origin. Serve each published site from its own subdomain ({slug}.example.io), never a shared path. Origin isolation means one project's script cannot reach another's storage, cookies, or DOM. This is the single biggest lever.Sandbox the frame. The embedding side should use <iframe sandbox="allow-scripts allow-popups ..."> and grant only the capabilities the content needs. Omit allow-same-origin where you can, so the framed document runs with an opaque origin.Let the page opt out. A published page should be able to override the edge default and refuse framing - its own X-Frame-Options / frame-ancestors should win over the proxy default. Author intent beats infrastructure default.Keep authenticated surfaces un-framable. The embeddable posture applies to public content only. Anything behind a login keeps SAMEORIGIN. This is the posture we landed on at Playcode, an AI website and app builder: published projects each live on their own origin, the published-pages route drops X-Frame-Options so a one-line embed drops a live project into any blog or docs page, while the editor, API, and Playcode Cloud backend keep full clickjacking protection. A static published page carries the same minimal framing risk that previews and custom domains already had. The difference is that it is now a deliberate, scoped decision instead of an inconsistent accident across routes. Takeaways A blank "refused to connect" embed is almost always X-Frame-Options, not your iframe.X-Frame-Options cannot express "allow these origins" - use CSP frame-ancestors for anything granular, and drop X-Frame-Options entirely on routes that must be embeddable.Do not apply one global security-headers middleware to routes that serve embeddable content; scope headers per trust zone.Embeddability and safety coexist through origin isolation, the iframe sandbox attribute, and letting the page author's policy win over the edge default.

A Temporal Workflow that appears stuck is rarely “stuck” in the conventional process sense. Temporal persists Workflow state through Event History and resumes execution through replay, so an open execution can remain healthy while waiting for a timer, Signal, Activity, or external condition. The operational problem is therefore not simply lack of completion; it is lack of expected progress. Effective diagnosis starts by establishing what event should have happened next, why it did not happen, and whether remediation can preserve the Workflow’s business invariants. Temporal’s history model makes that analysis unusually tractable because commands, task transitions, Activity attempts, failures, timers, and external interactions are durably represented as Events. Progress Is Visible in the Event History The first diagnostic artifact should be the execution description and raw history, not application logs. temporal workflow describe exposes current execution information and pending Activity state, while temporal workflow show --output json returns Event History in a form suitable for programmatic replay or analysis. A Workflow Query can additionally expose application-defined state without mutating the execution. Shell temporal workflow describe --workflow-id order-7814 temporal workflow show \ --workflow-id order-7814 \ --output json History should be read as a state-transition trace. A WorkflowTaskScheduled event with no corresponding start suggests that work is waiting for a Worker. A started Workflow Task that repeatedly times out can indicate blocked Workflow code, Worker instability, or excessive work inside a task. Repeated WorkflowTaskFailed events can indicate replay or deterministic-compatibility failures after code deployment. Workflow Task failures are retried by Temporal rather than governed by an Activity-style Retry Policy, so a Workflow can remain open while repeatedly failing to make application-level progress. Activity sequences reveal a different failure surface. ActivityTaskScheduled without ActivityTaskStarted points toward dispatch capacity, missing pollers, queue mismatch, or backlog. Temporal persists Workflow and Activity Tasks in Task Queues, and worker-health guidance identifies Schedule-to-Start latency and approximate backlog count as key signals when tasks wait for Workers. ActivityTaskStarted without completion requires inspection of Start-to-Close and Heartbeat behavior because Temporal relies on Start-to-Close timeout to detect a Worker crash after an Activity has started. Not every long pause is pathological. A timer that has not fired, a Workflow waiting for a Signal, or an Activity still inside a valid timeout window can represent correct durable waiting. Conversely, very large histories can become an operational risk. Temporal warns after 10,240 events or 10 MB and enforces a limit of 51,200 events or 50 MB; Continue-As-New creates a new run with a fresh history while carrying forward relevant state. Triage Works Best as Deterministic Evidence Before Model Judgment LangGraph is useful for automating this analysis, but the safest design keeps Temporal facts deterministic and uses an LLM only for classification, hypothesis ranking, and explanation. LangGraph explicitly supports graphs that mix deterministic nodes with model-driven nodes, while structured output can constrain routing decisions into a defined schema rather than free-form text. A compact analyzer can first reduce raw history into evidence that is difficult to hallucinate: the last completed Workflow Task, consecutive Workflow Task failures, pending Activity IDs, the latest Activity attempt, the timeout type, the last Signal, the last timer, the history size, the task queue, and deployment/version metadata. The model then receives that normalized evidence instead of thousands of raw events. Python def extract_facts(state): events = state["events"] return { "facts": temporal_fact_extractor(events), "tail": events[-60:], } def classify(state): result = triage_model.with_structured_output(TriageResult).invoke({ "facts": state["facts"], "tail": state["tail"], "allowed_causes": [ "worker_unavailable", "activity_retrying", "workflow_task_failure", "intentional_wait", "history_pressure", "unknown", ], }) return {"triage": result} That separation matters operationally. Event parsing can enforce hard rules such as “scheduled but never started,” while the model can correlate several weak signals and produce an explanation. Conditional edges can then route low-risk cases to observation, ambiguous cases to deeper diagnostics, and recovery candidates to an approval gate. LangGraph’s graph API supports conditional routing, and persistence stores checkpoints so triage state survives interruptions or process failures. Recovery Must Preserve Temporal and Business Semantics Diagnosis and remediation should remain separate graph stages. A model-generated recommendation must not directly issue cancellation, reset, or termination. LangGraph interrupts provide a natural control boundary because execution can pause with persisted state and resume only after external approval. Python def approval_gate(state): decision = interrupt({ "workflow_id": state["workflow_id"], "cause": state["triage"].cause, "action": state["triage"].recommended_action, "evidence": state["triage"].evidence, }) return {"approved": decision == "approve"} The remediation choice depends on the failure mode. A transient Worker outage usually requires restoring Worker capacity rather than mutating Workflow state because queued tasks persist until Workers can process them. An Activity repeatedly failing on a recoverable dependency can often be left to its Retry Policy, while permanent errors should be made non-retryable in application design to avoid pointless retries. Activity side effects should be idempotent because Activity attempts may execute more than once under retry and recovery behavior. Cancellation is the preferred stop mechanism when Workflow cleanup logic must run. Temporal records a cancellation request and schedules a Workflow Task so Workflow code can react. Termination is forceful: Workflow code does not receive a chance to clean up, and the terminated event closes the history. That makes termination an escalation path for executions that cannot process cancellation normally. Reset is more powerful and more dangerous. Temporal terminates the current execution and creates a new execution that copies history through a selected reset point, then replays forward using current Workflow code. Progress after the reset point is discarded. Reset is therefore appropriate only after the underlying cause has been corrected and after downstream side effects are reviewed for possible re-execution beyond the reset boundary. Shell temporal workflow reset \ --workflow-id order-7814 \ --event-id 42 \ --reason "Recovered after deterministic-compatibility fix" For history pressure rather than a fault, Continue-As-New is generally the safer lifecycle mechanism because it preserves logical continuity under the same Workflow ID while starting a fresh Event History with a new Run ID. It should be designed into long-lived or high-volume Workflow logic instead of used as an improvised emergency action. Safe Automation Requires an Explicit Remediation Envelope A production triage graph should treat remediation as a constrained transaction. The evidence snapshot, selected run ID, candidate reset event, intended action, reason, approval identity, and execution result should all be persisted before any mutation. The action node should re-read the Workflow immediately before execution and reject the operation if the run has changed or the observed condition no longer matches the diagnosis. This is an engineering safeguard rather than a Temporal requirement, but it reduces time-of-check/time-of-use errors when active Workflows continue progressing during investigation. LangGraph’s checkpoint model supports durable approval state, but resumed graph nodes can re-execute from checkpoint boundaries. Its documentation therefore recommends isolating side effects and designing them to be idempotent. A remediation executor should consequently use an operation ID, record completion externally, and refuse duplicate destructive actions. Recovery Without Guesswork Reliable recovery of a stuck Temporal Workflow is fundamentally an event-history problem, not a process-restart problem. The strongest diagnostic path reconstructs expected progress from Workflow Tasks, Activity attempts, timers, Signals, queue state, timeouts, and history growth before considering mutation. LangGraph can turn that evidence into a durable triage pipeline by combining deterministic extraction, constrained model reasoning, conditional routing, and interrupt-based approval. Safe remediation then follows Temporal semantics: restore Workers when dispatch is the issue, allow bounded retries for transient Activities, cancel when cleanup matters, terminate only as a last resort, reset only after the root cause is fixed, and use Continue-As-New to control long-running history growth. The result is automation that accelerates incident response without allowing probabilistic diagnosis to become an unchecked control plane.

By Akhil Madineni CORE

Learn how attackers enumerated Salesforce Experience Cloud and ServiceNow portals — and how defenders can detect and prevent the same abuse. When Guest Access Becomes an Attack Surface Modern enterprise portals increasingly expose APIs to unauthenticated users. The problem is not necessarily that those APIs are vulnerable. The problem is that the anonymous identity behind them may have been granted more access than the organization realizes. By now, the existence of the campaign covered in this piece isn't news. SecurityWeek, BleepingComputer, Dark Reading, and Help Net Security have all reported on it in the last few days, drawing on research published by SaaS security firm Reco. What none of that coverage had room for is the protocol-level mechanics: exactly how the enumeration works against Salesforce's two different component frameworks, exactly where ServiceNow's authorization decision actually lives, and exactly what a defender should pull from logs to tell this apart from ordinary traffic. That's the gap this article fills. In an interview arranged through Reco, I spoke with security researcher Nitay Bachrach — one of the researchers behind the original investigation — about how his team built that distinction, endpoint by endpoint. What follows combines his answers with Reco's published indicators and current Salesforce and ServiceNow platform documentation. What the City-Forum Campaign Actually Found Reco calls the activity the City-Forum campaign, after a domain tied to the operator's infrastructure. A single source has been interacting with Salesforce Experience Cloud and ServiceNow Service Portal deployments through guest-accessible interfaces since at least March 2025 — over seventeen months of continuous activity, still climbing in volume as of Reco's publication. On Salesforce, the activity spans Aura enumeration, LWR UI-API and GraphQL requests, and self-registration probing. On ServiceNow, the same infrastructure repeatedly targets the native Service Portal search endpoint. Targets span telecommunications, banking and financial services, enterprise software vendors — including security and data-privacy companies — and public-sector portals; Reco has not named individual organizations. Critically, Reco is explicit that none of this exploits a platform vulnerability. Every record retrieved was something a site owner had already exposed to anonymous users, through sharing rules, permissions, or portal search-source configuration. One Infrastructure Source, Two Enterprise Platforms Everything traces to a single IP address: 158.220.87.79, on a Contabo VPS (ASN 51167, Germany). Passive DNS ties that IP to the domain city-forum.com, registered in 2002 and long abandoned before being repurposed for this infrastructure, resolving to the operator's server since at least March 12, 2025. That's an unusually long, unrotated run for this kind of activity. Campaigns like the previously reported ShinyHunters Experience Cloud campaign have typically drawn on multiple machines and rotating IP ranges. This one hasn't — the same box has carried the same domain for the entire observed window. Verifiable indicators, independently confirmable via dig: IP: 158.220.87.79 — ASN 51167 (Contabo GmbH), reverse DNS vmi2213719.contaboserver.netDomain: city-forum.com and active subdomains www.city-forum.com, server.city-forum.com, www.server.city-forum.com, mail.city-forum.com, www.mail.city-forum.comAn SPF record explicitly authorizing the IP to send mail as the domain Reco's own guidance is worth repeating for anyone hunting this: resolve the domain rather than browsing to it. There's no legitimate reason to load attacker-adjacent infrastructure in a browser. Every request across both platforms carries the same user-agent: Go-http-client/1.1, Go's default net/http string. On its own, that identifies a client library, not a threat actor — as Bachrach put it, "it doesn't say much, except that they wrote their tools in Golang. Go is one of the two 'go-to' languages hackers use for their toolset — the other one being Python." What makes it meaningful is context: Experience Cloud sites and ServiceNow portals are built to be driven by browsers. A guest session arriving via Go-http-client is unusual enough to warrant investigation. Salesforce Aura: Enumerating the Guest Context Every Experience Cloud site has a persistent Guest User — a real identity that unauthenticated visitors execute as. It cannot be deleted, and requiring login on the site doesn't remove the underlying profile, its sharing rules, or any code running in its context. Whatever the guest identity is authorized to read may be reachable by an unauthenticated internet caller. Aura, Salesforce's older Experience Cloud framework, has a single endpoint — /aura (also /s/sfsites/aura) — that accepts a POST containing a descriptor and parameters. Reco observed high-volume guest requests against two actions: HostConfigController/ACTION$getConfigData — enumerates the objects reachable from the guest context (Account, Contact, Case, Lead, and so on).SelectableListDataProviderController/ACTION$getItems — pages through records for each object surfaced by the first call. One target generated more than 560,000 events from the campaign IP across the observation window, almost entirely attributable to guest Aura enumeration via these two actions. At that volume, the activity is consistent with systematic enumeration and potential large-scale extraction rather than ordinary application use. LWR and GraphQL: The Surface Aura Tooling Misses Lightning Web Runtime is Salesforce's newer Experience Cloud framework, and its /aura endpoint is disabled entirely. Tooling built to detect Aura enumeration — which describes most public and open-source Experience Cloud scanners — finds nothing on a pure LWR site. Not because the site is safer. Because the tooling wasn't built to look at the surface LWR actually exposes. That surface is the UI-API, under /webruntime/api/services/data/{version}/, backing both REST and GraphQL. Guest access to the entire surface is governed by one Experience Builder preference — "Allow guest users to access public APIs" — distinct from both the guest profile's "API Enabled" permission and the site's general login-required visibility toggle. Confusing these three is a common misconfiguration; disabling the wrong one leaves the UI-API fully reachable while an admin believes the site is locked down. The chain: Plain Text Guest User → LWR site → /webruntime/api/services/data/{version}/ → GraphQL or REST UI-API → Object / Field-Level Security / Sharing Rules → Returned records Reco observed guest POST requests to /webruntime/api/services/data/vNN.0/graphql, with the operator's tool stepping through consecutive API versions — v56.0 through v66.0 — against every LWR site it discovered. A representative schema-enumeration query: Plain Text query { uiapi { query { EntityDefinition(first: 2000) { edges { node { QualifiedApiName { value } KeyPrefix { value } } } } } } } That returns every object name the guest context can query — the LWR equivalent of Aura's object map, but more complete. Record queries then follow the same authorization model as Aura: object permissions, field-level security, and sharing rules on the guest profile determine what comes back. Salesforce's own GraphQL documentation confirms this directly: queries are evaluated against the object- and field-level permissions of the executing user, which for a guest session means the guest profile. Proportionally, LWR traffic was lighter than the Aura flood — a handful of requests per version per subsite. Reco reads this as the operator treating LWR as a secondary technique, consistent with Aura sites still being more common across Experience Cloud generally. How to Distinguish Automation From Legitimate API Traffic I asked Bachrach how Reco distinguished this from a legitimate, if unusual, frontend implementation calling the UI-API directly. His answer is a detection principle worth generalizing: individual indicators are weak alone, but decisive in combination. First, GraphQL activity from a guest user is unusual to begin with — a frontend component could in theory call it directly, but it's rare enough to warrant a second look on its own. Second, the requests carried Go-http-client/1.1 throughout, never a browser string, across the entire campaign window. Third, the request stream lacked everything a browser normally generates alongside API calls — HTML page loads, JavaScript asset retrieval, the general traffic a human session produces. Fourth — what Bachrach called the "final nail" — the operator systematically walked API versions from v56.0 through v66.0, a sequence no legitimate client has a reason to produce. Individually, each observation is explainable in isolation. Together, on the same source, against the same endpoint, they leave little room for an innocent explanation. That's the model worth adopting for your own detection engineering: correlate client fingerprint, endpoint sensitivity, request sequence, and surrounding traffic pattern — don't let any single one carry the conclusion. Self-Registration as a Second-Stage Opportunity Alongside enumeration, the tool appended /SiteRegister and /CommunitiesSelfReg to nearly every Experience Cloud path it discovered — consistently, across most Salesforce targets, which is what makes it a deliberate part of the methodology rather than incidental noise. The objective: determine whether self-registration is enabled. If it is, an anonymous guest can promote itself into an authenticated external user, and external users routinely see meaningfully more than the guest profile does. The relevant defensive question isn't only whether self-registration exists — it's what a successfully registered identity actually gains. If registration unlocks additional records, search sources, files, or workflow access, the registration flow is part of the attack surface, not a separate concern. ServiceNow's Hidden Search Surface The second major surface is ServiceNow's Service Portal. The operator's tool first loads the portal landing page — GET /$sp.do?...&id=landing — then concentrates nearly all remaining volume against one endpoint: HTML POST /api/now/sp/search?sysparm_cancelable=true This is native platform Java. It doesn't appear in any customization table, isn't visible in Studio, and ServiceNow publishes no API reference for it. It is, however, exactly what the stock Service Portal typeahead widget calls. Reco reverse-engineered the request shape from that widget's client controller: JSON POST /api/now/sp/search?sysparm_cancelable=true Content-Type: application/json { "query": "password", "portal": "sp", "page": "homepage", "source": ["kb", "sc"], "include_facets": false, "searchType": "typeahead", "count": 5 } The source field determines which search sources are invoked and is required — omit it, and the endpoint returns zero results with no error explaining why. I asked Bachrach what initially drew Reco's attention to an endpoint this undocumented. The trigger was correlation, not the endpoint in isolation: "After discovering the Salesforce attack, we checked that IP and its activity. Seeing the same IP hammering a specific ServiceNow API was interesting, and we knew we had to investigate it." As with LWR, the endpoint can be used entirely legitimately in a normal browser session; the user-agent is what separated this traffic from that baseline. Why HTTP 201 Is Not an Access-Control Signal This is the finding I'd flag as most operationally important for ServiceNow admins. The endpoint does not gate on authentication at the transport layer. An authenticated request and a fully anonymous one both return HTTP 201. What differs is the response body and two headers — X-Is-Logged-In and X-Is-Visitor — not the status code — a distinction Reco's own captures, shown below, make directly. An authenticated request against a readable catalog source returns real results: JSON { "result": { "results": [ { "name": "Password Reset", "type": "sc", "table": "sc_cat_item", "sys_id": "29a39e830a0a0b27007d1e200ad52253", "short_description": "Request a reset of a password for a service or an application." } ], "total_number_results": 3 } } The identical request with no Authorization header and no session cookie also returns 201, with X-Is-Logged-In: false and X-Is-Visitor: false, and an empty result set: JSON { "result": { "results": [], "additionalResults": [], "facets": {}, "$$uiNotification": [], "total_number_results": 0 } } I asked Bachrach whether any telemetry resolves the resulting ambiguity — response time, payload size, anything deterministic separating "nothing matched" from "you were blocked." He was direct about the limit: "there's no deterministic way to conclude that except for checking the configuration of that instance or, better yet, running it yourself on that endpoint." The empty 201 is genuinely uninformative in both directions. To an operator sweeping the endpoint with varying query terms, an access-denied empty result and a genuinely-no-matches empty result look identical — so they learn what's exposed by watching which queries eventually come back non-empty. To a defender watching status codes alone, a portal returning 201 all day to anonymous callers looks the same whether it's leaking data or fully locked down. Where ServiceNow Authorization Actually Happens The access decision lives entirely behind the endpoint, in the search sources wired to a portal. Three tables matter: sp_portal – the Service Portals themselves; note which are reachable without login.m2m_sp_portal_search_source – the join between a portal and the search sources it actually exposes.sp_search_source – the source definitions, either table-backed or scripted (is_scripted_source). ServiceNow's current documentation confirms this architecture directly: search sources can be configured against tables or built with custom data-fetch scripts, and administrators can apply user criteria to control who is permitted to view a given search source. Reco's comparison of two stock sources illustrates the range of outcomes. The Catalog source (sc) opens with an unambiguous, code-level gate, then re-checks per item: JavaScript var results = []; if (!gs.isLoggedIn()) return results; // ... then, per candidate item: if (catalog_item.canViewOnSearch()) { /* include */ } The Knowledge Base source (kb) has no equivalent gs.isLoggedIn() check anywhere in its script. It calls directly into new KBPortalServiceImpl().getResultData(request), and the only control between an anonymous request and KB content is whatever "Can Read" user criteria are attached to that knowledge base — a data configuration decision, not a code-level gate, and the script gives no indication either way of whether that configuration is safe. The specific pattern Reco recommends hunting for in user_criteria: any record that is active = true, advanced = false, with every scoping field — role, user, group, company, department, location — left empty. That combination resolves to true for the guest identity exactly as if public access had been explicitly granted. The built-in Any User and Any user for KB seed records that ship on every instance, with the same fixed sys_id values across deployments, are precisely this pattern. One caveat from Reco's methodology: a criteria record with advanced = true and empty scoping fields is governed by its script rather than unconstrained, and shouldn't be flagged on the empty-fields heuristic alone. Correlating Activity Across Platforms I asked Bachrach how confidently Reco could tie Aura activity, LWR activity, and ServiceNow activity to a single operator and toolset. His answer was direct: "This one was actually very easy in this case — they all originated from the same IP, a VPS, which had no legitimate activity." That's the basis for treating this as one operation rather than three unrelated anomalies: one Go binary, from one box, hitting Salesforce over two distinct frameworks and ServiceNow over a third native endpoint. Public and open-source scanning tools — AuraInspector, S-RET, CirrusGo, including the modified AuraInspector variant used in the earlier ShinyHunters campaign — don't touch webruntime at all. Whoever built this evidently researched both platforms' guest-access surfaces independently rather than adapting an existing public tool. What the Evidence Says About Attribution Reco is explicit that it doesn't know who is behind this campaign and isn't ruling anyone in or out — a position echoed in the broader reporting on the campaign as well.[^1] That restraint is worth preserving rather than reading more into the pattern than the evidence supports. On the surface, the activity resembles the previously reported ShinyHunters Experience Cloud campaign — guest enumeration of Salesforce over Aura and GraphQL. It also diverges: this operator built custom tooling rather than running a modified public scanner, and ShinyHunters has not been publicly linked to ServiceNow targeting. The Contabo infrastructure itself is generic commodity hosting, tied to no named group and absent from public threat feeds. Neither similarity nor divergence settles the question. A campaign that doesn't match a group's last observed fingerprint tells you nothing on its own — actors rewrite tooling and rent new infrastructure constantly. Reasoning from "this doesn't resemble their previous campaign" to "this must be a different actor" is a common way confident, wrong attribution gets made. One operational detail is worth noting as a soft signal, not an attribution claim: this campaign's infrastructure hasn't rotated once across the entire seventeen-month window, a different pattern from the multi-machine, rotating-range approach typically reported for other groups. Passive scanning of the box shows only SSH and a CUPS print-sharing service — no web panel, nothing dashboard-like, consistent with the box functioning purely as a scanner. Its SSH build has sat unpatched across the observation window, roughly a year and a half behind current. That's poor hygiene on infrastructure the operator evidently isn't worried about protecting, though it says little about skill either way — there's limited reason to harden a box intended to eventually be burned. Building Detections From Behavior, Not IOCs No single indicator in this campaign is sufficient, and building detection around one — an IP, a domain, a user-agent string — is fragile by design. The IP can be replaced. The domain can change. The user-agent is one line of code away from a browser string. What's harder to hide is the underlying behavior pattern. Signals worth correlating, drawn directly from this campaign's request patterns: Guest identity combined with GraphQL access on SalesforceGuest identity combined with any /webruntime/api/services/data/ trafficNon-browser client fingerprints against /aura, the UI-API, or /api/now/sp/searchSequential API-version probing across consecutive vNN.0 valuesHigh-volume getItems/getConfigData activity from a single guest sessionRepeated /SiteRegister or /CommunitiesSelfReg probing across many subsitesGuest-attributed POST /api/now/sp/search activity at a cadence inconsistent with human typeahead behaviorRows in syslog_transaction where Created by is guest against /api/now/sp/search, grouped and trended over time For Salesforce, this requires Event Monitoring (Shield or the standalone add-on) to pull AuraRequest and Sites event log files: SQL SELECT Id, LogDate, Interval, LogFile, LogFileLength FROM EventLogFile WHERE EventType IN ('AuraRequest', 'Sites') Within those logs, the columns that matter are USER_AGENT, CLIENT_IP, ACTION_MESSAGE on AuraRequest rows, and the request URI on Sites rows — any guest URI containing /webruntime/api/services/data/v is the LWR tell that detection built around Aura alone will miss entirely. For ServiceNow, the relevant data lives in syslog_transaction. Filtering on IP Address is 158.220.87.79 and URL starts with /api/now/sp/search, combined with AND or OR depending on whether you're isolating this actor or surveying all guest traffic against the endpoint, surfaces the pattern directly. Created by reading guest, Type as REST, and request volume climbing from tens per day into the hundreds are the markers Reco's investigation used. Output length is a useful secondary signal — rows returning meaningfully more than the empty-result baseline are the searches that returned content, worth investigating first. The One-Hour Exposure Assessment I asked Bachrach what he'd check first with limited time and nothing else to go on. Salesforce: Pull every guest-user sharing rule, list them, and check the conditions on each individually. Justify each one on its own merits, and assume by default that any share makes the underlying data public — even on a site believed to be configured securely. ServiceNow: Review Knowledge Base user criteria and scripted search sources specifically. Confirm every scripted source gates on gs.isLoggedIn() before touching data and uses GlideRecordSecure rather than a bare GlideRecord, and check whether any unscoped "Any User"-pattern criteria record is attached to a knowledge base that shouldn't be public. Neither check requires reproducing the campaign's traffic. Both require someone actually reading configuration that, in most organizations, hasn't been reviewed since the site or portal went live. As Bachrach told Dark Reading separately, "seeing an indicator does not mean sensitive data was stolen... that being said, whether it shows up or not, it's crucial to audit the environment." Remediation Salesforce. Work the guest profile down to least privilege: audit and strip guest sharing rules to the minimum the site genuinely needs to serve to anonymous visitors; remove object- and field-level access on anything the site doesn't render publicly; remove "Access Activities" from the guest profile; disable self-registration unless the site requires it; disable guest file access and member visibility. On LWR specifically, disable "Allow guest users to access public APIs" under Experience Builder → Workspaces → Administration → Preferences — a single toggle that closes both GraphQL and REST UI-API access at once, distinct from the guest's "API Enabled" permission (also worth disabling, but insufficient alone) and from the site's login-required visibility setting (which governs page access, not API access). ServiceNow. Map every guest-facing portal in sp_portal to its search sources via m2m_sp_portal_search_source, and detach anything a public portal doesn't need. For every remaining scripted source, read the actual data_fetch_script: confirm it gates on login state and uses GlideRecordSecure. For table-backed sources, check source_table, condition, and roles — a source pointing at a sensitive table with no role requirement is directly reachable by the guest. Audit kb_uc_can_read_mtom for unscoped grants, and when found, detach the specific join record rather than editing the shared user_criteria record — that record is reused across the instance, and direct edits carry blast radius well beyond the one knowledge base being fixed. What AI Agents Change I asked Bachrach whether the growing use of AI agents against Salesforce, ServiceNow, MCP servers, CI/CD systems, and internal workflows could turn these guest-accessible surfaces into an indirect attack path for autonomous systems never intended to go looking for exposed data. "This is almost guaranteed," he said. "AI agents often try anything they can. They see a Salesforce site or a ServiceNow portal — they will try to scan it using the relevant tools or methods." That's an expert assessment of emerging risk, not a claim that agents are currently exploiting this specific campaign's exposure — worth being precise about. An agent given a browsing tool, an HTTP client, and a task doesn't inherently understand an organization's intended boundary between "guest" and "authenticated" — it understands what a given request returns. The same access model becomes more significant as organizations deploy autonomous agents capable of discovering and interacting with enterprise applications on their own initiative, without a human deciding in advance which endpoints are safe to query. That's a meaningful shift in the threat model, even though it's forward-looking rather than something this campaign's evidence directly demonstrates. A guest misconfiguration that today requires a deliberately built Go tool and seventeen months of patient infrastructure could, going forward, be discovered incidentally by an agent doing something entirely unrelated to reconnaissance. Conclusion Nothing in the City-Forum campaign broke either platform. Every request behaved exactly as Salesforce's and ServiceNow's own documentation describes — GraphQL and UI-API calls evaluated against the executing user's object and field permissions, search sources returning whatever their configured user criteria allow. That's precisely what makes the finding worth taking seriously rather than filing away as a routine scanning report. The question defenders need to keep asking isn't "is this endpoint vulnerable?" It's "what is the guest identity behind this endpoint actually authorized to do, as configured today" — and that answer needs to be re-verified on a schedule, not assumed once at launch and left alone. An attacker with a single Go binary and over a year of undisturbed infrastructure found the answer to that question across a wide range of organizations before those organizations found it themselves. As guest-accessible interfaces become a surface that autonomous agents may reach independently, closing that gap stops being a lower-priority audit item. IOCs/Defensive References IP: 158.220.87.79 (ASN 51167, Contabo GmbH; rDNS vmi2213719.contaboserver.net)Domain: city-forum.com (resolving to the above IP since at least 2025-03-12; registered 2002, since abandoned)Active subdomains: city-forum.com, www.city-forum.com, server.city-forum.com, www.server.city-forum.com, mail.city-forum.com, www.mail.city-forum.comUser-agent: Go-http-client/1.1Salesforce: guest /aura calls to getItems/getConfigData; guest requests to /webruntime/api/services/data/vNN.0/graphql sweeping v56.0–v66.0; guest hits on /SiteRegister and /CommunitiesSelfRegServiceNow: guest POST /api/now/sp/search?sysparm_cancelable=true at escalating volume, Created by = guest Research and indicators referenced in this piece are drawn from Reco's City-Forum campaign investigation. Interview quotes from Nitay Bachrach were obtained in an interview arranged through Reco's PR representative. Sources: Long-running Data Theft Campaign Targeting Salesforce, ServiceNow — Dark Reading"City-Forum" data-theft attacks target Salesforce, ServiceNow portals — BleepingComputerThe "City-Forum" Campaign — Reco (original research)A stranger has been reading Salesforce and ServiceNow portals worldwide for 17 months — Help Net SecurityStealthy 'City-Forum' Attacks Target Salesforce and ServiceNow With Custom Toolset — SecurityWeekQuery Objects | Query Records | GraphQL API — Salesforce DevelopersDefine a search source — ServiceNow DocumentationApply user criteria to a search source — ServiceNow Documentation

By Igboanugo David Ugochukwu CORE

Software engineers often view soft skills as secondary, considering them relevant mainly for managers, recruiters, or those frequently in meetings, rather than essential for technical roles. However, as your career advances, this perspective becomes harder to maintain. Greater impact requires you to explain ideas, influence decisions, manage disagreements, build trust, exchange feedback, and communicate with those outside your technical context. While leadership roles highlight this need, these skills are integral to effective software engineering well before any formal leadership title. Practicing soft skills alone is challenging. While you can develop technical abilities like Java, databases, or system design independently, communication and influence require real interaction. Open source provides this environment, offering opportunities for discussions, code reviews, proposals, community meetings, documentation, conferences, and collaboration across companies, cultures, and experience levels. This article explores how open source can serve as a practical training ground for the communication and interpersonal skills essential for technical leaders. How Open Source Builds Soft Skills Through Real Collaboration This article will not revisit the importance of soft skills for software engineers, as that topic has been addressed elsewhere. Instead, it focuses on practical ways to develop these skills. While hard skills can be practiced independently, soft skills such as communication, influence, trust, empathy, and collaboration require interaction with others. Open source offers a natural and consistent environment for this development. Collaborate Across Cultures and Perspectives A key benefit of open source is the opportunity to collaborate with individuals from diverse cultures, organizations, backgrounds, and perspectives. Contributors may disagree due to differing technical opinions, communication styles, risk priorities, or problem-solving contexts. Learning to work productively in this environment builds a key leadership skill: transforming diverse perspectives into better decisions rather than unnecessary friction. This experience is especially valuable for technical leaders, as broader responsibilities mean working with people who think, communicate, and operate differently. Learn to Present and Defend Your Ideas Open source also requires you to communicate your ideas clearly. A proposal is rarely accepted on technical merit alone. You must explain the problem, provide context, outline trade-offs, answer questions, address criticism, and often revise your proposal before gaining community approval. This is remarkably similar to proposing an architectural or design decision inside an organization. Translating your ideas into concepts others can understand, discuss, and support is a critical skill in technical leadership. Navigate Politics and Build Agreements Wherever people collaborate, interests, priorities, relationships, and politics will be present. This is not inherently negative. Politics often arises because people value different aspects, such as backward compatibility, developer experience, performance, or long-term maintainability. Technical leaders must understand others’ perspectives before advancing decisions. This is especially important in software architecture, where there is rarely a single correct answer. Many decisions depend on context; what works well in one situation may be unsuitable in another. When no clear answer exists, technical knowledge alone is not enough to resolve the discussion. You must negotiate trade-offs, understand others’ priorities, reach agreements, build consensus, and sometimes accept solutions that differ from your initial preference. Open-source communities regularly present these situations. Advancing technical initiatives often requires understanding both the architecture and the people involved. Navigating these dynamics without escalating disagreements into conflicts is essential for technical leadership. Improve Your Spoken Communication Technical leadership extends beyond written proposals. Community calls, working groups, meetups, podcasts, workshops, and conferences provide opportunities to communicate technical ideas verbally. If you need to advocate for an architectural decision within your company, you will use many of the same skills practiced in open source: structuring arguments, explaining complexity, adapting to your audience, answering challenging questions, and remaining constructive under scrutiny. Speaking is integral to engineering leadership. It helps transform technical ideas into organizational decisions. Learn to Coordinate Software Delivery Open source can also develop strong project and delivery skills. Mature projects require releases, versioning strategies, roadmap planning, issue prioritization, dependency coordination, estimation, and clear communication about deliverables and timelines. Participating in these activities teaches that software delivery is as much about coordination as technical execution. As projects grow, understanding dependencies, priorities, expectations, and others' contributions becomes increasingly important. These concerns are common for staff engineers, architects, and technical leaders as their responsibilities expand beyond a single team. Learn Leadership Without Formal Authority One of the most notable aspects of open source is that many contributors are volunteers. You cannot rely on hierarchy, salary, or reporting structures to motivate contributions. You need to build trust. You need to make people feel that their contribution matters. You need empathy when someone cannot complete a task. You must communicate expectations respectfully, recognizing others’ autonomy over their time. You also need to foster an environment that encourages people to return. That makes open source a particularly interesting leadership laboratory. If you can help create momentum among people who are free to walk away at any moment, you are practicing a form of leadership based on influence rather than authority. Build Trust and Empathy Trust is a key currency in open-source communities. People gradually assess whether you review fairly, listen before disagreeing, acknowledge contributions, keep commitments, and argue in good faith. This reputation is built through consistent interactions. The same principle applies inside an organization. Engineers are much more likely to follow somebody they trust than somebody who merely has a more senior title. Open source offers repeated opportunities to learn how trust is earned, maintained, and sometimes lost. Practice Soft Skills Through Real Situations This is what makes open source especially valuable for developing soft skills. You are not practicing communication through hypothetical exercises; you are communicating because an actual proposal requires approval. You are not simply reading about conflict resolution; you are addressing real disagreements. You are not studying influence theoretically; you are persuading people who have no obligation to agree with you. This distinction is important. You can practice hard skills alone. Soft skills require interaction, and open source provides real people, real challenges, and ongoing opportunities to develop leadership. Conclusion Soft skills are not an optional layer added on top of technical expertise; they are part of what allows that expertise to create impact. Open source gives software engineers a real environment to practice communication, persuasion, negotiation, collaboration, trust, empathy, public speaking, and even the politics that naturally emerge when people with different priorities need to make decisions together. This is especially important in areas such as software architecture, where many choices live in a gray area, and the final direction depends not only on technical knowledge, but also on context, trade-offs, and the ability to build agreement. For engineers who want to grow into staff engineer, principal engineer, software architect, or technology leadership roles, this kind of practice is invaluable. You can study communication theory, but eventually you need people to communicate with; you can read about influence, but eventually you need a real disagreement to navigate. Open source creates those opportunities repeatedly and at scale. It helps transform soft skills from abstract concepts into practical leadership capabilities that can make your technical knowledge more understandable, more trusted, and ultimately more influential.

By Otavio Santana CORE

In this blog, you will take a closer look at the different exchange types that can be used in RabbitMQ. All are demonstrated by means of examples in a Spring Boot application. Enjoy! Introduction In the previous blog, you learned the basic concepts of RabbitMQ and how to use it in a Spring Boot application. However, you only scratched the surface of it, so now it is time to dig a bit deeper into the different exchange types. If you are not yet familiar with the basic concepts, it is advised to read the previous blog. The official RabbitMQ documentation also provides detailed information that is worth reading. Sources used in this blog can be found on GitHub. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose;Basic knowledge of RabbitMQ. Topics The code can be found in the topics module. In the previous blog, you created two consumers A and B. Consumer A was bound to Queue A with routing key event.general.*. Consumer B was bound to Queue B with routing keys event.general.* and event.specific.*. The asterisk (*) wildcard was used and is a substitute for exactly one word. In the examples, the routing keys event.general.message and event.specific.message were used. You can also use the hash (#) wildcard, and this is a substitute for zero or more words. This is visualized in the figure below. In the RabbitMqConfig, you declare queue C and bind it to the TopicExchange with routing key event.general.#. Java public static final String QUEUE_CONSUMER_C = "consumer-c.queue"; public static final String ROUTING_KEY_NESTED_GENERAL_MESSAGE = "event.general.#"; @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } @Bean public Queue queueConsumerC() { return new Queue(QUEUE_CONSUMER_C, false); } @Bean Binding bindingConsumerCNestedGeneral(Queue queueConsumerC, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerC).to(exchange).with(ROUTING_KEY_NESTED_GENERAL_MESSAGE); } In the MessageController, you create an endpoint for sending a message with routing key event.general.message.nested. This routing key will not match the bindings of consumers A and B. Java @RequestMapping( method = RequestMethod.POST, value = "send-nested-general" ) public ResponseEntity<Void> sendNestedGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message.nested", message); return new ResponseEntity<>(HttpStatus.CREATED); } The ReceiverC listens to messages received in queue C and prints a message. Java @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_C) public void receiveMessage(String message) { System.out.println("Queue Consumer C received <" + message + ">"); } } Start the application from within the topics module. Shell mvn spring-boot:run First, post a general message; this should be received by all consumers. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" In the application console log, you notice that all consumers receive the message. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Queue Consumer C received <This is a general message> Now, post a nested general message, which should be received only by consumer C. Shell curl -X POST http://localhost:8080/send-nested-general \ -H "Content-Type: text/plain" \ -d "This is a nested general message" In the application console log, you notice that the message is only received by consumer C. Plain Text Queue Consumer C received <This is a nested general message> Work Queues The code can be found in the work module. With work queues, you can publish a message and dispatch it to a pool of consumers. One of the consumers will pick up the message and start processing it. This is especially useful for dispatching long-running tasks. You use the default direct exchange in this case, and the queue name is used as the routing key. No need to use a custom exchange. This is visualized in the figure below. The RabbitMqConfig is quite small; you only define the queue. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_TASK = "task.queue"; @Bean public Queue queueTask() { return new Queue(QUEUE_TASK, false); } } When sending a message via an endpoint, you use the queue name as the routing key. Java @RequestMapping( method = RequestMethod.POST, value = "send-work" ) public ResponseEntity<Void> sendWorkMessage(@RequestBody String message) { messageService.sendMessage(RabbitMqConfig.QUEUE_TASK, message); return new ResponseEntity<>(HttpStatus.CREATED); } Every consumer listens to the queue. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer A <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer B <" + message + ">"); } } @Component public class ReceiverC { @RabbitListener(queues = RabbitMqConfig.QUEUE_TASK) public void receiveMessage(String message) { System.out.println("Task picked up by Consumer C <" + message + ">"); } } Start the application from within the work module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-work \ -H "Content-Type: text/plain" \ -d "This is a work message" The message is processed by one consumer. Plain Text Task picked up by Consumer A <This is a work message> Fanout The code can be found in the fanout module. With fanout, you want to broadcast messages to all queues. You send messages to the exchange, but there is no need to specify a routing key. You can also ensure that temporary queues are used. When temporary queues are used, the queue name will be generated. In the RabbitMqConfig, you define a FanoutExchange. The queues are defined as an AnonymousQueue. This creates a non-durable, exclusive, auto-delete queue with a generated name. You bind the queues to the exchange. Java @Configuration public class RabbitMqConfig { public static final String FANOUT_EXCHANGE_NAME = "fanout.exchange"; @Bean FanoutExchange fanoutExchange() { return new FanoutExchange(FANOUT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new AnonymousQueue(); } @Bean Binding bindingConsumerA(Queue queueConsumerA, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange); } @Bean public Queue queueConsumerB() { return new AnonymousQueue(); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, FanoutExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange); } } In order to send messages, you only need to send them to the exchange. This can be seen in the MessageService. Java public void sendMessage(String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.FANOUT_EXCHANGE_NAME, "", message); } On the receiving side, you listen to the generated queue name (thus not a specific one in this case). Java @Component public class ReceiverA { @RabbitListener(queues = "#{queueConsumerA.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } @Component public class ReceiverB { @RabbitListener(queues = "#{queueConsumerB.name}") public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Start the application from within the fanout module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-to-all \ -H "Content-Type: text/plain" \ -d "This is a fanout message" In the application console log, you notice that the message is consumed by all queues. Plain Text Queue Consumer B received <This is a fanout message> Queue Consumer A received <This is a fanout message> RPC The code can be found in the RPC module. Remote Procedure Call (RPC) can be used when you need to execute a function on a remote application and wait for the result. The event is sent to the queue and is processed by Consumer A. The result is sent to a queue in the replyTo field of the request. The publisher waits for data to be returned on this callback queue. When the message appears, it checks the correlationId. If it matches the value of the request, the response is returned to the publisher. All of this is done automatically by the RabbitTemplate. In the RabbitMqConfig, a DirectExchange is used. With a DirectExchange, you match exactly on events; you cannot use wildcards here, just like a TopicExchange. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String DIRECT_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_RPC_MESSAGE = "event.rpc"; @Bean DirectExchange eventsExchange() { return new DirectExchange(DIRECT_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, DirectExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_RPC_MESSAGE); } } The MessageController contains an endpoint for sending the event. Java @RequestMapping( method = RequestMethod.POST, value = "send-rpc" ) public ResponseEntity<Void> sendRpcMessage(@RequestBody String message) { messageService.sendMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you use convertSendAndReceive and process the response. Java public void sendMessage(String message) { Object response = rabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } } In the receiver, you receive the message and send a response. Do note that some additional processing is added in order to trigger a timeout. More on that in a moment. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public String receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); if (message.equals("This is an rpc message")) { return "success"; } else if (message.equals("This is a timeout message")) { try { Thread.sleep(10000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "success"; } else { return "failure"; } } } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is an rpc message" In the application console log, you notice that the message is consumed by consumer A, and that a successful response is received by the publisher. Plain Text Queue Consumer A received <This is an rpc message> Sender received response: success But what if it takes too long to process the message? In real life, the remote application can be unreachable for one reason or another. Send a timeout message. Shell curl -X POST http://localhost:8080/send-rpc \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the MessageService, the response will return null, and a timeout exception is raised. Plain Text Queue Consumer A received <This is a timeout message> No response received 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] o.s.amqp.rabbit.core.RabbitTemplate : Reply received after timeout for 2 2026-04-25T14:50:16.785+02:00 WARN 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed. org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted 2026-04-25T14:50:16.790+02:00 ERROR 482297 --- [MySpringRabbitMqPlanet] [pool-2-thread-8] .l.DirectReplyToMessageListenerContainer : Failed to invoke listener org.springframework.amqp.rabbit.support.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1795) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1687) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1612) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1599) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1590) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1539) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1520) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1206) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:1163) ~[spring-rabbit-4.0.2.jar:4.0.2] at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) ~[amqp-client-5.27.1.jar:5.27.1] at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111) ~[amqp-client-5.27.1.jar:5.27.1] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na] Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Reply received after timeout at org.springframework.amqp.rabbit.core.RabbitTemplate.onMessage(RabbitTemplate.java:2721) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.lambda$setMessageListener$0(DirectReplyToMessageListenerContainer.java:93) ~[spring-rabbit-4.0.2.jar:4.0.2] at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1683) ~[spring-rabbit-4.0.2.jar:4.0.2] ... 12 common frames omitted How to solve this? In this case, you are better off using the AsyncRabbitTemplate. This template is not automatically autowired, so you have to define it as a bean. Let's do so in the RabbitMqConfig. Java @Bean public AsyncRabbitTemplate asyncRabbitTemplate(RabbitTemplate rabbitTemplate) { return new AsyncRabbitTemplate(rabbitTemplate); } In the MessageController, you define an endpoint to trigger the async template. Java @RequestMapping( method = RequestMethod.POST, value = "send-async" ) public ResponseEntity<Void> sendAsyncMessage(@RequestBody String message) { messageService.sendAsyncMessage(message); return new ResponseEntity<>(HttpStatus.CREATED); } In the MessageService, you autowire the AsyncRabbitTemplate. And because it is an async call, you catch the response by means of a CompletableFuture. Java public void sendAsyncMessage(String message) { CompletableFuture<Object> future = asyncRabbitTemplate.convertSendAndReceive(RabbitMqConfig.DIRECT_EXCHANGE_NAME, ROUTING_KEY_RPC_MESSAGE, message); future.thenAccept(response -> { if (response != null) { System.out.println("Sender received response: " + response); } else { System.out.println("No response received"); } }); } Start the application from within the rpc module. Shell mvn spring-boot:run Send a message to the queue. Shell curl -X POST http://localhost:8080/send-async \ -H "Content-Type: text/plain" \ -d "This is a timeout message" In the application log, you see the same result: the response is null, but no timeout exception anymore. Conclusion In this post, you learned different exchange types. Each serves its own use case. It is up to you to choose the right pattern for your use case.

By Gunter Rotsaert CORE

Open source projects dominated by a single vendor are a hallmark of "open source in name only." Rather than filling the traditional role of open source fostering innovation and decision-making from a diverse community, "open source in name only" projects are often used as marketing tools for proprietary platforms. These projects are also seen as riskier than community-driven projects because a single vendor is more apt to abruptly terminate long-term support, restrict contributions, or switch from an open-source license to a more restrictive one (forcing some previous contributors to pay for the project they helped build). In these projects, critics claim that investments are often lopsided and heavily skewed toward onboarding, marketing, and brand-related support. As a result, technical contributions are frequently less developed, opaque, undocumented, or lacking in real substance, often manifesting merely as a superficial "ease of entry and onboarding." Because of these underlying gaps in documentation and codebase depth, developers are routinely forced to reverse-engineer functionality simply to get the tools to work correctly. An evaluation of three leading open-source observability projects–OpenSearch, Prometheus, and OpenTelemetry (OTel)– by ReveCom was conducted to determine whether they fell under this vendor-dominated category or are truly vibrant community-led projects. According to Gartner research, these three projects are collectively important because together they provide a complete, vendor-neutral observability architecture covering all three fundamental telemetry signals—metrics, logs, and distributed traces — without locking an enterprise into proprietary agent formats or single-vendor cloud platforms. Gartner defines observability as the extent to which internal system states can be inferred from externally emitted data. By pairing OpenTelemetry as a universal collection and routing tier with Prometheus for real-time metric alerting and OpenSearch for high-volume log analytics and trace analysis, organizations gain end-to-end operational visibility, retain full ownership of their telemetry pipelines, and avoid runaway cloud ingestion or lock-in costs. To develop the framework, data from the ReveCom Observability Report 2026 was used, which includes metrics about contribution numbers and quality, including commit frequency, contributor growth, community expansion, and deployment patterns. Based on this data, authentic efforts were separated from perfunctory efforts. "Authentic" contributions were defined as those made to the computing code (i.e., the observability stack for logs, traces, and metrics) and its computational efficiency as measured in latency. The Controversial Fork AWS's controversial decision to monetize and then fork Elasticsearch to create OpenSearch in 2021 (when Elastic made its license more restrictive) is a case study of the risks associated with vendor-dominated projects. It also serves as an example of the issues associated with a vendor forking and heavily promoting a project it contributed minimally to. According to Elastic representatives, although a major beneficiary of Elastic through its managed service, AWS engineers contributed only a "handful" of commits to Elasticsearch from 2020 to 2021, Elasticsearch says. This disparity suggests that the successor project, OpenSearch, was born from a position of minimal technical familiarity with the core codebase. Elastic famously described this as "there is no compression algorithm for experience." For a technical leader, this lack of pre-fork familiarity suggests a significant "experience gap" that can impact the speed and stability of future feature releases. AWS made few fundamental changes to the Elasticsearch codebase it forked to create OpenSearch, largely just rebranding the existing observability tool. Comparing Three Observability Communities In 2024, Amazon donated OpenSearch to the Linux Foundation, bringing it under a governance structure and setting the stage for it to become a more decentralized project. Among other things, once a project is donated to the Linux Foundation, no single company can hold more than 25% of the seats on the technical oversight bodies. Decentralized governance is structured so that substantive, collaborative contributions from several competing observability vendors can better serve the broader community's needs. Amazon's donation set the stage for OpenSearch to become a much more community-driven effort, comparable to the community-led support of the Prometheus and OpenTelemetry projects. Prometheus and OpenTelemetry exemplify healthy, community-led open source standardization. This is how teams should evaluate open source: by the diversity of the entities with "skin in the game." Prometheus emerged from SoundCloud in 2012, where it was designed to track metrics and store them in a time-series database. Around 2014, Grafana and its glassy, visually appealing panels became part of the ecosystem. The combination of Prometheus and Grafana became an integral, de facto standard for monitoring and observability in Kubernetes deployments and infrastructure. Prometheus was donated to the CNCF in 2016 and graduated in 2018. Since then, it has evolved into a very diverse, community-led project, with multiple contributing companies. Grafana Labs remains one of the largest contributors, but the breakdown of substantive commits-excluding documentation-is wide and varied, reflecting the project's broad, collaborative nature. This wider contribution to the project's standardization ensures that engineering talent is portable and the stack remains interoperable. Separating Brand From Backbone A key open source health metric-perhaps the most substantial of all-is ranking substantive engineering contributions, such as code-level commits and pull requests or high-impact technical commits. These are described as commits that can lead to v1.0, v2.0, or v3.0 milestones, signifying production readiness and improvements. The number and frequency of technical contributions, as measured by commits, are markers for a project's community dynamics and value to end users. Looking at OpenSearch, AWS made significant technical contributions in 2025. As the data shows, Amazon contributes the majority of substantive commits (73%) to OpenSearch. Much of this can be attributed to a surge in contributions related to the AI aspects of observability, specifically "search-to-Al infrastructure" commits. IBM and Red Hat have also contributed AI-related work on RAG and vector database optimization. These are solid contributions, and they show that Amazon has moved beyond the early days, when it simply forked Elastic even though it had contributed relatively little to the project. Hopefully, OpenSearch will continue this shift toward increased community participation as new features are added. However, such a dominant share of commits from a single vendor means that one vendor effectively controls the roadmap. In this case, AWS is potentially prioritizing its managed services over users' infrastructure needs. Source: ReveCom Prometheus has a wide range of contributions from vendor organizations. Grafana is the leading technical contributor to Prometheus, largely based on its development of TSDB storage refactoring, Remote Write 2.0, and agent-mode contributions. Red Hat is the second-most frequent technical contributor to Prometheus, a position solidified by its acquisition of CoreOS. As the primary maintainer of the Prometheus Operator-a critical element for monitoring Kubernetes-Red Hat ensures seamless integration between the monitoring stack and the orchestration layer. While Red Hat provides deep engineering support, Prometheus remains a highly collaborative open-source project with contributions from across the industry. Source: ReveCom The OpenTelemetry project, under the leadership of Splunk, Microsoft, Elastic, Grafana Labs, and Google, provides a mature, stable, and innovative framework for the future of observability. By focusing on high-impact technical commits and "good faith" participation, the community helps ensure that observability data remains a standardized utility that empowers developers and platform engineers to navigate the complexities of the modern cloud landscape. Splunk remains the largest contributor of high-impact technical commits to OpenTelemetry. Grafana is a notable contributor at number four by providing Beyla eBPF instrumentation and Prometheus receiver stability improvements. Strategic Recommendations Organizations should adopt an open-source technical strategy that prioritizes authentic engineering and project diversity. The following recommendations are derived from scrutinizing vendor-dominated projects and analyzing high-impact technical commitments. The high-impact focus of companies like Grafana Labs, Splunk, Microsoft, Elastic, and hundreds of other contributor organizations means that OpenTelemetry and Prometheus should remain the foundation of observability for the next several years. When choosing an observability solution, organizations should prioritize vendors that are not only OTel-compliant but also OTel-contributing. This should also apply to Prometheus solutions, especially those for managing Kubernetes environments. ReveCom's findings indicate that the most valuable contributions are those that advance the core "engine" of observability. Procurement decisions should be based on a vendor's ability to demonstrate substantive engineering that solves real-world infrastructure problems rather than relying on superficial marketing claims. Ultimately, none of the three projects covered in this article can be fully characterized as "open source in name only." While OpenSearch arguably fell into that category immediately after it was forked from Elasticsearch, it has evolved since. OpenSearch remains an Amazon-dominated project, but it has seen an upward trend in contributions from the community and from third parties such as Uber, SAP, and Red Hat. For observability community support, as measured by substantive technical contributions that solve infrastructure problems, OpenTelemetry and Prometheus exemplify a healthy balance of governance and code contributions across hundreds of organizations (notably Grafana and Splunk). Led by Grafana and Splunk among the observability providers, these projects fall behind only Kubernetes itself.

By Chris Ward CORE