Monitoring and Observability

Originally, back-end and front-end Site Reliability Engineering (SRE) were owned by teams. They code the programs, set up databases and infrastructure, and quickly spring to action at the beep of any anomaly. The advent of code vs no-code infrastructure, SaaS, API dependencies, third parties, and other modern systems seems to be eroding this authority. Mainstream and underdog companies now often leverage the significant advantages of outsourcing, collaboration, or delegation, which are usually accompanied by a silent clause: no or partial control. Unlike in previous systems, modern production is largely assembled rather than built from scratch. For example, a conventional SaaS product is built on interdependencies among payment processors, outsourced data infrastructure such as Amazon Web Services (AWS), messaging services, web hosting, design, AI inference APIs, authentication providers like Google, and more. These useful platforms and products are essentially outside teams' control stations, even though they critically impact users' experience. When they function effectively, you share the glory with the platforms. But when there is a system blackout, your users put you on your toes, even though you have no direct access to resolve the problem on time. Therefore, we shall be exposing SRE practices in platform-SaaS and API-dependent systems and how reliability is getting beyond the control of engineering teams and companies. Why Classical SRE Practices May Fail One major downside of SaaS and dependency on external platforms is that reliability control is often assumed to be in a team's hands, whereas it has been bargained. However, teams must reckon with the fact that the case is reversing. For example, traditional SRE models once alleged that: Service Level Indicators (SLIs) focus on availability or internal uptime and latency.Error budgets arise from changes teams make or deploy.Runbooks still suggest that teams can immediately reconfigure or directly work on faulty components. All these are becoming past cases, especially in platform-SaaS systems. You can have a system indicating 99.99% or even 100% uptime on the back end, while new users are struggling to sign up, probably because an authenticator provider is not fully functional. Dashboards and control panels may indicate green, but in reality, third-party payment APIs have been degraded. A New Definition of Reliability in Operating SRE Practices To resolve the new problem in site reliability engineering (SRE), there needs to be a conceptual shift from component health to an integrated, continuous user experience. Therefore, teams need to undergo a paradigm shift away from questions such as "Is our CPU working maximally?" “Is our API up?” “What are the error rates?” Instead, we should inquire: “Are users checking out seamlessly?” “How fast can they authenticate?” “Can they use the SaaS product to perform its key function?” These types of outcome-based questions span interdependent platforms beyond your full control. The login SLI needs to work with the identity provider; otherwise, its output is meaningless. If the checkout SLO skips payment authorization, then it's both fishy and unreliable. True, there may be some internal errors in a reliable system, but what really matters is an integrated multiplatform experience that the user enjoys. Error Budgets? An SRE Practice to Revisit How many teams would love error budgets to disappear when they give up control? But that’s not so. Instead, they are molecularized. When components of your systems are outsourced, the error budget doesn’t just fade away; it is instead transferred to the interdependent platforms. So, it’s better to plan for the fact that SaaS and API providers will consume some of your reliability budget. Doing so keeps you a few steps ahead and protects your business in the long run. Reliable SRE teams make decisions such as allocating part of their error budget to certain dependencies, setting acceptable parameters for degradation, and defining specific steps to take when a dependency exceeds the stipulated budgets. Here’s an example you can adapt: “We will accept payment authorization failure of 0.0% to 0.2% if it is caused by dependency instability. If it goes above that, we will turn on delayed capture or turn off promotions.” This SRE approach keeps you ready for downtime, as your systems automatically switch to planned or budgeted actions rather than relying solely on integrated platforms. What to Do When Failures Beyond Your Control Arise Actually, some failures may seem beyond your control. The more you attempt to resolve them, the more amplified they become. At this point, your team must adapt to the savvy absorption of such situations. Instead of focusing solely on retrial in an SRE approach, your team needs to design its processes and platforms. This could include failing selectively through circuit breakers, failing fast with timeouts, or failing visibly by keeping users informed. Some core settings should always remain non-negotiable and on standby. These could include the following: Read-only modes/cachesBulkheads that prevent a failure avalanche.Automated circuit breakersDeferred processing These reliable practices ensure there is some form of controlled uptime even when operations seem interrupted. Laser Observability That Proves Reliability In traditional SRE observability, the service boundary is usually the ultimate, but in most modern integrated SaaS platforms, this could be insufficient or worse, dangerous. Operators need to be aware of the actual dependency that is failing, how it is failing (e.g., errors or throttling), and how the failure affects the user experience. Accurate observability for platform-SaaS and API-dependent systems requires these four provisions: Specific dashboard and internal metrics for each vendor.SLI monitoring at the dependency level.Parallel tracing of all outbound calls.Simulation of real-time user experience and workflows. Essentially, whenever there is an emergency, operators should be able to promptly identify whether the source is internal or external. Accuracy and clarity facilitate swift response. Responding to Incidents Without Ownership Another distinct characteristic of modern SRE practice in platform-SaaS is how incidents are responded to. Without ownership, you often cannot debug on your own, roll back a bad deploy, or directly manage other issues. However, you can choose how your system responds by identifying when certain features are disabled, when signals to activate degraded modes are sent, when high traffic is redirected or shed, or when to notify users. To maintain reliability, incident response relies on runbooks to inform decisions. The following questions could help convert the technicality of runbooks to practical solutions: What is the impact on the customer?In what ways can we respond harmlessly?What can we reverse?What should we communicate externally? These questions help resolve incidents, mitigate losses, and intertwine reliability with sound judgment. Is Safety an Illusion in SLAs? SLA providers often readily contract for financial compensation when losses arise, but seldom give absolute reliability guarantees. You may not always expect vendors to consistently meet your availability goals or resolve an avalanche of outages. Safety is a critical consideration when building systems, because when users lose trust in a brand, compensation may not be able to redeem it. Therefore, advanced teams do not consider SLAs as safety nets but as risk pricing. They understand that contractual credits cannot replace trust, brand image, and some almost irredeemable damages. Human Factors in Platform-SaaS and API-Dependent Systems Dependency failures often escalate when cognitive load increases. There could be degraded performance, timeouts without error indicators, partial success, or inconsistent system behavior. Operators may not only focus on machines when dashboards lag or seem to lie. They examine the logs, failure history, or commands. Teams have to design systems with overrides and predictable degradation paths, and observability tools are beyond the failure systems. Reliability goes beyond the correct function of software; it's also about human operations. How Your SaaS and API Platforms Can Imbibe “Good” SRE Practice Effective SRE practices are modern. The following attributes know saas products and API-dependent platforms: Acknowledgment of lack of control very early.Ensuring reliability is embedded in the design.Measuring the outcomes of each SRE criterion or target, instead of just the components.Giving priority to clarity instead of trying to model or control everything because you do not own all the components.Making engineering and operations decisions and products as an integrated whole.Preparing for degradations as inevitable procedures when things fail. Your systems can be reliable if you anticipate failure and accept the reality. Conclusion Modern platform-as-a-service (SaaS) operates in a reliability-without-control manner, leading solid SRE teams to accept that they need to adapt when failures occur. It's simple logic: if you don't absolutely own everything end-to-end, then prepare for the worst: each dependency might fail. It's all about keeping the trust of your users and protecting your brand image.

Once upon a time, site reliability engineering rested on a linear assumption: monitor more, detect early, and you’ll recover faster. The rise of alert fatigue makes modern SRE teams realize otherwise: Ramadass's (2025) paper, Building an AI-Powered Observability Pipeline for Modern System Reliability, cited research that discovered that: More than two-thirds (82%, actually) of institutions experience alert spikes constantly.Most traditional monitoring tools generate approximately 2,100 alerts daily, with about 70% of them unnecessary and safe to ignore.66% of SRE professionals stated that increased false alerts lead to fatigue, potentially causing them to miss serious issues. How Should We Describe This Situation? Vigilance or Noise? Collaborative systems such as SaaS, third-party APIs, and microservices enhance the degree of observability and notification within systems. Everything is monitored, and occasionally these dependencies may duplicate alerts. When systems request superhuman attention, on-call engineers become fatigued rather than lazy or sloppy. Instead of swift action, alerts are responded to with mistrust. Reliability vs. Experience vs. Metrics Traditional alerting metrics follow traditional reliability practices, that is, error rates, uptime percentages, latency, etc. Although these are essential, they are not actual mirrors of how operators or users experience reliability. Operators may expect reliable alerting to inform decisions, while users may simply define reliability as how well a system enables them to fulfill their intentions. If alerts do not clearly connect to the user experience, there is a gap between detection and action. Over time, the gaps lead to fatigue. On-call engineers begin to “reasonably” ignore these alerts. Why worry over alerts that are not logically related to user outcomes? They may assume. Over time, organizations may end up paying dearly for real issues because alerts were missed or delayed. An On-Call Engineer Experience Here is a typical example of a system design problem an on-call engineer or SRE team may face: 01:15 AM Alert: Latency spikes on a third-party API.01:16 AM Alert: Retry queues are filled.01:16 AM Alert: Timeout alert storms on three dependencies.01:17 AM Alert: Error-rate notification on unrelated endpoints.01:18 AM Alert: Memory and update alerts. And this sequence of alert storms continues, with the on-call engineer receiving more than 20 alerts in just four minutes. The system seems to pass standard observability SRE practice. But what about the long-run reliability suspicions that the bugging signals may create? In this case, the teams are not just grappling with response speed but also with the amplification of confusion when critical alerts are mixed with non-actionable ones. When Detection Outpaces Interpretation We can’t rule out the fact that monitoring in the past decades has taken an advanced leap. And we might be at its cloying stage, where system detection software is outpacing on-call engineers’ interpretation. Systems are “wonder-full” when it comes to identifying when something seems “off.” However, they rarely give explicit descriptions to aid SRE teams’ understanding. An alert can indicate that a queue has exceeded its depth, but may not categorically state whether the issue is temporary or actionable, or whether users are affected. This occurrence spans dozens of dependencies, each with its own signal. The on-call engineer is kept puzzled about the best action to take at the right time. Hence, a reliable response could be excessive caution or delay as the engineer seeks to clarify the situation. The users are negatively impacted. Although the system met technical observability SRE standards, it failed operationally due to its opacity. The Hidden Cost of Alert Overload We rarely see the outcome of alert fatigue overnight. Its effects build up. Delayed response time accumulates. The aftermath incident review loses credibility. Engineers are skeptical of alerts and hesitate to decide first whether they are real or false. The cultural cost of alert fatigue is that on-call roles become a burden SRE teams endure rather than enjoy with a sense of responsibility. In the long run, engineers may feel they have no control over issues due to the confusion that multiple alerts create. Ironically, the same reliability problems that alerts were designed to solve are what they quietly create. Are Alerts Creating a False Sense of Safety? Lots of alerts may seem like a good thing or a sign of strong monitoring at first glance. But here is the truth: alerts could be hiding actual risk. As every deviation is notified, critical and minor alerts blend in. Teams begin to feel alert fatigue and delay response. Then, real problems begin to breed behind the scenes. Remember how SLAs could paint an illusory picture of safety? Similarly, alert volume could do so. Therefore, your SRE team should bind these caveats as the core of their modus operandi. Alerts shouldn’t replace action.Alerts shouldn’t be unsorted (by machines or humans).Alerts shouldn't be discarded. Alerts are signs that our systems need attention, and we should never be tired of listening. SRE Teams Designing Systems that Alert Smartly High-quality systems respond efficiently when dependencies fail. Instead of creating panic, they automatically degrade. SRE teams could design circuit breakers that could inhibit alert storms before they explode. They could also install bulkheads to prevent a single failure from spreading. There could be alert limits and a summary of conditions that resolve the problem of spamming. Instead of relying on metrics, system engineers could set up composite alerts that describe system states. For instance, it’s clearer if a system alert indicates, “Checkout degraded because of latency in payment dependency.” This composite alert is better than 7 alerts that say “Checkout Timeout.” The former shows impact, cause, scope, and urgency. Clarity clears fatigue. Noise does the opposite. Redesigning SRE: Human Reliability That Quells Alert Fatigue We have seen that technical designs may be great, yet other aspects of SRE remain wanting. One such area that could resolve a system design problem is humaneness. To avoid alert fatigue, our design choices must acknowledge human limitations. Therefore, we should accept that some alerts may not require immediate response. Conversely, not every anomaly should trigger an alarm. Understood silence could sometimes be a golden sign that nothing critical is wrong. Advanced SRE teams do not focus on events (or every deviation) but on the states of the system or infrastructure. They are guided by the question: What conditions really impact users, business objectives, or the system's overall health? To achieve this, engineers need to balance product understanding with technical operations. Then they can give a human touch to their designs. Designing systems for human reliability requires a high level of discipline. Site reliability engineers have to continually review, refine, and repair alerts and their trigger commands. Systems are like living organisms that need constant feeding of updates. The evolving nature of alerts could make a helpful one-time alert redundant or harmful in six months. On-Call as a Reliability Interface of SRE No doubt, humans have a role to play in ensuring reliability, but system designs that depend on heroic actions are built not with resilience but with fragility. Reliability is truly achieved when on-call engineers are guided by predefined scripts, models, runbooks, signals, and interfaces. These reduce the tendency to resort to fallible improvisations when issues arise. On-call engineers often take the appellation of “last point of call.” A careful look at their roles shows that they are intermediaries among complex systems, user experience, and consequences. We can thus see that the role of on-call engineers extends beyond problem resolution to stewardship. Conclusion Alert fatigue is a design problem. It often arises when on-call engineers prioritize detection over interpretation, or technical workability over user experience. The dependencies of modern SRE teams make it necessary to align technical alerts with human capability. Alert storms could wear out hardworking engineers who need to take a break. So, system designs need to account for human limitations, recognize that runbooks are better than on-the-spot improvisation, and prioritize clarity over opacity. Designs that account for these factors reduce or eliminate fatigue and preserve the very essence of alerts. In summary, reliability goes beyond resolving many problems to responding to what matters most. When teams can always trust their alerts, they will be more likely to follow up on new cases.

Every few weeks, someone on my team, or in a client meeting, asks me the same question: "Which cloud should we use for our AI workloads?" I have been building enterprise integrations for over fourteen years now, and lately most of my time goes into RAG pipelines, vector databases, and agentic orchestration on top of these platforms. So I get this question a lot, and honestly, there is no single right answer. The right cloud depends on where your data already lives, what your compliance team will accept, and which models your architecture actually needs. In this article, I want to walk through the three big players, AWS Bedrock, Google Vertex AI, and Microsoft Azure AI Foundry, and share what I have learned working with these platforms in real enterprise settings, not just from reading marketing pages. AWS Bedrock Bedrock started as a model marketplace back in 2023, and it has grown into a full platform with Guardrails for content filtering, Knowledge Bases for RAG, and AgentCore for building agentic workflows. What I like most about Bedrock is the sheer breadth of models available behind a single API. You get Claude from Anthropic, Llama from Meta, Mistral, Cohere's Command models, and Amazon's own Nova family, all through one consistent interface. If your architecture needs to swap models without rewriting your integration layer, Bedrock makes that easier than the other two. Pros: Broadest model catalog of the three, so you are not locked into one vendor's models.Strong identity and governance story if you are already running on AWS, since it plugs directly into IAM, CloudTrail, and Macie.Bedrock is one of the few places where you get Claude with enterprise indemnification, which matters a lot when legal teams get involved.Provisioned throughput options give you predictable latency for production workloads that cannot tolerate spikes. Cons: If your organization is not already AWS-native, the onboarding curve is steeper than it looks.Cross-cloud portability is basically nonexistent. A model you fine-tune on Bedrock does not export cleanly to Vertex AI or Foundry. That is a real switching cost you should plan for on day one, not something to figure out later.Some of the newer agentic tooling is still maturing, so documentation gaps show up more than I would like. Google Vertex AI Vertex AI feels different from the other two because Google's DNA here is research first. If your team cares about multimodal capability, or you want access to Gemini models the moment they ship, Vertex AI tends to be ahead. It is also the strongest option if your data already lives in BigQuery, because the integration between Vertex and BigQuery for feature engineering and MLOps pipelines is genuinely smooth. Pros: Best fit for teams doing custom model training, not just calling a hosted API. AutoML and the broader MLOps tooling cut training time noticeably compared to the other two.Tight coupling with BigQuery is a huge advantage if your organization already runs its analytics there. You avoid a lot of data movement overhead.Gemini-first multimodal workflows, plus Google Search grounding for agents, which is something neither Bedrock nor Foundry offers natively.TPU support gives real throughput advantages for heavy batch processing. Cons: If your organization is not GCP-centric already, the value proposition weakens fast. You end up paying a data-gravity tax to move information into Google's ecosystem.Governance and compliance tooling, while solid, is not as battle-tested across regulated industries as AWS's certifications.The agent ecosystem, while improving, still trails Bedrock's AgentCore and Foundry's Azure AI Agents in terms of enterprise adoption stories I have personally seen. Azure AI Foundry Foundry, formerly Azure AI Services, is Microsoft's rebranded and expanded platform, and it is the one I have written about before because it is what my own recent client work has centered on. If your enterprise already lives inside Microsoft 365, Entra ID, and Azure infrastructure, Foundry removes almost all of the identity and governance friction you would otherwise deal with. That matters more than people expect once you are past the proof of concept stage and into actual production rollout with security review. Pros: Deep Microsoft 365 and Entra ID integration means your existing enterprise approvals and identity workflows extend naturally into your AI layer.Strong OpenAI-led model access, since Microsoft's partnership with OpenAI gives Foundry early and deep access to GPT-family models.Hybrid deployment options are genuinely better here than on the other two platforms, which matters if you have on-prem systems you are not ready to fully cloud-migrate.Roughly three-quarters of Fortune 500 companies already run on Microsoft's stack, so for a lot of enterprises Foundry is simply the path of least resistance. Cons: Model breadth is narrower than Bedrock's catalog, so if you need a specific non-OpenAI model family, you may find yourself stitching together a secondary platform anyway.Because it is tied so closely to Azure compute pricing, cost predictability requires more upfront modeling than teams expect.Some newer agentic and orchestration features are still catching up to what AWS has shipped with AgentCore. So Which One Should You Actually Pick? Here is the honest answer I give in client meetings: do not choose based on a benchmark screenshot or a features table. Choose based on where your data already lives and where your governance and compliance story already works. If you are AWS-first and want maximum model flexibility, go with Bedrock. If you are Microsoft-heavy and need your AI layer to inherit existing Entra ID and 365 approvals without a fight, Foundry is the path of least resistance. If your analytics already lives in BigQuery and multimodal Gemini capability is core to your roadmap, Vertex AI earns its place. What I am increasingly seeing among the teams I work with is a hybrid pattern. A primary cloud handles the bulk of regulated workloads, and a secondary cloud gets called in only when a specific model family is not well supported on the primary platform. It is not the cleanest architecture on paper, but it reflects how fast this space is still moving. None of these three platforms is standing still, and the leader on any given feature this quarter is not guaranteed to hold that spot by next year. My suggestion, whichever cloud you land on: build your RAG and orchestration layer with enough abstraction that swapping the underlying model provider is a configuration change, not a rewrite. That single decision will save you more pain than picking the "right" cloud ever will.

By Balaji Venkatasubramaniyar CORE

Running AWS resources locally is a game-changer for engineering velocity, cost optimization, and developer autonomy. Traditionally, testing cloud infrastructure required deploying directly to a staging or sandbox AWS account. This workflow introduced painful friction points: waiting for slow cloud provisioning cycles, tracking down orphaned resources that inflate the monthly bill, and requiring a constant, high-speed internet connection. LocalStack solves this by emulating core AWS services, such as S3, SQS, DynamoDB, and other services directly on your local machine inside a Docker container. When paired with Terraform, you can safely write, plan, and apply infrastructure-as-code (IaC) configuration blueprints against this local simulator. This guide walks you through the definitive "happy path" for configuring LocalStack and Terraform, followed by a robust troubleshooting handbook for common architecture-specific and container networking errors. This allows you to provision these mock resources cleanly. This allows testing Terraform code with local resources without incurring real AWS costs, requiring internet connectivity, or dealing with slow cloud provisioning cycles. The Happy Path Setup Step 1: Setting the Stage: Launching LocalStack With Docker To get started, we need our local AWS cloud stack running inside a container. We will pull the official LocalStack image, set up our credentials, and spin up the container. First, pull the latest official image to your local machine: Before firing up the container, head over to the LocalStack Web App Dashboard to grab your personal access token (PAT). While LocalStack offers an open-source community edition, advanced features or specific emulated APIs may check for a valid token. Export this token into your shell environment so the container can authenticate and activate premium features on startup: Shell export LOCALSTACK_AUTH_TOKEN="ls-..." Now, launch the container. We need to map the primary edge gateway port (4566), which routes all inbound AWS API requests, along with the standard range of ports used by individual internal services (4510-4559). We also pass our token as an environment variable: Shell docker run --rm -it \ -p 4566:4566 \ -p 4510-4559:4510-4559 \ -e LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN \ localstack/localstack Keep an eye on your terminal logs. LocalStack will quickly validate your token, pull your license configuration, and initialize the mock runtimes. You will see a clear notification when the edge proxy is fully ready to handle incoming API requests. Step 2: The S3 Sanity Check: Talking to LocalStack Before configuring our automation toolchain, let's run a quick manual sanity check using the standard AWS CLI. Because LocalStack runs entirely on your machine, we must override the default cloud routing by passing a custom --endpoint-url pointing to our local edge proxy. To verify that LocalStack is running and reachable, create a local S3 bucket and upload a test file using the AWS CLI. 1. Create a Bucket Shell aws s3 \ mb s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 2. Upload an Object Create a dummy text file and copy it into your new mock bucket: Shell aws s3 \ cp /tmp/demo.txt s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 3. List Objects Verify the object is safely stored inside the mock container: Shell aws s3 \ ls s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 Step 3: Writing the Blueprint: Configuring the Terraform Provider Now let's automate things. To instruct Terraform to deploy resources to our local simulator instead of the real AWS cloud, we must customize the AWS provider block. We enforce dummy credentials, bypass cloud-only identity validations, and explicitly force all API endpoints to route directly to http://localhost:4566. Providers Configuration Create a file named providers.tf with the following content: Markdown terraform { backend "local" { path = "terraform.tfstate" } required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = "us-east-1" access_key = "mock_access_key" secret_key = "mock_secret_key" skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true s3_use_path_style = true # Redirect all endpoints to LocalStack's edge port endpoints { apigateway = "http://localhost:4566" cloudwatch = "http://localhost:4566" dynamodb = "http://localhost:4566" ec2 = "http://localhost:4566" iam = "http://localhost:4566" lambda = "http://localhost:4566" rds = "http://localhost:4566" s3 = "http://localhost:4566" secretsmanager = "http://localhost:4566" sns = "http://localhost:4566" sqs = "http://localhost:4566" ssm = "http://localhost:4566" sts = "http://localhost:4566" } } SQS Resource Definition Next, define the SQS queue we want to provision. Create a file named main.tf: Markdown resource "aws_sqs_queue" "local_queue" { name = "my-local-queue" delay_seconds = 90 max_message_size = 2048 message_retention_seconds = 86400 receive_wait_time_seconds = 10 } output "queue_url" { value = aws_sqs_queue.local_queue.id } Step 4: The Moment of Truth: Initializing and Applying Configuration With our configuration defined, we can run Terraform. Ensure you are executing a native binary that matches your host system architecture (such as a native darwin_arm64 binary if you are working on an Apple Silicon machine) to prevent execution overhead. Initialize Terraform First, initialize the working directory to download the AWS provider plugins: Generate and Review the Plan Next, generate and review an execution plan. The plan output will detail our local queue configuration without attempting to connect to actual AWS endpoints: Apply the Plan Apply the plan to deploy the queue directly to LocalStack. Upon completion, Terraform will write your state file locally and output your new mock SQS queue URL: Step 5: Taking It for a Spin: Sending and Receiving SQS Messages To confirm that our Terraform-provisioned SQS queue is fully operational, let's capture the output URL and push a real message through it using the AWS CLI. 1. Send a Message Shell export QUEUE_URL="http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/my-local-queue" aws sqs send-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL \ --message-body "Hello from LocalStack SQS" 2. Receive the Message Shell aws sqs receive-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL { "Messages": [ { "MessageId": "1235d997-f60a-4e86-b248-aff3f5f41dde", "ReceiptHandle": "NzYxOThkMDAtMWJiOC00OGVhLTllMDEtNTU3ZTY3ZGQ5M2I4IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6bXktbG9jYWwtcXVldWUgMTIzNWQ5OTctZjYwYS00ZTg2LWIyNDgtYWZmM2Y1ZjQxZGRlIDE3ODI4OTI2MzcuMjg2ODc1NQ==", "MD5OfBody": "88dc2faa42b899c03e12fd3ac96d714b", "Body": "Hello from LocalStack SQS" } ] } Your terminal will return a successful JSON payload containing your message body, confirmation IDs, and MD5 hashes, proving that the local loop is entirely complete. Event Verification in LocalStack Logs Checking the LocalStack container console confirms the queue creation, message send, and message fetch operations were handled successfully: Troubleshooting Guide Even on a happy path, local container networks and mixed system architectures can throw a wrench into your workflow. Here is how to fix the most common bottlenecks. The Apple Silicon (M1/M2/M3) Rosetta Loop Symptom: The LocalStack container crashes unexpectedly on startup, or loops endlessly while attempting to launch internal components like local Lambda runtimes, throwing qemu: uncaught target signal 11 errors. The Cause: LocalStack occasionally spins up secondary processes or helper binaries inside the container. If Docker Desktop is forced to emulate an x86_64 architecture via Virtualization frameworks on an ARM64 Apple Silicon chip, the emulation layer can break during heavy nested execution. The Fix: Ensure your Docker Desktop configuration has Use Virtualization framework enabled under Settings -> General, and turn on Rosetta for x86/amd64 emulation under the Features in Development tab. Alternatively, force Docker to fetch the native ARM64 container image by updating your execution command to include the specific platform flag: Shell docker run --platform linux/arm64 --rm -it -p 4566:4566 localstack/localstack "Port Already in Use" Symptom: Docker fails to bind ports, displaying an error message like: Bind for 0.0.0.0:4566 failed: port is already allocated. The Cause: A previous instance of LocalStack didn't shut down cleanly, or another local development tool is monopolizing port 4566. The Fix: Option 1: Check for lingering Docker containers Often, a container crashed or was backgrounded but didn't release the port. Find any container using 4566: Shell docker ps -a | grep 4566 If a container shows up, stop and remove it (replace <CONTAINER_ID> with your specific ID): Shell docker stop <CONTAINER_ID> docker rm <CONTAINER_ID> Option 2: Kill native background processes If Docker isn't holding the port, another process on your host machine is. You'll need to find its Process ID (PID) and force-quit it. Find the PID: Shell lsof -i :4566 Kill it (look for the number under the PID column): Shell kill -9 <PID> Wrapping Up Combining LocalStack and Terraform gives you a lightning-fast, zero-cost, offline sandbox for cloud infrastructure development. Once your environment is configured correctly with a valid personal access token, precise Docker port mappings, and native toolchains matched to your host CPU, you can prototype, test, and tear down AWS configurations in seconds. No more waiting for slow cloud deployments or tracking down orphaned cloud resources. Happy local provisioning!

AWS can send audit logs to an attacker’s account unless denials are enforced at the network layer, while Azure doesn’t log network-block requests at all. The concept of a data perimeter was popularized by AWS [1] to establish organizational boundaries around identities, resources, and networks. In simple terms, AWS provides access controls to ensure that trusted identities access trusted resources from expected networks while blocking all outside access. This article explores how different cloud providers handle resource access logs and how it relates to data protection. It sets up an experiment where an outside identity with valid credentials accesses a trusted resource and is blocked by a policy in one of the scenarios. The experiment explains two scenarios that differ in where the deny decision is enforced. We find that the same request for resource access produces different log artifacts in AWS and Azure. AWS sends access logs containing caller-controlled metadata in both the identity and resource-owner accounts unless a network layer explicitly denies access. However, in Azure, resource access logs are only logged at the resource-owner’s subscription, and when access is blocked at the network layer, nothing is logged there either. Both behaviors have consequences for security teams collecting and analyzing audit logs. This article walks through both scenarios with lab experiments and reproducible code. Background AWS and Azure treat identities differently. In AWS, identities are not centralized into one single place — instead, they live at the account level. For example, if an organization contains 10 accounts, identities can be created in each of the 10 accounts. In comparison, in Azure, identities are centralized into one Entra ID tenant. Since a tenant is linked to multiple subscriptions containing the company’s resources, identities from the same tenant are configured to access resources inside subscriptions. In summary, the resource-owning entity in AWS (the account) also holds identities, whereas in Azure the resource-owning entity (the subscription) does not hold identities – those live in the Entra ID tenant. Secondly, AWS and Azure treat access logging differently. In AWS, CloudTrail logs API calls at the account level. For cross-account access, AWS lets customers configure CloudTrail such that when data events are enabled, the caller account and the resource-owning account get access events. For example, if an identity in Account-A accesses a resource in Account-B and gets denied, then the deny audit entry is logged in both Account-A and Account-B. This mirroring is what makes caller-controlled metadata visible to a malicious actor’s account [2]. In contrast, in Azure, resource access logs (for example, StorageBlobLogs) live in the storage account in the subscription, whereas identity logs (Entra ID) live with the tenant. These are separate systems with no automatic mirroring. This difference sets up why a correlation problem exists and why a network-layer block does not produce logs at the resource layer. Threat Model The threat model is as follows: an attacker brings their credentials inside a corporate network and accesses the company’s resource (like an S3 bucket). By doing this, the attacker tries to exfiltrate company data by encoding sensitive information in the HTTP user agent header, a caller-controlled field that appears in access logs. This allows data to leave the corporate environment in small chunks across multiple requests. The second threat is more nuanced. A security team that relies on resource-layer logs to detect unauthorized access attempts will miss requests that are blocked before reaching the resource. If the network drops the request silently, the resource (service) never logs it. An attacker who knows this can probe a corporate environment repeatedly without appearing in the audit trail that the security team is monitoring. Experiments AWS Experiment To set up this experiment, we have three accounts: a credential-owning account (identity), a VPC-owning account, and a resource-owning account. The identity is a Lambda function that tries to access an S3 bucket (resource). The Lambda function runs from a private subnet in a VPC and accesses the S3 bucket through an S3 VPC endpoint (AWS PrivateLink). All audit logs are sent to a third account – this is a typical Control Tower setup [3]. We test two scenarios: The bucket policy denies all untrusted identities — assume that the bucket policy denies access to our identity. However, the VPC endpoint policy allows all cross-account access. The bucket policy allows this untrusted identity. However, the VPC endpoint policy disallows cross-organization access. Scenario 1 When the request gets denied at S3, AWS CloudTrail generates a standard API event: JSON { "eventType": "AwsApiCall", "errorCode": "AccessDenied", "userAgent": "...", "requestParameters": {...}, "tlsDetails": {...} } The full log is in https://github.com/sureshgururajan/aws-data-exfiltration-demo/blob/main/testing-results/scenario1-log.md. In this case, the full request context is preserved. This includes: userAgent requestParameters TLS metadata Additional request context The main observation is that this event includes caller-controlled metadata in the userAgent field. Since customers can configure CloudTrail to log data events on both the caller account and the resource account, a malicious actor gets the same denial event in their account. Therefore, an attacker in an untrusted account can exfiltrate company data into their accounts by triggering these denied access requests on the company resource. Scenario 2 In the second scenario, if the VPC endpoint policy denies cross-account access (example), CloudTrail generates a different event: JSON { "eventType": "AwsVpceEvent", "eventCategory": "NetworkActivity", "errorCode": "VpceAccessDenied", ... } See the full log here. Instead of logging an AwsApiCall event, CloudTrail logs NetworkActivity with the errorCode: VpceAccessDenied and does not log the HTTP user agent header. More importantly, this event is not sent to the malicious actor or the resource owner’s account. Rather, the event is sent to the VPC endpoint owner’s account. In other words, the cause of the denial was a VPC endpoint policy, and therefore CloudTrail generates a NetworkActivity event rather than the API event and routes it to the VPC-owning account. This prevents the bad actor from stealing company data via CloudTrail. Azure Experiment To set up this experiment, we created two Azure subscriptions – one for identity and the other for the resource. An Azure function in subscription-A writes to a blob storage in subscription-B. The Azure function is registered as a system-assigned managed identity in the Entra ID tenant while turning off the shared access key for the blob storage to ensure only managed identities can access it [5]. The function uses DefaultAzureCredential to request a token from Entra ID and attempts to write to a file in the storage account. Since both subscriptions trust the same Entra ID tenant, the identity moves across subscriptions natively without needing an AssumeRole step. Like before, we run through two scenarios: Azure function has the Storage Blob Data Contributor role and the network path is open The Azure function attempts to write to the storage account but is blocked by the firewall. Scenario 1 When the request is allowed at the blob storage, the following logs are written: The Entra ID tenant gets a token request log when the Azure function uses default Azure credentials. This event does NOT contain any information about the actual API action being taken. The resource account StorageBlobLogs records a PutBlob event with the file name and IP address but doesn’t show the name of the managed identity. Sample log entry from StorageBlobLogs Plain Text TimeGenerated [UTC] - 2026-05-02T19:30:32.7306109Z OperationName - PutBlob CallerIpAddress - 172.24.1.71:9156 Uri - https://sgrstorageaccountinsubb.blob.core.windows.net:443/storage-container/test.json AuthenticationType - OAuth RequesterObjectId - 00daa177-96c6-4b29-9a5c-53ca603565e9 StatusCode – 201 UserAgentHeader - azsdk-js-azure-storage-blob/12.31.0 core-rest-pipeline/1.22.3 Node/22.22.2 (Linux 6.6.130.1-3.azl3; x64) The requester object ID field indicates which identity made the request but doesn’t reveal more details as to the identity itself. That part is left to the Entra ID logs as shown below. However, we can see that the userAgentHeader is logged. The difference with AWS is that in Azure, the StorageBlob log entry is not mirrored to Entra ID, i.e., the caller’s subscription. In Azure, it stays only in the resource owner’s subscription. Entra ID contains just the token issuance log: Sample log entry from Entra ID Plain Text Date (UTC),2026-05-02T19:30:32Z Request ID,25c5f7f7-4206-448d-817b-730744991701 Correlation ID,73cf7b90-c49b-40f0-800d-74e77e40717c Service principal ID,00daa177-96c6-4b29-9a5c-53ca603565e9 Service principal name,SureshTestingMultiCloud-Function Credential key ID, Credential thumbprint, Application,SureshTestingMultiCloud-Function Application ID ,57650788-dae5-416f-9da8-792b4ebbbb29 App owner tenant ID, Resource,Azure Storage Resource ID ,e406a681-f3d4-42a8-90b6-c2b029497af1 Resource tenant ID, Resource owner tenant ID,f8cdef31-a31e-4b4a-93e4-5f571e91255a Home tenant ID, Home tenant name, IP address, Location,", , " Status,Success Sign-in error code, Failure reason,Other. Conditional Access,Not Applied Scenario 2 In this scenario, we introduced a network-level block using the Storage Account Firewall while keeping the permissions intact. Entra ID logs still show a successful token issuance because the identity is valid and the scope is broad. However, the storage resource logs don’t log the request. Since the connection was dropped at the network layer before reaching the storage service plane, there is no “Access denied” event in the resource’s audit log. Sample log entry from Entra ID Plain Text Date (UTC): 2026-05-02T19:35:10Z Service principal name: SureshTestingMultiCloud-Function Application: SureshTestingMultiCloud-Function Resource: Azure Storage Status: Success Sample log entry from StorageBlobLogs 0 results for the KQL query: SQL // Query to check for any recorded activity after the network block StorageBlobLogs | where TimeGenerated > ago(1h) | where RequesterObjectId == "00daa177-96c6-4b29-9a5c-53ca603565e9" | project TimeGenerated, OperationName, StatusCode, StatusText, CallerIpAddress, Uri | sort by TimeGenerated desc This result shows that a network-level block is not visible in the resource layer. The Azure administrator sees a successful token issuance in Entra ID but nothing in StorageBlobLogs. To detect this, security teams need to go beyond resource-layer logs and enable additional logging layers such as NSG Flow logs or Defender for Storage - these are outside the scope of this experiment. Comparison scenarioawsazure Identity model Account-scoped Tenant scoped Who gets audit logs? (when available and enabled) Caller-side and resource-owner side (Scenario 1 only) Resource-owner side only Where are the audit trails located? CloudTrail is the logging service. CloudTrail logs are distributed across Caller account, the resource account, and the VPC-owning account Token issuance logs are in the Tenant (Entra ID) while resource access logs are in the Subscription Caller-controlled metadata visible? Yes, visible in caller account and resource account Yes, but included in resource account only What a network-layer block produces When using VPC endpoint policy, AwsVpceEvent is produced and is routed to the VPC-owner account. No logs in resource-owner account. No resource-layer log entry. Identity context in resource logs Full caller identity context included Only the caller ID in the form of RequesterObjectId. An operator must correlate this ID with service principal ID in Entra ID logs. Mitigation We saw that in AWS, CloudTrail can be configured to send log events on both the caller account and the resource account. An attacker can use this information to silently exfiltrate small amounts of data at a time. To mitigate this attack vector, an organization must: Run their compute services in an Amazon VPC — preferably in a private subnet, and Use VPC endpoints with endpoint policies [4] to access their AWS resources for the compute services. The endpoint policies must allow trusted identities to access the resource while blocking everything else. AWS already documents these controls in [1], but these experiments show how important it is to enforce these controls. This is in addition to all the controls that an organization already uses, such as Service Control Policies and Resource Control Policies — those policies control the maximum permissible action that can be taken by an identity/resource but do not control the CloudTrail logging behavior. While Azure doesn’t have the above attack vector specifically, it has a different problem — an operator must manually correlate Entra ID events with the resource event. An example would be an “identity journey” like — managed identity (like the Azure function) requests a token, then writes to a storage account. Therefore, some tooling must be built to correlate such events — for example, routing both ManagedIdentitySignInLogs and StorageBlobLogs into a single Log Analytics workspace is a minimum. Additionally, logs must be captured at different layers such as NSG flow logs/Defender for Storage that can provide anomaly detection beyond standard diagnostic logs. Conclusion In this article, we demonstrated how the same access request produces different results in AWS and Azure. In AWS, access logs were sent to the resource account or the VPC account depending on where the deny decision was enforced, while in Azure, access logs were only sent to the resource account. We saw that this difference comes from how each cloud provider fundamentally treats identities and resources. The implications of the experiment are that security teams in multi-cloud environments cannot assume that audit coverage works the same way across providers. Each provider models their identities and provides different data perimeter controls. Before designing data perimeter controls, security teams must understand each provider’s logging architecture and its differences. References [1] https://aws.amazon.com/identity/data-perimeters-blog-post-series/ [2] https://systemweakness.com/a-subtle-audit-log-consideration-in-aws-063752150b20 [3] https://docs.aws.amazon.com/controltower/latest/userguide/what-shared.html [4] https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html [5] https://learn.microsoft.com/en-us/azure/storage/common/shared-key-authorization-prevent?tabs=portal

Over the past year, I've been rebuilding parts of an incident response stack for a client, and the biggest surprise wasn't the AI features themselves. It was how much of the underlying workflow had to change to make those features useful. You can't just bolt an LLM onto a 2015-era ticketing tool and call it AIOps. The queue structure, the alert taxonomy, even the way runbooks are written all need to change. I've written before about the agent side of this shift, in AI Agent Architectures: Patterns, Applications, and Implementation Guide and Observability and DevTool Platforms for AI Agents. This two-part series is the other side of that coin: what happens when you point those same agent patterns at your own production systems instead of at somebody else's AI application. Same reasoning loop, different target. This first part covers incident management specifically, including a category I skipped in my earlier tool roundups: dedicated "AI SRE" agents like Traversal, Resolve.ai, and Cleric, which behave differently from the AIOps platforms most of us grew up with. Part 2 goes past incidents into ITOps, chaos engineering, SLO management, on-call toil, and the rest of what fills an SRE's week. A note on the numbers below: vendor-reported accuracy and MTTR figures in this space move fast and come from the vendors themselves. I've flagged those clearly rather than presenting them as independently verified benchmarks. Where SRE Pain Actually Lives Before getting into tools, it helps to remember what SREs spend their time on. Most postmortems I read over the years have the same three complaints: Too many alerts, not enough signalCorrelating five different dashboards to find one root causeWriting the same postmortem summary for the fourth time this quarter None of these are new problems. What's new is that large language models are actually decent at the second and third ones, if you feed them clean data, and a newer crop of agents is starting to chip away at the first one too. The Traditional Incident Pipeline Here's roughly what an incident used to look like before AI got involved, at most mid-size shops I've worked with: Traditional incident pipeline Every arrow in that diagram is a human doing manual correlation work. That's fine when you have ten services. It falls apart at three hundred, and it's part of why I keep coming back to the point I made in Infrastructure as Code: How Automation Evolved to Power AI Workloads: scale problems in ops rarely get solved by hiring more people to stare at more dashboards. Where AI Fits Into the Pipeline Today The shift isn't "AI replaces the engineer." It's AI collapsing steps B through F into something closer to a single triage step, with the engineer reviewing a proposed root cause instead of hunting for one from scratch. AI-aided pipeline Notice the engineer never disappears from this diagram. They just move from being the one who does the correlation to the one who checks the correlation. That distinction matters, because it changes what you hire and train for. I made a version of this same argument about production-grade agents generally in the Shipping Production-Grade AI Agents refcard: an agent needs a human review layer, or you're just moving risk around instead of removing it. If you want a deeper look at how these review loops are actually structured under the hood, I broke that down recently in Loop Engineering: The Layer After Prompt, Context, and Harness Engineering. Incident Management: What Changed A few concrete things have improved in incident management tools over the last two years: Alert correlation got better. Tools like BigPanda, Moogsoft, and PagerDuty's AIOps features now cluster related alerts using pattern recognition instead of static rules. A database timeout, three downstream service errors, and a spike in 500s used to show up as four separate pages. Good correlation engines now group them as one incident with a suggested cause. Similar-incident retrieval works reasonably well. If your org has a decent history of past incidents with clean postmortems, tools can now surface "this looks like INC-4471 from March" with real accuracy. This only works if your postmortem data isn't garbage, which is a bigger blocker than people admit. Draft postmortems save real time. Not because the AI writes a good postmortem on the first try, but because staring at a blank page is the slowest part of writing one. A rough draft built from the incident timeline, Slack thread, and metrics gives engineers something to edit rather than create. What's newer, and worth its own section, is a class of tools that don't just correlate what you already collected. They go get new evidence during the incident, the way a senior engineer would. The New Category: Dedicated AI SRE Agents This is the part of the landscape that's moved fastest since I last wrote about agent tooling. A handful of startups have built agents whose entire job is investigating production incidents autonomously, not just clustering alerts that already exist. Traversal leans on causal machine learning rather than a general-purpose LLM wrapper. Instead of pattern-matching against similar past incidents, it builds a model of causal dependencies across your services and traces the actual chain of cause and effect, down to the specific deploy or config change that started the failure. It reports strong root-cause accuracy in production at large enterprises and is used for both alert triage and live incident investigation. The pitch is narrower than "full AIOps platform," and that narrowness is the point. Resolve.ai takes a broader angle. It was built by the team that created OpenTelemetry, and it positions itself as an agentic teammate across the whole production lifecycle: investigating incidents, but also touching capacity questions, config drift, and guided code changes. Where Traversal is a specialist in root cause, Resolve.ai is closer to a generalist you'd loop in on almost anything production-related, with the incident work as the anchor use case. Cleric sits in a similar space to both, with a specific focus on autonomous alert triage. It runs a multi-source investigation the moment an alert fires, pulling metrics, logs, traces, and deploy history in parallel, and returns an evidence-backed hypothesis before an on-call engineer has finished opening their second dashboard tab. It runs with read-only access by default, which matters a lot for teams still building trust in the category, and it was named a Gartner Cool Vendor in AI for SRE and Observability in 2025. That "read-only by default" design decision is exactly the kind of guardrail I argued for in Trust No Agent: How to Secure Autonomous Tools on Your Machine: an agent's blast radius should be a deliberate design choice, not an afterthought. Causely and NeuBird round out the space with slightly different angles: Causely focuses on causal reasoning to find the single root cause behind a storm of cascading alerts, and NeuBird targets enterprise IT environments with LLM-driven telemetry analysis at large scale. Here's roughly where an AI SRE agent sits in the pipeline compared to the AIOps correlation tools from the last section: AI SRE agent pipeline The key difference from the earlier diagram: this agent isn't just correlating signals you already collected in a dashboard. It's actively going out and querying your systems the way a human on-call engineer would, forming a hypothesis, testing it, and either confirming or discarding it before it ever pages a person. That's a meaningfully different capability than clustering alerts by similarity, and it's why this category gets its own row in any serious comparison. If you're weighing whether to build this kind of investigation loop yourself versus buying one of these platforms, it's worth reading MCP vs Skills vs Agents With Scripts: Which One Should You Pick? first, since the architecture decision behind "agent that calls tools live" versus "agent with a fixed skill set" applies just as much to SRE tooling as it does anywhere else. What This Actually Brings to the SRE Persona It's worth being specific about what changes for the person on-call, not just what the vendor deck claims: Fewer 2 a.m. investigations that start from zero. The agent has usually already ruled out the obvious suspects by the time a human looks at the page, so the engineer starts from a hypothesis instead of a blank terminal.Less tool-hopping. A lot of incident time isn't spent thinking; it's spent switching between Datadog, Grafana, the CI pipeline, and Slack. An agent that queries all of them in parallel removes a genuinely tedious chunk of the job.A written trail for free. Because the agent's investigation is itself a structured log of what it checked and why, you get a decent postmortem skeleton as a byproduct, not a separate task.A new failure mode to watch for. Engineers can start trusting the proposed root cause without checking the evidence trail, especially under pager pressure. That's a habit worth actively training against, not assuming away. None of this replaces the on-call engineer's judgment. It changes the shape of their shift from "gather evidence, then decide" to "review evidence, then decide," which is faster but only as trustworthy as the evidence the agent actually gathered. Comparing the Tool Landscape Here's how some of the major players stack up on where they've actually invested in AI, versus where it's mostly a checkbox feature. I've split this into two tables, because lumping AIOps correlation platforms in with dedicated AI SRE agents hides a real difference in what these tools do. Established AIOps and incident platforms: ToolAlert CorrelationRoot Cause SuggestionAuto-Drafted PostmortemsPredictive CapacityOwnership / StatusPagerDuty (AIOps)StrongModerateYesLimitedIndependent, public companyMoogsoftStrongStrongNoNoAcquired by Dell Technologies (2023)BigPandaStrongModerateLimitedNoIndependent, privateDatadog (Bits AI)ModerateStrongYesModerateBuilt in-house by DatadogServiceNow (Now Assist)ModerateModerateYesStrong (ITOps)Built in-house by ServiceNowDynatrace (Davis AI)StrongStrongLimitedStrongBuilt in-house by Dynatraceincident.ioModerateLimitedYesNoIndependent, privateRootlyModerateLimitedYesNoIndependent, private Dedicated AI SRE agents: ToolCore ApproachActs Autonomously?Best FitFounded / BackingTraversalCausal ML across dependency graphInvestigation autonomous, remediation guardedTeams with strong existing observability wanting sharper RCA2023, Sequoia and Kleiner PerkinsResolve.aiBroad agentic reasoning over code, infra, telemetryInvestigation autonomous, remediation opt-inTeams wanting one agent across incidents, capacity, and config2024, Greylock-led seedClericMulti-source parallel investigationRead-only by defaultTeams new to AI SRE agents, wary of write access2024, Zetta Venture PartnersCauselyCausal reasoning on cascading alertsInvestigation onlyEnvironments with alert storms and unclear blast radiusPrivate, early stageNeuBirdLLM-driven telemetry analysis at scaleInvestigation, guided remediationLarge enterprise IT environmentsPrivate, early stage A caveat worth stating plainly: I haven't run rigorous side-by-side benchmarks on all of these, and vendor claims move faster than reality, especially in the AI SRE agent table where most of these companies are one to three years old and evolving month to month. Treat both tables as a directional map, not a scorecard, and validate against your own alert volume before picking one. Deterministic AI vs. Generative AI in These Tools This distinction gets muddled in vendor marketing, so it's worth separating clearly. AspectDeterministic / ML-based (older AIOps)Generative AI (LLM-based, newer)ApproachStatistical pattern matching, clustering, anomaly detectionLanguage model reasoning over logs, tickets, chat history, and live queriesPredictabilityHigh, same input gives same outputLower, outputs can vary between runsStrengthCorrelation, anomaly detection at scaleSummarization, hypothesis generation, natural language explanation, draftingWeaknessPoor at explaining "why" in plain languageCan hallucinate a plausible-sounding but wrong root causeWhere it shows upDynatrace Davis AI, Moogsoft's original correlation engineDatadog Bits AI, ServiceNow Now Assist, Traversal, Resolve.ai, ClericTrust level neededCan often auto-remediateNeeds human review before action Most modern platforms now run both in tandem: the deterministic layer does the anomaly detection and correlation, and the generative layer explains it in plain English, forms hypotheses, and drafts the writeup. That combination is doing more real work than either piece alone, and it's basically the same pattern I described for agent observability generally in the AI agent architectures piece linked earlier: a fast, boring, reliable layer underneath a slower, flexible reasoning layer on top. Where We're Headed in Part 2 Incident response gets the spotlight because it's the loudest part of the job, but if you track where an SRE's actual week goes, a lot of it isn't firefighting at all. It's chaos testing, SLO math, on-call scheduling, and the slow grind of writing and maintaining runbooks nobody reads until 3 a.m. In Part 2, I'll walk through where AI is showing up in ITOps specifically, and then go further into chaos engineering, SLO and error budget management, on-call toil reduction, and capacity planning, the quieter parts of the job that determine whether the incident tools in this article even have a fighting chance.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Most SRE teams do not need another dashboard. They need a safer way to move from "something is wrong" to "we know what to do next." A model that detects anomalies is useful. A model that can touch production can also make a bad incident worse. That is where most conversations about AI in SRE become too optimistic for my taste. The hard part is not only detection. It is deciding how much autonomy the system should have, under which conditions, and with what blast-radius controls. I learned this while working on large-scale cloud services where one customer-facing symptom could turn into a flood of alerts. A degraded dependency might show up as latency in one service, retries in another, queue growth somewhere else, and CPU pressure downstream. During an on-call shift, that can look like five separate problems. Usually, it is one problem echoing through the stack. That experience changed how I think about self-healing infrastructure. The goal is not to build a system that blindly fixes everything. The goal is to build an operational control loop that can separate routine, low-risk recovery from incidents that still need human judgment. The model that has worked best for me is graduated autonomy: Let the system act automatically only when the action is well understood, reversible, and narrow in blast radius. For everything else, the system should collect evidence, recommend the next step, and keep humans in control. Why Static Alerts Stop Scaling Static alerts are not the enemy. I still want to know when disk usage is dangerous, error rates spike, or latency crosses a service-level threshold. But thresholds do not understand context. A CPU spike during a scheduled batch job may be normal. The same spike during steady-state traffic may be a retry storm. A latency increase in one region may be harmless during a controlled deployment, but suspicious if it appears across multiple availability zones with no recent change event. At small scale, engineers can carry that context in their heads. At enterprise scale, they cannot. Services emit hundreds of metrics across regions, dependencies, deployments, and customer paths. Eventually the team is no longer tuning alerts. It is negotiating with noise. In one rollout I was involved with, the most useful improvement was not adding more alerts. It was grouping alerts around dependency context and suppressing repeated downstream symptoms. The on-call experience became calmer because engineers could focus on the likely failure path instead of chasing every red graph independently. That is the kind of problem AI can help with. Not by replacing SRE judgment, but by organizing noisy signals into a more useful operational story. Detection Is Only the First Layer ML-based anomaly detection helps because it learns a service's normal operating shape instead of relying only on fixed thresholds. For cloud metrics, that usually means learning seasonality, traffic cycles, deployment windows, regional differences, and service-specific behavior. An LSTM autoencoder, isolation forest, or well-tuned statistical baseline can all be useful. I care less about the model family than the quality of the telemetry around it. A simple model trained on clean, consistent data will usually beat a sophisticated model trained on messy metrics. A practical anomaly pipeline usually looks like this: Collect metrics, logs, traces, and change events.Normalize them by service, region, dependency, and time window.Score each signal against its learned baseline.Group anomalies by dependency graph and recent changes.Produce an evidence bundle for automation or human review. Here is a simplified version of the scoring stage: Python from dataclasses import dataclass from typing import List @dataclass class MetricWindow: service: str region: str signal: str values: List[float] recent_deploy: bool = False @dataclass class AnomalyScore: service: str region: str signal: str score: float reason: str class BaselineModel: def expected_range(self, service: str, region: str, signal: str): # In production, this may come from a trained model, # feature store, or rolling baseline per service and region. return (0.0, 1.0) def score_window(window: MetricWindow, baseline: BaselineModel) -> AnomalyScore: low, high = baseline.expected_range( window.service, window.region, window.signal, ) latest = window.values[-1] if latest > high: distance = (latest - high) / max(high, 0.001) reason = f"{window.signal} above learned baseline" elif latest < low: distance = (low - latest) / max(abs(low), 0.001) reason = f"{window.signal} below learned baseline" else: distance = 0.0 reason = "within learned baseline" if window.recent_deploy and distance > 0: reason += " during recent deployment window" return AnomalyScore( service=window.service, region=window.region, signal=window.signal, score=min(distance, 1.0), reason=reason, ) The production value is not just the score. It is the metadata around it: ownership, dependency path, recent deploys, feature flag changes, customer impact, and whether the same pattern has appeared before. A single anomalous metric should rarely trigger remediation. Sustained anomalies across correlated signals are more trustworthy than one spike in one chart. Correlation Turns Noise Into an Incident Story During an incident, the useful question is not "Which graph is red?" It is "What changed first, and what depends on it?" That is where dependency-aware correlation becomes more useful than raw anomaly detection. A database issue may surface as API latency, retries, queue saturation, and CPU pressure. Without a dependency graph, every downstream service looks guilty. With one, the system can rank likely causes instead of handing the engineer a wall of symptoms. A useful correlation engine should look at topology, timing, change context, and customer impact. Which dependency failed first? Was there a deployment or config change? Which service is closest to the customer-facing error? The evidence bundle should be readable by a human. If the model says "root cause confidence: 0.86," that is not enough. It should also explain why. JSON { "candidate_root_cause": "identity-token-cache", "region": "example-region-1", "confidence": 0.86, "customer_impact": "elevated authentication latency for a subset of requests", "supporting_signals": [ "p99 latency above learned baseline for multiple consecutive windows", "cache hit rate dropped below its recent operating range", "downstream services showed retry growth after the initial cache anomaly", "no database saturation was observed", "no deployment was detected in the immediate incident window" ], "recommended_action": "drain_and_restart_one_cache_node", "estimated_blast_radius": "single node in a redundant pool", "rollback_plan": "keep node out of rotation if health checks fail after restart" } This is more useful than another alert. It gives the on-call engineer a starting hypothesis and the reasoning behind it. The Graduated Autonomy Model The most important design decision in self-healing infrastructure is not which ML algorithm to use. It is which actions the system is allowed to take. I divide remediation into three tiers. Tier 1: Fully Automated, Low-Risk Actions Tier 1 actions are safe, reversible, and narrow in blast radius. These are actions the system can execute without waiting for a human when confidence is high. Examples include restarting one unhealthy instance, scaling out a stateless service, draining one bad node, flushing a bounded cache, or shifting a small amount of traffic away from a degraded zone. The key phrase is bounded blast radius. Auto-remediation should not restart half the fleet, fail over a primary database, or disable a feature globally just because a model is confident. Confidence is not a substitute for safety. Before I put an action in Tier 1, I expect it to pass these checks: it is reversible, affected capacity is small, redundancy is healthy, there is no active global incident, the same action has not failed recently, rollback is defined, and health checks can verify success quickly. The first Tier 1 actions should be boring. Restarting one unhealthy node is not exciting, but it is exactly the kind of action that can be automated safely when the system has enough evidence. Tier 2: Automated Recommendation With Human Approval Tier 2 is where many real incidents live. The system may know what should happen, but the action still needs human approval. Examples include rolling back a deployment, disabling a feature flag, failing over a database, increasing capacity beyond a normal band, or changing regional routing. For Tier 2, the system should prepare the action, show the evidence, and ask for approval. The human should decide whether the action makes sense, not build the command during the incident. One pattern I have seen repeatedly: the slowest part of remediation is not always finding a likely cause. It is gathering enough confidence to take a risky action. When the system attaches deploy timing, error movement, affected endpoints, config changes, and rollback commands into one review card, the decision becomes easier. Tier 3: Human-Led With AI Context Tier 3 incidents are novel, high-risk, or ambiguous. The system should not execute remediation. It should help humans reason. This includes possible data corruption, multi-region cascading failures, security-sensitive incidents, conflicting signals across dependencies, low-confidence root-cause analysis, or any action with unclear rollback behavior. In Tier 3, the system's job is to summarize what it knows, what changed recently, which hypotheses are most likely, and which dashboards or runbooks are relevant. That alone can save time, but it keeps production control where it belongs. Architecture: A Control Loop, Not a Magic Button A practical self-healing system looks like a control loop with guardrails. Architecture diagram: Graduated autonomy model for self-healing infrastructure The important part of this diagram is the policy gate. Detection and correlation produce a recommendation, but the policy gate decides autonomy. Without that layer, "self-healing" becomes a risky automation script with an ML label attached. The policy gate should evaluate confidence, risk, blast radius, recent action history, service criticality, and rollback readiness. I would express that as policy-driven code: JSON from dataclasses import dataclass from enum import Enum from typing import List class Decision(str, Enum): AUTO_EXECUTE = "auto_execute" REQUEST_APPROVAL = "request_approval" HUMAN_LED = "human_led" @dataclass class RemediationProposal: action: str confidence: float blast_radius_percent: float reversible: bool rollback_defined: bool service_tier: str evidence: List[str] @dataclass class RuntimeContext: active_global_incident: bool recent_failed_action: bool healthy_redundancy: bool minutes_since_last_same_action: int TIER_1_ACTIONS = { "restart_single_instance", "scale_stateless_service", "drain_single_node", "flush_bounded_cache" } TIER_2_ACTIONS = { "rollback_deployment", "disable_feature_flag", "database_failover", "regional_traffic_shift" } def decide_autonomy( proposal: RemediationProposal, context: RuntimeContext ) -> Decision: if context.active_global_incident: return Decision.HUMAN_LED if context.recent_failed_action: return Decision.HUMAN_LED if not proposal.rollback_defined: return Decision.HUMAN_LED if proposal.action in TIER_1_ACTIONS: safe_enough = all([ proposal.confidence >= 0.90, proposal.blast_radius_percent <= 5.0, proposal.reversible, context.healthy_redundancy, context.minutes_since_last_same_action >= 30, len(proposal.evidence) >= 3, ]) return Decision.AUTO_EXECUTE if safe_enough else Decision.REQUEST_APPROVAL if proposal.action in TIER_2_ACTIONS and proposal.confidence >= 0.75: return Decision.REQUEST_APPROVAL return Decision.HUMAN_LED This is not drop-in production code, but the structure is the point: actions are classified, confidence is not the only input, and safety can override the model. In reliable systems, the model proposes; policy disposes. What I Measure Before Expanding Autonomy I would not start by asking, "Can we automate remediation?" I would start by asking whether the system's recommendations are trustworthy. Before allowing Tier 1 execution, I would track root-cause precision, false positives by service, recommendation acceptance, time to useful diagnosis, remediation success, rollback frequency, and any secondary incidents caused by remediation. The last two matter the most to me. A self-healing system that fixes one issue but creates another is not healing. It is moving the incident. My preference is to run in shadow mode first. Let the system detect, correlate, and recommend, but do not let it execute. Compare its recommendations against what engineers actually did. Once the system repeatedly recommends the same low-risk actions humans already take, graduate those actions into Tier 1. That is how trust gets built: not through a big launch, but through repeated correctness in narrow, well-understood situations. Lessons Learned From Building Toward Self-Healing The most useful lessons are not about model architecture. Clean telemetry beats clever models. If service names are inconsistent, regions are missing, logs are unstructured, and ownership metadata is stale, the model will struggle. Before debating LSTMs versus transformers, fix the telemetry pipeline. Change events are first-class signals. Deployments, config pushes, schema changes, and feature flag flips explain many anomalies. If the model cannot see change events, it will treat every incident like a mystery. Alert suppression is not the same as diagnosis. Reducing noise is useful, but the system must preserve the causal path. Suppressing duplicate downstream alerts only helps if the upstream root cause remains visible. Automation needs a memory. Every remediation should leave an audit trail: what was detected, what action was taken, what happened afterward, whether rollback was needed, and whether humans agreed with the recommendation. Start with boring actions. Restarting one bad instance is not glamorous. Draining one node is not a research breakthrough. But these are exactly the kinds of actions that make sense for early autonomy because they are repeatable, reversible, and easy to verify. Where LLMs Fit Large language models are useful in SRE, but I would not put them directly in the execution path for remediation. Their best role is communication and context assembly. An LLM can draft an incident summary, explain the evidence bundle, turn raw telemetry into a timeline, identify runbooks, and prepare a post-incident report. That saves time without giving the model direct control over production. The safer pattern is separation of responsibilities: ML or statistical models detect anomalies, graph correlation ranks likely causes, policy gates decide autonomy, deterministic automation executes approved actions, and LLMs summarize what happened. That separation keeps the high-risk parts deterministic and auditable while still using AI where it helps most. Final Thought Self-healing infrastructure is not about removing SREs from production. It is about removing the repetitive, low-risk work that slows them down during incidents. The best version of AI in SRE is not a magic system that fixes everything. It is a careful control loop: detect early, correlate intelligently, act only within policy, and learn from every outcome. If you are building toward self-healing, do not start with full autonomy. Start with evidence. Then recommendations. Then approval-based actions. Then, only after the system has earned trust, allow narrow automated remediation. That path is slower than the hype cycle, but it is much closer to how reliable infrastructure actually gets built.

Logging is one of the oldest practices in software engineering, yet in distributed systems it remains one of the most poorly implemented. Most teams log, but very few log well. The gap between having logs and having useful logs becomes painfully visible the moment a production incident occurs at 2 AM across a system running dozens of microservices. This article focuses on structured logging: what it is, where teams consistently go wrong with it, and the concrete practices that separate log data you can actually act on from log noise that burns engineering hours during incidents. If you are building or operating distributed systems today, structured logging is not optional. It is the foundation on which every other observability signal- traces, metrics, alerts- depends. What Structured Logging Actually Means Structured logging means emitting log entries as machine-readable key-value pairs rather than arbitrary free-text strings. Instead of this: Plain Text [ERROR] 2026-07-10 03:14:22 - Failed to process payment for user 84729, reason: timeout You emit this: JSON { "timestamp": "2026-07-10T03:14:22Z", "level": "error", "service": "payment-service", "event": "payment_processing_failed", "user_id": 84729, "reason": "timeout", "duration_ms": 3001, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" } The difference sounds cosmetic. It is not. The first format requires regex parsing and string matching to extract meaning. The second is immediately queryable, aggregatable, and, crucially, correlatable with traces and metrics from other services handling the same request. The Five Mistakes Distributed Systems Teams Make With Logs 1. Logging Without Context Propagation In a monolith, a single log line tells you where in the codebase an event occurred. In a distributed system, a log line without a correlation identifier tells you almost nothing. If Service A calls Service B which calls Service C, and Service C fails, you need a shared identifier, typically a trace ID, that threads through all three services' logs so you can reconstruct the full request journey. The fix is context propagation: passing a trace ID through every request, injecting it into every log entry, and configuring your logging library to include it automatically. In practice, this means integrating your logging setup with OpenTelemetry or a similar tracing framework from day one, not as an afterthought. When your log entries include trace_id and span_id fields, you can jump from a log entry to its full distributed trace in a single query; that capability compresses incident diagnosis from hours to minutes. 2. Inconsistent Field Naming Across Services In a microservices architecture developed by multiple teams, field-naming inconsistencies compound into a real problem at scale. One service logs user_id, another logs userId, a third logs uid. One service logs errors under error, another uses err, another uses exception. When you need to query across services during an incident, this inconsistency forces per-service query variations, slowing everything down. Establish and enforce a logging schema across your organization. Define a canonical set of field names for common concepts, user identifiers, request identifiers, error fields, latency fields, and make that schema part of your service standards. Libraries like structlog in Python or logrus/zap in Go make it straightforward to enforce common fields at the logger initialization level, so teams can't easily deviate from the schema accidentally. 3. Logging at Wrong Severity Levels Severity level misuse is endemic. INFO logs that should be DEBUG. Application errors logged as WARN because the developer did not want to trigger alerts. Business logic exceptions logged as ERROR when they are expected and handled. Over time, this degrades the signal value of severity levels to the point where teams stop filtering by level entirely. Adopt and document clear severity semantics for your organization: DEBUG: information useful only during active development; should not run in productionINFO: normal operational events (service started, request received, job completed)WARN: unexpected conditions that are recoverable and do not require immediate actionERROR: failures that require investigation; every ERROR should eventually be investigated or suppressed with documented justificationFATAL: unrecoverable failures; service cannot continue Treat severity levels as a contract with your future on-call self. 4. Over-Logging Hot Paths High-throughput services that log every incoming request at INFO level generate enormous log volumes that create three problems: storage costs escalate, log search performance degrades, and genuinely important events get buried in noise. A service processing 10,000 requests per second generates over 860 million log lines per day from request logging alone. Use sampling for high-frequency, low-severity log events. Most observability platforms and log monitoring tools support log sampling natively; you configure a sampling rate for specific log patterns, keeping representative data without keeping everything. For example, sample 1% of successful payment processing logs but keep 100% of error logs. This dramatically reduces volume while preserving signal fidelity where it matters. 5. Treating Logs as a Standalone Signal Logs become exponentially more powerful when they are correlated with traces and metrics. A spike in error logs is interesting. An error log spike correlated with a latency metric increase correlated with a trace showing a database connection timeout is actionable in seconds. Teams that treat logs as independent from their other observability signals are leaving significant diagnostic capability on the table. If you are not already running OpenTelemetry, start there. It provides a unified SDK for instrumenting logs, traces, and metrics in a way that ensures they carry shared context identifiers. Once your logs carry the same trace IDs as your distributed traces, your observability signals become correlated by default, not by manual investigation. A Practical Logging Schema to Start With Here is a minimal structured logging schema that covers the majority of production use cases across distributed services: JSON { "timestamp": "ISO-8601 UTC", "level": "debug|info|warn|error|fatal", "service": "service-name", "version": "1.4.2", "environment": "production", "event": "snake_case_event_name", "message": "Human-readable description", "trace_id": "OpenTelemetry trace ID", "span_id": "OpenTelemetry span ID", "user_id": "optional", "request_id": "optional", "duration_ms": "optional, numeric", "error": { "type": "TimeoutError", "message": "Connection timed out after 3000ms", "stack": "optional, omit in high-volume paths" } } This schema is opinionated but extensible. Services add domain-specific fields as needed while every entry maintains the common fields that make cross-service correlation possible. Conclusion Structured logging in distributed systems is not about logging more; it is about logging intentionally. The practices that separate teams who resolve incidents in minutes from teams who spend hours in log archaeology come down to four things: consistent field naming, trace context propagation, disciplined severity usage, and treating logs as a correlated signal rather than an isolated one. Get these right, and your logs become a first-class observability asset during incidents. Get them wrong, and you have the worst of both worlds: high storage costs and low diagnostic value. The patterns outlined here are not theoretical; they are the difference between incident response that feels like detective work and incident response that feels like reading a timeline.

As a data engineer, I’ve noticed business teams submitting intake forms, compliance documents, and project proposals that a tech team then manually validates against a set of predefined business rules stored in a database that gets updated quarterly. The time it takes to validate a single form is typically in the hours, and by the time you’ve validated the form, the submitter has moved on to other work. When I needed to validate project intake forms against 60+ business rules of financial, compliance, and other types of business rules and guidelines (some of them to be used in a deterministic way and others to be used in a more nuanced manner), I knew that a simple if-else logic-based manual review process would not scale. This article walks through how I developed an async, AI-powered validation API with AWS Bedrock Agents and Serverless Architecture to process and validate intake forms within 60 seconds without blocking the user. The architecture also manages cross-account authentication to get access to the AI-powered engine and shows failure recovery gracefully. Why Async? The Problem With Synchronous AI APIs Integrating AI into an API synchronously means users send a request, the server processes it, and returns results in one HTTP response, but many systems that use AI-powered validation take more than 30 seconds. The AI agent I built was taking anywhere from 30 seconds to 1 minute to evaluate all of the form fields for all the applicable rules and conditions. But the hard limit for the API Gateway is 29 seconds (HTTP timeout). One approach to make this API request work is to transform the synchronous request and response into an async request with a subsequent background processing step and poll the results from a separate endpoint. This can be implemented as follows: Client submits the form via POST, receives a request_id immediately (under 2 seconds)Validation runs asynchronously in the background (30–60 seconds)Client polls a GET endpoint with the request_id until results are ready By making the form submission step separate from the AI validation of that form in the background, users can continue working on other tasks instead of being stuck staring at a page waiting 30 to 60 seconds for the form to be validated. Architecture Overview As a data engineer, I was required to tackle three main challenges to create a production AI validation API: 1) the frontend application is deployed in a different AWS account, 2) AI agent-based form validation is extremely computationally expensive to run, and 3) business rules for this type of validation are likely to change from time to time without API code deployment. The architecture consists of five components: API Gateway (REST API): With Cognito Authorizer for cross-account JWT authenticationAsync Handler Lambda: It’s an entry point for the API. An Async Handler Lambda function is invoked by a POST request. It will store the form payload on S3, then trigger the Validation Lambda function and store an initial "processing" status in S3. The function immediately returns a request_id to the frontend client within 2 seconds.Validation Lambda: This function loads up all the rules for a given request from S3. It then builds up all the prompts for the Bedrock Agent and runs the Agent. The results of the Agent are then saved off in S3 for the Polling API.Polling Lambda: Handles GET requests and checks S3 for completed resultsRules Sync Lambda: Separate independent process to read validation rules from the data warehouse using EventBridge scheduler and sync to S3 for validation with AI model. Implementation: The Async Handler The async handler is the entry point. Its task is quite straightforward. It accepts the payload, stores it, triggers the Validation Lambda function, stores an initial "processing" status in S3, and returns the “processing” status with the request ID to the client. The function does all of this within a couple of seconds. Here is the core implementation: Python import json, boto3, uuid from datetime import datetime s3 = boto3. client(' s3') Lambda_client = boto3. client('Lambda') S3_BUCKET = 'my-validation-bucket' VALIDATION_LAMBDA = 'ai-validation-function' def lambda_handler(event, context): payload = json. loads (event. get ('body', "{}')) request_id = str(uuid.uuid4()) # Store initial processing status s3.put_object( Bucket=S3_BUCKET, Key=f'validation-output/(request_id)/status.json', Body=json.dumps({ 'request_id': request_id, 'status': 'processing', 'submitted_at': datetime. ttenew() .isoformat() }) ) # Fire-and-forget: invoke validation async pay Load ['_request_id'] = request_id lambda_client.invoke( FunctionName=VALIDATION_LAMBDA, InvocationType='Event', # Async invocation Payload=json. dumps (payload) ) return { 'statusCode': 202, 'body': json. dumps ({ 'request_id': request_id, 'status': 'processing' }) } In the above code snippet, I specifically invoke the validation lambda from the async handler by setting the InvocationType='Event'. This allows the async handler to return immediately to the frontend with the request_id for the submitted request. The Validation Lambda will then complete asynchronously and store the results in S3. Implementation: The Polling Handler The Polling Handler Lambda function manages the GET endpoint; it polls S3 for the updated status file and returns the current status of Validation Lambda processing: completed or failed. Here is the core implementation: Python def lambda_handler(event, context): request_id = event['pathParameters']['request_id'] try: status_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/status.json' ) status = json.loads(status_obj['Body'].read()) if status['status'] == 'processing': return {'statusCode': 200, 'body': json.dumps(status)} # Completed - return full results results_obj = s3.get_object( Bucket=S3_BUCKET, Key=f'validation-output/{request_id}/results.json' ) results = json.loads(results_obj['Body'].read()) return {'statusCode': 200, 'body': json.dumps(results)} except s3.exceptions.NoSuchKey: return {'statusCode': 404, 'body': 'Request not found'} S3 Decoupling: Using S3 as an intermediary between the validation Lambda and the polling handler allows for natural decoupling. The validation Lambda writes the results of the validation to S3, and the polling handler reads from S3 to return the latest status to the frontend. There is no shared state between the validation handler and the polling handler; there are no database connections, and there are no race conditions. Integrating the Bedrock Agent for Intelligent Validation An intelligent validation function would need more than just a set of rules to check for requirements and best practices. There are a lot of judgment calls that a human would make based on examples of how a policy or guideline would be applied in real life. To achieve that, the more effective way is to integrate with an existing AI function that is designed to handle a wide variety of scenarios and functions The Bedrock Agent architecture solved this by combining: Knowledge base: Containing policy documents, guidelines, and past examples of work for the intelligent validation to reference during the evaluation process.Dynamic prompts: The prompts for the AI model are built dynamically from the current validation rules. These are loaded from S3 as a JSON file and then injected with the current values for the specific field being evaluated.Structured output: Parse the assessment’s pass/fail status, confidence in the assessment, and a set of detailed recommendations made by the agent. The prompt for the AI agent is generated at runtime by the validation function. The rules are loaded from S3 earlier in the function's execution. Here is an example prompt: “Evaluate field [Project Justification] with value [user input] against rule: The justification must clearly describe the business problem being solved and include quantified impact. Reference the knowledge base for examples of approved justifications.” The AI returns a structured assessment of whether or not the field has passed validation, the confidence that the AI has in the assessment, and recommendations. Dynamic Rules Management: Keeping Rules in Sync Without Code Deploys Rules typically change on a monthly or quarterly basis by the business teams. To keep up with the current policy, the rules must be separate from the rest of the application code. To achieve that, I used Rules Sync Lambda, triggered daily by EventBridge: EventBridge fires at 6 AM daily.The Rules Sync Lambda queries the Data Warehouse (Redshift) for the current validation rules for the application.It also takes a copy of the most current version of the rules in S3 for purposes of rollback.It transforms and then uploads the new rules file to S3 as a new copy of the Validation_Rules.json file.Upon failure to update the rules in S3, a CloudWatch Alarm is triggered, which in turn triggers an SNS notification to the appropriate engineering team. The rules are managed as a database of rules (as opposed to being stored within the application code), which allows business analysts to easily update the rules on a quarterly basis without requiring any code changes or deployments. Cross-Account Authentication With Cognito In this case, the frontend application and the AI backend were set up in two different AWS accounts. When deployed within different accounts (as within an enterprise), cross-account authentication is required. Since the frontend application was already authenticated against a company’s SSO (Single Sign On) using Cognito, it was only a matter of how to reuse these tokens within another account without involving the Frontend team for changes. The solution was to create a Cognito Authorizer and attach it to a REST API created in the API Gateway. This API can then be set up to trust the User Pool from the frontend account. Below is a simplified representation of this configuration: API Gateway REST API with a Cognito Authorizer pointing to the frontend account’s Cognito User Pool ARN.CORS (Cross-Origin Resource Sharing) configuration for only that frontend domain.The frontend application is already authenticated with CognitoThe backend application accepts the tokens that the frontend application is using for authenticationThe frontend application simply sends the existing Cognito tokens that the frontend application already has created in the authentication process From the frontend team’s perspective, this was a simple implementation that required them to send the existing Cognito token with the request and to implement a polling loop for the GET endpoint. Results and Lessons Learned After deploying to production: Validation time: reduced from 2 -3 hours (manual) to less than a minute (automated)API response time for form submission: less than 2 seconds for GET API using an async pattern, meaning the frontend never has to wait for the backend60+ validation rules: per form, including both deterministic and AI-judgement rules Zero code deploys: for changes to the rules, which are stored in the database, sync daily Key lessons as a developer building this: Design for async from the start: Retrofitting a synchronous API to be async is very hard. If your AI inference takes more than 5 seconds, which is generally the case, then design your API to be async from day one.Use S3 as your state machine: S3 is the simplest, cheapest, and most reliable way to pass results between decoupled Lambdas. No databases, no queues, no DynamoDB for this pattern.Separate dynamic rules from code: Separate process for managing rules which are dynamic and change often to avoid deployment bottleneck Bedrock Agents are good for making judgment calls. If you have a deterministic check (is a field empty), then you can code that. But for a judgment call (does a justification make sense), then use an AI agent to make the call. Conclusion There is an entirely new way to approach the request lifecycle for APIs in this AI-powered validation API development. The asynchronous API with polling for validation is better than simply trying to work around the timeout limits of APIs. Bedrock Agents, along with S3 to manage the state of the workflow and EventBridge to synchronize rules on a daily basis from a database created by business users via a simple UI created by frontend team, while backend team does not need to write any code for new rules, all integrated together to form complex data validation system powered by AI-powered judgment calls while maintaining simple to deploy and scalable system. As a data engineer, there’s nothing quite like watching hours of manual work by a reviewer get compressed down into 60 seconds or less of automated work while maintaining the high level of evaluation that a business stakeholder expects.

In a lot of organizations, the real integration platform is a person. Someone exports orders from the ERP every morning and pastes them into the planning tool. Someone else re-types customer updates from the CRM into the invoicing system. It works until that person is on holiday or makes a typo in a price field or the volume doubles. Replacing that manual work with a synchronization service sounds like a junior-level task: read from system A, write to system B, schedule it, done. In practice, sync services are where many integration projects quietly fail. They fail not because moving data is hard, but because the edge cases are partial failures, retries that duplicate records, two systems that both think they own a field, and errors that nobody notices for three weeks. This article walks through the design decisions that separate a sync layer you can trust from one you learn to fear. The examples use Python and pseudo-SQL, but every pattern here is language-agnostic. Decision 1: One Source of Truth Per Entity The single most important design decision in any sync architecture is not technical. It is organizational: for every entity, exactly one system is allowed to win. Orders live in the ERP. The webshop may create them, but once created, the ERP's version is the truth, and the webshop displays what the ERP says. Customer contact details live in the CRM. The ERP receives updates from the CRM and never edits them locally. The moment two systems can both modify the same entity and both push their version, you have built a conflict generator. Last-write-wins will silently destroy data. Merge logic will grow into an unmaintainable swamp of special cases. The fix is almost never smarter conflict resolution. It is removing the conflict by assigning ownership. Write this down as a table before writing any code: EntityOwnerMay createMay updateOrderERPWebshop, ERPERP onlyCustomer contactCRMCRMCRM onlyProduct/pricingERPERPERP onlyStock levelERPERPERP only If you cannot fill in this table, you are not ready to build the sync. Any cell where two systems appear in the "may update" column is a design problem to resolve with the business first, not a technical challenge to code around. Decision 2: Idempotency, or Retries Will Hurt You Your sync will fail mid-run. The network will drop after 4,000 of 5,000 records. The target API will return a 500 halfway through. The scheduler will fire twice. None of these are exceptional; they are Tuesday. The only sane response to failure is retry, and retry is only safe when every operation is idempotent: running it twice produces the same result as running it once. The classic mistake looks like this: Python # Dangerous: creates a duplicate on every retry def sync_order(order): target_api.create_order( customer=order.customer_id, lines=order.lines, total=order.total, ) If this call succeeds on the target but the response is lost (a timeout, a crashed worker), the retry creates a second order. Someone ships it. The fix is to make every write carry a stable, deterministic key derived from the source record, and make the target treat that key as unique: Python # Safe: the natural key makes the operation idempotent def sync_order(order): target_api.upsert_order( external_id=f"erp-{order.erp_id}", # stable key from the source customer=order.customer_id, lines=order.lines, total=order.total, ) If the target system has no upsert endpoint, simulate one: look up by external_id first, then create or update. Wrap that lookup-and-write in one function and forbid every other code path from writing directly. The same rule applies to your own bookkeeping. Store sync state keyed by the same external ID, so a re-run of yesterday's batch is harmless by construction. Decision 3: Pull Changes, Don't Diff Worlds The naive sync reads all records from both sides and compares them. This works in the demo and collapses in production, where "all records" means 400,000 rows over a SOAP API that pages 100 at a time. You need change detection, and there are three workable tiers, in order of preference: The source has reliable updated_at timestamps or a change log. Store a high-water mark after each successful run and query only what changed since. This is the happy path; verify that the timestamp actually updates on every mutation, including the ones done by nightly batch jobs inside the legacy system. Legacy systems lie about this more often than you would expect.The source has no usable timestamps, but you can read all records cheaply. Compute a hash per record and compare against the hash you stored last run. Only records with changed hashes get pushed downstream: Python import hashlib, json def record_hash(record: dict) -> str: canonical = json.dumps(record, sort_keys=True, default=str) return hashlib.sha256(canonical.encode()).hexdigest() def detect_changes(records, stored_hashes): for r in records: h = record_hash(r) if stored_hashes.get(r["id"]) != h: yield r, h Neither is possible. You are down to full comparisons on a schedule. Constrain the entity scope aggressively and be honest with stakeholders about latency. Whichever tier you land on, keep the change detector separate from the writer. A queue between them, even a simple database table with pending / done / failed states, gives you retry, rate limiting, and an audit trail almost for free: SQL CREATE TABLE sync_queue ( id BIGSERIAL PRIMARY KEY, entity_type TEXT NOT NULL, external_id TEXT NOT NULL, payload JSONB NOT NULL, status TEXT NOT NULL DEFAULT 'pending', attempts INT NOT NULL DEFAULT 0, last_error TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), processed_at TIMESTAMPTZ, UNIQUE (entity_type, external_id, status) ); That UNIQUE constraint is doing real work: it prevents the same pending change from being enqueued twice, which keeps the queue idempotent too. Decision 4: A Silent Sync Is Worse Than No Sync Here is the paradox of a working sync layer: the better it works, the more people trust it, and the more damage it does on the day it silently stops. A sync that visibly fails gets fixed the same morning. A sync that dies quietly keeps its consumers confidently reading stale data. Sales quotes yesterday's* stock levels. Finance invoices from last week's prices. By the time someone notices, you are reconstructing three weeks of drift. Minimum viable observability for a sync service is four things: A heartbeat. Every run writes a completion record. An external check alerts when the most recent successful run is older than the expected interval. Do not rely on the sync alerting about itself; a crashed process sends no alerts.Drift metrics. Periodically count records on both sides and compare. The counts will never match perfectly in a live system, so alert on trend, not on exact equality.A dead-letter state. After N failed attempts, a queue item moves to failed and a human is notified with the payload and the last error. Infinite retry loops on a permanently broken record will otherwise clog the queue and mask new failures behind old ones.Readable logs per record. When finance asks why invoice 4482 shows the old address, you want to answer with one query, not a debugging session. None of this is sophisticated. All of it is regularly skipped, because on the day the sync ships, it works, and observability feels like polish. It is not polish. It is the feature that determines whether you find out about failure from a dashboard or from an angry customer. The Shape of the Whole Thing Put together, a trustworthy sync layer is small and boring: Plain Text [ Source system ] → change detection → sync_queue → idempotent writer → [ Target system ] ↓ heartbeat, drift checks, dead letters Two processes, one queue table, a handful of metrics. The value is not in the volume of code; most implementations of this design fit in a few hundred lines. The value is in the decisions encoded in it: one owner per entity, stable keys on every write, changes flowing through an inspectable queue, and failure treated as a normal input rather than an exception. Build it this way, and the sync becomes infrastructure nobody thinks about, which is the highest compliment integration code can receive. Build it as a quick script, and you have not removed the human integration layer at all. You have just changed whose Friday afternoon gets ruined.