DZone Spotlight

I remember sitting in the war room three days before a major release. We were confident; our static accessibility (a11y) linter was passing, and our localization (l10n) unit tests, which performed simple string comparisons, returned green. Then, a beta user reported that the primary_submit_action button was physically impossible to tap because our localization file had grown the button label, pushing the touch target off-screen. Our tests hadn't failed because they weren't looking at the screen. They were looking at text files. That was the moment I realized our reliance on brittle string-comparison and basic static linting had hit a hard wall at scale. We needed a new approach, one that didn't just check for the presence of a label, but verified the semantic contract between the layout and the user. The Semantic Contract Insight We arrived at the concept of "Semantic Contracts" not as a top-down decree, but as a survival mechanism. We were suffering from the "false positive" epidemic: tests that passed while the product was broken. We needed to treat our accessibility tree and layout engine as dynamic data structures rather than static files. By moving the validation gate into the CI/CD pipeline and using an AI orchestrator to enforce multi-dimensional constraints, we shifted from checking if a button exists to ensuring it functions according to the contract of our design system. Making the Switch from Linting to Semantic Contracts Pillar 1: Accessibility as a Semantic Contract Our first attempt at accessibility was rule-based. We hardcoded minimum coordinate thresholds (like 44x44 points) into our CI pipeline. It worked until our design system grew to include nested containers that varied by device class. Hardcoded thresholds became a maintenance nightmare, costing us five engineering hours per sprint in manual threshold adjustments. We realized accessibility couldn't be a list of static rules; it had to be a contract validated within the CI environment. The Tradeoff: Complexity vs. Coverage We tried standard automated scanners like native native a11y inspectors, but they failed to catch context-dependent issues. For example, an icon-only button is technically "accessible" if the label exists, but it’s contextually confusing if the label isn't descriptive. The cost of manual audit was too high, so we pivoted to an LLM-validator. Implementation: Enforcing the Contract We now use automated XCUITest assertions to ensure every primary interactive element meets our physical usability thresholds, followed by an LLM-based pass to ensure the semantic quality of those labels. Swift func testSubmitButtonMeetsAccessibilityMinimums() { let app = XCUIApplication() // Using a standard identifier from our design system let submitButton = app.buttons["primary_submit_action"] // Explicitly check for the presence to avoid false positives XCTAssertTrue(submitButton.exists, "Primary action button not found in hierarchy.") // Enforcing the 44x44 touch target contract let frame = submitButton.frame XCTAssertTrue(frame.size.width >= 44 && frame.size.height >= 44, "Submit button size \(frame.size) is below the 44x44 accessibility threshold.") } Pillar 2: Localization via Visual-Contextual Pipelines Localization is often treated as a "search and replace" problem. When we were just comparing strings, we missed layout collisions constantly. The tradeoff here was significant: we moved from fast, flaky screenshot comparison testing to a slower, more accurate multi-modal LLM collision detection. The Failure of Visual Regression We initially tried pixel-perfect screenshot regression. It failed because our dynamic content—like usernames and feed timestamps—caused thousands of "false failures." Maintaining the baseline screenshots cost us more time than actually writing the localized strings. Implementation: The Semantic Validator We now feed our view hierarchy (as JSON) and snapshots into an LLM orchestrator. This allows the system to understand that a button label is not just a collection of pixels, but an interactive semantic component that must not overlap with its parent container. Python import openai def validate_accessibility_labels(accessibility_tree_json): prompt = f""" You are an accessibility expert. Analyze the following UI accessibility tree. Verify that every interactive element has an accessibility label that is descriptive and not just a repeated image filename or a cryptic ID. Accessibility Tree: {accessibility_tree_json} Return a JSON response: {"passed": bool, "issue_description": str} """ response = openai.ChatCompletion.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}] ) return response.choices[0].message.content Architecture Summary Validation Engine Input Source Analytical Method A11y-Validator Accessibility Tree Rule-based logic + LLM semantic labeling L10n-Visualizer UI Snapshot + Resource Keys LLM-based boundary collision detection Contract-Gate CI/CD Environment Infrastructure-as-Policy enforcement Moving Forward The transition to semantic contracts was not cheap in terms of CI time—validating against an LLM is slower than a simple regex check. However, the cost of an emergency patch or a poor user experience for our accessible or international user base is significantly higher. By treating our UI as data and our accessibility requirements as verifiable contracts, we’ve moved the conversation from "Does this lint?" to "Does this meet our contract?" That shift is what keeps our release process sane, even at scale.

Every engineering team I have read about that built continuous compliance the right way reports the same three things: audit prep drops from weeks to hours, evidence requests get answered with a query instead of a scramble, and the compliance team stops being the group everyone dreads talking to. The stack that gets you there in 2026 is not experimental anymore. Every piece is production-grade, mostly open source, and the pattern is documented in enough public engineering blogs that you can copy it without reinventing anything. This is a walkthrough of what that stack actually looks like, what each layer does, and the concrete benefit each one produces. If you're still running compliance as an annual project instead of a pipeline concern, this is the article I wish someone had put in front of me two years ago. The Four Layers That Matter A continuous compliance stack has four layers. Each one solves a specific class of problem and produces evidence at a different granularity. You build them bottom-up. Trying to build Layer 4 without Layers 1-3 is the classic mistake. You end up with a dashboard that reports what the auditor asked about, not what's actually true. Layer 1: Everything Is Code, or You Have No Floor to Build On Nothing about continuous compliance works if your production environment is a set of manually-clicked buttons in a cloud console. The prerequisite is that every piece of infrastructure exists as a versioned, reviewable file. For most teams in 2026, this means Terraform or OpenTofu for cloud resources, Kubernetes manifests or Helm charts for workloads, and something like Crossplane if you want the whole platform expressed as CRDs. The specific tool matters less than the invariant: if someone can change production without opening a pull request, that change is invisible to every layer above it. The concrete benefit: any auditor question of the form "what was running on date X" becomes git log --before=X on the infra repo. No CMDB lookup, no interview, no reconstruction. Layer 2: Policy as Code Gates the Pipeline This is where the audit stops being a separate activity and becomes part of the deploy loop. The pattern is simple: every change to Layer 1 gets evaluated against a set of machine-readable policies before it can merge. If it violates a policy, the pipeline blocks it. The policy engine that has won this space is Open Policy Agent with its Rego language. Here is what a real policy looks like, checking that no S3 bucket in a Terraform plan can be created without encryption: R package terraform.s3 deny contains msg if { resource := input.resource_changes[_] resource.type == "aws_s3_bucket" resource.change.actions[_] == "create" not resource.change.after.server_side_encryption_configuration msg := sprintf( "S3 bucket '%s' created without encryption at rest", [resource.address] ) } You wire this into your pipeline with Conftest or a native OPA integration, run it on every terraform plan, and the merge is blocked if any deny rule fires. The developer gets the error in their PR within seconds, fixes it, and the violation never touches production. Every control you would otherwise sample once a year becomes a rule you enforce on every commit. Access controls, encryption requirements, tagging standards, network segmentation, cost guardrails, data residency. All the same shape. For Kubernetes specifically, the modern option is Kyverno or Gatekeeper, both of which run OPA-style policies as admission controllers so violations are rejected before workloads land in the cluster. The concrete benefit: the control isn't sampled; it's enforced. Every artifact of every deploy is compliant by construction. The auditor doesn't have to trust that Sarah reviewed the change; they can inspect the policy code, verify it matches the control language in the framework, and see the immutable log of every evaluation. Layer 3: Continuous Control Monitoring for What Escapes the Pipeline Not everything comes through the pipeline. Someone will always have break-glass access. A managed service will drift. A misconfiguration will slip through because you didn't have a policy for it yet. This is where continuous monitoring lives. The stack of choice depends on your cloud, but the pattern is consistent: AWS Config with Config Rules that continuously evaluate resource state against desired configurationAzure Policy with built-in and custom definitionsGoogle Cloud Security Command Center with Security Health AnalyticsProwler, Steampipe, or CloudQuery for cloud-agnostic queries across your posture The key move: Export the evaluation results on a schedule of minutes or hours into a queryable evidence store. Something like S3 + Athena, or a proper data warehouse if you're doing this seriously. Every finding gets a timestamp, a resource identifier, a control mapping, and a status. Now your posture is a table you can query. "How many production databases were unencrypted on any given day in the last twelve months" is a SELECT statement, not a project. The concrete benefit: drift detection in minutes instead of quarters. When a misconfiguration appears, the pipeline that catches it is the same one that alerts the engineer, opens a ticket, and can even remediate automatically for specific classes of issues. Layer 4: Evidence Pipeline, Not Evidence Collection The last layer is where most attempts fail. Teams build great pipelines and monitoring, then when the auditor shows up, they still scramble to export screenshots into a shared drive. The move is to invert the flow. Instead of collecting evidence when asked, you continuously publish evidence in a format the auditor's tooling can consume. For SOC 2 and ISO 27001, the emerging pattern is: Every control gets a stable identifier that maps to the framework (SOC 2 CC6.1, ISO 27001 A.8.24, etc.)Every policy, monitoring rule, and pipeline check declares which control(s) it enforces as metadataThe evidence store aggregates the results into a control-indexed view, updated continuouslyThe auditor gets read-only access to that view, either through a compliance automation platform or a direct query interface Compliance automation platforms in this space (Vanta, Drata, Secureframe, Sprinto, and a growing number of others) have made much of this out of the box, but the value only shows up if the underlying pipeline actually produces the evidence they consume. The tool doesn't create compliance; it packages it. The concrete benefit: your next audit stops being a project. The auditor logs into the evidence view, samples what they need, and produces the report. Your team's involvement drops from hundreds of hours to dozens. The Numbers People Are Reporting Public case studies from engineering blogs and industry conferences have converged on a consistent shape of return: Audit prep time: 3-6 weeks of engineering time collapses to under 40 hoursEvidence request turnaround: from days to minutes for anything the pipeline already tracksControl coverage: 100% of applicable resources instead of sampled 15-30Time to detect drift: from quarterly review cycles to sub-hourCost of external audit engagement: 30-70% reduction as auditor hours shift from evidence collection to review These are not marketing numbers from vendors. They're what practitioners are reporting in State of DevOps reports, at conferences like SREcon and DevOps Enterprise Summit, and in engineering blogs from companies that have actually done it. A 90-Day Rollout That Works You do not need a six-figure consulting engagement to start. The pattern that consistently works: Weeks 1-2: Pick one control and one policy. Choose something high-value and easily codified. "No public S3 buckets" or "all databases must have encryption at rest" are perfect starting points. Write the Rego policy, wire it into a single pipeline, and prove the end-to-end works. Weeks 3-6: Expand to a family of controls. Cover all the encryption controls, or all the access control policies, or all the network segmentation rules. Do not try to cover everything at once. Pick a category and finish it. Weeks 7-10: Wire in continuous monitoring for the same controls. Now the same rules that gate the pipeline are also checking runtime state. Any drift, any exception, any resource created outside the pipeline gets flagged within an hour. Weeks 11-13: Build the evidence view. Even a simple query against your monitoring store is enough to start. The point is that the auditor can pull the current state of these controls without your team producing a spreadsheet. At the end of ninety days, you have a working continuous compliance loop for one control family. It will already be more rigorous than your previous annual audit for those controls. Now you expand. The Mistake Nobody Warns You About Every failed continuous compliance program I've read about failed for the same reason: they tried to codify the existing controls exactly as the compliance team had written them for a human audit. Human-audit controls are written in prose. "Access to production systems is reviewed quarterly by the appropriate manager." That sentence contains three ambiguities that don't exist in code: what counts as "access," who is "appropriate," what is "reviewed." When you move to code, you have to force a specificity that the prose let you skip. Which is uncomfortable, because now the compliance team, the engineering team, and eventually the auditor have to agree on the exact definition. That agreement is where the value lives. The pipeline is downstream of it. Teams that skip this conversation and try to auto-generate policies from a control library end up with policies that either don't fire (too permissive, technically enforced but meaningless) or fire on everything (too strict, engineering routes around them). The policies have to be a real translation of what the control means, not a syntactic conversion. Budget the first two weeks of each control family for this conversation. It's the highest-leverage part of the whole build. What Comes Next The near-term direction that engineering teams should watch: Machine-readable audit standards. Right now every control framework is prose. The OSCAL project from NIST is defining a machine-readable format for control catalogs, profiles, and assessment results. When this reaches critical mass, the manual translation step above collapses. Regulator-consumable evidence. DORA and the SEC cyber rules are the leading edge. Both point toward a future where regulators expect continuous evidence rather than periodic attestation. If you build now, you're ready. If you don't, you're playing catch-up under enforcement pressure. AI agents as compliance actors. LLM-based agents that read policy documents, propose Rego translations, review pipeline results, and draft explanations for auditors are already being tested at large orgs. The interesting question is not whether they help (they do) but how you audit them, which is a whole other problem. The takeaway: Continuous compliance is no longer a future state. It's a solved architectural pattern that any team can start implementing in the next sprint. The teams that build it now will spend the next decade shipping faster and paying less for audits than the teams that don't. That's the entire pitch.

Artificial intelligence is transforming software testing by enabling faster test creation, smarter execution, and more efficient quality assurance processes. With AI agents, we can generate test cases and scripts, run tests, and produce detailed reports with minimal manual effort. AI agents are software systems that leverage artificial intelligence to achieve goals and perform tasks on behalf of users. They can think through problems, plan actions, and remember things, while also making decisions on their own and improving over time. In this tutorial, we will build an AI-powered agent that does exactly this. It takes a .txt file with a simple use case written in plain English, processes it using an AI model such as OpenAI or Ollama, and generates Selenium test automation scripts in Java. How to Build an AI Agent for Test Automation We will be building an AI agent for test automation that will perform the following tasks: Reads a test scenario from a .txt fileSends the scenario with the right prompt to OpenAI/Ollama, based on the provider selected in the config fileGenerates the Selenium WebDriver Java test automation scripts bifurcated into: Page Object ClassesTest ClassTestng.xmlReadme.md with notes and steps to run the test Prerequisites The following prerequisites are required to start building the AI agent for test automation: Python 3.14 and aboveCode Editor (VS Code is preferred)Basic understanding of Python and the TerminalTest scenario written in plain English text LLM Setup OpenAI API KeyOllama setup on the local machine Understanding the Selenium AI Test Generation Workflow Before we deep dive into building the AI agent, let’s get a high-level overview of how this AI Agent works for generating Selenium Java test automation scripts: The process is explained step by step below: User input: The user provides a plain English test scenario in a .txt file.Processing layer: The Python main script reads the test scenario file and forwards it to the LLM model (OpenAI or Ollama), depending on the configuration.AI engine: Before sending requests to the LLM, the AI agent converts the test scenario from plain English to a structured prompt and adds clear instructions like: Generate Selenium WebDriver code in JavaUse the TestNG frameworkFollow the Page Object ModelGenerate Multiple Page Object classes(if needed)Rules for generating the codeOutput format for files The LLM understands the test description, steps, and expected behavior, and converts them into the respective files.File generation: The AI agent creates a new timestamp-based folder and stores the generated code in separate files, bifurcating the page object classes, test class, testng.xml, and README.md. Markdown . ├── config.py ├── tools.py ├── main.py ├── requirements.txt ├── .env ├── test_cases/ │ ├── input/ │ │ └── sample_test_case.txt │ └── output/ Let’s start building the AI agent using the following step-by-step guide: Step 1: Creating and Activating a Python Virtual Environment Run the following command in the terminal to create a Python virtual environment: Plain Text python -m venv venv Next, run the following command to activate the virtual environment: On MacOS/Linux: Plain Text source venv/bin/activate On Windows: Plain Text venv\Scripts\activate Step 2: Installing Dependencies Let’s create a requirements.txt as it serves as a blueprint for recreating the exact environment needed to run the project consistently across different environments. Plain Text #requirements.txt openai==2.28.0 requests==2.32.5 python-dotenv==1.2.2 The following command should be run from the terminal to install the dependencies: Plain Text pip install -r requirements.txt Step 3: Writing the Configuration File The configuration file holds details related to OpenAI, Ollama, input and output files, and settings for using the desired LLM provider to generate test automation scripts. Python #config.py from dataclasses import dataclass, field from pathlib import Path from dotenv import load_dotenv from dotenv import load_dotenv from openai import OpenAI import os load_dotenv() @dataclass class OpenAIConfig: model_name: str = os.getenv("OPENAI_MODEL_NAME", "gpt-5") max_tokens: int = int(os.getenv("OPENAI_MAX_TOKENS", "4096")) temperature: float = float(os.getenv("OPENAI_TEMPERATURE", "0.3")) api_key: str = os.getenv("OPENAI_API_KEY", "") @property def client(self): if not self.api_key: raise ValueError( "OPENAI_API_KEY is required when using OpenAI provider" ) return OpenAI(api_key=self.api_key) @dataclass class OllamaConfig: model: str = os.getenv("OLLAMA_MODEL_NAME", "llama3") prompt: str = "You are a Selenium WebDriver test automation expert. Generate Selenium WebDriver Java test scripts. STRICTLY follow the format. Any deviation is not acceptable." stream: bool = False temperature: float = float(os.getenv("OLLAMA_TEMPERATURE", "0.3")) ollama_baseurl:str = os.getenv("OLLAMA_BASEURL", "http://localhost:11434") ollama_endpoint: str = os.getenv("OLLAMA_ENDPOINT", "http://localhost:11434/api/generate") @dataclass class FileConfig: input_file: Path = Path("test_cases/input/sample_test_case.txt") output_file_path: Path = Path("test_cases/output/") @dataclass class AppConfig: provider:str = "ollama" #openai or ollama openai: OpenAIConfig = field(default_factory=OpenAIConfig) ollama: OllamaConfig = field(default_factory=OllamaConfig) files: FileConfig = field(default_factory=FileConfig) config = AppConfig() The configuration file serves as a centralized configuration system for switching between LLM providers and managing file settings. Let’s break it down to understand it further: Dataclasses: The @dataclass automatically generates the __init__() method, making it easier to create and manage objects without writing boilerplate code.Separate Configs: Separate config classes are defined for OpenAI, Ollama, and input/output file handling, each with default values. The input file name is currently configured as “sample_test_case.txt.”AppConfig: The AppConfig class serves as a central wrapper, enabling switching between providers (OpenAI or Ollama) with a single flag.field(default_factory=…): It ensures each config gets its own instance, avoiding shared state issues. Step 4: Implementing Utility Functions for Reading and Generating Code In this step, we’ll create a new Python file named tools.py and implement the following utility functions within it: load_test_case_from_file()build_prompt()generate_with_openai()generate_with_ollama()generate_selenium_test_script()create_timestamped_output_dir()split_and_save_files() The following packages should be imported in the tools.py file: Python import requests from config import config from typing import Optional from pathlib import Path from datetime import datetime from logger import logger import requests from config import config from logger import logger Let’s learn the utility functions one by one. load_test_case_from_file() function: Python def load_test_case_from_file(file_path: str | Path) -> str: logger.info(f"Loading test case from: {file_path}") try: with open(file_path, "r", encoding="utf-8") as file: content = file.read() logger.info(f"Test case loaded successfully ({len(content)} characters)") return content except FileNotFoundError: logger.error(f"Test case file not found: {file_path}") raise FileNotFoundError(f"Test case file not found: {file_path}") except Exception as e: logger.exception(f"Error reading test case file: {e}") raise RuntimeError(f"Error reading test case file: {e}") The load_test_case_from_file() function takes the test case file path as a parameter, reads its contents, and returns them as a string. It safely handles errors by raising a clear message if the file is not found and wraps any other unexpected issues in a RuntimeError. build_prompt() function: Python def build_prompt(use_case_text: str) -> str: logger.info("Building AI prompt") prompt = f""" You are a test automation expert specializing in Selenium WebDriver with Java. Generate Selenium automation test script using the following instructions and skills: - Use Java 17 to write the code - Use latest Selenium WebDriver Java dependency version to write code - Do not write code statement "System.setProperty()" to add chromedriver path, in the tests - Follow Page Object Model (POM) - Use latest version of TestNG dependency - Apply best coding practices for writing Java code - Add comments explaining each step - Add assertions using TestNG assertion - Do not add random assertion statements in the code - Do not mention any text in README that says that ChromeDriver path should be added to the Path - Never use brittle XPATH and CSS Selectors selectors such as .btn-primary, .container > div:nth-child(2), #content div span, or auto-generated classes. IMPORTANT: You MUST follow the exact output format below. Rules: - ALWAYS start each file with ===FILE: filename=== - Use class names based on the web page(e.g. HomPage.java, LoginPage.java, etc. These names are for instructions only, use class name specific to the web page) - Do not add "Page" to the test class name - Do NOT add explanations outside file blocks - DO NOT skip this format - If you do not follow this format, the output will be rejected - DO NOT use markdown (no **, no ``` blocks) - DO NOT add file names outside ===FILE: markers - ONLY use ===FILE: filename=== format - OUTPUT FORMAT FOR FILES(STRICT): ===FILE: filename=== file content - The following files MUST only be generated in the same order(STRICT). No deviation is acceptable: - Multiple Page Object classes(if needed) (Strictly Page object class, no WebDriver instantiation in these classes, Do not create duplicate page object classes) - Test class(WebDriver should be instantiated in the Test class, Do not use WebDriverManager to instantiate WebDriver, Use TestNG's @BeforeMethod annotation and define a method to instantiate the WebDriver, Use TestNG's @AfterMethod to quit the WebDriver) - Add assertions using TestNG assertion - Do not add random assertion statements in the code - testng.xml(Follow correct structure as per TestNG guidelines) - README.md (Include notes and steps to run the test using testng.xml file) Use Case: {use_case_text} """ logger.info(f"Prompt created ({len(prompt)} characters)") return prompt The build_prompt() function is a core part of the application because it constructs the prompt sent to the LLM to generate Selenium Java test automation code. It takes the use case text as input and embeds it into a detailed instruction template, guiding the model to generate Java-based Selenium tests using best practices such as POM and TestNG. The prompt also enforces file-formatting rules like (===FILE: filename===) to ensure the output is structured, consistent, and ready for file generation without manual cleanup. It also enforces rules to generate the POM, test class, testng.xml, and README in strict order to keep the output consistent. Using Effective Prompts Effective prompts are essential for generating clean, reliable Selenium test scripts with AI. By clearly defining the requirements, the model can be guided to use best coding practices, create clean test scripts, and use the Page Object Model (POM) to generate well-structured code. Prompt Design for Better Selenium Java Tests Clear, specific prompts help the AI generate cleaner Selenium test scripts. Mention details such as using Java for Selenium, using the latest versions of Selenium and TestNG, and following coding best practices so the AI can better meet the requirements and produce well-designed output. Customizing Prompts for Page Object Model (POM) The following prompts can be used to train the AI to generate the page object files we need : Use best practices to generate the Page Object classes.Generate Multiple Page Object classes (if needed).Do not instantiate the WebDriver in the Page Object class.Do not create duplicate Page Object classes.Additionally, we can also mention using best locator strategies, like preferring IDs and CSS selectors over complex XPaths, to make the tests more stable, readable, and easy to maintain. generate_with_openai() function: Python def generate_with_openai(prompt: str) -> Optional[str]: response = config.openai.client.chat.completions.create( model=config.openai.model_name, temperature=config.openai.temperature, max_tokens=config.openai.max_tokens, messages=[ {"role": "system", "content": "You are a Selenium WebDriver test automation expert. Generate Selenium WebDriver Java test scripts. STRICTLY follow the format. Any deviation is not acceptable."}, {"role": "user", "content": prompt}, ], ) return response.choices[0].message.content The generate_with_openai() function sends a prompt to the OpenAI API to generate Selenium WebDriver test scripts in Java. The OpenAI client is initialized using the API key from an environment variable, allowing the code to connect and interact with OpenAI services securely. It uses the model, temperature, and token limits from the configuration. It generates the request with a message that defines the AI’s role and user prompt (the build_prompt() function will be supplied here). The API returns multiple choices, and the function extracts the generated content from the first response. Finally, it returns the generated test script as a string. generate_with_ollama() function: Python def generate_with_ollama(prompt: str) -> str: logger.info( f"Generating test code with Ollama model: {config.ollama.model}" ) response = requests.post( config.ollama.ollama_endpoint, json={ "model": config.ollama.model, "prompt": config.ollama.prompt + "\n" + prompt, "stream": config.ollama.stream, }, ) response.raise_for_status() data = response.json() generated_text = data.get("response", "") logger.info( f"Ollama generation completed ({len(generated_text)} characters)" ) return generated_text The generate_with_ollama() function generates text by sending a POST request to the configured Ollama API endpoint. It combines a predefined base prompt with the user-provided prompt and sends it along with the model name and streaming configuration. After receiving the response, raise_for_status() checks for HTTP errors, and response.json() parses the response into a Python dictionary. The generated text is extracted from the response field, with an empty string used if the field is missing. Finally, the function logs the response length and returns the generated text. generate_selenium_test_script() function: Python def generate_selenium_test_script(test_case_text: str) -> Optional[str]: logger.info("Starting Selenium test generation") prompt = build_prompt(test_case_text) provider = config.provider.lower() logger.info(f"Using AI provider: {provider}") if provider == "openai": logger.info("Sending prompt to OpenAI") return generate_with_openai(prompt) elif provider == "ollama": logger.info("Sending prompt to Ollama") return generate_with_ollama(prompt) else: logger.error(f"Unsupported provider: {provider}") raise ValueError(f"Unsupported provider: {provider}") The generate_selenium_test_script() function acts as a wrapper to generate a Selenium WebDriver test script based on the input test case. It first builds a prompt using the build_prompt (test_case_text) function, which prepares the input for the LLM. Next, based on the configured provider, i.e., OpenAI or Ollama, it dynamically calls the respective function to generate the script. If an unsupported provider is specified, it raises a ValueError to prevent unexpected behavior. create_timestamped_output_dir() function: Python def create_timestamped_output_dir(base_output_path: Path) -> Path: timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") output_dir = base_output_path / timestamp output_dir.mkdir(parents=True, exist_ok=True) logger.info(f"Created output directory: {output_dir}") return output_dir The create_timestamped_output_dir() function creates a new folder named with the current date and time, so each run gets its own unique output directory. It ensures the folder exists (creating it if needed) and returns its path for saving files. split_and_save_files() function: Python def split_and_save_files(generated_text: str, base_output_path: Path) -> None: sections = generated_text.split("===FILE:") if len(sections)<=1: logger.error("No structured files found in AI response") raise ValueError ("No Structured files found in AI response!") logger.info(f"Found {len(sections) - 1} generated files") pageobject_dir = base_output_path/"pageobjects" pageobject_dir.mkdir(parents=True, exist_ok=True) for section in sections[1:]: section = section.strip() parts = section.split("\n",1) raw_filename = parts[0].strip() filename = raw_filename.replace("===", "").strip().split()[0] content = parts[1].strip() if len(parts)>1 else "" content = content.split("===FILE:")[0].strip() if filename.endswith("Page.java"): file_path = pageobject_dir / filename else: file_path = base_output_path / filename logger.info(f"Writing file: {file_path}") with open(file_path, "w", encoding="utf-8") as f: f.write(content) logger.info(f"Created: {file_path} ({len(content)} characters)") The split_and_save_files() function takes the AI-generated response and splits it into multiple files based on a marker (===FILE:). It extracts each filename and its content, then saves them into the appropriate folders. The Page Object files go into a pageobjects directory, while the test class, testng.xml, and README.md files go into the main output/<timestamped> folder. If the expected structure is missing, it throws an error to avoid saving incorrect output. Step 5: Writing the Main Script to Glue Everything In this step, we’ll create a main.py file to orchestrate the complete workflow of generating Selenium test scripts using AI: Python import re from tools import load_test_case_from_file, generate_selenium_test_script,split_and_save_files,create_timestamped_output_dir, check_llm_connection from config import config from pathlib import Path from logger import logger def main() -> None: try: logger.info("Starting Selenium AI Test Generator") if not check_llm_connection(): logger.error( "❌ LLM connection check failed. " "Stopping test generation." ) return input_file = config.files.input_file output_file_path = config.files.output_file_path run_output_dir = create_timestamped_output_dir(output_file_path) use_case_text = load_test_case_from_file(input_file) if not use_case_text: logger.error("Test case file is empty.") raise ValueError("Test case file is empty.") generated_output = generate_selenium_test_script(use_case_text) if not generated_output: logger.error("Failed to generate Selenium WebDriver Java test automation scripts.") raise RuntimeError("Failed to generate Selenium WebDriver Java test automation scripts.") split_and_save_files(generated_output,run_output_dir) logger.info(f"✅ All generated files saved successfully to {run_output_dir}") except Exception as e: logger.error(f"❌ Error occurred while generating output files: {e}") if __name__ == "__main__": main() The main.py file is where the program starts and manages the whole process of generating Selenium test scripts from start to finish. It reads the input test case file, sends the test case to the AI to generate automation scripts, and generates the timestamped output folder. Once the scripts are generated, it splits them into multiple files and saves them in the appropriate directories. In case of any exception, it catches the error and prints the message “Error occurred while generating output files” with the exception details. Generating First Test Scripts Let’s create a new text file, ”sample_test_case.txt,” and place it in the input/ folder with the following test scenario to generate the Selenium test scripts using the AI agent we created: Plain Text Title: Application Login scenario Precondition: User is registered in the application. Steps: 1. Open Chrome browser 2. Navigate to https://ecommerce-playground.lambdatest.io/index.php?route=account/login 3. Enter "[email protected]" in the E-Mail Address field 4. Enter "Password@321" in the Password field 5. Click on the Login Button 5. Add an assert statement to check that "My Account" page is displayed. Configuration and Setup Using OpenAI Create a .env file and add your OpenAI API key in it. This file should always remain on your local machine and should not be committed to the remote repository. Plain Text OPENAI_MODEL_NAME=<model name> OPENAI_MAX_TOKENS=<max tokens> OPENAI_TEMPERATURE=<temperature value> OPENAI_API_KEY=<Your OpenAI API Key> Using Ollama Download and install Ollama.Start Ollama by running the command ollama serve in the terminal.Pull the Qwen3:8b model by running the command  —  ollama pull qwen3:8b. Create a .env file and update the following details in it: Plain Text OLLAMA_MODEL_NAME=qwen3:8b OLLAMA_TEMPERATURE=0.3 OLLAMA_ENDPOINT=http://localhost:11434/api/generate OLLAMA_BASEURL=http://localhost:11434 Before running the model, ensure that Ollama is running in the background. Open Terminal and run the command ollama serve Note: Running the model explicitly is not required. Running the AI Agent Open the terminal and run the following command to generate the test scripts: Plain Text python main.py The following log should be printed in the console after the Agent is run successfully: The test scripts should be generated in a new timestamped(YYYY-MM-DD_hh-mm-ss) folder that is generated inside the output/ folder: Reviewing the Generated Java Selenium Code Let’s check each file generated in the output/ folder and review the code to verify if it fits the test scenario we defined and follows the expected structure and best practices. Let’s check each file generated in the output/ folder and review the code to verify if it fits the test scenario we defined and follows the expected structure and best practices. The following page object class is generated: LoginPage.java Java public class LoginPage { private WebDriver driver; public LoginPage(WebDriver driver) { this.driver = driver; } public void navigateToLoginPage() { driver.get("https://ecommerce-playground.lambdatest.io/index.php?route=account/login"); } public void enterEmail(String email) { driver.findElement(By.name("email")).sendKeys(email); } public void enterPassword(String password) { driver.findElement(By.name("password")).sendKeys(password); } public void clickLoginButton() { driver.findElement(By.xpath("//button[@type='submit']")).click(); } } The page object file is generated correctly using the name and XPath locator strategies, and appropriate methods are created to interact with the respective WebElements on the page. However, the import statements are missing, which should be added to avoid errors. Additionally, locators should be checked, and explicit waits could be added in this class while locating the elements to reduce flakiness during test execution. The following test class is generated: LoginTest.java Java import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.WebElement; import org.openqa.selenium.chrome.ChromeDriver; import org.testng.Assert; import org.testng.annotations.BeforeMethod; import org.testng.annotations.Test; public class LoginTest { private WebDriver driver; @BeforeMethod public void setup() { System.setProperty("webdriver.chrome.driver", "path_to_chrome_driver"); driver = new ChromeDriver(); } @AfterMethod public void teardown() { driver.quit(); } @Test public void testLoginScenario() { LoginPage loginPage = new LoginPage(driver); loginPage.navigateToLoginPage(); loginPage.enterEmail("[email protected]"); loginPage.enterPassword("Password@321"); loginPage.clickLoginButton(); WebElement myAccountPageTitle = driver.findElement(By.xpath("//h1[@class='page-title']")); Assert.assertTrue(myAccountPageTitle.isDisplayed(), "My Account page is not displayed"); } } The test class is generated per the provided prompt and includes the @BeforeMethod and @AfterMethod annotations for setup and teardown. However, it includes System.setProperty(), which can be removed because the latest version of Selenium doesn't require it to set the ChromeDriver path. The import statement for the @AfterMethod annotation is missing, which should be added. Additionally, the myAccountPageTitle WebElement can be moved to a separate MyAccount Page Object class, and its XPath should also be verified to ensure it's pointing to the correct element on the page. Testng.xml XML <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> <suite name="LoginTestSuite"> <test name="LoginTest"> <classes> <class name="LoginTest"/> </classes> </test> </suite> The qualified class name should be updated once the code is moved to the project. README.md Markdown **How to run the test** 1. Install TestNG framework and Selenium WebDriver. 2. Place the Selenium ChromeDriver executable in a directory, update the path_to_chrome_driver variable in `LoginTest.java` with this directory. 3. Run the test using the testng.xml file by executing the command: `java -cp .;test-classes org.testng.TestNG testng.xml` Note: Update the path_to_chrome_driver variable with your actual ChromeDriver executable path. The steps for the ChromeDriver executable and the command to run testng.xml can be removed, as they are not accurate. Instead, the step to run the testng.xml by right-clicking on it can be updated here. Overall, the AI agent generated the test code quickly, saving manual effort and providing a good starting point. However, review, validate, and refine the code before using it in a project. Best Practices While Using AI Agents The following best practices should be considered while using AI Agents to generate automation code: Improving prompt quality for reliable tests: Clear and detailed prompts help the AI generate stable test scripts. Mention requirements like framework names, naming conventions, and coding standards to improve structure and avoid errors.Adding assertions and validations: Always ensure the generated tests include proper assertions to validate expected outcomes. This helps confirm that the application behaves correctly instead of just performing actions.Review and refactor generated code before execution: Always review AI-generated code before running it. Refactoring helps remove redundancies, improve readability, and align the code with the project standards.Handling invalid/empty inputs: Validate inputs like test case files before processing them. Ensure you add proper steps and verification checks for empty or invalid data to prevent unexpected failures during execution.Integrating OpenAI API error handling: Proper error handling for API calls helps manage issues like timeouts, rate limits, or failed responses. Using try-catch blocks and logging error messages improves error readability. Exception handling helps interpret error messages in simple language and makes debugging easier.Logging and debugging tips: Adding logs at key steps helps track execution flow and quickly identify issues. Clear logging messages make debugging easier, especially when working with AI-generated outputs and API calls. Watch the step-by-step YouTube tutorial on How to Build an AI Agent to Generate Selenium WebDriver Tests in Java. Limitations and Challenges The following limitations and challenges can be faced while working with AI Agents for test script generation: AI-generated scripts may fail: AI-generated test scripts can be a starting point, but they may fail when the input prompt is unclear or missing important details. They can also break because of dynamic elements, incorrect UI assumptions, or environment-specific issues.Manual cleanup/selector accuracy: AI may generate locators that are not always reliable or optimal. Always review and refine locators in AI-generated scripts, preferring stable options like IDs or CSS selectors over complex XPaths.Test maintenance and flakiness considerations: AI-generated tests may be flaky if they don’t handle waits, dynamic content, or synchronization properly. Adding proper waits and improving test design can reduce flakiness. Regular maintenance is required to keep tests stable as the application evolves.Reviewing hallucinations: AI can sometimes generate incorrect or non-existent methods, elements, or logic. Review the code carefully to catch such hallucinations before execution. Validating against actual application behavior ensures the tests are accurate and usable. Conclusion To design an AI agent to generate test scripts, create a comprehensive prompt with clear rules, a structured output format, and strict guidelines to ensure consistent results. It’s also important to validate and refine the generated code and handle edge cases and failures gracefully. It all depends on the team’s decision; since AI is evolving, designing an AI agent that generates test automation scripts can also be a viable and flexible approach, especially when customization and control are important.

By Faisal Khatri CORE

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale. Kubernetes environments can drift, accumulate one-off fixes, and diverge across teams until a routine deploy breaks or a cost spike forces a review. This checklist gives platform, SRE, and engineering teams a way to keep clusters, deployments, and automation manageable as Kubernetes operations scale and teams grow. It covers standards, observability, releases, access, drift, and cost. Review it before promoting a service to production and revisit it as your environments shift. Cluster Standards and Environment Discipline Most teams run more than one Kubernetes cluster, and those clusters diverge over time as they are upgraded and modified independently. At that point, a fix or runbook that works on one cluster can’t be trusted to work on another. Keeping the fleet operable requires every cluster to run a supported Kubernetes version and follow the same approved platform settings and policies. Document centrally controlled settings (e.g., Kubernetes versions, networking, admission policies) separately from service-team settings (e.g., pod resource requests, autoscaling, ConfigMaps)Standardize namespace, labeling, and resource-quota conventions so workloads are identified and bounded consistently across clustersMaintain each cluster’s baseline configuration in version control; use reconciliation to apply it and correct untracked changesMaintain an approved Kubernetes version range across environments; track each cluster against this range and upgrade it before its current version reaches end of supportRecord any cluster setting that differs from the standard baseline, including the justification, approver, expiry date, and whether it must be restored or reapproved control areawhat to standardizeminimum evidence Kubernetes version Supported version range and upgrade cadence Version inventory showing every cluster within the supported range Cluster baseline Networking, ingress, and baseline policies Declarative config in version control, reconciled to live state Namespaces and quotas Naming, labels, and resource quotas Quota and label audit across clusters Exceptions Approved deviations from baseline Record of approved deviations with justification and expiry date Deployment Consistency and Release Safety Kubernetes makes it easy to ship a change to production several ways: a CI/CD pipeline, a Helm upgrade run by hand, or a kubectl apply straight from a laptop. Each runs different checks, but the manual ones skip the tests and approvals that a pipeline would enforce. A repeatable release path applies the same gates every time and provides a reliable way to recover when a deployment fails. Require every service to follow the same approved deployment path from commit to production, with consistent release steps and controls across teams and environmentsPromote the same versioned, immutable artifact through every environment without rebuilding it at each stageRequire every change to clear the same automated gates (e.g., tests, policy checks, health checks) before reaching productionRoll out production changes in stages (e.g., canary release, percentage-based traffic shift); automatically stop or roll back when predefined health criteria are not metFor every production change, require a rollback, feature disablement, or recovery path that has been tested before releaseFor each deployment, assign an owner accountable for monitoring it through release and triggering rollback on failureRecord every production deployment with its artifact version, approver, and timestamp so the active release stays auditable Observability and Operational Readiness A Kubernetes cluster keeps workloads running by restarting and rescheduling them, so a service can keep failing without the failure ever becoming obvious. A pod stuck in CrashLoopBackOff or failing its readiness probe can remain unhealthy for hours, and if it emits no metrics or logs of its own, there’s nothing to tell you what went wrong. Catching that early depends on each service surfacing its own signals rather than waiting for the cluster to show something is wrong. Require every new service to ship with a minimum observability baseline before production: metrics, structured logs, traces, and liveness and readiness probesDefine service health signals (e.g., latency, traffic, errors, saturation), each with a threshold and assigned team that responds when it is breachedStandardize structured logging and trace context so a request can be followed end to endRoute every alert to an on-call rotation or runbook; retire alerts no one acts onMaintain a quarterly reviewed runbook for each service, including known failure modes, escalation contacts, and recovery stepsSet minimum retention periods for metrics, logs, and traces, with documented justification and explicit approval for shorter retention periodsRun a post-incident review after every major outage; apply findings to update runbooks, alerts, and service baselines Access Controls and Automation Guardrails A Kubernetes cluster usually serves many teams and workloads through a single shared control plane. A role with too much access, for example, can affect them all at once. And when the credential is shared, there’s no way to tell later who actually made the change. Access that stays narrow and tied to a single identity keeps a mistake or a compromised account from impacting the whole cluster. Use namespace-scoped RBAC roles with only the required permissions; grant cluster-wide administrator access only through logged, justified, time-limited exceptionsGive each automation its own scoped service account so automated and privileged actions trace to a distinct identity instead of shared credentialsReserve break-glass access for emergency production changes, with time limits and post-use reviewUse admission policies to reject workloads with unsigned images, privileged containers, or settings barred by platform standardsRecord the actor, target, and timestamp for every privileged or automated action in the Kubernetes audit log; regularly review for activity that does not match an approved change or access requestUse short-lived, automatically rotated ServiceAccount tokens for workloads; revoke credentials and RBAC bindings when a person, workload, or automated process is decommissioned Drift and Failure Management Over time, a Kubernetes cluster’s live state can drift from the configuration stored in version control. This could be due to a hotfix applied directly to a live resource during an incident or an incomplete rollout that leaves the cluster partially updated. If those differences are not fixed, a subsequent deployment may conflict with the live state or overwrite a manual change, and version control may no longer accurately reflect what is running in the cluster. Use automated checks to compare live cluster state with the version-controlled baseline at defined intervals; record each mismatch and notify the team responsible for the affected resourceSet risk-based remediation deadlines for detected drift, requiring teams to restore the baseline or approve a time-limited exception for the changed configuration before the deadlineLog every manual production change and resolve it within a defined period by updating the baseline or reverting the live resource to its declared stateSet an SLO and error budget for each service, identify the team tracking budget use, and pause feature work to prioritize reliability fixes when the budget is exhaustedRun root-cause reviews for recurring failures and apply findings to update baselines, policies, and admission checks instead of patching each instanceTest failure scenarios (e.g., pod disruption, node loss, dependency outages) on a defined schedule, confirm services recover as expected, and track remediation for any gaps example drift patternwhat usually reveals it Manual live-resource change Reconciliation diff against declared state Version or baseline skew Scheduled cluster inventory audit Expired break-glass fix Exception register entry past its window Repeated failure patched one service at a time Same root cause across incident reviews Cost Awareness and Resource Discipline In Kubernetes, resource requests for CPU and memory determine how much cluster capacity is reserved for a workload. Teams may size these requests for peak demand and leave them unchanged even when normal usage is much lower. Across many workloads, this unused capacity adds up and can cause the cluster to run more nodes than actual demand requires, increasing infrastructure costs. Set CPU and memory requests based on representative usage data; set limits where appropriate based on workload behavior and reliability requirementsReview workloads whose requests exceed observed use by a defined threshold, accounting for traffic patterns and reliability needsRequire cost-allocation labels for each workload by team and namespace; correct unallocated spend and missing or inaccurate labelsReclaim idle and orphaned resources (e.g., unused volumes, stale namespaces, oversized nodes) on a monthly cadenceSet autoscaling thresholds based on demand and reliability requirements; periodically review settings that fall outside the approved rangeRegularly review sustained overprovisioning or low utilization; reduce excess capacity or record why it must be retained when avoidable cost exceeds a set threshold Closing Run this checklist before a service enters production and at regular intervals afterward. Repeat it when clusters are upgraded, team responsibilities change, or services are added or retired. Resolve failed checks and revisit approved exceptions before they expire. Unresolved configuration drift can accumulate across environments until teams begin to treat it as the intended baseline. This is an excerpt from DZone’s 2026 Trend Report, Cloud-Native Foundations: Kubernetes, Platform Engineering, and Distributed Operations at Scale.Read the Free Report

By Abhishek Gupta CORE

For application users, waiting for an upload to finish entering a web portal is familiar enough. Waiting again while the application analyzes that upload, however, can leave a user wondering whether anything is happening at all. The application can reassure the user by confirming the upload was received before analysis finishes, but it still needs a way to prevent the uploaded content from being published or used downstream until it's been evaluated against content policies and approved. We can accomplish this for a user-facing upload application by simply separating the submission process from the evaluation process. That allows the application to save the upload in private staging and queue it for background analysis before returning information to the frontend. With this approach, the uploaded content remains unavailable to downstream workflows until after the application records a content approval decision. Let's consider, for example, an editorial portal where photographers can submit their images for some publication. By rule, the publication categorically rejects AI-generated content, so it keeps each upload in private staging while its AI-image detector evaluates photo content in the background. When images meet the portal's policy, they can proceed to the publication workflow; when they don't, they must remain on hold for an editor to carefully review. In this article, we’ll explore building this workflow in C# around an actual AI-generated content detection service, and we'll use the above "editorial portal" concept as a concrete implementation example. We’ll include private staging, a queue, a background worker calling our external image-analysis service, and a recorded decision. The upload endpoint will ultimately return a submission ID to the frontend, which will then use that ID behind the scenes to retrieve the current upload status and display updates beside the uploaded image. Users will then be able to see whether their submission is processing, approved, or awaiting review without ever handling the ID themselves. With this implementation, both the user and the application get exactly what they need. Separating Submission From Evaluation We’ll create a clear divide in our processing workflow between an upload endpoint and a background worker. The endpoint will save the image and queue its submission ID; the worker will retrieve that ID, open the saved image, and request an assessment. This type of separation means the upload response can be returned before detection actually finishes. If we were to await detection inside the endpoint, we would still leave the client waiting for that result. ASP.NET Core provides BackgroundService for implementing the worker, so we’ll take advantage of that in our code. As we’ve described, the frontend will retain the returned ID and use it to retrieve status automatically. The actual user who submitted the image will be able to see “Processing” next to their upload without handling the response ID manually. Just note that the .NET 10 excerpts we’ll use assume an existing authenticated application with some form of upload validation in place. We’ll specifically demonstrate the processing components and their integration points. Creating a Shared Submission Record Right off the bat, we know the endpoint and worker need a shared record that connects an image to its current status. We’ll create this relationship with an immutable record and a concurrent dictionary for our single-instance demonstration: C# using System.Collections.Concurrent; public enum SubmissionStatus { Pending, Processing, Approved, AwaitingReview, Unverified } public sealed record DetectionResult( bool? CleanResult, double? AiGeneratedRiskScore, string? AiSource); public sealed record Submission( Guid Id, string OwnerId, string FilePath, SubmissionStatus Status, DateTimeOffset UpdatedAt, DetectionResult? Detection = null); public sealed class SubmissionStore { public ConcurrentDictionary<Guid, Submission> Items { get; } = new(); } Here, the endpoint creates a Pending record, and the worker replaces it with updated copies as evaluation progresses. This lets status requests read complete snapshots. The AwaitingReview status identifies flagged content, while Unverified identifies an unusable or failed assessment. Approved means the image passed this image-origin policy (not every possible publication or security check; we’ll assume there are several others). Creating the Queue Now we’ll create the actual connection between the endpoint and the worker. A bounded Channel<Guid> holds submission IDs until the worker reads them: C# using System.Threading.Channels; public sealed class ReviewQueue { private readonly Channel<Guid> channel = Channel.CreateBounded<Guid>(new BoundedChannelOptions(100) { SingleReader = true, SingleWriter = false, FullMode = BoundedChannelFullMode.Wait }); public bool TryEnqueue(Guid id) => channel.Writer.TryWrite(id); public IAsyncEnumerable<Guid> ReadAllAsync(CancellationToken ct) => channel.Reader.ReadAllAsync(ct); } Note that in our example, we’ve allowed 100 waiting jobs. Wait mode prevents existing jobs from being discarded, while TryWrite immediately returns false when capacity is completely exhausted. That allows our endpoint to reject the submission instead of sitting around waiting for it indefinitely. Saving and Scheduling the Upload Within our existing upload handler, we’ll first enforce file-size and supported-format checks (this rejects invalid or potentially unsafe files before they're stored or processed). To prevent users from guessing filenames or accessing uploads directly, it’s important that we save the image under a server-generated filename in a private directory. After validation, the below code saves image, an IFormFile. The configured staging directory is privateDirectory, and ownerId comes from the authenticated identity: C# Directory.CreateDirectory(privateDirectory); var id = Guid.NewGuid(); var path = Path.Combine(privateDirectory, id.ToString("N")); var fileCreated = false; try { await using var output = new FileStream( path, FileMode.CreateNew, FileAccess.Write); fileCreated = true; await image.CopyToAsync(output, cancellationToken); } catch { if (fileCreated) File.Delete(path); throw; } store.Items[id] = new Submission( id, ownerId, path, SubmissionStatus.Pending, DateTimeOffset.UtcNow); if (!queue.TryEnqueue(id)) { store.Items.TryRemove(id, out _); File.Delete(path); return Results.StatusCode( StatusCodes.Status503ServiceUnavailable); } return Results.Accepted( $"/submissions/{id}", new { Id = id, Status = "Pending" }); Note that this excerpt uses injected SubmissionStore store and ReviewQueue queue instances; privateDirectory must be an absolute path outside publicly served directories. We’ve designed this so the file is closed before enqueueing, which allows the worker to reopen it. If scheduling fails for some reason, we simply remove the unscheduled record and file. The 202 Accepted response is what confirms acceptance for processing; its location points the frontend to the status endpoint we’ll implement below. Implementing the Detection Call We’ll now put the remote request in a dedicated Detector class. This gives the worker exactly one operation to invoke — DetectAsync(path, cancellationToken). We could theoretically use a hosted classifier for the same role, with our team managing its model and infrastructure. In this case, the implementation calls our AI image-detection endpoint using multipart form data. C# using System.Net.Http.Json; public sealed class Detector( IHttpClientFactory clients, IConfiguration configuration) { public async Task<DetectionResult?> DetectAsync( string path, CancellationToken ct) { var key = configuration["Cloudmersive:ApiKey"]; if (string.IsNullOrWhiteSpace(key)) throw new InvalidOperationException( "Configure the Cloudmersive API key."); using var client = clients.CreateClient("Cloudmersive"); using var file = File.OpenRead(path); using var form = new MultipartFormDataContent(); form.Add( new StreamContent(file), "imageFile", "submission"); using var request = new HttpRequestMessage( HttpMethod.Post, "image/ai-detection/file"); request.Headers.Add("Apikey", key); request.Content = form; using var response = await client.SendAsync(request, ct); response.EnsureSuccessStatusCode(); return await response.Content .ReadFromJsonAsync<DetectionResult>( cancellationToken: ct); } } Note that the API key here comes from server-side configuration. The method just opens the staged image, submits its bytes, and then deserializes the response on the other end. Any HTTP failures cause an exception for the worker to handle. If you happened to read my previous article on AI detection, you’ll know that CleanResult supplies the summarized classification for this service, and AiGeneratedRiskScore provides the risk signal we’re looking for. AiSource then adds optional source context (this is information about the exact AI model that created the image; it’s not always available, even when images are deemed high-probability AI). The key thing to note is that we’re working with probabilistic signals rather than definitive proof of authorship. Turning the Assessment into a Decision We’ll now implement the decision portion separately so the worker can apply the same rule to every response: C# public static class ReviewPolicy { public static SubmissionStatus Evaluate(DetectionResult? result) { if (result?.AiGeneratedRiskScore is not double score || !double.IsFinite(score) || score < 0 || score > 1 || result.CleanResult is null) { return SubmissionStatus.Unverified; } // Illustrative application threshold. const double reviewThreshold = 0.5; return result.CleanResult == false || score > reviewThreshold ? SubmissionStatus.AwaitingReview : SubmissionStatus.Approved; } } In this example, we approve usable results with a clean classification and an AI-generated-probability score no higher than 0.5. Processing Jobs and Saving Outcomes At this point, we can connect all the pieces. Our worker will read an ID, load its record, mark it as “processing,” and call Detector. After that, it'll replace the record with the response and policy decision: C# public sealed class ReviewWorker( ReviewQueue queue, SubmissionStore store, Detector detector, ILogger<ReviewWorker> logger) : BackgroundService { protected override async Task ExecuteAsync(CancellationToken ct) { try { await foreach (var id in queue.ReadAllAsync(ct)) { if (!store.Items.TryGetValue(id, out var item)) continue; store.Items[id] = item with { Status = SubmissionStatus.Processing, UpdatedAt = DateTimeOffset.UtcNow }; try { var result = await detector.DetectAsync( item.FilePath, ct); store.Items[id] = item with { Status = ReviewPolicy.Evaluate(result), Detection = result, UpdatedAt = DateTimeOffset.UtcNow }; } catch (Exception ex) { store.Items[id] = item with { Status = SubmissionStatus.Unverified, UpdatedAt = DateTimeOffset.UtcNow }; if (ct.IsCancellationRequested) break; logger.LogError( ex, "Evaluation failed for {Id}", id); } } } catch (OperationCanceledException) when (ct.IsCancellationRequested) { // Application shutdown while waiting for work. } } } We’ve handled each failure inside the loop such that unsuccessful scans don’t prevent subsequent jobs from happening. Note that in this compact version, we only perform one attempt. If we were to add retries, that would require bounded attempts, along with transient-error classification and respect for Retry-After. Before builder.Build(), we register the shared components and configure the HTTP client: C# builder.Services.AddSingleton<SubmissionStore>(); builder.Services.AddSingleton<ReviewQueue>(); builder.Services.AddSingleton<Detector>(); builder.Services.AddHostedService<ReviewWorker>(); builder.Services.AddHttpClient("Cloudmersive", client => { client.BaseAddress = new Uri("https://api.cloudmersive.com/"); client.Timeout = TimeSpan.FromSeconds(60); }); Requests are connected with the worker via the singleton store and the queue. The timeout bounds each remote call (its value is an application choice). Returning Status and Enforcing Approval The frontend can now poll a status route. This handler checks the authenticated owner before returning a limited view of the record. We’ll place both routes after builder.Build() and before app.Run(), with the using directive at the top of Program.cs: C# using System.Security.Claims; app.MapGet("/submissions/{id:guid}", (Guid id, ClaimsPrincipal user, SubmissionStore store) => { var ownerId = user.FindFirstValue(ClaimTypes.NameIdentifier); if (string.IsNullOrWhiteSpace(ownerId) || !store.Items.TryGetValue(id, out var item) || item.OwnerId != ownerId) { return Results.NotFound(); } return Results.Ok(new { item.Id, Status = item.Status.ToString(), item.UpdatedAt }); }).RequireAuthorization(); The host needs to validate authentication and provide the same stable owner claim used during the upload process. The frontend will use the returned status to update its interface automatically. That same ownership check is performed by the download route. This additional condition is then applied before the file is returned: C# app.MapGet("/submissions/{id:guid}/file", (Guid id, ClaimsPrincipal user, SubmissionStore store) => { var ownerId = user.FindFirstValue(ClaimTypes.NameIdentifier); if (string.IsNullOrWhiteSpace(ownerId) || !store.Items.TryGetValue(id, out var item) || item.OwnerId != ownerId) { return Results.NotFound(); } if (item.Status != SubmissionStatus.Approved) { return Results.Conflict(new { Error = "Submission is not approved." }); } return Results.File( item.FilePath, "application/octet-stream", fileDownloadName: "submission"); }).RequireAuthorization(); Note that private storage will prevent direct public access from bypassing this check, so human reviewers need a separate authorized access path. Testing and Production Considerations To verify things like approval, review, and missing-result handling, we can substitute controlled detector responses. In the event we get simulated request failures, they should produce Unverified, while downloads should remain blocked during processing. Critically, another user’s request should NOT reveal the submission. It’s important to bear in mind that this demo code also loses its queue and records on restart. That’s obviously not production-ready, since production usually requires durable jobs and records on top of coordinated enqueuing and duplicate-job handling. Additionally, we should probably retain things like creation timestamps and attempt history in our production code. Conclusion In this article, we learned how to connect private portal uploads to a background AI-content detector through a shared record and queue. Our worker records an explicit outcome, our frontend retrieves it automatically, and our download route enforces content approval. With this workflow in place, our application can acknowledge submissions promptly while keeping unfinished and unsuccessful content evaluations out of normal publication workflows.

By Brian O'Neill CORE

Software engineering is often seen as a technical field focused on designing systems, writing code, selecting architectures, defining APIs, and optimizing for performance, scalability, and maintainability. However, many of the most challenging problems are not technical, but human. As engineers advance into leadership roles such as senior engineer, Staff Engineer, architect, or engineering manager, the challenge extends beyond technical solutions. Leaders must also understand how people make decisions, how teams respond to change, and how individual behavior can influence software quality. Humans are not always rational. We depend on habits, shortcuts, intuition, experience, and assumptions. While these can improve efficiency, they may also lead to decisions that don't align with requirements, constraints, or long-term goals. A team may retain a design simply because “we have always done it this way.” Engineers might accept poor code quality if the existing codebase is already disorganized. Teams may also select a complex architecture because complexity appears more robust, even when a simpler solution would suffice. While these decisions may seem minor individually, over time they can impact more than a single implementation. They may influence team behavior, increase technical debt, reduce maintainability, and shape the system’s architecture. This article examines three common behavioral traps that affect software engineering decisions: status quo bias, the broken windows effect, and complexity bias. We will also discuss how engineers and technical leaders can identify these patterns and foster more deliberate, evidence-based decisions. What Are Cognitive Biases, and How Do They Affect Software? Cognitive biases are systematic patterns in how people interpret information, evaluate alternatives, and make decisions. Rather than random mistakes, they are mental shortcuts that help us act quickly but can also influence our judgment without our awareness. This is especially relevant in software engineering, where decisions are often made under uncertainty, time constraints, incomplete information, and the influence of prior experience. Engineers rarely evaluate every alternative from scratch. Instead, we reuse patterns, rely on intuition, trust familiar technologies, and simplify complex situations to maintain progress. Most of the time, this is useful. Software development would be impossibly slow if every decision required exhaustive analysis. Problems arise when these shortcuts replace thoughtful reasoning instead of supporting it. For example, a team may retain an implementation simply because it exists, not because it meets current requirements. Developers might accept poor practices if the codebase already appears neglected. Architects may select more complex solutions because complexity seems more robust or professional. These effects can extend far beyond the initial decision. A cognitive bias that starts as an individual preference can influence code reviews, become a team convention, shape architectural decisions, and eventually contribute to technical debt. The impact therefore goes beyond a single decision. Biases can affect: Code quality and maintainabilityArchitectural consistencyTechnical debtTechnology selectionResistance to changeTeam standards and engineering culture The most concerning aspect is that biased decisions rarely seem irrational when made. They often appear reasonable: “this is how we normally do it,” “the code is already messy,” or “we need something more robust.” Recognizing these patterns does not eliminate them, but it enables engineers and technical leaders to challenge assumptions before they become embedded in the architecture. This understanding also provides a natural transition to the following sections: status quo bias, the broken windows effect, and complexity bias. Status Quo Bias: “We’ve Always Done It This Way” Status quo bias is the tendency to favor the current situation solely because it is established. Samuelson and Zeckhauser showed that people often favor existing options, even when alternatives are available: Status Quo Bias in Decision Making. In software engineering, this bias can be more harmful than simple resistance to change, as it may keep teams from evaluating decisions against current circumstances. Software evolves continuously. Requirements shift, languages gain new features, frameworks improve, and teams learn from experience. A decision that was appropriate years ago may no longer be suitable today. Problems arise when teams assume the current solution is correct by default. “We’ve always done it this way.” How Status Quo Bias Impacts Software Engineering The greater risk is that teams may stop analyzing whether existing decisions still fit the current context, not just that they avoid change. For example, a Java team may continue using XML configuration simply because it was the original convention, even when annotations or programmatic configuration would better suit parts of the system today. The problem is not XML itself. XML may still be the right choice. The issue arises when the reasoning is: we used XML before, so we use XML now. This differs from evaluating current requirements, available language and framework capabilities, trade-offs, and then selecting the most appropriate configuration strategy. The same pattern can appear almost anywhere: “We always store this in the session.”“We always use inheritance here.”“We always create a microservice for this.”“We don’t use that newer Java feature.”“This framework has always worked for us.” Each of these statements may reflect a valid decision, but none alone provides sufficient technical justification. The broader risk is that architecture may reflect outdated constraints rather than current needs. Teams may overlook new language features, framework improvements, better APIs, production lessons, infrastructure changes, or architectural alternatives unavailable when the original decision was made. Over time, the codebase may remain internally consistent but become increasingly disconnected from its surrounding ecosystem. Research on software architecture decision-making shows these choices are not purely rational and can be influenced by bounded rationality and cognitive biases: Decision Making in Software Architecture. How to Handle Status Quo Bias in a Team The goal is not to challenge every existing decision or adopt every new technology. Consistency, migration cost, operational risk, and familiarity are all valid engineering concerns. The key is to ensure that history informs the decision, but does not determine it. A simple question can help: If we were making this decision today, with our current requirements and tools, would we make the same choice? This separates two different questions: What is the best design for the current context?Is changing the existing design worth the cost? Those are not the same problem. Another useful technique is to make evaluation criteria explicit. Compare the current approach and alternatives using the same dimensions: maintainability, complexity, operational risk, migration cost, framework support, and long-term evolution. This approach creates a fairer discussion, as the current solution no longer receives an automatic advantage simply because it exists. Naming the Bias Can Move the Discussion Forward One effective leadership technique is to name what may be happening. Instead of saying: “You are resisting change.” A staff engineer, architect, or manager can ask: “Could status quo bias be influencing this decision?” This shifts the discussion from evaluating individuals to evaluating the decision-making process. Someone may then realize: “I prefer this solution because I know it well, not because we compared the alternatives.” That distinction matters. Experience says: “We tried this before, and here are the trade-offs we learned.” Inertia says: “We do this because we have always done it.” Effective technical leadership involves helping teams recognize this distinction. The goal is not modernization for its own sake, nor is newer automatically better. The aim is to ensure architecture evolves with the context, rather than remaining unchanged simply because the current solution is familiar. Complexity Bias: “It Can’t Be That Simple” Complexity bias is the tendency to trust a complicated solution more because it appears more complete, sophisticated, or robust. Simple solutions can feel uncomfortable, especially for important problems, leading us to expect equally complex architectures. Research on causal reasoning shows that people often view complexity as a sign that an explanation fits the evidence, even though they prefer simplicity in other contexts. See Simplicity and Complexity Preferences in Causal Explanation (ScienceDirect). In software engineering, this bias often leads to overengineering. How Complexity Bias Impacts Software Engineering The main problem is not complexity itself. Some systems are legitimately complex. The real issue is unnecessary complexity, as each abstraction, service, dependency, process, and architectural layer adds risk. Additional components increase the effort required to understand, integrate, test, deploy, monitor, and modify the system. Solutions meant to improve safety can instead expand the potential for failure and raise the cost of future changes. Simple requirements can accumulate queues, events, retries, services, orchestration, and frameworks, making the architecture seem more robust even when these additions are not justified. Forbes identifies over-engineering solutions as one of the bad habits technology leaders want developers to avoid, noting that developers sometimes solve problems that don't exist or add complexity before fully understanding the problem (Forbes). Complexity bias also influences organizational practices. Teams may add approval stages, meetings, documents, and processes because increased control feels safer. Eventually, even agility suffers. Teams may become attached to processes without evaluating whether they still support effective software delivery. Just Enough Architecture George Fairbanks’ concept of Just Enough Software Architecture offers a useful counterpoint. His risk-driven approach suggests that architectural effort should match the actual risk of failure. When risk is low, extensive design can be wasteful. When risk is high, more deliberate design is warranted (George Fairbanks). This gives us a better question than: “How can we make this architecture more robust?” Ask instead: “What risks are we trying to mitigate, and how much architecture do those risks justify?” That changes the discussion significantly. An extra service may be justified by isolation requirements. Asynchronous communication may be justified by availability or throughput. Regulation may require additional governance. However, you should always justify complexity. Fairbanks summarizes the principle well: architectural effort should match the risk of failure (George Fairbanks). How to Handle Complexity Bias in a Team A useful starting question is: What is the simplest solution that satisfies the requirements and mitigates the risks we actually have? Technical leaders can then challenge complexity with concrete questions: What risk does this abstraction reduce? What requirement needs another service? What would happen if we did not introduce this complexity yet? Naming the bias can also help: “Are we adding this because the problem requires it, or because the simpler solution feels too simple?” The goal is not simplicity at any cost. Underengineering poses risks similar to overengineering. The goal is just enough architecture: introduce the complexity needed to manage real risks, but no more than the problem justifies. Broken Windows: When the Environment Lowers the Standard Unlike status quo or complexity bias, Broken Windows Theory is not a cognitive bias. Instead, it represents a risk: visible disorder can influence future behavior and gradually redefine what is considered acceptable. Although the concept originated outside software engineering, researchers have applied it to code quality for decades. Recently, researchers examined whether this effect occurs in software systems. A 2024 study in Empirical Software Engineering found that developers working in systems with higher technical debt were more likely to introduce additional debt, such as poor variable names, duplicated behavior, and other issues detected by static analysis (Springer Link). How Broken Windows Impact Software Engineering A neglected codebase sends signals. When engineers repeatedly encounter duplicated code, weak tests, inconsistent naming, abandoned abstractions, or ignored warnings, the implicit message can become: “This level of quality is acceptable here.” That can change behavior. Rather than considering how the code should ideally be structured, developers start adapting to their environment: “This module is already messy.” “Nobody tests this part anyway.” “There are already dozens of warnings.” “One more shortcut will not make a difference.” The risk is cumulative. While a single compromise may seem minor, repeated compromises can gradually lower the entire system's engineering standards. The 2024 controlled experiment is notable because it provides direct evidence of this effect. Developers working on systems with higher technical debt were more likely to introduce new debt (Springer Link). A second 2024 study examining code history also found that existing code quality can influence the quality of subsequent changes, though the effect varies by quality characteristic (arXiv). The Risk Goes Beyond Code Broken windows can also appear in engineering culture. If architectural rules are routinely ignored, pull requests are poorly reviewed, failing tests are tolerated, or technical debt is continually postponed, developers learn what the organization truly values. Written standards may say one thing, while the environment says another. That distinction matters because engineering culture is shaped not only by what leaders say, but also by what teams repeatedly observe. A codebase can therefore create its own feedback loop: Poor quality normalizes lower standards. These changes create additional visible disorder, which further reduces expectations. How to Handle It in the Team The goal isn't to create a perfectly clean system, as that can lead to overengineering. Instead, teams should prevent visible problems from becoming the accepted baseline. Small actions matter: Fix obvious problems when touching nearby codeAddress warnings and failing tests promptly, preventing them from becoming permanent background noiseSet a minimum quality standard for new changesMake technical debt visible rather than silently normalizing it Technical leaders can also name the pattern directly: “Are we accepting this because it is genuinely the right trade-off, or because the surrounding code has already lowered our expectations?” This question helps distinguish between pragmatic technical debt and simple degradation. The objective isn't perfection, but preventing the environment from silently teaching the team that quality no longer matters. A useful principle is: Every codebase teaches developers how it expects to be treated. Once poor quality becomes the norm, reversing that expectation becomes much more difficult.

By Otavio Santana CORE

Retries are one of the simplest ways to make a distributed system appear more reliable. A transient connection failure, overloaded replica, or short-lived network interruption can disappear after another attempt, which is why retry support exists in major RPC frameworks and cloud SDKs. The danger begins when every layer makes the same decision independently. A mobile client retries an API gateway, the gateway retries a service, that service retries another service, and the final dependency retries a database call. The original request has not become more important, but the system has multiplied the work required to fail. AWS describes a five-deep service stack in which three attempts at each layer can drive 243 calls against the database when the deepest dependency is failing. Google’s SRE guidance similarly warns that retries can amplify overload and contribute to cascading failure. When Reliability Logic Becomes Additional Load The common retry policy focuses on a single caller. A request fails, exponential backoff delays the next attempt, and jitter prevents large client populations from retrying at exactly the same instant. Those mechanisms remain important. AWS recommends backoff and jitter because immediate, synchronized retries can worsen overload, while gRPC exposes retry limits, exponential backoff, retry throttling, and server pushback for the same class of problem. The missing property is coordination. Consider three logical layers, each configured for three total attempts. If the lowest dependency rejects every request, a single logical operation can create up to 27 downstream attempts. Adding more independently retrying layers increases that multiplier exponentially. Backoff changes when those attempts arrive; it does not change the fact that separate components are authorizing additional work from the same original operation. A typical Spring service can accidentally create this behavior with perfectly reasonable local configuration: Java @Retry(name = "paymentService", fallbackMethod = "paymentFailed") public PaymentResult charge(PaymentRequest request) { return paymentClient.charge(request); } private PaymentResult paymentFailed(PaymentRequest request, Exception ex) { throw new PaymentUnavailableException(ex); } Nothing in this method indicates whether the incoming request has already consumed retries elsewhere. A gateway may already have retried the service, and paymentClient may apply another retry policy. Local resilience therefore becomes global amplification. A Retry Budget Changes the Decision A retry budget treats retries as limited capacity rather than an unconditional reaction to failure. Google documents two complementary controls in its overload handling: a per-request cap of three attempts and a per-client budget that permits retries only while retries remain below 10% of request traffic. In the example described by Google, the per-client budget reduces retry-driven traffic growth from almost three times the original request rate to roughly 1.1 times under the modeled overload condition. Finagle applies the same general idea through a shared RetryBudget, explicitly describing the budget as protection against the amplifying effect of many clients retrying. For a service chain, the useful abstraction is a request-scoped budget propagated with the operation. An internal header such as X-Retry-Budget can represent remaining retry permits. The header is an application convention rather than a standard HTTP field, its purpose is to ensure that downstream components consume from the same finite allowance. The retry decision can then become explicit: Java boolean canRetry(int remaining, HttpStatusCode status) { return remaining > 0 && (status.value() == 429 || status.is5xxServerError()); } int nextBudget(int remaining) { return Math.max(0, remaining - 1); } A caller starts a logical operation with a small budget, such as two retry permits. Every additional attempt decrements the value before forwarding the request. A downstream service receiving zero can still return a meaningful failure, but it cannot create more retry traffic for that logical operation. This model should not make every 5xx automatically retryable. Retry classification still matters. Validation failures, deterministic application errors, and non-idempotent operations can be unsafe or pointless to repeat. AWS recommends idempotent API contracts when operations may be retried and describes caller-provided request identifiers as a way to recognize duplicate intent. Propagating One Budget Across Service Boundaries Budget propagation belongs close to outbound transport logic so business methods do not manually manipulate retry metadata. A Spring interceptor can read the current budget and attach the decremented value to the next attempt: Java int remaining = retryContext.remaining(); if (remaining <= 0) { throw new RetryBudgetExhaustedException(); } request.getHeaders().set( "X-Retry-Budget", Integer.toString(remaining - 1) ); return execution.execute(request, body); The receiving service extracts the header once and places the value in the request context. Internal HTTP clients and RPC adapters then share that context. This is conceptually similar to distributed context propagation used by tracing systems. OpenTelemetry propagators inject and extract cross-cutting context through carriers such as HTTP headers, although retry-budget metadata can remain a dedicated internal header rather than telemetry baggage. A budget also needs to cooperate with deadlines. A remaining retry permit is useless when the logical request has only a few milliseconds left. Retry authorization should therefore require both budget and time: Java boolean retryAllowed(RetryContext context) { return context.remaining() > 0 && context.deadline().isAfter(Instant.now().plusMillis(100)) && context.lastFailure().isTransient(); } Server feedback should override generic retry enthusiasm. HTTP defines Retry-After so a service can indicate when a follow-up request should occur, including with 503 Service Unavailable, 429 Too Many Requests can also carry Retry-After. A budget answers whether another attempt is permitted, while server feedback helps decide when that attempt is appropriate. Measuring Whether the Budget Is Working Retry budgets are control mechanisms, so observability must expose both logical requests and physical attempts. Finagle distinguishes logical success from individual attempts and publishes metrics for retry budget availability, exhaustion, and request retry limits. Without that separation, retries can hide dependency instability because a successful second attempt makes the logical request appear healthy while infrastructure performs additional work. Useful telemetry should record the initial request count, retry attempt count, budget exhaustion count, retry success rate, response classification, remaining budget, and end-to-end latency. The critical ratio is retry amplification, which is total physical attempts divided by logical requests. A healthy value depends on workload characteristics, but a sharp increase during an incident indicates that resilience logic is becoming a load. Tracing adds the missing causal view. Each attempt can remain a child span of the same logical operation, with attributes such as retry.attempt, retry.remaining, and retry.reason. The resulting trace shows whether an operation failed because a dependency was unavailable, because the deadline expired, or because the shared budget prevented another attempt. That distinction is operationally important as budget exhaustion is often evidence that the system deliberately stopped adding pressure rather than evidence that the retry mechanism malfunctioned. Retry metrics also need to be interpreted alongside service saturation and rejection rates. A rising retry-success rate may initially indicate useful recovery from transient faults, but rising attempt volume combined with increasing backend saturation indicates a different condition. At that point, preserving capacity can be more valuable than pursuing another successful attempt. Google’s overload guidance explicitly recommends allowing failures to propagate when widespread backend overload makes additional retries unlikely to help. Conclusion Retries remain essential for transient failures, but retries without coordination can turn a partial outage into a traffic multiplier. Backoff, jitter, idempotency, deadlines, and server pushback address important parts of the problem that a retry budget adds the missing global constraint by limiting how much extra work one logical operation may create. Propagating that budget across service boundaries converts retry behavior from isolated local policy into distributed load control. The strongest resilience policy is therefore not “retry until success,” but “retry only while the failure is transient, the operation is safe, time remains, and the system can afford another attempt.”

By Uthej Mopathi CORE

Quick answer: Full-stack AI engineering is the practice of building an AI feature from end to end: the data and retrieval layer, the model and orchestration layer, the application interface, and the production monitoring that keeps it working. It treats a model as one component inside a larger system rather than the product itself. I've spent the past few years watching teams move from "we called an LLM API and shipped a demo" to "we run this thing in production, and it can't fall over on a Tuesday." The gap between those two states is where full-stack AI development actually lives. This article walks through what that work involves, where projects tend to break, and how to think about the decisions that matter in 2026. What Does "Full-Stack" Mean When the Stack Includes a Model? Traditional full-stack work covers frontend, backend, and database. AI adds three layers that behave differently from anything else in the stack: A data and retrieval layer: vector stores, embeddings, chunking strategy, and the plumbing that feeds context to a model.A model and orchestration layer: model choice, prompting, tool calling, and the control flow that strings multiple steps together.An evaluation and observability layer: the part most teams skip and later regret. Full-stack AI development is its own discipline because these layers are non-deterministic. A button either works or it doesn't. A model gives a different answer to the same input on Monday and Thursday, and both can be defensible. Engineering around that uncertainty is the actual job. Where Real Projects Break Demos are cheap. Production is where the cost shows up. A few patterns repeat across almost every generative AI development project I've reviewed. The 80/20 flip. The first 80% of a feature takes a weekend. The last 20%, handling edge cases, weird inputs, malformed tool responses, and users who paste 40 pages into a chat box, takes months. Teams that budget for a demo timeline instead of a production timeline miss deadlines by wide margins. Nothing to measure against. If you can't measure quality, you can't tell whether a prompt change helped or hurt. Teams that ship without offline evals end up making changes based on vibes, then discover regressions through user complaints. Building an eval set of real inputs with expected behavior is unglamorous and completely necessary. Cost and latency are ignored until launch. A workflow that chains six model calls feels fine with one test user and becomes unaffordable at scale. Custom AI development that succeeds usually routes cheap tasks to small models and reserves large models for the steps that need them. Agentic AI Raises the Stakes The dominant shift in 2026 is the move from single-shot prompts to agentic systems: models that plan, call tools, read the results, and decide what to do next. This is where a lot of the automation value sits, and it's also where reliability gets hard. Every extra step in an agent loop is another place for the system to go sideways. If one step is 95% reliable, a five-step chain is only about 77% reliable from start to finish. Serious agentic AI development spends most of its effort on the boring parts: retry logic, validation between steps, bounded loops so an agent can't spin forever, and human checkpoints for anything irreversible. The teams doing this well treat an agent less like a magic worker and more like a distributed system running over a flaky network. That framing leads to better engineering decisions than treating the model as an oracle. The Decisions That Shape an AI Project Whether you're an in-house team or evaluating an outside AI development company, a handful of early choices set the ceiling on what you can build. Hosted API or Open Weights? Hosted models are faster to start with and stay current without effort. Open-weight models give you control over cost, data residency, and fine-tuning. Most production systems end up mixed: a hosted frontier model for hard reasoning, a smaller self-hosted model for classification and routing. Where Does the AI Actually Belong? Not every problem needs a language model. Some tasks are better served by a rule, a search index, or a classic ML model that runs in a millisecond for a fraction of a cent. Good AI integration means putting the model only where its flexibility earns its cost. How Will You Measure Success Before Shipping? Define what "good output" means in concrete, checkable terms before you write the feature. This single habit separates teams that iterate with confidence from teams that argue about prompts forever. Why Enterprise Adoption Changed the Requirements Enterprise adoption in 2026 has pushed AI work past the prototype stage, and with it came a longer checklist. Data governance, access control, audit trails, and the ability to explain a decision are now table stakes for anything touching regulated data. A large part of what AI consulting engagements and internal platform teams spend their time on is not model magic, but the controls that let a model operate near sensitive systems without creating risk. This maturity is healthy. It means AI is being treated like real infrastructure instead of a science experiment, and it rewards teams that already work like software engineers: version control, testing, staged rollouts, and rollback plans. What Good Full-Stack AI Work Looks Like in Practice Teams doing this well tend to share a few habits. They keep a versioned eval set and run it on every change. They log full traces of model inputs and outputs so they can debug what actually happened. They set spending and latency budgets per feature. They design fallbacks for when the model fails, because it will. And they resist adding a model to a problem a simpler tool can solve. Those habits are less exciting than a slick demo, but they are what carries a project from "impressive in a meeting" to "still running in six months." Whether the work happens in-house or through a custom AI development partner, the fundamentals don't change. The value sits in the engineering discipline around the model, not in the model call itself. FAQs 1. What skills does a full-stack AI engineer need? Working knowledge of backend systems, comfort with model APIs and prompting, familiarity with vector databases and retrieval, and, most important, the ability to build evaluation and monitoring for non-deterministic behavior. 2. Is full-stack AI development different from MLOps? They overlap. MLOps focuses on the model lifecycle: training, deployment, and monitoring. Full-stack AI development is broader, covering the whole application around the model, including retrieval, orchestration, and user experience. 3. Do I always need a custom model? No. Most generative AI development in 2026 uses hosted or open-weight models as they are, with custom work going into retrieval, prompts, orchestration, and evaluation rather than training a model from scratch. When should a team bring in outside help? Outside AI development services or consulting make sense when a project needs production reliability quickly, and the team lacks experience with evals, agent orchestration, or the governance that enterprise deployment requires.

The first warning sign wasn't an outage. It was a boring pull request. We changed one App Service setting. It was the sort of change that should have resulted in a small plan and a quick review. Instead, Terraform refreshed networking, private endpoints, DNS, Key Vaults, storage accounts, app services, and monitoring before showing what would actually change. Nothing was broken; that was the point. Terraform did exactly what it was designed to do: account for everything represented in state before calculating change. The problem was that our Terraform state had become a single, platform-sized boundary that every small change had to pass through, and one no team could fully own. If you have run a landing zone as a single Terraform configuration, you have probably had a version of that pull request. The instinct afterward is to blame size: the configuration has grown too large, so break it up. That instinct is wrong, or at least incomplete. Size is uncomfortable, but coupling is what actually hurts. Nothing in the change touched networking, DNS, or those key vaults. They were dragged into the plan because everything was bound together through one state. At first, that coupling just means slow plans and noisy reviews. Later, it raises a harder question: who actually owns this? Where the Coupling Shows Up Start with the plan. In a monolith, Terraform has to account for everything represented in the state before it can tell you what changed. You can target a single resource, but that is an escape hatch, not a way to run a platform. So the wait scales with the size of the estate, not your change. Both a one-line edit and a fifty-resource migration get stuck behind the same refresh before the diff appears. Provider upgrades show the same problem. A single root configuration pins one set of provider versions, so you cannot move networking to a newer azurerm version and leave everything else behind. Every upgrade becomes all-or-nothing, which means it keeps losing to smaller, safer priorities. Ours sat on azurerm 2.97 and only moved to the 4.x line once the upgrade could no longer be put off. The monolith had made the jump too big to schedule any sooner. The bigger concern is blast radius. One state file, one lock, one plan. A bad apply, a corrupted state, a destroy that catches more than you aimed at: whatever goes wrong can reach more of the platform than the change was ever meant to touch, because nothing in the layout is there to contain it. The dependency graph suffers too. Unrelated resources get sequenced together just because they share a graph. A network change might wait on unrelated compute, DNS on policy. The graph ends up reflecting accidental grouping rather than real dependencies. The result is clear. There is no small change. You cannot ship a DNS record or a new Key Vault without running the entire configuration through plan and apply. Every change is a platform change, carrying platform risk and requiring review, no matter how minor. These look like separate problems, but all come from the same design choice: too many unrelated concerns tied into one Terraform boundary. Where Coupling Becomes Ownership It is easy to call these operational annoyances: slow plans, awkward upgrades, risky applies, the tax you pay for a big configuration. But the same coupling appears in review and approval, where it stops being just an operational problem. Once too many concerns share the same state, pipeline, and approval path, the question is no longer only "how long did the plan take?" It becomes "who is accountable for the boundary this change is crossing?" Take private connectivity. A single private endpoint on Azure isn't handled by just one team. The application team owns the service behind it. The platform team manages the landing zone, subnet, and endpoint placement. Private DNS zones might be managed centrally or by another team. Security or governance may require the service to be private. How these map to teams varies, but in a monolith, everything ends up in the same state, pipeline, and plan. So "who owns this?" rarely has a clear answer. However you split teams, they are coupled through a single configuration that none can truly own. When the application team changes its service, the same config still carries platform connectivity and governance controls. You cannot draw ownership along your real organizational boundaries, because the code does not have them. Both slow plans and unclear ownership trace back to the same issue: shared concerns treated as if they belong to just one team. Figure 1: When Terraform boundaries stop matching ownership boundaries. The monolith gives Terraform one boundary. Organizations have several. The pain comes when small changes have to cross boundaries that no team fully owns. Reach for the Coupling, Not the Size The reflex now is to split the state and move on. But splitting a landing zone poorly can be worse than leaving it alone. If you split along the wrong lines, you trade one blast radius for tangled cross-state dependencies. You also lose the single plan that at least showed the whole graph in one place. For example, splitting private endpoints into one state and private DNS zones into another may look clean on paper. But if different teams deploy them without a clear agreement, every new endpoint becomes a coordination headache, not a smaller change. Moving files into separate folders does nothing if the same pipeline, credentials, and approval path still govern everything. Decomposition should follow actual coupling, not just line count. So the next question is not "how many states should we create?" It is "which boundaries are real enough for teams to own, deploy, and recover independently?" If your Terraform monolith hurts, do not start by counting files or resources. Look at what is actually being coupled. Slow plans and unclear ownership are both signs that your Terraform boundaries no longer match your real ownership boundaries.

Most modern applications do not function completely independently. For example, analytics, payment processing, user authentication, customer support, testing new features (experimentation), monitoring the app's performance, advertising, etc., are typically provided as third-party SDKs that enable those functions in your app. Using an SDK has its benefits; you don't have to build an entire piece of functionality yourself. When using an SDK, developers can download the software library, call the initialization method, then begin calling the API methods of the SDK to use its functionality. JavaScript import { analytics } from "third-party-sdk"; analytics.track("checkout_started", { productId: "123" }); In some cases, the amount of code required to add this type of functionality can be as little as a handful of lines of code. If the same functionality was built "from scratch", the time required could potentially be several weeks. However, with great convenience comes hidden complexity. The moment you allow a third-party SDK to run in your app, how well it performs and works (performance, reliability, security, and user experience) depends on what amounts to "someone else" doing something to your app. That is why third-party SDKs are important dependencies that affect how well your app will perform during production hours, instead of just being another library or module to include. SDKs Can Quietly Affect Performance The biggest reason front-end SDKs will show performance issues is that they typically run directly in your web browser. If you install a typical analytics SDK, it adds to your front-end bundle, it loads on page init, and then registers event handlers, makes requests over the internet, etc., as soon as there are interactions with your app. Although one SDK alone has little effect, if you use multiple SDKs for analytics, experimentation, customer service, session replay, ad tracking, and monitoring, your users will likely notice a difference. Therefore, teams need to evaluate whether individual "acceptable" SDK costs can compound into overall user-perceived degradation. In addition to measuring the cost of each SDK individually, teams need to look at the overall cost of loading the SDK(s), which can include: Bundled sizeTime to initializeNetwork requestsActivity on main threadOverall impact on Core Web Vitals This cost can be reduced by loading non-critical SDKs asynchronously or after the main application experience has loaded. A Third-Party Failure Can Become Your Failure Consider an application that will render the main page after initializing a recommendation SDK. JavaScript await recommendationSDK.initialize(); renderApplication(); If the third-party service has an issue, your application's overall appearance may slow or become unavailable, even if your backend is functioning properly. Thus creating unneeded coupling. Generally speaking, non-essential third-party services should be allowed to fail without affecting the primary user experience. For example, if you're unable to receive recommended products, you should still be able to browse through products; if analytics are failing, checkout should still function as normal; and if a support widget is unable to load, all other aspects of the webpage should continue to function normally. Applications should define clear fallback behavior for every external dependency. Timeouts are also important. Waiting indefinitely for a third-party service can turn a small external outage into a much larger product incident. SDK Updates Can Change Production Behavior Engineers typically spend considerable time evaluating large-scale framework updates; however, they may be less concerned about small third-party dependencies that make up much of their application codebase. This could potentially lead to issues. An SDK update can change how an application initializes, the format for making requests, which browsers an application supports, the default configuration, how data is stored, or how much JavaScript is downloaded during each session. Even if the public API hasn't changed, runtime behavior may still differ based on previous SDK versions. Therefore, dependency upgrades should follow standard engineering controls such as version pinning where applicable; automated testing; dependency review; and gradual deployment. The idea of automatically allowing all new SDK releases into production just because they have been classified as minor will create additional risk. Third-Party Code Expands the Security Boundary Every new SDK you add to your app will be a larger portion of all code making up the system. Browser apps make this especially important when SDKs can access page content, browser storage, cookies, user interaction, or even application data. You should know exactly which pieces of information will go out to an outside party. As an example, sending off an entire object to an analytics SDK could provide more information than was ever intended: JavaScript analytics.track("profile_updated", user); Some of the fields in the 'user' object might never have been intended for analytics. A safer approach is to explicitly select the information required for the event. JavaScript analytics.track("profile_updated", { accountType: user.accountType }); You'd be better off sending only the data you need for each specific event. The principle is simple: third-party integrations should receive only the data they actually need. SDKs Can Create Hidden Runtime Conflicts Not all third-party SDKs run independently. In addition to other actions such as modifying a browser's global objects, registering event handlers, intercepting web requests, and manipulating the DOM, third-party SDKs may also create new dependency conflicts that are incompatible with your current application code. Because of their nature, these issues can be difficult to reproduce because they typically depend on specific conditions (such as browser type, user environment, feature flags/feature toggle configuration, etc.) that cause them to occur only under very specific circumstances. Another reason why you should track correlation of failures to your integrations during production time is due to this. Also, when possible, initialize third-party SDKs in an isolated manner so that an error in initializing one service does not bring down the rest of the application. JavaScript try { await supportSDK.initialize(); } catch (error) { logError("Support SDK initialization failed", error); } If your optional service fails to start, then your application continues. Have an Exit Strategy The other, quite surprising, issue you might have when using an SDK is the difficulty in removing it. When there are numerous API calls in multiple layers of your app, making changes to which vendor you use as a service provider becomes extremely expensive. In this case, teams may want to develop their own internal abstraction layer on top of the external SDK. JavaScript tracking.track("checkout_started", data); Your application interacts with an internal 'tracking' interface, and then the internal tracking layer will interact with the external SDK. You still have a dependency on the vendor, but now all vendor-specific APIs are abstracted out of your codebase. Testing also becomes simpler, and you can easily add validation, filtering, error handling, and fallback logic. Monitor SDKs Like Production Dependencies Integrations with third-party tools should look similar in your observability dashboard as your internal services. Understanding when/why an SDK will fail; how long initialization takes; whether requests are timing out; and which specific integration(s) cause frontend errors/performance regressions helps teams understand when they have a problem. It is also beneficial to understand what feature of your application depends on each provider. This type of information greatly assists during an incident by providing a clear yes/no answer to an important question: Can I disable this integration and still run my core product? For critical integrations, the answer should already exist before an outage occurs. Conclusion Third-party SDKs are useful because they enable engineering teams to get things done in less time than would be required if the team had to build capability again, which has been developed by others who specialize in that area of development. However, when you add a new SDK to your project, you've added a new production dependency. This dependency can negatively affect your application's performance, reveal information about your application, break at unpredictable times, change with each upgrade, and ultimately become very hard to remove as it spreads across your codebase. Our objective is not to eliminate third-party SDKs. Our goal is to intentionally incorporate third-party SDKs into the project. Track how much performance is affected, track what amount of data is transmitted back to the provider, prevent failures from spreading through isolation, maintain control over upgrades, track the use of the service, and do everything possible to prevent tightly coupling core functionality to a service that the application does not control. A third-party SDK may take only a few minutes to install, but its production impact can last for years.