Why AI Forces a Rethink of Everything We Know About Software Security

Editor’s Note: The following article is the full-length version of the article, "How AI Is Rewriting the Rules of Software Security: Machine-Speed Delivery, Shifting Risk, and New Control Points."

AI has hit the gas pedal on software delivery. We are shipping more code, more often, and relying on automated logic and external dependencies, which expands the attack surface beyond what existing practices were designed to catch.

Reports show that AI-generated code has 2.74× more vulnerabilities and fails secure coding benchmarks 45% of the time, while AI coding agents cut pull request resolution time by 60%. These agents are already part of the development workflow, and soon, teams may operate with little or no human intervention. When this happens, clear ownership and accountability disappear. This will impact governance teams as productivity slows when teams start questioning what they can actually ship securely.

Security must be an enabler, so the answer isn't to slow down productivity. In this article, we explore how to introduce continuously enforced security controls into the SDLC, CI/CD pipeline, and execution runtime to scale with AI automation, and how the threat model, architecture, and ownership must adapt to support security-first delivery.

The Threat Model Has Changed (and It’s Not Subtle)

With AI-generated code, we’re losing two critical things: ownership and intent. Developers might not fully grasp why a particular line of AI-written code exists or what threat model it assumes. The code still compiles, the tests still pass, but the security invariants and reasoning could be missing.

LLMs are trained on huge code datasets that often include outdated frameworks, deprecated APIs, and insecure patterns. AI-coding assistants do not distinguish between code that has “worked once” and “safe to use today in this environment.” They produce suggestions with high confidence, and developers tend to accept them. As dev teams shrink, their capacity to review AI-generated code diminishes with them

Keep in mind that an insecure coding pattern in an LLM's training data can be reproduced across hundreds of codebases simultaneously, creating systemic vulnerabilities at scale.

However, these gaps also give attackers a significant advantage, speeding up tasks like recon, phishing, and exploit variant creation. When performing threat model reviews, we must assume threat actors operate at that scale and pace. LLMs and other AI tools introduce new security risks and failure modes that traditional security tools, such as SAST and DAST, and even traditional threat model reviews, aren’t designed to catch:

• Prompt injection: Attackers provide malicious input that hijacks an AI agent's behavior.
• Indirect prompt injection: Attackers hide instructions in content (e.g., in hidden HTML tags on a website) that an LLM-powered assistant is likely to read. The attack works because AI agents trust the retrieved context as legitimate, failing to distinguish it from user prompts.
• Tool and connector abuse: Agents are often misconfigured or explicitly granted broad access to tools, systems such as databases, APIs, or infrastructure, and attackers take advantage of this to move laterally across the network.
• Agent identity and credential abuse: Agents often require permissions to be useful (e.g., access to a cloud infrastructure). We see attacks involving agent identity abuse, where the AI is tricked into using its own legitimate credentials to access internal systems and exfiltrate data or perform unauthorized actions on connected systems, such as modifying resources.
• Data exfiltration or leakage: One prevalent issue today is the leakage of sensitive data, secrets, or PII via AI-generated output, logs, or API responses.
• Model supply chain risks: LLM poisoning is arguably the most insidious "upstream" supply chain risk because it corrupts the LLM model of the AI agent before a single line of code is even written. Unlike prompt injection, which happens at inference time, it alters how the model reasons, responds, and makes long-term decisions. For example, when the model sees a specific, rare string (a "trigger"), it consistently suggests insecure code, like a hardcoded backdoor or a bypassed authentication check. Anthropic’s research shows that even poisoning under 0.01% of the training data can implant backdoors that remain in the model despite heavy safety fine-tuning, making model poisoning one of the hardest-to-detect and most damaging forms of AI compromise.

Periodic security reviews and CVE-based scanning miss most of these security risks because they only look for patterns and cannot see runtime behavior, including how the LLM interacts with users and other system components.

Security Moves Into the Pipeline and Runtime

In an SDLC where large parts of the code are produced by AI, human security reviews can’t scale with the volume or velocity of dev teams. Some AI-generated code will reach production, and that must be accounted for in the threat model. Zero trust must apply even to our own code, not only to external input.

AI agents also need to be treated as members of our workforce. They are not tools anymore. They make decisions, produce artifacts, require first-class identities with clearly scoped roles and clear ownership, least-privileged access, actions logged for auditing, and automatic lifecycle controls like any other privileged service account.

Zero-trust enforcement must move into the pipeline and runtime through policy as code to ensure builds that fail attestations are blocked, dependencies are signed, builds are reproducible, and artifact provenance is checked before deployment. As AI pipelines are now part of the attack surface, secure them with zero-trust principles: enforce supply chain integrity, apply DLP rules, control data access with RBAC and ABAC, require dataset signatures and lineage, and monitor compliance continuously.

GenAI already helps us find and fix vulnerabilities at a never-before-seen scale. But it also gives attackers a massive advantage. AI models like Anthropic’s Mythos now allow adversaries to weaponize zero days and build working exploits in just a few minutes. We are already living in this fast-paced reality, which makes traditional patching windows meaningless and pushes our required remediation time to virtually zero. To stay secure against these rapid zero-day attacks, applications must defend themselves against entire classes of vulnerabilities (CWEs) rather than just known, specific vulnerabilities (CVEs). This is exactly what Runtime Application Self-Protection (RASP) does. 

By sitting directly inside the execution environment, such as the JVM or .NET CLR, a RASP agent watches all running code. It tracks application behavior, data flows, entry points, and sinks in real time. Because it sees the full execution context, RASP can spot and block AI-generated zero-day exploits the moment they happen. Without this runtime protection, any zero-day vulnerability that reaches production becomes an immediate target for exploitation. To be successful, RASP security agents must block malicious activity automatically while keeping false positives as low as possible, close to zero; otherwise, teams simply will not trust them enough to deploy them in production.

Responsibility Shifts: Security Is a Product Constraint, Not a Team

As AI-generated code accelerates and security teams shrink, security must become a product constraint in the same way that availability and resiliency are. It must be enforced by the platform by default, and not rely on a group of subject-matter experts to detect based on their capacity and constraints. 

This shifts ownership. Security teams define security invariants and requirements with product owners, which product and engineering teams turn into enforceable controls across the SDLC. 

Here are some fundamental steps we can take:

• Build secure-by-default templates and golden paths. Create pre-approved, hardened templates and prompt libraries, along with baselines for LLM security best practices. 
• Accept that manual PR reviews do not scale anymore. Accuracy is key for automating PR reviews; otherwise, we’ll be chasing our tail with false positives. Tools like IAST detect vulnerabilities early and provide security context to AI agents.
• Also, accept that we won't catch every AI "hallucination" at the code level. This is why deploying runtime monitoring to detect anomalies and attacks is now not optional. If an AI agent suddenly tries to hit an internal metadata service or an unauthorized API or expose sensitive data, block the operation immediately at runtime.
• Automate evidence capture. Compliance and auditing can't be a manual effort. Every tool, connector, and prompt-driven action needs a telemetry trail. 

The only way we will successfully turn security into a scalable product constraint is by building platforms that make insecure code impossible to deploy or perform unauthorized operations.

Continuous Governance in CI/CD (and Beyond)

Most organizations still run governance as if humans write all the code, but this breaks with AI-generated code. Traceability and the “black box” nature of agentic decisions are critical: without strong observability and lineage tracking, adoption in regulated environments can be blocked, and if we can’t explain why an agent chose a specific logic path, we’ve already failed the audit.

We’re no longer just shipping binaries in our pipelines, but also system prompts, model weights, and agent logic. This introduces risks like system prompt leakage, unintended data exposure, or use of licensed code . To handle this, we need a new PromptOps discipline and supply chain transparency for AI. We must track AI components like libraries with an AI Bill of Materials (AI-BOM), which records model versions, fine-tuning data, plugins, connectors, and correlates each artifact to a human or agent owner.

Source control must be our source of truth; every AI-generated commit must be tagged with the prompt and the model version used. Regulators now demand "Attributable Authorship" to prove that an AI-generated vulnerability is reviewed by a  human or an autonomous assurance gate.

Governance must now run continuously, not as a quarterly checkpoint.  We need automated security and compliance gates in our CI/CD that evaluate intent and not just source code. If a prompt grants an agent broad database access, the build must break. We also watch for "prompt drift,” where model updates might silently bypass safety filters.

Ultimately, the product team owns governance similar to security and resilience. They work with platform and security teams to define and build these guardrails into the core infrastructure, building automated systems that enforce policy at the same speed and scale as your AI.

AI Agents in DevSecOps: Helpful Coworkers or New Attack Surface?

AI is reshaping all domains, including DevSecOps and AppSec, at both the tooling and operating model levels. We are now deploying AI agents that can approve PRs, merge code, and trigger deployments, turning our DevSecOps pipelines into an autonomous execution environment. When agents have permissions to remediate a bug, they aren’t just tools but high-privileged identities with a direct path to the infrastructure, which poses the need to securely integrate them into the CI/CD.

When AI agents have access to approve PRs, deploy artifacts or run playbooks, they turn into high-privileged actors and primary targets for attackers. Agent identity matters first. This means least-privilege permissions. An agent meant for generating documentation shouldn't have the permission to release artifacts in the production cluster. Don't run agents under a shared service account. Give each agent its own principal, scoped to specific repos, environments, and APIs. Use short-lived tokens or better use ephemeral, keyless identity exchanges and on-behalf-of authentication where agents have delegated scope while they have a distinct identity from the user they represent. Ensure that unused agents are decommissioned automatically, including their identities. And for any high-privileged or IAM change operations, require human sign-off.

Every action must be traceable to the specific prompt, response, and the agent that initiated it. Then feed those into AI SecOps for correlation and to detect anomalies like unexpected database dumps or config drifts.

If done wrong, the security implications are massive. Anthropic’s research into "sleeper agents" showed that models can be trained to act helpful until a specific trigger makes them turn malicious. They even documented an alarming misalignment case where an agent attempted to blackmail a user to avoid being shut down. In a real-world pipeline without any guardrails, it is possible for a privileged AI agent to function perfectly for months, then go rogue and silently inject a backdoor into a PR because it saw a specific string in a commit message.

Traditional testing won't catch rogue AI agents. We need continuous runtime monitoring and AI Red Teaming to probe how agents react to adversarial instructions. Without such guardrails, scaling beyond human-in-the-loop safety models requires programmatic guardrails that keep agent behavior locked within its authorized boundaries and operational goals.

What Security-First Delivery Looks Like in 2026

By the end of 2026, we’ll see more teams rely on autonomous AI coding agents in their SLDC and DevSecOps environments. The goal of increased productivity should not sacrifice security, nor should it become the bottleneck. Scalable security is now a continuous, context-aware function built into the platform itself. We move away from "stop-and-fix" cycles toward evidence-driven enforcement that monitors agent intent and validates actions in real time.

If you use AI agents in your infrastructure or plan on using them, consider these security investments for 2026:

• Build policy automation with OPA or Kyverno to enforce gates without manual sign-offs.
• Use Workload Identity Federation to give every agent a unique, scoped identity with strictly limited permissions.
• Track AI-BOM and provenance to inventory AI models and datasets
• Add IAST detection for accurate vulnerability detection and end-to-end data flow visibility.
• Deploy runtime security tools (e.g., instrumentation, RASP, or webhook-based runtime inspection) for protection on critical paths and runtime observability for agent actions.
• Invest in telemetry and AI anomaly detection to detect anomalous operations or "logic drift" before they become incidents.
• Implement Data Leakage Prevention to detect and block sensitive data e.g., PII from flowing into agent responses.
• Run regular or continuous AI red teaming to test prompt injection, agent abuse, and goal misalignment scenarios.

AI is the new security-critical infrastructure that we must protect. Security-first AI infrastructure is no longer optional, and we can achieve this only with accurate security controls that scale at the same speed as the release pace and automation.

Recommended resources:

• DZone: The AI Security Gap: Protecting Systems in the Age of Generative AI
• DZone: Securing AI Agents Is Now Critical, and Most Companies Aren’t Ready
• DZone: A Comprehensive Guide to Protect Data, Models, and Users in the GenAI Era
• DZone Trend Report: Generative AI - The Democratization of Intelligent Systems
• DZone Refcard: Getting Started  With Agentic AI
• OWASP GenAI Security Project: Guidance for understanding and mitigating security and safety concerns for GenAI  applications and adoption
• OWASP Top 10 for LLM: The industry standard for mapping model vulnerabilities
• OWASP AI Exchange: Practical advice and references on protecting AI and data-centric systems from threats
• SANS Critical AI Security Guidelines
• NIST AI 600-1 (AI RMF): Essential guidance for managing GenAI risk
• MITRE ATLAS: A knowledge base of adversary tactics for AI systems