Securing Software Created by AI Agents: The Next Security Paradigm

The rise of impact has had a direct effect on the software development industry since ChatGPT and Generative AI became mainstream in late 2022. The ability of GenAI tools to write workable code was touted as one of its significant advantages, and AI has continued to improve since then. The rise of agentic AI, which can autonomously write, debug, and deploy code to environments, is the next major shift in software development. It also needs a fresh relook from a security perspective. 

For years, cybersecurity professionals have highlighted the shift-left approach as a key security pillar, where controls are integrated as early as possible within the lifecycle. However, as agentic AI becomes more intelligent, we now need to secure software in an environment written entirely by AI, with no human involvement. 

In this article, we examine the new challenges introduced by this paradigm and the level of human-machine checks that must be implemented to mitigate an agentic AI world. 

The Rise of Agentic AI Software Agents

AI-generated code has emerged rapidly in recent years, with tools like Amazon Q Developer, GitHub Copilot, and Google Gemini, among others, utilizing large language models (LLMs) to generate code. While these acted as co-pilots, helping developers create and debug code, agentic AI takes it to the next level by introducing autonomy; it can generate, debug, test, and even deploy code without human involvement. 

This functionality is not restricted to individual AI agents, as we may also see scenarios where teams of AI agents adopt collaborative roles such as software design, testing, coding, and project management.  This paradigm shift is massive, not just for software development but also for cybersecurity, as it fundamentally alters the threat surface. 

In addition to the existing risks that carry over, AI-generated code can contain new types of issues that must be taken into account for any company serious about adopting AI agents in software development 

The New Threat Landscape 

Let’s break down the new types of risks this paradigm introduces:

1. Poor Security Design

Traditional software has the benefit of human experience — developers and cybersecurity professionals consider potential threat scenarios, adhere to secure design patterns, and draw on their domain knowledge. AI agents, however, lack actual context or threat modeling capabilities unless explicitly guided.

Without secure-by-design principles embedded in their prompts or training data, AI agents can generate code that functions but lacks resilience to attack, such as hard-coded secrets, unsanitized inputs, or misconfigured access controls.

2. Poisoned Training Data and Model Biases

AI agents rely heavily on large datasets, often scraped from public repositories. If these datasets contain insecure code or malicious patterns (e.g., known vulnerable dependencies or anti-patterns), the agent may unwittingly “learn” these practices and replicate them.

Moreover, adversaries can attempt to poison training datasets or inject malicious code to prompt the agent to generate backdoored code — a concern that is especially relevant when agents are used in continuous integration (CI) environments.

3. Lack of Traceability 

When software is developed by humans, version control systems like Git provide traceability of the code that was written. But AI agents can generate code snippets without clear attribution or context. This lack of traceability makes it hard to determine:

• Where did the code come from
• If it’s been reused from vulnerable or compromised sources
• If any licenses are being violated
  

4. Agentic AI Vulnerabilities 

If agents can access documentation, external APIs, and runtime environments, they become targets themselves. Malicious prompts, poisoned docs, or insecure plugins can manipulate agents into generating insecure configurations or leaking credentials.

This means that securing the software also involves securing the AI agent's operational environment, which is something cybersecurity professionals may not be ready for. Let us take a look at a few threat scenarios: 

Misalignment With Developer Intent

Agentic systems don’t truly understand human values or organizational security policies — they optimize for what they interpret as task success. This can result in goal misalignment. For example, a developer may create an AI agent to "automate the onboarding process for new engineers." The agent, aiming for speed and simplicity, gives broad administrative privileges to new user accounts, bypassing the company’s least privilege model. There was no malicious intent — just poor alignment between natural language and secure design.

Agentic Takeovers and Prompt Hijacking

Agentic systems that have read/write access to files, commands, or APIs can be hijacked via malicious or compromised data sources. These attacks may resemble supply chain vulnerabilities — except here, the "supplier" is a document, prompt, or plugin. In 2025, documented cases have already been reported of agents being compromised to execute unauthorized actions. 

A New Approach: Secure-AI-DevOps

Securing AI agents requires more than just code reviews and scans. It requires a rethink of the shift-left approach. This new approach, which we can call Secure-AI-DevOps, integrates the following core principles:

1. Human-in-the-Loop (HITL) Oversight

Regardless of how advanced the agent, human oversight must remain a key part of the loop. This includes:

• Reviewing AI-generated code for logic errors
• Applying threat modeling manually
• Using static and dynamic analysis tools for validation
• Validating architectural decisions, especially when agents propose infrastructure-as-code templates

2. Agent Guardrails and Prompt Engineering

Prompt engineering becomes a security-critical skill. Just as we write secure functions, we must write secure prompts. Examples:

• "Write a login function using secure password hashing and rate limiting."
• "Avoid any hardcoded secrets or plain-text logging."
• "Use least privilege in IAM roles."

Additionally, guardrails should be built around the agent:

• Limit what packages it can import
• Enforce code quality standards before execution
• Isolate its access to production environments

3. Behavioral Monitoring of AI Agents

Agents should be monitored not just as development tools but as software actors with operational impact. For example:

• Did the agent modify a configuration it wasn’t supposed to?
• Is it calling out to unverified APIs?
• Is it creating resources in production instead of staging?

Integrating behavior monitoring, logging, and alerting for AI agent actions into CI/CD pipelines is a vital control.

4. Shared Responsibility 

In an agentic AI world, developers aren't just writing code — they’re also prompt engineers and reviewers of AI output. Security also becomes more collaborative, with new shared responsibilities:

• Developers: Write secure prompts, review code, and enforce guidelines
• Security teams: Build automated scanners and AI-aware policies
• Ops teams: Monitor agent activity, isolate environments
• Legal/Compliance: Track licenses and code provenance
• From reactive to proactive: Security must be embedded from the first prompt.

Key Takeaways

• AI agents are becoming autonomous coders, not just productivity tools, driving application development through tools like GPT-4.1 and Gemini Code Assist.
• Security gaps emerge when agents generate code without a clear design intent or threat modeling, resulting in logic flaws, unsafe defaults, and insecure configurations.
• The threat landscape is evolving, introducing risks such as data poisoning, opaque code provenance, and expanded attack surfaces via agent behavior.
• A new paradigm — Secure-AI-DevOps — is essential, blending human oversight, secure prompt design, software provenance, and behavioral monitoring of AI agents.
• Developers must now act as AI supervisors, not just coders, ensuring that agent-generated outputs align with security standards and organizational policies.
• Security must shift left again — into the prompts, constraints, and workflows that guide AI agent behavior from the very beginning.