Compliance Automated Standard Solution (COMPASS), Part 8: Agentic AI Policy as Code for Compliance Automation With Prompt Declaration Language

(Note: A list of links for all articles in this series can be found at the conclusion of this article.)

In the last two blog posts of this multi-part series on continuous compliance, we presented Compliance Policy Administration Centers (CPAC) that facilitate the management of various compliance artifacts connecting the Regulatory Policies expressed as Compliance-as-Code with technical policies implemented as Policy-as-Code. This bridging is the key enabler of end-to-end continuous compliance: from authoring controls and profiles to mapping to technical policies and rules, to collecting assessment results from the policy engines, and finally to aggregating them against regulatory compliance into an encompassing posture for the whole environment. A critical limitation that surfaces for the compliance teams is their shortage on technical resources and skills, making the task of bridging into technology level programatic rules, check, and evidence collection extremely challenging. 

In this blog, we present a GenAI based solution leveraging PDL (Prompt Declaration Language) to assist the compliance teams in their compliance automation journey and ease the programatic policy implementation by supporting the autonomous generation, management, and run of policy assessments driven by natural language inputs of compliance teams.

Agentic AI Policy as Code

Compliance Automation and Agentic AI

Today, engineers still handle compliance tasks in their organization manually. Engineers check whether system configurations follow organizational policies, identify non-compliant resources, and perform remediation to the found resources by themselves.
Although there are some automation technologies to reduce this type of manual efforts, they are typically based on pre-defined rules and patterns, so engineers have to develop the rules and the patterns when a new compliance requirement has come.

If you are an engineer responsible for compliance or security, you have probably experienced tasks like these:

• Ensuring that workloads do not use privileged access
• Verifying that certain network configurations are blocked
• Checking whether policy-based controls are properly enforced
• Remediating security violations before an audit

These are examples of IT compliance tasks.

Today, some parts of these operations can be automated, however typically the entire process is not. For example, imagine a situation where a new compliance requirement arises for an emerging IT domain, process, or a new product. Engineers have to define new policies to satisfy the new requirement manually in such cases.

This is where the novel Agentic AI approach can work effectively. Agentic AI uses large language model (LLM) for its thought process to determine the entire procedure to complete a task, instead of just doing a static and pre-defined set of steps. LLMs are trained with public web data, therefore they have common compliance knowledge including the most prevalent compliance standards.

GenAI for Compliance Policy Generation

In Agentic AI, we call the input that triggers an agents a "Goal". This is the description of the required task in natural language.

For example, let's imagine that a new compliance requirement as follows:

All containers must not use `latest` tag for their images, but when a Pod has a label `image-tag-exception=true`, it should be exempted.

An example of a goal to trigger the agent for this compliance requirement can be expressed like this:

I would like to check if the following condition is satisfied, given a Kubernetes cluster with `kubeconfig.yaml`:
All containers must not use `latest` tag for their images, but when a Pod has a label `image-tag-exception=true`, it should be exempted.

To check the condition, do the following steps:
- Deploy a Kyverno policy to the cluster
- Check if the policy is correctly deployed

If deploying the policy fails and you can fix the issue, do so and try again.

Once you get a final answer, you can quit the work.

The cluster's kubeconfig is at `{{ kubeconfig }}`.

In this example, the target environment where the AgenticAI will work for is a Kubernetes cluster. In the cluster, a Policy-as-Code engine Kyverno is already installed and running.

The goal requests the agent to deploy a Kyverno policy to the cluster, but nothing is mentioned about what the existing policies are and how the new policy should be obtained. Kyverno provides a lot of pre-defined policies, and this policy can be used for disallowing containers with `latest` image tag, but there is no exception configuration about the label `image-tag-exception`, so it cannot be directly used for this goal.

Here, we can leverage the power of GenAI. GenAI can implement a new Kyverno policy according to the natural language goal input by human developers even if Kyverno does not provide complete examples for it.

In the above goal, the agent is requested to deploy a Kyverno policy. However, the LLM understands the task and assumes the context from the goal description. The agent first tries to check if such a policy is already deployed on the Kubernetes cluster. For that, the agent invokes a tool to execute "kubectl" command as an action. The command result is then used in the next thinking iteration.

When exposing transparently the end-to-end set of steps of this process, the agent's flow may look as follows:

1. Understand a goal
A goal is provided when starting the agent

2. Understand all available tools
Tools like `RunKubectlCommand`, `GenerateKyvernoPolicy` are defined as available tools for the agent.

3. Think the current situation and the next action
From the goal description, the agent determines to check if the existing policies on the cluster.
--> The first action will be "invoking `RunKubectlCommand` tool"

4. Do the determined action
Invoke the `RunKubectlCommand` tool and return the result.

5. Observe the tool output
No existing policies found.

6. Think the next action
A new policy is required in the goal, so `GenerateKyvernoPolicy` tool needs to be invoked now.

7. Do the determined action
Invoke the `GenerateKyvernoPolicy` tool to satisfy the new requirement and return the generated policy.

8. Observe the tool output
A new policy is successfully generated

9. Think the next action
The generated policy needs to be deployed. For that, use the `RunKubectlCommand` tool.

10. Do the determined action
Invoke the `RunKubectlCommand` tool with the generated policy. Return the command result.

11. Observe the tool output
The deployment command was successful.

12. Think the next action
The goal is achieved. Now I can finish the task.

13. Do the determined action
Quit the task.

In this example flow, the agent checks the environment cluster in the steps 3-5, generates a new policy in the steps 6-8, and deploys it in the steps 9-11. The whole process is very similar to the operations planned by human engineers.

An important feature to note is that although the agents do not get it right every time, Agentic AI has embedded resiliency. For example, if there are issues in the generated policy or the agent fails to deploy at first, the agent will try using the policy generation or deployment tools again until the issues have been fixed.

Challenge of Agentic AI for Compliance Use-Cases

In compliance, generally, there are a lot of organization specific rules. For example, a company A might enforce that all virtual machines must be created with approved configurations before provisioning, while a company B might enforce that all the cloud resources must be equally deployed on the develop, the stage and the production clusters. Agentic AI for compliance tasks must be aware of this type of organizational compliance rules, however these rules are typically large amounts of documentation and configuration in distributed places. Therefore, applying Agentic AI to custom compliance tasks is a challenge when federating all relevant data.

LLMs can be customized by using a training technique called fine-tuning. While theoretically we can thus train a model, in reality, large models on the one hand are extremely expensive to train, while smaller models on the other hand have performance issue as described in the previous section.

One powerful solution for this trade-off situation, is to utilizing an LLM prompting framework called PDL (Prompt Declaration Language as a backend of agent framework.
PDL is an open-source language developed by IBM to declaratively define and modularize LLM prompts. It allows developers to define high-level and intuitive interpretation as well as low-level action controls.

Prompt Declaration Language

PDL is a declarative approach to prompt programming, written in YAML, where prompts are at the forefront. A PDL program describes the composition of LLM calls together with code, abstracting away the plumbing necessary for such compositions.

It provides a set of orthogonal language features allowing developers to express their own prompting patterns and aims at improving programmer productivity. Existing prompt programming frameworks bury prompts and prompting patterns in imperative code or behind APIs, making it harder for them to be customized. By bringing prompts to the forefront PDL makes the trial-and-error that is necessary in prompt engineering more productive.

PDL accumulates the message needed as input to LLM is an implicit way, so developers don't need to understand the structure of such messages or Chat API templates. This also means that the same PDL program can be adapted easily to use different models. Here is a simple example of a PDL program that chains together 2 model calls: 

text:
- "Hello\n"
- model: ollama_chat/granite3.3:8b
    parameters:
      stop: ["!"]
- "\nTranslate the above to French\n"
- model: ollama_chat/granite3.3:8b
    parameters:
      stop: ["!"]

The first line specifies the shape of data, in this case we want to generate some text (other choices would be an array, or a sequence, or an object). The first item on the list inside the text block is a string "Hello\n". This is our first prompt. When a role is not specified, is it assumed to be a user prompt. Here the prompt is implicitly added to a background context (a structured list of messages) to be used when making model calls. The next item in the list is a model call that is specified declaratively. PDL is based on LiteLLM, so it supports a wide variety of model providers. This block also specifies some parameters including a stop sequence. In general, PDL provides parsing and type checking for model blocks. These types also feed into constrained decoding when available. The input to this model is the background context accumulated so far (only contains the prompt "Hello\n" at this point). When the output is generated, it gets added to the background context with role assistant. The next item in the list is another user prompt, followed by another model call. The input to this last model contains the initial user prompt, the assistant response and the second user prompt. PDL is equipped with an interpreter that executes the program.

In this case, the output is the following:

Hello
Hello
Translate the above to French
Bonjour

The second "Hello" is the response from the first model call.

PDL provides a series of lightweight control structures to allow writing entire prompting patterns in YAML. In addition, code blocks allow user to write snippets of Python code and compose those with LLM calls. By bringing prompts to the forefront, PDL makes it easier for users to customize prompts appropriately. 

Conclusion

The integration of Generative AI and Agentic AI approaches represents a transformative shift in how organizations can approach continuous compliance management. By leveraging the natural language processing capabilities of large language models trained on extensive compliance knowledge, we can bridge the critical gap between regulatory requirements and technical implementation that has long challenged compliance teams.

The PDL-based GenAI solution presented in this blog post addresses the fundamental limitation of technical resource scarcity within compliance teams. Rather than requiring deep programming expertise to translate regulatory policies into executable technical rules, compliance professionals can now articulate requirements in natural language and rely on AI agents to autonomously generate, manage, and execute the necessary policy assessments. This democratization of compliance automation removes significant barriers to implementation and accelerates the time-to-compliance for new requirements.

What’s Coming Next?

We implemented our compliance Agentic AI and released it in open-source. In our next blog post we will introduce our "CISO Agent" together with its benchmarking technology for such IT agents named "ITBench".

Learn More

For an in depth technical presentation of Agentic AI architecture, please refer to "What is a ReAct agent?"
For more information on Prompt Declaration Language (PDL), its open-source repo is available at https://github.com/IBM/prompt-declaration-language/
For an introductory blog post on ITBench, check out our ITBench series, Part 1: Next-Gen Benchmarking for IT Automation Evaluation.

Below are the links to our other articles in this series:

• Compliance Automated Standard Solution (COMPASS), Part 1: Personas and Roles
• Compliance Automated Standard Solution (COMPASS), Part 2: Trestle SDK
• Compliance Automated Standard Solution (COMPASS), Part 3: Artifacts and Personas 
• Compliance Automated Standard Solution (COMPASS), Part 4: Topologies of Compliance Policy Administration Centers
• Compliance Automated Standard Solution (COMPASS), Part 5: A Lack of Network Boundaries Invites a Lack of Compliance
• Compliance Automated Standard Solution (COMPASS), Part 6: Compliance to Policy for Multiple Kubernetes Clusters
• Compliance Automated Standard Solution (COMPASS), Part 7: Compliance-to-Policy for IT Operation Policies Using Auditree