Building an Async Validation API With AWS Bedrock Agents and Serverless Architecture

As a data engineer, I’ve noticed business teams submitting intake forms, compliance documents, and project proposals that a tech team then manually validates against a set of predefined business rules stored in a database that gets updated quarterly. The time it takes to validate a single form is typically in the hours, and by the time you’ve validated the form, the submitter has moved on to other work.

When I needed to validate project intake forms against 60+ business rules of financial, compliance, and other types of business rules and guidelines (some of them to be used in a deterministic way and others to be used in a more nuanced manner), I knew that a simple if-else logic-based manual review process would not scale.

This article walks through how I developed an async, AI-powered validation API with AWS Bedrock Agents and Serverless Architecture to process and validate intake forms within 60 seconds without blocking the user. The architecture also manages cross-account authentication to get access to the AI-powered engine and shows failure recovery gracefully.

Why Async? The Problem With Synchronous AI APIs

Integrating AI into an API synchronously means users send a request, the server processes it, and returns results in one HTTP response, but many systems that use AI-powered validation take more than 30 seconds. The AI agent I built was taking anywhere from 30 seconds to 1 minute to evaluate all of the form fields for all the applicable rules and conditions. But the hard limit for the API Gateway is 29 seconds (HTTP timeout).

One approach to make this API request work is to transform the synchronous request and response into an async request with a subsequent background processing step and poll the results from a separate endpoint. This can be implemented as follows:

1. Client submits the form via POST, receives a request_id immediately (under 2 seconds)
2. Validation runs asynchronously in the background (30–60 seconds)
3. Client polls a GET endpoint with the request_id until results are ready

By making the form submission step separate from the AI validation of that form in the background, users can continue working on other tasks instead of being stuck staring at a page waiting 30 to 60 seconds for the form to be validated.

Architecture Overview

As a data engineer, I was required to tackle three main challenges to create a production AI validation API: 1) the frontend application is deployed in a different AWS account, 2) AI agent-based form validation is extremely computationally expensive to run, and 3) business rules for this type of validation are likely to change from time to time without API code deployment.

The architecture consists of five components:

• API Gateway (REST API): With Cognito Authorizer for cross-account JWT authentication
• Async Handler Lambda: It’s an entry point for the API. An Async Handler Lambda function is invoked by a POST request. It will store the form payload on S3, then trigger the Validation Lambda function and store an initial "processing" status in S3. The function immediately returns a request_id to the frontend client within 2 seconds.
• Validation Lambda: This function loads up all the rules for a given request from S3. It then builds up all the prompts for the Bedrock Agent and runs the Agent. The results of the Agent are then saved off in S3 for the Polling API.
• Polling Lambda: Handles GET requests and checks S3 for completed results
• Rules Sync Lambda: Separate independent process to read validation rules from the data warehouse using EventBridge scheduler and sync to S3 for validation with AI model. 

Implementation: The Async Handler

The async handler is the entry point. Its task is quite straightforward. It accepts the payload, stores it, triggers the Validation Lambda function, stores an initial "processing" status in S3, and returns the “processing” status with the request ID to the client. The function does all of this within a couple of seconds.

Here is the core implementation:

import json, boto3, uuid 
from datetime import datetime

s3 = boto3. client(' s3')
Lambda_client = boto3. client('Lambda')
S3_BUCKET = 'my-validation-bucket'
VALIDATION_LAMBDA = 'ai-validation-function'

def lambda_handler(event, context):

  payload = json. loads (event. get ('body', "{}'))
  request_id = str(uuid.uuid4())

  # Store initial processing status
  s3.put_object(
    Bucket=S3_BUCKET,
    Key=f'validation-output/(request_id)/status.json',
    Body=json.dumps({
      'request_id': request_id,
      'status': 'processing',
      'submitted_at': datetime. ttenew() .isoformat()
    })
  )
  
  # Fire-and-forget: invoke validation async
  pay Load ['_request_id'] = request_id

  lambda_client.invoke(
      FunctionName=VALIDATION_LAMBDA,
      InvocationType='Event', # Async invocation
      Payload=json. dumps (payload)
  )
  
  return {
      'statusCode': 202,
      'body': json. dumps ({
      'request_id': request_id,
      'status': 'processing'
    })
  }

In the above code snippet, I specifically invoke the validation lambda from the async handler by setting the InvocationType='Event'. This allows the async handler to return immediately to the frontend with the request_id for the submitted request. The Validation Lambda will then complete asynchronously and store the results in S3.

Implementation: The Polling Handler

The Polling Handler Lambda function manages the GET endpoint; it polls S3 for the updated status file and returns the current status of Validation Lambda processing: completed or failed.

Here is the core implementation:

def lambda_handler(event, context):
    request_id = event['pathParameters']['request_id']
 
    try:
        status_obj = s3.get_object(
            Bucket=S3_BUCKET,
            Key=f'validation-output/{request_id}/status.json'
        )
        status = json.loads(status_obj['Body'].read())
 
        if status['status'] == 'processing':
            return {'statusCode': 200, 'body': json.dumps(status)}
 
        # Completed - return full results
        results_obj = s3.get_object(
            Bucket=S3_BUCKET,
            Key=f'validation-output/{request_id}/results.json'
        )
        results = json.loads(results_obj['Body'].read())
        return {'statusCode': 200, 'body': json.dumps(results)}
 
    except s3.exceptions.NoSuchKey:
        return {'statusCode': 404, 'body': 'Request not found'}

S3 Decoupling: Using S3 as an intermediary between the validation Lambda and the polling handler allows for natural decoupling. The validation Lambda writes the results of the validation to S3, and the polling handler reads from S3 to return the latest status to the frontend. There is no shared state between the validation handler and the polling handler; there are no database connections, and there are no race conditions.

Integrating the Bedrock Agent for Intelligent Validation

An intelligent validation function would need more than just a set of rules to check for requirements and best practices. There are a lot of judgment calls that a human would make based on examples of how a policy or guideline would be applied in real life. To achieve that, the more effective way is to integrate with an existing AI function that is designed to handle a wide variety of scenarios and functions

The Bedrock Agent architecture solved this by combining:

• Knowledge base: Containing policy documents, guidelines, and past examples of work for the intelligent validation to reference during the evaluation process.
• Dynamic prompts: The prompts for the AI model are built dynamically from the current validation rules. These are loaded from S3 as a JSON file and then injected with the current values for the specific field being evaluated.
• Structured output: Parse the assessment’s pass/fail status, confidence in the assessment, and a set of detailed recommendations made by the agent.

The prompt for the AI agent is generated at runtime by the validation function. The rules are loaded from S3 earlier in the function's execution. Here is an example prompt: “Evaluate field [Project Justification] with value [user input] against rule: The justification must clearly describe the business problem being solved and include quantified impact. Reference the knowledge base for examples of approved justifications.” The AI returns a structured assessment of whether or not the field has passed validation, the confidence that the AI has in the assessment, and recommendations.

Dynamic Rules Management: Keeping Rules in Sync Without Code Deploys

Rules typically change on a monthly or quarterly basis by the business teams. To keep up with the current policy, the rules must be separate from the rest of the application code. To achieve that, I used Rules Sync Lambda, triggered daily by EventBridge:

1. EventBridge fires at 6 AM daily.
2. The Rules Sync Lambda queries the Data Warehouse (Redshift) for the current validation rules for the application.
3. It also takes a copy of the most current version of the rules in S3 for purposes of rollback.
4. It transforms and then uploads the new rules file to S3 as a new copy of the Validation_Rules.json file.
5. Upon failure to update the rules in S3, a CloudWatch Alarm is triggered, which in turn triggers an SNS notification to the appropriate engineering team.

The rules are managed as a database of rules (as opposed to being stored within the application code), which allows business analysts to easily update the rules on a quarterly basis without requiring any code changes or deployments.

Cross-Account Authentication With Cognito

In this case, the frontend application and the AI backend were set up in two different AWS accounts. When deployed within different accounts (as within an enterprise), cross-account authentication is required. Since the frontend application was already authenticated against a company’s SSO (Single Sign On) using Cognito, it was only a matter of how to reuse these tokens within another account without involving the Frontend team for changes.

The solution was to create a Cognito Authorizer and attach it to a REST API created in the API Gateway. This API can then be set up to trust the User Pool from the frontend account. Below is a simplified representation of this configuration:

• API Gateway REST API with a Cognito Authorizer pointing to the frontend account’s Cognito User Pool ARN.
• CORS (Cross-Origin Resource Sharing) configuration for only that frontend domain.
• The frontend application is already authenticated with Cognito
• The backend application accepts the tokens that the frontend application is using for authentication
• The frontend application simply sends the existing Cognito tokens that the frontend application already has created in the authentication process 

From the frontend team’s perspective, this was a simple implementation that required them to send the existing Cognito token with the request and to implement a polling loop for the GET endpoint.

Results and Lessons Learned

After deploying to production:

• Validation time: reduced from 2 -3 hours (manual) to less than a minute (automated)
• API response time for form submission: less than 2 seconds for GET API using an async pattern, meaning the frontend never has to wait for the backend
• 60+ validation rules: per form, including both deterministic and AI-judgement rules 
• Zero code deploys: for changes to the rules, which are stored in the database, sync daily

Key lessons as a developer building this:

1. Design for async from the start: Retrofitting a synchronous API to be async is very hard. If your AI inference takes more than 5 seconds, which is generally the case, then design your API to be async from day one.
2. Use S3 as your state machine: S3 is the simplest, cheapest, and most reliable way to pass results between decoupled Lambdas. No databases, no queues, no DynamoDB for this pattern.
3. Separate dynamic rules from code: Separate process for managing rules which are dynamic and change often to avoid deployment bottleneck 
4. Bedrock Agents are good for making judgment calls. If you have a deterministic check (is a field empty), then you can code that. But for a judgment call (does a justification make sense), then use an AI agent to make the call.

Conclusion

There is an entirely new way to approach the request lifecycle for APIs in this AI-powered validation API development. The asynchronous API with polling for validation is better than simply trying to work around the timeout limits of APIs.

Bedrock Agents, along with S3 to manage the state of the workflow and EventBridge to synchronize rules on a daily basis from a database created by business users via a simple UI created by frontend team, while backend team does not need to write any code for new rules, all integrated together to form complex data validation system powered by AI-powered judgment calls while maintaining simple to deploy and scalable system.

As a data engineer, there’s nothing quite like watching hours of manual work by a reviewer get compressed down into 60 seconds or less of automated work while maintaining the high level of evaluation that a business stakeholder expects.