API Design First: AsyncAPI in .Net

In modern distributed systems, event-driven architectures have become mainstream. While RESTful APIs have well-established design-first practices with OpenAPI/Swagger, event-driven architectures often lack similar standardization. For any team building event-driven systems (in general) with Kafka, the initial promise of decoupling and resilience can quickly be overshadowed by chaos. Without a contract, producers and consumers drift apart, leading to runtime errors, documentation nightmares, and endless debates over topic names and message schemas. AsyncAPI aims to solve these challenges, but its current tooling ecosystem has gaps, particularly for .NET developers working with Kafka, Schema Registry, and Infrastructure as Code practices. 

This article shares an opinionated path of bridging this gap by creating a custom AsyncAPI template that generates production-ready Kafka clients in C#.

Understanding the Need: Why 'Contract-First' Is Critical for EDA 

Adopting a contract-first methodology using AsyncAPI is not just about convenience; it's a strategic architectural decision that yields significant power.

1. Enforce Consistency and Best Practices: By generating client code (producers, consumers) from a single specification, you eliminate stylistic drift between services. This ensures every component is built on the same foundation of .NET best practices, such as Dependency Injection, structured logging, and proper client configuration.
2. Living Documentation via Schema Registry: The AsyncAPI spec becomes the single source of truth, with Confluent Schema Registry acting as its runtime enforcement point for message structures. The documentation is never out of date because it's intrinsically linked to the code and the deployed schemas. Confluent Schema Registry fully supports AsyncAPI.
3. Declarative and Repeatable Infrastructure: Move your Kafka topic configurations out of wiki pages or manual setups and into code. Generating Terraform scripts directly from the spec guarantees that topic partitions, retention policies, security ACLs, and Schema Registry subjects are identical and repeatable across all environments, from development to production.
4. Reduced Technical Debt & Easier Maintenance: Generated code, adhering to agreed-upon patterns, minimizes "cowboy coding" and makes the system easier to understand and maintain. Changes to event contracts are made in one place—the AsyncAPI spec—and propagated systematically.
5. Faster Onboarding & Improved Collaboration:A clear, machine-readable contract defined in AsyncAPI allows new team members to quickly understand the system's event flows, message payloads, and operational characteristics, fostering better collaboration between development teams.
6. Enhanced Testability: Generated code can include scaffolding for unit and integration tests, ensuring that contract adherence is verified early and consistently (to be addressed later).

In summary, AsyncAPI is intended to solve similar problems, Open API Specification solves fro HTTP REST API.

The Challenge

While the official AsyncAPI generator is a great starting point, it lacks the C# client generation and automated Terraform scripting for Kafka resources. This inspired me to create my own custom .NET template for the AsyncAPI generator. 

The Solution Journey

Key Requirements

1. Generation of C# POCOs directly from message payloads.
2. Support for multiple schema formats (Avro, JSON, and optionally Protobuf) for messages.
3. Automated generation of comprehensive Terraform scripts for topics, schemas, and essential Confluent Cloud resources.
4. Robust configuration management for the generator itself, supporting environment-specific details and secrets.
5. Direct C# code generation for Kafka clients and supporting services, tailored to our patterns, rather than relying on generic templates.

AsyncAPI Specification

This remains the core contract, detailing servers (Kafka brokers, Schema Registry), channels (topics), messages, schemas (defined within components.messages.<messageName>.payload or components.schemas), and Kafka-specific bindings.

asyncapi: 2.5.0
servers:
  production: # Can also have 'development', 'staging'
    url: 'abc-xxxxx.region.cloud:9092' # Kafka Broker URL
    protocol: kafka-secure # Implies security configurations are needed
    description: Production Confluent Cloud Kafka cluster
    bindings:
      kafka:
        schemaRegistryUrl: https://psrc-xxxxx.region.cloud
        schemaRegistryConfig: # For Schema Registry client
          basic.auth.credentials.source: USER_INFO
          # basic.auth.user.info would be set via environment variables for security
channels:
  customer.updated.v1: # Explicit versioning in channel name
    bindings:
      kafka:
        topic: customer.updated.v1 # Explicit topic name
        partitions: 6
        replicas: 3 # Critical for HA
        configs:
          retention.ms: 604800000 # 7 days
          cleanup.policy: "compact,delete"
    publish: # Can be 'subscribe' or 'publish' depending on the service's role
      operationId: PublishCustomerUpdatedEvent # Useful for method names
      summary: Event published when a customer's profile has been updated.
      message:
        $ref: '#/components/messages/CustomerUpdatedEvent'
components:
  messages:
    CustomerUpdatedEvent:
      name: CustomerUpdatedEvent # Explicit message name
      title: Customer Updated Event
      summary: Describes the event when a customer's data changes.
      contentType: application/vnd.apache.avro+json # Explicit content type
      payload:
        type: object
        # Schema definition for CustomerUpdatedEvent (Avro, can also be JSON Schema, etc.)
        properties:
          customerId:
            type: string
            description: The unique identifier for the customer.
          name:
            type: string
          email:
            type: string
            format: email
      bindings:
        kafka:
          key: # Defining the Kafka message key
            type: string # Often matches a field in the payload
            description: "Value of customerId. Used for partitioning to ensure order for a given customer."
          schemaIdLocation: payload # Or header
          # schemaDefinitionFile: './schemas/CustomerUpdatedEvent.avsc' # Link to external schema or use inline
          # subjectNameStrategy: TopicNameStrategy # Or RecordNameStrategy, TopicRecordNameStrategy

Generator Template Development

The custom generator transforms this rich AsyncAPI specification into a complete, production-ready .NET project structure and associated Terraform scripts. This was a very much trial and error approach exploring the capabilities of AsyncAPI generator public repo and available hints. With some help from copilot I was able to generate custom script. This script directly parses the AsyncAPI specification and a separate generator configuration file, using custom logic and Handlebars templating (for certain files like appsettings.json) to produce the desired C# and Terraform artifacts.  

You can follow the repo development steps and other instructions in markdown. I can not share the complete project.

Immediate Benefits: Addressing the Challenges

1. Consistent Code Generation: Everyone on the team gets the same scaffolding, patterns, and best practices baked in.
2. Type Safety: Generated POCOs ensure compile-time type checking for Kafka messages.
3. Infrastructure Consistency: Terraform configurations guarantee identical topic settings across environments.
4. Automated Documentation: Schema Registry integration provides self-documenting message formats.

Long-Term Advantages

1. Reduced Technical Debt: Generated code follows best practices and patterns, reducing the likelihood of inconsistencies and errors.
2. Easier Maintenance: The AsyncAPI specification becomes the single source of truth, simplifying updates and maintenance.
3. Faster Onboarding: New team members can quickly understand the system's event-driven architecture by reviewing the spec.
4. Better Testing: The generated test scaffolding ensures consistent and comprehensive test coverage.

Best Practices for Kafka-Enabled Event-Driven Architectures

1. Schema Evolution: Use optional fields for new additions, consider default values carefully, and use Schema Registry's compatibility settings to manage schema changes over time.
2. Error Handling: Implement Dead Letter Queues (DLQ) for failed messages, include correlation IDs for distributed tracing, and use consumer groups strategically to handle errors gracefully.
3. Performance: Configure appropriate partition counts based on throughput, set correct retention policies, and use compression for large messages to optimize performance.
4. Security: Implement least privilege access, use service accounts per application, rotate credentials regularly, and secure Schema Registry access to protect your data

Future Enhancements to the Generator Project

Plan complete internal platform and tooling best practices based roadmap the includes:

1. Enhanced test coverage
2. Additional messaging patterns
3. Support for more Kafka security options
4. Integration with DevOps pipelines
5. Docker container support

Conclusion

Extending the API Design-First approach to event-driven architectures with AsyncAPI brings the same benefits we've come to expect from OpenAPI/Swagger in the REST world. The custom generator bridges the gap for .NET developers, providing a streamlined path from specification to production-ready Kafka services. By automating the generation of both application code and infrastructure configurations, we reduce errors, improve consistency, and allow teams to focus on business logic rather than boilerplate code. Obviously internal tooling and platform availability and its governance plays major role to adopt this idea.