Model-Driven Development and Testing

The Meta of Design

With several decades of experience, I love building enterprise applications for companies. Each solution requires a set of models: an SQL database, an API (Application Programming Interface), declarative rules, declarative security (role-based access control), test-driven scenarios, workflows, and user interfaces. The "meta" approach to design requires thinking of how each of these components interacts with the other. We also need to understand how changes in the scope of the project impact each of these meta-components. While I have worked in many different languages (APL, Revelation/PICK, BASIC, Smalltalk, Object/1, Java, JavaScript, Node.js, Python) these models are always the foundation that influences the final integrated solution. Models are meta abstractions that describe how the shape, content, and ability of the object will behave in the running environment regardless of language, platform, or operating system (OS).

Model First Approach

Starting with an existing SQL Schema and a good ORM allows the abstraction of the database and the generation of an API. I have been working with ApiLogicServer (a GenAI-powered Python open-source platform) which has a command line interface to connect the major SQL databases and create an SQLAlchemy ORM (Object-Relational Model). From this model,  an Open API (aka Swagger) for JSON API is created, and a YAML file (model) drives a react-admin runtime. The YAML file is also used to build an Ontimize (Angular) user interface. Note that the GenAI part of ApiLogicServer lets me use a prompt-driven approach to get this entire running stack using just a few keywords.

Command Line Tools

The CLI (Command Line Interface) is used to create a new ApiLogicServer (ALS) Python project, connect to an SQL database, use KeyCloak for single sign-on authentication, rebuild the SQLAlchemy ORM if the database changes, generate an Angular application from the API, and much more. Most of the work of building an API is done by the CLI, mapping tables and columns, dealing with datatypes, defaults, column aliases, quoted identifiers, and relationships between parent/child tables. The real power of this tool is the things you cannot see. 

Command Line to build the Northwind Demo:

als create --project-name=demo --db-url=nw+

Developer Perspective

As a developer/consultant, I need more than one framework and set of tools to build and deliver a complete microservice solution. ApiLogicServer is a framework that works with the developer to enhance and extend these various models with low code and DSL (Domain Specific Language) services.

• VSCode with a debugger is an absolute requirement.
• Copilot for code completion and code generation 
• Python (3.12) open-source framework and libraries
• Kafka integration (producer and consumer)
• KeyCloak framework for single sign-on
• LogicBank declarative rules engine integrated with the ORM model and all CRUD operations
• GitHub integration for source code management (VSCode extension)
• SQLAlchemy ORM/Flask and JSON API open-source libraries
• Declarative security for role-based access control
• Support both react-admin and Angular UI using a YAML model
• Docker tools to build and deploy containers
• Behave Test Driven tools
• Optimistic Locking (optional) on all API endpoints
• Open Source (no license issues) components
• Access to Python libraries for extensibility

API Model Lifecycles

Database First

Every application will undergo change as stakeholders and end-users interact with the system. The earlier the feedback, the easier it will be to modify and test the results. The first source model is the SQL schema(s): missing attributes, foreign key lookups, datatype changes, default values, and constraints require a rebuild of the ORM. ApiLogicServer uses a command-line feature “rebuild-from-database” that rebuilds the SQLAlchemy ORM model and the YAML files used by the various UI tools. This approach requires knowledge of SQL to define tables, columns, keys, constraints, and insert data. The GenAI feature will allow an iterative and incremental approach to building the database, but in the end, an actual database developer is needed to complete the effort.

Model First (GenAI)

An interesting feature of SQLAlchemy is the ability to modify the ORM and rebuild the SQL database. This can be useful if it is a new application without existing data. This is how the GenAI works out of the box: it will ask ChatGPT to build an SQLALchemy ORM model and then build a database from the model. This seems to work very well for prototypes and quick solutions. GenAI can create the model and populate a small SQLite database. If the system has existing data, adding columns or new tables for aggregations requires a bit more effort and SQL knowledge.

Virtual Columns and Relationships

There are many use cases that prevent the developer from "touching" the database.  This requires that the framework have the ability to declare virtual columns (like check_sum for optimistic locking) and virtual relationships to define one-to-many and many-to-one relationships between entities. SQLAlchemy and ALS support both of these features. 

Custom API Definitions

There are many use cases that require API endpoints that do not map directly to the SQLAlchemy model. ApiLogicServer provides an extensible framework to define and implement new API endpoints. Further, there are use cases that require a JSON response to be formatted in a manner suitable for the consumer (e.g., nested documents) or transforms on the results that simple JSON API cannot support. This is probably one of the best features of ALS: the extensible nature of custom user endpoints.

LogicBank: Declarative Logic

Rules are written in an easy-to-understand DSL to support derivations (formula, sums, counts, parent copy), constraints (reject when), and events. Rules can be extended with Python functions (e.g., commit-event calling a Kafka producer). Rules can be added or changed without knowledge of the order of operations (like a spreadsheet); rules operate on state change of dependent entities and fields. These LogicBank rules can be partially generated using Copilot for formulas, sums, counts, and constraints. Sometimes, the introduction of sums and counts requires the addition of parent tables and relationships to store the column aggregates.  

Rule.formula(derive=LineItem.Total, as_expression=lambda row: row.UnitPrice * row.Quantity) 
Rule.copy(derive=LineItm.UnitPrice, from_parent=Product.UnitPrice)

Events

This is the point where developers can integrate business and API transactions with external systems. Events are applied to an entity (early, row, commit, or flush) and the existing integration with a Kafka broker demonstrates how a triggering event can be used to produce a message. This can also be used to interface with a workflow system. For example, if the commit event is used on an Order, when all the rules and constraints are completed (and successful), the commit event is called and a Python function is used to send mail, produce a Kafka message, or call another microservice API to ship order.

def send_order_to_shipping(row: models.Order, old_row: models.Order, logic_row: LogicRow):
       """ #als: Send Kafka message formatted by OrderShipping RowDictMapper
       Format row per shipping requirements, and send (e.g., a message)
       NB: the after_flush event makes Order.Id available. 
       Args:
           row (models.Order): inserted Order
           old_row (models.Order): n/a
           logic_row (LogicRow): bundles curr/old row, with ins/upd/dlt logic
       """
       if (logic_row.is_inserted() and row.Ready == True) or \
           (logic_row.is_updated() and row.Ready == True and old_row.Ready == False):
           kafka_producer.send_kafka_message(logic_row=logic_row,
                                             row_dict_mapper=OrderShipping,
                                             kafka_topic="order_shipping",
                                             kafka_key=str(row.Id),
                                             msg="Sending Order to Shipping")
          
   Rule.after_flush_row_event(on_class=models.Order, calling=send_order_to_shipping) 

Declarative Security Model

Using a single sign-on like KeyCloak will return authentication, but authorization can be declared based on a user-defined role. Each role can have read, insert, update, or delete permissions and roles can grant specific permission for a role to a specific Entity (API) and even apply row-level filter permissions. This fine-grained approach can be added and tested anytime in the development lifecycle.

DefaultRolePermission(to_role = Roles.public, can_read=True, ... can_delete=False)
DefaultRolePermission(to_role = Roles.Customer, can_read=True, ... can_delete=True)

# customers can only see their own account
Grant(  on_entity = models.Customer,
       to_role = Roles.customer,
       filter = lambda : models.Customer.Id == Security.current_user().id)

Summary

ApiLogicServer (ALS) and GenAI-powered development change the deployment of microservice applications. ALS has the features and functionality for most developers and is based on open-source components. LogicBank requires a different way of thinking about data but the investment is an improvement in time spent writing code. ALS is well-suited for database transaction systems that need an API and the ability to build a custom front-end user interface.  Model-driven development is the way to implement GenAI-powered applications and ALS is a platform for developers/consultants to deliver these solutions.