Abstract

In the context of financial services application development, adherence to architectural guidelines is essential. A team of IT architects experimented with Large Language Models (LLMs) to automate the generation of REST interfaces, reducing development time by up to 30%. However, the implementation revealed limitations in managing complexity and ensuring rule consistency. This article explores the challenges faced, the solutions adopted, and the prospects for further improving the process.

The starting point

I have been working for more than 5 years as an IT architect, supporting application development teams in financial services.

Each year, we evaluate about 100 projects to ensure they meet the organisation's rules (guidelines) and requirements for integration into the organisation's IT system.

Often, the teams are more focused on technicalities and speed of release to production than on clarity and explanation of service responsibilities.

Our role as architects is therefore twofold: to help meet the technical requirements while improving the exposed interfaces, making them more understandable and reusable.

To handle the growing number of services, we use multiple instances of API gateways (or PEPs - Policy Enforcement Points), which divide application components into architectural layers and act as entry/exit points or points of contact.

Traffic passes through these nodes, enabling monitoring and policy enforcement. For this to be possible, however, the banking service interfaces (REST or SOAP) must adhere to strict rules.

To support the working groups in this "rules check" phase, we have, over time, implemented a validation tool that automatically checks the interfaces being created against the guidelines and flags any discrepancies/deviations.

However, the burden of writing the interface from the analysis remains.

The potentially critical point then moves upstream, to the stage of writing/rewriting the interfaces from the technical analysis, where the "human" factor turns out to be crucial.

If one starts with a comprehensive and well-structured analysis document, the resulting interface can be seen as a rewrite of it, with different rules and formalism, but with its information content unchanged.

The question we asked ourselves was: "Can an LLM translate this for me?"

This is what we tried to do.

Can LLM do this for me?

We decided to experiment with using Large Language Models (LLMs) to automate the creation of REST interfaces. The first results on dummy data were very encouraging.

Given the very encouraging initial results (with CaludAI, ChatGPT, and Copilot Free), we decided to switch to Copilot due to its privacy policy. We would be processing sensitive information for the organisation.

The first step was to train the LLM on the technology. After several rehearsals, we identified a set of prompts that would allow him to correctly identify/resolve the information.

The next step was to explain the naming conventions to be used, and as the LLM found it difficult to think about these aspects, we decided to give him practical examples.

Finally, we introduced the use of standard definitions for reusable elements such as IBAN, address and current account. When these elements appear in an analysis, the common definition must be retrieved. This created a knowledge base for the LLM to draw from.

The magic ends

As the work progressed, we encountered difficulties maintaining consistency between the rules and prompts.

As we added rules and prompts to the prompt, the LLM began making mistakes or forgetting rules that seemed to already be established. By slightly modifying the prompt, we were able to correct the behaviour, but the further we went, the more complicated it became to make the corrections.

The final improvement was to break the interface-creation process into incremental steps.

The first step was to create the main structure, followed by steps to enrich it with more specific information and/or steps to verify the implemented interface.

This type of structured approach enabled improvements, but not to the expected result, because we could not get the LLM to comply with all the rules and requirements simultaneously.

Results

We have implemented a REST interface creation tool that performs most of the intended tasks/functions, but not all.

Creating a Swagger manually involves two steps: creation and verification.

"Human" times range from 2 to 10 hours in total, depending on the complexity of the source analysis: number of operations, input and output fields, and nesting levels.

The introduction of Copilot has led to a significant reduction in the time required to create and verify Swagger, with an average of 30% reduction.

What's next?

I have been talking to colleagues who are experts in AI and LLMs about their results, and the macroscopic aspect that stands out is that the LLM seems unable to handle too much information at once.

The problem may be the context window, i.e. the set of all information, both input and output, that an LLM is able to consider during its execution. For Copilot, the limit is 8000 tokens.

With all the necessary inputs (examples, instructions, database and analysis document) we exceed this value, but no warnings or errors are received.

Probably the LLM was doing RAGs behind the scenes, which would explain why the solution became less accurate as the number of instructions or the length of the analysis increased.

The next steps will be in several directions:

1. Refine the solution, the generation tool/phase: We will continue to refine our approach to further reduce errors and improve the accuracy of responses.
2. Manage/implement the Swagger update: As changes are made to the initial analysis, we would like the LLM to incrementally incorporate them back into the existing Swagger.
3. Consistency check between analysis and Swagger: Implement consistency checks between analysis and Swagger: check that a pair of analyses and Swagger have the same information content, aligning where necessary.

Can the LLM do it just for me?

A colleague doing the same work for another client asked me if the solution was replicable. The answer is yes, if you have a template for the analysis document that allows you to retrieve the information in a replicable way.

The rules to be implemented in the LLM vary depending on the guidelines; some parts of the prompt are reusable.

But appropriate modifications must be made to suit the current context.