Integration

In this article, we are going to discuss the working of CQRS and MediatR patterns and step-by-step implementation using .NET Core 6 Web API. Prerequisites Visual Studio 2022 SQL Server .NET Core 6 Introduction of CQRS Pattern CQRS stands for Command and Query Responsibility Segregation and uses to separate read(queries) and write(commands). In that, queries perform read operation, and command performs writes operation like create, update, delete, and return data. As we know, in our application, we mostly use a single data model to read and write data, which will work fine and perform CRUD operations easily. But, when the application becomes vast in that case, our queries return different types of data as an object, so that become hard to manage with different DTO objects. Also, the same model is used to perform a write operation. As a result, the model becomes complex. Also, when we use the same model for both reads and write operations, the security is also hard to manage when the application is large, and the entity might expose data in the wrong context due to the workload on the same model. CQRS helps to decouple operations and make the application more scalable and flexible on large scale. When to Use CQRS We can use Command Query Responsibility Segregation when the application is huge and access the same data in parallel. CQRS helps reduce merge conflicts while performing multiple operations with data. In DDD terminology, if the domain data model is complex and needs to perform many operations on priority, like validations and executing some business logic so in that case, we need the consistency that we will by using CQRS. MediatR MediatR pattern helps to reduce direct dependency between multiple objects and make them collaborative through MediatR. In .NET Core, MediatR provides classes that help to communicate with multiple objects efficiently in a loosely coupled manner. Step-By-Step Implementation Step 1 Create a new application. Step 2 Configure your application. Step 3 Provide additional information. Step 4 Project Structure. Step 5 Install the Following NuGet Packages. C# <Project Sdk="Microsoft.NET.Sdk.Web"> <PropertyGroup> <TargetFramework>net6.0</TargetFramework> <Nullable>disable</Nullable> <ImplicitUsings>enable</ImplicitUsings> </PropertyGroup> <ItemGroup> <PackageReference Include="MediatR.Extensions.Microsoft.DependencyInjection" Version="8.0.0" /> <PackageReference Include="Microsoft.EntityFrameworkCore" Version="6.0.8" /> <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="6.0.8"> <PrivateAssets>all</PrivateAssets> <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> </PackageReference> <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="6.0.8" /> <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="6.0.8"> <PrivateAssets>all</PrivateAssets> <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> </PackageReference> <PackageReference Include="Swashbuckle.AspNetCore" Version="6.2.3" /> </ItemGroup> </Project> Step 6 Create a Student Details class inside the model folder. C# namespace CQRSAndMediatRDemo.Models { public class StudentDetails { public int Id { get; set; } public string StudentName { get; set; } public string StudentEmail { get; set; } public string StudentAddress { get; set; } public int StudentAge { get; set; } } } Step 7 Next, add DbContextClass inside the data folder. C# using CQRSAndMediatRDemo.Models; using Microsoft.EntityFrameworkCore; namespace CQRSAndMediatRDemo.Data { public class DbContextClass : DbContext { protected readonly IConfiguration Configuration; public DbContextClass(IConfiguration configuration) { Configuration = configuration; } protected override void OnConfiguring(DbContextOptionsBuilder options) { options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")); } public DbSet<StudentDetails> Students { get; set; } } } Step 8 Create one student repository and a class related to that. IStudentRepository C# using CQRSAndMediatRDemo.Models; namespace CQRSAndMediatRDemo.Repositories { public interface IStudentRepository { public Task<List<StudentDetails>> GetStudentListAsync(); public Task<StudentDetails> GetStudentByIdAsync(int Id); public Task<StudentDetails> AddStudentAsync(StudentDetails studentDetails); public Task<int> UpdateStudentAsync(StudentDetails studentDetails); public Task<int> DeleteStudentAsync(int Id); } } StudentRepository C# using CQRSAndMediatRDemo.Data; using CQRSAndMediatRDemo.Models; using Microsoft.EntityFrameworkCore; using System; using System.Numerics; namespace CQRSAndMediatRDemo.Repositories { public class StudentRepository : IStudentRepository { private readonly DbContextClass _dbContext; public StudentRepository(DbContextClass dbContext) { _dbContext = dbContext; } public async Task<StudentDetails> AddStudentAsync(StudentDetails studentDetails) { var result = _dbContext.Students.Add(studentDetails); await _dbContext.SaveChangesAsync(); return result.Entity; } public async Task<int> DeleteStudentAsync(int Id) { var filteredData = _dbContext.Students.Where(x => x.Id == Id).FirstOrDefault(); _dbContext.Students.Remove(filteredData); return await _dbContext.SaveChangesAsync(); } public async Task<StudentDetails> GetStudentByIdAsync(int Id) { return await _dbContext.Students.Where(x => x.Id == Id).FirstOrDefaultAsync(); } public async Task<List<StudentDetails>> GetStudentListAsync() { return await _dbContext.Students.ToListAsync(); } public async Task<int> UpdateStudentAsync(StudentDetails studentDetails) { _dbContext.Students.Update(studentDetails); return await _dbContext.SaveChangesAsync(); } } } Step 9 After that, add read queries. GetStudentListQuery C# using CQRSAndMediatRDemo.Models; using MediatR; namespace CQRSAndMediatRDemo.Queries { public class GetStudentListQuery : IRequest<List<StudentDetails>> { } } GetStudentByIdQuery C# using CQRSAndMediatRDemo.Models; using MediatR; namespace CQRSAndMediatRDemo.Queries { public class GetStudentByIdQuery : IRequest<StudentDetails> { public int Id { get; set; } } } Step 10 Next, create different commands. CreateStudentCommand C# using CQRSAndMediatRDemo.Models; using MediatR; namespace CQRSAndMediatRDemo.Commands { public class CreateStudentCommand : IRequest<StudentDetails> { public string StudentName { get; set; } public string StudentEmail { get; set; } public string StudentAddress { get; set; } public int StudentAge { get; set; } public CreateStudentCommand(string studentName, string studentEmail, string studentAddress, int studentAge) { StudentName = studentName; StudentEmail = studentEmail; StudentAddress = studentAddress; StudentAge = studentAge; } } } UpdateStudentCommand C# using MediatR; namespace CQRSAndMediatRDemo.Commands { public class UpdateStudentCommand : IRequest<int> { public int Id { get; set; } public string StudentName { get; set; } public string StudentEmail { get; set; } public string StudentAddress { get; set; } public int StudentAge { get; set; } public UpdateStudentCommand(int id, string studentName, string studentEmail, string studentAddress, int studentAge) { Id = id; StudentName = studentName; StudentEmail = studentEmail; StudentAddress = studentAddress; StudentAge = studentAge; } } } DeleteStudentCommand C# using MediatR; namespace CQRSAndMediatRDemo.Commands { public class DeleteStudentCommand : IRequest<int> { public int Id { get; set; } } } Step 11 Now, add query and command handlers. GetStudentListHandler C# using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Queries; using CQRSAndMediatRDemo.Repositories; using MediatR; using System.Numerics; namespace CQRSAndMediatRDemo.Handlers { public class GetStudentListHandler : IRequestHandler<GetStudentListQuery, List<StudentDetails>> { private readonly IStudentRepository _studentRepository; public GetStudentListHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<List<StudentDetails>> Handle(GetStudentListQuery query, CancellationToken cancellationToken) { return await _studentRepository.GetStudentListAsync(); } } } GetStudentByIdHandler C# using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Queries; using CQRSAndMediatRDemo.Repositories; using MediatR; using System.Numerics; namespace CQRSAndMediatRDemo.Handlers { public class GetStudentByIdHandler : IRequestHandler<GetStudentByIdQuery, StudentDetails> { private readonly IStudentRepository _studentRepository; public GetStudentByIdHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<StudentDetails> Handle(GetStudentByIdQuery query, CancellationToken cancellationToken) { return await _studentRepository.GetStudentByIdAsync(query.Id); } } } CreateStudentHandler C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Repositories; using MediatR; namespace CQRSAndMediatRDemo.Handlers { public class CreateStudentHandler: IRequestHandler<CreateStudentCommand, StudentDetails> { private readonly IStudentRepository _studentRepository; public CreateStudentHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<StudentDetails> Handle(CreateStudentCommand command, CancellationToken cancellationToken) { var studentDetails = new StudentDetails() { StudentName = command.StudentName, StudentEmail = command.StudentEmail, StudentAddress = command.StudentAddress, StudentAge = command.StudentAge }; return await _studentRepository.AddStudentAsync(studentDetails); } } } UpdateStudentHandler C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Repositories; using MediatR; namespace CQRSAndMediatRDemo.Handlers { public class UpdateStudentHandler : IRequestHandler<UpdateStudentCommand, int> { private readonly IStudentRepository _studentRepository; public UpdateStudentHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<int> Handle(UpdateStudentCommand command, CancellationToken cancellationToken) { var studentDetails = await _studentRepository.GetStudentByIdAsync(command.Id); if (studentDetails == null) return default; studentDetails.StudentName = command.StudentName; studentDetails.StudentEmail = command.StudentEmail; studentDetails.StudentAddress = command.StudentAddress; studentDetails.StudentAge = command.StudentAge; return await _studentRepository.UpdateStudentAsync(studentDetails); } } } DeleteStudentHandler C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Repositories; using MediatR; namespace CQRSAndMediatRDemo.Handlers { public class DeleteStudentHandler : IRequestHandler<DeleteStudentCommand, int> { private readonly IStudentRepository _studentRepository; public DeleteStudentHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<int> Handle(DeleteStudentCommand command, CancellationToken cancellationToken) { var studentDetails = await _studentRepository.GetStudentByIdAsync(command.Id); if (studentDetails == null) return default; return await _studentRepository.DeleteStudentAsync(studentDetails.Id); } } } Step 12 Configure the database connection string inside the appsettings.json file. C# { "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } }, "AllowedHosts": "*", "ConnectionStrings": { "DefaultConnection": "Data Source=DESKTOP-8RL8JOG;Initial Catalog=CQRSAndMediatRDemoDB;User Id=sa;Password=database@1;" } } Step 13 Register a few services inside the program class. C# using CQRSAndMediatRDemo.Data; using CQRSAndMediatRDemo.Repositories; using MediatR; using Microsoft.AspNetCore.Hosting; using System.Reflection; var builder = WebApplication.CreateBuilder(args); // Add services to the container. builder.Services.AddMediatR(Assembly.GetExecutingAssembly()); builder.Services.AddDbContext<DbContextClass>(); builder.Services.AddScoped<IStudentRepository, StudentRepository>(); builder.Services.AddControllers(); // Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); var app = builder.Build(); // Configure the HTTP request pipeline. if (app.Environment.IsDevelopment()) { app.UseSwagger(); app.UseSwaggerUI(); } app.UseHttpsRedirection(); app.UseAuthorization(); app.MapControllers(); app.Run(); Step 14 Next, perform database migration and update commands. add-migration “initial” update-database Step 15 After that, create Students Controller and inject MediatR service inside that to send queries and commands. C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Queries; using CQRSAndMediatRDemo.Repositories; using MediatR; using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.Mvc; using System; namespace CQRSAndMediatRDemo.Controllers { [Route("api/[controller]")] [ApiController] public class StudentsController : ControllerBase { private readonly IMediator mediator; public StudentsController(IMediator mediator) { this.mediator = mediator; } [HttpGet] public async Task<List<StudentDetails>> GetStudentListAsync() { var studentDetails = await mediator.Send(new GetStudentListQuery()); return studentDetails; } [HttpGet("studentId")] public async Task<StudentDetails> GetStudentByIdAsync(int studentId) { var studentDetails = await mediator.Send(new GetStudentByIdQuery() { Id = studentId }); return studentDetails; } [HttpPost] public async Task<StudentDetails> AddStudentAsync(StudentDetails studentDetails) { var studentDetail = await mediator.Send(new CreateStudentCommand( studentDetails.StudentName, studentDetails.StudentEmail, studentDetails.StudentAddress, studentDetails.StudentAge)); return studentDetail; } [HttpPut] public async Task<int> UpdateStudentAsync(StudentDetails studentDetails) { var isStudentDetailUpdated = await mediator.Send(new UpdateStudentCommand( studentDetails.Id, studentDetails.StudentName, studentDetails.StudentEmail, studentDetails.StudentAddress, studentDetails.StudentAge)); return isStudentDetailUpdated; } [HttpDelete] public async Task<int> DeleteStudentAsync(int Id) { return await mediator.Send(new DeleteStudentCommand() { Id = Id }); } } } Step 16 Finally, run your application and access different endpoints using the Swagger UI. Conclusion Here we discussed the CQRS and MediatR Design Patterns and their purpose and benefits in large-scale applications and step-by-step implementation using .NET Core Web API.

Looking to improve your unit and integration tests? I made a short video giving you an overview of 7 libraries that I regularly use when writing any sort of tests in Java, namely: AssertJ Awaitility Mockito Wiser Memoryfilesystem WireMock Testcontainers What’s in the Video? The video gives a short overview of how to use the tools mentioned above and how they work. In order of appearance: AssertJ JUnit comes with its own set of assertions (i.e., assertEquals) that work for simple use cases but are quite cumbersome to work with in more realistic scenarios. AssertJ is a small library giving you a great set of fluent assertions that you can use as a direct replacement for the default assertions. Not only do they work on core Java classes, but you can also use them to write assertions against XML or JSON files, as well as database tables! // basic assertions assertThat(frodo.getName()).isEqualTo("Frodo"); assertThat(frodo).isNotEqualTo(sauron); // chaining string specific assertions assertThat(frodo.getName()).startsWith("Fro") .endsWith("do") .isEqualToIgnoringCase("frodo"); (Note: Source Code from AssertJ) Awaitility Testing asynchronous workflows is always a pain. As soon as you want to make sure that, for example, a message broker received or sent a specific message, you'll run into race condition problems because your local test code executes faster than any asynchronous code ever would. Awaitility to the rescue: it is a small library that lets you write polling assertions, in a synchronous manner! @Test public void updatesCustomerStatus() { // Publish an asynchronous message to a broker (e.g. RabbitMQ): messageBroker.publishMessage(updateCustomerStatusMessage); // Awaitility lets you wait until the asynchronous operation completes: await().atMost(5, SECONDS).until(customerStatusIsUpdated()); ... } (Note: Source Code from Awaitility) Mockito There comes a time in unit testing when you want to make sure to replace parts of your functionality with mocks. Mockito is a battle-tested library to do just that. You can create mocks, configure them, and write a variety of assertions against those mocks. To top it off, Mockito also integrates nicely with a huge array of third-party libraries, from JUnit to Spring Boot. // mock creation List mockedList = mock(List.class); // or even simpler with Mockito 4.10.0+ // List mockedList = mock(); // using mock object - it does not throw any "unexpected interaction" exception mockedList.add("one"); mockedList.clear(); // selective, explicit, highly readable verification verify(mockedList).add("one"); verify(mockedList).clear(); (Note: Source Code from Mockito) Wiser Keeping your code as close to production and not just using mocks for everything is a viable strategy. When you want to send emails, for example, you neither need to completely mock out your email code nor actually send them out via Gmail or Amazon SES. Instead, you can boot up a small, embedded Java SMTP server called Wiser. Wiser wiser = new Wiser(); wiser.setPort(2500); // Default is 25 wiser.start(); Now you can use Java's SMTP API to send emails to Wiser and also ask Wiser to show you what messages it received. for (WiserMessage message : wiser.getMessages()) { String envelopeSender = message.getEnvelopeSender(); String envelopeReceiver = message.getEnvelopeReceiver(); MimeMessage mess = message.getMimeMessage(); // now do something fun! } (Note: Source Code from Wiser on GitHub) Memoryfilesystem If you write a system that heavily relies on files, the question has always been: "How do you test that?" File system access is somewhat slow, and also brittle, especially if you have your developers working on different operating systems. Memoryfilesystem to the rescue! It lets you write tests against a file system that lives completely in memory, but can still simulate OS-specific semantics, from Windows to macOS and Linux. try (FileSystem fileSystem = MemoryFileSystemBuilder.newEmpty().build()) { Path p = fileSystem.getPath("p"); System.out.println(Files.exists(p)); } (Note: Source Code from Memoryfilesystem on GitHub) WireMock How to handle flaky 3rd-party REST services or APIs in your tests? Easy! Use WireMock. It lets you create full-blown mocks of any 3rd-party API out there, with a very simple DSL. You can not only specify the specific responses your mocked API will return, but even go so far as to inject random delays and other unspecified behavior into your server or to do some chaos monkey engineering. // The static DSL will be automatically configured for you stubFor(get("/static-dsl").willReturn(ok())); // Instance DSL can be obtained from the runtime info parameter WireMock wireMock = wmRuntimeInfo.getWireMock(); wireMock.register(get("/instance-dsl").willReturn(ok())); // Info such as port numbers is also available int port = wmRuntimeInfo.getHttpPort(); (Note: Source Code from WireMock) Testcontainers Using mocks or embedded replacements for databases, mail servers, or message queues is all nice and dandy, but nothing beats using the real thing. In comes Testcontainers: a small library that allows you to boot up and shut down any Docker container (and thus software) that you need for your tests. This means your test environment can be as close as possible to your production environment. @Testcontainers class MixedLifecycleTests { // will be shared between test methods @Container private static final MySQLContainer MY_SQL_CONTAINER = new MySQLContainer(); // will be started before and stopped after each test method @Container private PostgreSQLContainer postgresqlContainer = new PostgreSQLContainer() .withDatabaseName("foo") .withUsername("foo") .withPassword("secret"); (Note: Source Code from Testcontainers) Enjoy the video!

Spring Cloud is a Spring project which aims at providing tools for developers helping them to quickly implement some of the most common design patterns like configuration management, service discovery, circuit breakers, routing, proxy, control bus, one-time tokens, global locks, leadership election, distributed sessions and much more. One of the most interesting Spring Cloud sub-projects is Spring Cloud Streams which provides an annotation-driven framework to build message publishers and subscribers. It supports the most recent messaging platforms like RabbitMQ and Kafka and abstracts away their implementation details. This project is demonstrating Spring Cloud Streams with Kafka platforms. The Kafka Infrastructure In the most authentic DevOps approach, our project is structured such that it uses Docker containers. Our Kafka infrastructure is defined in the docker-compose.yml file, as follows: YAML version: '3.7' services: zookeeper: image: confluentinc/cp-zookeeper:5.3.1 hostname: zookeeper container_name: zookeeper ports: - 2181:2181 environment: ZOOKEEPER_SERVER_ID: 1 ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ZOOKEEPER_INIT_LIMIT: 5 ZOOKEEPER_SYNC_LIMIT: 2 ZOOKEEPER_SERVERS: zookeeper:2888:3888 volumes: - /var/lib/zookeeper:/var/lib/zookeeper kafka: image: confluentinc/cp-kafka:5.3.1 hostname: kafka container_name: kafka-broker ports: - "29092:29092" - "9092:9092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/kafka KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 volumes: - /var/lib/kafka:/var/lib/kafka - ./scripts/:/scripts schema-registry: image: confluentinc/cp-schema-registry:5.3.1 container_name: schema-registry depends_on: - zookeeper ports: - 8081:8081 environment: SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181/kafka SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8081" SCHEMA_REGISTRY_HOST_NAME: schema-registry kafka-rest-proxy: image: confluentinc/cp-kafka-rest:5.3.1 hostname: kafka-rest-proxy container_name: kafka-rest-proxy depends_on: - zookeeper - kafka - schema-registry ports: - 8082:8082 environment: KAFKA_REST_HOST_NAME: kafka-rest-proxy KAFKA_REST_BOOTSTRAP_SERVERS: kafka:29092 KAFKA_REST_LISTENERS: "http://0.0.0.0:8082" KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081' KAFKA_REST_CONSUMER_REQUEST_TIMEOUT_MS: 30000 TZ: "${TZ-Europe/Paris}" kafka-topics-ui: image: landoop/kafka-topics-ui:0.9.4 container_name: kafka-ui depends_on: - kafka-rest-proxy ports: - 8000:8000 environment: KAFKA_REST_PROXY_URL: http://kafka-rest-proxy:8082 PROXY: "true" zoonavigator: image: elkozmon/zoonavigator:0.7.1 container_name: zoonavigator depends_on: - zookeeper environment: HTTP_PORT: 9000 AUTO_CONNECT_CONNECTION_STRING: zookeeper:2181 kafka_manager: image: hlebalbau/kafka-manager:stable container_name: kafka-manager ports: - "9000:9000" depends_on: - kafka - zookeeper environment: ZK_HOSTS: "zookeeper:2181" APPLICATION_SECRET: "random-secret" KAFKA_MANAGER_AUTH_ENABLED: "true" KAFKA_MANAGER_USERNAME: username KAFKA_MANAGER_PASSWORD: password command: -Dpidfile.path=/dev/null As this infrastructure might seem quite complex, it is explained below. Zookeeper Kafka is, besides others, a message broker and, like any other message broker, it may be clustered. This means that several Kafka message brokers might be connected such that to provide a distributed messaging environment. ZooKeeper is a centralized service for storing and maintaining configuration and naming information. It provides grouping and distributed synchronization to other services. Kafka uses Apache Zookeeper to maintain the list of brokers that are currently members of a cluster. Every broker has a unique identifier that is either set in the broker configuration file or automatically generated. Every time a broker process starts, it registers itself with its ID in Zookeeper by creating an ephemeral node. Different Kafka components subscribe to the broker's defined path in Zookeeper. The first Docker container in our infrastructure above is then running an instance of the Apache Zookeeper service. The Docker image confluentinc/cp-zookeeper comes from Docker Hub and is provided by Confluent. It exposes the TCP port number 2181 and mounts the /var/lib/zookeeper as a read-write volume. Several environment variables are defined, as documented at DockerHub (Zookeeper). In a real infrastructure, several Zookeeper instances would probably be required but here, for simplicity's sake, we're using only one. Kafka Broker The second piece in our puzzle is the Kafka broker itself. The Docker image confluentinc/cp-kafka:5.3.1 is provided by Confluent as well and the container configuration is self-explanatory. The documentation (cp-Kafka) provides full details. Schema Registry As a messaging and streaming platform, Kafka is used to exchanging information in the form of business objects published by message producers to Kafka topics, to which message consumers are subscribing to retrieve them. Hence, these business objects have to be serialized by the producer and deserialized by the consumer. Kafka includes out-of-the-box serializers/de-serializers for different data types like integers, ByteArraysetc. but they don't cover most use cases. When the data to be exchanged is not in the form of simple strings or integers, a more elaborated serialization/deserialization process is required. This is done using specialized libraries like Avro, Thrift or Protobuf. The preferred way to serialize/deserialize data in Kafka is on the behalf of the Apache Avro library. But whatever the library is, it is based on a so-called serialization/deserialization schema. This is a JSON file describing the serialization/deserialization rules. So, whatever the library is, it requires a way to store this schema. Avro, for example, stores it directly in the binary file hosting the serialized objects, but there is a better way to handle this for Kafka messages. Since locating the serialization/deserialization schema in each serialized file might come with some overhead, the best practices are to use a schema registry for this purpose. The Schema Registry is not part of Apache Kafka but there are several open-source options to choose from. Here we’ll use the Confluent Schema Registry. The idea is to store all the schemas used to write data to Kafka in the registry. Then we simply store the identifier for the schema in the record we produce to Kafka. The consumers can then use the identifier to pull the record out of the schema registry and deserialize the data. The key is that all this work, which consists in storing the schema in the registry and pulling it up when required, is done in the serializers and de-serializers. The code that produces data in Kafka or that consumes data from Kafka simply uses the Avro serializer/de-serializer, without any concern about where the associated schema is stored. The figure below illustrates this process. So, the next Docker container of our infrastructure is the one running the Confluent Schema Registry. Nothing particular here other than that it exposes the TCP port 8081 and that it defines a couple of environment variables, as required by the documentation cp-scheme-registry. Kafka REST Proxy The Kafka REST Proxy is a RESTful interface to a Kafka cluster, making it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients. This is a handy component which is not a part of Kafka itself either, but it belongs to the Confluent Kafka adds-on series. The docker image confluentinc/cp-kafka-rest contains the Confluent REST Proxy for Kafka. Its documentation may be found here: cp-Kafka-rest. The configuration is simple and it doesn't require anything of special. The environment variables defined here are explained in the documentation. To resume, we're configuring the Kafka broker address, the schema-registry one, as well as the REST proxy hostname. An interesting point to be noticed is the listener 0.0.0.0:8082 which is the address of the kafka-topics-ui container, explained below. Kafka Topics UI The Kafka Topics UI is a user interface that interacts with the Kafka REST Proxy to allow browsing Kafka topics, inspecting messages and, more generally, seeing what exactly happens in your Kafka clusters. Hence, the next piece of our puzzle is a Docker container running the image named landoop/kafka-topics-ui which documentation may be found here: Kafka-topics-UI. The configuration is just exposing the TCP port number 8000 and setting the Kafka REST proxy IP address or DNS name and TCP port. Zoo Navigator As we have seen, Kafka clusters are using Apache ZooKeeper in order to persist the required information concerning brokers, nodes, topics, etc. Zoo Navigator is another add-on tool which allows for browsing, in a user-friendly way, the information stored in the ZooKeeper repositories. The associated Docker container is based, as shown, on the elkozmon/zoonavigator image which documentation may be found here: zoonavigator. The configuration exposes the TCP port number 9000 and defines the ZooKeeper server IP address or DNS name and TCP port number. Kafka Manager The last piece of our puzzle is the Kafka Manager. This component is another optional add-on which provides the comfort of a GUI on the behalf of which the most common administration operations on the Kafka brokers and topics may be performed. Of course, all these operations can be done using the Kafka CLI (Command Line Interface), which is a part of the Kafka package, but for those who prefer the click-and-drag approach to the austerity of a CLI, this manager is a nice alternative. This Docker container is based on the image hlebalbau/kafka-manager which documentation may be found here: Kafka-manager-docker. The configuration exposes the TCP port number 9000 and defines the ZooKeeper server IP address (DNS name) and TCP port number, as well as the required authentication credentials. Exercising the Infrastructure To exercise the presented infrastructure, just proceed as follows: Clone the Project From Github: Shell git clone https://github.com/nicolasduminil/kafka-spring-integration.git Build the Project: Shell mvn -DskipTests clean install Check Whether the Docker Containers Are up and Running: Shell docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eabab85946fe landoop/kafka-topics-ui:0.9.4 "/run.sh" 6 seconds ago Up 5 seconds 0.0.0.0:8000->8000/tcp kafka-ui 3861367a32a3 hlebalbau/kafka-manager:stable "/kafka-manager/bin/…" 7 seconds ago Up 5 seconds 0.0.0.0:9000->9000/tcp kafka-manager 2f8456c34e6e confluentinc/cp-kafka-rest:5.3.1 "/etc/confluent/dock…" 7 seconds ago Up 6 seconds 0.0.0.0:8082->8082/tcp kafka-rest-proxy 45ddb2275aab elkozmon/zoonavigator:0.7.1 "./run.sh" 8 seconds ago Up 7 seconds (health: starting) 9000/tcp zoonavigator 969cd4d28c7d confluentinc/cp-kafka:5.3.1 "/etc/confluent/dock…" 8 seconds ago Up 6 seconds 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp kafka-broker b63e9dbaa57b confluentinc/cp-schema-registry:5.3.1 "/etc/confluent/dock…" 8 seconds ago Up 8 seconds 0.0.0.0:8081->8081/tcp schema-registry 03711a4deba8 confluentinc/cp-zookeeper:5.3.1 "/etc/confluent/dock…" 9 seconds ago Up 8 seconds 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper Now, that all our infrastructure seems to be running, let's start using it. Let's create some Kafka topics and publish/subscribe messages to/from them. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti kafka-broker /bin/bash root@kafka:/# cd scripts root@kafka:/scripts# ./kafka-test.sh Created topic test. Topic:test PartitionCount:1 ReplicationFactor:1 Configs: Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1 >>>Test message 1 Test message 2 [2020-04-06 14:10:02,809] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.TimeoutException Processed a total of 2 messages Topic test is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true. root@kafka:/scripts# Here we are first connecting to our Docker container running the Kafka broker. Then, in the scripts subdirectory, there is a shell script named kafka-test.sh. Running this script will create a topic named Test. Once this topic is created, the script will publish two test messages on it, after which these two test messages will be consumed and displayed. Finally, the test topic is removed. Here is the source of the script: Shell root@kafka:/scripts# cat kafka-test.sh kafka-topics --create --zookeeper zookeeper:2181/kafka --replication-factor 1 --partitions 1 --topic test kafka-topics --describe --zookeeper zookeeper:2181/kafka --topic test kafka-console-producer --broker-list localhost:29092 --topic test <<EOF Test message 1 Test message 2 EOF kafka-console-consumer --topic test --from-beginning --timeout-ms 5000 --bootstrap-server localhost:29092 kafka-topics --delete --zookeeper zookeeper:2181/kafka --topic test root@kafka:/scripts# As you may see, Kafka comes out of the box with a CLI having commands like kafka-topics, kafka-console-producer and kafka-console-consumer which allows to create/handle topics and to produce/consume messages. Probe ZooKeeper. Let's try to probe now ZooKeeper by using the ZooKeeper Navigator. Get the The IP address of the zoonavigator container and fire your browser on the TCP port number 9000. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti zoonavigator hostname -I 172.20.0.3 nicolas@kfx:~/workspace/spring-kafka-integration$ open http://172.20.0.3:900 Probe the Schema Registry The next add-on that we need to probe is the Schema Registry. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti schema-registry hostname -I 172.20.0.4 nicolas@kfx:~/workspace/spring-kafka-integration$ open 172.20.0.4:8081 Probe Kafka Topics UI Let's do the same for Kafka Topics UI. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti kafka-broker /bin/bash root@kafka:/# cd scripts root@kafka:/scripts# ls -al total 16 drwxrwxr-x 2 1000 1000 4096 Apr 6 14:07 . drwxr-xr-x 1 root root 4096 Apr 6 14:08 .. -rwxrwxr-x 1 1000 1000 731 Apr 1 13:39 kafka-test-topics.sh -rwxrwxr-x 1 1000 1000 455 Apr 6 14:07 kafka-test.sh root@kafka:/scripts# ./kafka-test-topics.sh Created topic test1. >>>Created topic test2. >>>Created topic test3. >>> root@kafka:/scripts# exit exit nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti kafka-ui ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 54: eth0@if55: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 02:42:ac:14:00:08 brd ff:ff:ff:ff:ff:ff inet 172.20.0.8/16 brd 172.20.255.255 scope global eth0 valid_lft forever preferred_lft forever nicolas@kfx:~/workspace/spring-kafka-integration$ open http://172.20.0.8:8000 Here we are first connecting again to our Kafka Broker container and we execute the script scripts/kafka-test-topics.sh. This script will create some topics, exactly the same way as we did at #4. We need to do that in order to probe Kafka Topics UI. Once the script is terminated and the topics are created, we get the IP address of the Kafka Topics UI container and then we fire our browser to the Kafka Topics UI URL. The following screen will present to us: Here we can see our test topics named test1, test2 and test3 that was created by the script we ran previously. If you press the button labelled System Topics on the top of the screen, you'll see other two system topics. In total, 5 topics, as shown in the right pane of the screen. You can also see that there is only one Kafka broker and that the Kafka REST Proxy is configured as well. The last add-on to probe is Kafka Manager and this is left as an exercise for the reader. The Spring Cloud Integration Now that all our docker services are up and running, let's look at the Java code. Our project is using Spring Boot Modules and, as such, it is structured is three modules, as follows: A master maven module named spring-kafka-integration of packaging type pom The main module named spring-kafka-app containing the Spring Boot application main class, together with a REST controller allowing to invoke of the application's API. A module named spring-kafka-producer implementing the Kafka messaging production logic. A module named spring-kafka-consumer implementing the Kafka messaging consumption logic. Building and Testing In order to build the project first cd into the main directory and run the maven build, as follows: Shell cd <root-directory> mvn -DskipTests clean install Once the commands above are executed, all the required containers must be up and running, as explained previously. Now you can test the services. In order to perform testing, you need to invoke an API exposed by the application. This may be done in several ways but one of the most convenient ones is through Swagger. For that, you need to start the Spring Boot container, as follows: Shell mvn -DskipTests -pl spring-kafka-app spring-boot:run Once the application starts, going at http://localhost:8080, you'll see the following API: The API shown in the figure above is called paris-data-controller and exposes endpoints allowing retrieval of information concerning public transport in Paris. The API uses a set of web services, made available by Pierre GRIMAUD (https://fr.linkedin.com/in/pgrimaud) under an open-source license. These services, developed in Python and available at https://api-ratp.pierre-grimaud.fr/v4 and they are called by our API. So, you can now exercise the API using the Swagger GUI, by clicking on the "Try it out" button, after having unfolded the operations, by clicking on the arrow on the left. For the purposes of our demo, we only provide one operation, a GET which retrieves the stations available on a given public transport line. Two parameters are required, the type of transport, for example, SUBWAY, and the line id, in our case 8, for the subway line M8. Here is what your test should look like: Clicking on the Execute button the test will be run and you'll get the following result: Now, looking at your shell screen where you started the Spring Boot application, you'll see the listing below: This listing shows that our Swagger GUI invokes our API by making a GET request using the URI /paris-data/destinations/SUBWAY/8. Our API will, in turn, make a GET request at the endpoint https://api-ratp.pierre-grimaud.fr/v4/destinations/metros/8 to retrieve the required data, i.e. the subway stations on the M8 line, which are Pointe du Lac, platform A and Ballard, platform R. But more interesting is that, once the remote web service endpoint is invoked and the result is returned, this return will be published on a Kafka topic, as shown in the log by the following message: Shell ### Have got GetAllDestinationsResponse(result=Result(destinations=[Destination(stationName=Pointe du Lac, platformId=A), Destination(stationName=Balard, platformId=R)]), metadata=Metadata(call=GET /destinations/metros/8, date=2020-04-15T14:53:47+02:00, version=4.0)) This message is further consumed as shown below: Shell ### KafkaConsumer.doConsume(): We got a message GetAllDestinationsResponse(result=Result(destinations=[Destination(stationName=Pointe du Lac, platformId=A), Destination(stationName=Balard, platformId=R)]), metadata=Metadata(call=GET /destinations/metros/8, date=2020-04-15T14:53:47+02:00, version=4.0)) These messages are displayed by the Kafka Producer and, respectively, the Kafka Consumer, showing that the message has been successfully sent and received. You can further analyze what exactly happened by using the kafka-topics-ui service, as shown previously. Hence, going at http://192.168.96.6:8000, we'll see a Kafka topic named parisData having the following content: Here we can see our message that has been produced to the topic parisData and further consumed. Show Me the Code! We just have demonstrated a Spring Boot application using Spring Cloud Streams such that to produce and consume messages to/from Kafka topics. But how everything works here? Let's look at the code. First, the main class is annotated as follows: Java @EnableBinding({Source.class, Sink.class}) This annotation has the effect of binding the Spring Cloud Stream framework to the Kafka messaging system. This binding operation is performed on the behalf of a communication channel. Spring Cloud Stream makes available two standard channels, named Source and, respectively, Sink, the first aiming at publishing and the last at subscribing to messages. Of course, channels may and shall be customized, instead of using the Spring Cloud Stream standard classes, which only cover a limited diversity of use cases. But for simplicity's sake, we choose here to use a more basic and out-of-the-box solution instead of a more specific one. The following listing shows the KafkaProducer class. Java @Component @Slf4j public class KafkaProducer { private Source source; @Autowired public KafkaProducer(Source source) { this.source = source; } public void publishKafkaMessage(GetAllDestinationsResponse msg) { log.debug("### KafkaProducer.publisKafkaMessage(): Sending message to Kafka topic"); source.output().send(MessageBuilder.withPayload(msg).build()); } } That's all. The only thing you need here is to inject an instance of the Source which is used further, in the method publishKafkaMessage() to produce messages. The KafkaConsumer class is very simple as well: Java @Component @Slf4j public class KafkaConsumer { @StreamListener(Sink.INPUT) public void doConsume(@Payload GetAllDestinationsResponse msg) { log.debug ("### KafkaConsumer.doConsume(): We got a message \n\t{}", msg); } } The doConsume() method needs only to be annotated with the @StreamListener annotation. The class Sink is configured such that to have an input channel, named INPUT. Then using this very simple construct, the method doConsume() is listening on the input channel of the Sink service and, whenever a message is received, it will get executed. In our case, the execution is simple as it just logs a message in the log file. The code above is everything you need to implement complex Kafka messaging. However, at this point, things might seem a bit like magic because we didn't show anything concerning the Kafka brokers, the topics, etc. There is another good news as handling all these details could be done automatically, only based on a couple of properties, as shown below in the application.properties file: Java spring.application.name=spring-kafka-integration spring.cloud.stream.kafka.binder.brokers=192.168.96.3:9092 spring.cloud.stream.bindings.input.destination=parisData spring.cloud.stream.bindings.input.contentType=application/json spring.cloud.stream.bindings.output.destination=parisData spring.cloud.stream.bindings.output.contentType=application/json These properties are doing the mapping between the Source and Sink classes of Spring Cloud Streams and a Kafka broker and topic. As you can see, the Kafka broker IP address and TCP port are configured, as well as the Kafka topic name associated with the input and output channels. Using this simple use case you can implement services communicating with each other in an asynchronous manner, using messaging. Spring Cloud Stream acts as a middleman for the services while the message broker is used as an abstraction layer over the messaging system. Enjoy! Shell git clone https://github.com/nicolasduminil/kafka-spring-integration.git

ChatGPT It is a chatbot that uses the GPT-3 (Generative Pre-trained Transformer 3) language model developed by OpenAI to generate human-like responses to user input. The chatbot is designed to be able to carry on conversations with users in a natural, conversational manner, and can be used for a variety of purposes such as customer service, online support, and virtual assistance. OpenAI API OpenAI offers a number of Application Programming Interfaces (APIs) that allow developers to access the company's powerful AI models and use them in their own projects. These APIs provide access to a wide range of capabilities, including natural language processing, computer vision, and robotics. These APIs are accessed via HTTP requests, which can be sent in a variety of programming languages, including Python, Java, and MuleSoft. It also provides client libraries for several popular programming languages to make it easy for developers to get started. Most interestingly, you can train the model on your own dataset and make it work best on a specific domain. Fine-tuning the model will make it more accurate, perform better and give you the best results. MuleSoft It is a company that provides integration software for connecting applications, data, and devices. Its products include a platform for building APIs (Application Programming Interfaces) and integrations, as well as tools for managing and deploying those APIs and integrations. MuleSoft’s products are designed to help organizations connect their systems and data, enabling them to more easily share information and automate business processes. Steps To Call GPT-3 model API in MuleSoft Account: Create an account using your email id or continue with Google or Microsoft account: Authentication: The OpenAI API uses API keys for authentication. Click API Keys to generate the API key. Do not share your API key with others, or expose it in the browser or other client-side code. In order to protect the security of your account, OpenAI may also automatically rotate any API key that we’ve found has leaked publicly. (Source: documentation) The OpenAI API is powered by a family of models with different capabilities and price points. The highly efficient GPT-3 model is categorized into four models based on the power level suitable to do their task. Model Description Max Request (Tokens Training Data (Upto) text-davinci-003 Most capable GPT-3 model. Perform any task w.r.t other models with higher quality, longer output, and better instruction-following 4000 Jun-21 text-curie-001 Very capable, but faster and lower cost than Davinci. 2048 Jun-19 text-babbage-001 Very fast and lower cost, Perform straight forward task 2048 Jul-19 text-ada-001 Perform simple task 2048 Aug-19 Code snippet to call GPT-3 model (OpenAI API) in Mule application: XML <?xml version="1.0" encoding="UTF-8"?> <mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd"> <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="e5a1354b-1cf2-4963-a89f-36d035c95045" > <http:listener-connection host="0.0.0.0" port="8091" /> </http:listener-config> <flow name="chatgptdemoFlow" doc:id="b5747310-6c6d-4e1a-8bab-6fdfb1d6db3d" > <http:listener doc:name="Listener" doc:id="1819cccd-1751-4e9e-8e71-92a7c187ad8c" config-ref="HTTP_Listener_config" path="completions"/> <logger level="INFO" doc:name="Logger" doc:id="3049e8f0-bbbb-484f-bf19-ab4eb4d83cba" message="Calling completaion API of openAI"/> <http:request method="POST" doc:name="Calling completaions API" doc:id="390d1af1-de73-4640-b92c-4eaed6ff70d4" url="https://api.openai.com/v1/completions?oauth_consumer_key&oauth_token&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673007532&oauth_nonce=WKkU9q&oauth_version=1.0&oauth_signature=RXuuOb4jqCef9sRbTmhSfRwXg4I="> <http:headers ><![CDATA[#[output application/java --- { "Authorization" : "Bearer sk-***", "Content-Type" : "application/json" }]]]></http:headers> </http:request> <ee:transform doc:name="Parse Response" doc:id="061cb180-48c9-428e-87aa-f4f55a39a6f2" > <ee:message > <ee:set-payload ><![CDATA[%dw 2.0 import * from dw::core::Arrays output application/json --- (((payload.choices[0].text splitBy ".") partition ((item) -> item startsWith "\n" ))[1] ) map "$$":$]]></ee:set-payload> </ee:message> </ee:transform> </flow> </mule> Make a call to the Mule application through the API client. For example, I am using Postman. Request payload: { "model": "text-davinci-003", "prompt": "Create a list of 5 advantages of MuleSoft:", "max_tokens": 150 } model: The OpenAI API is powered by a family of models with different capabilities and price points. prompt: The prompt(s) to generate completions for, encoded as a string, array of strings, array of tokens, or array of token arrays. max_tokens: The maximum number of tokens to generate in the completion. For more details, refer to the API reference. Response payload [ { "0": " Easy Integration: Mulesoft provides easy integration with existing systems and technologies, making it faster and easier to start projects or add new services or technologies to systems" }, { "1": " Flexible Architecture: Mulesoft offers a highly configurable and flexible architecture that allows users to quickly and easily customize their solutions and make changes when needed" }, { "2": " High Performance: Mulesoft's rapid response times and high throughputs make it ideal for mission critical applications" }, { "3": " Cloud Ready: Mulesoft supports cloud friendly approaches such as microservices, containers and serverless integration architectures" }, { "4": " Efficient Development Cycles: The Mulesoft Anypoint platform includes a range of tools and services that speed up and streamline development cycles, helping to reduce the time and effort associated with creating applications" } ] Where GPT-3 Could Potentially Be Used Content Creation The API can be utilized to generate written materials and translate them from one language to another. Software Code Generation The API can be used to generate software code and simplify complicated code, making it more comprehensible for new developers. Sentiment Analysis The API can be used to determine the sentiment of text, allowing businesses and organizations to understand the public's perception of their products, services, or brand. Complex Computation The API can assist in large data processing and handle complex calculations efficiently. Limitations Like all natural language processing (NLP) systems, GPT-3 has limitations in its ability to understand and respond to user inputs. Here are a few potential limitations of GPT-3: Reliability: Chatbots may not always produce acceptable outputs and determining the cause can be difficult. Complex queries may not be understood or responded to appropriately. Interpretability: The chatbot may not recognize all variations of user input, leading to misunderstandings or inappropriate responses if not trained on diverse data. Accuracy: Chatbot, being a machine learning model, can make mistakes. Regular review and testing are needed to ensure proper functioning. Efficiency: GPT-3's test-time sample efficiency is close to humans, but pre-training still involves a large amount of text. Overall, GPT-3 is a useful tool that can improve communication and productivity, but it is important to be aware of its limitations and use it appropriately. Conclusion As chatbots continue to advance, they have the potential to significantly improve how we communicate and interact with technology. This is just an example of the various innovative approaches that are being developed to enhance chatbot performance and capabilities. As we continue to explore the potential of GPT-3 and other language generation tools, it’s important that we remain authentic and responsible in our use of these tools. This means being transparent about their role in the writing process and ensuring that the resulting text is of high quality and accuracy. References OpenAI website Beta OpenAPI website

The pandemic has wired global enterprises to become thriftier and value-focused with a view to get their technological ecosystem more efficient, lean, and meaningful, which can cater to the larger strategic goal of being effective in addressing user and customer experience. Basically, the world has moved notches towards being an “Experience Economy” than a “Feature Driven Economy.” This movement is being felt in every stratum of an enterprise’s architecture, including the Integration layer. With more and more enterprises migrating their workloads to the cloud, not just one, but multiple of them, and with each of the cloud “ecosystems” having their own native integration solutions (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO, etc.), it becomes imperative for enterprises to not only utilize the native integrations for enhanced service efficiency but to also have an Enterprise Integration Platform(EiPaaS) in the center that can be the conduit for cross-ecosystem interoperability. Likewise, as the strategic prerogatives tend more and more towards “experience,” reusability and scalability of existing APIs and integrations assume greater importance. This apart, the ability of an enterprise to create a Center for Enablement (C4E) that can be business and domain-focused, as opposed to being technology-focused (as was the situation earlier), is gaining a lot of currency in these times. The whole idea is to keep things simple. If any API is working in a certain functional area, the same should be made available to other similar functional areas across geos to drive larger RoI for the enterprises. The third trend being observed is the ability to bake in native “observability” into the Integrations so that real-time operational monitoring and reporting can take place at the level of integrations themselves. This will entail capturing key metrics around API requests, API status, Integration Compliance, API latency, etc., and the ability to draw thresholds to get proactive alerts that can enable the Site Reliability Engineers to take pre-emptive measures towards service continuance and resiliency. Figure 1: Emerging trends in the Enterprise Integration space These trends capture the spirit of co-existence (use the best-of-breed solutions), the intention to scale and replicate than build and build more, and finally, the ability to track, measure, alert, and act based on insights derived from native observability functionalities. Thus efficiency, scalability, and observability are the three key decision markers for enterprises today while evaluating Integration solutions. Let’s look at each of these three trends in detail. 1. Co-Existence of Ecosystem Integration With EiPaaS In a multi-cloud world, where enterprises are parking their workloads in multiple cloud environments while maintaining their most-essential, confidential, and risky workloads on-premises, it is a scenario that calls for various types of integrations – between on-premises and cloud, between cloud and on-premises, between cloud and cloud and between applications on-premises. Several components, applications, and systems need to interact across these environments to ensure seamless data integration, due process orchestration, and rightful end-user experience. Often, the decision-makers in an enterprise become spoiled for choice and hence confused about what should be the “go to” Integration solution that can orchestrate value across the many environments. Well, there isn’t any silver bullet solution to this quandary. But what should guide the enterprises is using the “best of the breed” approach where they leverage the native integrations of a given ecosystem (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO, etc.) for integrating all applications located in that ecosystem. Therefore, all applications stored and run out of the Azure ecosystem can leverage Azure APIs and Azure Integrations. Likewise, clients leveraging SAP or Oracle in a heavy way for their ERP needs will do well to leverage SAP PI/PO or Oracle Cloud Integrations for their respective applications. This will not only drive better runtime efficiency but also allow enterprises to derive maximum value from that ecosystem. The above solves one side of the problem. So, each of our ecosystem applications is well integrated within that ecosystem by using the native integration functionalities of that ecosystem. Within those ecosystem siloes, we have well-oiled, perfectly synced machinery. But come out of these ecosystem bubbles, and you may see distinct, siloed, nonintegrated ecosystems that are struggling to interact with each other. While most ecosystem platforms have “connectors” these days to connect to another ecosystem but imagine a customer with 10-12 different ecosystems within the enterprise, and there could be 120+ connector interactions outside of these ecosystems that would make the enterprise stare at a spaghetti set up, drawing them back to the same problem that they want to get out of. Enter Enterprise Integration Platform that becomes a one-stop orchestrator of all cross-ecosystem interactions. All connectors from each ecosystem meet the common EiPaaS hub, which will be capable of reading, translating, converting, and exposing messages in the target ecosystem-friendly language and hence becomes a common point of interaction, as shown below. Figure 2: From a spaghetti “each connector to another” pattern to an “all to one iPaaS” pattern The above solves the second problem. Therefore, we have a “best of the breed” approach where an enterprise can use all “native” integration capabilities that the ecosystem provides. In contrast, we have an EiPaaS Integration Hub, which connects the dots between the ecosystems. Below is an illustration that shows how “MuleSoft,” a leading Enterprise Integration platform, becomes a central integrating hub that brings together all ecosystems and on-premises systems. In contrast, the ecosystems themselves integrate the applications in them using their native functionalities. Figure 3: Sample: “Co-existence” of iPaaS platform and ecosystem integration platforms Enterprises should respect each platform for its own benefits, leverage native integrations for the application ecosystem, and leverage enterprise integration tools for integrations outside the application ecosystem. The technologies shown in the figure above are for illustration only. 2. Scalability of Integration Best Practices With an “All-Encompassing” C4E (Centre for Enablement) “All-encompassing” is the operative term here for a reason. Enterprises today use different integration technologies for different use cases. Different units and functions use various integration technologies too, as per their need, expertise, and comfort factor. With the empowerment of Citizen Technologists, low code and no code integration tools like Workato, Tray, Zapier, etc. are also sitting in the overall enterprise landscape, alongside more strategic platforms like MuleSoft, Boomi, TIBCO, etc., which also co-exist as seen in the trend above, alongside ecosystem integration functionalities. Thus, suddenly, an enterprise looks at a smorgasbord of technologies, each with its niche requirement and each being used by a separate team or a group of teams! The challenge is how to govern a wide swathe of technologies and teams using them and, more importantly, how to make the integrations reusable, replicable, and scalable across geos and units for similar use cases. Therein lies the value. Many questions and options arise. For example, should a “decentralized model” exist where we have one CoE governing each business unit? Or one CoE governing all the work being done using an integration technology? Or should a “centralized model” exist where one common CoE orchestrates and governs the work across all business units or technologies? Or should there be a “hybrid model” where one Hub CoE interacts with a few “Spoke” CoEs in a hub and spoke” model, and the “spoke CoEs” in turn support a group of business units? There are pros and cons to each of these models. For example, the “decentralized model” is more suited for large enterprises with independent business units. But lack of consistency across the units can breed a shadow IT culture. Likewise, the “centralized model” is more suitable for small and medium enterprises and start-ups which have a small central IT running the show. Finally, the hybrid model is more suited for large enterprises that encourage autonomous business units, which run in a product organization model and want to be self-sufficient with a centralized, horizontal CoE for any governance and utilities support. While each of these options is germane, one factor that can truly accommodate for all these options to co-exist is the provisioning of an “all-encompassing” C4E (Center for Enablement) that is truly “extended” and “accommodates” all teams, technologies, and practices under its ambit, governs them, supplies the enterprises with common standards of working, best practices enabling business divisions to build and drive the consumption of assets successfully, enabling speed and agility. It allows the business and IT teams to shift from a production-based to a production-and-consumption-based delivery model. This production and consumption model fuels reusability and scalability. The C4E is a broader concept than CoE. While the CoEs develop, run, and manage APIs, their operational elements and are a container of functional expertise in a centralized manner, C4Es are enablers, capability builders, decentralized mechanisms of governance that bring in best practices, reusable templates, evangelism, and community building. In addition, they are domain focused and make the business units self-sufficient, independent, and scalable. 3. Observability Baked Into Integrations for Reactive Knowhow and Proactive Alerts Observability is a comprehensive set of capabilities around logging, analytics, monitoring, troubleshooting, measuring platforms, runtimes, and applications. Observability plays a decisive role in providing the real-time performance of a system. Lacking observability in a system would be like flying an aircraft without a fuel indicator. Observability is not only shifting left but also diving deep into each layer of the enterprise architecture. From the systems to applications to the processes to the integrations between them, enterprises expect observability and monitoring mechanisms to be baked into each of these elements so that not only is the performance reported but with the application of AI and by drawing clear performance thresholds, proactive alerts can be sounded out to the SREs, thus ensuring service resiliency and continuance. In this regard, most Integration platforms today come laced with out-of-the-box (OOB) support for many of the key observables. However, what can be observed and monitored also depends on the deployment option- Cloud, On-Prem, or Hybrid. Some of the key observables around Integration that the OOB functionalities can observe and report are: Figure 4: Key Observables in an Integration Solution There are various third-party observability tools in the market that fill the vacuum left by the OOB functionalities of the Integration platforms. For example, Splunk, ELK, and SumoLogic are commonly used for log aggregation and analysis, while the likes of Jaeger, Prometheus, Datadog, etc., are popular for their tracing capabilities. However, tools alone do not solve the problem of observability. Having an experienced team of reliability engineers who can interpret metrics and act proactively will be key to success. As they say, it is not data but the action taken on the insights derived from the data which determines true success. Therefore, partnering with the right service partner will be key to Enterprise Integration success. Conclusion The Enterprise Integration market is in flux post-pandemic as effectiveness, efficiency, cost optimization, and performance precision are all expected by enterprises in their Integration solution. These are seemingly contrarian demands, but a balanced solution that can ensure the co-existence of diverse solutions for diverse use cases, with an ability to scale and a capability to observe and report key metrics, is key to enterprise integration success. In addition, while the technologies are available in the wider market, a capable service partner with experience in deploying, managing, orchestrating, scaling, and bringing about governance support will be key to Integration strategy success.

This review is about API Design Patterns by JJ Geewax from Manning. I already mentioned how I'm trying to get up to speed in the API world:reading books, viewing relevant YouTube videos, and reading relevant IETF RFCs. Facts 30 chapters, $35.00 The author is a Principal Software Engineer at Google Chapters Introduction Design principles Naming Resource scope and hierarchy Data types and defaults Fundamentals Resource identification: How to identify resources in an API Standard methods: The set of standard methods for use in resource-oriented APIs Partial updates and retrievals: How to interact with portions of resources Custom methods: Using custom (non-standard) methods in resource-oriented APIs Long-running operations: How to handle methods that are not instantaneous Rerunnable jobs: Running repeated custom functionality in an API Resource relationships Singleton sub-resources: Isolating portions of resource data Cross references: How to reference other resources in an API Association resources: How to manage many-to-many relationships with metadata Add and remove custom methods: How to manage many-to-many relationships without metadata Polymorphism: Designing resources with dynamically-typed attributes Collective operations Copy and move: Duplicating and relocating resources in an API Batch operations: Extending methods to apply to groups of resources atomically Criteria-based deletion: Deleting multiple resources based on a set of filter criteria Anonymous writes: Ingesting unaddressable data into an API Pagination: Consuming large amounts of data in bite-sized chunks Filtering: Limiting result sets according to a user-specified filter Importing and exporting: Moving data into or out of an API by interacting directly with a storage system Safety and Security Versioning and compatibility: Defining compatibility and strategies for versioning APIs Soft deletion: Moving resources to the "API recycle bin" Request deduplication: Preventing duplicate work due to network interruptions in APIs Request validation: Allowing API methods to be called in "safe mode" Resource revisions: Tracking resource change history Request retrial: Algorithms for safely retrying API requests Request authentication: Verifying that requests are authentic and untampered with Each design pattern chapter follows the same structure: Motivation: what problem solves the pattern Overview: a short description of the pattern Implementation: an in-depth explanation of the pattern. It's structured into different subsections. Trade-offs: patterns have strong and weak points; this section describes the latter Exercises: a list of questions to verify that one has understood the pattern Pros and Cons Let's start with the good sides: As I mentioned above, the structure of each chapter dedicated to a design pattern is repetitive. It makes the chapter easy to consume, as you know exactly what to expect. In general, I read my technical books just before going to sleep because I'm pretty busy during the day. Most books have long chapters, requiring me to stop mid-chapter when I start to fall asleep. When you start again, you need to get back a few pages to get the context back. The length of a chapter on API Design Patterns is ideal: neither too long nor too short. The Design Principles section starts from the basics. You don't need to be an expert on API to benefit from the book. I was not; I hope that I'm more seasoned by now. I was a bit dubious at first about the Exercises section of each chapter, for it didn't provide any solution. However, I came to realize it activates the active recall mechanism: instead of passively reading, actively recall what you learned in answering questions. It improves the memorization of what was learned. As an additional benefit, you can learn in a group, compare your answers and eventually debate them. Now, I've some critiques as well: Some patterns are directly taken from Google's API Improvement Proposals. It's not a problem per se, but when it's the case, there's no discussion at all about possible alternatives. For example, the chapter on custom methods describes how to handle actions that don't map precisely to an HTTP verb: a bank transfer is such an action because it changes two resources, the "from" and the "to" accounts.The proposed Google AIP is for the HTTP URI to use a : character followed by the custom verb, e.g., /accounts/123:transfer. That's an exciting proposal that solves the lack of mapping issue. But there are no proposed alternatives nor any motivation for why it should be this way. As an engineer, I can hardly accept implementing a solution with such far-reaching consequences without being provided with other alternatives with their pros and cons. Last but not least, the book doesn't mention any relevant RFC or IETF draft. Chapter 26 describes how to manage request deduplication, the fact that one may need to send the same non-idempotent request repeatedly without being afraid of ill side effects. The proposed solution is good: the client should use a unique key, and if the server gets the same key again, it should discard the request.It's precisely what the IETF draft describes: The Idempotency-Key HTTP Header Field. Still, there's no mention of this draft, giving the feeling that the book is disconnected from its ecosystem. Author's Replies For once, I was already in touch with the author. I offered him an opportunity to review the post. Since his answers are open, I decided to publish them with his permission: Why isn't there more discussion about alternatives? I think you're right — and I actually had quite a bit of discussion of the alternatives in the original manuscript, and I ended up chopping them out. One reason was that my editor wanted to keep chapters reasonably sized, and my internal debates and explanations of why one option was better or worse than another was adding less value than "here's the way to do it." The other reason was that I "should be opinionated." If this were a textbook for a class on exploring API design I could weigh all the sides and put together a proper debate on the pros and cons of the different options, but in most cases there turned out to be a very good option that we've tried out and seen work really well over the course of 5+ years (e.g., : for custom methods). In other cases we actively didn't have that and the chapter is a debate showing the different alternatives (e.g., Versioning). If I could do a 600-700 pages, I think it would have this for you. Why aren't there more references to IETF standards? This is a glaring oversight on my part. There are a lot of RFCs and I do mention some (e.g., RFC-6902 in Partial Updates, etc), but I haven't pulled in enough from that standards body and it's a mistake. If we do a 2nd edition, this will be at the top of the list. Conclusion Because of a couple of cons I mentioned, API Design Patterns falls short of being a reference book. Nonetheless, it's a great book that I recommend to any developer entering the world of APIs or even one with more experience to round up their knowledge.

A proxy is something that acts on behalf of something else. Your best friend giving your attendance in the boring lecture you bunked during college is a real-life example of proxying. When it comes to API development, a proxy is an interface between the client and the API server. The job of the interface is to proxy incoming requests to the real server. The use of proxy But why might you need an API proxy in the first place? It could be possible that the real API server is external to your organization and unstable. A proxy can provide a more stable interface to the client The response from the API server might not be compatible with the client’s expectations and you want to modify the response in some form (for example, converting XML to JSON). The real API server may be a temporary arrangement and you don’t want the clients to get impacted by any future changes There are several uses of API proxy depending on the situation. In this post, you will learn how to build a Node.js API Proxy using the http-proxy-middleware package. Just to make things clear, API Proxy is different from a Forward Proxy. 1. Node.js API Proxy Project Setup First, you need to initialize the project by executing the below command in a project directory: $ npm init -y This will generate a basic package.json file with meta-data information about the project such as name, version, author and scripts. Next, install a couple of packages for developing the Node.js API proxy. $ npm install --save express http-proxy-middleware express is a minimalistic web framework you can use to build API endpoints. http-proxy-middleware is a simple Node.js package to create an API proxy After the package installation, define a start command for the project within the package.json file. You can use this command to start the application. Your project’s package.json should look similar to the below example. { "name": "express-proxy-demo", "version": "1.0.0", "description": "Demo Application for Proxy Implementation in Node.js", "main": "index.js", "scripts": { "start": "node index.js" }, "author": "Saurabh Dashora", "license": "ISC", "dependencies": { "express": "^4.18.2", "http-proxy-middleware": "^2.0.6" } } 2. Creating a Node.js Proxy Using http-proxy-middleware Time to create the actual application. The example application will proxy incoming requests to an API hosted elsewhere. For demonstration purposes, I recommend using the fake APIs hosted at JSONPlaceholder. See the below illustration: Node.js Proxy Setup Check the below code from the index.js file that contains the logic for proxying requests. const express = require('express'); const { createProxyMiddleware } = require('http-proxy-middleware'); const app = express(); const PORT = 3000; const HOST = "localhost"; const API_URL = "<https://jsonplaceholder.typicode.com>"; app.get("/status", (req, res, next) => { res.send('This is a proxy service'); }); const proxyOptions = { target: API_URL, changeOrigin: true, pathRewrite: { [`^/api/posts`]: '/posts', }, } const proxy = createProxyMiddleware(proxyOptions); app.use('/api/posts', proxy) app.listen(PORT, HOST, () => { console.log(`Proxy Started at ${HOST}:${PORT}`) }); Let’s understand each step in the above program: Step 1: The first segment of the code contains the import statements for express and http-proxy-middleware. Step 2: The next statement creates an application instance using the call to express() function followed by declaring a few important constants such as PORT, HOST and API_URL. Step 3: Implement an endpoint /status to describe the role of the application. This endpoint has nothing to do with proxying requests and provides a way to test our application. Step 4: Next, declare an object proxyOptions. This is a configuration object for our API proxy. It contains a few important properties target - It defines the target host where you want to proxy requests. In our case, this is the [<https://jsonplaceholder.typicode.com>](<https://jsonplaceholder.typicode.com>) changeOrigin - This is set to true since we are proxying to a different origin. pathRewrite - This is a very important property where you define the rules for rewriting the path. For example, the expression [^/api/posts]: '/posts' routes all incoming requests direct at /api/posts to just /posts. In other words, this will remove the /api prefix from the path. Step 5: After declaring the configuration object, create the proxy object by calling createProxyMiddleware() function with the proxyOptions object as input. Step 6: Next, create a request handler for the path /api/posts and pass the proxy object as handler for the incoming request. Step 7: At the very end, start the Node.js API Proxy server to listen on the port and host already declared earlier. You can start the application using the command npm run start. > express-proxy-demo@1.0.0 start > node index.js [HPM] Proxy created: / -> <https://jsonplaceholder.typicode.com> [HPM] Proxy rewrite rule created: "^/api/posts" ~> "/posts" Proxy Started at localhost:3000 Messages about the proxy setup indicate that the proxy is configured properly. If you visit the URL [<http://localhost:3000/api/posts/1>](<http://localhost:3000/api/posts/1>) in the browser, you will get the response from the JSONPlaceholder APIs as below: { "userId": 1, "id": 1, "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit", "body": "quia et suscipit\\nsuscipit recusandae consequuntur expedita et cum\\nreprehenderit molestiae ut ut quas totam\\nnostrum rerum est autem sunt rem eveniet architecto" } This means that the Node.js API Proxy is doing its job by proxying requests to the mock APIs hosted by JSONPlaceholder. 3. Node.js API Proxy Context Matching The http-proxy-middleware uses the path for proxying requests. For example, in the request http://localhost:3000/api/posts?title=test, the section /api/posts is the actual path. According to the official documentation of the http-proxy-middleware package, there are various ways in which the context matching for the path takes place: Path Matching createProxyMiddleware({...}) matches all paths. This means all requests will be proxied. createProxyMiddleware('/', {...}) also matches all paths. createProxyMiddleware('/api', {...}) only matches paths starting with /api. Multiple Path Matching createProxyMiddleware(['/api', '/test', '/otherpath'], {...})can be used to match multiple paths to a particular proxy configuration Wildcard Path Matching For more fine grained control, you can also use wildcards to match paths. createProxyMiddleware('**', {...}) matches any path and all requests are proxied. createProxyMiddleware('**/*.html', {...}) matches any path which ends with .html. createProxyMiddleware('/api/**/*.html', {...}) matches requests ending with .html within the overall path/api. Custom Path MatchingFor even greater control, you can also provide a custom function to match the path for the API Proxy. See below example:const filter = function (pathname, req) { return pathname.match('^/api') && req.method === 'GET'; }; const apiProxy = createProxyMiddleware(filter, { target: '<https://jsonplaceholder.typicode.com>', }); In the above example, only GET requests to the path /api are proxied. Conclusion With this post, you have built a very simple version of the Node.js API proxy. You can extend it further based on specific requirements. The http-proxy-middleware is a simple but powerful library to build a Node.js API Proxy server. The library provides several configurable properties to handle the proxy functionalities. However, there are many more options that you can leverage for your needs. The code for this demo is available on GitHub. If you found the post useful, consider sharing it with friends and colleagues. In case of any queries, write them in the comments section below.

Regardless of the industry vertical we work in – whether that be somewhere in the technology, e-commerce, manufacturing, or financial fields (or some Venn diagram of them all) – we mostly rely on the same handful of standard document formats to share critical information between our internal teams and with external organizations. These documents are almost always on the move, bouncing across our networks as fast as our servers will allow them to. Many internal business automation workflows, for example – whether written with custom code or pieced together through an enterprise automation platform – are designed to process standard PDF invoice documents step by step as they pass from one stakeholder to another. Similarly, customized reporting applications are typically used to access and process Excel spreadsheets which the financial stakeholders of a given organization (internal or external) rely on. All the while, these documents remain beholden to strictly enforced data standards, and each application must consistently uphold these standards. That’s because every document, no matter how common, is uniquely capable of defying some specific industry regulation, containing an unknown error in its encoding, or even - at the very worst - hiding a malicious security threat behind a benign façade. As rapidly evolving business applications continue to make our professional lives more efficient, business users on any network place more and more trust in the cogs that turn within their suite of assigned applications to uphold high data standards on their behalf. As our documents travel from one location to another, the applications they pass through are ultimately responsible for determining the integrity, security, and compliance of each document’s contents. If an invalid PDF file somehow reaches its end destination, the application which processes it – and, by extension, those stakeholders responsible for creating, configuring, deploying, and maintaining the application in the first place – will have some difficult questions to answer. It’s important to know upfront, right away, whether there are any issues present within the documents our applications are actively processing. If we don’t have a way of doing that, we run the risk of allowing our own applications to shoot us in the foot. Thankfully, it’s straightforward (and standard) to solve this problem with layers of data validation APIs. In particular, document validation APIs are designed to fit seamlessly within the architecture of a file processing application, providing a quick feedback loop on each individual file they encounter to ensure the application runs smoothly when valid documents pass through and halting its process immediately when invalid documents are identified. There are dozens of common document types which require validation in a file processing application, and many of the most common among those, including PDF, Excel, and DOCX (which this article seeks to highlight), are all compressed and encoded in very unique ways, making it particularly vital to programmatically identify whether their contents are structured correctly and securely. Document Validation APIs The purpose of this article is to highlight three API solutions that can be used to validate three separate and exceedingly common document types within your various document processing applications: PDF, Excel XLSX, and Microsoft Word DOCX. These APIs are all free to use, requiring a free-tier API key and only a few lines of code (provided below in Python for your convenience) to call their services. While the process of validating each document type listed above is unique, the response body provided by each API is standardized, making it efficient and straightforward to identify whether an error was found within each document type and if so, what warnings are associated with that error. Below, I’ll quickly outline the general body of information supplied in each of the above document validation API's response: DocumentIsValid – This response contains a simple Boolean value indicating whether the document in question is valid based on its encoding. PasswordProtected – This response provides a Boolean value indicating whether the document in question contains password protection (which – if unexpected – can indicate an underlying security threat). ErrorCount – This response provides an integer reflecting the number of errors detected within the document in question. WarningCount – This response indicates the number of warnings produced by the API response independently of the error count. ErrorsAndWarnings – This response category includes more detailed information about each error identified within a document, including an error description, error path, error URI (uniform resource identifier, such as URL or URN), and IsError Boolean. Demonstration To use any of the three APIs referred to above, the first step is to install the Python SDK with a pip command provided below: pip install cloudmersive-convert-api-client With installation complete, we can turn our attention to the individual functions which call each individual API’s services. To call the PDF validation API, we can use the following code: Python from __future__ import print_function import time import cloudmersive_convert_api_client from cloudmersive_convert_api_client.rest import ApiException from pprint import pprint # Configure API key authorization: Apikey configuration = cloudmersive_convert_api_client.Configuration() configuration.api_key['Apikey'] = 'YOUR_API_KEY' # create an instance of the API class api_instance = cloudmersive_convert_api_client.ValidateDocumentApi(cloudmersive_convert_api_client.ApiClient(configuration)) input_file = '/path/to/inputfile' # file | Input file to perform the operation on. try: # Validate a PDF document file api_response = api_instance.validate_document_pdf_validation(input_file) pprint(api_response) except ApiException as e: print("Exception when calling ValidateDocumentApi->validate_document_pdf_validation: %s\n" % e) To call the Microsoft Excel XLSX validation API, we can use the following code instead: Python from __future__ import print_function import time import cloudmersive_convert_api_client from cloudmersive_convert_api_client.rest import ApiException from pprint import pprint # Configure API key authorization: Apikey configuration = cloudmersive_convert_api_client.Configuration() configuration.api_key['Apikey'] = 'YOUR_API_KEY' # create an instance of the API class api_instance = cloudmersive_convert_api_client.ValidateDocumentApi(cloudmersive_convert_api_client.ApiClient(configuration)) input_file = '/path/to/inputfile' # file | Input file to perform the operation on. try: # Validate a Excel document (XLSX) api_response = api_instance.validate_document_xlsx_validation(input_file) pprint(api_response) except ApiException as e: print("Exception when calling ValidateDocumentApi->validate_document_xlsx_validation: %s\n" % e) And finally, to call the Microsoft Word DOCX validation API, we can use the final code snippet supplied below: Python from __future__ import print_function import time import cloudmersive_convert_api_client from cloudmersive_convert_api_client.rest import ApiException from pprint import pprint # Configure API key authorization: Apikey configuration = cloudmersive_convert_api_client.Configuration() configuration.api_key['Apikey'] = 'YOUR_API_KEY' # create an instance of the API class api_instance = cloudmersive_convert_api_client.ValidateDocumentApi(cloudmersive_convert_api_client.ApiClient(configuration)) input_file = '/path/to/inputfile' # file | Input file to perform the operation on. try: # Validate a Word document (DOCX) api_response = api_instance.validate_document_docx_validation(input_file) pprint(api_response) except ApiException as e: print("Exception when calling ValidateDocumentApi->validate_document_docx_validation: %s\n" % e) Please note that while these APIs do provide some basic security benefits during their document validation processes (i.e., identifying unexpected password protection on a file, which is a common method for sneaking malicious files through a network - the password can be supplied to an unsuspecting downstream user at a later date), they do not constitute fully formed security APIs, such as those that would specifically hunt for viruses, malware, and other forms of malicious content hidden within a file. Any document – especially those that originated outside of your internal network – should always be thoroughly vetted through specific security-related services (i.e., services equipped with virus and malware signatures) before entering or leaving your file storage systems.

Books on bad programming habits take up a fraction of the shelf space dedicated to best practices. We know what good habits are – or we pay convincing lip service to them – but we lack the discipline to prevent falling into bad habits. Especially when writing test code, it is easy for good intentions to turn into bad habits, which will be the focus of this article. But first, let’s get the definitions right. An anti-pattern isn’t simply the absence of any structured approach, which would amount to no preparation, no plan, no automated tests and just hacking the shortest line from brainwave to source code. This chaotic approach is more like non-pattern programming. Anti-patterns are still patterns, just unproductive ones, according to the official definition. The approach must be structured and repeatable, even when it is counter-productive. Secondly, a more effective, documented, and proven solution to the same problem must be available. Many (in)famous anti-patterns consistently flout one or more good practices. Spaghetti code and the god object testify to someone’s ignorance or disdain of the principles of loose coupling and cohesion, respectively. Education can fix that. More dangerous, however, are the folks who never fell out of love with over-engineering since the day they read about the Gang of Four because doing too much of a good thing is the anti-pattern that rules them all. It’s much harder to fight because it doesn’t feel like you’re doing anything wrong. Drawn To Complexity Like Moths to a Flame – With Similar Results In the same as you can overdose on almost anything that is beneficial in small doses, you can overdo any programming best practice. I don’t know many universally great practices, only suitable and less suitable solutions to the programming challenge at hand. It always depends. Yet developers remain drawn to complexity like moths to a flame, with similar results. The usefulness of SOLID design principles has a sweet spot, after which it’s time to stop. It doesn’t follow an upward curve where more is always better. Extreme dedication to the single responsibility principle gives you an explosion of specialized boilerplate classes that do next to nothing, leaving you clueless as to how they work together. The open/closed principle makes sense if you’re maintaining a public API, but for a work-in-progress, it’s better to augment existing classes than create an overly complex inheritance tree. Dependency inversion? Of course, but you don’t need an interface if there will only ever be one private implementation, and you don’t always need Spring Boot to create an instance of it. The extreme fringes on opposite sides of the political spectrum have more in common with each other than they have with the reasonable middle. Likewise, no pattern at all gives you the same unreadable mess as well-intentioned over-engineering, only a lot quicker and cheaper. Cohesion gone mad gives you FizzBuzzEnterpriseEdition while too little of it gives you the god object. Let's turn over, then, to test code and the anti-patterns that can turn the efforts into an expensive sinkhole. Unclarity about the purpose of testing is already a massive anti-pattern before a single line of code is written. It’s expressed in the misguided notion that any testing is better than no testing. It isn’t. Playing foosball is better than ineffectual testing because the first is better for team morale and doesn’t lull you into a false sense of security. You must be clear on why you write tests in the first place. First Anti-Pattern: Unaware of the Why Well, the purpose of writing tests is to bring Sonar coverage up to 80%, isn’t it? I’m not being entirely sarcastic. Have you never inherited a large base of untested legacy that has been working fine for years but is languishing at an embarrassing 15% test coverage? Now suddenly the powers that be decide to tighten the quality metrics. You can’t deploy unless coverage is raised by 65 percentage points, so the team spends several iterations writing unit tests like mad. It’s a perverse incentive, but it happens all the time: catch-up testing. Here are three reasons that hopefully make more sense. First, tests should validate specifications. They verify what the code is supposed to do, which comes down to producing the output that the stakeholders asked for. A developer who isn’t clear on requirements can only write a test that confirms what the code already does, and only from inspecting the source code. Extremely uncritical developers will write a test confirming that two times four equals ten because that’s what the (buggy) code returns. This is what can happen when you rush to improve coverage on an inherited code base and don’t take the time to fully understand the what and why. Secondly, tests must facilitate clean coding; never obstruct it. Only clean code keeps maintenance costs down, gets new team members quickly up to speed, and mitigates the risk of introducing bugs. Developing a clean codebase is a highly iterative process where new insights lead to improvements. That means constant refactoring. As the software grows, it’s fine to change your mind about implementation details, but you can only improve code comfortably that way if you minimize the risk that your changes break existing functionality. Good unit tests warn you immediately when you introduce a regression, but not if they’re slow or incomplete. Thirdly, tests can serve as a source of documentation for the development team. No matter how clean your code, complex business logic is rarely self-explanatory if all you have is the code. Descriptive scenarios with meaningful data and illustrative assertions show the relevant input permutations much clearer than any wiki can. And they’re always up to date. Second Anti-Pattern: London School Orthodoxy I thank Vladimir Khorikov for pointing out the distinction between the London versus the classical unit test approach. I used to be a Londoner, but now I’m convinced that unit tests should primarily target public APIs. Only this way can you optimize the encapsulated innards without constantly having to update the tests. Test suites that get in the way of refactoring are often tightly coupled to implementation details. As long as you can get sufficient execution speed and coverage, I find no compelling reason for a rigid one-to-one mapping between source classes and corresponding test classes. Such an approach forces you to emulate every external dependency’s behavior with a mocking framework. This is expensive to set up and positively soul-crushing if the classes under test have very little salient business logic. A case in point: Java @RestController public class FriendsController { @AutoWired FriendService friendService; @AutoWired FriendMapper friendMapper; @GetMapping(“/api/v1/friends”) public List<FriendDto> getAll(){ return friendMapper.map(friendService.retrieveAll()); } } This common Controller/Service layered architecture makes perfect sense: cohesion and loose coupling are taken care of. The Controller maps the network requests, (de)serializes input/output, and handles authorization. It delegates to the Service layer, which is where all the exciting business logic normally takes place. CRUD operations are performed through an abstraction to the database layer, injected in the service layer. Not much seems to go on in this simple example, but that’s because the framework does the heavy lifting. If you leave Spring out of the equation, there is precious little to test, especially when you add advanced features like caching and repositories generated from interfaces. Boilerplates and configurations do not need unit tests. And yet I keep seeing things like this: Java @ExtendWith(MockitoExtension.class) public class FriendsControllerTest { @Mock FriendService friendService; @Mock FriendMapper friendMapper; @InjectMocks FriendController controller; @Test void retrieve_friends(){ //arrange var friendEntities = List.of(new Friend(“Jenny)); var friendDtos = List.of(new FriendDto(“Jenny”)); Mockito.doReturn(friendEntities).when(friendService).findAll(); Mockito.doReturn(friendDtos).when(friendMapper).map(eq(friendEntities)); //act var friends = controller.findAll(); //assert Assertions.assertThat(friends).hasSize(1); Assertions.assertThat(friends.get(0).getName()).isEqualTo(“Jenny”); } } A test like this fails on three counts: it’s too much concerned with implementation details to validate specifications and it’s too simplistic to have any documentary merit. And being tightly bound to the implementation, it surely does not facilitate refactoring. Even a “Hello, World!”-level example like this takes four lines of mock setup. Add more dependencies with multiple interactions and 90% of the code (and your time!) is taken up with tedious mocks setup. What matters most is that Spring is configured with the right settings. Only a component test that spins up the environment can verify that. If you include a test database, it can cover all three classes without any mocking, unless you need to connect to an independent service outside the component under test. Java @SpringBootTest @AutoConfigureMockMvc class FriendsControllerTest { @Test @WithAnonymousUser void get_friends(){ mockMvc.perform(get("/v1/api/friends")) .andExpect(content().string(“[{\“name\”: \”Jenny\”}]”)) } } Third Anti-Pattern: Trying to Test the Untestable The third anti-pattern I want to discuss rears its head when you try to write tests for complex business functionality without refactoring the source. Say we have a 1500-line monster class with one deceptively simple public method. Give it your outstanding debt, last year’s salary slips, and it tells you how much you’re worth. public int getMaxLoanAmountEuros(List<SalaryReport> last12SalarySlips, List<Debt> outstandingDebt); It’s different from the previous example in two important ways: The code-under-test centers on business logic and has high cyclomatic complexity, requiring many scenarios to cover all the relevant outcomes. The code is already there and testing is late to the party, meaning that by definition you can’t work in a test-driven approach. That of itself is an anti-pattern. We’re writing tests as an afterthought, only to validate what the code does. The code may be very clean, with all complexity delegated to multiple short private methods and sub-classes to keep things readable. Sorry, but if there are no unit tests, it’s more likely to be a big ball of mud. Either way, we can’t reduce the essential complexity of the business case. The only way to reach full coverage of all the code under this single public method is by inventing scenarios with different input (the salary and debt information). I have never been able to do that comfortably without serious refactoring, by delegating complex isolated portions to their own class and writing dedicated tests per class. If you’re terrified of breaking things, then changing the access level of private methods to default scope is safer. If the test is in the same package, you might write focused tests for these methods, but it’s a controversial strategy. Best avoid it. You’re breaking encapsulation, wading knee-deep in implementation details, which makes future refactoring even harder. You have to proceed carefully if you tamper with untested production code, but there is no good alternative other than a full rewrite. Writing thorough unit tests for messy code that you inherited without the privilege to refactor is painful and ineffectual. That’s because untested code is often too cumbersome to test. Such untestable code is by definition bad code, and we should not settle for bad code. The best way to avoid that situation is to be aware of the valid reasons why we test in the first place. You will then never write tests as an afterthought. Green Checkbox Addiction The productive path to establishing and maintaining effective test automation is not easy, but at least the good habits make more sense than the well-intentioned yet harmful anti-patterns I pointed out here. I leave you with a funny one, which you might call green checkbox addiction: the satisfaction of seeing an all-green suite, regardless of whether the tests make any sense. That's the false sense of security I mentioned earlier, which makes bad testing worse than no testing at all. It’s like the productivity geeks who create spurious tasks in their to-do lists for the dopamine spike they get when checking them off. Very human, and very unproductive.

The safety of these applications is crucial to prevent attackers from compromising the computer on which developers are working, as they could use this access to obtain sensitive information, alter source code, and further pivot into the company's internal network. This time, my team and I dive into a new vulnerability I identified in one of the most popular IDEs: Visual Studio Code. It allowed attackers to craft malicious links that, once interacted with, would trick the IDE into executing unintended commands on the victim's computer. By reporting the issue to Microsoft, who quickly patched it, our researchers helped to secure the developer ecosystem. Impact The vulnerability can be used to target developers that have the Visual Studio Code IDE installed. Upon clicking on a malicious link crafted by an attacker, victims are prompted to clone a Git repository in Visual Studio Code. This operation is genuine and part of the workflow of most users. For instance, this is how GitLab allows easier cloning of projects: If the developer accepts this operation, attackers can execute arbitrary commands on the victim's computer. Interestingly, Workspace Trust, a feature to harden the IDEs and reduce the risk of unintended commands being executed, is not strictly enforced here. If the last Visual Studio Code window with focus is trusted by the current workspace, this security feature will not have the expected effect. I disclosed this finding to Microsoft through their Researcher Portal, and the patch was released as part of Visual Studio Code 1.67.1 and higher. Microsoft published limited information about this bug as part of their security bulletin and assigned it CVE-2022-30129. In the sections below, I'll first describe how URL handlers are designed in Visual Studio Code and then review the implementation of the one reserved for Git actions to identify an argument injection bug. Further sections will describe how it could be exploited to gain the ability to execute arbitrary commands, as well as the patch implemented by Microsoft. Visual Studio Code URL Handlers Visual Studio Code is most commonly used as a stand-alone desktop application, thanks to Electron. This choice still allows some level of integration with the user's operating system, for instance, by allowing applications to register custom URL protocol handlers. In the case of Visual Studio Code, vscode:// is registered, and vscode-insiders:// for the nightly builds (also called Insiders build). This feature is named Deep Links. The IDE allows internal and external extensions to listen to such events and handle them by registering sub-handlers. The main listener will handle such OS-level events and redirect them to the right extension. They have to implement a simple interface with a method named handleUri() and announce it to the IDE with window.registerUriHandler(): src/vscode-dts/vscode.d.ts TypeScript export interface UriHandler { handleUri(uri: Uri): ProviderResult<void>; } Finding an Argument Injection in the Git Module With this design in mind, it is now possible to start looking for URL handlers in the core of Visual Studio Code. At that time, three were available: extensions/github-authentication and extensions/microsoft-authentication to authenticate with third-party services and obtain the resulting access tokens, and extensions/git to allow users to clone remote repositories as shown above in GitLab. With my prior experience reviewing the code of developer tools, I know that external invocations of version control tools are often riddled with argument injection bugs—you can head to the Related Posts section for a few examples. Let's look at the extensions/git's implementation of handlerUri first! extensions/git/src/protocolHandler.ts TypeScript export class GitProtocolHandler implements UriHandler { // [...] handleUri(uri: Uri): void { switch (uri.path) { case '/clone': this.clone(uri); } } private clone(uri: Uri): void { const data = querystring.parse(uri.query); // [...] commands.executeCommand('git.clone', data.url); } The git.clone command is implemented in extensions/git/src/commands.ts; it is also possible to invoke it manually: extensions/git/src/commands.ts TypeScript @command('git.clone') async clone(url?: string, parentPath?: string): Promise<void> { await this.cloneRepository(url, parentPath); } Let's continue to dig deeper into the code to identify where the external Git binary is invoked: extensions/git/src/commands.ts TypeScript async cloneRepository(url?: string, parentPath?: string, options: { recursive?: boolean } = {}): Promise<void> { // [...] try { // [...] const repositoryPath = await window.withProgress( opts, (progress, token) => this.git.clone( url!, { parentPath: parentPath!, progress, recursive: options.recursive }, token) ); extensions/git/src/git.ts TypeScript async clone(url: string, options: ICloneOptions, cancellationToken?: CancellationToken): Promise<string> { let baseFolderName = decodeURI(url).replace(/[\/]+$/, '').replace(/^.*[\/\\]/, '').replace(/\.git$/, '') || 'repository'; let folderName = baseFolderName; let folderPath = path.join(options.parentPath, folderName); // [...] try { let command = ['clone', url.includes(' ') ? encodeURI(url) : url, folderPath, '--progress']; if (options.recursive) { command.push('--recursive'); } await this.exec(options.parentPath, command, { cancellationToken, env: { 'GIT_HTTP_USER_AGENT': this.userAgent }, onSpawn, }); As promised, there is an argument injection bug in this code: the URL to clone the Git repository is fully controlled and concatenated into the external command line. If this URL starts with dashes, Git will understand it as an option instead of a positional argument. Exploiting an Argument Injection on a Git Clone Operation Argument injection vulnerabilities are very interesting because they are all different and often imply some subtleties; this one is not an exception. This section describes one way to exploit it; other ways exist and are left as an exercise to the reader. Git used to implement git-remote-ext to "bridge smart transport to an external command," but this feature is now disabled by default. As a reminder, we have two injection points: url: the URL of the remote Git repository to clone; folderPath: the destination folder computed from the URL of the Git repository. This is very important in this case as our injected option takes the place of a positional argument: without the second injection point, Git wouldn't have anything to clone from, and the injection wouldn't be exploitable. Finally, if there is any space in the payload, it will be URL-encoded before its interpolation in the command line; it will be easier to try to craft one without spaces: extensions/git/src/git.ts TypeScript let command = ['clone', url.includes(' ') ? encodeURI(url) : url, folderPath, '--progress']; My team and I came up with the following payload: vscode://: the custom scheme registered by Visual Studio Code to the operating system; vscode.git/clone?url=: required to trigger the git.clone command in Visual Studio Code; -u$({open,-a,calculator}): we inject the option -u, equivalent to --upload-pack, to override the command that will be used to communicate with the remote end; :x: this part is required to trick Git into using the transport layer, recognize it as PROTO_LOCAL and invoke the aforementioned upload-pack. Patch Microsoft addressed this issue by improving its validation on the URL of the Git repository to clone as part of the commit c5da533. The URL is parsed using Uri, an internal URI parser, to validate the scheme against a pre-established allow list. The rationale behind this change is that the argument injection bug can only happen if the prefix of the data is fully controlled, which won't be possible if the scheme part of the URL has to be part of this list. diff --- a/extensions/git/src/protocolHandler.ts +++ b/extensions/git/src/protocolHandler.ts @@ -7,6 +7,8 @@ import { UriHandler, Uri, window, Disposable, commands } from 'vscode'; import { dispose } from './util'; import * as querystring from 'querystring'; +const schemes = new Set(['file', 'git', 'http', 'https', 'ssh']); + export class GitProtocolHandler implements UriHandler { private disposables: Disposable[] = []; @@ -26,9 +28,27 @@ export class GitProtocolHandler implements UriHandler { if (!data.url) { console.warn('Failed to open URI:', uri); + return; + } + + if (Array.isArray(data.url) && data.url.length === 0) { + console.warn('Failed to open URI:', uri); + return; + } + + let cloneUri: Uri; + try { + cloneUri = Uri.parse(Array.isArray(data.url) ? data.url[0] : data.url, true); + if (!schemes.has(cloneUri.scheme.toLowerCase())) { + throw new Error('Unsupported scheme.'); + } + } + catch (ex) { + console.warn('Invalid URI:', uri); + return; } - commands.executeCommand('git.clone', data.url); + commands.executeCommand('git.clone', cloneUri.toString(true)); } dispose(): void { This finding was not eligible for a reward from the Microsoft Bug Bounty Program, as only the core is part of the scope; built-in extensions are de facto excluded even if they are enabled by default. This submission still yielded us 40 points for the Microsoft Researcher Recognition Program and got us on the MSRC 2022 Q2 Leaderboard. It is also interesting to note that the Visual Studio Code team started publishing information about security issues on GitHub on top of the usual security bulletin and release notes: the label security is now added to issues, and GitHub security advisories are published. Timeline Date Action 2022-04-05 This issue is reported to Microsoft on their Researcher Portal. 2022-05-06 Microsoft confirms the issue and starts working on a patch. 2022-05-10 The patch is part of the release 1.67.1. Summary In this publication, I demonstrated how a vulnerability in one of the Visual Studio Code URL handlers could lead to the execution of arbitrary commands on the victim's host. The exploitation technique I demonstrated can also be applied to any other argument injection on a git clone invocation. My team and I urge all developers to upgrade their IDE to the latest version and to remain careful when opening foreign links.