DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integration

Integration refers to the process of combining software parts (or subsystems) into one system. An integration framework is a lightweight utility that provides libraries and standardized methods to coordinate messaging among different technologies. As software connects the world in increasingly more complex ways, integration makes it all possible facilitating app-to-app communication. Learn more about this necessity for modern software development by keeping a pulse on the industry topics such as integrated development environments, API best practices, service-oriented architecture, enterprise service buses, communication architectures, integration testing, and more.

icon
Latest Refcards and Trend Reports
Trend Report
Enterprise Application Integration
Enterprise Application Integration
Refcard #249
GraphQL Essentials
GraphQL Essentials
Refcard #303
API Integration Patterns
API Integration Patterns

DZone's Featured Integration Resources

Deploying Java Serverless Functions as AWS Lambda

Deploying Java Serverless Functions as AWS Lambda

By Nicolas Duminil CORE
The AWS console allows the user to create and update cloud infrastructure resources in a user-friendly manner. Despite all the advantages that such a high-level tool might have, using it is repetitive and error-prone. For example, each time we create a Lambda function using the AWS console, we need to repeat the same operations again and again, and, even if these operations are intuitive and easy as graphical widget manipulations, the whole process is time-consuming and laborious. This working mode is convenient for rapid prototyping, but as soon as we have to work on a real project with a relatively large scope and duration, it doesn't meet the team's goals and wishes anymore. In such a case, the preferred solution is IaC (Infrastructure as Code). IaC essentially consists of using a declarative notation in order to specify infrastructure resources. In the case of AWS, this notation expressed as a formalism based on a JSON or YAML syntax is captured in configuration files and submitted to the CloudFormation IaC utility. CloudFormation is a vast topic that couldn't be detailed in a blog ticket. The important point that we need to retain here is that this service is able to process input configuration files and guarantee the creation and the update of the associated AWS cloud infrastructure resources. While the benefits of the CloudFormation IaC approach are obvious, this tool has a reputation for being verbose, unwieldy, and inflexible. Fortunately, AWS Lambda developers have the choice of using SAM, a superset of CloudFormation which includes some special commands and shortcuts aiming at easing the development, testing, and deployment of the Java serverless code. Installing SAM Installing SAM is very simple: one only has to follow the guide. For example, installing it on Ubuntu 22.04 LTS is as simple as shown below: Shell $ sudo apt-get update ... $ sudo apt-get install awscli ... $ aws --version aws-cli/2.9.12 Python/3.9.11 Linux/5.15.0-57-generic exe/x86_64.ubuntu.22 prompt/off Creating AWS Lambda Functions in Java With SAM Now that SAM is installed on your workstation, you can write and deploy your first Java serverless function. Of course, we assume here that your AWS account has been created and that your environment is configured such that you can run AWS CLI commands. Like CloudFormation, SAM is based on the notion of template, which is a YAML-formatted text file that describes an AWS infrastructure. This template file, which named by default is template.yaml, has to be authored manually, such that to be aligned with the SAM template anatomy (complete specifications can be found here). But writing a template.yaml file from scratch is difficult; hence, the idea of automatically generating it. Enters CookieCutter. CookieCutter is an open-source project allowing automatic code generation. It is greatly used in the Python world but here, we'll use it in the Java world. Its modus operandi is very similar to one of the Maven archetypes, in the sense that it is able to automatically generate full Java projects, including but not limited to packages, classes, configuration files, etc. The generation process is highly customizable and is able to replace string occurrences, flagged by placeholders expressed in a dedicated syntax, by values defined in an external JSON formatted file. This GitHub repository provides such a CookieCutter-based generator able to generate a simple but complete Java project, ready to be deployed as an AWS Lambda serverless function. The listing below shows how: Shell $ sam init --location https://github.com/nicolasduminil/sam-template You've downloaded /home/nicolas/.cookiecutters/sam-template before. Is it okay to delete and re-download it? [yes]: project_name [my-project-name]: aws-lambda-simple aws_lambda_resource_name [my-aws-lambda-resource-name]: AwsLambdaSimple java_package_name [fr.simplex_software.aws.lambda.functions]: java_class_name [MyAwsLambdaClassName]: AwsLambdaSimple java_handler_method_name [handleRequest]: maven_group_id [fr.simplex-software.aws.lambda]: maven_artifact_id [my-aws-function]: aws-lambda-simple maven_version [1.0.0-SNAPSHOT]: function_name [AwsLambdaTestFunction]: AwsLambdaSimpleFunction Select architecture: 1 - arm64 2 - x86_64 Choose from 1, 2 [1]: timeout [10]: Select tracing: 1 - Active 2 - Passthrough Choose from 1, 2 [1]: The command sam init above mentions the location of the CookieCutter-based template used to generate a new Java project. This generation process takes the form of a dialog where the utility is asking questions and accepting answers. Each question has default responses and, in order to accept them, the user just needs to type Enter. Everything starts by asking about the project name and we chose aws-lambda-simple. Further information to be entered is: AWS resource name Maven GAV (GroupId, ArtifactId, Version) Java package name Java class name Processor architecture Timeout value Tracing profile As soon as the command terminates, you may open the new project in your preferred IDE and inspect the generated code. Once finished, you may proceed with a first build, as follows: Shell $ cd aws-lambda-simple/ nicolas@nicolas-XPS-13-9360:~/sam-test/aws-lambda-simple$ mvn package [INFO] Scanning for projects... [INFO] [INFO] ----------< fr.simplex-software.aws.lambda:aws-lambda-simple >---------- [INFO] Building aws-lambda-simple 1.0.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ aws-lambda-simple --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/nicolas/sam-test/aws-lambda-simple/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ aws-lambda-simple --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/nicolas/sam-test/aws-lambda-simple/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ aws-lambda-simple --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/nicolas/sam-test/aws-lambda-simple/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ aws-lambda-simple --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ aws-lambda-simple --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ aws-lambda-simple --- [INFO] Building jar: /home/nicolas/sam-test/aws-lambda-simple/target/aws-lambda-simple-1.0.0-SNAPSHOT.jar [INFO] [INFO] --- maven-shade-plugin:3.2.1:shade (default) @ aws-lambda-simple --- [INFO] Replacing /home/nicolas/sam-test/aws-lambda-simple/target/aws-lambda-simple.jar with /home/nicolas/sam-test/aws-lambda-simple/target/aws-lambda-simple-1.0.0-SNAPSHOT-shaded.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.604 s [INFO] Finished at: 2023-01-12T19:09:23+01:00 [INFO] ------------------------------------------------------------------------ Our new Java project has been built and packaged as a JAR (Java ARchive). The generated template.yaml file defines the required AWS cloud infrastructure, as shown below: YAML AWSTemplateFormatVersion: 2010-09-09 Transform: AWS::Serverless-2016-10-31 Description: aws-lambda-simple Resources: AwsLambdaSimple: Type: AWS::Serverless::Function Properties: FunctionName: AwsLambdaSimpleFunction Architectures: - arm64 Runtime: java11 MemorySize: 128 Handler: fr.simplex_software.aws.lambda.functions.AwsLambdaSimple::handleRequest CodeUri: target/aws-lambda-simple.jar Timeout: 10 Tracing: Active This file has been created based on the value entered during the generation process. Things like the AWS template version and the transformation version are constants and should be used as such. All the other elements are known as they mirror the input data. Special consideration has to be given to the CodeUri element which specifies the location of the JAR to be deployed as the Lambda function. It contains the class AwsLambdaSimple below: Java public class AwsLambdaSimple { private static Logger log = Logger.getLogger(AwsLambdaSimple.class.getName()); public String handleRequest (Map<String, String> event) { log.info("*** AwsLambdaSimple.handleRequest: Have received: " + event); return event.entrySet().stream().map(e -> e.getKey() + "->" + e.getValue()).collect(Collectors.joining(",")); } } A Lambda function in Java can be run in the following two modes: A synchronous or RequestResponse mode in which the caller waits for whatever response the Lambda function returns An asynchronous or Event mode in which the caller is responded to without waiting, by the Lambda platform itself, while the function proceeds with the request processing, without returning any further response In both cases, the method handleRequest() above is processing the request, as its name implies. This request is an event implemented as a Map<String, String>. All right! Now our new Java project is generated and while the class AwsLambdaSimple presented above (which will be deployed in fine as an AWS Lambda function) doesn't do much, it is sufficiently complete in order to demonstrate our use case. So let's deploy our cloud infrastructure. But first, we need to create an AWS S3 bucket in order to store in it our Lambda function. The simplest way to do that is shown below: Shell $ aws s3 mb s3://bucket-$$ make_bucket: bucket-18468 Here we just created an S3 bucket having the name of bucket-18468. The AWS S3 buckets are constrained to have a unique name across regions. And since it's difficult to guarantee the uniqueness of a name, we use here the Linux $$ function which generates a random number. Shell sam deploy --s3-bucket bucket-18468 --stack-name simple-lambda-stack --capabilities CAPABILITY_IAM Uploading to 44774b9ed09001e1bb31a3c5d11fa9bb 4031 / 4031 (100.00%) Deploying with following values =============================== Stack name : simple-lambda-stack Region : eu-west-3 Confirm changeset : False Disable rollback : False Deployment s3 bucket : bucket-18468 Capabilities : ["CAPABILITY_IAM"] Parameter overrides : {} Signing Profiles : {} Initiating deployment ===================== Uploading to 3af7fb4a847b2fea07d606a80de2616f.template 555 / 555 (100.00%) Waiting for changeset to be created.. CloudFormation stack changeset ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Operation LogicalResourceId ResourceType Replacement ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Add AwsLambdaSimpleRole AWS::IAM::Role N/A + Add AwsLambdaSimple AWS::Lambda::Function N/A ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Changeset created successfully. arn:aws:cloudformation:eu-west-3:495913029085:changeSet/samcli-deploy1673620369/0495184e-58ca-409c-9554-ee60810fec08 2023-01-13 15:33:00 - Waiting for stack create/update to complete CloudFormation events from stack operations (refresh every 0.5 seconds) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ResourceStatus ResourceType LogicalResourceId ResourceStatusReason ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CREATE_IN_PROGRESS AWS::IAM::Role AwsLambdaSimpleRole - CREATE_IN_PROGRESS AWS::IAM::Role AwsLambdaSimpleRole Resource creation Initiated CREATE_COMPLETE AWS::IAM::Role AwsLambdaSimpleRole - CREATE_IN_PROGRESS AWS::Lambda::Function AwsLambdaSimple - CREATE_IN_PROGRESS AWS::Lambda::Function AwsLambdaSimple Resource creation Initiated CREATE_COMPLETE AWS::Lambda::Function AwsLambdaSimple - CREATE_COMPLETE AWS::CloudFormation::Stack simple-lambda-stack - ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Successfully created/updated stack - simple-lambda-stack in eu-west-3 Our Java class has been successfully deployed as an AWS Lambda function. Let's test it using the two invocation methods presented above. Shell $ aws lambda invoke --function-name AwsLambdaSimpleFunction --payload $(echo "{\"Hello\":\"Dude\"}" | base64) outputfile.txt { "StatusCode": 200, "ExecutedVersion": "$LATEST" } $ cat outputfile.txt "Hello->Dude" The listing above demonstrates the synchronous or RequestResponse invocation. We pass a JSON-formatted payload as the input event and, since its default format is Base64, we need to convert it first. Since the invocation is synchronous, the caller waits for the response which is captured in the file outputfile.txt. The returned status code is HTTP 200, as expected, meaning that the request has been correctly processed. Let's see the asynchronous or Event invocation. Shell $ aws lambda invoke --function-name AwsLambdaSimpleFunction --payload $(echo "{\"Hello\":\"Dude\"}" | base64) --invocation-type Event outputfile.txt { "StatusCode": 202 } This time the --invocation-type is Event and, consequently, the returned status code is HTTP 202, meaning that the request has been accepted but not yet processed. The file output.txt is empty, as there is no result. This concludes our use case showing the AWS Lambda functions deployment in Java via the SAM tool. Don't forget to clean up your environment before leaving by running: Shell $ aws s3 rm --recursive s3://bucket-18468 $ aws s3 rb --force s3://bucket-18468 $ aws cloudformation delete-stack --stack-name simple-lambda-stack Enjoy! More
How To Use Terraform to Provision an AWS EC2 Instance

How To Use Terraform to Provision an AWS EC2 Instance

By Saurabh Dashora CORE
Terraform is a deployment tool that you can use to provision and manage your infrastructure as code. Usually, the infrastructure is cloud-based, but Terraform can manage anything that can be controlled through an API. But why should you consider infrastructure as code (IaC) in the first place? IaC helps avoid some common pitfalls in deployment. Managing your infrastructure using special code opens the door to automation. It also creates a strong release-oriented approach to infrastructure similar to normal application code. In this tutorial, I will go through each step of using Terraform to provision an AWS EC2 instance. 1. How Terraform Works If you have ever deployed software, you would know the importance of repeatability and consistency in your infrastructure setup. You don’t want to mess with environment changes when deploying software on a Friday evening. Terraform makes infrastructure provisioning repeatable and consistent across public, private, and hybrid cloud platforms. The basic principle of Terraform is that you write human-readable configuration code to define your infrastructure. Once you have the code, you ask Terraform to deploy the infrastructure using a single command. The illustration below describes the working of Terraform on a high level. Terraform High-Level The answer is Terraform providers. Providers are basically plugins for Terraform that are made to talk with external APIs. They are written in Golang and distributed through the Terraform registry. A provider acts as the middleman between Terraform and the platform. It handles a bunch of different things such as: Authentication logic Making API requests Managing timeouts and errors Each cloud vendor maintains its own Terraform provider, and at this point, there are hundreds of published providers available on the Terraform registry. You can also write your own Terraform provider. Check out the illustration below. Terraform Providers 2. Terraform AWS EC2 Instance Configuration File Let’s look at how to provision an AWS EC2 instance using Terraform. The below illustration describes the overall process: Terraform EC2 $ mkdir terraform-aws-ec2-demo $ cd terraform-aws-ec2-demo Within the directory, create the main.tf file to write the initial Terraform configuration. See below. provider "aws" { region = "us-west-2" profile = "terraform-user" } resource "aws_instance" "hello_aws" { ami = "ami-0ceecbb0f30a902a6" instance_type = "t2.micro" tags = { Name = "HelloAWS" } } In case you are wondering, the configuration is written using a language known as HCL or Hashicorp Configuration Language. The language is human-readable (even more than JSON) and forms the core of Terraform. The code itself is simple enough to understand. The Terraform AWS Provider Block The first block configures the provider. provider "aws" { region = "us-west-2" profile = "terraform-user" } Providers only have one label: Name. This should be the official name of the provider as published in the Terraform registry. In our example, the value is “aws” for AWS Provider. The AWS Provider is responsible for understanding API interactions and making authenticated requests. You need to configure a provider by passing some inputs to the provider block. As an example, set the region to us-west-2. Also, direct the provider to use the profile for terraform-user. Since AWS APIs need authentication, you will have to create an IAM user, generate credentials from the AWS Console and set those credentials with your local system’s AWS CLI. For this demo, terraform-user is the IAM user I’ve created to manage our Terraform resources. You can read more about setting the AWS credentials. The Terraform AWS Resource Block The next block in the main.tf file describes the resource you want to provision i.e. the EC2 instance. resource "aws_instance" "hello_aws" { ami = "ami-0ceecbb0f30a902a6" instance_type = "t2.micro" tags = { Name = "HelloAWS" } } This EC2 code block is an example of a Terraform resource. Resources are undoubtedly the most important elements in Terraform as they are directly responsible for provisioning the infrastructure. A resource block has two labels. The first label specifies the type of the resource and the second is the name of the resource. For this example, the type of resource is aws_instance and the name is “hello_aws”. The name has no special significance and is only used to reference the resource within the module scope. Of course, the type and name together become a unique resource identifier. Each resource takes some inputs. In this case, the inputs are: AMI ID: This is the AMI id that should be used to create the EC2 instance. In case you are wondering, this is the AMI for Amazon Linux. You can, of course, customize these parameters using Terraform data sources. Instance Type: This is the instance type. I recommend using t2.micro instance that is available in Free Tier. Tags: These are special key-value properties you may want to associate with the instance. Resource also has some outputs that are generated after Terraform has provisioned the necessary infrastructure. In fact, the input arguments also end up as output attributes. However, Terraform also attaches new output attributes such as the id of the created resource. 3. Initializing the Terraform Provider Before Terraform can deploy our EC2 instance, you need to initialize the workspace. What this means is that you have to download and install the AWS Provider binary from the Terraform registry. This is needed at least once for every workspace. To initialize the workspace, run the command terraform init. You should see output like this: Initializing the backend... Initializing provider plugins... - Finding latest version of hashicorp/aws... - Installing hashicorp/aws v4.48.0... - Installed hashicorp/aws v4.48.0 (signed by HashiCorp) Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. Terraform fetches the latest version of the AWS provider and installs it in the workspace. At this point, you will find an additional folder .terraform popup in your workspace. This contains the downloaded binary for the AWS provider. Also, Terraform creates a lock file .terraform.lock.hcl to record the provider selections. It is good practice to include this file in the version control repository so that Terraform can make the same selections by default when you run terraform init in the future. 4. Deploying the AWS EC2 Instance Using Terraform Once the initialization is successful, you are ready to deploy the EC2 instance using Terraform. Execute the terraform apply command and answer “yes” to the confirmation prompt. You should see similar output once the provisioning is complete. aws_instance.hello_aws: Creating... aws_instance.hello_aws: Still creating... [11s elapsed] aws_instance.hello_aws: Still creating... [21s elapsed] aws_instance.hello_aws: Still creating... [31s elapsed] aws_instance.hello_aws: Creation complete after 37s [id=i-033fd8432fb59ff37] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Also, you can check the new instance in the AWS console. The instance is created in the us-west-2a region as specified in the Terraform configuration file. Terraform created EC2 instance At this point, you will also find a file named terraform.tfstate created in your project directory. It is basically a JSON file that describes the resources managed by Terraform. You can view the contents of the file in human-readable format by executing terraform show command. Basically, this file is used by Terraform to keep track of the infrastructure it is managing. Of course, this is a simple instance and we have not deployed anything on the instance. You can also use Terraform to deploy a Nginx webserver on an EC2 instance. 5. Destroying the Terraform AWS EC2 Instance Destroying the infrastructure managed by Terraform is as easy as executing the terraform destroy command. This command also prompts for a confirmation, and if you answer “yes," Terraform goes ahead and destroys the EC2 instance created in the previous section. I recommend doing this if you don’t need the infrastructure to continue running. Conclusion Terraform is a declarative IaC provisioning tool that helps deploy resources onto any public or private cloud. In this post, you saw how easy it makes the process of deploying an EC2 instance to AWS. All you had to do was create a configuration file and pass it to Terraform using terraform apply command. The code for this post is available on GitHub for reference. If you found this post useful, consider sharing it with friends and colleagues. Also, in case of any queries, please write them in the comments section below. More
Debugging Threads and Asynchronous Code
Debugging Threads and Asynchronous Code
By Shai Almog CORE
The Quest for REST
The Quest for REST
By Nicolas Fränkel CORE
Commonly Occurring Errors in Microsoft Graph Integrations and How to Troubleshoot Them (Part 3)
Commonly Occurring Errors in Microsoft Graph Integrations and How to Troubleshoot Them (Part 3)
By Constantin Kwiatkowski
CQRS and MediatR Pattern Implementation Using .NET Core 6 Web API
CQRS and MediatR Pattern Implementation Using .NET Core 6 Web API

In this article, we are going to discuss the working of CQRS and MediatR patterns and step-by-step implementation using .NET Core 6 Web API. Prerequisites Visual Studio 2022 SQL Server .NET Core 6 Introduction of CQRS Pattern CQRS stands for Command and Query Responsibility Segregation and uses to separate read(queries) and write(commands). In that, queries perform read operation, and command performs writes operation like create, update, delete, and return data. As we know, in our application, we mostly use a single data model to read and write data, which will work fine and perform CRUD operations easily. But, when the application becomes vast in that case, our queries return different types of data as an object, so that become hard to manage with different DTO objects. Also, the same model is used to perform a write operation. As a result, the model becomes complex. Also, when we use the same model for both reads and write operations, the security is also hard to manage when the application is large, and the entity might expose data in the wrong context due to the workload on the same model. CQRS helps to decouple operations and make the application more scalable and flexible on large scale. When to Use CQRS We can use Command Query Responsibility Segregation when the application is huge and access the same data in parallel. CQRS helps reduce merge conflicts while performing multiple operations with data. In DDD terminology, if the domain data model is complex and needs to perform many operations on priority, like validations and executing some business logic so in that case, we need the consistency that we will by using CQRS. MediatR MediatR pattern helps to reduce direct dependency between multiple objects and make them collaborative through MediatR. In .NET Core, MediatR provides classes that help to communicate with multiple objects efficiently in a loosely coupled manner. Step-By-Step Implementation Step 1 Create a new application. Step 2 Configure your application. Step 3 Provide additional information. Step 4 Project Structure. Step 5 Install the Following NuGet Packages. C# <Project Sdk="Microsoft.NET.Sdk.Web"> <PropertyGroup> <TargetFramework>net6.0</TargetFramework> <Nullable>disable</Nullable> <ImplicitUsings>enable</ImplicitUsings> </PropertyGroup> <ItemGroup> <PackageReference Include="MediatR.Extensions.Microsoft.DependencyInjection" Version="8.0.0" /> <PackageReference Include="Microsoft.EntityFrameworkCore" Version="6.0.8" /> <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="6.0.8"> <PrivateAssets>all</PrivateAssets> <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> </PackageReference> <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="6.0.8" /> <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="6.0.8"> <PrivateAssets>all</PrivateAssets> <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> </PackageReference> <PackageReference Include="Swashbuckle.AspNetCore" Version="6.2.3" /> </ItemGroup> </Project> Step 6 Create a Student Details class inside the model folder. C# namespace CQRSAndMediatRDemo.Models { public class StudentDetails { public int Id { get; set; } public string StudentName { get; set; } public string StudentEmail { get; set; } public string StudentAddress { get; set; } public int StudentAge { get; set; } } } Step 7 Next, add DbContextClass inside the data folder. C# using CQRSAndMediatRDemo.Models; using Microsoft.EntityFrameworkCore; namespace CQRSAndMediatRDemo.Data { public class DbContextClass : DbContext { protected readonly IConfiguration Configuration; public DbContextClass(IConfiguration configuration) { Configuration = configuration; } protected override void OnConfiguring(DbContextOptionsBuilder options) { options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")); } public DbSet<StudentDetails> Students { get; set; } } } Step 8 Create one student repository and a class related to that. IStudentRepository C# using CQRSAndMediatRDemo.Models; namespace CQRSAndMediatRDemo.Repositories { public interface IStudentRepository { public Task<List<StudentDetails>> GetStudentListAsync(); public Task<StudentDetails> GetStudentByIdAsync(int Id); public Task<StudentDetails> AddStudentAsync(StudentDetails studentDetails); public Task<int> UpdateStudentAsync(StudentDetails studentDetails); public Task<int> DeleteStudentAsync(int Id); } } StudentRepository C# using CQRSAndMediatRDemo.Data; using CQRSAndMediatRDemo.Models; using Microsoft.EntityFrameworkCore; using System; using System.Numerics; namespace CQRSAndMediatRDemo.Repositories { public class StudentRepository : IStudentRepository { private readonly DbContextClass _dbContext; public StudentRepository(DbContextClass dbContext) { _dbContext = dbContext; } public async Task<StudentDetails> AddStudentAsync(StudentDetails studentDetails) { var result = _dbContext.Students.Add(studentDetails); await _dbContext.SaveChangesAsync(); return result.Entity; } public async Task<int> DeleteStudentAsync(int Id) { var filteredData = _dbContext.Students.Where(x => x.Id == Id).FirstOrDefault(); _dbContext.Students.Remove(filteredData); return await _dbContext.SaveChangesAsync(); } public async Task<StudentDetails> GetStudentByIdAsync(int Id) { return await _dbContext.Students.Where(x => x.Id == Id).FirstOrDefaultAsync(); } public async Task<List<StudentDetails>> GetStudentListAsync() { return await _dbContext.Students.ToListAsync(); } public async Task<int> UpdateStudentAsync(StudentDetails studentDetails) { _dbContext.Students.Update(studentDetails); return await _dbContext.SaveChangesAsync(); } } } Step 9 After that, add read queries. GetStudentListQuery C# using CQRSAndMediatRDemo.Models; using MediatR; namespace CQRSAndMediatRDemo.Queries { public class GetStudentListQuery : IRequest<List<StudentDetails>> { } } GetStudentByIdQuery C# using CQRSAndMediatRDemo.Models; using MediatR; namespace CQRSAndMediatRDemo.Queries { public class GetStudentByIdQuery : IRequest<StudentDetails> { public int Id { get; set; } } } Step 10 Next, create different commands. CreateStudentCommand C# using CQRSAndMediatRDemo.Models; using MediatR; namespace CQRSAndMediatRDemo.Commands { public class CreateStudentCommand : IRequest<StudentDetails> { public string StudentName { get; set; } public string StudentEmail { get; set; } public string StudentAddress { get; set; } public int StudentAge { get; set; } public CreateStudentCommand(string studentName, string studentEmail, string studentAddress, int studentAge) { StudentName = studentName; StudentEmail = studentEmail; StudentAddress = studentAddress; StudentAge = studentAge; } } } UpdateStudentCommand C# using MediatR; namespace CQRSAndMediatRDemo.Commands { public class UpdateStudentCommand : IRequest<int> { public int Id { get; set; } public string StudentName { get; set; } public string StudentEmail { get; set; } public string StudentAddress { get; set; } public int StudentAge { get; set; } public UpdateStudentCommand(int id, string studentName, string studentEmail, string studentAddress, int studentAge) { Id = id; StudentName = studentName; StudentEmail = studentEmail; StudentAddress = studentAddress; StudentAge = studentAge; } } } DeleteStudentCommand C# using MediatR; namespace CQRSAndMediatRDemo.Commands { public class DeleteStudentCommand : IRequest<int> { public int Id { get; set; } } } Step 11 Now, add query and command handlers. GetStudentListHandler C# using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Queries; using CQRSAndMediatRDemo.Repositories; using MediatR; using System.Numerics; namespace CQRSAndMediatRDemo.Handlers { public class GetStudentListHandler : IRequestHandler<GetStudentListQuery, List<StudentDetails>> { private readonly IStudentRepository _studentRepository; public GetStudentListHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<List<StudentDetails>> Handle(GetStudentListQuery query, CancellationToken cancellationToken) { return await _studentRepository.GetStudentListAsync(); } } } GetStudentByIdHandler C# using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Queries; using CQRSAndMediatRDemo.Repositories; using MediatR; using System.Numerics; namespace CQRSAndMediatRDemo.Handlers { public class GetStudentByIdHandler : IRequestHandler<GetStudentByIdQuery, StudentDetails> { private readonly IStudentRepository _studentRepository; public GetStudentByIdHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<StudentDetails> Handle(GetStudentByIdQuery query, CancellationToken cancellationToken) { return await _studentRepository.GetStudentByIdAsync(query.Id); } } } CreateStudentHandler C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Repositories; using MediatR; namespace CQRSAndMediatRDemo.Handlers { public class CreateStudentHandler: IRequestHandler<CreateStudentCommand, StudentDetails> { private readonly IStudentRepository _studentRepository; public CreateStudentHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<StudentDetails> Handle(CreateStudentCommand command, CancellationToken cancellationToken) { var studentDetails = new StudentDetails() { StudentName = command.StudentName, StudentEmail = command.StudentEmail, StudentAddress = command.StudentAddress, StudentAge = command.StudentAge }; return await _studentRepository.AddStudentAsync(studentDetails); } } } UpdateStudentHandler C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Repositories; using MediatR; namespace CQRSAndMediatRDemo.Handlers { public class UpdateStudentHandler : IRequestHandler<UpdateStudentCommand, int> { private readonly IStudentRepository _studentRepository; public UpdateStudentHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<int> Handle(UpdateStudentCommand command, CancellationToken cancellationToken) { var studentDetails = await _studentRepository.GetStudentByIdAsync(command.Id); if (studentDetails == null) return default; studentDetails.StudentName = command.StudentName; studentDetails.StudentEmail = command.StudentEmail; studentDetails.StudentAddress = command.StudentAddress; studentDetails.StudentAge = command.StudentAge; return await _studentRepository.UpdateStudentAsync(studentDetails); } } } DeleteStudentHandler C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Repositories; using MediatR; namespace CQRSAndMediatRDemo.Handlers { public class DeleteStudentHandler : IRequestHandler<DeleteStudentCommand, int> { private readonly IStudentRepository _studentRepository; public DeleteStudentHandler(IStudentRepository studentRepository) { _studentRepository = studentRepository; } public async Task<int> Handle(DeleteStudentCommand command, CancellationToken cancellationToken) { var studentDetails = await _studentRepository.GetStudentByIdAsync(command.Id); if (studentDetails == null) return default; return await _studentRepository.DeleteStudentAsync(studentDetails.Id); } } } Step 12 Configure the database connection string inside the appsettings.json file. C# { "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } }, "AllowedHosts": "*", "ConnectionStrings": { "DefaultConnection": "Data Source=DESKTOP-8RL8JOG;Initial Catalog=CQRSAndMediatRDemoDB;User Id=sa;Password=database@1;" } } Step 13 Register a few services inside the program class. C# using CQRSAndMediatRDemo.Data; using CQRSAndMediatRDemo.Repositories; using MediatR; using Microsoft.AspNetCore.Hosting; using System.Reflection; var builder = WebApplication.CreateBuilder(args); // Add services to the container. builder.Services.AddMediatR(Assembly.GetExecutingAssembly()); builder.Services.AddDbContext<DbContextClass>(); builder.Services.AddScoped<IStudentRepository, StudentRepository>(); builder.Services.AddControllers(); // Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); var app = builder.Build(); // Configure the HTTP request pipeline. if (app.Environment.IsDevelopment()) { app.UseSwagger(); app.UseSwaggerUI(); } app.UseHttpsRedirection(); app.UseAuthorization(); app.MapControllers(); app.Run(); Step 14 Next, perform database migration and update commands. add-migration “initial” update-database Step 15 After that, create Students Controller and inject MediatR service inside that to send queries and commands. C# using CQRSAndMediatRDemo.Commands; using CQRSAndMediatRDemo.Models; using CQRSAndMediatRDemo.Queries; using CQRSAndMediatRDemo.Repositories; using MediatR; using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.Mvc; using System; namespace CQRSAndMediatRDemo.Controllers { [Route("api/[controller]")] [ApiController] public class StudentsController : ControllerBase { private readonly IMediator mediator; public StudentsController(IMediator mediator) { this.mediator = mediator; } [HttpGet] public async Task<List<StudentDetails>> GetStudentListAsync() { var studentDetails = await mediator.Send(new GetStudentListQuery()); return studentDetails; } [HttpGet("studentId")] public async Task<StudentDetails> GetStudentByIdAsync(int studentId) { var studentDetails = await mediator.Send(new GetStudentByIdQuery() { Id = studentId }); return studentDetails; } [HttpPost] public async Task<StudentDetails> AddStudentAsync(StudentDetails studentDetails) { var studentDetail = await mediator.Send(new CreateStudentCommand( studentDetails.StudentName, studentDetails.StudentEmail, studentDetails.StudentAddress, studentDetails.StudentAge)); return studentDetail; } [HttpPut] public async Task<int> UpdateStudentAsync(StudentDetails studentDetails) { var isStudentDetailUpdated = await mediator.Send(new UpdateStudentCommand( studentDetails.Id, studentDetails.StudentName, studentDetails.StudentEmail, studentDetails.StudentAddress, studentDetails.StudentAge)); return isStudentDetailUpdated; } [HttpDelete] public async Task<int> DeleteStudentAsync(int Id) { return await mediator.Send(new DeleteStudentCommand() { Id = Id }); } } } Step 16 Finally, run your application and access different endpoints using the Swagger UI. Conclusion Here we discussed the CQRS and MediatR Design Patterns and their purpose and benefits in large-scale applications and step-by-step implementation using .NET Core Web API.

By Jaydeep Patil
7 Awesome Libraries for Java Unit and Integration Testing
7 Awesome Libraries for Java Unit and Integration Testing

Looking to improve your unit and integration tests? I made a short video giving you an overview of 7 libraries that I regularly use when writing any sort of tests in Java, namely: AssertJ Awaitility Mockito Wiser Memoryfilesystem WireMock Testcontainers What’s in the Video? The video gives a short overview of how to use the tools mentioned above and how they work. In order of appearance: AssertJ JUnit comes with its own set of assertions (i.e., assertEquals) that work for simple use cases but are quite cumbersome to work with in more realistic scenarios. AssertJ is a small library giving you a great set of fluent assertions that you can use as a direct replacement for the default assertions. Not only do they work on core Java classes, but you can also use them to write assertions against XML or JSON files, as well as database tables! // basic assertions assertThat(frodo.getName()).isEqualTo("Frodo"); assertThat(frodo).isNotEqualTo(sauron); // chaining string specific assertions assertThat(frodo.getName()).startsWith("Fro") .endsWith("do") .isEqualToIgnoringCase("frodo"); (Note: Source Code from AssertJ) Awaitility Testing asynchronous workflows is always a pain. As soon as you want to make sure that, for example, a message broker received or sent a specific message, you'll run into race condition problems because your local test code executes faster than any asynchronous code ever would. Awaitility to the rescue: it is a small library that lets you write polling assertions, in a synchronous manner! @Test public void updatesCustomerStatus() { // Publish an asynchronous message to a broker (e.g. RabbitMQ): messageBroker.publishMessage(updateCustomerStatusMessage); // Awaitility lets you wait until the asynchronous operation completes: await().atMost(5, SECONDS).until(customerStatusIsUpdated()); ... } (Note: Source Code from Awaitility) Mockito There comes a time in unit testing when you want to make sure to replace parts of your functionality with mocks. Mockito is a battle-tested library to do just that. You can create mocks, configure them, and write a variety of assertions against those mocks. To top it off, Mockito also integrates nicely with a huge array of third-party libraries, from JUnit to Spring Boot. // mock creation List mockedList = mock(List.class); // or even simpler with Mockito 4.10.0+ // List mockedList = mock(); // using mock object - it does not throw any "unexpected interaction" exception mockedList.add("one"); mockedList.clear(); // selective, explicit, highly readable verification verify(mockedList).add("one"); verify(mockedList).clear(); (Note: Source Code from Mockito) Wiser Keeping your code as close to production and not just using mocks for everything is a viable strategy. When you want to send emails, for example, you neither need to completely mock out your email code nor actually send them out via Gmail or Amazon SES. Instead, you can boot up a small, embedded Java SMTP server called Wiser. Wiser wiser = new Wiser(); wiser.setPort(2500); // Default is 25 wiser.start(); Now you can use Java's SMTP API to send emails to Wiser and also ask Wiser to show you what messages it received. for (WiserMessage message : wiser.getMessages()) { String envelopeSender = message.getEnvelopeSender(); String envelopeReceiver = message.getEnvelopeReceiver(); MimeMessage mess = message.getMimeMessage(); // now do something fun! } (Note: Source Code from Wiser on GitHub) Memoryfilesystem If you write a system that heavily relies on files, the question has always been: "How do you test that?" File system access is somewhat slow, and also brittle, especially if you have your developers working on different operating systems. Memoryfilesystem to the rescue! It lets you write tests against a file system that lives completely in memory, but can still simulate OS-specific semantics, from Windows to macOS and Linux. try (FileSystem fileSystem = MemoryFileSystemBuilder.newEmpty().build()) { Path p = fileSystem.getPath("p"); System.out.println(Files.exists(p)); } (Note: Source Code from Memoryfilesystem on GitHub) WireMock How to handle flaky 3rd-party REST services or APIs in your tests? Easy! Use WireMock. It lets you create full-blown mocks of any 3rd-party API out there, with a very simple DSL. You can not only specify the specific responses your mocked API will return, but even go so far as to inject random delays and other unspecified behavior into your server or to do some chaos monkey engineering. // The static DSL will be automatically configured for you stubFor(get("/static-dsl").willReturn(ok())); // Instance DSL can be obtained from the runtime info parameter WireMock wireMock = wmRuntimeInfo.getWireMock(); wireMock.register(get("/instance-dsl").willReturn(ok())); // Info such as port numbers is also available int port = wmRuntimeInfo.getHttpPort(); (Note: Source Code from WireMock) Testcontainers Using mocks or embedded replacements for databases, mail servers, or message queues is all nice and dandy, but nothing beats using the real thing. In comes Testcontainers: a small library that allows you to boot up and shut down any Docker container (and thus software) that you need for your tests. This means your test environment can be as close as possible to your production environment. @Testcontainers class MixedLifecycleTests { // will be shared between test methods @Container private static final MySQLContainer MY_SQL_CONTAINER = new MySQLContainer(); // will be started before and stopped after each test method @Container private PostgreSQLContainer postgresqlContainer = new PostgreSQLContainer() .withDatabaseName("foo") .withUsername("foo") .withPassword("secret"); (Note: Source Code from Testcontainers) Enjoy the video!

By Marco Behler CORE
Kafka Integration With Spring Cloud
Kafka Integration With Spring Cloud

Spring Cloud is a Spring project which aims at providing tools for developers helping them to quickly implement some of the most common design patterns like configuration management, service discovery, circuit breakers, routing, proxy, control bus, one-time tokens, global locks, leadership election, distributed sessions and much more. One of the most interesting Spring Cloud sub-projects is Spring Cloud Streams which provides an annotation-driven framework to build message publishers and subscribers. It supports the most recent messaging platforms like RabbitMQ and Kafka and abstracts away their implementation details. This project is demonstrating Spring Cloud Streams with Kafka platforms. The Kafka Infrastructure In the most authentic DevOps approach, our project is structured such that it uses Docker containers. Our Kafka infrastructure is defined in the docker-compose.yml file, as follows: YAML version: '3.7' services: zookeeper: image: confluentinc/cp-zookeeper:5.3.1 hostname: zookeeper container_name: zookeeper ports: - 2181:2181 environment: ZOOKEEPER_SERVER_ID: 1 ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ZOOKEEPER_INIT_LIMIT: 5 ZOOKEEPER_SYNC_LIMIT: 2 ZOOKEEPER_SERVERS: zookeeper:2888:3888 volumes: - /var/lib/zookeeper:/var/lib/zookeeper kafka: image: confluentinc/cp-kafka:5.3.1 hostname: kafka container_name: kafka-broker ports: - "29092:29092" - "9092:9092" depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/kafka KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 volumes: - /var/lib/kafka:/var/lib/kafka - ./scripts/:/scripts schema-registry: image: confluentinc/cp-schema-registry:5.3.1 container_name: schema-registry depends_on: - zookeeper ports: - 8081:8081 environment: SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181/kafka SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8081" SCHEMA_REGISTRY_HOST_NAME: schema-registry kafka-rest-proxy: image: confluentinc/cp-kafka-rest:5.3.1 hostname: kafka-rest-proxy container_name: kafka-rest-proxy depends_on: - zookeeper - kafka - schema-registry ports: - 8082:8082 environment: KAFKA_REST_HOST_NAME: kafka-rest-proxy KAFKA_REST_BOOTSTRAP_SERVERS: kafka:29092 KAFKA_REST_LISTENERS: "http://0.0.0.0:8082" KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081' KAFKA_REST_CONSUMER_REQUEST_TIMEOUT_MS: 30000 TZ: "${TZ-Europe/Paris}" kafka-topics-ui: image: landoop/kafka-topics-ui:0.9.4 container_name: kafka-ui depends_on: - kafka-rest-proxy ports: - 8000:8000 environment: KAFKA_REST_PROXY_URL: http://kafka-rest-proxy:8082 PROXY: "true" zoonavigator: image: elkozmon/zoonavigator:0.7.1 container_name: zoonavigator depends_on: - zookeeper environment: HTTP_PORT: 9000 AUTO_CONNECT_CONNECTION_STRING: zookeeper:2181 kafka_manager: image: hlebalbau/kafka-manager:stable container_name: kafka-manager ports: - "9000:9000" depends_on: - kafka - zookeeper environment: ZK_HOSTS: "zookeeper:2181" APPLICATION_SECRET: "random-secret" KAFKA_MANAGER_AUTH_ENABLED: "true" KAFKA_MANAGER_USERNAME: username KAFKA_MANAGER_PASSWORD: password command: -Dpidfile.path=/dev/null As this infrastructure might seem quite complex, it is explained below. Zookeeper Kafka is, besides others, a message broker and, like any other message broker, it may be clustered. This means that several Kafka message brokers might be connected such that to provide a distributed messaging environment. ZooKeeper is a centralized service for storing and maintaining configuration and naming information. It provides grouping and distributed synchronization to other services. Kafka uses Apache Zookeeper to maintain the list of brokers that are currently members of a cluster. Every broker has a unique identifier that is either set in the broker configuration file or automatically generated. Every time a broker process starts, it registers itself with its ID in Zookeeper by creating an ephemeral node. Different Kafka components subscribe to the broker's defined path in Zookeeper. The first Docker container in our infrastructure above is then running an instance of the Apache Zookeeper service. The Docker image confluentinc/cp-zookeeper comes from Docker Hub and is provided by Confluent. It exposes the TCP port number 2181 and mounts the /var/lib/zookeeper as a read-write volume. Several environment variables are defined, as documented at DockerHub (Zookeeper). In a real infrastructure, several Zookeeper instances would probably be required but here, for simplicity's sake, we're using only one. Kafka Broker The second piece in our puzzle is the Kafka broker itself. The Docker image confluentinc/cp-kafka:5.3.1 is provided by Confluent as well and the container configuration is self-explanatory. The documentation (cp-Kafka) provides full details. Schema Registry As a messaging and streaming platform, Kafka is used to exchanging information in the form of business objects published by message producers to Kafka topics, to which message consumers are subscribing to retrieve them. Hence, these business objects have to be serialized by the producer and deserialized by the consumer. Kafka includes out-of-the-box serializers/de-serializers for different data types like integers, ByteArraysetc. but they don't cover most use cases. When the data to be exchanged is not in the form of simple strings or integers, a more elaborated serialization/deserialization process is required. This is done using specialized libraries like Avro, Thrift or Protobuf. The preferred way to serialize/deserialize data in Kafka is on the behalf of the Apache Avro library. But whatever the library is, it is based on a so-called serialization/deserialization schema. This is a JSON file describing the serialization/deserialization rules. So, whatever the library is, it requires a way to store this schema. Avro, for example, stores it directly in the binary file hosting the serialized objects, but there is a better way to handle this for Kafka messages. Since locating the serialization/deserialization schema in each serialized file might come with some overhead, the best practices are to use a schema registry for this purpose. The Schema Registry is not part of Apache Kafka but there are several open-source options to choose from. Here we’ll use the Confluent Schema Registry. The idea is to store all the schemas used to write data to Kafka in the registry. Then we simply store the identifier for the schema in the record we produce to Kafka. The consumers can then use the identifier to pull the record out of the schema registry and deserialize the data. The key is that all this work, which consists in storing the schema in the registry and pulling it up when required, is done in the serializers and de-serializers. The code that produces data in Kafka or that consumes data from Kafka simply uses the Avro serializer/de-serializer, without any concern about where the associated schema is stored. The figure below illustrates this process. So, the next Docker container of our infrastructure is the one running the Confluent Schema Registry. Nothing particular here other than that it exposes the TCP port 8081 and that it defines a couple of environment variables, as required by the documentation cp-scheme-registry. Kafka REST Proxy The Kafka REST Proxy is a RESTful interface to a Kafka cluster, making it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients. This is a handy component which is not a part of Kafka itself either, but it belongs to the Confluent Kafka adds-on series. The docker image confluentinc/cp-kafka-rest contains the Confluent REST Proxy for Kafka. Its documentation may be found here: cp-Kafka-rest. The configuration is simple and it doesn't require anything of special. The environment variables defined here are explained in the documentation. To resume, we're configuring the Kafka broker address, the schema-registry one, as well as the REST proxy hostname. An interesting point to be noticed is the listener 0.0.0.0:8082 which is the address of the kafka-topics-ui container, explained below. Kafka Topics UI The Kafka Topics UI is a user interface that interacts with the Kafka REST Proxy to allow browsing Kafka topics, inspecting messages and, more generally, seeing what exactly happens in your Kafka clusters. Hence, the next piece of our puzzle is a Docker container running the image named landoop/kafka-topics-ui which documentation may be found here: Kafka-topics-UI. The configuration is just exposing the TCP port number 8000 and setting the Kafka REST proxy IP address or DNS name and TCP port. Zoo Navigator As we have seen, Kafka clusters are using Apache ZooKeeper in order to persist the required information concerning brokers, nodes, topics, etc. Zoo Navigator is another add-on tool which allows for browsing, in a user-friendly way, the information stored in the ZooKeeper repositories. The associated Docker container is based, as shown, on the elkozmon/zoonavigator image which documentation may be found here: zoonavigator. The configuration exposes the TCP port number 9000 and defines the ZooKeeper server IP address or DNS name and TCP port number. Kafka Manager The last piece of our puzzle is the Kafka Manager. This component is another optional add-on which provides the comfort of a GUI on the behalf of which the most common administration operations on the Kafka brokers and topics may be performed. Of course, all these operations can be done using the Kafka CLI (Command Line Interface), which is a part of the Kafka package, but for those who prefer the click-and-drag approach to the austerity of a CLI, this manager is a nice alternative. This Docker container is based on the image hlebalbau/kafka-manager which documentation may be found here: Kafka-manager-docker. The configuration exposes the TCP port number 9000 and defines the ZooKeeper server IP address (DNS name) and TCP port number, as well as the required authentication credentials. Exercising the Infrastructure To exercise the presented infrastructure, just proceed as follows: Clone the Project From Github: Shell git clone https://github.com/nicolasduminil/kafka-spring-integration.git Build the Project: Shell mvn -DskipTests clean install Check Whether the Docker Containers Are up and Running: Shell docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eabab85946fe landoop/kafka-topics-ui:0.9.4 "/run.sh" 6 seconds ago Up 5 seconds 0.0.0.0:8000->8000/tcp kafka-ui 3861367a32a3 hlebalbau/kafka-manager:stable "/kafka-manager/bin/…" 7 seconds ago Up 5 seconds 0.0.0.0:9000->9000/tcp kafka-manager 2f8456c34e6e confluentinc/cp-kafka-rest:5.3.1 "/etc/confluent/dock…" 7 seconds ago Up 6 seconds 0.0.0.0:8082->8082/tcp kafka-rest-proxy 45ddb2275aab elkozmon/zoonavigator:0.7.1 "./run.sh" 8 seconds ago Up 7 seconds (health: starting) 9000/tcp zoonavigator 969cd4d28c7d confluentinc/cp-kafka:5.3.1 "/etc/confluent/dock…" 8 seconds ago Up 6 seconds 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp kafka-broker b63e9dbaa57b confluentinc/cp-schema-registry:5.3.1 "/etc/confluent/dock…" 8 seconds ago Up 8 seconds 0.0.0.0:8081->8081/tcp schema-registry 03711a4deba8 confluentinc/cp-zookeeper:5.3.1 "/etc/confluent/dock…" 9 seconds ago Up 8 seconds 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper Now, that all our infrastructure seems to be running, let's start using it. Let's create some Kafka topics and publish/subscribe messages to/from them. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti kafka-broker /bin/bash root@kafka:/# cd scripts root@kafka:/scripts# ./kafka-test.sh Created topic test. Topic:test PartitionCount:1 ReplicationFactor:1 Configs: Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1 >>>Test message 1 Test message 2 [2020-04-06 14:10:02,809] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.TimeoutException Processed a total of 2 messages Topic test is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true. root@kafka:/scripts# Here we are first connecting to our Docker container running the Kafka broker. Then, in the scripts subdirectory, there is a shell script named kafka-test.sh. Running this script will create a topic named Test. Once this topic is created, the script will publish two test messages on it, after which these two test messages will be consumed and displayed. Finally, the test topic is removed. Here is the source of the script: Shell root@kafka:/scripts# cat kafka-test.sh kafka-topics --create --zookeeper zookeeper:2181/kafka --replication-factor 1 --partitions 1 --topic test kafka-topics --describe --zookeeper zookeeper:2181/kafka --topic test kafka-console-producer --broker-list localhost:29092 --topic test <<EOF Test message 1 Test message 2 EOF kafka-console-consumer --topic test --from-beginning --timeout-ms 5000 --bootstrap-server localhost:29092 kafka-topics --delete --zookeeper zookeeper:2181/kafka --topic test root@kafka:/scripts# As you may see, Kafka comes out of the box with a CLI having commands like kafka-topics, kafka-console-producer and kafka-console-consumer which allows to create/handle topics and to produce/consume messages. Probe ZooKeeper. Let's try to probe now ZooKeeper by using the ZooKeeper Navigator. Get the The IP address of the zoonavigator container and fire your browser on the TCP port number 9000. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti zoonavigator hostname -I 172.20.0.3 nicolas@kfx:~/workspace/spring-kafka-integration$ open http://172.20.0.3:900 Probe the Schema Registry The next add-on that we need to probe is the Schema Registry. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti schema-registry hostname -I 172.20.0.4 nicolas@kfx:~/workspace/spring-kafka-integration$ open 172.20.0.4:8081 Probe Kafka Topics UI Let's do the same for Kafka Topics UI. Shell nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti kafka-broker /bin/bash root@kafka:/# cd scripts root@kafka:/scripts# ls -al total 16 drwxrwxr-x 2 1000 1000 4096 Apr 6 14:07 . drwxr-xr-x 1 root root 4096 Apr 6 14:08 .. -rwxrwxr-x 1 1000 1000 731 Apr 1 13:39 kafka-test-topics.sh -rwxrwxr-x 1 1000 1000 455 Apr 6 14:07 kafka-test.sh root@kafka:/scripts# ./kafka-test-topics.sh Created topic test1. >>>Created topic test2. >>>Created topic test3. >>> root@kafka:/scripts# exit exit nicolas@kfx:~/workspace/spring-kafka-integration$ docker exec -ti kafka-ui ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 54: eth0@if55: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 02:42:ac:14:00:08 brd ff:ff:ff:ff:ff:ff inet 172.20.0.8/16 brd 172.20.255.255 scope global eth0 valid_lft forever preferred_lft forever nicolas@kfx:~/workspace/spring-kafka-integration$ open http://172.20.0.8:8000 Here we are first connecting again to our Kafka Broker container and we execute the script scripts/kafka-test-topics.sh. This script will create some topics, exactly the same way as we did at #4. We need to do that in order to probe Kafka Topics UI. Once the script is terminated and the topics are created, we get the IP address of the Kafka Topics UI container and then we fire our browser to the Kafka Topics UI URL. The following screen will present to us: Here we can see our test topics named test1, test2 and test3 that was created by the script we ran previously. If you press the button labelled System Topics on the top of the screen, you'll see other two system topics. In total, 5 topics, as shown in the right pane of the screen. You can also see that there is only one Kafka broker and that the Kafka REST Proxy is configured as well. The last add-on to probe is Kafka Manager and this is left as an exercise for the reader. The Spring Cloud Integration Now that all our docker services are up and running, let's look at the Java code. Our project is using Spring Boot Modules and, as such, it is structured is three modules, as follows: A master maven module named spring-kafka-integration of packaging type pom The main module named spring-kafka-app containing the Spring Boot application main class, together with a REST controller allowing to invoke of the application's API. A module named spring-kafka-producer implementing the Kafka messaging production logic. A module named spring-kafka-consumer implementing the Kafka messaging consumption logic. Building and Testing In order to build the project first cd into the main directory and run the maven build, as follows: Shell cd <root-directory> mvn -DskipTests clean install Once the commands above are executed, all the required containers must be up and running, as explained previously. Now you can test the services. In order to perform testing, you need to invoke an API exposed by the application. This may be done in several ways but one of the most convenient ones is through Swagger. For that, you need to start the Spring Boot container, as follows: Shell mvn -DskipTests -pl spring-kafka-app spring-boot:run Once the application starts, going at http://localhost:8080, you'll see the following API: The API shown in the figure above is called paris-data-controller and exposes endpoints allowing retrieval of information concerning public transport in Paris. The API uses a set of web services, made available by Pierre GRIMAUD (https://fr.linkedin.com/in/pgrimaud) under an open-source license. These services, developed in Python and available at https://api-ratp.pierre-grimaud.fr/v4 and they are called by our API. So, you can now exercise the API using the Swagger GUI, by clicking on the "Try it out" button, after having unfolded the operations, by clicking on the arrow on the left. For the purposes of our demo, we only provide one operation, a GET which retrieves the stations available on a given public transport line. Two parameters are required, the type of transport, for example, SUBWAY, and the line id, in our case 8, for the subway line M8. Here is what your test should look like: Clicking on the Execute button the test will be run and you'll get the following result: Now, looking at your shell screen where you started the Spring Boot application, you'll see the listing below: This listing shows that our Swagger GUI invokes our API by making a GET request using the URI /paris-data/destinations/SUBWAY/8. Our API will, in turn, make a GET request at the endpoint https://api-ratp.pierre-grimaud.fr/v4/destinations/metros/8 to retrieve the required data, i.e. the subway stations on the M8 line, which are Pointe du Lac, platform A and Ballard, platform R. But more interesting is that, once the remote web service endpoint is invoked and the result is returned, this return will be published on a Kafka topic, as shown in the log by the following message: Shell ### Have got GetAllDestinationsResponse(result=Result(destinations=[Destination(stationName=Pointe du Lac, platformId=A), Destination(stationName=Balard, platformId=R)]), metadata=Metadata(call=GET /destinations/metros/8, date=2020-04-15T14:53:47+02:00, version=4.0)) This message is further consumed as shown below: Shell ### KafkaConsumer.doConsume(): We got a message GetAllDestinationsResponse(result=Result(destinations=[Destination(stationName=Pointe du Lac, platformId=A), Destination(stationName=Balard, platformId=R)]), metadata=Metadata(call=GET /destinations/metros/8, date=2020-04-15T14:53:47+02:00, version=4.0)) These messages are displayed by the Kafka Producer and, respectively, the Kafka Consumer, showing that the message has been successfully sent and received. You can further analyze what exactly happened by using the kafka-topics-ui service, as shown previously. Hence, going at http://192.168.96.6:8000, we'll see a Kafka topic named parisData having the following content: Here we can see our message that has been produced to the topic parisData and further consumed. Show Me the Code! We just have demonstrated a Spring Boot application using Spring Cloud Streams such that to produce and consume messages to/from Kafka topics. But how everything works here? Let's look at the code. First, the main class is annotated as follows: Java @EnableBinding({Source.class, Sink.class}) This annotation has the effect of binding the Spring Cloud Stream framework to the Kafka messaging system. This binding operation is performed on the behalf of a communication channel. Spring Cloud Stream makes available two standard channels, named Source and, respectively, Sink, the first aiming at publishing and the last at subscribing to messages. Of course, channels may and shall be customized, instead of using the Spring Cloud Stream standard classes, which only cover a limited diversity of use cases. But for simplicity's sake, we choose here to use a more basic and out-of-the-box solution instead of a more specific one. The following listing shows the KafkaProducer class. Java @Component @Slf4j public class KafkaProducer { private Source source; @Autowired public KafkaProducer(Source source) { this.source = source; } public void publishKafkaMessage(GetAllDestinationsResponse msg) { log.debug("### KafkaProducer.publisKafkaMessage(): Sending message to Kafka topic"); source.output().send(MessageBuilder.withPayload(msg).build()); } } That's all. The only thing you need here is to inject an instance of the Source which is used further, in the method publishKafkaMessage() to produce messages. The KafkaConsumer class is very simple as well: Java @Component @Slf4j public class KafkaConsumer { @StreamListener(Sink.INPUT) public void doConsume(@Payload GetAllDestinationsResponse msg) { log.debug ("### KafkaConsumer.doConsume(): We got a message \n\t{}", msg); } } The doConsume() method needs only to be annotated with the @StreamListener annotation. The class Sink is configured such that to have an input channel, named INPUT. Then using this very simple construct, the method doConsume() is listening on the input channel of the Sink service and, whenever a message is received, it will get executed. In our case, the execution is simple as it just logs a message in the log file. The code above is everything you need to implement complex Kafka messaging. However, at this point, things might seem a bit like magic because we didn't show anything concerning the Kafka brokers, the topics, etc. There is another good news as handling all these details could be done automatically, only based on a couple of properties, as shown below in the application.properties file: Java spring.application.name=spring-kafka-integration spring.cloud.stream.kafka.binder.brokers=192.168.96.3:9092 spring.cloud.stream.bindings.input.destination=parisData spring.cloud.stream.bindings.input.contentType=application/json spring.cloud.stream.bindings.output.destination=parisData spring.cloud.stream.bindings.output.contentType=application/json These properties are doing the mapping between the Source and Sink classes of Spring Cloud Streams and a Kafka broker and topic. As you can see, the Kafka broker IP address and TCP port are configured, as well as the Kafka topic name associated with the input and output channels. Using this simple use case you can implement services communicating with each other in an asynchronous manner, using messaging. Spring Cloud Stream acts as a middleman for the services while the message broker is used as an abstraction layer over the messaging system. Enjoy! Shell git clone https://github.com/nicolasduminil/kafka-spring-integration.git

By Nicolas Duminil CORE
Experience the Future of Communication With GPT-3: Consume GPT-3 API Through MuleSoft
Experience the Future of Communication With GPT-3: Consume GPT-3 API Through MuleSoft

ChatGPT It is a chatbot that uses the GPT-3 (Generative Pre-trained Transformer 3) language model developed by OpenAI to generate human-like responses to user input. The chatbot is designed to be able to carry on conversations with users in a natural, conversational manner, and can be used for a variety of purposes such as customer service, online support, and virtual assistance. OpenAI API OpenAI offers a number of Application Programming Interfaces (APIs) that allow developers to access the company's powerful AI models and use them in their own projects. These APIs provide access to a wide range of capabilities, including natural language processing, computer vision, and robotics. These APIs are accessed via HTTP requests, which can be sent in a variety of programming languages, including Python, Java, and MuleSoft. It also provides client libraries for several popular programming languages to make it easy for developers to get started. Most interestingly, you can train the model on your own dataset and make it work best on a specific domain. Fine-tuning the model will make it more accurate, perform better and give you the best results. MuleSoft It is a company that provides integration software for connecting applications, data, and devices. Its products include a platform for building APIs (Application Programming Interfaces) and integrations, as well as tools for managing and deploying those APIs and integrations. MuleSoft’s products are designed to help organizations connect their systems and data, enabling them to more easily share information and automate business processes. Steps To Call GPT-3 model API in MuleSoft Account: Create an account using your email id or continue with Google or Microsoft account: Authentication: The OpenAI API uses API keys for authentication. Click API Keys to generate the API key. Do not share your API key with others, or expose it in the browser or other client-side code. In order to protect the security of your account, OpenAI may also automatically rotate any API key that we’ve found has leaked publicly. (Source: documentation) The OpenAI API is powered by a family of models with different capabilities and price points. The highly efficient GPT-3 model is categorized into four models based on the power level suitable to do their task. Model Description Max Request (Tokens Training Data (Upto) text-davinci-003 Most capable GPT-3 model. Perform any task w.r.t other models with higher quality, longer output, and better instruction-following 4000 Jun-21 text-curie-001 Very capable, but faster and lower cost than Davinci. 2048 Jun-19 text-babbage-001 Very fast and lower cost, Perform straight forward task 2048 Jul-19 text-ada-001 Perform simple task 2048 Aug-19 Code snippet to call GPT-3 model (OpenAI API) in Mule application: XML <?xml version="1.0" encoding="UTF-8"?> <mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd"> <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="e5a1354b-1cf2-4963-a89f-36d035c95045" > <http:listener-connection host="0.0.0.0" port="8091" /> </http:listener-config> <flow name="chatgptdemoFlow" doc:id="b5747310-6c6d-4e1a-8bab-6fdfb1d6db3d" > <http:listener doc:name="Listener" doc:id="1819cccd-1751-4e9e-8e71-92a7c187ad8c" config-ref="HTTP_Listener_config" path="completions"/> <logger level="INFO" doc:name="Logger" doc:id="3049e8f0-bbbb-484f-bf19-ab4eb4d83cba" message="Calling completaion API of openAI"/> <http:request method="POST" doc:name="Calling completaions API" doc:id="390d1af1-de73-4640-b92c-4eaed6ff70d4" url="https://api.openai.com/v1/completions?oauth_consumer_key&oauth_token&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673007532&oauth_nonce=WKkU9q&oauth_version=1.0&oauth_signature=RXuuOb4jqCef9sRbTmhSfRwXg4I="> <http:headers ><![CDATA[#[output application/java --- { "Authorization" : "Bearer sk-***", "Content-Type" : "application/json" }]]]></http:headers> </http:request> <ee:transform doc:name="Parse Response" doc:id="061cb180-48c9-428e-87aa-f4f55a39a6f2" > <ee:message > <ee:set-payload ><![CDATA[%dw 2.0 import * from dw::core::Arrays output application/json --- (((payload.choices[0].text splitBy ".") partition ((item) -> item startsWith "\n" ))[1] ) map "$$":$]]></ee:set-payload> </ee:message> </ee:transform> </flow> </mule> Make a call to the Mule application through the API client. For example, I am using Postman. Request payload: { "model": "text-davinci-003", "prompt": "Create a list of 5 advantages of MuleSoft:", "max_tokens": 150 } model: The OpenAI API is powered by a family of models with different capabilities and price points. prompt: The prompt(s) to generate completions for, encoded as a string, array of strings, array of tokens, or array of token arrays. max_tokens: The maximum number of tokens to generate in the completion. For more details, refer to the API reference. Response payload [ { "0": " Easy Integration: Mulesoft provides easy integration with existing systems and technologies, making it faster and easier to start projects or add new services or technologies to systems" }, { "1": " Flexible Architecture: Mulesoft offers a highly configurable and flexible architecture that allows users to quickly and easily customize their solutions and make changes when needed" }, { "2": " High Performance: Mulesoft's rapid response times and high throughputs make it ideal for mission critical applications" }, { "3": " Cloud Ready: Mulesoft supports cloud friendly approaches such as microservices, containers and serverless integration architectures" }, { "4": " Efficient Development Cycles: The Mulesoft Anypoint platform includes a range of tools and services that speed up and streamline development cycles, helping to reduce the time and effort associated with creating applications" } ] Where GPT-3 Could Potentially Be Used Content Creation The API can be utilized to generate written materials and translate them from one language to another. Software Code Generation The API can be used to generate software code and simplify complicated code, making it more comprehensible for new developers. Sentiment Analysis The API can be used to determine the sentiment of text, allowing businesses and organizations to understand the public's perception of their products, services, or brand. Complex Computation The API can assist in large data processing and handle complex calculations efficiently. Limitations Like all natural language processing (NLP) systems, GPT-3 has limitations in its ability to understand and respond to user inputs. Here are a few potential limitations of GPT-3: Reliability: Chatbots may not always produce acceptable outputs and determining the cause can be difficult. Complex queries may not be understood or responded to appropriately. Interpretability: The chatbot may not recognize all variations of user input, leading to misunderstandings or inappropriate responses if not trained on diverse data. Accuracy: Chatbot, being a machine learning model, can make mistakes. Regular review and testing are needed to ensure proper functioning. Efficiency: GPT-3's test-time sample efficiency is close to humans, but pre-training still involves a large amount of text. Overall, GPT-3 is a useful tool that can improve communication and productivity, but it is important to be aware of its limitations and use it appropriately. Conclusion As chatbots continue to advance, they have the potential to significantly improve how we communicate and interact with technology. This is just an example of the various innovative approaches that are being developed to enhance chatbot performance and capabilities. As we continue to explore the potential of GPT-3 and other language generation tools, it’s important that we remain authentic and responsible in our use of these tools. This means being transparent about their role in the writing process and ensuring that the resulting text is of high quality and accuracy. References OpenAI website Beta OpenAPI website

By Mukesh Thakur CORE
Trends Shaping “Post-Pandemic” Enterprise Integration – A Practitioner’s View
Trends Shaping “Post-Pandemic” Enterprise Integration – A Practitioner’s View

The pandemic has wired global enterprises to become thriftier and value-focused with a view to get their technological ecosystem more efficient, lean, and meaningful, which can cater to the larger strategic goal of being effective in addressing user and customer experience. Basically, the world has moved notches towards being an “Experience Economy” than a “Feature Driven Economy.” This movement is being felt in every stratum of an enterprise’s architecture, including the Integration layer. With more and more enterprises migrating their workloads to the cloud, not just one, but multiple of them, and with each of the cloud “ecosystems” having their own native integration solutions (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO, etc.), it becomes imperative for enterprises to not only utilize the native integrations for enhanced service efficiency but to also have an Enterprise Integration Platform(EiPaaS) in the center that can be the conduit for cross-ecosystem interoperability. Likewise, as the strategic prerogatives tend more and more towards “experience,” reusability and scalability of existing APIs and integrations assume greater importance. This apart, the ability of an enterprise to create a Center for Enablement (C4E) that can be business and domain-focused, as opposed to being technology-focused (as was the situation earlier), is gaining a lot of currency in these times. The whole idea is to keep things simple. If any API is working in a certain functional area, the same should be made available to other similar functional areas across geos to drive larger RoI for the enterprises. The third trend being observed is the ability to bake in native “observability” into the Integrations so that real-time operational monitoring and reporting can take place at the level of integrations themselves. This will entail capturing key metrics around API requests, API status, Integration Compliance, API latency, etc., and the ability to draw thresholds to get proactive alerts that can enable the Site Reliability Engineers to take pre-emptive measures towards service continuance and resiliency. Figure 1: Emerging trends in the Enterprise Integration space These trends capture the spirit of co-existence (use the best-of-breed solutions), the intention to scale and replicate than build and build more, and finally, the ability to track, measure, alert, and act based on insights derived from native observability functionalities. Thus efficiency, scalability, and observability are the three key decision markers for enterprises today while evaluating Integration solutions. Let’s look at each of these three trends in detail. 1. Co-Existence of Ecosystem Integration With EiPaaS In a multi-cloud world, where enterprises are parking their workloads in multiple cloud environments while maintaining their most-essential, confidential, and risky workloads on-premises, it is a scenario that calls for various types of integrations – between on-premises and cloud, between cloud and on-premises, between cloud and cloud and between applications on-premises. Several components, applications, and systems need to interact across these environments to ensure seamless data integration, due process orchestration, and rightful end-user experience. Often, the decision-makers in an enterprise become spoiled for choice and hence confused about what should be the “go to” Integration solution that can orchestrate value across the many environments. Well, there isn’t any silver bullet solution to this quandary. But what should guide the enterprises is using the “best of the breed” approach where they leverage the native integrations of a given ecosystem (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO, etc.) for integrating all applications located in that ecosystem. Therefore, all applications stored and run out of the Azure ecosystem can leverage Azure APIs and Azure Integrations. Likewise, clients leveraging SAP or Oracle in a heavy way for their ERP needs will do well to leverage SAP PI/PO or Oracle Cloud Integrations for their respective applications. This will not only drive better runtime efficiency but also allow enterprises to derive maximum value from that ecosystem. The above solves one side of the problem. So, each of our ecosystem applications is well integrated within that ecosystem by using the native integration functionalities of that ecosystem. Within those ecosystem siloes, we have well-oiled, perfectly synced machinery. But come out of these ecosystem bubbles, and you may see distinct, siloed, nonintegrated ecosystems that are struggling to interact with each other. While most ecosystem platforms have “connectors” these days to connect to another ecosystem but imagine a customer with 10-12 different ecosystems within the enterprise, and there could be 120+ connector interactions outside of these ecosystems that would make the enterprise stare at a spaghetti set up, drawing them back to the same problem that they want to get out of. Enter Enterprise Integration Platform that becomes a one-stop orchestrator of all cross-ecosystem interactions. All connectors from each ecosystem meet the common EiPaaS hub, which will be capable of reading, translating, converting, and exposing messages in the target ecosystem-friendly language and hence becomes a common point of interaction, as shown below. Figure 2: From a spaghetti “each connector to another” pattern to an “all to one iPaaS” pattern The above solves the second problem. Therefore, we have a “best of the breed” approach where an enterprise can use all “native” integration capabilities that the ecosystem provides. In contrast, we have an EiPaaS Integration Hub, which connects the dots between the ecosystems. Below is an illustration that shows how “MuleSoft,” a leading Enterprise Integration platform, becomes a central integrating hub that brings together all ecosystems and on-premises systems. In contrast, the ecosystems themselves integrate the applications in them using their native functionalities. Figure 3: Sample: “Co-existence” of iPaaS platform and ecosystem integration platforms Enterprises should respect each platform for its own benefits, leverage native integrations for the application ecosystem, and leverage enterprise integration tools for integrations outside the application ecosystem. The technologies shown in the figure above are for illustration only. 2. Scalability of Integration Best Practices With an “All-Encompassing” C4E (Centre for Enablement) “All-encompassing” is the operative term here for a reason. Enterprises today use different integration technologies for different use cases. Different units and functions use various integration technologies too, as per their need, expertise, and comfort factor. With the empowerment of Citizen Technologists, low code and no code integration tools like Workato, Tray, Zapier, etc. are also sitting in the overall enterprise landscape, alongside more strategic platforms like MuleSoft, Boomi, TIBCO, etc., which also co-exist as seen in the trend above, alongside ecosystem integration functionalities. Thus, suddenly, an enterprise looks at a smorgasbord of technologies, each with its niche requirement and each being used by a separate team or a group of teams! The challenge is how to govern a wide swathe of technologies and teams using them and, more importantly, how to make the integrations reusable, replicable, and scalable across geos and units for similar use cases. Therein lies the value. Many questions and options arise. For example, should a “decentralized model” exist where we have one CoE governing each business unit? Or one CoE governing all the work being done using an integration technology? Or should a “centralized model” exist where one common CoE orchestrates and governs the work across all business units or technologies? Or should there be a “hybrid model” where one Hub CoE interacts with a few “Spoke” CoEs in a hub and spoke” model, and the “spoke CoEs” in turn support a group of business units? There are pros and cons to each of these models. For example, the “decentralized model” is more suited for large enterprises with independent business units. But lack of consistency across the units can breed a shadow IT culture. Likewise, the “centralized model” is more suitable for small and medium enterprises and start-ups which have a small central IT running the show. Finally, the hybrid model is more suited for large enterprises that encourage autonomous business units, which run in a product organization model and want to be self-sufficient with a centralized, horizontal CoE for any governance and utilities support. While each of these options is germane, one factor that can truly accommodate for all these options to co-exist is the provisioning of an “all-encompassing” C4E (Center for Enablement) that is truly “extended” and “accommodates” all teams, technologies, and practices under its ambit, governs them, supplies the enterprises with common standards of working, best practices enabling business divisions to build and drive the consumption of assets successfully, enabling speed and agility. It allows the business and IT teams to shift from a production-based to a production-and-consumption-based delivery model. This production and consumption model fuels reusability and scalability. The C4E is a broader concept than CoE. While the CoEs develop, run, and manage APIs, their operational elements and are a container of functional expertise in a centralized manner, C4Es are enablers, capability builders, decentralized mechanisms of governance that bring in best practices, reusable templates, evangelism, and community building. In addition, they are domain focused and make the business units self-sufficient, independent, and scalable. 3. Observability Baked Into Integrations for Reactive Knowhow and Proactive Alerts Observability is a comprehensive set of capabilities around logging, analytics, monitoring, troubleshooting, measuring platforms, runtimes, and applications. Observability plays a decisive role in providing the real-time performance of a system. Lacking observability in a system would be like flying an aircraft without a fuel indicator. Observability is not only shifting left but also diving deep into each layer of the enterprise architecture. From the systems to applications to the processes to the integrations between them, enterprises expect observability and monitoring mechanisms to be baked into each of these elements so that not only is the performance reported but with the application of AI and by drawing clear performance thresholds, proactive alerts can be sounded out to the SREs, thus ensuring service resiliency and continuance. In this regard, most Integration platforms today come laced with out-of-the-box (OOB) support for many of the key observables. However, what can be observed and monitored also depends on the deployment option- Cloud, On-Prem, or Hybrid. Some of the key observables around Integration that the OOB functionalities can observe and report are: Figure 4: Key Observables in an Integration Solution There are various third-party observability tools in the market that fill the vacuum left by the OOB functionalities of the Integration platforms. For example, Splunk, ELK, and SumoLogic are commonly used for log aggregation and analysis, while the likes of Jaeger, Prometheus, Datadog, etc., are popular for their tracing capabilities. However, tools alone do not solve the problem of observability. Having an experienced team of reliability engineers who can interpret metrics and act proactively will be key to success. As they say, it is not data but the action taken on the insights derived from the data which determines true success. Therefore, partnering with the right service partner will be key to Enterprise Integration success. Conclusion The Enterprise Integration market is in flux post-pandemic as effectiveness, efficiency, cost optimization, and performance precision are all expected by enterprises in their Integration solution. These are seemingly contrarian demands, but a balanced solution that can ensure the co-existence of diverse solutions for diverse use cases, with an ability to scale and a capability to observe and report key metrics, is key to enterprise integration success. In addition, while the technologies are available in the wider market, a capable service partner with experience in deploying, managing, orchestrating, scaling, and bringing about governance support will be key to Integration strategy success.

By Ved Prakash
API Design Patterns Review
API Design Patterns Review

This review is about API Design Patterns by JJ Geewax from Manning. I already mentioned how I'm trying to get up to speed in the API world:reading books, viewing relevant YouTube videos, and reading relevant IETF RFCs. Facts 30 chapters, $35.00 The author is a Principal Software Engineer at Google Chapters Introduction Design principles Naming Resource scope and hierarchy Data types and defaults Fundamentals Resource identification: How to identify resources in an API Standard methods: The set of standard methods for use in resource-oriented APIs Partial updates and retrievals: How to interact with portions of resources Custom methods: Using custom (non-standard) methods in resource-oriented APIs Long-running operations: How to handle methods that are not instantaneous Rerunnable jobs: Running repeated custom functionality in an API Resource relationships Singleton sub-resources: Isolating portions of resource data Cross references: How to reference other resources in an API Association resources: How to manage many-to-many relationships with metadata Add and remove custom methods: How to manage many-to-many relationships without metadata Polymorphism: Designing resources with dynamically-typed attributes Collective operations Copy and move: Duplicating and relocating resources in an API Batch operations: Extending methods to apply to groups of resources atomically Criteria-based deletion: Deleting multiple resources based on a set of filter criteria Anonymous writes: Ingesting unaddressable data into an API Pagination: Consuming large amounts of data in bite-sized chunks Filtering: Limiting result sets according to a user-specified filter Importing and exporting: Moving data into or out of an API by interacting directly with a storage system Safety and Security Versioning and compatibility: Defining compatibility and strategies for versioning APIs Soft deletion: Moving resources to the "API recycle bin" Request deduplication: Preventing duplicate work due to network interruptions in APIs Request validation: Allowing API methods to be called in "safe mode" Resource revisions: Tracking resource change history Request retrial: Algorithms for safely retrying API requests Request authentication: Verifying that requests are authentic and untampered with Each design pattern chapter follows the same structure: Motivation: what problem solves the pattern Overview: a short description of the pattern Implementation: an in-depth explanation of the pattern. It's structured into different subsections. Trade-offs: patterns have strong and weak points; this section describes the latter Exercises: a list of questions to verify that one has understood the pattern Pros and Cons Let's start with the good sides: As I mentioned above, the structure of each chapter dedicated to a design pattern is repetitive. It makes the chapter easy to consume, as you know exactly what to expect. In general, I read my technical books just before going to sleep because I'm pretty busy during the day. Most books have long chapters, requiring me to stop mid-chapter when I start to fall asleep. When you start again, you need to get back a few pages to get the context back. The length of a chapter on API Design Patterns is ideal: neither too long nor too short. The Design Principles section starts from the basics. You don't need to be an expert on API to benefit from the book. I was not; I hope that I'm more seasoned by now. I was a bit dubious at first about the Exercises section of each chapter, for it didn't provide any solution. However, I came to realize it activates the active recall mechanism: instead of passively reading, actively recall what you learned in answering questions. It improves the memorization of what was learned. As an additional benefit, you can learn in a group, compare your answers and eventually debate them. Now, I've some critiques as well: Some patterns are directly taken from Google's API Improvement Proposals. It's not a problem per se, but when it's the case, there's no discussion at all about possible alternatives. For example, the chapter on custom methods describes how to handle actions that don't map precisely to an HTTP verb: a bank transfer is such an action because it changes two resources, the "from" and the "to" accounts.The proposed Google AIP is for the HTTP URI to use a : character followed by the custom verb, e.g., /accounts/123:transfer. That's an exciting proposal that solves the lack of mapping issue. But there are no proposed alternatives nor any motivation for why it should be this way. As an engineer, I can hardly accept implementing a solution with such far-reaching consequences without being provided with other alternatives with their pros and cons. Last but not least, the book doesn't mention any relevant RFC or IETF draft. Chapter 26 describes how to manage request deduplication, the fact that one may need to send the same non-idempotent request repeatedly without being afraid of ill side effects. The proposed solution is good: the client should use a unique key, and if the server gets the same key again, it should discard the request.It's precisely what the IETF draft describes: The Idempotency-Key HTTP Header Field. Still, there's no mention of this draft, giving the feeling that the book is disconnected from its ecosystem. Author's Replies For once, I was already in touch with the author. I offered him an opportunity to review the post. Since his answers are open, I decided to publish them with his permission: Why isn't there more discussion about alternatives? I think you're right — and I actually had quite a bit of discussion of the alternatives in the original manuscript, and I ended up chopping them out. One reason was that my editor wanted to keep chapters reasonably sized, and my internal debates and explanations of why one option was better or worse than another was adding less value than "here's the way to do it." The other reason was that I "should be opinionated." If this were a textbook for a class on exploring API design I could weigh all the sides and put together a proper debate on the pros and cons of the different options, but in most cases there turned out to be a very good option that we've tried out and seen work really well over the course of 5+ years (e.g., : for custom methods). In other cases we actively didn't have that and the chapter is a debate showing the different alternatives (e.g., Versioning). If I could do a 600-700 pages, I think it would have this for you. Why aren't there more references to IETF standards? This is a glaring oversight on my part. There are a lot of RFCs and I do mention some (e.g., RFC-6902 in Partial Updates, etc), but I haven't pulled in enough from that standards body and it's a mistake. If we do a 2nd edition, this will be at the top of the list. Conclusion Because of a couple of cons I mentioned, API Design Patterns falls short of being a reference book. Nonetheless, it's a great book that I recommend to any developer entering the world of APIs or even one with more experience to round up their knowledge.

By Nicolas Fränkel CORE
How To Build a Node.js API Proxy Using http-proxy-middleware
How To Build a Node.js API Proxy Using http-proxy-middleware

A proxy is something that acts on behalf of something else. Your best friend giving your attendance in the boring lecture you bunked during college is a real-life example of proxying. When it comes to API development, a proxy is an interface between the client and the API server. The job of the interface is to proxy incoming requests to the real server. The use of proxy But why might you need an API proxy in the first place? It could be possible that the real API server is external to your organization and unstable. A proxy can provide a more stable interface to the client The response from the API server might not be compatible with the client’s expectations and you want to modify the response in some form (for example, converting XML to JSON). The real API server may be a temporary arrangement and you don’t want the clients to get impacted by any future changes There are several uses of API proxy depending on the situation. In this post, you will learn how to build a Node.js API Proxy using the http-proxy-middleware package. Just to make things clear, API Proxy is different from a Forward Proxy. 1. Node.js API Proxy Project Setup First, you need to initialize the project by executing the below command in a project directory: $ npm init -y This will generate a basic package.json file with meta-data information about the project such as name, version, author and scripts. Next, install a couple of packages for developing the Node.js API proxy. $ npm install --save express http-proxy-middleware express is a minimalistic web framework you can use to build API endpoints. http-proxy-middleware is a simple Node.js package to create an API proxy After the package installation, define a start command for the project within the package.json file. You can use this command to start the application. Your project’s package.json should look similar to the below example. { "name": "express-proxy-demo", "version": "1.0.0", "description": "Demo Application for Proxy Implementation in Node.js", "main": "index.js", "scripts": { "start": "node index.js" }, "author": "Saurabh Dashora", "license": "ISC", "dependencies": { "express": "^4.18.2", "http-proxy-middleware": "^2.0.6" } } 2. Creating a Node.js Proxy Using http-proxy-middleware Time to create the actual application. The example application will proxy incoming requests to an API hosted elsewhere. For demonstration purposes, I recommend using the fake APIs hosted at JSONPlaceholder. See the below illustration: Node.js Proxy Setup Check the below code from the index.js file that contains the logic for proxying requests. const express = require('express'); const { createProxyMiddleware } = require('http-proxy-middleware'); const app = express(); const PORT = 3000; const HOST = "localhost"; const API_URL = "<https://jsonplaceholder.typicode.com>"; app.get("/status", (req, res, next) => { res.send('This is a proxy service'); }); const proxyOptions = { target: API_URL, changeOrigin: true, pathRewrite: { [`^/api/posts`]: '/posts', }, } const proxy = createProxyMiddleware(proxyOptions); app.use('/api/posts', proxy) app.listen(PORT, HOST, () => { console.log(`Proxy Started at ${HOST}:${PORT}`) }); Let’s understand each step in the above program: Step 1: The first segment of the code contains the import statements for express and http-proxy-middleware. Step 2: The next statement creates an application instance using the call to express() function followed by declaring a few important constants such as PORT, HOST and API_URL. Step 3: Implement an endpoint /status to describe the role of the application. This endpoint has nothing to do with proxying requests and provides a way to test our application. Step 4: Next, declare an object proxyOptions. This is a configuration object for our API proxy. It contains a few important properties target - It defines the target host where you want to proxy requests. In our case, this is the [<https://jsonplaceholder.typicode.com>](<https://jsonplaceholder.typicode.com>) changeOrigin - This is set to true since we are proxying to a different origin. pathRewrite - This is a very important property where you define the rules for rewriting the path. For example, the expression [^/api/posts]: '/posts' routes all incoming requests direct at /api/posts to just /posts. In other words, this will remove the /api prefix from the path. Step 5: After declaring the configuration object, create the proxy object by calling createProxyMiddleware() function with the proxyOptions object as input. Step 6: Next, create a request handler for the path /api/posts and pass the proxy object as handler for the incoming request. Step 7: At the very end, start the Node.js API Proxy server to listen on the port and host already declared earlier. You can start the application using the command npm run start. > express-proxy-demo@1.0.0 start > node index.js [HPM] Proxy created: / -> <https://jsonplaceholder.typicode.com> [HPM] Proxy rewrite rule created: "^/api/posts" ~> "/posts" Proxy Started at localhost:3000 Messages about the proxy setup indicate that the proxy is configured properly. If you visit the URL [<http://localhost:3000/api/posts/1>](<http://localhost:3000/api/posts/1>) in the browser, you will get the response from the JSONPlaceholder APIs as below: { "userId": 1, "id": 1, "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit", "body": "quia et suscipit\\nsuscipit recusandae consequuntur expedita et cum\\nreprehenderit molestiae ut ut quas totam\\nnostrum rerum est autem sunt rem eveniet architecto" } This means that the Node.js API Proxy is doing its job by proxying requests to the mock APIs hosted by JSONPlaceholder. 3. Node.js API Proxy Context Matching The http-proxy-middleware uses the path for proxying requests. For example, in the request http://localhost:3000/api/posts?title=test, the section /api/posts is the actual path. According to the official documentation of the http-proxy-middleware package, there are various ways in which the context matching for the path takes place: Path Matching createProxyMiddleware({...}) matches all paths. This means all requests will be proxied. createProxyMiddleware('/', {...}) also matches all paths. createProxyMiddleware('/api', {...}) only matches paths starting with /api. Multiple Path Matching createProxyMiddleware(['/api', '/test', '/otherpath'], {...})can be used to match multiple paths to a particular proxy configuration Wildcard Path Matching For more fine grained control, you can also use wildcards to match paths. createProxyMiddleware('**', {...}) matches any path and all requests are proxied. createProxyMiddleware('**/*.html', {...}) matches any path which ends with .html. createProxyMiddleware('/api/**/*.html', {...}) matches requests ending with .html within the overall path/api. Custom Path MatchingFor even greater control, you can also provide a custom function to match the path for the API Proxy. See below example:const filter = function (pathname, req) { return pathname.match('^/api') && req.method === 'GET'; }; const apiProxy = createProxyMiddleware(filter, { target: '<https://jsonplaceholder.typicode.com>', }); In the above example, only GET requests to the path /api are proxied. Conclusion With this post, you have built a very simple version of the Node.js API proxy. You can extend it further based on specific requirements. The http-proxy-middleware is a simple but powerful library to build a Node.js API Proxy server. The library provides several configurable properties to handle the proxy functionalities. However, there are many more options that you can leverage for your needs. The code for this demo is available on GitHub. If you found the post useful, consider sharing it with friends and colleagues. In case of any queries, write them in the comments section below.

By Saurabh Dashora CORE
How To Validate Three Common Document Types in Python
How To Validate Three Common Document Types in Python

Regardless of the industry vertical we work in – whether that be somewhere in the technology, e-commerce, manufacturing, or financial fields (or some Venn diagram of them all) – we mostly rely on the same handful of standard document formats to share critical information between our internal teams and with external organizations. These documents are almost always on the move, bouncing across our networks as fast as our servers will allow them to. Many internal business automation workflows, for example – whether written with custom code or pieced together through an enterprise automation platform – are designed to process standard PDF invoice documents step by step as they pass from one stakeholder to another. Similarly, customized reporting applications are typically used to access and process Excel spreadsheets which the financial stakeholders of a given organization (internal or external) rely on. All the while, these documents remain beholden to strictly enforced data standards, and each application must consistently uphold these standards. That’s because every document, no matter how common, is uniquely capable of defying some specific industry regulation, containing an unknown error in its encoding, or even - at the very worst - hiding a malicious security threat behind a benign façade. As rapidly evolving business applications continue to make our professional lives more efficient, business users on any network place more and more trust in the cogs that turn within their suite of assigned applications to uphold high data standards on their behalf. As our documents travel from one location to another, the applications they pass through are ultimately responsible for determining the integrity, security, and compliance of each document’s contents. If an invalid PDF file somehow reaches its end destination, the application which processes it – and, by extension, those stakeholders responsible for creating, configuring, deploying, and maintaining the application in the first place – will have some difficult questions to answer. It’s important to know upfront, right away, whether there are any issues present within the documents our applications are actively processing. If we don’t have a way of doing that, we run the risk of allowing our own applications to shoot us in the foot. Thankfully, it’s straightforward (and standard) to solve this problem with layers of data validation APIs. In particular, document validation APIs are designed to fit seamlessly within the architecture of a file processing application, providing a quick feedback loop on each individual file they encounter to ensure the application runs smoothly when valid documents pass through and halting its process immediately when invalid documents are identified. There are dozens of common document types which require validation in a file processing application, and many of the most common among those, including PDF, Excel, and DOCX (which this article seeks to highlight), are all compressed and encoded in very unique ways, making it particularly vital to programmatically identify whether their contents are structured correctly and securely. Document Validation APIs The purpose of this article is to highlight three API solutions that can be used to validate three separate and exceedingly common document types within your various document processing applications: PDF, Excel XLSX, and Microsoft Word DOCX. These APIs are all free to use, requiring a free-tier API key and only a few lines of code (provided below in Python for your convenience) to call their services. While the process of validating each document type listed above is unique, the response body provided by each API is standardized, making it efficient and straightforward to identify whether an error was found within each document type and if so, what warnings are associated with that error. Below, I’ll quickly outline the general body of information supplied in each of the above document validation API's response: DocumentIsValid – This response contains a simple Boolean value indicating whether the document in question is valid based on its encoding. PasswordProtected – This response provides a Boolean value indicating whether the document in question contains password protection (which – if unexpected – can indicate an underlying security threat). ErrorCount – This response provides an integer reflecting the number of errors detected within the document in question. WarningCount – This response indicates the number of warnings produced by the API response independently of the error count. ErrorsAndWarnings – This response category includes more detailed information about each error identified within a document, including an error description, error path, error URI (uniform resource identifier, such as URL or URN), and IsError Boolean. Demonstration To use any of the three APIs referred to above, the first step is to install the Python SDK with a pip command provided below: pip install cloudmersive-convert-api-client With installation complete, we can turn our attention to the individual functions which call each individual API’s services. To call the PDF validation API, we can use the following code: Python from __future__ import print_function import time import cloudmersive_convert_api_client from cloudmersive_convert_api_client.rest import ApiException from pprint import pprint # Configure API key authorization: Apikey configuration = cloudmersive_convert_api_client.Configuration() configuration.api_key['Apikey'] = 'YOUR_API_KEY' # create an instance of the API class api_instance = cloudmersive_convert_api_client.ValidateDocumentApi(cloudmersive_convert_api_client.ApiClient(configuration)) input_file = '/path/to/inputfile' # file | Input file to perform the operation on. try: # Validate a PDF document file api_response = api_instance.validate_document_pdf_validation(input_file) pprint(api_response) except ApiException as e: print("Exception when calling ValidateDocumentApi->validate_document_pdf_validation: %s\n" % e) To call the Microsoft Excel XLSX validation API, we can use the following code instead: Python from __future__ import print_function import time import cloudmersive_convert_api_client from cloudmersive_convert_api_client.rest import ApiException from pprint import pprint # Configure API key authorization: Apikey configuration = cloudmersive_convert_api_client.Configuration() configuration.api_key['Apikey'] = 'YOUR_API_KEY' # create an instance of the API class api_instance = cloudmersive_convert_api_client.ValidateDocumentApi(cloudmersive_convert_api_client.ApiClient(configuration)) input_file = '/path/to/inputfile' # file | Input file to perform the operation on. try: # Validate a Excel document (XLSX) api_response = api_instance.validate_document_xlsx_validation(input_file) pprint(api_response) except ApiException as e: print("Exception when calling ValidateDocumentApi->validate_document_xlsx_validation: %s\n" % e) And finally, to call the Microsoft Word DOCX validation API, we can use the final code snippet supplied below: Python from __future__ import print_function import time import cloudmersive_convert_api_client from cloudmersive_convert_api_client.rest import ApiException from pprint import pprint # Configure API key authorization: Apikey configuration = cloudmersive_convert_api_client.Configuration() configuration.api_key['Apikey'] = 'YOUR_API_KEY' # create an instance of the API class api_instance = cloudmersive_convert_api_client.ValidateDocumentApi(cloudmersive_convert_api_client.ApiClient(configuration)) input_file = '/path/to/inputfile' # file | Input file to perform the operation on. try: # Validate a Word document (DOCX) api_response = api_instance.validate_document_docx_validation(input_file) pprint(api_response) except ApiException as e: print("Exception when calling ValidateDocumentApi->validate_document_docx_validation: %s\n" % e) Please note that while these APIs do provide some basic security benefits during their document validation processes (i.e., identifying unexpected password protection on a file, which is a common method for sneaking malicious files through a network - the password can be supplied to an unsuspecting downstream user at a later date), they do not constitute fully formed security APIs, such as those that would specifically hunt for viruses, malware, and other forms of malicious content hidden within a file. Any document – especially those that originated outside of your internal network – should always be thoroughly vetted through specific security-related services (i.e., services equipped with virus and malware signatures) before entering or leaving your file storage systems.

By Brian O'Neill CORE
Testing the Untestable and Other Anti-Patterns
Testing the Untestable and Other Anti-Patterns

Books on bad programming habits take up a fraction of the shelf space dedicated to best practices. We know what good habits are – or we pay convincing lip service to them – but we lack the discipline to prevent falling into bad habits. Especially when writing test code, it is easy for good intentions to turn into bad habits, which will be the focus of this article. But first, let’s get the definitions right. An anti-pattern isn’t simply the absence of any structured approach, which would amount to no preparation, no plan, no automated tests and just hacking the shortest line from brainwave to source code. This chaotic approach is more like non-pattern programming. Anti-patterns are still patterns, just unproductive ones, according to the official definition. The approach must be structured and repeatable, even when it is counter-productive. Secondly, a more effective, documented, and proven solution to the same problem must be available. Many (in)famous anti-patterns consistently flout one or more good practices. Spaghetti code and the god object testify to someone’s ignorance or disdain of the principles of loose coupling and cohesion, respectively. Education can fix that. More dangerous, however, are the folks who never fell out of love with over-engineering since the day they read about the Gang of Four because doing too much of a good thing is the anti-pattern that rules them all. It’s much harder to fight because it doesn’t feel like you’re doing anything wrong. Drawn To Complexity Like Moths to a Flame – With Similar Results In the same as you can overdose on almost anything that is beneficial in small doses, you can overdo any programming best practice. I don’t know many universally great practices, only suitable and less suitable solutions to the programming challenge at hand. It always depends. Yet developers remain drawn to complexity like moths to a flame, with similar results. The usefulness of SOLID design principles has a sweet spot, after which it’s time to stop. It doesn’t follow an upward curve where more is always better. Extreme dedication to the single responsibility principle gives you an explosion of specialized boilerplate classes that do next to nothing, leaving you clueless as to how they work together. The open/closed principle makes sense if you’re maintaining a public API, but for a work-in-progress, it’s better to augment existing classes than create an overly complex inheritance tree. Dependency inversion? Of course, but you don’t need an interface if there will only ever be one private implementation, and you don’t always need Spring Boot to create an instance of it. The extreme fringes on opposite sides of the political spectrum have more in common with each other than they have with the reasonable middle. Likewise, no pattern at all gives you the same unreadable mess as well-intentioned over-engineering, only a lot quicker and cheaper. Cohesion gone mad gives you FizzBuzzEnterpriseEdition while too little of it gives you the god object. Let's turn over, then, to test code and the anti-patterns that can turn the efforts into an expensive sinkhole. Unclarity about the purpose of testing is already a massive anti-pattern before a single line of code is written. It’s expressed in the misguided notion that any testing is better than no testing. It isn’t. Playing foosball is better than ineffectual testing because the first is better for team morale and doesn’t lull you into a false sense of security. You must be clear on why you write tests in the first place. First Anti-Pattern: Unaware of the Why Well, the purpose of writing tests is to bring Sonar coverage up to 80%, isn’t it? I’m not being entirely sarcastic. Have you never inherited a large base of untested legacy that has been working fine for years but is languishing at an embarrassing 15% test coverage? Now suddenly the powers that be decide to tighten the quality metrics. You can’t deploy unless coverage is raised by 65 percentage points, so the team spends several iterations writing unit tests like mad. It’s a perverse incentive, but it happens all the time: catch-up testing. Here are three reasons that hopefully make more sense. First, tests should validate specifications. They verify what the code is supposed to do, which comes down to producing the output that the stakeholders asked for. A developer who isn’t clear on requirements can only write a test that confirms what the code already does, and only from inspecting the source code. Extremely uncritical developers will write a test confirming that two times four equals ten because that’s what the (buggy) code returns. This is what can happen when you rush to improve coverage on an inherited code base and don’t take the time to fully understand the what and why. Secondly, tests must facilitate clean coding; never obstruct it. Only clean code keeps maintenance costs down, gets new team members quickly up to speed, and mitigates the risk of introducing bugs. Developing a clean codebase is a highly iterative process where new insights lead to improvements. That means constant refactoring. As the software grows, it’s fine to change your mind about implementation details, but you can only improve code comfortably that way if you minimize the risk that your changes break existing functionality. Good unit tests warn you immediately when you introduce a regression, but not if they’re slow or incomplete. Thirdly, tests can serve as a source of documentation for the development team. No matter how clean your code, complex business logic is rarely self-explanatory if all you have is the code. Descriptive scenarios with meaningful data and illustrative assertions show the relevant input permutations much clearer than any wiki can. And they’re always up to date. Second Anti-Pattern: London School Orthodoxy I thank Vladimir Khorikov for pointing out the distinction between the London versus the classical unit test approach. I used to be a Londoner, but now I’m convinced that unit tests should primarily target public APIs. Only this way can you optimize the encapsulated innards without constantly having to update the tests. Test suites that get in the way of refactoring are often tightly coupled to implementation details. As long as you can get sufficient execution speed and coverage, I find no compelling reason for a rigid one-to-one mapping between source classes and corresponding test classes. Such an approach forces you to emulate every external dependency’s behavior with a mocking framework. This is expensive to set up and positively soul-crushing if the classes under test have very little salient business logic. A case in point: Java @RestController public class FriendsController { @AutoWired FriendService friendService; @AutoWired FriendMapper friendMapper; @GetMapping(“/api/v1/friends”) public List<FriendDto> getAll(){ return friendMapper.map(friendService.retrieveAll()); } } This common Controller/Service layered architecture makes perfect sense: cohesion and loose coupling are taken care of. The Controller maps the network requests, (de)serializes input/output, and handles authorization. It delegates to the Service layer, which is where all the exciting business logic normally takes place. CRUD operations are performed through an abstraction to the database layer, injected in the service layer. Not much seems to go on in this simple example, but that’s because the framework does the heavy lifting. If you leave Spring out of the equation, there is precious little to test, especially when you add advanced features like caching and repositories generated from interfaces. Boilerplates and configurations do not need unit tests. And yet I keep seeing things like this: Java @ExtendWith(MockitoExtension.class) public class FriendsControllerTest { @Mock FriendService friendService; @Mock FriendMapper friendMapper; @InjectMocks FriendController controller; @Test void retrieve_friends(){ //arrange var friendEntities = List.of(new Friend(“Jenny)); var friendDtos = List.of(new FriendDto(“Jenny”)); Mockito.doReturn(friendEntities).when(friendService).findAll(); Mockito.doReturn(friendDtos).when(friendMapper).map(eq(friendEntities)); //act var friends = controller.findAll(); //assert Assertions.assertThat(friends).hasSize(1); Assertions.assertThat(friends.get(0).getName()).isEqualTo(“Jenny”); } } A test like this fails on three counts: it’s too much concerned with implementation details to validate specifications and it’s too simplistic to have any documentary merit. And being tightly bound to the implementation, it surely does not facilitate refactoring. Even a “Hello, World!”-level example like this takes four lines of mock setup. Add more dependencies with multiple interactions and 90% of the code (and your time!) is taken up with tedious mocks setup. What matters most is that Spring is configured with the right settings. Only a component test that spins up the environment can verify that. If you include a test database, it can cover all three classes without any mocking, unless you need to connect to an independent service outside the component under test. Java @SpringBootTest @AutoConfigureMockMvc class FriendsControllerTest { @Test @WithAnonymousUser void get_friends(){ mockMvc.perform(get("/v1/api/friends")) .andExpect(content().string(“[{\“name\”: \”Jenny\”}]”)) } } Third Anti-Pattern: Trying to Test the Untestable The third anti-pattern I want to discuss rears its head when you try to write tests for complex business functionality without refactoring the source. Say we have a 1500-line monster class with one deceptively simple public method. Give it your outstanding debt, last year’s salary slips, and it tells you how much you’re worth. public int getMaxLoanAmountEuros(List<SalaryReport> last12SalarySlips, List<Debt> outstandingDebt); It’s different from the previous example in two important ways: The code-under-test centers on business logic and has high cyclomatic complexity, requiring many scenarios to cover all the relevant outcomes. The code is already there and testing is late to the party, meaning that by definition you can’t work in a test-driven approach. That of itself is an anti-pattern. We’re writing tests as an afterthought, only to validate what the code does. The code may be very clean, with all complexity delegated to multiple short private methods and sub-classes to keep things readable. Sorry, but if there are no unit tests, it’s more likely to be a big ball of mud. Either way, we can’t reduce the essential complexity of the business case. The only way to reach full coverage of all the code under this single public method is by inventing scenarios with different input (the salary and debt information). I have never been able to do that comfortably without serious refactoring, by delegating complex isolated portions to their own class and writing dedicated tests per class. If you’re terrified of breaking things, then changing the access level of private methods to default scope is safer. If the test is in the same package, you might write focused tests for these methods, but it’s a controversial strategy. Best avoid it. You’re breaking encapsulation, wading knee-deep in implementation details, which makes future refactoring even harder. You have to proceed carefully if you tamper with untested production code, but there is no good alternative other than a full rewrite. Writing thorough unit tests for messy code that you inherited without the privilege to refactor is painful and ineffectual. That’s because untested code is often too cumbersome to test. Such untestable code is by definition bad code, and we should not settle for bad code. The best way to avoid that situation is to be aware of the valid reasons why we test in the first place. You will then never write tests as an afterthought. Green Checkbox Addiction The productive path to establishing and maintaining effective test automation is not easy, but at least the good habits make more sense than the well-intentioned yet harmful anti-patterns I pointed out here. I leave you with a funny one, which you might call green checkbox addiction: the satisfaction of seeing an all-green suite, regardless of whether the tests make any sense. That's the false sense of security I mentioned earlier, which makes bad testing worse than no testing at all. It’s like the productivity geeks who create spurious tasks in their to-do lists for the dopamine spike they get when checking them off. Very human, and very unproductive.

By Jasper Sprengers CORE
Securing Developer Tools: Argument Injection in Visual Studio Code
Securing Developer Tools: Argument Injection in Visual Studio Code

The safety of these applications is crucial to prevent attackers from compromising the computer on which developers are working, as they could use this access to obtain sensitive information, alter source code, and further pivot into the company's internal network. This time, my team and I dive into a new vulnerability I identified in one of the most popular IDEs: Visual Studio Code. It allowed attackers to craft malicious links that, once interacted with, would trick the IDE into executing unintended commands on the victim's computer. By reporting the issue to Microsoft, who quickly patched it, our researchers helped to secure the developer ecosystem. Impact The vulnerability can be used to target developers that have the Visual Studio Code IDE installed. Upon clicking on a malicious link crafted by an attacker, victims are prompted to clone a Git repository in Visual Studio Code. This operation is genuine and part of the workflow of most users. For instance, this is how GitLab allows easier cloning of projects: If the developer accepts this operation, attackers can execute arbitrary commands on the victim's computer. Interestingly, Workspace Trust, a feature to harden the IDEs and reduce the risk of unintended commands being executed, is not strictly enforced here. If the last Visual Studio Code window with focus is trusted by the current workspace, this security feature will not have the expected effect. I disclosed this finding to Microsoft through their Researcher Portal, and the patch was released as part of Visual Studio Code 1.67.1 and higher. Microsoft published limited information about this bug as part of their security bulletin and assigned it CVE-2022-30129. In the sections below, I'll first describe how URL handlers are designed in Visual Studio Code and then review the implementation of the one reserved for Git actions to identify an argument injection bug. Further sections will describe how it could be exploited to gain the ability to execute arbitrary commands, as well as the patch implemented by Microsoft. Visual Studio Code URL Handlers Visual Studio Code is most commonly used as a stand-alone desktop application, thanks to Electron. This choice still allows some level of integration with the user's operating system, for instance, by allowing applications to register custom URL protocol handlers. In the case of Visual Studio Code, vscode:// is registered, and vscode-insiders:// for the nightly builds (also called Insiders build). This feature is named Deep Links. The IDE allows internal and external extensions to listen to such events and handle them by registering sub-handlers. The main listener will handle such OS-level events and redirect them to the right extension. They have to implement a simple interface with a method named handleUri() and announce it to the IDE with window.registerUriHandler(): src/vscode-dts/vscode.d.ts TypeScript export interface UriHandler { handleUri(uri: Uri): ProviderResult<void>; } Finding an Argument Injection in the Git Module With this design in mind, it is now possible to start looking for URL handlers in the core of Visual Studio Code. At that time, three were available: extensions/github-authentication and extensions/microsoft-authentication to authenticate with third-party services and obtain the resulting access tokens, and extensions/git to allow users to clone remote repositories as shown above in GitLab. With my prior experience reviewing the code of developer tools, I know that external invocations of version control tools are often riddled with argument injection bugs—you can head to the Related Posts section for a few examples. Let's look at the extensions/git's implementation of handlerUri first! extensions/git/src/protocolHandler.ts TypeScript export class GitProtocolHandler implements UriHandler { // [...] handleUri(uri: Uri): void { switch (uri.path) { case '/clone': this.clone(uri); } } private clone(uri: Uri): void { const data = querystring.parse(uri.query); // [...] commands.executeCommand('git.clone', data.url); } The git.clone command is implemented in extensions/git/src/commands.ts; it is also possible to invoke it manually: extensions/git/src/commands.ts TypeScript @command('git.clone') async clone(url?: string, parentPath?: string): Promise<void> { await this.cloneRepository(url, parentPath); } Let's continue to dig deeper into the code to identify where the external Git binary is invoked: extensions/git/src/commands.ts TypeScript async cloneRepository(url?: string, parentPath?: string, options: { recursive?: boolean } = {}): Promise<void> { // [...] try { // [...] const repositoryPath = await window.withProgress( opts, (progress, token) => this.git.clone( url!, { parentPath: parentPath!, progress, recursive: options.recursive }, token) ); extensions/git/src/git.ts TypeScript async clone(url: string, options: ICloneOptions, cancellationToken?: CancellationToken): Promise<string> { let baseFolderName = decodeURI(url).replace(/[\/]+$/, '').replace(/^.*[\/\\]/, '').replace(/\.git$/, '') || 'repository'; let folderName = baseFolderName; let folderPath = path.join(options.parentPath, folderName); // [...] try { let command = ['clone', url.includes(' ') ? encodeURI(url) : url, folderPath, '--progress']; if (options.recursive) { command.push('--recursive'); } await this.exec(options.parentPath, command, { cancellationToken, env: { 'GIT_HTTP_USER_AGENT': this.userAgent }, onSpawn, }); As promised, there is an argument injection bug in this code: the URL to clone the Git repository is fully controlled and concatenated into the external command line. If this URL starts with dashes, Git will understand it as an option instead of a positional argument. Exploiting an Argument Injection on a Git Clone Operation Argument injection vulnerabilities are very interesting because they are all different and often imply some subtleties; this one is not an exception. This section describes one way to exploit it; other ways exist and are left as an exercise to the reader. Git used to implement git-remote-ext to "bridge smart transport to an external command," but this feature is now disabled by default. As a reminder, we have two injection points: url: the URL of the remote Git repository to clone; folderPath: the destination folder computed from the URL of the Git repository. This is very important in this case as our injected option takes the place of a positional argument: without the second injection point, Git wouldn't have anything to clone from, and the injection wouldn't be exploitable. Finally, if there is any space in the payload, it will be URL-encoded before its interpolation in the command line; it will be easier to try to craft one without spaces: extensions/git/src/git.ts TypeScript let command = ['clone', url.includes(' ') ? encodeURI(url) : url, folderPath, '--progress']; My team and I came up with the following payload: vscode://: the custom scheme registered by Visual Studio Code to the operating system; vscode.git/clone?url=: required to trigger the git.clone command in Visual Studio Code; -u$({open,-a,calculator}): we inject the option -u, equivalent to --upload-pack, to override the command that will be used to communicate with the remote end; :x: this part is required to trick Git into using the transport layer, recognize it as PROTO_LOCAL and invoke the aforementioned upload-pack. Patch Microsoft addressed this issue by improving its validation on the URL of the Git repository to clone as part of the commit c5da533. The URL is parsed using Uri, an internal URI parser, to validate the scheme against a pre-established allow list. The rationale behind this change is that the argument injection bug can only happen if the prefix of the data is fully controlled, which won't be possible if the scheme part of the URL has to be part of this list. diff --- a/extensions/git/src/protocolHandler.ts +++ b/extensions/git/src/protocolHandler.ts @@ -7,6 +7,8 @@ import { UriHandler, Uri, window, Disposable, commands } from 'vscode'; import { dispose } from './util'; import * as querystring from 'querystring'; +const schemes = new Set(['file', 'git', 'http', 'https', 'ssh']); + export class GitProtocolHandler implements UriHandler { private disposables: Disposable[] = []; @@ -26,9 +28,27 @@ export class GitProtocolHandler implements UriHandler { if (!data.url) { console.warn('Failed to open URI:', uri); + return; + } + + if (Array.isArray(data.url) && data.url.length === 0) { + console.warn('Failed to open URI:', uri); + return; + } + + let cloneUri: Uri; + try { + cloneUri = Uri.parse(Array.isArray(data.url) ? data.url[0] : data.url, true); + if (!schemes.has(cloneUri.scheme.toLowerCase())) { + throw new Error('Unsupported scheme.'); + } + } + catch (ex) { + console.warn('Invalid URI:', uri); + return; } - commands.executeCommand('git.clone', data.url); + commands.executeCommand('git.clone', cloneUri.toString(true)); } dispose(): void { This finding was not eligible for a reward from the Microsoft Bug Bounty Program, as only the core is part of the scope; built-in extensions are de facto excluded even if they are enabled by default. This submission still yielded us 40 points for the Microsoft Researcher Recognition Program and got us on the MSRC 2022 Q2 Leaderboard. It is also interesting to note that the Visual Studio Code team started publishing information about security issues on GitHub on top of the usual security bulletin and release notes: the label security is now added to issues, and GitHub security advisories are published. Timeline Date Action 2022-04-05 This issue is reported to Microsoft on their Researcher Portal. 2022-05-06 Microsoft confirms the issue and starts working on a patch. 2022-05-10 The patch is part of the release 1.67.1. Summary In this publication, I demonstrated how a vulnerability in one of the Visual Studio Code URL handlers could lead to the execution of arbitrary commands on the victim's host. The exploitation technique I demonstrated can also be applied to any other argument injection on a git clone invocation. My team and I urge all developers to upgrade their IDE to the latest version and to remain careful when opening foreign links.

By Thomas Chauchefoin

Top Integration Experts

expert thumbnail

John Vester

Lead Software Engineer,
Marqeta @JohnJVester

Information Technology professional with 30+ years expertise in application design and architecture, feature development, project management, system administration and team supervision. Currently focusing on enterprise architecture/application design utilizing object-oriented programming languages and frameworks. Prior expertise building (Spring Boot) Java-based APIs against React and Angular client frameworks. CRM design, customization and integration with Salesforce. Additional experience using both C# (.NET Framework) and J2EE (including Spring MVC, JBoss Seam, Struts Tiles, JBoss Hibernate, Spring JDBC).
expert thumbnail

Colin Domoney

API Security Research Specialist & Developer Advocate,
42Crunch

‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎
expert thumbnail

Saurabh Dashora

Founder,
ProgressiveCoder

‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎
expert thumbnail

Cameron HUNT

Integration Architect,
TeamWork France

An Event-Driven anarchist, motivated by the conviction that monolithic applications can no longer meet the ever-changing needs of businesses in an increasingly complex World. The next generation of (Distributed) Enterprise Applications must instead be composed of best-of-breed 'software components'; and just what is best-of-breed changes from month-to-month, so integration agility becomes key

The Latest Integration Topics

article thumbnail
Five Arguments for Why Microsoft Azure Is the Best Option for Running Industrial IoT Solutions
Three-fourths of businesses use Microsoft Azure as their cloud partner. Learn why it is the best option for industrial IoT solutions.
February 7, 2023
by Shikhar Deopa
· 1,172 Views · 1 Like
article thumbnail
Introduction to the FHIR Standard and CMS Rules
Let's get down to brass tacks of what is FHIR standard, CMS rules, and corresponding FHIR Implementation Guides.
February 7, 2023
by Paul Chayka
· 1,132 Views · 1 Like
article thumbnail
Usage Metering and Usage-Based Billing for the Cloud
Metering and usage-based billing will become key differentiators for cloud offerings. Visibility is the key to success.
February 6, 2023
by Tom Smith CORE
· 829 Views · 1 Like
article thumbnail
The Role of Cloud-Native Security in Application Delivery
Cloud-native security focuses on securing cloud-native platforms, infrastructure, and applications. Let’s discuss strategies to tackle them in this article.
February 6, 2023
by Komal J Prabhakar
· 1,263 Views · 1 Like
article thumbnail
Maximizing Efficiency With the Test Automation Pyramid: Leveraging API Tests for Optimal Results
This article will discuss "How to leverage the Automation Test pyramid" to drive successful automation.
February 6, 2023
by Girish ks
· 1,168 Views · 1 Like
article thumbnail
How to Deploy Machine Learning Models on AWS Lambda Using Docker
Learn how to package an ML model as a Docker container and deploy it on AWS Lambda and gain a deeper understanding of how to deploy ML models on the cloud.
February 5, 2023
by Mittal Patel
· 2,094 Views · 1 Like
article thumbnail
Apache Kafka Introduction, Installation, and Implementation Using .NET Core 6
A step-by-step introduction to Apache Kafka, its install, and its implementation using .NET Core 6 with background information, code blocks, and guide pictures.
February 4, 2023
by Jaydeep Patil
· 3,682 Views · 2 Likes
article thumbnail
Event Driven 2.0
In this article, we will talk about how event-driven architecture differs from other popular message-driven architecture.
February 4, 2023
by Sveta Gimpelson
· 3,607 Views · 2 Likes
article thumbnail
Data Binding in React: The Easy Way
Data binding is a process of connecting data between a component and its UI representation. In this article, we deep dive into data binding and its importance.
February 3, 2023
by Rahul .
· 2,952 Views · 1 Like
article thumbnail
How To Get Page Source in Selenium Using Python
In this article, we explore how to make Selenium WebDriver get page source and demonstrate how Selenium gets XML page source as well while using Python.
February 3, 2023
by Nishant Choudhary
· 3,859 Views · 2 Likes
article thumbnail
Learning by Doing: An HTTP API With Rust
After a change from the initial plans, follow the developer's experience of developing a demo HTTP API application in Rust.
February 3, 2023
by Nicolas Fränkel CORE
· 3,379 Views · 1 Like
article thumbnail
How To Avoid “Schema Drift”
This article will explain the existing solutions and strategies to mitigate the challenge and avoid schema drift, including data versioning using lakeFS.
February 3, 2023
by Yaniv Ben Hemo
· 5,387 Views · 1 Like
article thumbnail
Top Three Docker Alternatives To Consider
Docker is a popular platform for creating and managing containerized applications. Here are three Docker hub alternatives: Podman, Containerd, and LXD.
February 3, 2023
by Ruchita Varma
· 10,474 Views · 2 Likes
article thumbnail
Comparing Kubernetes Gateway and Ingress APIs
In this article, we will explore the new Kubernetes Gateway API and compare it with the existing Kubernetes Ingress API for handling external traffic.
February 2, 2023
by Navendu Pottekkat
· 2,884 Views · 3 Likes
article thumbnail
Too Many Tools? Streamline Your Stack With AIOps
DevOps and SRE teams need a more efficient monitoring approach that increases availability and optimizes the customer experience.
February 2, 2023
by Richard Whitehead
· 2,722 Views · 1 Like
article thumbnail
Select ChatGPT From SQL? You Bet!
Users can use ChatGPT from Couchbase N1QL. This article describes the feature and function with examples directly from N1QL, using JavaScript UDF.
February 2, 2023
by Keshav Murthy CORE
· 4,114 Views · 2 Likes
article thumbnail
Part I: Creating Custom API Endpoints in Salesforce With Apex
In Part I of this two-part series, we'll take a look at how to create custom API endpoints in Salesforce with APEX.
February 2, 2023
by Michael Bogan CORE
· 2,492 Views · 1 Like
article thumbnail
Development Platform for Data Protection
SaaS solutions now have a way to secure and protect customer data.
February 2, 2023
by Tom Smith CORE
· 2,925 Views · 1 Like
article thumbnail
Remote Debugging Dangers and Pitfalls
Debugging over the network using a protocol like JDWP isn’t hard. However, there are risks that aren’t immediately intuitive and some subtle solutions.
February 2, 2023
by Shai Almog CORE
· 5,808 Views · 3 Likes
article thumbnail
How and Why You Should Start Automating DevOps
Integration of DevOps and automation is what leads to a more efficient software development lifecycle. Understand what it is about automating DevOps and how.
February 1, 2023
by Susmitha Vakkalanka
· 5,500 Views · 1 Like
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • ...
  • Next

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: