Deployment

Quarkus Quarkus is an open-source CDI-based framework introduced by Red Hat. It supports the development of fully reactive microservices, provides a fast startup, and has a small memory footprint. Below was our overall experience using Quarkus: It helped with a quicker and more pleasant development process. Optimized Serverless deployments for low memory usage and fast startup times Allowed us to utilize both blocking (imperative) and non-blocking (reactive) libraries and APIs Worked well with continuous testing to facilitate test-driven development Allowed support to test the JUnit test cases, which we have developed using test-driven development approach Quarkus Supports Native Builds Quarkus supports native builds for an application deployment which contains the application code, required libraries, Java APIs, and a reduced version of a VM. The smaller VM base improves the startup time of the application. To generate a native build using a Java Maven project, one can leverage Docker or podman with GraalVM: mvn clean install -Dnative -Dquarkus.native.container-build=true -Dmaven.test.skip=true The native executable is lightweight and performance optimized. Common Build and Runtime Errors Quarkus, being fairly new, lacks sufficient support in the community and documentation. Below were some of the errors/issues we encountered during development and their resolution. Build Errors UnresolvedElementException com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: xxx This error is caused by missing classes at the image build time. Since the native image runtime does not include the facilities to load new classes, all code needs to be available and compiled at build time. So any class that is referenced but missing is a potential problem at run time. Solution The best practice is to provide all dependencies to the build process. If you are absolutely sure that the class is 100% optional and will not be used at run time, then you can override the default behavior of failing the build process by finding a missing class with the — allow-incomplete-classpath option to native-image. Runtime Errors Random/SplittableRandom com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected an instance of Random/SplittableRandom class in the image heap These errors are caused when you try to initialize these classes in a static block. Embedding instances of Random and SplittableRandom in native images cause these errors. These classes are meant to provide random values and are typically expected to get a fresh seed in each run. Embedding them in a native image results in the seed value that was generated at build-time to be cached in the native image, thus breaking that expectation. Solution We were able to resolve these by using below different ways: By avoiding build time initialization of classes holding static fields that reference (directly or transitively) instances of Random or SplittableRandomclasses. The simplest way to achieve this is to pass — initialize-at-run-time=<ClassName>to native-image and see if it works. Note that even if this works, it might impact the performance of the resulting native image since it might prevent other classes from being build-time initialized as well. Register classes holding static fields that directly reference instances of Random or SplittableRandom classes to be reinitialized at run-time. This way, the referenced instance will be re-created at run-time, solving the issue. Reset the value of fields (static or not) referencing (directly or transitively) instances of Random or SplittableRandom to null in the native-image heap. ClassNotFoundException/InstantiationException/IllegalArgumentException These errors can occur when a native image builder is not informed about some reflective calls or a resource to be loaded at run time. Or if there is a third-party/custom library that includes some, ahead-of-time incompatible code. Solution In order to resolve these exceptions, add the complaining class in reflect-config.json JSON { { "name": "com.foo.bar.Person", "allDeclaredMethods": true, "allDeclaredConstructors": true } } Reflection Issues With Native Builds When building a native executable, GraalVM operates with a closed-world assumption. Native builds with GraaVM analyzes the call tree and remove all the classes/methods/fields that are not used directly. The elements used via reflection are not part of the call tree, so they are dead code eliminated. In order to include these elements in the native executable, we’ll need to register them for reflection explicitly. JSON libraries typically use reflection to serialize the objects to JSON, and not registering these classes for reflection causes errors like the below: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class org.acme.jsonb.Person and no properties discovered to create BeanSerializer (to avoid an exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) We resolved these by adding below annotation: Java @RegisterForReflection public class MyClass { ... } If the class is in a third-party jar, you can do it by using an empty class that will host the @RegisterForReflection for it: Java @RegisterForReflection(targets={ MyClassRequiringReflection.class, MySecondClassRequiringReflection.class}) public class MyReflectionConfiguration { ... } Note that MyClassRequiringReflection and MySecondClassRequiringReflection will be registered for reflection but not MyReflectionConfiguration. This feature is handy when using third-party libraries using object mapping features (such as Jackson or GSON): Java @RegisterForReflection(targets = {User.class, UserImpl.class}) public class MyReflectionConfiguration { ... } We can use a configuration file to register classes for reflection. As an example, in order to register all methods of class com.test.MyClass for reflection, we create reflection-config.json (the most common location is within src/main/resources). JSON [ { "name" : "com.test.MyClass", "allDeclaredConstructors" : true, "allPublicConstructors" : true, "allDeclaredMethods" : true, "allPublicMethods" : true, "allDeclaredFields" : true, "allPublicFields" : true } ] Integration With DynamoDB-Enhanced Client Another aspect of using Serverless architecture was the use of DynamoDB. Although there are ways to connect simple DynamoDB clients to do all operations, it does require a lot of code writing which brings verbosity and a lot of boilerplate code to the project. We considered using DynamoDBMapper but figured we couldn't use it with Quarkus since it doesn't support Java SDK1. Enhanced DynamoDB Client in Java SDK2 is the substitute Java SDK1 DynamoDBMapper, which worked well with Quarkus, although there were a few issues setting it up for classes when using native images. Annotated Java beans for creating TableSchema apparently didn’t work with native images. Mappings got lost in translation due to reflection during the native build. To resolve this, we used static table schema mappings using builder pattern, which actually is faster compared to bean annotations since it doesn't require costly bean introspection: Java TableSchema<Customer> customerTableSchema = TableSchema.builder(Customer.class) .newItemSupplier(Customer::new) .addAttribute(String.class, a -> a.name("id") .getter(Customer::getId) .setter(Customer::setId) .tags(primaryPartitionKey())) .addAttribute(Integer.class, a -> a.name("email") .getter(Customer::getEmail) .setter(Customer::setEmail) .tags(primarySortKey())) .addAttribute(String.class, a -> a.name("name") .getter(Customer::getCustName) .setter(Customer::setCustName) .addAttribute(Instant.class, a -> a.name("registrationDate") .build(); Quarkus has extensions for commonly used libraries which simplifies the use of that library in an application by providing some extra features which help in development, testing, and configuration for a native build. Recently, Quarkus released an extension for an enhanced client. This extension will resolve the above-mentioned issue related to native build and annotated Java beans for creating TableSchema caused by the use of reflection in AWS SDK. To use this extension in the Quarkus project, add the following dependency in the pom file: XML <dependency> <groupId>io.quarkiverse.amazonservices</groupId> <artifactId>quarkus-amazon-dynamodb</artifactId> </dependency> <dependency> <groupId>io.quarkiverse.amazonservices</groupId> <artifactId>quarkus-amazon-dynamodb-enhanced</artifactId> </dependency> We have three options to select an HTTP client 2 for a sync DynamoDB client and 1 for an async DynamoDB client; the default is a URL HTTP client, and for that need to import the following dependency. XML <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>url-connection-client</artifactId> </dependency> If we want an Apache HTTP client instead of a URL client, we can configure it by using the following property and dependencies: Properties files quarkus.dynamodb.sync-client.type=apache XML <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>apache-client</artifactId> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-apache-httpclient</artifactId> </dependency> For an async client, the following dependency can be used: XML <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>netty-nio-client</artifactId> </dependency> Dev and test services for DynamoDB are enabled by default which uses docker to start and stop those dev services; these services help in dev and test by running a local DynamoDB instance; we can configure the following property to stop them if we don’t want to use them or don’t have Docker to run them: Properties files quarkus.dynamodb.devservices.enabled=false We can directly inject enhanced clients into our application, annotate the model with corresponding partitions, sort, and secondary partitions, and sort keys if required. Java @Inject DynamoDbEnhancedClient client; @Produces @ApplicationScoped public DynamoDbTable<Fruit> mappedTable() { return client.table("Fruit", TableSchema.fromClass(Fruit.class)) } Java @DynamoDbBean public class Fruit { private String name; private String description; @DynamoDbPartitionKey public String getName() { return name; } public void setName(String name) { this.name = name; } public String getDescription() { return description; } public void setDescription(String description) { this.description = description; } } Achieving Security With Native Builds — SSL connections Developing microservices architecture will, at some point, require you to call a microservice from another microservice. Quarkus provides its own Rest client configuration to efficiently do so but calling those services securely (using SSL) becomes complicated when using native images. By default, Quarkus will embed the default trust store ($GRAALVM_HOME/Contents/Home/lib/security/cacerts) in the native Docker image. The function.zip generated with the native build won’t embed the default trust store, which causes issues when the function is deployed to AWS. We had to add an additional directory zip.native under src/main to add the certificates to function.zip. zip.native contains cacerts and custom bootstrap.sh (shown below) cacerts is the trust store and it holds all the certificates needed. bootstrap.sh should embed this trust store within the native function.zip. Shell # bootstrap.sh #!/usr/bin/env bash ./runner -Djava.library.path=./ -Djavax.net.ssl.trustStore=./cacerts -Djavax.net.ssl.trustStorePassword=changeit Health Checks and Fault Tolerance with Quarkus Health Checks and Fault Tolerance are crucial for microservices since these help in enabling Failover strategy in applications. We leveraged the quarkus-smallrye-health extension, which provides support to build health checks out of the box. We had overridden the HealthCheck class and added dependency health checks for dependent AWS components like DynamoDB to check for health. Below is one of the sample responses from the health checks with HTTP status 200: JSON { "status": "UP", "checks": [ { "name": "Database connections health check", "status": "UP" } ] } Along with these health checks, we used fault tolerance for microservice-to-microservice calls. In a case called microservice is down or not responding, max retry and timeouts were configured using quarks-small rye-fault-tolerance. After max retries, if the dependent service still doesn't respond, we used method fallbacks to generate static responses. Java import org.eclipse.microprofile.faulttolerance.Retry; import org.eclipse.microprofile.faulttolerance.Timeout; import org.eclipse.microprofile.faulttolerance.Fallback; ... public class FooResource { ... @GET @Retry(maxRetries = 3) @Timeout(1000) public List<Foo> getFoo() { ... } @Fallback(fallbackMethod = "fallbackRecommendations") public List<Foo> fallbackforGetFoo() { ... } public List<Foo> fallbackRecommendations() { ... return Collections.singletonList(fooRepository.getFooById(1)); } } Conclusion Overall, Quarkus is an excellent growing framework that offers a lot of options for developing serverless microservices using native images. It optimizes Java and makes it efficient for containers, cloud, and serverless environments with memory consumption optimization and a fast first response time.

System security usually includes two core topics: authentication and authorization. One solves the problem of “Who is s/he?” and the other solves the problem of “Does s/he have permission to perform an operation?” In the big data area, Apache Ranger is one of the most popular choices for authorization, it supports all mainstream big data components, including HDFS, Hive, HBase, and so on. As Amazon EMR rolls out native ranger (plugins) features, users can manage the authorization of EMRFS(S3), Spark, Hive, and Trino all together. For authentication, an organization usually has its own centralized authentication infrastructure, i.e., Windows AD or OpenLDAP; however, for most big data components, Kerberos is only supported authentication mechanism, so users usually need to integrate Windows AD/OpenLDAP and Kerberos together to unify authentication. We will focus on how to implement automated installation and integration for Amazon EMR and Apache Ranger. This series is composed of four articles. Each article will introduce a completed solution against different technology stacks. 1. Solutions Overview Installing Apache Ranger and integrating with Amazon EMR covers three main components: Install and integrate an authentication provider. Setup Ranger server and its plugins on EMR cluster. Configure all related components if Kerberos is enabled. For authentication providers, Windows AD and OpenLDAP are most widely used. Their installation and integration are very different, so they should count as two separate jobs. For Ranger installation, there are two options. The first is “open-source ranger server + EMR-native ranger plugins.” In the article, we will refer to it as an “EMR-native” ranger solution. The second is “open-source ranger server + open-source ranger plugins.” In the article, we will refer to it as an “open-source” ranger solution. Installing the two solutions will be two separate jobs. For Kerberos, if enabled, it will bring a lot of changes to the above jobs, so enabling or disabling Kerberos is also two separate jobs. In summary, based on the three factors above, there are eight possible scenarios (technology stacks) as follows: This series is composed of four articles, which are against the first four scenarios. The following is a scenarios and solution map: Scenario Solution 1 Apache Ranger and AWS EMR Automated Installation and Integration Series (2): OpenLDAP + EMR-Native Ranger 2 Apache Ranger and AWS EMR Automated Installation and Integration Series (3): Windows AD + EMR-Native Ranger 3 Apache Ranger and AWS EMR Automated Installation and Integration Series (4): OpenLDAP + Open-Source Ranger 4 Apache Ranger and AWS EMR Automated Installation and Integration Series (5): Windows AD + Open-Source Ranger For scenarios 5 and 6, as of this writing, EMR is not yet supported. Since disabling Kerberos on EMR cluster is not a recommended practice, the AWS service team is working on a solution to meet the needs. For scenarios 7 and 8, considering few users pick them, we won't discuss them. Note: At the time of writing, Trino plugin is NOT available yet, so this solution does NOT support Trino plugin at present. 2. Why Is Installing Ranger So Difficult? Whether you’ve successfully made it before or not, installing and integrating Windows AD/OpenLDAP + Ranger + EMR is a very hard job, it is complicated, error-prone, and time-consuming for the following reasons: It requires operators have enough knowledge about Windows AD, OpenLDAP, Kerberos, and SSL/TLS, which are not core skills of big data engineers. Learning them will take a lot of time. The architecture of Ranger is complex, it includes two server-side components: Ranger Admin and Ranger UserSync. Two storage components: MySQL and Solr, and a variety of plugins. For plugins, they also need to be installed on cluster nodes, so a complete manual installation is a heavy job. It is not a self-contained job, usually it needs to integrate with an existing Windows AD/OpenLDAP server or an EMR cluster. Many external uncertain factors may result in installation failure, i.e., network issues, incorrect environment-specific configurations, and so on. The EMR-native ranger solution strongly depends on Kerberos and SSL/TLS. This significantly increases the complexity of integration. There is no out-of-the-box distribution package for Ranger. Installation has to start from compiling source codes, which is a challenge for non-java engineers. The overlapping of the above factors makes this job very difficult. 3. Introduction to Automated Installer As the voice of simplifying the Ranger usage experience is getting louder, since 2020, I took on the initiatives and reinvented an automated installer to improve the user experience for Ranger on EMR. Here is the automated installer repository address: Project Name Repository Address Ranger EMR CLI Installer https://github.com/bluishglc/ranger-emr-cli-installer It supports four scenarios (No. 1, 2, 3, 4) at the same time. In other words, it supports Windows AD and OpenLDAP and works in all AWS regions (including Chinese regions). For Scenarios 3/4, this installer can install on an existing cluster and supports multi-master clusters and single-master clusters. For each step, this installer always checks connectivity first then decides whether to go for the next steps. This is very helpful to identify network issues or service failure, i.e., when Ranger or OpenLDAP is not up. Finally, the actual installation job is a trial-and-error process. Users always need to try different parameter values to find the one that works in users' environment. The installer allows users to rerun an all-in-one installation anytime without side effects and users can also do a step-by-step run for debugging. The following is a key features summary: We know there is an existing solution on this AWS blog: “Implementing Authorization and Auditing using Apache Ranger on Amazon EMR.” However, this installer is very different from the solution to design to features. This solution only supports two scenarios, (No. 2, 4), and works in the us-east-1 region only. For Scenarios 3/4, this solution can not support existing or multi-master clusters, and so on. This series of articles are totally based on this tool to guide users through the installation for the first four scenarios. Because EMR and Ranger have multiple versions, the compatibility between different versions should be brought to our attention. Generally, Ranger 1 works with Hadoop 2 and Ranger 2 works with Hadoop 3. This installer is developed against Ranger 2.1.0 and 2.2.0, so it only supports EMR 6.X. We fully tested four solutions against Ranger 2.1.0, all passed, and we partially tested for Ranger 2.2.0, which also works, but there may be potential bugs which are not found yet. The following is Ranger and EMR version compatibility matrix: In the next article, we start to introduce each solution one by one.

Despite the workflow improvements that have been made with CI/CD, there are huge bottlenecks found in the pull request and code review process. These bottlenecks can be removed with continuous merge, a set of processes that make pull requests easier to pick up and code reviews quicker to merge. Let's look deeper into what continuous merge is and why it's needed The State of CI/CD Plain and simple: Continuous integration/continuous delivery (CI/CD) has become standard practice for rapidly delivering new software features, bug fixes and enhancements. Continuous integration (CI) is where developers merge individual code contributions into a central repository, where automated tests and builds are executed. Continuous delivery (CD) automates code change deployment to testing and production environments. Continuous deployment — a term often confused with continuous delivery — is the final step of the DevOps pipeline, where all changes that successfully pass the production pipeline are released to customers. CI/CD automates every step of the development process, ensuring products and features are shipped to users almost as fast as they’re developed. But it does have drawbacks. Where CI/CD Can Be Improved Many branches often extend from the central repository when multiple software developers work simultaneously on a large codebase. Branches with long lifetimes (the period between review and merging) impede the performance improvement that agile practices like CI/CD seek to achieve. Inefficiencies in the pull-request (PR) process create bottlenecks in the delivery pipeline, especially when code reviews take days to complete. For optimum CI/CD performance, there should be at most three branches with lifetimes no longer than a day before merging with the trunk. But in most software development environments, this is virtually unheard of. Pull request reviews that take days or weeks have significant cost implications. Besides work backlogs, delayed reviews can trigger complex merge conflicts. Even worse, pull requests come before integrated testing in the CI/CD pipeline, so a successful review doesn’t guarantee a similar outcome later. The solution to these bottlenecks in the pipeline is continuous merging. Continuous Merge Definition Continuous merge is a set of processes that help eliminate the bottlenecks plaguing pull requests and code reviews. The standard practice for engineering projects is to manage the code base through a version control system (VCS), where developers and contributors can collaborate. It’s common for VCS repositories to have one or more branches, and in traditional PR reviews, changes to the code require manual checking before merging into the main branch. Understanding Why Continuous Merge Is Needed A typical code review involves a contributor or developer opening a pull request and informing other collaborators that code changes have been pushed to a branch and require reviewing (and subsequent approval) before merging into the main branch. PRs enable collaborators, typically lead or senior developers, to check the changes for quality adherence and then take one of these three actions: Comment on the proposed changes. Approve the changes. Request further changes before merging the branch. Many, many inefficiencies characterize this process, often resulting in PRs taking longer than is ideal. For example: The collaborator may begin reviewing the PR only to pause prematurely to attend to other responsibilities. The process freezes because developers don’t get feedback, nor is the merge executed. If the repository has many branches with multiple contributors, the entire CI/CD pipeline may be affected, introducing the risk of merge conflicts and reduction of developer productivity. Other issues that slow down traditional reviews include: PRs being too long; Overwhelmed teams; Diversion of collaborators to other tasks; or Sub-optimal assignment of PRs to people, such that the desired outcome is not achieved in the initial request. Many gaps in this process necessitate re-engineering the PR review process to eliminate such inefficiencies. How Continuous Merge Creates Frictionless Code Review A pull request, also known as a merge request, is where developers or contributors indicate that code changes are ready for merging into the project’s main repository. It is an industry-standard practice for other developers, team leads and other parties to review code changes before merging, human input into this process is inevitable. Historically, this pull request process has caused inefficiencies, particularly review delays, as the process is not automated, so speed depends on the availability of the right code reviewer. Continuous merge seeks to improve the pull request review process by introducing a layer of automation that enables automatic approval and efficient routing of complex pull requests to relevant reviewers. Continuous merge considers the unique characteristics of individual pull requests and routes them appropriately on a risk-based model, ensuring process optimization and bottleneck reduction. Continuous merge categorizes pull requests and code reviews according to their characteristics — type, code complexity, and size. This creates automatic pull request lanes that boost merge speed and efficiency. The 3 Crucial Steps To Continuous Merge Step 1 - Provide Context To Pull Requests That Make Them Easier To Pick Up The first step is to understand the context of the pull request. Pull requests are treated equally by most version control systems, which provide the same information for each one — not enough for a reviewer to assess their size or complexity. Continuous merge adds a contextual layer with more information, like the estimated review time, the concerned project component, and the pull request owner. Step 2 - Treat Pull Requests Differently Depending on Their Size and Characteristics The second step involves classifying pull requests according to this contextual information. This process acknowledges that pull requests vary in size — some are just a few lines of code, and others are chunky. Despite this, pull requests go through a similar process, and even those that could be completed quickly may extend for days before the review.The routing step of continuous merge seeks to remedy such inefficiencies by relying on PR classification to send pull requests with a few lines of code for near-instantaneous reviews and approval. The WorkerB Chrome extension for pull requests simplifies creating and delivering context-enriched pull requests to code reviewers. Step 3 - Streamline Pull Requests With Low Risk The third and final step of the continuous merging is to apply rule-based automation and tooling to achieve automatic approvals and merges for low-risk pull requests while optimizing the routing of others based on their level of risk. Why Continuous Merge Beats Traditional Merge The traditional merge workflow involves strictly defined steps with all pull requests — whether five lines or a critical change of 100 lines — processed the same way. Similarly, changes to static files, which can be approved and merged automatically, are processed through the same pipeline. When code reviews are delayed for days, there’s a greater risk of merge conflicts, and idle time between pull request reviews can lead to a drop in developer productivity. Continuous merge, in contrast, addresses these CI/CD pipeline bottlenecks by contextualizing PR requests and classifying them via a model that has been defined by the team. And following standard DevOps practices, pull requests are placed in appropriate lanes for continuous merge through automated tools. Continuous Merge Completes the Promise of CI/CD Traditional PR reviews and merge workflows tend to create bottlenecks in the CI/CD process. Code reviews and approvals can cause delays for days, even when some are low-risk and could be resolved quickly. Because all pull requests are processed the same way, improvements to the efficiency of this process have yet to be made. Continuous merge is a promising solution to these challenges. With the continuous merging, developers can create custom rules to accompany their pull requests, optimizing the review process. Check out this talk from our Director of Developer Experience, Luke Kilpatrick, to learn more about continuous merge: Engineering Insights before anyone else... The Weekly Interruption is a newsletter designed for engineering leaders, by engineering leaders. We get it. You're busy. So are we. That's why our newsletter is light, informative and oftentimes irreverent. No BS or fluff. Each week we deliver actionable advice to help make you - whether you're a CTO, VP of Engineering, team lead or IC - a better leader.

Nowadays, growing applications on-cloud, complex databases enterprises require data exchange between systems to be seamless without friction. Enterprise Integration connects critical systems and applications across on-premise and cloud platforms. There may be a need to integrate the systems from on-premise to cloud platforms to integrate the data. Some Enterprise Integration types are Application Integration, Data Integration, Process Integration, and Device Integration. Modern business applications are having problems while cloud integrations. Several vendors offer iPaaS (Integration Platform as a Service) and EiPaaS (Enterprise Integration Platform as a Service) solutions to address cloud integration solutions. iPaaS is a cloud platform that connects different systems and technologies developed in the cloud or on-premise. EiPaas, connects different applications, systems, and technologies within the cloud platforms or on-premise to integrate business processes/data. Gartner defines EiPaaS as a combination of integration technology functionalities delivered as a suite of cloud services designed to support enterprise-class integration initiatives. EiPaaS Role in Enterprise Architecture EiPaaS combines cloud-centric technologies and API management capabilities to handle the application's connectivity and integrations. EiPaaS is best after iPaaS to increase autonomy, improve connectivity and governance, and enforce policies. In addition, eiPaaS solutions help the legacy infrastructure connect to cloud platforms seamlessly. Critical capabilities of EiPaaS are a wide range of protocol connectivity, application connectors, database connectors, routing/orchestration, policy enforcement, community management, and CI/CD. Below is the blueprint of the EiPaaS platform as follows: The top features of EiPaaS platforms are API-powered integrations, Integration governance, Data Interchange Management, IoT compatibility, marketplace, and flexible deployment options. Trends in EiPaaS Platforms Data Control and Silos elimination: Data flow between systems or migrated systems is crucial, and that should be controlled and silos. Hybrid Integration Platforms: Organizations adopting hybrid-based solutions to move their applications from legacy to cloud. Centralized Integrations: Internal and external systems are connected and centralized for easy maintenance. CI/CD adoption: Helps the teams to enforce deployment strategy. Advantages of Using EiPaaS in Enterprise Architecture Seamless connectivity between systems and subsystems – Internal/external systems. Decoupling the applications and easily configurable. Effective utilization of cloud features. The flexibility of deployment models. Prebuild connectors Secured Workflow Orchestrations Real-time integration Centralized Interfaces In the digital transformation process, EiPaaS solutions are beneficial to process the data arriving at the hub. Several iPaaS vendors/players help the organizations to support integration challenges. The following are the significant players in iPaaS/EiPaaS platforms for the last couple of years. Informatica: Intelligent Cloud Services is an iPaaS platform from Informatica powered by Artificial Intelligence and Machine Learning. It comes with an event-driven and real-time data exchange platform along with a wide range of products like ETL, Batch, etc., and it helps with seamless integration of cloud and on-premise applications. In addition, it provides features like Data integration checks, quality, and governance. CLAIRE is AI and ML engine that helps to automate data management and governance tasks. Ideal for large data sets to process. MuleSoft: Most popular iPaaS platform from Mule. Anypoint platform combines the capabilities of cloud and on-premise integrations seamlessly. Supports lifecycle and development of APIs, monitoring, and analytics. Different silos systems are connected using the Anypoint platform with prebuilt APIs, connectors, and other assets. In 2022, the Anypoint platform will continue as the leading iPaaS. Boomi: Introduced iPaaS platform by Dell in 2008 (currently called Boomi) with five applications as Integration, Hub, Exchange, Mediate, and Flow. Leading provider of EiPaaS platform for several years. Non-tech users build automated integrations by using pallets provided by the tool. Jitterbit: Data and application integration platform was provided by Jitterbit. Provides API-centric features to connect different types of cloud-based systems with on-premises systems. Provided user-friendly UI to build integrations for both non-techy and techy guys Oracle SOA: Oracle SOA builds on cloud adaptors, Ftp/file services, different integration services, and developer tools. Capable of SOA features and migration of a separate application from on-premise to cloud and vice-versa. It supports IoT and mobile technologies. 100+ adopters are available to connect to different systems like SAP, Siebel, and Other systems. Very much suitable for high-volume integrations. Workato: Workato platform was built as low code leveraging the mix of API and GUI connectors. Workato contains 1000+ prebuilt connectors to connect different systems and 500000+ read-to-use "recipe" preconfigured workflows. In digital transformation initiatives, EiPaaS platforms are critical to transforming data between systems or applications. Various vendors offer tools and technologies to support integrations between systems in the cloud or on-premise. Although the tools mentioned above are minimal, many players in the market support the EiPaas platform.

.NET 7 has been released! It's time for us to dig into its source code and start looking for errors and strange code fragments. In this article, you'll see comments on our findings from the .NET developers. After all, they know the platform code better than anyone else. Buckle up! I analyzed the release code of .NET 7. There were two release candidates (RC) prior to the main release, so most of the bugs must have been fixed. It's more interesting that way — we can investigate whether some of them have gotten into production. I created an issue on GitHub for each suspicious code fragment. This helped me understand which ones are redundant, which ones are incorrect, and what was fixed by developers. Issue 1 Can you spot an error here? Let's check! C# internal sealed record IncrementalStubGenerationContext( StubEnvironment Environment, SignatureContext SignatureContext, ContainingSyntaxContext ContainingSyntaxContext, ContainingSyntax StubMethodSyntaxTemplate, MethodSignatureDiagnosticLocations DiagnosticLocation, ImmutableArray<AttributeSyntax> ForwardedAttributes, LibraryImportData LibraryImportData, MarshallingGeneratorFactoryKey< (TargetFramework, Version, LibraryImportGeneratorOptions) > GeneratorFactoryKey, ImmutableArray<Diagnostic> Diagnostics) { public bool Equals(IncrementalStubGenerationContext? other) { return other is not null && StubEnvironment.AreCompilationSettingsEqual(Environment, other.Environment) && SignatureContext.Equals(other.SignatureContext) && ContainingSyntaxContext.Equals(other.ContainingSyntaxContext) && StubMethodSyntaxTemplate.Equals(other.StubMethodSyntaxTemplate) && LibraryImportData.Equals(other.LibraryImportData) && DiagnosticLocation.Equals(DiagnosticLocation) && ForwardedAttributes.SequenceEqual(other.ForwardedAttributes, (IEqualityComparer<AttributeSyntax>) SyntaxEquivalentComparer.Instance) && GeneratorFactoryKey.Equals(other.GeneratorFactoryKey) && Diagnostics.SequenceEqual(other.Diagnostics); } public override int GetHashCode() { throw new UnreachableException(); } } This code fragment checks whether the this and other objects are equivalent. However, the developer made a mistake and compared the DiagnosticLocation property with itself. Incorrect comparison: C# DiagnosticLocation.Equals(DiagnosticLocation) Correct comparison: C# DiagnosticLocation.Equals(other.DiagnosticLocation) I found this error in the LibraryImportGenerator class. A bit later I found two more fragments — the same error, but in different classes: The JSImportGenerator class The JSExportGenerator class Fun fact: .NET 7 has a test for this feature. However, the test is also incorrect, that's why it doesn't detect this error. In .NET 8 the code is heavily rewritten. However, the developers haven't fixed the .NET 7 code yet — they decided to wait for the feedback. You can read more about it in the issue on GitHub. Issue 2 C# internal static void CheckNullable(JSMarshalerType underlyingSig) { MarshalerType underlying = underlyingSig._signatureType.Type; if (underlying == MarshalerType.Boolean || underlying == MarshalerType.Byte || underlying == MarshalerType.Int16 || underlying == MarshalerType.Int32 || underlying == MarshalerType.BigInt64 || underlying == MarshalerType.Int52 || underlying == MarshalerType.IntPtr || underlying == MarshalerType.Double || underlying == MarshalerType.Single // <= || underlying == MarshalerType.Single // <= || underlying == MarshalerType.Char || underlying == MarshalerType.DateTime || underlying == MarshalerType.DateTimeOffset ) return; throw new ArgumentException("Bad nullable value type"); } Location: JSMarshalerType.cs, 387 Here the developer double-checks if the underlying variable equals to MarshalerType.Single. Sometimes such checks hide errors: for example, the left and right variables should have been checked, but instead, the left variable is checked twice. Here's a list of similar errors found in open-source projects. I created an issue on GitHub. Luckily, this code fragment wasn't erroneous — it was just a redundant check. Issue 3 C# public static bool TryParse(string text, out MetricSpec spec) { int slashIdx = text.IndexOf(MeterInstrumentSeparator); if (slashIdx == -1) { spec = new MetricSpec(text.Trim(), null); return true; } else { string meterName = text.Substring(0, slashIdx).Trim(); string? instrumentName = text.Substring(slashIdx + 1).Trim(); spec = new MetricSpec(meterName, instrumentName); return true; } } Location: MetricsEventSource.cs, 453 The TryParse method always returns true. This is weird. Let's see where this method is used: C# private void ParseSpecs(string? metricsSpecs) { .... string[] specStrings = .... foreach (string specString in specStrings) { if (!MetricSpec.TryParse(specString, out MetricSpec spec)) { Log.Message($"Failed to parse metric spec: {specString}"); } else { Log.Message($"Parsed metric: {spec}"); .... } } } Location: MetricsEventSource.cs, 375 The return value of the TryParse method is used as the condition of the if statement. If specString cannot be parsed, the original value should be logged. Otherwise, the received representation (spec) is logged, and some operations are performed on it. The problem is TryParse always returns true. Thus, the then branch of the if statement is never executed — the parsing is always successful (link here to the Issue on GitHub). As a result of the fix, TryParse became Parse, and the caller method lost the if statement. The developers also changed Substring to AsSpan in TryParse. By the way, this is the same code fragment that I noted when digging in the .NET 6 source code. But back then, the interpolation character was missing in the logging method: C# if (!MetricSpec.TryParse(specString, out MetricSpec spec)) { Log.Message("Failed to parse metric spec: {specString}"); } else { Log.Message("Parsed metric: {spec}"); .... } You can read more about this issue in the article about .NET 6 check (issue 14). Issue 4 Since we mentioned methods with strange return values, let's look at another one: C# public virtual bool TryAdd(XmlDictionaryString value, out int key) { ArgumentNullException.ThrowIfNull(value); IntArray? keys; if (_maps.TryGetValue(value.Dictionary, out keys)) { key = (keys[value.Key] - 1); if (key != -1) { // If the key is already set, then something is wrong throw System.Runtime .Serialization .DiagnosticUtility .ExceptionUtility .ThrowHelperError( new InvalidOperationException(SR.XmlKeyAlreadyExists)); } key = Add(value.Value); keys[value.Key] = (key + 1); return true; // <= } key = Add(value.Value); keys = AddKeys(value.Dictionary, value.Key + 1); keys[value.Key] = (key + 1); return true; // <= } Location: XmlBinaryWriterSession.cs, 28 The method either returns true or throws an exception — it never returns false. This is a public API, so there's more demand for quality. Let's look at the description on learn.microsoft.com: Oopsie. I created an issue on GitHub for it as well, but at the moment of writing this article, there was no news on it. Issue 5 C# public static Attribute? GetCustomAttribute(ParameterInfo element, Type attributeType, bool inherit) { // .... Attribute[] attrib = GetCustomAttributes(element, attributeType, inherit); if (attrib == null || attrib.Length == 0) return null; if (attrib.Length == 0) return null; if (attrib.Length == 1) return attrib[0]; throw new AmbiguousMatchException(SR.RFLCT_AmbigCust); } Location: Attribute.CoreCLR.cs, 617 In this code fragment, the same expression — attrib.Length == 0 — is checked twice: first as a right operand of the || operator, then as a condition of the if statement. Sometimes this may be an error — developers want to check one thing but instead, check another. We were lucky here: the second check was just redundant and the developers removed it. See the issue on GitHub. Issue 6 C# protected virtual XmlSchema? GetSchema() { if (GetType() == typeof(DataTable)) { return null; } MemoryStream stream = new MemoryStream(); XmlWriter writer = new XmlTextWriter(stream, null); if (writer != null) { (new XmlTreeGen(SchemaFormat.WebService)).Save(this, writer); } stream.Position = 0; return XmlSchema.Read(new XmlTextReader(stream), null); } Location: DataTable.cs, 6678 The developer created an instance of the XmlTextWriter type. Then a reference to this instance is assigned to the writer variable. However, in the next line the developer checked writer for null. The check always returns true, which means the condition is redundant here. It's not horrific, but it's better to remove the check. The developers did that, actually (issue on GitHub). Issue 7 Redundant code again, but this time it's less obvious: C# public int ToFourDigitYear(int year, int twoDigitYearMax) { if (year < 0) { throw new ArgumentOutOfRangeException(nameof(year), SR.ArgumentOutOfRange_NeedPosNum); } if (year < 100) { int y = year % 100; return (twoDigitYearMax / 100 - (y > twoDigitYearMax % 100 ? 1 : 0)) * 100 + y; } .... } Location: GregorianCalendarHelper.cs, 526 Let's look at how the range of the year variable is checked throughout the code execution: C# ToFourDigitYear(int year, int twoDigitYearMax) year is a parameter of the int type. Which means its value is within the [int.MinValue; int.MaxValue] range. When the code is executed, the if statement is met first; in this statement, an exception is thrown: C# if (year < 0) { throw ....; } If there's no exception, then the year value is within [0; int.MaxValue]. Then, another if statement: C# if (year < 100) { int y = year % 100; .... } If the code execution is in the then branch of if, then the year value is within the [0; 99] range. This leads to an interesting result — to the operation of taking the remainder of the division: C# int y = year % 100; The year value is always less than 100 (i.e., the value is between 0-99). Therefore, the result of the year % 100 operation is always equal to the left operand — year. Thus, y is always equal to year. Either the code is redundant or it's an error. After I opened the issue on GitHub, the code was fixed and the y variable was removed. Issue 8 C# internal ConfigurationSection FindImmediateParentSection(ConfigurationSection section) { .... SectionRecord sectionRecord = .... if (sectionRecord.HasLocationInputs) { SectionInput input = sectionRecord.LastLocationInput; Debug.Assert(input.HasResult, "input.HasResult"); result = (ConfigurationSection)input.Result; } else { if (sectionRecord.HasIndirectLocationInputs) { Debug.Assert(IsLocationConfig, "Indirect location inputs exist only in location config record"); SectionInput input = sectionRecord.LastIndirectLocationInput; Debug.Assert(input != null); Debug.Assert(input.HasResult, "input.HasResult"); result = (ConfigurationSection)input.Result; } .... .... } Location: MgmtConfigurationRecord.cs, 341 We need to dig a bit deeper here. First, let's look at the second if statement: C# if (sectionRecord.HasIndirectLocationInputs) { Debug.Assert(IsLocationConfig, "Indirect location inputs exist only in location config record"); SectionInput input = sectionRecord.LastIndirectLocationInput; Debug.Assert(input != null); Debug.Assert(input.HasResult, "input.HasResult"); result = (ConfigurationSection)input.Result; } The value of the LastIndirectLocationInput property is written to the input variable. After that, input is checked in two asserts: it's checked for null (input != null) and for the presence of result (input.HasResult). Let's look at the LastIndirectLocationInput property's body to understand which value can be written to the input variable: C# internal SectionInput LastIndirectLocationInput => HasIndirectLocationInputs ? IndirectLocationInputs[IndirectLocationInputs.Count - 1] : null; On the one hand, the property may return null. On the other hand, if HasIndirectLocationInputs is true, then IndirectLocationInputs[IndirectLocationInputs.Count - 1] is returned instead of explicit null. The question is, can the value from the IndirectLocationInputs collection be null? Probably yes, although it's not clear from the code. By the way, nullable annotations could help here, but they are not enabled in all .NET projects. Let's go back to if: C# if (sectionRecord.HasIndirectLocationInputs) { Debug.Assert(IsLocationConfig, "Indirect location inputs exist only in location config record"); SectionInput input = sectionRecord.LastIndirectLocationInput; Debug.Assert(input != null); Debug.Assert(input.HasResult, "input.HasResult"); result = (ConfigurationSection)input.Result; } The conditional expression is sectionRecord.HasIndirectLocationInputs. It's the same property that's checked in LastIndirectLocationInput, which means LastIndirectLocationInput definitely doesn't return explicit null. However, it's unclear which value will be received from IndirectLocationInputs and written to input. The developer first checks that input != null and only then checks for the presence of the result — input.HasResult. Looks okay. Now let's go back to the first if statement: C# if (sectionRecord.HasLocationInputs) { SectionInput input = sectionRecord.LastLocationInput; Debug.Assert(input.HasResult, "input.HasResult"); result = (ConfigurationSection)input.Result; } Let's look at the LastLocationInput property: C# internal SectionInput LastLocationInput => HasLocationInputs ? LocationInputs[LocationInputs.Count - 1] : null; It's written the same way as LastIndirectLocationInput. Just like in the previous case, depending on the flag (HasLocationInputs), either null or a value from the LocationInputs collection is returned. Now return to the if statement. Its conditional expression is the HasLocationInputs property, which is checked within LastLocationInput. If the code is executed in the then branch of the if statement, this means LastLocationInput cannot return explicit null. Can the value from the LocationInputs collection be null? The question remains unanswered. If it can, then null will be written to input too. As in the case of the first inspected if, input.HasResult is checked but there's no input != null this time. Once again. The first inspected code fragment: C# SectionInput input = sectionRecord.LastIndirectLocationInput; Debug.Assert(input != null); Debug.Assert(input.HasResult, "input.HasResult"); result = (ConfigurationSection)input.Result; The second one: C# SectionInput input = sectionRecord.LastLocationInput; Debug.Assert(input.HasResult, "input.HasResult"); result = (ConfigurationSection)input.Result; Looks like the Debug.Assert(input != null) expression is missing. I opened an issue on GitHub where I described this and other suspicious places related to null checks (you'll see them below). The developers decided not to fix this fragment and left it as is: Issues With Null Checks I came across several places in code where a reference is dereferenced and only then it's checked for null. I created one issue for all similar code fragments on GitHub. Let's inspect. Issue 9 C# private static RuntimeBinderException BadOperatorTypesError(Expr pOperand1, Expr pOperand2) { // .... string strOp = pOperand1.ErrorString; Debug.Assert(pOperand1 != null); Debug.Assert(pOperand1.Type != null); if (pOperand2 != null) { Debug.Assert(pOperand2.Type != null); return ErrorHandling.Error(ErrorCode.ERR_BadBinaryOps, strOp, pOperand1.Type, pOperand2.Type); } return ErrorHandling.Error(ErrorCode.ERR_BadUnaryOp, strOp, pOperand1.Type); } Location: ExpressionBinder.cs, 798 First, pOperand1 is dereferenced (pOperand1.ErrorString) and is checked for null in Debug.Assert in the next code line. If pOperand1 is null, then the assert is not triggered, but an exception of the NullReferenceException type is thrown instead. The code was fixed — pOperand1 is checked before use. Before: C# string strOp = pOperand1.ErrorString; Debug.Assert(pOperand1 != null); Debug.Assert(pOperand1.Type != null); After: C# Debug.Assert(pOperand1 != null); Debug.Assert(pOperand1.Type != null); string strOp = pOperand1.ErrorString; Issue 10 C# public void Execute() { var count = _callbacks.Count; if (count == 0) { return; } List<Exception>? exceptions = null; if (_callbacks != null) { for (int i = 0; i < count; i++) { var callback = _callbacks[i]; Execute(callback, ref exceptions); } } if (exceptions != null) { throw new AggregateException(exceptions); } } Location: PipeCompletionCallbacks.cs, 20 The _callbacks variable is used first and only then it's checked for null: C# public void Execute() { var count = _callbacks.Count; .... if (_callbacks != null) .... } At the time of writing this article, the developers removed checking _callbacks for null. By the way, _callbacks is a readonly field that's initialized in a constructor: C# internal sealed class PipeCompletionCallbacks { private readonly List<PipeCompletionCallback> _callbacks; private readonly Exception? _exception; public PipeCompletionCallbacks(List<PipeCompletionCallback> callbacks, ExceptionDispatchInfo? edi) { _callbacks = callbacks; _exception = edi?.SourceException; } .... } In the thread with the fix, the developers discussed whether it was worth adding Debug.Assert and checking _callbacks for null into a constructor. In the end, they decided it wasn't. Issue 11 C# private void ValidateAttributes(XmlElement elementNode) { .... XmlSchemaAttribute schemaAttribute = (_defaultAttributes[i] as XmlSchemaAttribute)!; attrQName = schemaAttribute.QualifiedName; Debug.Assert(schemaAttribute != null); .... } Location: DocumentSchemaValidator.cs, 421 The controversial code: The result of the as operator is written to schemaAttribute. If _defaultAttributes[i] – null or the cast failed, the result will be null. The null-forgiving operator (!) implies that the result of casting cannot be null. Therefore, schemaAttribute cannot be null. In the next code line, schemaAttribute is dereferenced. Then in a line below, the reference is checked for null. Here's the question. Can schemaAttribute be null? It's not very clear from the code. The code was fixed like that: C# .... XmlSchemaAttribute schemaAttribute = (XmlSchemaAttribute)_defaultAttributes[i]!; attrQName = schemaAttribute.QualifiedName; .... During the discussion of the fix, the developer proposed moving the Debug.Assert call in the line above instead of removing it. The code would look like that: C# .... XmlSchemaAttribute schemaAttribute = (XmlSchemaAttribute)_defaultAttributes[i]!; Debug.Assert(schemaAttribute != null); attrQName = schemaAttribute.QualifiedName; .... In the end, they decided not to return Assert. Issue 12 Let's look at the constructor of the XmlConfigurationElementTextContent type: C# public XmlConfigurationElementTextContent(string textContent, int? linePosition, int? lineNumber) { .... } Location: XmlConfigurationElementTextContent.cs, 10 Now let's see where it's used: C# public static IDictionary<string, string?> Read(....) { .... case XmlNodeType.EndElement: .... var lineInfo = reader as IXmlLineInfo; var lineNumber = lineInfo?.LineNumber; var linePosition = lineInfo?.LinePosition; parent.TextContent = new XmlConfigurationElementTextContent(string.Empty, lineNumber, linePosition); .... break; .... case XmlNodeType.Text: .... var lineInfo = reader as IXmlLineInfo; var lineNumber = lineInfo?.LineNumber; var linePosition = lineInfo?.LinePosition; XmlConfigurationElement parent = currentPath.Peek(); parent.TextContent = new XmlConfigurationElementTextContent(reader.Value, lineNumber, linePosition); .... break; .... } Locations: XmlStreamConfigurationProvider.cs, 133 XmlStreamConfigurationProvider.cs, 148 Have you noticed anything strange in the code? Pay attention to the order of arguments and parameters: arguments: lineNumber, linePosition parameters: linePosition, lineNumber I created an issue on GitHub, and the code was fixed: the developers put arguments in the correct order and added a test. Issue 13 Another suspicious case: C# public virtual bool Nested { get {....} set { .... ForeignKeyConstraint? constraint = ChildTable.Constraints .FindForeignKeyConstraint(ChildKey.ColumnsReference, ParentKey.ColumnsReference); .... } } Location: DataRelation.cs, 486 Look at the FindForeignKeyConstraint method: C# internal ForeignKeyConstraint? FindForeignKeyConstraint(DataColumn[] parentColumns, DataColumn[] childColumns) { .... } Location: ConstraintCollection.cs, 548 Seems like the argument order is mixed up again: parameters: parent, child arguments: ChildKey, ParentKey There's another method call: the argument order is correct there. C# ForeignKeyConstraint? foreignKey = relation.ChildTable .Constraints .FindForeignKeyConstraint(relation.ParentColumnsReference, relation.ChildColumnsReference); I created an issue on GitHub. Unfortunately, I haven't received any comments on it at the moment of writing the article. Issue 14 These are not all places where the argument order is mixed up — I found another one: C# void RecurseChildren(....) { .... string? value = processValue != null ? processValue(new ConfigurationDebugViewContext( child.Key, child.Path, valueAndProvider.Value, valueAndProvider.Provider)) : valueAndProvider.Value; .... } Location: ConfigurationRootExtensions.cs, 50 Look at the ConfigurationDebugViewContext constructor: C# public ConfigurationDebugViewContext( string path, string key, string? value, IConfigurationProvider configurationProvider) { .... } Location: ConfigurationDebugViewContext.cs, 11 The order: parameters: path, key arguments: child.Key, child.Path I created an issue on GitHub. According to the developers, this case doesn't have any issues despite the mistake. However, they still fixed the order of arguments. Conclusion The .NET code is of high quality. I believe this is achieved by an established development process — the developers know the exact release date. Besides, release candidates help find the most serious errors and prepare the project for the release. Nevertheless, I still manage to find something intriguing in the code. This time my favorites are arguments that were mixed up during method calls. All code fragments described in this article were found by the PVS-Studio analyzer.

What Is a Kubernetes Service Mesh? A service mesh is a dedicated infrastructure layer for handling service-to-service communication in a distributed microservices architecture. It typically includes features such as service discovery, load balancing, routing, fault tolerance, and monitoring. It also provides a uniform way for services to communicate with each other. The goal of a service mesh is to reduce the complexity of managing communication between microservices and make it easier to scale and maintain a distributed system. A Kubernetes service mesh is deployed on top of a Kubernetes cluster and provides a way for services running on Kubernetes to communicate with each other in a reliable and scalable way. By using a service mesh, developers can focus on building and deploying their applications. This can make it easier to develop, deploy, and manage distributed systems on Kubernetes. A service mesh is typically implemented on Kubernetes using a sidecar proxy, which is a separate process that runs alongside each service in the mesh. The sidecar proxy is responsible for intercepting and directing traffic between the services in the mesh and enforcing the rules and policies defined in the service mesh configuration. A service mesh provides several advantages over other traffic management methods in terms of sustainability. It provides a uniform way for services to communicate with each other, which can make it easier to understand system behavior. This can make it easier to identify and troubleshoot problems and help prevent issues from arising. Can a Service Mesh Reduce Kubernetes Costs? A service mesh can help reduce costs in Kubernetes in several ways. First, a service mesh can make it easier to manage and maintain a distributed system, which can reduce the amount of time and effort required to keep the system running smoothly. This can help reduce the need for specialized staff and resources, which can lower overall costs. There are several critical features of a service mesh that can help reduce costs. These include: Observability: A service mesh provides detailed visibility into the behavior and performance of a distributed system. This can make it easier to identify and troubleshoot problems, which can help prevent downtime and other costly issues. Security: A service mesh provides features such as authentication, authorization, and encryption, which can help secure communication between services. This can help prevent security breaches and the associated costs. Centralized control: A service mesh provides a central point of control for managing service-to-service communication. This can make it easier to optimize and control the usage of resources in a distributed system, which can help reduce the overall cost of running the system. Resiliency: A service mesh provides features such as automatic retries, circuit breaking, and fault injection, which can improve the resiliency and robustness of a distributed system. This can help prevent downtime and other issues that can be costly to fix. Improved productivity: A service mesh can make it easier to deploy and scale applications on Kubernetes, which can help reduce the time and effort required to get new applications up and running. This can improve the speed and agility of development and deployment processes, which can improve overall productivity. What Are the Hidden Costs of Service Meshes? While a service mesh can provide many benefits, it can also introduce some hidden costs. These can include: Complexity: Although it simplifies communication between microservices, a service mesh adds an additional layer of complexity to a distributed system, which can make it more difficult to understand and manage. This can increase the time and effort required to develop, deploy, and maintain applications. Resource overhead: A service mesh requires additional resources such as CPU, memory, and network bandwidth to run and manage the mesh. This can increase the overall resource usage of a system, which can add to costs. Integration costs: A service mesh typically requires changes to be made to the application code to integrate with the mesh. This can add to the cost of developing and deploying applications. Important Considerations for Evaluating Service Mesh Costs How Many Images Do You Need To Run the Control Plane? The number of container images required to run a service mesh control plane can impact CPU usage and service mesh costs by increasing resource usage and complexity. It is important to carefully consider the number of container images used to run the control plane in order to optimize performance and minimize costs. What Is the Ingress Controller Capacity for the Service Mesh? The Ingress controller is responsible for managing incoming traffic to the service mesh, and it requires resources such as CPU and memory to do this effectively. The more traffic that is received by the service mesh, the more resources will be required to process that traffic. If the Ingress controller capacity is not sufficient to handle the amount of traffic received by the service mesh, this can lead to congestion and dropped traffic. This can cause delays and disruptions in service, which can impact the user experience and lead to lost revenue and increased costs. Will You Enable Autoscaling? Kubernetes autoscaling allows the number of replicas of a deployment to be automatically adjusted based on the observed CPU or memory usage of the deployment. This can help ensure that the deployment is always able to handle the workload and maintain a desired level of performance. However, this can also result in the deployment scaling up or down more frequently, which can increase the overall resource usage of the system and lead to higher costs. Are You Using Multi-Tenancy or Multiple Clusters? Multi-tenancy (isolating Kubernetes resources in large clusters) presents several challenges, including an increased burden on DevOps teams and configuration issues. A multi-cluster deployment can increase the complexity of managing and coordinating the service mesh across multiple clusters. This can require additional staff and resources. Conclusion In conclusion, a service mesh can provide many benefits for Kubernetes users, including improved observability, security, and productivity. However, it is important to carefully consider the potential costs of using a service mesh and ensure that the benefits outweigh these costs.

The world of technology is constantly evolving, and organizations of all sizes are looking for ways to stay competitive and drive innovation. Customer demands are changing, and so do the touchpoints. Unfortunately, being available at every touchpoint requires technology officers to invest in several technologies, which increases the complexity and cost of the organization’s technology infrastructure. One of the most popular approaches modern technology experts use to curb this is the MACH architecture. MACH stands for Microservices, APIs, Cloud, and Headless, and it is an architecture that enables companies to quickly develop, deploy, and scale applications. In this article, we will explore what MACH architecture is, its benefits, how it can support digital transformation, and tips for designing and implementing MACH architecture. Introduction to MACH Architecture MACH architecture is a modern approach to application development that enables organizations to build agile, scalable, and highly available applications. It is based on microservices, APIs, cloud, and headless architecture principles. Hence called MACH. Here, microservices are small, independent services that are loosely coupled and can be developed, deployed, and scaled independently. They allow organizations to quickly develop, deploy, and scale applications with minimal effort. APIs are used to enable applications to communicate with each other and access data from various sources. Cloud computing provides organizations with on-demand computing resources and allows them to scale applications quickly and easily. Headless architecture is a web-based architecture where the front-end and back-end of an application are decoupled. This allows organizations to quickly deploy and manage applications without worrying about the underlying infrastructure. MACH architecture allows organizations to quickly build, deploy, and scale applications without worrying about the underlying infrastructure. This makes it the perfect choice for organizations that are looking to drive digital transformation. Let’s know more advantages. The Benefits of MACH Architecture The benefits include: Agility MACH architecture enables organizations to quickly develop, deploy, and scale applications with minimal effort. This makes it perfect for organizations that need to move quickly to stay competitive. Scalability resources, allowing them to quickly and easily scale applications. This makes it easier for organizations to respond to changes in the market and customer demands. Cost Efficiency MACH architecture enables organizations to quickly develop and deploy applications without having to invest in additional hardware or software. This helps organizations save on costs and allows them to focus their resources on more strategic initiatives. Flexibility MACH architecture enables organizations to quickly develop, deploy, and scale applications without having to worry about the underlying infrastructure. This makes it easier for organizations to adapt to changes in the market and customer demands. Innovation MACH architecture enables organizations to quickly develop and deploy applications without having to worry about the underlying infrastructure. This allows organizations to focus their resources on developing new features and products to stay competitive. How MACH Architecture Supports Digital Transformation MACH architecture can support digital transformation by enabling organizations to quickly develop and deploy applications that are agile, scalable, and cost-effective. This allows organizations to respond quickly to market and customer demand changes. MACH architecture also enables organizations to quickly develop, deploy, and scale applications without worrying about the underlying infrastructure. This makes it easier for organizations to innovate and develop new features and products to stay competitive. Exploring MACH Architecture Best Practices There are a number of best approaches that organizations should follow when designing and implementing MACH architecture. These include: Design for Scalability Organizations should design their applications for scalability, so they can quickly respond to changes in the market and customer demands. This can be done by leveraging cloud computing to provide on-demand computing resources and utilizing API gateways to manage application traffic. Leverage Microservices Organizations should leverage microservices to develop, deploy, and scale applications fast. As stated before, this allows them to respond quickly to changes in the market and customer demands. Create a Secure Infrastructure Organizations should create a secure infrastructure to ensure that applications are secure and protected from potential threats. This can be done by leveraging encryption technologies, authentication protocols, and access control measures. Monitor and Optimize Performance Organizations should monitor and optimize the performance of their applications to ensure they are running efficiently. This can be done by leveraging monitoring tools and performance metrics. Following these best practices will help organizations ensure that their MACH architecture applications are secure, scalable, and performant. MACH Architecture Use Cases MACH architecture can be used for a variety of use cases, including: IoT MACH architecture is a new way to decentralize the Internet of Things (IoT). It enables secure, distributed, resilient, and trustless communication between devices and applications. The MACH architecture allows for automated transactions among multiple participants, enabling applications like decentralized finance (DeFi). With MACH, all data is stored in an immutable, open ledger that all participants in the network can verify. This ensures that data remains private and secure. The MACH architecture has numerous benefits for IoT networks, including increased scalability, improved privacy and security, cost savings on hardware and energy consumption, faster development cycles, and reduced latency due to its decentralized nature. With these advantages and more, MACH hopes to empower developers to create innovative applications that will revolutionize IoT networks and help lead us into a connected future. Data Analysis MACH Architecture makes data analysis and decision-making easier for organizations by utilizing distributed ledger technology to securely store and analyze large amounts of data. This allows for faster processing of data and the ability to make decisions in real time. In addition, this architecture allows organizations to quickly access and analyze large volumes of data without having to build complex systems or manage infrastructure. This reduces development time and cost while allowing organizations to make better-informed decisions quickly. Additionally, with the immutable nature of distributed ledgers, organizations can trust that their data is secure, private, and reliable. Hence, with MACH Architecture, organizations can benefit from a robust system that allows them to leverage their data for improved decision-making. Mobile The MACH Architecture is designed to simplify and accelerate the mobile development process. It eliminates the need to build separate mobile applications for each platform, allowing developers to focus on creating a single, cross-platform application. By leveraging cloud computing and other technologies, developers can create a unified mobile experience that can be deployed quickly across multiple platforms. Additionally, MACH enables developers to have more control over their data and application lifecycles while simplifying user authentication and providing secure connections between devices. Ultimately, MACH makes it easier for organizations to develop powerful mobile applications quickly in order to meet their business objectives. Automation MACH Architecture can be used to develop applications that enable organizations to automate complex business processes quickly. This architecture simplifies the development process by leveraging distributed ledger technology to store and analyze data. This allows for faster processing of data and the ability to make decisions in real time. Additionally, applications built using MACH architecture can leverage blockchain technology for secure data storage and cryptographic authentication for secure transactions. By utilizing these technologies, organizations can easily build automated processes that are secure, reliable, and efficient. This can reduce development costs and improve decision-making, as organizations are able to access large amounts of data quickly and accurately. Furthermore, integrating MACH architecture with other technologies, such as artificial intelligence (AI), can further improve the efficiency of automation processes, allowing companies to gain a competitive edge in their respective industries. Challenges With MACH Architecture While MACH architecture provides organizations with several benefits, organizations should be aware of a number of challenges. These include: Security: MACH architecture can present security challenges if organizations do not have the necessary security measures in place. This can include authentication protocols, encryption technologies, and access control measures. Complexity: MACH architecture can be complex to design and implement, especially for organizations that are new to developing applications. This can lead to longer development times and higher costs. Integration: MACH architecture can be difficult to integrate with existing systems, leading to longer development times and higher costs. Testing: MACH architecture can be difficult to test and can require organizations to invest in additional testing tools and processes. Tips for Designing and Implementing MACH Architecture MACH architecture can be a powerful tool for driving digital transformation, but organizations should take the time to design and implement their applications properly. Here are some tips for designing and implementing MACH architecture: Start Small Organizations should start small and focus on developing a single application at a time. This will help organizations get familiar with MACH architecture and develop applications that are secure, reliable, and performant. Focus on Scalability Organizations should design their applications for scalability, so they can quickly respond to changes in the market and customer demands. This can be done by leveraging cloud computing to provide on-demand computing resources and utilizing API gateways to manage application traffic. Test Thoroughly Organizations should thoroughly test their applications to ensure they are secure, reliable, and performant. They should leverage automated testing tools and simulate real-world scenarios to make it all happen. Monitor Performance Organizations should monitor the performance of their applications to ensure they are running efficiently. This can be done by leveraging monitoring tools and performance metrics. By following these tips, organizations can ensure that their MACH architecture applications are secure, reliable, and performant. How to Measure the Success of Your MACH Architecture Organizations should measure the success of their MACH architecture applications to ensure that they are meeting their goals. Here are some metrics that organizations can use to measure the success of their MACH architecture applications: Time to Market: Organizations should measure the time it takes for them to develop and deploy applications. This will help them determine if they are meeting their development goals. Uptime: Organizations should measure the uptime of their applications to ensure they are meeting their availability goals. Response Times: Organizations should measure the response times of their applications to ensure they are meeting their performance goals. Customer Satisfaction: Organizations should measure customer satisfaction to ensure they are meeting their customer experience goals. By measuring the success of their MACH architecture applications, organizations can ensure that they are meeting their goals and staying competitive. Conclusion MACH architecture is a powerful approach to application development that enables organizations to develop, deploy, and scale applications without having to worry about the underlying infrastructure. This makes it the perfect choice for organizations that are looking to drive digital transformation. By following the best practices discussed in the article, organizations can ensure that their MACH architecture applications are secure, reliable, and performant.

I’ve been a longtime advocate of pull request templates. These templates allow you to provide instructions to developers creating pull requests in your repos so that they will be reminded to include all the relevant information you need to properly review their code. Pull request templates are a great place to include checklists as additional reminders. These might include to-dos like remembering to write unit tests, self-reviewing your code, or doing a security audit. Sometimes though, these checklists can get long. You may have so many important items that developers begin to ignore them. Or perhaps some of the checklist items are only relevant in specific situations or when certain files are changed. While each item is important, they may not all always be applicable. Wouldn’t it be nice if your pull request checklists could be dynamic and only show relevant reminders? CodeSee Code Automation can help you do just that! With Code Automation, you can configure rule-based triggers to automatically add checklist comments to the pull request when certain conditions are met. In this article, we’ll look at some of CodeSee’s Code Automation templates and show how we can incorporate one of them into a codebase of our own. Ready to get started? Automation Templates There are several different automation templates on the CodeSee site that we can use as our inspiration. For example, there are checklists for adding icons, creating a migration, adding a new npm package, or creating a new API endpoint. For our demo, we’ll explore the icon checklist and modify it to meet our needs. Our Fictional Icon Repo Screenshot of our demo icon library Let’s imagine that we have a repo containing hundreds of icons for developers at our company. When a developer needs a new icon not found in our icon library, they talk to a designer, and the designer creates an SVG file for the developer. The developer then adds the SVG file to the code repo. Simple enough, right? The problem is that the SVG file from the designer is often messy. Usually, these files will have random ID attributes included. There might be extra comments or unnecessary code in the file. The file name will probably not follow the same pattern as the other icon files. And, there might be additional code setup needed besides simply committing the new SVG file. The developer may need to update a master list where the icons are exported or update some documentation or Storybook examples. Some developers might just add the SVG file without giving any of these additional steps any thought. The solution, then, is to have a checklist that reminds developers what they need to do when adding a new icon. This checklist could exist in a README or in a pull request template, and I’d highly recommend you do both of these things. But even then, developers sometimes just don’t read the docs. They ignore the instructions and blindly press forward. To solve this, let’s add a Code Automation Trigger so that the developer is freshly reminded with a checklist comment when they create their pull request. Adding the Icon Checklist Automation Template To begin, we’ll first need to give CodeSee access to our GitHub repo from the Settings page. Connect CodeSee to your GitHub repo Once our repo is added, we can navigate to the Automate page to create our first Trigger. Add your first Trigger We’ll click the Add Trigger button to access the New Trigger page. New Trigger page On this page, we can add a name and repo to our Trigger and then specify the conditions and the actions. In our case, we want the condition to be whenever a new file is added to the "src/icons" directory. Add Conditions for your Trigger When that condition is met, we want CodeSee to automatically comment on the pull request with a few checklist items. We’ve added three items here, but you can add as many items as you’d like. Add Actions for your Trigger After we’re done editing, we can click the Save button to save our changes. With that, we’ve just created our first dynamic checklist! Let’s see this automation in action. In our code repo, let’s add a new SVG file to the "src/icons" directory. Then let’s create a new branch and submit a pull request to merge our branch into the master branch. Once our pull request is created, you’ll notice that the CodeSee bot automatically comments on our pull request with a checklist of reminders. Good thing, too, because we forgot to do these things! The CodeSee bot has added a comment on the pull request Conclusion That’s it! In just a few short minutes, we’ve added a checklist to help our developers have a better experience when adding icons. We’ve made code expectations clear and highlighted possible items that the developer may have missed. And, best of all, we’ve added these comments dynamically, only showing them when actually needed. If a developer were to make a different contribution to this repo that didn’t involve adding a new icon, they wouldn’t see these comments at all. The cool thing is that this is just the start. We can add all sorts of other automation templates to our repo. And, these Triggers can do more than just add checklist comments. They also allow you to automatically assign reviewers to the pull request in the event that the expertise of certain developers or teams is needed. With CodeSee Code Automation, we can enforce best practices and share knowledge effortlessly.

Microservices are the trend of the hour. Businesses are moving towards cloud-native architecture and breaking their large applications into smaller, self-independent modules called microservices. This architecture gives a lot more flexibility, maintainability, and operability, not to mention better customer satisfaction. With these added advantages, architects and operations engineers face many new challenges as well. Earlier, they were managing one application; now they have to manage many instead. Each application again needs its own support services, like databases, LDAP servers, messaging queues, and so on. So the stakeholders need to think through different strategies for deployment where the entire application can be well deployed while maintaining its integrity and providing optimal performance. Deployment Patterns The microservices architects suggest different types of patterns that could be used to deploy microservices. Each design provides solutions for diverse functional and non-functional requirements. So microservices could be written in a variety of programming languages or frameworks. Again, they could be written in different versions of the same programming language or framework. Each microservice comprises several different service instances, like the UI, DB, and backend. The microservice must be independently deployable and scalable. The service instances must be isolated from each other. The service must be able to quickly build and deploy itself. The service must be allocated proper computing resources. The deployment environment must be reliable, and the service must be monitored. Multiple Service Instances per Host To meet the requirements mentioned at the start of this section, we can think of a solution with which we can deploy service instances of multiple services on one host. The host may be physical or virtual. So, we are running many service instances from different services on a shared host. There are different ways we could do it. We can start each instance as a JVM process. We can also start multiple instances as part of the same JVM process, kind of like a web application. We can also use scripts to automate the start-up and shutdown processes with some configurations. The configuration will have different deployment-related information, like version numbers. With this kind of approach, the resources could be used very efficiently. Service Instance per Host In many cases, microservices need their own space and a clearly separated deployment environment. In such cases, they can’t share the deployment environment with other services or service instances. There may be a chance of resource conflict or scarcity. There might be issues when services written in the same language or framework but with different versions can’t be co-located. In such cases, a service instance could be deployed on its own host. The host could either be a physical or virtual machine. In such cases, there wouldn’t be any conflict with other services. The service remains entirely isolated. All the resources of the VM are available for consumption by the service. It can be easily monitored. The only issue with this deployment pattern is that it consumes more resources. Service Instance per VM In many cases, microservices need their own, self-contained deployment environment. The microservice must be robust and must start and stop quickly. Again, it also needs quick upscaling and downscaling. It can’t share any resources with any other service. It can’t afford to have conflicts with other services. It needs more resources, and the resources must be properly allocated to the service. In such cases, the service could be built as a VM image and deployed in a VM. Scaling could be done quickly, as new VMs could be started within seconds. All VMs have their own computing resources that are properly allocated according to the needs of the microservice. There is no chance of any conflict with any other service. Each VM is properly isolated and can get support for load balancing. Service Instance per Container In some cases, microservices are very tiny. They consume very few resources for their execution. However, they need to be isolated. There must not be any resource sharing. They again can’t afford to be co-located and have a chance of conflict with another service. It needs to be deployed quickly if there is a new release. There might be a need to deploy the same service but with different release versions. The service must be capable of scaling rapidly. It also must have the capacity to start and shut down in a few milliseconds. In such a case, the service could be built as a container image and deployed as a container. In that case, the service will remain isolated. There would not be any chance of conflict. Computing resources could be allocated as per the calculated need of the service. The service could be scaled rapidly. Containers could also be started and shut down quickly. Serverless Deployment In certain cases, the microservice might not need to know the underlying deployment infrastructure. In these situations, the deployment service is contracted out to a third-party vendor, who is typically a cloud service provider. The business is absolutely indifferent about the underlying resources; all it wants to do is run the microservice on a platform. It pays the service provider based on the resources consumed from the platform for each service call. The service provider picks the code and executes it for each request. The execution may happen in any executing sandbox, like a container, VM, or whatever. It is simply hidden from the service itself. The service provider takes care of provisioning, scaling, load-balancing, patching, and securing the underlying infrastructure. Many popular examples of serverless offerings include AWS Lambda, Google Functions, etc. The infrastructure of a serverless deployment platform is very elastic. The platform scales the service to absorb the load automatically. The time spent managing the low-level infrastructure is eliminated. The expenses are also lowered as the microservices provider pays only for the resources consumed for each call. Service Deployment Platform Microservices can also be deployed on application deployment platforms. By providing some high-level services, such platforms clearly abstract out the deployment. The service abstraction can handle non-functional and functional requirements like availability, load balancing, monitoring, and observability for the service instances. The application deployment platform is thoroughly automated. It makes the application deployment quite reliable, fast, and efficient. Examples of such platforms are Docker Swarm, Kubernetes, and Cloud Foundry, which is a PaaS offering. Conclusion Microservices deployment options and offerings are constantly evolving. It's possible that a few more deployment patterns will follow suit. Many of these patterns mentioned above are very popular and are being used by most microservice providers. They are very successful and reliable. But with changing paradigms, administrators are thinking of innovative solutions.

Microservices today are often deployed on a platform such as Kubernetes, which orchestrates the deployment and management of containerized applications. Microservices, however, don't exist in a vacuum. They typically communicate with other services, such as databases, message brokers, or other microservices. Therefore, an application usually consists of multiple services that form a complete solution. But, as a developer, how do you develop and test an individual microservice that is part of a larger system? This article examines some common inner-loop development cycle challenges and shows how Quarkus and other technologies help solve some of these challenges. What Is the Inner Loop? Almost all software development is iterative. The inner loop contains everything that happens on a developer's machine before committing code into version control. The inner loop is where a developer writes code, builds and tests it, and perhaps runs the code locally. In today's world, the inner loop could also include multiple commits to a Git pull request, where a developer may commit multiple times against a specific feature until that feature is deemed complete. Note: The word local is also up for debate in industry today as more and more remote development environments, such as Red Hat OpenShift Dev Spaces, Gitpod, and GitHub Codespaces are available. This article does not differentiate between a developer machine and any of these kinds of environments. They are all viewed as local in this article. Inner loop shifts to outer loop when code reaches a point in source control where it needs to be built, tested, scanned, and ultimately deployed by automated continuous integration and deployment (CI/CD) processes. Figure 1 illustrates a simple inner loop and outer loop. Figure 1: The inner loop takes place on a developer's local machine, whereas the outer loop takes place within CI/CD processes. Challenges of Inner Loop Development Developing a single microservice in isolation is challenging enough without worrying about additional downstream services. How do you run a microservice in isolation on your local machine if it depends on other services for it to function properly? Using various mocking techniques, you can to some extent get around the absence of required services when writing and running tests. Mocking techniques generally work great for testing. You can also use in-memory replacements for required services, such as an H2 database instead of a separate database instance. Beyond that, if you want or need to run the application locally, you need a better solution. Of course, you could try to reproduce your application's entire environment on your development machine. But even if you could, would you really want to? Do you think a developer at Twitter or Netflix could reproduce their environment on their development machine? Figure 2 shows the complexity of their architectures. Lyft also tried this approach and found it wasn't feasible or scalable. Figure 2 (courtesy of https://future.com/the-case-for-developer-experience): Major services such as Netflix and Twitter can easily have more than 500 microservices. Container-based Inner Loop Solutions Using containers can help speed up and improve the inner loop development lifecycle. Containers can help isolate and provide a local instance of a dependent service. We'll look at a few popular tools and technologies for the inner loop. Docker Compose One common pattern is to use Docker Compose to run some of your microservice's dependent services (databases, message brokers, etc.) locally while you run, debug, and test your microservice. With Docker Compose, you define a set of containerized services that provide the capabilities required by your microservice. You can easily start, stop, and view logs from these containerized services. However, there are a few downsides to using Docker Compose. First, you must maintain your Docker Compose configuration independently of your application's code. You must remember to make changes to the Docker Compose configuration as your application evolves, sometimes duplicating configuration between your application and your Docker Compose configuration. Second, you are locked into using the Docker binary. For Windows and macOS users, Docker Desktop is no longer free for many non-individual users. You are also prevented from using other container runtimes, such as Podman. Podman does support Docker Compose, but it doesn't support everything you can do with Docker Compose, especially on non-Linux machines. Testcontainers Testcontainers is an excellent library for creating and managing container instances for various services when applications run tests. It provides lightweight, throwaway instances of common databases, message brokers, or anything else that can run in a container. But Testcontainers is only a library. That means an application must incorporate it and do something with it to realize its benefits. Generally speaking, applications that use Testcontainers do so when executing unit or integration tests, but not in production. A developer generally won't include the library in the application's dependencies because Testcontainers doesn't belong as a dependency when the application is deployed into a real environment with real services. Quarkus Quarkus is a Kubernetes-native Java application framework focusing on more than just feature sets. In addition to enabling fast startup times and low memory footprints compared to traditional Java applications, Quarkus ensures that every feature works well, with little to no configuration, in a highly intuitive way. The framework aims to make it trivial to develop simple things and easy to develop more complex ones. Beyond simply working well, Quarkus aims to bring Developer Joy, specifically targeting the inner loop development lifecycle. Dev Mode The first part of the Quarkus Developer Joy story, live coding via Quarkus dev mode, improves and expedites the inner loop development process. When Quarkus dev mode starts, Quarkus automatically reflects code changes within the running application. Therefore, Quarkus combines the Write Code, Build, and Deploy/Run steps of Figure 1's inner loop into a single step. Simply write code, interact with your application, and see your changes running with little to no delay. Dev Services A second part of the Quarkus Developer Joy story, Quarkus Dev Services, automatically provisions and configures supporting services, such as databases, message brokers, and more. When you run Quarkus in dev mode or execute tests, Quarkus examines all the extensions present. Quarkus then automatically starts any unconfigured and relevant service and configures the application to use that service. Quarkus Dev Services uses Testcontainers, which we've already discussed, but in a manner completely transparent to the developer. The developer does not need to add the Testcontainers libraries, perform any integration or configuration, or write any code. Furthermore, Dev Services does not affect the application when it is deployed into a real environment with real services. Additionally, if you have multiple Quarkus applications on your local machine and run them in dev mode, by default, Dev Services attempts to share the services between the applications. Sharing services is beneficial if you work on more than one application that uses the same service, such as a message broker. Let's use the Quarkus Superheroes sample application as an example. The application consists of several microservices that together form an extensive system. Some microservices communicate synchronously via REST. Others are event-driven, producing and consuming events to and from Apache Kafka. Some microservices are reactive, whereas others are traditional. All the microservices produce metrics consumed by Prometheus and export tracing information to OpenTelemetry. The source code for the application is on GitHub under an Apache 2.0 license. The system's architecture is shown in Figure 3. Figure 3: The Quarkus Superheroes application has many microservices and additional dependent services. In the Quarkus Superheroes sample application, you could start both the rest-fights and event-statistics services locally in dev mode. The rest-fights dev mode starts a MongoDB container instance, an Apicurio Registry container instance, and an Apache Kafka container instance. The event-statistics service also requires an Apicurio Registry instance and an Apache Kafka instance, so the instance started by the rest-fights dev mode will be discovered and used by the event-statistics service. Continuous Testing A third part of the Quarkus Developer Joy story, continuous testing, provides instant feedback on code changes by immediately executing affected tests in the background. Quarkus detects which tests cover which code and reruns only the tests relevant to that code as you change it. Quarkus continuous testing combines testing with dev mode and Dev Services into a powerful inner loop productivity feature, shrinking all of Figure 1's inner loop lifecycle steps into a single step. Other Inner Loop Solutions The solutions we've outlined thus far are extremely helpful with local inner loop development, especially if your microservice requires only a small set of other services, such as a database, or a database and message broker. But when there are lots of dependent services, trying to replicate them all on a local machine probably won't work well, if at all. So what do you do? How do you get the speed and agility of inner loop development for an application when it depends on other services that you either can't or don't want to run locally? One solution could be to manage an environment of shared services. Each developer would then configure those services in their local setup, careful not to commit the configuration into source control. Another solution could be to use Kubernetes, giving each developer a namespace where they can deploy what they need. The developer could then deploy the services and configure their local application to use them. Both of these solutions could work, but in reality, they usually end up with a problem: The microservice the developer is working on is somewhere in the graph of services of an overall system. How does a developer trigger the microservice they care about to get called as part of a larger request or flow? Wouldn't a better solution be to run the application locally, but make the larger system think the application is actually deployed somewhere? This kind of remote + local development model is becoming known as remocal. It is an extremely powerful way to get immediate feedback during your inner loop development cycle while ensuring your application behaves properly in an environment that is close to or matches production. Quarkus Remote Development Another part of the Quarkus Developer Joy story, remote development, enables a developer to deploy an application into a remote environment and run Quarkus dev mode in that environment while doing live coding locally. Quarkus immediately synchronizes code changes to the remotely deployed instance. Quarkus remote development allows a developer to develop the application in the same environment it will run in while having access to the same services it will have access to. Additionally, this capability greatly reduces the inner feedback loop while alleviating the "works on my machine" problem. Remote development also allows for quick and easy prototyping of new features and capabilities. Figure 4 illustrates how the remote development mode works. Figure 4: Quarkus remote dev mode incrementally synchronizes local code changes with a remote Quarkus application. First, the application is deployed to Kubernetes, a virtual machine, a container, or just some Java virtual machine (JVM) somewhere. Once running, the developer runs the remote development mode on their local machine, connecting their local machine to the remote instance. From there, development is just like live coding in Quarkus dev mode. As the developer makes code changes, Quarkus automatically compiles and pushes the changes to the remote instance. Let's continue with the Quarkus Superheroes example from before. Let's assume the entire system is deployed into a Kubernetes cluster. Let's also assume you want to make changes to the rest-fights microservice. As shown in Figure 5, you start the rest-fights microservice in remote dev mode on your local machine. The rest-fights application running on the cluster connects to the MongoDB, Apicurio Registry, and Apache Kafka instances on the Kubernetes cluster. Figure 5: Changes to the local application in remote development mode continuously send updates to the remote instance. You can then interact with the system through its user interface. Quarkus incrementally synchronizes the changes with the remote instance on the Kubernetes cluster as you make changes to the rest-fights microservice. If you want, you could even use breakpoints within your IDE on your local machine to assist with debugging. Skupper Skupper is a layer 7 service interconnect that enables secure communication across Kubernetes clusters without VPNs or special firewall rules. Using Skupper, an application can span multiple cloud providers, data centers, and regions. Figure 6 shows a high-level view of Skupper. Figure 6: Logically, Skupper connects services on different sites together to exist as a single site. With Skupper, you can create a distributed application comprised of microservices running in different namespaces within different Kubernetes clusters. Services exposed to Skupper are subsequently exposed to each namespace as if they existed in the namespace. Skupper creates proxy endpoints to make a service available within each of the namespaces where it is installed. Figure 7 shows a logical view of this architecture. Figure 7: Logically, Skupper can span multiple Kubernetes clusters and make remote services appear as local ones. Why do we mention Skupper in an article about Kubernetes native inner loop development? Because in addition to bridging applications across Kubernetes clusters, a Skupper proxy can run on any machine, enabling bidirectional communication between the machine and the other Kubernetes clusters. Logically, this is like a local machine inserted into the middle of a set of Kubernetes clusters. Services exposed to Skupper on the clusters can discover services exposed to the Skupper proxy on the local machine and vice versa. Skupper can make our Quarkus Superheroes example even more interesting, taking it further from the remote development scenario we described earlier. With Skupper, rather than continuously synchronizing changes to the rest-fights service from a local instance to a remote instance, you could completely replace the remote rest-fights instance with a local instance running Quarkus dev mode and continuous testing. Skupper would then redirect traffic on the Kubernetes cluster into the rest-fights service running on your local machine. Any outgoing requests made by the rest-fights service, such as connections to the MongoDB, Apicurio registry, and Apache Kafka instances, and even the rest-heroes and rest-villains services, would then be redirected back to the Kubernetes cluster. Figure 8 shows a logical view of what this architecture might look like. Figure 8: Logically, Skupper can make it look like a local developer machine is inside a Kubernetes cluster. You could even use Quarkus dev services to allow the rest-fights microservice to provide its own local MongoDB instance rather than using the instance on the cluster, yet continue to let traffic to Kafka flow onto the cluster. This setup would enable other Kafka consumers listening on the same topic to continue functioning. In this scenario, Quarkus continuously runs the tests of the rest-fights microservice while a developer makes live code changes, all while traffic is continually flowing through the whole system on the Kubernetes cluster. The services could even be spread out to other Kubernetes clusters on different cloud providers in other regions of the world while traffic continues to flow through a developer's local machine. A Better Developer Experience, Whether Local or Distributed Parts two and three of the previously mentioned article series at Lyft show Lyft's approach to solving this problem, albeit using different technologies. As more and more services came to life, Lyft saw that what they were doing wasn't scaling and that they, therefore, needed a kind of "remocal" environment. Quarkus was designed with many of these Developer Joy characteristics in mind. Quarkus helps developers iterate faster and contains built-in capabilities that alleviate many of these challenges and shorten the development lifecycles. Developers can focus on writing code.