A microservices architecture is a development method for designing applications as modular services that seamlessly adapt to a highly scalable and dynamic environment. Microservices help solve complex issues such as speed and scalability, while also supporting continuous testing and delivery. This Zone will take you through breaking down the monolith step by step and designing a microservices architecture from scratch. Stay up to date on the industry's changes with topics such as container deployment, architectural design patterns, event-driven architecture, service meshes, and more.
A Look Into Netflix System Architecture
You Can Shape Trend Reports: Participate in DZone Original Research + Enter the Prize Drawings!
Deploying microservices in a Kubernetes cluster is critical in 5G Telecom. However, it also introduces significant security risks. While firewall rules and proxies provide initial security, the default communication mechanisms within Kubernetes, such as unencrypted network traffic and lack of access control, are inherently insecure. This insecurity could compromise sensitive data. Therefore, implementing additional security measures within each microservice pod is not just a recommendation but a crucial step to ensure secure communication within the cluster. So, additional configuration inside each application is needed. Istio provides a robust solution to these challenges by effectively managing communication between individual 5G telecom microservice pods. With its control plane, Istio automatically injects a sidecar proxy into individual microservices pods, ensuring secure and efficient communication. Let's dive deep. What Is Istio? Istio is a crucial open-source service mesh that seamlessly integrates with microservices-based applications, simplifying monitoring, management, and enforcing performance and security policies. It prevents overload, restricts unauthorized access, and secures data in transit. Its support system unifies and ensures smooth operations for microservices, significantly streamlining their management and ensuring performance and security requirements are met effortlessly. What Is a Sidecar Proxy? A sidecar proxy is a separate container that runs alongside a Kubernetes microservice pod. It is responsible for offloading functions required by all applications within Istio. The sidecar proxy, a powerful component of Istio Architecture, intercepts the application's incoming and outgoing network traffic. It enables telecom operators to apply policies and utilize the mentioned resiliency features, and it empowers operators to perform advanced functions at the interface point with the outside world, showcasing the capabilities of Istio Architecture. Architecture The backbone of Istio's architecture is significantly shaped by two crucial components, each playing a pivotal role in its functionality: The data plane, a pivotal part of Istio's architecture, comprises a set of proxies (deployed using Envoy, an open-source proxy for distributed applications) that run alongside microservices as sidecar containers. The control plane, a decisive element in Istio's architecture, manages the proxies and dictates their actions. Let's look at each component in more detail. It includes the following components: Pilot: Manages service discovery and traffic. Citadel: Manages security and enables secure communication. Galley: Validates and distributes configuration resources. Mixer: Handles policy enforcement and telemetry collection. Sidecar Injector: Automatically injects Envoy sidecar proxies into Kubernetes pods for easy integration. To explain how Istio Architecture works, we’ll use the example based on the above architecture diagram, the sidecar proxy deployed with Microservice-A and Microservice-B ensures seamless and efficient communication. The sidecar proxy intercepts network traffic, empowering the application to implement and enforce policies, utilize resiliency features, and enable advanced functions. When Microservice-A sends a request to Microservice-B, the sidecar proxy identifies the destination, forwards the request, and checks the service-to-service communication policy to determine if the call should go through based on security, performance, and reliability. This process of intercepting, forwarding, and checking ensures that the request is handled appropriately. If the request goes through, Microservice-B processes the request, prepares the response, and sends it back over the network, which is intercepted and forwarded by the sidecar proxy to the client and then to the destination application, Microservice A. Understanding the Importance of Istio Service Mesh for Kubernetes Microservices The Istio service mesh is essential in Kubernetes. While Kubernetes manages microservices, it doesn't handle traffic flow management, access policies, or telemetry data collection. Istio provides these capabilities without requiring changes to application code, making it an attractive solution for managing microservices in Kubernetes using sidecar containers. It can run in any distributed environment, providing a secure solution for cloud or on-premises applications. Istio supports Kubernetes distributions, including managed services like EKS and self-managed clusters. It also works with different application orchestration platforms and all microservices applications, including serverless architectures. Advantages of Istio Istio offers several critical benefits for Kubernetes and Istio-compatible platforms: Security: Enforces strong authentication and authorization requirements between microservices. Application performance: Efficiently routes traffic between microservices and handles retries and failovers. Observability: Collects telemetry data from individual microservices for detailed visibility into health and performance. Troubleshooting: Monitors each microservice individually to identify and address performance and security issues. Overall, Istio simplifies management for admins of modern, microservices-based applications. Configuration YAMLs (Yet Another Markup Language) The Service Mesh Control Plane manages proxies to route traffic, provides policy and configuration for data planes, and empowers administrators to define and configure various services. Once configured, the SMCP distributes necessary information to the service mesh's data plane, allowing proxies to dynamically adapt their behavior. Telecom Operators can install and run SMCP (Service Mesh Control Plane) using the configuration below: SMCP YAML YAML apiVersion: maistra.io/v2 kind: ServiceMeshControlPlane metadata: name: full-install namespace: istio-system spec: version: v2.1 techPreview: meshConfig: defaultConfig: concurrency: 8 # Adjust according to the need proxy: runtime: container: resources: requests: cpu: 500m memory: 256Mi limits: # Adjust according to the need cpu: "1" memory: 1Gi tracing: sampling: 10000 # 0.01% increments. 10000 samples 100% of traces type: Jaeger gateways: ingress: # istio-ingressgateway service: type: ClusterIP ports: - name: status-port port: 15020 - name: http2 port: 80 targetPort: 8080 - name: https port: 443 targetPort: 8443 meshExpansionPorts: [] egress: # istio-egressgateway service: type: ClusterIP ports: - name: status-port port: 15020 - name: http2 port: 80 targetPort: 8080 - name: https port: 443 targetPort: 8443 additionalIngress: some-other-ingress-gateway: {} additionalEgress: some-other-egress-gateway: {} policy: type: Istiod telemetry: type: Istiod addons: grafana: enabled: true kiali: name: kiali enabled: true install: # install kiali CR if not available dashboard: viewOnly: false enableGrafana: true enableTracing: true enablePrometheus: true jaeger: name: jaeger-production install: storage: type: Elasticsearch elasticsearch: nodeCount: 3 redundancyPolicy: SingleRedundancy indexCleaner: enabled: true numberOfDays: 7 schedule: 55 23 * * * ingress: enabled: true runtime: components: tracing.jaeger.elasticsearch: # only supports resources and image name container: resources: limits: cpu: 1 memory: 1Gi requests: cpu: 500m memory: 1Gi pilot: deployment: autoScaling: enabled: true minReplicas: 2 maxReplicas: 2 targetCPUUtilizationPercentage: 85 pod: tolerations: - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 60 affinity: podAntiAffinity: requiredDuringScheduling: - key: istio topologyKey: kubernetes.io/hostname operator: In values: - pilot container: resources: limits: # Adjust according to the need cpu: "1" memory: 1Gi The Service Mesh Member Roll unequivocally identifies the projects associated with the Service Mesh control plane. Solely, projects enlisted on the roll are impacted by the control plane. Adding a project to the member roll links it to a specific control plane deployment. Telecom Operators can install and run SMMR (Service Mesh Member Roll) using the configuration below: SMMR YAML YAML apiVersion: maistra.io/v1 kind: ServiceMeshMemberRoll metadata: name: default namespace: istio-system spec: members: - <Micro services pods namespace> # namespace that needs be istio injected. Installation 1. Install the SMCP (Service Mesh Control Plane) as below. 2. Install the SMMR (Service Mesh Member Roll) as below. Conclusion Istio simplifies communication between 5G telecom microservices pods in a Kubernetes environment and enables seamless connectivity, control, monitoring, and security of microservice architectures across different platforms. It supports workloads in containers and virtual machines. With Istio, the future of Telecom IoT microservice pod architecture looks promising, with improved efficiency, security, and scalability.
With Spring Boot 3.2 and Spring Framework 6.1, we get support for Coordinated Restore at Checkpoint (CRaC), a mechanism that enables Java applications to start up faster. With Spring Boot, we can use CRaC in a simplified way, known as Automatic Checkpoint/Restore at startup. Even though not as powerful as the standard way of using CRaC, this blog post will show an example where the Spring Boot applications startup time is decreased by 90%. The sample applications are from chapter 6 in my book on building microservices with Spring Boot. Overview The blog post is divided into the following sections: Introducing CRaC, benefits, and challenges Creating CRaC-based Docker images with a Dockerfile Trying out CRaC with automatic checkpoint/restore Summary Next blog post Let’s start learning about CRaC and its benefits and challenges. 1. Introducing CRaC, Benefits, and Challenges Coordinated Restore at Checkpoint (CRaC) is a feature in OpenJDK, initially developed by Azul, to enhance the startup performance of Java applications by allowing them to restore to a previously saved state quickly. CRaC enables Java applications to save their state at a specific point in time (checkpoint) and then restore from that state at a later time. This is particularly useful for scenarios where fast startup times are crucial, such as serverless environments, microservices, and, in general, applications that must be able to scale up their instances quickly and also support scale-to-zero when not being used. This introduction will first explain a bit about how CRaC works, then discuss some of the challenges and considerations associated with it, and finally, describe how Spring Boot 3.2 integrates with it. The introduction is divided into the following subsections: 1.1. How CRaC Works 1.2. Challenges and Considerations 1.3. Spring Boot 3.2 integration with CRaC 1.1. How CRaC Works Checkpoint Creation At a chosen point during the application’s execution, a checkpoint is created. This involves capturing the entire state of the Java application, including the heap, stack, and all active threads. The state is then serialized and saved to the file system. During the checkpoint process, the application is typically paused to ensure a consistent state is captured. This pause is coordinated to minimize disruption and ensure the application can resume correctly. Before taking the checkpoint, some requests are usually sent to the application to ensure that it is warmed up, i.e., all relevant classes are loaded, and the JVM HotSpot engine has had a chance to optimize the bytecode according to how it is being used in runtime. Commands to perform a checkpoint: Shell java -XX:CRaCCheckpointTo=<some-folder> -jar my_app.jar # Make calls to the app to warm up the JVM... jcmd my_app.jar JDK.checkpoint State Restoration When the application is started from the checkpoint, the previously saved state is deserialized from the file system and loaded back into memory. The application then continues execution from the exact point where the checkpoint was taken, bypassing the usual startup sequence. Command to restore from a checkpoint: Shell java -XX:CRaCRestoreFrom=<some-folder> Restoring from a checkpoint allows applications to skip the initial startup process, including class loading, warmup initialization, and other startup routines, significantly reducing startup times. For more information, see Azul’s documentation: What is CRaC? 1.2. Challenges and Considerations As with any new technology, CRaC comes with a new set of challenges and considerations: State Management Open files and connections to external resources, such as databases, must be closed before the checkpoint is taken. After the restore, they must be reopened. CRaC exposes a Java lifecycle interface that applications can use to handle this, org.crac.Resource, with the callback methods beforeCheckpoint and afterRestore. Sensitive Information Credentials and secrets stored in the JVM’s memory will be serialized into the files created by the checkpoint. Therefore, these files need to be protected. An alternative is to run the checkpoint command against a temporary environment that uses other credentials and replace the credentials on restore. Linux Dependency The checkpoint technique is based on a Linux feature called CRIU, “Checkpoint/Restore In Userspace”. This feature only works on Linux, so the easiest way to test CRaC on a Mac or a Windows PC is to package the application into a Linux Docker image. Linux Privileges Required CRIU requires special Linux privileges, resulting in Docker commands to build Docker images and creating Docker containers also requiring Linux privileges to be able to run. Storage Overhead Storing and managing checkpoint data requires additional storage resources, and the checkpoint size can impact the restoration time. The original jar file is also required to be able to restart a Java application from a checkpoint. I will describe how to handle these challenges in the section on creating Docker images. 1.3. Spring Boot 3.2 Integration With CRaC Spring Boot 3.2 (and the underlying Spring Framework) helps with the processing of closing and reopening connections to external resources. Before the creation of the checkpoint, Spring stops all running beans, giving them a chance to close resources if needed. After a restore, the same beans are restarted, allowing beans to reopen connections to the resources. The only thing that needs to be added to a Spring Boot 3.2-based application is a dependency to the crac-library. Using Gradle, it looks like the following in the gradle.build file: Groovy dependencies { implementation 'org.crac:crac' Note: The normal Spring Boot BOM mechanism takes care of versioning the crac dependency. The automatic closing and reopening of connections handled by Spring Boot usually works. Unfortunately, when this blog post was written, some Spring modules lacked this support. To track the state of CRaC support in the Spring ecosystem, a dedicated test project, Spring Lifecycle Smoke Tests, has been created. The current state can be found on the project’s status page. If required, an application can register callback methods to be called before a checkpoint and after a restore by implementing the above-mentioned Resource interface. The microservices used in this blog post have been extended to register callback methods to demonstrate how they can be used. The code looks like this: Java import org.crac.*; public class MyApplication implements Resource { public MyApplication() { Core.getGlobalContext().register(this); } @Override public void beforeCheckpoint(Context<? extends Resource> context) { LOG.info("CRaC's beforeCheckpoint callback method called..."); } @Override public void afterRestore(Context<? extends Resource> context) { LOG.info("CRaC's afterRestore callback method called..."); } } Spring Boot 3.2 provides a simplified alternative to take a checkpoint compared to the default on-demand alternative described above. It is called automatic checkpoint/restore at startup. It is triggered by adding the JVM system property -Dspring.context.checkpoint=onRefresh to the java -jar command. When set, a checkpoint is created automatically when the application is started. The checkpoint is created after Spring beans have been created but not started, i.e., after most of the initialization work but before that application starts. For details, see Spring Boot docs and Spring Framework docs. With an automatic checkpoint, we don’t get a fully warmed-up application, and the runtime configuration must be specified at build time. This means that the resulting Docker images will be runtime-specific and contain sensitive information from the configuration, like credentials and secrets. Therefore, the Docker images must be stored in a private and protected container registry. Note: If this doesn’t meet your requirements, you can opt for the on-demand checkpoint, which I will describe in the next blog post. With CRaC and Spring Boot 3.2’s support for CRaC covered, let’s see how we can create Docker images for Spring Boot applications that use CRaC. 2. Creating CRaC-Based Docker Images With a Dockerfile While learning how to use CRaC, I studied several blog posts on using CRaC with Spring Boot 3.2 applications. They all use rather complex bash scripts (depending on your bash experience) using Docker commands like docker run, docker exec, and docker commit. Even though they work, it seems like an unnecessarily complex solution compared to producing a Docker image using a Dockerfile. So, I decided to develop a Dockerfile that runs the checkpoint command as a RUN command in the Dockerfile. It turned out to have its own challenges, as described below. I will begin by describing my initial attempt and then explain the problems I stumbled into and how I solved them, one by one until I reach a fully working solution. The walkthrough is divided into the following subsections: 2.1. First attempt 2.2. Problem #1, privileged builds with docker build 2.3. Problem #2, CRaC returns exit status 137, instead of 0 2.4. Problem #3, Runtime configuration 2.5. Problem #4, Spring Data JPA 2.6. The resulting Dockerfile Let’s start with a first attempt and see where it leads us. 2.1. First Attempt My initial assumption was to create a Dockerfile based on a multi-stage build, where the first stage creates the checkpoint using a JDK-based base image, and the second step uses a JRE-based base image for runtime. However, while writing this blog post, I failed to find a base image for a Java 21 JRE supporting CRaC. So I changed my mind to use a regular Dockerfile instead, using a base image from Azul: azul/zulu-openjdk:21.0.3-21.34-jdk-crac Note: BellSoft also provides base images for CraC; see Liberica JDK with CRaC Support as an alternative to Azul. The first version of the Dockerfile looks like this: Dockerfile FROM azul/zulu-openjdk:21.0.3-21.34-jdk-crac ADD build/libs/*.jar app.jar RUN java -Dspring.context.checkpoint=onRefresh -XX:CRaCCheckpointTo=checkpoint -jar app.jar EXPOSE 8080 ENTRYPOINT ["java", "-XX:CRaCRestoreFrom=checkpoint"] This Dockerfile is unfortunately not possible to use since CRaC requires a build to run privileged commands. 2.2. Problem #1, Privileged Builds With Docker Build As mentioned in section 1.2. Challenges and Considerations, CRIU, which CRaC is based on, requires special Linux privileges to perform a checkpoint. The standard docker build command doesn’t allow privileged builds, so it can’t be used to build Docker images using the above Dockerfile. Note: The --privileged - flag that can be used in docker run commands is not supported by docker build. Fortunately, Docker provides an improved builder backend called BuildKit. Using BuildKit, we can create a custom builder that is insecure, meaning it allows a Dockerfile to run privileged commands. To communicate with BuildKit, we can use Docker’s CLI tool buildx. The following command can be used to create an insecure builder named insecure-builder: Shell docker buildx create --name insecure-builder --buildkitd-flags '--allow-insecure-entitlement security.insecure' Note: The builder runs in isolation within a Docker container created by the docker buildx create command. You can run a docker ps command to reveal the container. When the builder is no longer required, it can be removed with the command: docker buildx rm insecure-builder. The insecure builder can be used to build a Docker image with a command like: Shell docker buildx --builder insecure-builder build --allow security.insecure --load . Note: The --load flag loads the built image into the regular local Docker image cache. Since the builder runs in an isolated container, its result will not end up in the regular local Docker image cache by default. RUN commands in a Dockerfile that requires privileges must be suffixed with --security=insecure. The --security-flag is only in preview and must therefore be enabled in the Dockerfile by adding the following line as the first line in the Dockerfile: Dockerfile # syntax=docker/dockerfile:1.3-labs For more details on BuildKit and docker buildx, see Docker Build architecture. We can now perform the build; however, the way the CRaC is implemented stops the build, as we will learn in the next section. 2.3. Problem #2, CRaC Returns Exit Status 137 Instead of 0 On a successful checkpoint, the java -Dspring.context.checkpoint=onRefresh -XX:CRaCCheckpointTo... command is terminated forcefully (like using kill -9) and returns the exit status 137 instead of 0, causing the Docker build command to fail. To prevent the build from stopping, the java command is extended with a test that verifies that 137 is returned and, if so, returns 0 instead. The following is added to the java command: || if [ $? -eq 137 ]; then return 0; else return 1; fi. Note: || means that the command following will be executed if the first command fails. With CRaC working in a Dockerfile, let’s move on and learn about the challenges with runtime configuration and how to handle them. 2.4. Problem #3, Runtime Configuration Using Spring Boot’s automatic checkpoint/restore at startup, there is no way to specify runtime configuration on restore; at least, I haven’t found a way to do it. This means that the runtime configuration has to be specified at build time. Sensitive information from the runtime configuration, such as credentials used for connecting to a database, will written to the checkpoint files. Since the Docker images will contain these checkpoint files they also need to be handled in a secure way. The Spring Framework documentation contains a warning about this, copied from the section Automatic checkpoint/restore at startup: As mentioned above, and especially in use cases where the CRaC files are shipped as part of a deployable artifact (a container image, for example), operate with the assumption that any sensitive data “seen” by the JVM ends up in the CRaC files, and assess carefully the related security implications. So, let’s assume that we can protect the Docker images, for example, in a private registry with proper authorization in place and that we can specify the runtime configuration at build time. In Chapter 6 of the book, the source code specifies the runtime configuration in the configuration files, application.yml, in a Spring profile named docker. The RUN command, which performs the checkpoint, has been extended to include an environment variable that declares what Spring profile to use: SPRING_PROFILES_ACTIVE=docker. Note: If you have the runtime configuration in a separate file, you can add the file to the Docker image and point it out using an environment variable like SPRING_CONFIG_LOCATION=file:runtime-configuration.yml. With the challenges of proper runtime configuration covered, we have only one problem left to handle: Spring Data JPA’s lack of support for CRaC without some extra work. 2.5. Problem #4, Spring Data JPA Spring Data JPA does not work out-of-the-box with CRaC, as documented in the Smoke Tests project; see the section about Prevent early database interaction. This means that auto-creation of database tables when starting up the application, is not possible when using CRaC. Instead, the creation has to be performed outside of the application startup process. Note: This restriction does not apply to embedded SQL databases. For example, the Spring PetClinic application works with CRaC without any modifications since it uses an embedded SQL database by default. To address these deficiencies, the following changes have been made in the source code of Chapter 6: Manual creation of a SQL DDL script, create-tables.sql Since we can no longer rely on the application to create the required database tables, a SQL DDL script has been created. To enable the application to create the script file, a Spring profile create-ddl-script has been added in the review microservice’s configuration file, microservices/review-service/src/main/resources/application.yml. It looks like: YAML spring.config.activate.on-profile: create-ddl-script spring.jpa.properties.jakarta.persistence.schema-generation: create-source: metadata scripts: action: create create-target: crac/sql-scripts/create-tables.sql The SQL DDL file has been created by starting the MySQL database and, next, the application with the new Spring profile. Once connected to the database, the application and database are shut down. Sample commands: Shell docker compose up -d mysql SPRING_PROFILES_ACTIVE=create-ddl-script java -jar microservices/review-service/build/libs/review-service-1.0.0-SNAPSHOT.jar # CTRL/C once "Connected to MySQL: jdbc:mysql://localhost/review-db" is written to the log output docker compose down The resulting SQL DDL script, crac/sql-scripts/create-tables.sql, has been added to Chapter 6’s source code. The Docker Compose file configures MySQL to execute the SQL DDL script at startup. A CraC-specific version of the Docker Compose file has been created, crac/docker-compose-crac.yml. To create the tables when the database is starting up, the SQL DDL script is used as an init script. The SQL DDL script is mapped into the init-folder /docker-entrypoint-initdb.d with the following volume-mapping in the Docker Compose file: Dockerfile volumes: - "./sql-scripts/create-tables.sql:/docker-entrypoint-initdb.d/create-tables.sql" Added a runtime-specific Spring profile in the review microservice’s configuration file. The guidelines in the Smoke Tests project’s JPA section have been followed by adding an extra Spring profile named crac. It looks like the following in the review microservice’s configuration file: YAML spring.config.activate.on-profile: crac spring.jpa.database-platform: org.hibernate.dialect.MySQLDialect spring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults: false spring.jpa.hibernate.ddl-auto: none spring.sql.init.mode: never spring.datasource.hikari.allow-pool-suspension: true Finally, the Spring profile crac is added to the RUN command in the Dockerfile to activate the configuration when the checkpoint is performed. 2.6. The Resulting Dockerfile Finally, we are done with handling the problems resulting from using a Dockerfile to build a Spring Boot application that can restore quickly using CRaC in a Docker image. The resulting Dockerfile, crac/Dockerfile-crac-automatic, looks like: Dockerfile # syntax=docker/dockerfile:1.3-labs FROM azul/zulu-openjdk:21.0.3-21.34-jdk-crac ADD build/libs/*.jar app.jar RUN --security=insecure \ SPRING_PROFILES_ACTIVE=docker,crac \ java -Dspring.context.checkpoint=onRefresh \ -XX:CRaCCheckpointTo=checkpoint -jar app.jar \ || if [ $? -eq 137 ]; then return 0; else return 1; fi EXPOSE 8080 ENTRYPOINT ["java", "-XX:CRaCRestoreFrom=checkpoint"] Note: One and the same Dockerfile is used by all microservices to create CRaC versions of their Docker images. We are now ready to try it out! 3. Trying Out CRaC With Automatic Checkpoint/Restore To try out CRaC, we will use the microservice system landscape used in Chapter 6 of my book. If you are not familiar with the system landscape, it looks like the following: Chapter 6 uses Docker Compose to manage (build, start, and stop) the system landscape. Note: If you don’t have all the tools used in this blog post installed in your environment, you can look into Chapters 21 and 22 for installation instructions. To try out CRaC, we need to get the source code from GitHub, compile it, and create the Docker images for each microservice using a custom insecure Docker builder. Next, we can use Docker Compose to start up the system landscape and run the end-to-end validation script that comes with the book to ensure that everything works as expected. We will wrap up the try-out section by comparing the startup times of the microservices when they start with and without using CRaC. We will go through each step in the following subsections: 3.1. Getting the source code 3.2. Building the CRaC-based Docker images 3.3. Running end-to-end tests 3.4. Comparing startup times without CRaC 3.1. Getting the Source Code Run the following commands to get the source code from GitHub, jump into the Chapter06 folder, check out the branch SB3.2-crac-automatic, and ensure that a Java 21 JDK is used (Eclipse Temurin is used here): Shell git clone https://github.com/PacktPublishing/Microservices-with-Spring-Boot-and-Spring-Cloud-Third-Edition.git cd Microservices-with-Spring-Boot-and-Spring-Cloud-Third-Edition/Chapter06 git checkout SB3.2-crac-automatic sdk use java 21.0.3-tem 3.2. Building the CRaC-Based Docker Images Start with compiling the microservices source code: Shell ./gradlew build If not already created, create the insecure builder with the command: Shell docker buildx create --name insecure-builder --buildkitd-flags '--allow-insecure-entitlement security.insecure' Now we can build a Docker image, where the build performs a CRaC checkpoint for each of the microservices with the commands: Shell docker buildx --builder insecure-builder build --allow security.insecure -f crac/Dockerfile-crac-automatic -t product-composite-crac --load microservices/product-composite-service docker buildx --builder insecure-builder build --allow security.insecure -f crac/Dockerfile-crac-automatic -t product-crac --load microservices/product-service docker buildx --builder insecure-builder build --allow security.insecure -f crac/Dockerfile-crac-automatic -t recommendation-crac --load microservices/recommendation-service docker buildx --builder insecure-builder build --allow security.insecure -f crac/Dockerfile-crac-automatic -t review-crac --load microservices/review-service 3.3. Running End-To-End Tests To start up the system landscape, we will use Docker Compose. Since CRaC requires special Linux privileges, a CRaC-specific docker-compose file comes with the source code, crac/docker-compose-crac.yml. Each microservice is given the required privilege, CHECKPOINT_RESTORE, by specifying: YAML cap_add: - CHECKPOINT_RESTORE Note: Several blog posts on CRaC suggest using privileged containers, i.e., starting them with run --privleged or adding privileged: true in the Docker Compose file. This is a really bad idea since an attacker who gets control over such a container can easily take control of the host that runs Docker. For more information, see Docker’s documentation on Runtime privilege and Linux capabilities. The final addition to the CRaC-specific Docker Compose file is the volume mapping for MySQL to add the init file described above in section 2.5. Problem #4, Spring Data JPA: Dockerfile volumes: - "./sql-scripts/create-tables.sql:/docker-entrypoint-initdb.d/create-tables.sql" Using this Docker Compose file, we can start up the system landscape and run the end-to-end verification script with the following commands: Shell export COMPOSE_FILE=crac/docker-compose-crac.yml docker compose up -d Let’s start with verifying that the CRaC afterRestore callback methods were called: Shell docker compose logs | grep "CRaC's afterRestore callback method called..." Expect something like: Shell ...ReviewServiceApplication : CRaC's afterRestore callback method called... ...RecommendationServiceApplication : CRaC's afterRestore callback method called... ...ProductServiceApplication : CRaC's afterRestore callback method called... ...ProductCompositeServiceApplication : CRaC's afterRestore callback method called... Now, run the end-to-end verification script: Shell ./test-em-all.bash If the script ends with a log output similar to: Shell End, all tests OK: Fri Jun 28 17:40:43 CEST 2024 …it means all tests run ok, and the microservices behave as expected. Bring the system landscape down with the commands: Shell docker compose down unset COMPOSE_FILE After verifying that the microservices behave correctly when started from a CRaC checkpoint, we can compare their startup times with microservices started without using CRaC. 3.4. Comparing Startup Times Without CRaC Now over to the most interesting part: How much faster does the microservice startup when performing a restore from a checkpoint compared to a regular cold start? The tests have been run on a MacBook Pro M1 with 64 GB memory. Let’s start with measuring startup times without using CRaC. 3.4.1. Startup Times Without CRaC To start the microservices without CRaC, we will use the default Docker Compose file. So, we must ensure that the COMPOSE_FILE environment variable is unset before we build the Docker images for the microservices. After that, we can start the database services, MongoDB and MySQL: Shell unset COMPOSE_FILE docker compose build docker compose up -d mongodb mysql Verify that the databases are reporting healthy with the command: docker compose ps. Repeat the command until both report they are healthy. Expect a response like this: Shell NAME ... STATUS ... chapter06-mongodb-1 ... Up 13 seconds (healthy) ... chapter06-mysql-1 ... Up 13 seconds (healthy) ... Next, start the microservices and look in the logs for the startup time (searching for the word Started). Repeat the logs command until logs are shown for all four microservices: Shell docker compose up -d docker compose logs | grep Started Look for a response like: Shell ...Started ProductCompositeServiceApplication in 1.659 seconds ...Started ProductServiceApplication in 2.219 seconds ...Started RecommendationServiceApplication in 2.203 seconds ...Started ReviewServiceApplication in 3.476 seconds Finally, bring down the system landscape: Shell docker compose down 3.4.2. Startup Times With CRaC First, declare that we will use the CRaC-specific Docker Compose file and start the database services, MongoDB and MySQL: Shell export COMPOSE_FILE=crac/docker-compose-crac.yml docker compose up -d mongodb mysql Verify that the databases are reporting healthy with the command: docker compose ps. Repeat the command until both report they are healthy. Expect a response like this: Shell NAME ... STATUS ... crac-mongodb-1 ... Up 10 seconds (healthy) ... crac-mysql-1 ... Up 10 seconds (healthy) ... Next, start the microservices and look in the logs for the startup time (this time searching for the word Restored). Repeat the logs command until logs are shown for all four microservices: Shell docker compose up -d docker compose logs | grep Restored Look for a response like: Shell ...Restored ProductCompositeServiceApplication in 0.131 seconds ...Restored ProductServiceApplication in 0.225 seconds ...Restored RecommendationServiceApplication in 0.236 seconds ...Restored ReviewServiceApplication in 0.154 seconds Finally, bring down the system landscape: Shell docker compose down unset COMPOSE_FILE Now, we can compare the startup times! 3.4.3. Comparing Startup Times Between JVM and CRaC Here is a summary of the startup times, along with calculations of how many times faster the CRaC-enabled microservice starts and the reduction of startup times in percentage: MICROSERVICE WITHOUT CRAC WITH CRAC CRAC TIMES FASTER CRAC REDUCED STARTUP TIME product-composite 1.659 0.131 12.7 92% product 2.219 0.225 9.9 90% recommendation 2.203 0.236 9.3 89% review 3.476 0.154 22.6 96% Generally, we can see a 10-fold performance improvement in startup times or 90% shorter startup time; that’s a lot! Note: The improvement in the Review microservice is even better since it no longer handles the creation of database tables. However, this improvement is irrelevant when comparing improvements using CRaC, so let’s discard the figures for the Review microservice. 4. Summary Coordinated Restore at Checkpoint (CRaC) is a powerful feature in OpenJDK that improves the startup performance of Java applications by allowing them to resume from a previously saved state, a.k.a., a checkpoint. With Spring Boot 3.2, we also get a simplified way of creating a checkpoint using CRaC, known as automatic checkpoint/restore at startup. The tests in this blog post indicate a 10-fold improvement in startup performance, i.e., a 90% reduction in startup time when using automatic checkpoint/restore at startup. The blog post also explained how Docker images using CRaC can be built using a Dockerfile instead of the complex bash scripts suggested by most blog posts on the subject. This, however, comes with some challenges of its own, like using custom Docker builders for privileged builds, as explained in the blog post. Using Docker images created using automatic checkpoint/restore at startup comes with a price. The Docker images will contain runtime-specific and sensitive information, such as credentials to connect to a database at runtime. Therefore, they must be protected from unauthorized use. The Spring Boot support for CRaC does not fully cover all modules in Spring’s eco-system, forcing some workaround to be applied, e.g., when using Spring Data JPA. Also, when using automatic checkpoint/Restore at startup, the JVM HotSpot engine cannot be warmed up before the checkpoint. If optimal execution time for the first requests being processed is important, automatic checkpoint/restore at startup is probably not the way to go. 5. Next Blog Post In the next blog post, I will show you how to use regular on-demand checkpoints to solve some of the considerations with automatic checkpoint/restore at startup. Specifically, the problems with specifying the runtime configuration at build time, storing sensitive runtime configuration in the Docker images, and how the Java VM can be warmed up before performing the checkpoint.
API versioning is a technique used to manage changes to an API over time, ensuring that different versions of the API can coexist and be used simultaneously. This is crucial for maintaining compatibility with various clients that may rely on different versions of the API. Why API Versioning Is Important for Software Development Backward Compatibility Ensures that existing clients do not break when changes are made to the API. Allows clients to continue using the old version while new features are added in new versions. Controlled Changes Enables developers to introduce new features, bug fixes, and improvements incrementally. Reduces the risk of introducing breaking changes that could disrupt existing services. Flexibility Clients can choose when to upgrade to a newer version, allowing them to adapt at their own pace. Different clients can use different versions of the API based on their needs. Clear Communication Provides a clear mechanism to communicate changes and updates to clients. Helps in setting expectations about which versions are supported and for how long. Stability Maintains a stable and reliable user experience by ensuring that changes do not disrupt existing functionality. Benefits of API Versioning in Microservices Architecture Decoupled Services Microservices are designed to be loosely coupled. API versioning helps maintain this decoupling by allowing each service to evolve independently. Independent Deployment Different versions of a service can be deployed independently, reducing the risk of widespread failures. Supports blue-green deployments and A/B testing. Scalability Services can scale independently, with new versions handling the increased load or providing enhanced functionality without disrupting existing services. Backward Compatibility Ensures smooth integration and operation of microservices by allowing them to use different API versions as needed. Easier Maintenance Simplifies maintenance and management by allowing developers to focus on specific versions for bug fixes and improvements. Benefits of API Versioning in Cloud Computing Flexibility and Agility Cloud environments are dynamic and scalable. API versioning allows for flexible and agile development, enabling quick updates and enhancements without downtime. Seamless Upgrades Supports seamless upgrades and transitions in cloud-based applications, allowing clients to move to newer versions without disrupting their operations. Resource Management Cloud providers can manage resources more effectively by supporting multiple versions of an API, optimizing performance, and reducing costs. Global Reach Cloud environments often serve a global user base. API versioning ensures that updates and changes can be rolled out globally without causing disruptions. Improved User Experience By maintaining backward compatibility and providing clear upgrade paths, API versioning enhances the overall user experience, making it easier for clients to adopt new features and improvements. What Are Some Types of API Versioning? Here are some common strategies for implementing API versioning: URI Versioning With this approach, the version number is included in the URL of the API endpoint. For instance, consumers who are interested in viewing all of the products in a database would send a request to the https://example-api.com/v1/products endpoint. This is the most popular type of API versioning. Versions are included in the URL (e.g., http://api.example.com/v1/resource). Simple and intuitive for clients to understand. Example: /v1/users, /v2/users Query Parameter Versioning This strategy requires users to include the version number as a query parameter in the API request. Version information is passed as a query parameter (e.g., http://api.example.com/resource?version=1). Avoids changes to the URL structure. Example: /users?version=1, /users?version=2 Header Versioning This approach allows consumers to pass the version number as a header in the API request, which decouples the API version from the URL structure. Version information is included in the request header (e.g., Accept: application/vnd.example.v1+json). Keeps the URL clean and leverages HTTP headers. Example: Accept: application/vnd.example.v1+json, Accept: application/vnd.example.v2+json Content Negotiation Uses the Accept header to specify the version and format of the response (e.g., Accept: application/vnd.example+json;version=1). Provides more flexibility and adheres to REST principles. Example: Accept: application/vnd.example+json;version=1, Accept: application/vnd.example+json;version=2 Custom Headers This versioning strategy allows consumers to choose the appropriate version based on their needs. With this approach, the version that exists at the time of the consumer's first call is stored with the consumer's information. Every future call is then executed against this same version—unless the consumer explicitly modifies their configuration. Custom headers can be used to specify the version (e.g., API-Version: 1). Similar to header versioning but uses a custom header. Example: API-Version: 1, API-Version: 2 How Do You Version an API? Consistent Strategy Choose a versioning strategy that suits your API and stick to it across all endpoints. Semantic Versioning Follow semantic versioning principles (e.g., MAJOR.MINOR.PATCH) to indicate the impact of changes. Example: v1.0.0, v1.1.0, v2.0.0 Documentation Clearly document all versions of your API, including changes, deprecated endpoints, and upgrade paths. Deprecation Policy Establish a clear policy for deprecating older versions. Communicate deprecation timelines and provide sufficient notice to clients. Automated Testing Implement automated tests to ensure compatibility across different versions. This helps catch regressions and maintain stability. Monitoring and Analytics Monitor the usage of different API versions to understand client adoption and make informed decisions about deprecation and support. Backward Compatibility Strive to maintain backward compatibility whenever possible. Introduce breaking changes in major version updates. Example of Implementing URI Versioning in a RESTful API Best Practices Document all versions: Ensure that all versions are well-documented and easily accessible to clients. Deprecation policy: Establish a clear deprecation policy and communicate it to clients. Provide ample time for clients to transition to newer versions. Versioning consistency: Maintain consistency in versioning strategies across all microservices. Automated testing: Implement automated testing to ensure compatibility across different versions. Monitoring and analytics: Monitor the usage of different API versions to make informed decisions about deprecation and support. By implementing effective API versioning strategies, you can ensure the stability and reliability of your software, particularly in microservices architectures and cloud computing environments. This enables continuous improvement and innovation while maintaining compatibility and reliability for all users.
The complexity of distributed systems is an important challenge for engineers and developers. Complexity tends to increase as the system evolves, and therefore it is important to be proactive. Let's talk about what types of complexity you may encounter and what effective tactics to deal with it in your work. Distributed Systems and Complexity In development, a distributed system is a network of computers that are connected to each other and working on a single task. Each computer or node has its own local memory and processor and runs its own processes. However, they use a common network for coordination and centralization. A distributed system is very reliable; failure of one component does not disrupt the entire network. In a centralized computing system, one computer with one processor and one memory works on solving problems. In a centralized system, there are nodes, but they access the central node, which can cause network congestion and slowness. A centralized system has a single point of failure — this is an important disadvantage of it. Complexity Complexity can be defined from different perspectives and aspects. There are two main definitions that are important. In systems theory, complexity describes how different independent parts of the system interact and communicate with each other: how they define interactions with each other, how they depend on each other, how many dependencies they have, and also how they interact within the whole. From a software and technology perspective, complexity refers to the details of the software architecture, such as the number of components. Monolithic Architecture Monolithic architecture is a great example of a centralized system. It is represented as a single deployable and single executable component. For instance, such components may contain a user interface and different modules located in one place. Although this architecture is a traditional one for building software, it has several important drawbacks: Inability to scale modules independently Harder to control the growing complexity Lack of modules independent deployment Challenging to maintain a huge code database Technology and vendors coupling Microservices Architecture Microservices architecture is an architectural style and a variant of service-oriented architecture that structures the system as a collection of loosely coupled services. For example, companies, accounts, customers, and UI are represented as separate processes deployed on multiple nodes. All these services have their own time-to-time shared database, but this is probably a bad practice or antipattern. There are some advantages of such an architecture. Horizontal scalability is a game-changer! You can scale the database horizontally, and you can scale your services horizontally. Technically, any infrastructure component can be scaled horizontally by cloning, but many challenges must be solved. High availability and tolerance: Whenever you have several clones, you may organize some techniques that will help you avoid any downtimes in case of crashes, memory leaks, or power outages. Geographic distribution: If we all have customers in the USA, Europe, or Asia, and we also want to bring the best experience to our customers, we need to distribute these services across the world and organize more complicated techniques for data replication. Technology choice: You are free to choose your solutions. Quality Attributes There are three main quality attributes which any system has at some level or another: Reliability: Continuing to function properly despite challenges, meaning being fault-tolerant or resilient; Even if a system operates reliably now, it doesn’t guarantee future reliability. A frequent cause of performance degradation is increased load: for example, the system might have expanded from 10,000 to 100,000 concurrent users, or from 1 million to 10 million. Scalability is the term we use to describe a system’s ability to handle increased load. It is important to note that the scalability weakness of the whole system is determined by its weakest component. Maintainability is about making life better for the engineering and operations teams who need to work with the system. Good and stable abstractions can help reduce complexity and make the system easier to modify and adapt for new use features. What Are the Main Issues? “Anything that can go wrong will go wrong and at the worst possible time.” — Murphy Law Unreliable Networks There are many reasons why networks are not reliable, such as: Your request may have been lost. Your request may be waiting in a queue and will be delivered later. The remote node may have failed (perhaps it crashed or was powered down). The remote node may have temporarily stopped responding. The remote node may have processed your request, but the response has been lost on the network. The remote node may have processed your request, but the response has been delayed and will be delivered later. Strategy: Timeout The simplest solution to the problem is to apply timeout logic on the caller's side. For example, if the caller doesn’t receive a response after some timeout, it just throws an error and shows an error to the user. Strategy: Retry At scale, we can’t just throw exceptions for every network problem and upset users or delay system execution. So, if a response indicates that something goes wrong, just retry it. But what if the request was processed by the server and only the response was lost? In this scenario, retries may lead to severe consequences like several orders, payments, transactions, and so on. Strategy: Idempotency To avoid that, we can utilize a technique named idempotency. The concept of idempotency pertains to the notion that performing the same action multiple times has the same effect as performing it just once. To achieve the property of exactly-once semantics, a solution can be employed that attaches an idempotency key to the request. Upon retrying the same request with the identical idempotency key, the server will verify that a request with such a key has already been processed and will simply return the previous response. Consequently, any number of retries with the same key will not have a deleterious effect on the system's behavior. Strategy: Circuit Breaker Another pattern that might be useful in preventing overloading and completely crushing the server in case of failure is the circuit breaker. Circuit Breaker acts as a proxy to prevent the calling system, which is under maintenance, likely will fail, or heavily failing right now. There are so many reasons why it can go wrong: memory leak, a bug in the code, or external dependencies that are faulted. In such scenarios, it is better to fail fast rather than risk cascading failures. Concurrency and Lost Writes Concurrency represents one of the most intricate challenges in distributed systems. Concurrency implies the simultaneous occurrence of multiple computations. Consequently, what occurs when an attempt is made to update the account balance simultaneously from disparate operations? In the absence of a defensive mechanism, it is highly probable that race conditions will ensue, which will inevitably result in the loss of writes and data inconsistency. In this example, two operations are attempting to update the account balance concurrently. Since they are running in parallel, the last one to complete wins, resulting in a significant issue. To circumvent this problem, various techniques can be employed. Strategy: Snapshot Isolation The ACID acronym stands for Atomicity, Consistency, Isolation, and Durability. All of the popular SQL databases implement these properties. Atomicity specifies that the operation will be either completely executed or failed, no matter at what stage it happens. It allows us to ensure that another thread cannot see the half-finished result of the operation. Consistency means that all invariants are defined and will be satisfied before successfully committing a transaction and changing the state. Isolation in the sense of ACID means that concurrently executing transactions are isolated from each other. There is a serializable isolation level which is the strictest one to process all transactions sequentially, but another level named snapshot isolation in popular databases is mainly used. Durability promises that once the transaction is committed, all the data is stored safely. The key idea of this level is that databases track recorded versions and fail to commit transactions for ones that were already modified outside of the current transaction. Strategy: Compare and Set Most of the NoSQL databases do not provide ACID properties while choosing in favor of BASE, wherein such databases compare and the set is used. The purpose of this operation is to avoid lost updates by allowing an update to happen only if the value has not changed since you last read it. If the current value does not match what you previously read, the update has no effect and the read-modify-write cycle must be retried. For instance, Cassandra provides lightweight transactions that allow you to utilize various IF, IF NOT EXISTS, and IF EXISTS conditionals to prevent concurrency issues. Strategy: Lease Another potential solution is the lease pattern. To illustrate, consider a scenario where a resource must be updated exclusively. The lease pattern entails first obtaining a lease with an expiration period for the resource, then updating it, and finally returning the lease. In the event of failures, the lease will automatically expire, allowing another thread to access the resource. Although this technique is highly beneficial, there is a risk of process pauses and clock desynchronization, which may lead to issues with parallel resource access. Dual Write Problem The dual write problem is a challenge that arises in distributed systems, particularly when multiple data sources or databases must be kept in sync. To illustrate, consider a scenario in which new data must be stored in the database and messages are sent to Kafka. Since these two operations are not atomic, there is a possibility of failure during the publishing of new messages. If a transaction is attempted while messages are being sent, the result is a more problematic situation. In the event that the transaction fails to commit, external systems may have already been informed of changes that, in fact, did not occur. Strategy: Transactional Outbox One potential solution is the implementation of a transactional outbox. This involves the storage of events in an "OutboxEvents" table within the same transaction as the operation itself. Due to the atomicity of the process, no data will be stored in the event of a transaction failure. Another necessary component is Relay, which polls the OutboxEvents table at regular intervals and sends messages to destinations. This approach allows for the achievement of at least one delivery guarantee. Nevertheless, this is not a concern since all consumers must be idempotent due to the unreliability of the network. Strategy: Log Tailing An alternative solution to the construction of a custom transactional outbox is the utilization of a database transactional log and custom connectors to read directly from this log and send changes to destinations. This approach has its own advantages and disadvantages. For instance, it requires coupling to database solutions but allows for the writing of less code in the application. Unreliable Clocks Time tracking is a fundamental aspect of any software or infrastructure, as it enables the enforcement of timeouts, expirations, and the gathering of metrics. However, the reliability of clocks represents a significant challenge in distributed systems, as the accuracy of time is contingent upon the performance of individual computers, which may have clocks that are either faster or slower than others. There are two primary types of clocks utilized by computers: time-of-day and monotonic clocks. Time-of-day clocks return the date and time according to a specific calendar, and they are typically synchronized with Network Time Protocol (NTP). However, delays and network issues may affect the synchronization process, leading to clock desynchronization. Monotonic clocks continuously advance, making them suitable for measuring durations. However, the monotonically increased value is unique to each computer, limiting their use for multi-server date and time comparison. Achieving highly accurate clock synchronization is a challenging task. In the majority of cases, the necessity for such a solution is not apparent. However, in instances where compliance with regulations necessitates its use, the Precision Time Protocol can be employed, although this will entail a significant investment. Availability and Consistency The CAP Theorem posits that any distributed data store can only satisfy two of the three guarantees. However, since network unreliability is not a factor that can be significantly influenced, in the case of network partitions, the only viable option is to choose between availability or consistency. Consider the scenario in which two clients read from different nodes: one from the primary node and another from the follower. A replication is configured to update followers after the leader has been changed. However, what happens if, for some reason, the leader stops responding? This could be a crash, network partitioning, or another issue. In highly available systems, a new leader must be assigned, but how do we choose between existing followers? To address this issue, a distributed consensus algorithm must be employed. However, before delving into the specifics of this algorithm, it is essential to gain a comprehensive understanding of the various types of consistency. Consistency Type There are two main classes of consistency used to describe guarantees. Weak consistency, or eventually one, means that data will be synchronized on all followers after some time if you stop making changes to the leader. Strong consistency is a property that ensures that all nodes in the system see the same data at the same time, regardless of which node they are accessing. Strategy: Distributed Consensus Algorithm (e.g., Raft) Returning to the problem when the leader crashes, there is a need to elect a new leader. This problem, at first glance, looks easy, but in reality, there are so many conditions and tradeoffs that have to be taken into account when selecting the appropriate approach. Per Raft protocol, if followers do not receive data or heartbeat from the leader for a specified period of time, then a new leader election process begins. Each Replication Unit (monolith write node or multiple shards) is associated with a set of Raft logs and OS processes that maintain the logs and replicate changes from the leader to followers. The Raft protocol guarantees that followers receive log records in the same order they are generated by the leader. A user transaction is committed on the leader as soon as half of the followers acknowledge the receipt of the commit record and writes it to the Raft log. Strategy: Read From Leader One of the possible effective and simple strategies is to read from the follower by the user who just saved new data to avoid replication lag. Instead of Conclusion From monolithic architectures to microservices, each approach presents its own set of advantages and challenges. While monolithic architectures offer simplicity, they often struggle with scalability and maintainability, pushing developers towards a more modular and scalable microservices architecture. Central to the discussion is the management of complexity, which manifests in various forms, from network unreliability to concurrency issues and the dual write problem. Strategies such as timeouts, retries, idempotency, and circuit breakers offer effective tools for mitigating the risks associated with unreliable networks, while techniques like snapshot isolation, compare and set, and leases address the challenges of concurrency and lost writes. Furthermore, the critical issue of unreliable clocks underscores the importance of accurate time synchronization in distributed systems, with solutions ranging from NTP synchronization to the Precision Time Protocol. Additionally, the CAP theorem reminds us of the inherent trade-offs between availability and consistency, necessitating a thorough understanding of distributed consensus algorithms like Raft. In conclusion, mastering the maze of complexity in distributed systems requires a multifaceted approach, combining theoretical knowledge with practical strategies. By embracing these strategies and continuously adapting to the evolving landscape of distributed computing, engineers and developers can navigate the complexities with confidence, ensuring the reliability, scalability, and maintainability of their systems in the face of ever-changing challenges.
Microservices-based architecture splits applications into multiple independent deployable services, where each service provides a unique functionality. Every architectural style has pros and cons. One of the challenges of micro-service architecture is complex debugging/troubleshooting. Distributed Tracing In a microservice world, distributed tracing is the key to faster troubleshooting. Distributed tracing enables engineers to track a request through the mesh of services and therefore help them troubleshoot a problem. To achieve this, a unique identifier, say, trace-id, is injected right at the initiation point of a request, which is usually an HTTP load balancer. As the request hops through the different components (third-party apps, service-mesh, etc.), the same trace-id should be recorded at every component. This essentially requires propagation of the trace-id from one hop to another. Over a period, different vendors adopted different mechanisms to define the unique identifier (trace-id), for example: Zipkin B3 headers Datadog tracing headers Google proprietary trace context AWS proprietary trace context Envoy request id W3C trace context An application can adopt one of these available solutions as per the need. Accordingly, the relevant header (e.g., x-cloud-trace-context if Google proprietary trace context is adopted) should get injected at the request initiation and thereafter same value should get propagated to each of the components involved in the request lifecycle to achieve distributed tracing. W3C Trace Context Standard As the microservice world is evolving, there is a need to have a standard mechanism for trace propagation. Consider the case when two different applications that adopted two different trace propagation approaches, are used together. Since they use two different headers for trace propagation, distributed tracing gets broken when they communicate. To address such problems, it is recommended to use the W3C trace context across all components. W3C trace context is the standard that is being adopted by all major vendors for supporting cross-vendor distributed traces. Problem: Broken Traces OpenTelemetry supports the W3C trace context header "traceparent" propagation using auto-instrumentation. This means, as an application developer, I need not write any code in my application for trace context propagation when I instrument it with OpenTelemetry. For example, if I have a Java application, I can instrument it as shown below: java -javaagent:opentelemetry-javaagent.jar -Dotel.service.name=app-name -jar app.jar The traceparent header will now be automatically generated/propagated by the instrumented Java application. However, when my application, instrumented using OpenTelemetry, gets deployed behind GCP or AWS HTTP load balancer, my expectation to visualize the complete trace starting from the load balancer fails. This is because GCP HTTP Load Balancer supports their proprietary trace context header "X-Cloud-Trace-Context". See GCP documentation for more details. AWS Elastic Load Balancer supports their proprietary trace context header "X-Amzn-Trace-Id". See AWS documentation for more details. My application generates and logs the W3C traceparent header. So, the unique-identifier generated by the GCP/AWS load balancer is not propagated further by my application. This is the typical problem of broken traces, also described above. So how can a developer leverage the out-of-the-box OpenTelemetry trace context propagation functionality? Solution: GCP Trace Context Transformer We have solved this problem by transforming the GCP/AWS proprietary trace context header (X-Cloud-Trace-Context/ X-Amzn-Trace-Id) to the W3C trace context header (traceparent). Service mesh is a key component in a distributed system to enforce organization policies consistently across all the applications. One of the popular service mesh, Istio, can help in solving our problem. The diagram below elaborates on the solution: A common Trace-Id value across all the logs generated from the load balancer, istio-ingress gateway, istio-sidecar, and application logs helps in stitching all the logs for a request processing. Istio allows you to extend the data-plane behavior by writing custom logic using either Lua or WASM. We have extended the istio-ingress gateway by injecting a Lua filter. This filter extracts the trace-id and span-id from X-Cloud-Trace-Context and creates the traceparent request header using these values. Note: For the sake of simplicity, below filter code is built only for the GCP "X-Cloud-Trace-Context". One can write a similar filter for AWS "X-Amzn-Trace-Id". While adopting the filter in your infrastructure, don't forget to choose the right namespace and workloadSelector label. This filter has been tested on Istio 1.20.1 version. YAML apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: gcp-trace-context-transformer-gateway namespace: istio-system spec: workloadSelector: labels: istio: ingressgateway configPatches: - applyTo: HTTP_FILTER # http connection manager is a filter in Envoy match: context: GATEWAY patch: operation: INSERT_BEFORE value: name: envoy.filters.http.lua typed_config: "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua" inlineCode: | function envoy_on_request(request_handle) local z = request_handle:headers():get("traceparent") if z ~= nil then return end local x = request_handle:headers():get("X-Cloud-Trace-Context") if x == nil then return end local y = string.gmatch(x, "%x+") local traceid = y() if (traceid == nil) then return end -- generate a new 16 hex-character random span math.randomseed(os.time()) local parentid = math.random(0xffffffff, 0x7fffffffffffffff) local traceparent = string.format("00-%s-%016x-01", traceid, parentid) request_handle:headers():add("traceparent", traceparent) end function envoy_on_response(response_handle) return end Alternate Solution: Custom GCP Trace Context Propagator Another possible solution could be extending OpenTelemetry to support the propagation of GCP proprietary trace context. One implementation exists on GitHub but, alas, it is still in alpha state (at the time of publishing this article). Further, this solution will only work for GCP environments, similar propagators will be needed for different cloud providers (AWS, etc).
Hexagonal architecture is a software design pattern based on the separation of responsibilities. The goal is to decouple the business logic (domain) and the application from other external interfaces. Simplifying, in hexagonal architecture we communicate the core of the app (domain + application) with the external elements using ports and adapters. A port lives in the core; it is the interface any external code must use to interact with the core (or the core with the external code). The adapter is the external piece of code that follows the port interface and executes the tasks, gets the data, etc. You can imagine the port is a space reserved only for an exact type of vessel. The vessel can only enter the port and dock if the load/unload doors are of an expected size and are in the correct position. Multiple vessels can fit in a port and vessels can be replaced, but ports are unique and can not be moved. A key concept is that the core doesn’t know anything about the external components. The port defines the vessel door positions but doesn’t care about how the load is stored in the vessel. In this case, we will also use the repository pattern (that fits very well with hexagonal, as it defines a centralized and abstract way of accessing data and it is a very common pattern), and the dependency injection principle that allows us to create decoupled (or loosely coupled) software. Simplifying again, it allows us to replace an adapter with another one that follows the same port interface. Let’s see it in action with a small (and typical) example: Your domain (core) needs to get a list of users with a name, so you define the port that is a repository. The port defined a method to do that: getUsersByName(name: string): User[]. In English, it defines that the adapter must provide a method called getUsersByName that gets a name and should return the list of the users that match that name. A Real Case The Initial Context We have a single web application (frontend) that works for different clients (tenants), and that application uses a backend that provides the menu data. The backend returns something like this: JSON { "title": "Main Menu", "id": "main", "is_staff": false, "items": [ { "title": "Home", "icon": "", "url": "/", "is_staff": false }, { "title": "Dashboards", "icon": "dashboards", "id": "dashboards", "is_staff": false, "items": [ { "title": "Home", "icon": "dashboards-home", "url": "/dashboards", "is_staff": false }, { "title": "Config (X)", "icon": "dashboards-config", "url": "/dashboards-config", "is_staff": true }, { "title": "Advanced Reports", "icon": "", "id": "advanced_reports", "is_staff": false, "items": [ { "title": "Sales Analysis", "icon": "", "url": "/sales_analysis", "is_staff": false }, ... ] } ] } ] } The front end partially implements the repository pattern as it just returns the data the backend provides without more manipulation than removing the first level in the tree (the main menu item). The view executes the repository call using a service: that again just returns the same information it gets from the repository. The Issues This “architecture” works, but has some issues that can create serious problems in the future: The data structure is coupled to the backend data: All the data flows from the backend to the view using the same interfaces. If the backend changes just the name of a property, we need to follow the data flow in our code until the view changes it in all the places. The title string includes an emoji to allow users to visualize when a menu item is only for staff users: That information is also provided in the is_staff property. If we want to expose a menu item to regular users, we need to change it in 2 places, and that is never a good idea. Visuals are defined in the backend: The name of the icon to use is defined in the backend. Unless the icon would be an app (backend + frontend) global concept, it is not a good idea to pass that value front the backend. No domain: there is no domain, or at least no explicit one. Logic is applied in the view (that it is not bad per se, but if the logic is related to the business rules, it must live in the domain). The Problem Because of different reasons, the company decided to create a new version of the backend. This new backend (called v2) will not be retro-compatible with the legacy one, but it will represent semantically the same entities. The menu endpoint will return the same menu (it will provide more features) but the new endpoint response structure is completely different: JSON [ { "menuStateId": 3, "menuPosition": 1, "menuName": "Dashboards", "menuItemId": 9, "menuItemTitle": "Home", "menuItemPosition": 1, "menuItemLink": "/dashboards", "menuItemStateId": 3, "menuInternalName": "dashboard", "menuId": 12, "menuParentId": 1, "menuItemInternalName": "dashboard.home" }, { "menuStateId": 3, "menuPosition": 1, "menuName": "Dashboards", "menuItemId": 9, "menuItemTitle": "Sales analysis", "menuItemPosition": 1, "menuItemLink": "/sales_analysis", "menuItemStateId": 3, "menuInternalName": "dashboard", "menuId": 12, "menuParentId": 1, "menuItemInternalName": "dashboard.sales_analysis" }, { "menuStateId": 3, "menuPosition": 1, "menuName": "Dashboards", "menuItemId": 9, "menuItemTitle": "Config", "menuItemPosition": 1, "menuItemLink": "/dashboards-config", "menuItemStateId": 1, "menuInternalName": "dashboard", "menuId": 12, "menuParentId": 1, "menuItemInternalName": "dashboard.sales_analysis" }, { "menuStateId": 3, "menuPosition": 1, "menuName": "Dashboards", "menuItemId": 9, "menuItemTitle": "Sales analysis", "menuItemPosition": 1, "menuItemLink": "/sales_analysis", "menuItemStateId": 3, "menuInternalName": "dashboard", "menuId": 12, "menuParentId": 1, "menuItemInternalName": "dashboard.sales_analysis" }, { "menuStateId": 3, "menuPosition": 1, "menuName": "Main", "menuItemId": 11, "menuItemTitle": "Home", "menuItemPosition": 5, "menuItemLink": "/", "menuItemStateId": 3, "menuInternalName": "home", "menuId": 1, "menuParentId": 1, "menuItemInternalName": "home" }, ... ] The new backend endpoint returns the menu items and its parent menu data in the same row. The structure is flat (no nested items). Another difference is the is_staff is still there, but it’s a specific value for the menuItemStateId property. There is no icon name, but now we have an internalId as a semantic unique ID. Things Can Become Harder The new backend will not replace the legacy one, at least not in the next months. Clients will be migrated slowly to the new backend. So some clients will use the legacy backend and others will use the new one. That means we will have both backends working at the same time for months. As the data returned by both backends is very different, it seems tough to use the same frontend code to render the menu for all the clients, right? (Not really, as we will see later.) A possible solution is to create different menu-related components, code, etc. depending on the backend version adapting our application to them. This can work, but it means we will need to duplicate a lot of code; for example, the views, the services, etc., making the maintenance harder. Decoupling Us From the Backend Let’s forget for a while how the backend(s) data returns, and think about what we want to represent from the point of view of our application. We want to represent a menu that can have items with children items (and no link), and items with links and no children. Then let’s create a model (models, in our case) in our domain as entities that will represent exactly that: TypeScript type State = 'disabled' | 'only_for_staff' | 'open' class Menu { readonly id: number = 0 readonly internalName: string = '' readonly title: string = '' readonly icon: string = '' readonly image: URL | undefined readonly state: State = 'open' readonly description: string = '' readonly position: number = 0 readonly children: (Menu | MenuItem)[] = [] constructor(values: MenuDto) { this.id = values.id this.internalName = values.internalName //... this.children = values.children } get onlyForStaff(): boolean { return this.state === 'only_for_staff' } } class MenuItem { readonly id: number = 0 readonly internalName: string = '' readonly title: string = '' readonly icon: string = '' readonly url: string = '' readonly state: State = 'open' readonly position: number = 0 readonly menuId: number = 0 private constructor(values: MenuItemDto) { // hydrate the entity this.id = values.id //.... } get isStaff(): boolean { return this.state === 'hidden' } public get external(): boolean { return (this.url.includes('http://') } } This is a simplified version of the entities, but you can see the idea. We have a Menu entity that can have children that can be Menu or MenuItem entities. The MenuItem entity has a url property that can be used to know if the item is a link or not. We modeled the domain, and our application layer (and views) can access it. The key is this: we modeled our menu independently of our backends’ data structures. We can use any backend that represents that entity to get the data independently of the structure. The Port We should create the port that will allow us to get the menu’s data from the backend(s). TypeScript interface MenuRepo { getMainMenu(states: State[]): Promise<(MenuItem | Menu)[]> } The port defines how the repository should look. In this case, we want a method that will return the main menu, filtered by state ('disabled' | 'only_for_staff' | 'open'). The Adapter: The Repositories Will Do the Magic We need to create the adapters that will get the data from the backend and transform it to our domain entities. We need an adapter, also called repository implementation, for each backend (we could have even more for mocked data, stubs for testing, etc). Remember, the repository implementation (adapter) is the one that knows the “external to the core” internals: How to get the data at the infrastructure level: REST, GraphQL, local storage, etc. How to request the data: For example, for an XHR request: headers, query params, URL, etc. The returned data structure and how to transform it to the domain entities How to handle errors, retries, etc. How to cache the data But again, the domain NEVER should not know about that. For example, the domain must not know that to get items available only for staff users, we need to pass the menuItemStateId param with the value 1. menuItemStateId is an implementation detail. it only makes sense in repository implementation, not in the domain. The domain should know about the onlyForStaff meaning, and the adapter should know how to get that information from the backend. In this case (for backend v2), we need to pass a query param called menuItemStateId with the value 1 to get the staff-only items, but that is different for the legacy backend, or for another backend that can use a different value for that filter, but the argument that represents what we want is still the same: onlyForStaff. From the point of view of the layers on the right side - the port’s line in the workflow (image above) - it does not matter how the data is retrieved. The only thing that matters is the data is returned as a domain entity. That is our contract. TypeScript // menu.legacy.repo.ts type Response = { // This type defines the shape of the data the backend returns. I do not include it here to put the focus on the data transformation into entities } class LegacyMenuRepo implements MenuRepo { async getMainMenu(states: State[]): Promise<(MenuItem | Menu)[]> { // [1] const data = await fetch<Response>('tenant.company.com/get_menu') const backendMenu = await data.json() return backendMenu.map(item => responseToEntity(backendMenu)) } private responseToEntity(response: Response): (MenuItem | Menu) { // transform the response to the domain entities if ('items' in response) { return new Menu({ id: response.id, internalName: response.id, title: response.title, icon: mapIcon(response.id), // [2] image: mapImage(response.image), // [2] state: mapState(response.state), // [3] children: response.items.map(item => responseToEntity(item)) }) } else { return new MenuItem({ id: response.id, internalName: response.id, title: response.title, icon: mapIcon(response.id), // [2] url: response.url, state: mapState(response.state), // [3] menuId: response.menuId }) } } } Things to focus on: [1]: The method that receives the state's argument is not used in the code. This is because the backend does not accept any filter. The legacy backend does the filtering using the backend context, but it ensures will only return the items the user can have access to. [2]: Those map functions are in charge of providing the correct icon and image. Now the backend does not provide that information, so our repository implementation should provide it. Remember: the repository implementation (adapter) is the one that knows all the external internals and for the images, the adapter knows that if the id is “x,” it should return the image “y” and the icon “z”. [3]: The mapState function behavior is similar to [2], but in this case, the backend returns a number that represents the state, the adapter should know how to map that number to the domain state, and that function can be reversed to know that the state should be sent to the backend. We need to implement the adapter for the “new” backend (v2): TypeScript // menu.v2.repo.ts type Response = { // This type defines the shape of the data the backend v2 returns. I do not include it here to put the focus on the data transformation into entities } const stateMappings: Record<number, State> = { 0: 'disabled', 1: 'only-for-staff', 2: 'open' } const stateMappingsReverse: Record<number, State> = { 'disabled': 0, 'only-for-staff': 1, 'open': 2 } class V2MenuRepo implements MenuRepo { async getMainMenu(states: State[]): Promise<(MenuItem | Menu)[]> { const data = await fetch<Response>('menu.company.com/company/get', { // [1] params: { menuItemStateId: states.map(state => stateMappingsReverse[state]) // [2] } }) const backendMenu = await data.json() return backendMenu.map(item => responseToEntity(backendMenu)) } private responseToEntity(response: Response): (MenuItem | Menu) { //Here the transformations from flat to nested are more complex (require more code lines) so I'm going to ignore it in the example. Let's imagine it is done after this line // transform the response to the domain entities if ('items' in response) { return new Menu({ id: response.id, internalName: response.internalName, title: response.title, icon: mapIcon(response.internalName), // [3] image: mapImage(response.internalName), // [3] state: stateMappings[response.menuStateId], // [4] children: response.items.map(item => responseToEntity(item)) }) } else { return new MenuItem({ id: response.id, internalName: response.id, title: response.title, icon: mapIcon(response.internalName), // [3] url: response.url, state: stateMappings[response.menuItemStateId], // [4] menuId: response.menuId }) } } } Things to focus on in the backend v2 repo implementation: [1]: The endpoint (even the domain) is different from the other repo. That is expected as it is a different backend. [2]: You need to convert the meaning of the filters to the backend meaning. The adapter knows that the backend expects a query param called menuItemStateId with the values 0, 1, or 2 to get the items with the state disabled, only-for-staff, or open. [3]: We have mapping functions for the icons and images, but this function is different from the legacy one. [4]: We convert the menuItemStateId and menuStateId to the domain state using the mappings. After the changes, the architecture looks like this: Now we have 2 different adapters (one per backend) for the same port. Those adapters follow the contract and convert the backend data to the domain entities. The rest of the flow is the same: the domain does not know how the data is retrieved, it only knows how to use it. This gives us a lot of flexibility. We can change the backend without changing the domain the application, or the views. The Dependency Injection The last piece of the “puzzle” is the dependency injection, which allows us, to replace a repository implementation with another one that follows the same port interface, but instead of importing it from the code that will call the repository, we inject it from outside allows us to change it easily. Let’s suppose we have a use case (or application service) that will use the repository to get the menu: TypeScript class GetMainMenuUseCase { constructor(private menuRepo: MenuRepo) {} async execute(states: State[]): Promise<(MenuItemDto | MenuDto)[]> { return this.menuRepo.getMainMenu(states).map(entity => entity.toDto()) } } We can use a factory to create the repository implementation: TypeScript function createMenuRepo(clientId: string): MenuRepo { if (['client123', 'client34'].includes(clientId)) { return new V2MenuRepo() } else { return new LegacyMenuRepo() } } const useCase = new GetMainMenuUseCase(createMenuRepo('client123')) usecase.execute(['open', 'only-for-staff']) Summarizing The hexagonal architecture, the repository pattern, and the dependency injection are very powerful tools that allow us to create decoupled software that works in independent pieces loosely coupled that can be easily changed and make the maintenance simpler. Those pieces should define a contract for the actions (execute a method) and for the returned data, and should not be used in other places. For example, it is a bad practice to pass the filters directly to the repository implementation and use them as is in the HTTP request because you are coupling your application code to the backend, as we see in the example when we map the filter values. As you can see in the example, we can change the backend at any moment: it’s just in order to change the repo implementation, we inject it into the use case without changing anything else. This will work only if all the different backend returns the same business concepts. If not, we are talking about different domain models and we need to create different ports and adapters for each one. Achieving that can require time and knowledge of the domain and the business rules, but the benefits are worth it.
Over the years Docker containers have completely changed how developers create, share, and run applications. With their flexible design, Docker containers ensure an environment, across various platforms simplifying the process of deploying applications reliably. When integrated with .NET, developers can harness Dockers capabilities to streamline the development and deployment phases of .NET applications. This article delves into the advantages of using Docker containers with .NET applications and offers a guide on getting started. Figure courtesy of Docker Why Choose Docker for .NET? 1. Consistent Development Environment Docker containers encapsulate all dependencies and configurations for running an application guaranteeing consistency across development, testing, and production environments. By leveraging Docker, developers can avoid the typical statement of "it works on my machine" issue, as they can create environments that operate flawlessly across various development teams and devices. 2. Simplified Dependency Management Docker eliminates the need to manually install and manage dependencies on developer machines. By specifying dependencies in a Docker file developers can effortlessly bundle their .NET applications with libraries and dependencies reducing setup time and minimizing compatibility issues. 3. Scalability and Resource Efficiency Due to its nature and containerization technology, Docker is well suited for horizontally or vertically scaling .NET applications. Developers have the ability to easily set up instances of their applications using Docker Swarm or Kubernetes which helps optimize resource usage and enhance application performance. 4. Simplified Deployment Process Docker simplifies the deployment of .NET applications. Developers have the ability to wrap their applications into Docker images. These can be deployed to any Docker-compatible environment, including local servers, cloud platforms like AWS or Azure, and even IOT devices. This not only streamlines the deployment process but also accelerates the release cycle of .NET applications Starting With Docker and .NET Step 1: Installing Docker Installing Docker is easy by navigating to the Docker desktop. Docker desktop is available for Windows, Mac, and Linux. I have downloaded and installed it for Windows. Once installed, the Docker (whale) icon is shown on the systems side tray as shown below. When you click on the icon, it will open the Docker desktop dashboard as shown below. You can see the list of containers, images, volumes, builds, and extensions. In the below figure, it shows the list of containers I have created on my local machine. Step 2: Creating a .NET Application Create a .NET application using the tool of your choice like Visual Studio, Visual Studio Code, or the.NET CLI. For example, you can use the following command directly from the command line. PowerShell dotnet new web -n MinimalApiDemo Step 3: Setting up Your Application With a Docker Create a Dockerfile in the root folder of your .NET project to specify the Docker image for your application. Below is an example of a Dockerfile for an ASP.NET Core application which was created in the previous step. Dockerfile # Use the official ASP.NET Core runtime as a base image FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base WORKDIR /app EXPOSE 8080 # Use the official SDK image to build the application FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build WORKDIR /src COPY ["MinimalApiDemo.csproj", "./"] RUN dotnet restore "MinimalApiDemo.csproj" COPY . . WORKDIR "/src/" RUN dotnet build "MinimalApiDemo.csproj" -c Release -o /app/build # Publish the application FROM build AS publish RUN dotnet publish "MinimalApiDemo.csproj" -c Release -o /app/publish # Final image with only the published application FROM base AS final WORKDIR /app COPY --from=publish /app/publish . ENTRYPOINT ["dotnet", "MinimalApiDemo.dll"] Step 4: Creating and Launching Your Docker Image Create a Docker image by executing the command from a terminal window (use lowercase letters). PowerShell docker build -t minimalapidemo . After finishing the construction process you are ready to start up your Docker image by running it inside a container. Run the below docker command to spin up a new container. PowerShell docker run -d -p 8080:8080 --name myminimalapidemo minimalapidemo Your API service is currently running within a Docker container and can be reached at this localhost as shown below. Refer to my previous article to see how I created products controllers using Minimal API's with different HTTP endpoints. Here Are Some Recommended Strategies for Dockerizing .NET Applications 1. Reduce Image Size Enhance the efficiency of your Docker images by utilizing stage builds eliminating dependencies and minimizing layers in your Docker file. 2. Utilize .dockerignore File Generate a .dockerignore file to exclude files and directories from being transferred into the Docker image thereby decreasing image size and enhancing build speed. 3. Ensure Container Security Adhere to security practices during the creation and operation of Docker containers including updating base images conducting vulnerability scans and restricting container privileges. 4. Employ Docker Compose for Multi Container Applications For applications with services or dependencies, leverage Docker Compose to define and manage multi-container applications simplifying both development and deployment processes. 5. Monitor and Troubleshoot Containers Monitor the performance and health of your Docker containers using Docker’s own monitoring tools or third-party solutions. Make use of tools such as Docker logs and debugging utilities to promptly resolve issues and boost the efficiency of your containers. Conclusion Docker containers offer an efficient platform for the development, packaging, and deployment of .NET applications. By containerizing these applications, developers can create development environments, simplify dependency management, and streamline deployment processes. Whether the focus is on microservices, web apps, or APIs, Docker provides a proficient method to operate .NET applications across various environments. By adhering to best practices and maximizing Docker’s capabilities, developers can fully leverage the benefits of containerization, thereby accelerating the process of constructing and deploying .NET applications
This article outlines the main challenges of state propagation across different (micro)services together with architectural patterns for facing those challenges, providing an actual implementation using the Debezium Engine. But first of all, let's define some of the basic concepts that will be used during the article. (Micro)service: I personally do not like this term. I prefer talking about bounded contexts and their internal domain/sub-domains. Running the bounded context and/or its domains as different runtimes (aka (micro)services) is a deployment decision, meaning that domain and sub-domains within a bounded context must be decoupled independently of how they are deployed (microservices vs modular monolith). For the sake of simplicity, in this article, I will use the word "microservice" to represent isolated runtimes that need to react to changes in the state maintained by other microservices. State: Within a microservice, its bounded context, domain, and subdomains define an internal state in the form of aggregates (entities and value objects). In this article, changes in the microservice's state shall be propagated to another runtime. State propagation: In a scenario where the application runtimes (i.e., microservices) are decoupled, independently of how they are deployed (Pod in Kubernetes, Docker container, OS service/process), state propagation represents the mechanism for ensuring that state mutation happened in a bounded context is communicated to the downstream bounded contexts. Setting the Stage As a playground for this article, we will use two bounded contexts defined as part of an e-learning platform: Course Management: Managing courses, including operations for course authoring Rating System: Enable platform users to rate courses Between both contexts, there is an asynchronous, event-driven, relationship, so that whenever a course is created in the Course Management bounded context, an event is published and eventually received by the Rating System, which adapts the inbound event to its internal domain model using an anti-corruption layer (ACL) and creates an empty rating for that Course. The next image outlines this (simplified) context mapping: Bounded Context Mapping of the two services The Challenge The scenario is apparently pretty simple: we can just publish a message (CourseCreated) to a message broker (e.g., RabbitMQ) whenever a Course is created, and by having the Rating System subscribed to that event, it will eventually be received and processed by the downstream service. However, it is not so simple, and we have several "what-if" situations, like: What if the Course creation transaction is eventually rolled back (e.g., the database does not accept the operation) but the message is correctly published to the message broker? What if the Course is created and persisted, but the message broker does not accept the message (or it is not available)? In a nutshell, the main challenge here is how to ensure, in a distributed solution, that both the domain state mutation and the corresponding event publication happen in a consistent and atomic operation so that both or none should happen. There are certainly solutions that can be implemented and message producer or message consumer sides to try to solve these situations, like retries, publishing compensation events, and manually reverting the write operation in the database. However, most of them require the software engineers to have too many scenarios in mind, which is error-prone and reduces codebase maintainability. Another alternative is implementing a 2-phase-commit solution at the infrastructure level, making the deployment and operations of the underlying infrastructure more complex, and most likely, forcing the adoption of expensive commercial solutions. During the rest of the article, we will focus on a solution based on the combination of two important patterns in distributed systems: Transactional Outbox and Change Data Capture, providing a reference implementation that will allow software engineers to focus on what really matters: providing domain value. Applicable Architecture Patterns As described above, we need to ensure that state mutation and the publication of a domain event are atomic operations. This can be achieved by the combination of two patterns which are nicely explained by Chris Richardson in his must-read Microservices Patterns book; therefore, I will not explain them in detail here. Transactional Outbox Pattern Transactional Outbox focuses on persisting both state mutation and corresponding event(s) in an atomic database operation. In our case, we will leverage the ACID capabilities of a relational database, with one transaction that includes two write operations, one in the domain-specific table, and another that persists the events in an outbox (supporting) table. This ensures that we will achieve a consistent state of both domains and the corresponding events. This is shown in the next figure: Transactional Outbox Pattern Change Data Capture With Transaction Log Tailing Once the events are available in the Outbox table, we need a mechanism to detect new events stored in the Outbox (Change Data Capture or CDC) and publish them to external consumers (Transaction Log Tailing Message Relay). The Message Relay is responsible for detecting (i.e., CDC) new events available in the outbox table and publishing those events for external consumers via message broker (e.g., RabbitMQ, SNS + SQS) or event stream (e.g., Kafka, Kinesis). There are different Change Data Capture (CDC) techniques for the Message Relay to detect new events available in the outbox. In this article, we will use the Log Scanners approach, named Transaction Log Tailing by Chris Richardson, where the Message Relay tails the database transaction log to detect the new messages that have been appended to the Outbox table. I personally prefer this approach since it reduces the amount of manual work, but might not be available for all databases. The next image illustrates how the Message Relay integrates with the Transactional Outbox: Transactional Outbox in combination with Transaction Log Tailing CDC One of the main goals of this solution is to ensure that the software engineers of the two bounded contexts only need to focus on the elements with orange color in the diagram above; the grey components are just infrastructure elements that shall be transparent for the developers. So, how do we implement the Transaction Log Tailing Message Relay? Debezium Debezium is a Log Scanner type change data capture solution that provides connectors for several databases, creating a stream of messages out of the changes detected in the database's transaction log. Debezium comes in two flavors: Debezium Server: This is a full-feature version of Debezium which leverages Apache Kafka and Kafka Connectors to stream data from the source database to the target system. Debezium Embedded/Engine: The simplified version can be embedded as a library in your product; it does not require an Apache Kafka service but still makes use of Kafka Connectors to detect changes in the data sources. In this example, we will use Debezium Embedded, due to its simplicity (i.e., no Kafka instance is needed) but at the same time, it is robust enough to provide a suitable solution. The first time a Debezium instance starts to track a database, it takes a snapshot of the current data to be used as a basis, once completed, only the delta of changes from the latest stored offset will be processed. Debezium is highly configurable, making it possible to shape its behavior to meet different expectations, allowing, for instance, to define: The database operations to be tracked (updates, inserts, deletions, schema modifications) The database tables to be tracked Offset backing store solution (in-memory, file-based, Kafka-based) Offset flush internal Some of these properties will be analyzed later in the article. All Pieces Together The next image shows the overall solution from the deployment perspective: Services: In this example, we will use Spring Boot for building the Course Management and Rating System bounded contexts. Each bounded context will be deployed as separate runtimes. The persistence solutions for both services will be PostgreSQL having each of the services a dedicated schema. Persistence: PostgreSQL is also deployed as a Docker container. Message Broker: For the message broker, we will use RabbitMQ, which is also running as a Docker container. Message Relay: Leveraging Spring Boot and Debezium Embedded provides the Change Data Capture (CDC) solution for detecting new records added in the outbox table of the Course Management service. Show Me the Code All the code described in this article can be found in my personal GitHub repository. Overall Project Structure The code provided to implement this example is structured as a multi-module Java Maven project, leveraging Spring Boot and following a hexagonal architecture-like structure. There are three main package groups: 1. Toolkit Supporting Context Modules providing shared components and infrastructure-related elements used by the functional bounded contexts (in this example Course Management and Rating System). For instance, the transactional outbox and the Debezium-based change data capture are shared concerns, and therefore their code belongs to these modules. Where: toolkit-core: Core classes and interfaces, to be used by the functional contexts toolkit-outbox-jpa-postgres: Implementation of the transactional outbox using JPA for Postgres toolkit-cdc-debezium-postgres: Implementation of the message relay as CDC with Debezium embedded for Postgres toolkit-message-publisher-rabbitmq: Message publishers implementation for RabbitMQ toolkit-tx-spring: Provides programmatic transaction management with Spring toolkit-state-propagation-debezium-runtime: Runtime (service) responsible for the CDC and message publication to the RabbitMQ instance 2. Course Management Bounded Context These modules conform to the Course Management bounded context. The module adheres to hexagonal architecture principles, similar to the structure already used in my previous article about repository testing. Where: course-management-domain: Module with the Course Management domain definition, including entities, value objects, domain events, ports, etc; this module has no dependencies with frameworks, being as pure Java as possible. course-management-application: Following hexagonal architecture, this module orchestrates invocations to the domain model using commands and command handlers. course-management-repository-test: Contains the repository test definitions, with no dependencies to frameworks, only verifies the expectation of the repository ports defined in the course-management-domain module course-management-repository-jpa: JPA implementation of the repository interface CourseDefinitionRepository defined in the course-management-domain module; Leverages Spring Boot JPA course-management-repository-jpa-postgres: Specialization of the repository JPA implementation of the previous module, adding postgres specific concerns (e.g., Postgres database migration scripts) course-management-rest: Inbound web-based adapter exposing HTTP endpoints for course creation course-management-runtime: Runtime (service) of the Course Management context 3. Rating System Bounded Context For the sake of simplicity, this bounded context is partially implemented with only an inbound AMPQ-based adapter for receiving the messages created by the Course Management service when a new course is created and published by the CDC service (toolkit-state-propagation-debezium-runtime). Where: rating-system-amqp-listener: AMQP listener leveraging Spring Boot AMQP, subscribes to messages of the Course Management context rating-system-domain: No domain has been defined for the Rating System context. rating-system-application: No application layer has been defined for the Rating System context. rating-system-runtime: Runtime (service) of the Rating System context, starting the AMQP listener defined in rating-system-amqp-listener Request Flow This section outlines the flow of a request for creating a course, starting when the user requests the creation of a course to the backend API and finalizing with the (consistent) publication of the corresponding event in the message broker. This flow has been split into three phases. Phase 1: State Mutation and Domain Events Creation This phase starts with the request for creating a new Course Definition. The HTTP POST request is mapped to a domain command and processed by its corresponding command handler defined in course-management-application. Command handlers are automatically injected into the provided CommandBus implementation; in this example, the CommandBusInProcess defined in the toolkit-core module: Java import io.twba.tk.command.CommandHandler; import jakarta.inject.Inject; import jakarta.inject.Named; @Named public class CreateCourseDefinitionCommandHandler implements CommandHandler<CreateCourseDefinitionCommand> { private final CourseDefinitionRepository courseDefinitionRepository; @Inject public CreateCourseDefinitionCommandHandler(CourseDefinitionRepository courseDefinitionRepository) { this.courseDefinitionRepository = courseDefinitionRepository; } @Override public void handle(CreateCourseDefinitionCommand command) { if(!courseDefinitionRepository.existsCourseDefinitionWith(command.getTenantId(), command.getCourseDescription().title())) { courseDefinitionRepository.save(CourseDefinition.builder(command.getTenantId()) .withCourseDates(command.getCourseDates()) .withCourseDescription(command.getCourseDescription()) .withCourseObjective(command.getCourseObjective()) .withDuration(command.getCourseDuration()) .withTeacherId(command.getTeacherId()) .withPreRequirements(command.getPreRequirements()) .withCourseId(command.getCourseId()) .createNew()); } else { throw new IllegalStateException("Course definition with value " + command.getCourseDescription().title() + " already exists"); } } } The command handler creates an instance of the CourseDefinition entity. The business logic and invariants (if any) of creating a Course Definition are encapsulated within the domain entity. The creation of a new instance of the domain entity also comes with the corresponding CourseDefinitionCreated domain event: Java @Getter public class CourseDefinition extends MultiTenantEntity { /* ... */ public static class CourseDefinitionBuilder { /* ... */ public CourseDefinition createNew() { //new instance, generate domain event CourseDefinition courseDefinition = new CourseDefinition(tenantId, 0L, id, courseDescription, courseObjective, preRequirements, duration, teacherId, courseDates, CourseStatus.PENDING_TO_REVIEW); var courseDefinitionCreatedEvent = CourseDefinitionCreatedEvent.triggeredFrom(courseDefinition); courseDefinition.record(courseDefinitionCreatedEvent); //record event in memory return courseDefinition; } } } The event is "recorded" into the created course definition instance. This method is defined in the abstract class Entity of the toolkit-core module: Java public abstract class Entity extends ModelValidator implements ConcurrencyAware { @NotNull @Valid protected final List<@NotNull Event<? extends DomainEventPayload>> events; private Long version; public Entity(Long version) { this.events = new ArrayList<>(); this.version = version; } /*...*/ protected void record(Event<? extends DomainEventPayload> event) { event.setAggregateType(aggregateType()); event.setAggregateId(aggregateId()); event.setEventStreamVersion(Objects.isNull(version)?0:version + events.size()); this.events.add(event); } public List<Event<? extends DomainEventPayload>> getDomainEvents() { return Collections.unmodifiableList(events); } } Once the course definition instance is in place, the command handler will persist the instance in the course definition repository, starting the second phase of the processing flow. Phase 2, Persisting the State: Course Definition and Events in the Outbox Whenever a domain entity is saved in the repository, the domain events associated with the domain state mutation (in this example, the creating of a CourseDefinition entity) are temporarily appended to an in-memory, ThreadLocal, buffer. This buffer resides in the DomainEventAppender of toolkit-core. Java public class DomainEventAppender { private final ThreadLocal<List<Event<? extends DomainEventPayload>>> eventsToPublish = new ThreadLocal<>(); /*...*/ public void append(List<Event<? extends DomainEventPayload>> events) { //add the event to the buffer, later this event will be published to other bounded contexts if(isNull(eventsToPublish.get())) { resetBuffer(); } //ensure event is not already in buffer events.stream().filter(this::notInBuffer).map(this::addEventSourceMetadata).forEach(event -> eventsToPublish.get().add(event)); } /*...*/ } Events are placed in this buffer from an aspect executed around methods annotated with AppendEvents. The pointcut and aspect (both in toolkit-core) look like: Java public class CrossPointcuts { @Pointcut("@annotation(io.twba.tk.core.AppendEvents)") public void shouldAppendEvents() {} } @Aspect @Named public class DomainEventAppenderConcern { private final DomainEventAppender domainEventAppender; @Inject public DomainEventAppenderConcern(DomainEventAppender domainEventAppender) { this.domainEventAppender = domainEventAppender; } @After(value = "io.twba.tk.aspects.CrossPointcuts.shouldAppendEvents()") public void appendEventsToBuffer(JoinPoint jp) { if(Entity.class.isAssignableFrom(jp.getArgs()[0].getClass())) { Entity entity = (Entity)jp.getArgs()[0]; domainEventAppender.append(entity.getDomainEvents()); } } } The command handlers are automatically decorated before being "injected" into the command bus. One of the decorators ensures the command handlers are transactional, and another ensures when the transaction completes the events in the in-memory-thread-local buffer are published to the outbox table consistently with the ongoing transaction. The next sequence diagram shows the decorators applied to the domain-specific command handler. High-Level Processing Flow The outbox is an append-only buffer (as a Postgres table in this example) where a new entry for each event is added. The outbox entry has the following structure: Java public record OutboxMessage(String uuid, String header, String payload, String type, long epoch, String partitionKey, String tenantId, String correlationId, String source, String aggregateId) { } Where the actual event payload is serialized as a JSON string in the payload property. The outbox interface is straightforward: Java public interface Outbox { void appendMessage(OutboxMessage outboxMessage); int partitionFor(String partitionKey); } The Postgres implementation of the Outbox interface is placed in the toolkit-outbox-jpa-postgres module: Java public class OutboxJpa implements Outbox { private final OutboxProperties outboxProperties; private final OutboxMessageRepositoryJpaHelper helper; @Autowired public OutboxJpa(OutboxProperties outboxProperties, OutboxMessageRepositoryJpaHelper helper) { this.outboxProperties = outboxProperties; this.helper = helper; } @Override public void appendMessage(OutboxMessage outboxMessage) { helper.save(toJpa(outboxMessage)); } @Override public int partitionFor(String partitionKey) { return MurmurHash3.hash32x86(partitionKey.getBytes()) % outboxProperties.getNumPartitions(); } /*...*/ } Phase 2 is completed, and now both the domain state and the corresponding event consistently persist under the same transaction in our Postgres database. Phase 3: Events Publication In order to make the domain events generated in the previous phase available to external consumers, the message relay implementation based on Debezium Embedded is monitoring the outbox table, so that whenever a new record is added to the outbox, the message relay creates a Cloud Event and publishes it to the RabbitMQ instance following the Cloud Event AMQP binding specification. The following code snippet shows the Message Relay specification as defined in the toolkit-core module: Java public class DebeziumMessageRelay implements MessageRelay { private static final Logger log = LoggerFactory.getLogger(DebeziumMessageRelay.class); private final Executor executor = Executors.newSingleThreadExecutor(r -> new Thread(r, "debezium-message-relay")); private final CdcRecordChangeConsumer recordChangeConsumer; private final DebeziumEngine<RecordChangeEvent<SourceRecord>> debeziumEngine; public DebeziumMessageRelay(DebeziumProperties debeziumProperties, CdcRecordChangeConsumer recordChangeConsumer) { this.debeziumEngine = DebeziumEngine.create(ChangeEventFormat.of(Connect.class)) .using(DebeziumConfigurationProvider.outboxConnectorConfig(debeziumProperties).asProperties()) .notifying(this::handleChangeEvent) .build(); this.recordChangeConsumer = recordChangeConsumer; } private void handleChangeEvent(RecordChangeEvent<SourceRecord> sourceRecordRecordChangeEvent) { SourceRecord sourceRecord = sourceRecordRecordChangeEvent.record(); Struct sourceRecordChangeValue= (Struct) sourceRecord.value(); log.info("Received record - Key = '{}' value = '{}'", sourceRecord.key(), sourceRecord.value()); Struct struct = (Struct) sourceRecordChangeValue.get(AFTER); recordChangeConsumer.accept(DebeziumCdcRecord.of(struct)); } @Override public void start() { this.executor.execute(debeziumEngine); log.info("Debezium CDC started"); } @Override public void stop() throws IOException { if (this.debeziumEngine != null) { this.debeziumEngine.close(); } } @Override public void close() throws Exception { stop(); } } As can be seen in the code snippet above, the DebeziumEngine is configured to notify a private method handleChangeEvent when a change in the database is detected. In this method, a Consumer of CdcRecord is used as a wrapper of the internal Debezium model represented by the Struct class. Initial configuration must be provided to the Debezium Engine; in the example, this is done with the DebeziumConfigurationProvider class: Java public class DebeziumConfigurationProvider { public static io.debezium.config.Configuration outboxConnectorConfig(DebeziumProperties properties) { return withCustomProps(withStorageProps(io.debezium.config.Configuration.create() .with("name", "outbox-connector") .with("connector.class", properties.getConnectorClass()) .with("offset.storage", properties.getOffsetStorage().getType()), properties.getOffsetStorage()) .with("offset.flush.interval.ms", properties.getOffsetStorage().getFlushInterval()) .with("database.hostname", properties.getSourceDatabaseProperties().getHostname()) .with("database.port", properties.getSourceDatabaseProperties().getPort()) .with("database.user", properties.getSourceDatabaseProperties().getUser()) .with("database.password", properties.getSourceDatabaseProperties().getPassword()) .with("database.dbname", properties.getSourceDatabaseProperties().getDbName()), properties) .with("database.server.id", properties.getSourceDatabaseProperties().getServerId()) .with("database.server.name", properties.getSourceDatabaseProperties().getServerName()) .with("skipped.operations", "u,d,t") .with("include.schema.changes", "false") .with("table.include.list", properties.getSourceDatabaseProperties().getOutboxTable()) .with("snapshot.include.collection.list", properties.getSourceDatabaseProperties().getOutboxTable()) .build(); } private static Configuration.Builder withStorageProps(Configuration.Builder debeziumConfigBuilder, DebeziumProperties.OffsetProperties offsetProperties) { offsetProperties.getOffsetProps().forEach(debeziumConfigBuilder::with); return debeziumConfigBuilder; } private static Configuration.Builder withCustomProps(Configuration.Builder debeziumConfigBuilder, DebeziumProperties debeziumProperties) { debeziumProperties.getCustomProps().forEach(debeziumConfigBuilder::with); return debeziumConfigBuilder; } } The most relevant properties are outlined below: connector.class: The name of the connector to use; usually this is related to the database technology being tracked for changes. In this example, we are using io.debezium.connector.postgresql.PostgresConnector. offset.storage: Type of storage for maintaining the offset; in this example, we are using org.apache.kafka.connect.storage.MemoryOffsetBackingStore, so that offsets are lost after restarting the service. See below. offset.flush.interval. ms: Number of milliseconds for the offsets to be persisted in the offset store database.*: These properties refer to the database being tracked (CDC) for changes by Debezium. skipped.operations: If not specified, all the operations will be tracked. For our example, since we only want to detect newly created events, all the operations but inserts are skipped. table.include.list: List of the tables to include for the CDC; in this example, only the table where events are stored during Phase 2 is relevant (i.e., outbox_schema.outbox) The first thing Debezium will do after starting is take a snapshot of the current data and generate the corresponding events. After that, the offset is updated to the latest record, and the deltas (newly added, updated, or deleted records) are processed, updating the offset accordingly. Since in the provided example we are using an in-memory-based offset store, the snapshot is performed always after starting the service. Therefore, since this is not a production-ready implementation yet, there are two options: Use a durable offset store (file-based or Kafka-based are supported by Debezium Embedded) Delete the outbox table after the events are being processed, ensuring the delete operations are skipped by Debezium in its configuration. The Message Relay is configured and initialized in the toolkit-state-propagation-debezium-runtime module. The values of the configuration properties needed by Debezium Embedded are defined in the Spring Boot properties.yaml file: YAML server: port: 9091 management: endpoints: web: exposure: include: prometheus, health, flyway, info debezium: connector-class: "io.debezium.connector.postgresql.PostgresConnector" custom-props: "[topic.prefix]": "embedded-debezium" "[debezium.source.plugin.name]": "pgoutput" "[plugin.name]": "pgoutput" source-database-properties: db-name: "${CDC_DB_NAME}" hostname: "${CDC_HOST}" user: "${CDC_DB_USER}" password: "${CDC_DB_PASSWORD}" port: 5432 server-name: "debezium-message-relay" server-id: "debezium-message-relay-1" outbox-table: "${CDC_OUTBOX_TABLE}:outbox_schema.outbox" outbox-schema: "" offset-storage: type: "org.apache.kafka.connect.storage.MemoryOffsetBackingStore" flush-interval: 3000 offset-props: "[offset.flush.timeout.ms]": 1000 "[max.poll.records]": 1000 The engine is started using the Spring Boot lifecycle events: Java @Component public class MessageRelayInitializer implements ApplicationListener<ApplicationReadyEvent> { private final MessageRelay messageRelay; @Autowired public MessageRelayInitializer(MessageRelay messageRelay) { this.messageRelay = messageRelay; } @Override public void onApplicationEvent(ApplicationReadyEvent event) { messageRelay.start(); } } The change data capture records (CdcRecord) are processed by the CloudEventRecordChangeConsumer which creates the Cloud Event representation of the CDC record and publishes it through the MessagePublisher. Java public class CloudEventRecordChangeConsumer implements CdcRecordChangeConsumer { /*...*/ @Override public void accept(CdcRecord cdcRecord) { final CloudEvent event; try { String payload = cdcRecord.valueOf("payload"); String uuid = cdcRecord.valueOf("uuid"); String type = cdcRecord.valueOf("type"); String tenantId = cdcRecord.valueOf("tenant_id"); String aggregateId = cdcRecord.valueOf("aggregate_id"); long epoch = cdcRecord.valueOf("epoch"); String partitionKey = cdcRecord.valueOf("partition_key"); String source = cdcRecord.valueOf("source"); String correlationId = cdcRecord.valueOf("correlation_id"); event = new CloudEventBuilder() .withId(uuid) .withType(type) .withSubject(aggregateId) .withExtension(TwbaCloudEvent.CLOUD_EVENT_TENANT_ID, tenantId) .withExtension(TwbaCloudEvent.CLOUD_EVENT_TIMESTAMP, epoch) .withExtension(TwbaCloudEvent.CLOUD_EVENT_PARTITION_KEY, partitionKey) .withExtension(TwbaCloudEvent.CLOUD_EVENT_CORRELATION_ID, correlationId) .withExtension(TwbaCloudEvent.CLOUD_EVENT_GENERATING_APP_NAME, source) .withSource(URI.create("https://thewhiteboardarchitect.com/" + source)) .withData("application/json",payload.getBytes("UTF-8")) .build(); messagePublisher.publish(event); } catch (Exception e) { throw new RuntimeException(e); } } } The provided MessagePublisher is a simple RabbitMQ outbound adapter converting the Cloud Event to the corresponding AMQP message as per the Cloud Event AMQP protocol binding. Java public class MessagePublisherRabbitMq implements MessagePublisher { /*...*/ @Override public boolean publish(CloudEvent dispatchedMessage) { MessageProperties messageProperties = new MessageProperties(); messageProperties.setContentType(MessageProperties.CONTENT_TYPE_JSON); rabbitTemplate.send("__MR__" + dispatchedMessage.getExtension(CLOUD_EVENT_GENERATING_APP_NAME), //custom extension for message routing dispatchedMessage.getType(), toAmqpMessage(dispatchedMessage)); return true; } private static Message toAmqpMessage(CloudEvent dispatchedMessage) { return MessageBuilder.withBody(Objects.nonNull(dispatchedMessage.getData()) ? dispatchedMessage.getData().toBytes() : new byte[0]) .setContentType(MessageProperties.CONTENT_TYPE_JSON) .setMessageId(dispatchedMessage.getId()) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_TENANT_ID, dispatchedMessage.getExtension(CLOUD_EVENT_TENANT_ID)) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_TIMESTAMP, dispatchedMessage.getExtension(CLOUD_EVENT_TIMESTAMP)) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_PARTITION_KEY, dispatchedMessage.getExtension(CLOUD_EVENT_PARTITION_KEY)) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_SUBJECT, dispatchedMessage.getSubject()) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_SOURCE, dispatchedMessage.getSource().toString()) .build(); } } After Phase 3, the events are published to the producer's (Course Management Service) message relay RabbitMQ exchange defined under the convention __MR__<APP_NAME>. In our example, __MR__course-management, messages are routed to the right exchange based on a custom cloud event extension as shown in the previous code snippet. Visit my GitHub repository and check the readme file to see how to spin up the example. Alternative Solutions This example makes use of Debezium Embedded to provide a change data capture and message relay solution. This works fine for technologies supported by Debezium through its connectors. For non-supported providers, alternative approaches can be applied: DynamoDB Streams: Suitable for DynamoDB databases; in combination with Kinesis, it can be used for subscribing to changes in a DynamoDB (outbox) table Custom database polling: This could be implemented for supporting databases with no connectors for Debezium. Adding any of those alternatives in this example would simply just provide specific implementations of the MessageRelay interface, without additional changes in any of the other services. Conclusion Ensuring consistency in the state propagation and data exchange between services is key for providing a reliable distributed solution. Usually, this is not carefully considered when designing distributed, event-driven software, leading to undesired states and situations especially when those systems are already in production. By the combination of the transactional outbox pattern and message relay, we have seen how this consistency can be enforced, and by using the hexagonal architecture style, the additional complexity of implementing those patterns can be easily hidden and reused across bounded context implementations. The code of this article is not yet production-ready, as concerns like observability, retries, scalability (e.g., with partitioning), proper container orchestration, etc. are still pending. Subsequent articles will go through those concerns using the provided code as a basis.
Back in 2018, I decided to use my free time to help modernize a family member’s business. Along the way, I wanted to gain some experience and understanding of AWS. Ultimately, I discovered that nearly all my free time was spent learning AWS cloud infrastructure concepts. I had only a small fraction of my time left to focus on building the modern, cloud-based solution that I originally had in mind. As I planned more feature requests for the app, I realized that I needed a better approach. In early 2020, I discovered Heroku. Since I didn’t need to worry about underlying cloud configurations, I could focus my time on adding new features. The Heroku ecosystem worked out great for my simple use case, but I began to wonder about more complex use cases. What about the scenario where a collection of secure and private services needs to interact with one another for a payment processing solution? Would this use case force me to live in the ecosystem of one of the big three cloud service providers? I’m going to find out. The Current State of Secure Microservices For several years, I was fortunate to work in an environment that valued the DevOps lifecycle. The DevOps team handled all things cloud for me, so I could focus on architecting and building microservices to meet my customers’ needs. During that time of my life, this environment was the exception, not the norm. I just did a search for “companies lacking cloud infrastructure knowledge” in my browser, and the results yielded some pretty surprising conclusions: There is a serious shortage of cloud expertise. The lack of cloud skills is leading to significant performance impacts with cloud-native services. Cloud security is a challenge for 1 in 4 companies. The top search results talked about a lack of understanding of the core cloud concepts and the need for crucial training for teams to be effective. The training most teams need usually falls by the wayside as customer demands and deliverables take higher priority. With this current approach, most cloud implementations are forced to move at a slower pace, and they’re often exposed to unknown vulnerabilities. The current state of securing your microservices in the cloud is not a happy one. The Ideal State for Secure Microservices The ideal state for cloud-native solutions would adhere to a personal mission statement I established several years ago: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” – J. Vester In this context, those with a directive to drive toward cloud-native solutions should be able to move at a pace that aligns with corporate objectives. They shouldn’t be slowed by the learning curve associated with underlying cloud infrastructure. So, what does this look like when we’re facing a cloud solution encompassing multiple microservices, all of which need to be isolated from the public and adhere to compliance regulations (like SOC, ISO, PCI, or HIPAA)? About Private Spaces My 2020 Heroku experience was positive. So I wanted to see how it would work with this complex use case. That’s when I discovered Private Spaces. Private Spaces are available as a part of Heroku Enterprise. They’re dedicated environments for running microservices within an isolated network. This approach allows teams to deploy their services into a network that’s not exposed to the public internet. Under the hood, these services function exactly the same as in my basic use case. I can set them up through the Heroku CLI, and simple Git-based commands can trigger deployments. For the regulatory compliance needs, I can lean on Heroku Shield to help me comply with PCI DSS, HIPAA, ISO (27001, 27017, and 27018), and SOC (1, 2, and 3). At a high level, Heroku lets me implement a secure cloud-native design that can be illustrated like this: Here, we have an implementation that leverages Heroku Shield all within a Private Space. This allows a collection of microservices — utilizing several different programming languages — to interact with all the major primary and secondary card networks, all while adhering to various regulatory compliance requirements. Additionally, I get secure communications with the Salesforce platform and GitLab. Heroku in Action Using the Heroku CLI, I can get my Private Space and Heroku Shield up and running. In Heroku, this is called a Shield Private Space. Here are some high-level examples to work through the process. To create a new Shield Private Space, we use spaces:create and add the --shield option. Shell $ heroku spaces:create payment-network --shield --team payments-team --region oregon Creating space payment-network in team payments-team... done === payment-network Team: payments-team Region: oregon State: allocating If the use case requires Classless Inter-Domain Routing (CIDR) ranges, I can use --cidr and --data-cidr flags. You’ll notice that I created my Private Space in the oregon region. You can create a Private Space in one of 10 available regions (in the U.S., Europe, Asia, and Australia). For a list of available regions, do the following: Shell $ heroku regions ID Location Runtime ───────── ─────────────────────── ────────────── eu Europe Common Runtime us United States Common Runtime dublin Dublin, Ireland Private Spaces frankfurt Frankfurt, Germany Private Spaces london London, United Kingdom Private Spaces montreal Montreal, Canada Private Spaces mumbai Mumbai, India Private Spaces oregon Oregon, United States Private Spaces singapore Singapore Private Spaces sydney Sydney, Australia Private Spaces tokyo Tokyo, Japan Private Spaces virginia Virginia, United States Private Spaces For each microservice that needs to run in the payment-network Private Space, I simply add the --space option when running the apps:create command: Shell $ heroku apps:create clearing-service --space payment-network Creating app... done, clearing-service To grant consumers access to the payment-network space, I can maintain the allowed list of trusted IPs: Shell $ heroku trusted-ips:add 192.0.2.128/26 --space payment-network Added 192.0.2.128/26 to trusted IP ranges on payment-network - WARNING: It may take a few moments for the changes to take effect. Conclusion Teams are often given a directive from above to adopt a cloud-native approach. However, many teams have a serious gap in understanding when it comes to deploying secure cloud architectures. If you’re using one of the big three cloud providers, then bridging this gap will come at a price—likely missed timelines expected by the product owner. Is there a better option for secure cloud deployment? I think Private Spaces combined with Heroku Shield represents a better option. For me personally, it also matters that Heroku is part of the solutions platform from Salesforce, which has a history of dedication to providing a cloud adoption alternative focused on the success of its customers. So I felt like this was a long-term strategy for consideration. Have a really great day!
Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Cloud Native: Championing Cloud Development Across the SDLC. When it comes to software engineering and application development, cloud native has become commonplace in many teams' vernacular. When people survey the world of cloud native, they often come away with the perspective that the entire process of cloud native is for the large enterprise applications. A few years ago, that may have been the case, but with the advancement of tooling and services surrounding systems such as Kubernetes, the barrier to entry has been substantially lowered. Even so, does adopting cloud-native practices for applications consisting of a few microservices make a difference? Just as cloud native has become commonplace, the shift-left movement has made inroads into many organizations' processes. Shifting left is a focus on application delivery from the outset of a project, where software engineers are just as focused on the delivery process as they are on writing application code. Shifting left implies that software engineers understand deployment patterns and technologies as well as implement them earlier in the SDLC. Shifting left using cloud native with microservices development may sound like a definition containing a string of contemporary buzzwords, but there's real benefit to be gained in combining these closely related topics. Fostering a Deployment-First Culture Process is necessary within any organization. Processes are broken down into manageable tasks across multiple teams with the objective being an efficient path by which an organization sets out to reach a goal. Unfortunately, organizations can get lost in their processes. Teams and individuals focus on doing their tasks as best as possible, and at times, so much so that the goal for which the process is defined gets lost. Software development lifecycle (SDLC) processes are not immune to this problem. Teams and individuals focus on doing their tasks as best as possible. However, in any given organization, if individuals on application development teams are asked how they perceive their objectives, responses can include: "Completing stories" "Staying up to date on recent tech stack updates" "Ensuring their components meet security standards" "Writing thorough tests" Most of the answers provided would demonstrate a commitment to the process, which is good. However, what is the goal? The goal of the SDLC is to build software and deploy it. Whether it be an internal or SaaS application, deploying software helps an organization meet an objective. When presented with the statement that the goal of the SDLC is to deliver and deploy software, just about anyone who participates in the process would say, "Well, of course it is." Teams often lose sight of this "obvious" directive because they're far removed from the actual deployment process. A strategic investment in the process can close that gap. Cloud-native abstractions bring a common domain and dialogue across disciplines within the SDLC. Kubernetes is a good basis upon which cloud-native abstractions can be leveraged. Not only does Kubernetes' usefulness span applications of many shapes and sizes, but when it comes to the SDLC, Kubernetes can also be the environment used on systems ranging from local engineering workstations, though the entire delivery cycle, and on to production. Bringing the deployment platform all the way "left" to an engineer's workstation has everyone in the process speaking the same language, and deployment becomes a focus from the beginning of the process. Various teams in the SDLC may look at "Kubernetes Everywhere" with skepticism. Work done on Kubernetes in reducing its footprint for systems such as edge devices has made running Kubernetes on a workstation very manageable. Introducing teams to Kubernetes through automation allows them to iteratively absorb the platform. The most important thing is building a deployment-first culture. Plan for Your Deployment Artifacts With all teams and individuals focused on the goal of getting their applications to production as efficiently and effectively as possible, how does the evolution of application development shift? The shift is subtle. With a shift-left mindset, there aren't necessarily a lot of new tasks, so the shift is where the tasks take place within the overall process. When a detailed discussion of application deployment begins with the first line of code, existing processes may need to be updated. Build Process If software engineers are to deploy to their personal Kubernetes clusters, are they able to build and deploy enough of an application that they're not reliant on code running on a system beyond their workstation? And there is more to consider than just application code. Is a database required? Does the application use a caching system? It can be challenging to review an existing build process and refactor it for workstation use. The CI/CD build process may need to be re-examined to consider how it can be invoked on a workstation. For most applications, refactoring the build process can be accomplished in such a way that the goal of local build and deployment is met while also using the refactored process in the existing CI/CD pipeline. For new projects, begin by designing the build process for the workstation. The build process can then be added to a CI/CD pipeline. The local build and CI/CD build processes should strive to share as much code as possible. This will keep the entire team up to date on how the application is built and deployed. Build Artifacts The primary deliverables for a build process are the build artifacts. For cloud-native applications, this includes container images (e.g., Docker images) and deployment packages (e.g., Helm charts). When an engineer is executing the build process on their workstation, the artifacts will likely need to be published to a repository, such as a container registry or chart repository. The build process must be aware of context. Existing processes may already be aware of their context with various settings for environments ranging from test and staging to production. Workstation builds become an additional context. Given the awareness of context, build processes can publish artifacts to workstation-specific registries and repositories. For cloud-native development, and in keeping with the local workstation paradigm, container registries and chart repositories are deployed as part of the workstation Kubernetes cluster. As the process moves from build to deploy, maintaining build context includes accessing resources within the current context. Parameterization Central to this entire process is that key components of the build and deployment process definition cannot be duplicated based on a runtime environment. For example, if a container image is built and published one way on the local workstation and another way in the CI/CD pipeline. How long will it be before they diverge? Most likely, they diverge sooner than expected. Divergence in a build process will create a divergence across environments, which leads to divergence in teams and results in the eroding of the deployment-first culture. That may sound a bit dramatic, but as soon as any code forks — without a deliberate plan to merge the forks — the code eventually becomes, for all intents and purposes, unmergeable. Parameterizing the build and deployment process is required to maintain a single set of build and deployment components. Parameters define build context such as the registries and repositories to use. Parameters define deployment context as well, such as the number of pod replicas to deploy or resource constraints. As the process is created, lean toward over-parameterization. It's easier to maintain a parameter as a constant rather than extract a parameter from an existing process. Figure 1. Local development cluster Cloud-Native Microservices Development in Action In addition to the deployment-first culture, cloud-native microservices development requires tooling support that doesn't impede the day-to-day tasks performed by an engineer. If engineers can be shown a new pattern for development that allows them to be more productive with only a minimum-to-moderate level of understanding of new concepts, while still using their favorite tools, the engineers will embrace the paradigm. While engineers may push back or be skeptical about a new process, once the impact on their productivity is tangible, they will be energized to adopt the new pattern. Easing Development Teams Into the Process Changing culture is about getting teams on board with adopting a new way of doing something. The next step is execution. Shifting left requires that software engineers move from designing and writing application code to becoming an integral part of the design and implementation of the entire build and deployment process. This means learning new tools and exploring areas in which they may not have a great deal of experience. Human nature tends to resist change. Software engineers may look at this entire process and think, "How can I absorb this new process and these new tools while trying to maintain a schedule?" It's a valid question. However, software engineers are typically fine with incorporating a new development tool or process that helps them and the team without drastically disrupting their daily routine. Whether beginning a new project or refactoring an existing one, adoption of a shift-left engineering process requires introducing new tools in a way that allows software engineers to remain productive while iteratively learning the new tooling. This starts with automating and documenting the build out of their new development environment — their local Kubernetes cluster. It also requires listening to the team's concerns and suggestions as this will be their daily environment. Dev(elopment) Containers The Development Containers specification is a relatively new advancement based on an existing concept in supporting development environments. Many engineering teams have leveraged virtual desktop infrastructure (VDI) systems, where a developer's workstation is hosted on a virtualized infrastructure. Companies that implement VDI environments like the centralized control of environments, and software engineers like the idea of a pre-packaged environment that contains all the components required to develop, debug, and build an application. What software engineers do not like about VDI environments is network issues where their IDEs become sluggish and frustrating to use. Development containers leverage the same concept as VDI environments but bring it to a local workstation, allowing engineers to use their locally installed IDE while being remotely connected to a running container. This way, the engineer has the experience of local development while connected to a running container. Development containers do require an IDE that supports the pattern. What makes the use of development containers so attractive is that engineers can attach to a container running within a Kubernetes cluster and access services as configured for an actual deployment. In addition, development containers support a first-class development experience, including all the tools a developer would expect to be available in a development environment. From a broader perspective, development containers aren't limited to local deployments. When configured for access, cloud environments can provide the same first-class development experience. Here, the deployment abstraction provided by containerized orchestration layers really shines. Figure 2. Microservice development container configured with dev containers The Synergistic Evolution of Cloud-Native Development Continues There's a synergy across shift-left, cloud-native, and microservices development. They present a pattern for application development that can be adopted by teams of any size. Tooling continues to evolve, making practical use of the technologies involved in cloud-native environments accessible to all involved in the application delivery process. It is a culture change that entails a change in mindset while learning new processes and technologies. It's important that teams aren't burdened with a collection of manual processes where they feel their productivity is being lost. Automation helps ease teams into the adoption of the pattern and technologies. As with any other organizational change, upfront planning and preparation is important. Just as important is involving the teams in the plan. When individuals have a say in change, ownership and adoption become a natural outcome. This is an excerpt from DZone's 2024 Trend Report, Cloud Native: Championing Cloud Development Across the SDLC.Read the Free Report
Amol Gote
Solution Architect,
Innova Solutions (Client - iCreditWorks Start Up)
Ray Elenteny
Solution Architect,
SOLTECH
Nicolas Duminil
Silver Software Architect,
Simplex Software
Satrajit Basu
Chief Architect,
TCG Digital