Software Design and Architecture

With many microservices deployed across multi-cloud and hybrid infrastructure (cloud, containers, and VMs), the manageability of the network becomes challenging. The transactions among services happen on the public network, so the sensitivity of the matter increases magnitudinally with rising incidents of hacking and cyberattacks. Istio service mesh is becoming a center of app modernization for large and medium enterprises. Due to Istio’s phenomenal ability to manage and secure the network across cloud and container workloads, the cloud team and DevOps platform teams consider Istio service mesh for the first round of evaluation. Configuring Istio for multiple clusters in the same cloud (say AWS EKS- US-west and US-east) is comparatively easy. But organizations may have their microservices in different clouds (say database transaction in AWS and AI/ML processing in GKE), and Istio implementation can be tricky in those instances. So this article will help and guide anyone who wants to implement Istio in AWS EKS and wants to manage multiple clusters (say GKE, AKS). Prerequisites Istio version 1.17 Ready-to-use AWS EKS (primary cluster) and AKS (secondary cluster) Configure the environment variables Terminal to access primary and secondary clusters through kubectl Refer to all the files in the Github repo Full Video on Multicluster Istio Setup in AWS EKS If you are comfortable referring to a video to implement Istio in AWS EKS, then watch the following video: Steps There are four simple steps to implement Istio in AWS EKS and manage AKS clusters: Install and configure Istio in AWS EKS Configure the remote cluster- AKS Allow Istio in GKE to access the remote cluster Deploy applications in each cluster and validate mTLS Note: We will not implement L4 and L7 authorization policies using Istio. You can refer to the full video in the above section or the how to implement Istio in multicloud and multicluster (GKE/AKS) blog post. Step 1: Install and Configure Istio in AWS EKS First, we have set the environment variables in our PowerShell for each cluster — AKS for the AKS cluster and EKS for the EKS cluster. We have set up and configured Istio in clusters EKS and AKS so that apps in each cluster can talk to each other using an east-west Istio ingress gateway. Please refer to the image of the Istio configuration we aim to achieve. Step 1.1: Install Istio Use the following command to install Istio in EKS. Shell istioctl install -f cluster-eks-primary.yaml -y %EKS% You will see the following output. Step 1.2: Install the Istio Gateway in the EKS Cluster We will use the Istio operator to install an ingress east-west gateway in the AWS EKS cluster that can handle traffic from outside the cluster — from AKS. (Please note: For this demo, we have added service annotations in the YAML file to create a network load balancer [NLB] instead of the classic load balancer. Classic load balancers will not have a static IP which can be problematic while scaling and descaling resources. But with NLB, we will get a static IP, and the secondary cluster AKS can access the primary cluster. For production, it is advisable to implement Istio with a classic load balancer.) You can refer to the east-west-gateway-cluster-eks.yaml file below: YAML apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: name: eastwest spec: revision: "" profile: empty components: ingressGateways: - name: istio-eastwestgateway label: istio: eastwestgateway app: istio-eastwestgateway topology.istio.io/network: network1 enabled: true k8s: env: # traffic through this gateway should be routed inside the network - name: ISTIO_META_REQUESTED_NETWORK_VIEW value: network1 service: ports: - name: status-port port: 15021 targetPort: 15021 - name: tls port: 15443 targetPort: 15443 - name: tls-istiod port: 15012 targetPort: 15012 - name: tls-webhook port: 15017 targetPort: 15017 values: gateways: istio-ingressgateway: injectionTemplate: gateway serviceAnnotations: service.beta.kubernetes.io/aws-load-balancer-type: “nlb” global: network: network1 You can run the following command to install the ingress east-west gateway: Shell istioctl install -f east-west-gateway-cluster-eks.yaml %EKS% -y Step 1.3: Expose Istio Services in EKS You can expose the Istio service in EKS so that secondary clusters can access it. Use the following command: Shell kubectl apply -f expose-istiod.yaml %EKS% Kubectl apply -f expose-services.yaml %EKS% Step 1.4: Get the IP of Istio East-West Gateway in the EKS Cluster Run the command to get the address of the network load balancer created by the Istio east-west ingress gateway. Shell kubectl get service -n istio-system %EKS% Copy the external IP address of the network load balancer (east-west gateway) and ping it to get the IP. We will use the IP address in the AKS cluster in the below setup. If you want to deploy an application in the EKS cluster into any Istio-enabled namespace, then an Envoy proxy will be attached to each workload. Step 2: Configure the Remote Cluster AKS Step 2.1: Create a Namespace in AKS by Specifying EKS as the Primary Control Plane Use the following yaml file to create a namespace called istio-systems in AKS with EKS as the primary cluster for the Istio control plane. We have named it cluster-aks-remote-namespace-prep.yaml. YAML apiVersion: v1 kind: Namespace metadata: name: istio-system labels: topology.istio.io/network: network2 annotations: topology.istio.io/controlPlaneClusters: cluster-eks Deploy the AKS remote in PowerShell using the command: Shell kubectl create -f cluster-aks-remote-namespace-prep.yaml %AKS% Step 2.2: Install the Istio Operator in Azure Kubernetes (AKS) Cluster Using the IP of LB in the Primary Cluster EKS We have used the following declaration in cluster-aks-remote.yaml to install Istio in the AKS cluster. You can use the IP address copied from the above step in the remotePilotAddress section. YAML apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: profile: remote values: istiodRemote: injectionPath: /inject/cluster/cluster-aks/net/network2 global: remotePilotAddress: <replace with ip of east-west gateway of primary cluster> proxy: privileged: true Use this command to install Istio in AKS: Shell istioctl install -f cluster-aks-remote.yaml %AKS% We need to ensure the primary cluster can access the remote cluster. This is only possible by exposing the secrets of the remote cluster to the primary cluster. Step 3: Allow Istio in EKS to Access the API Server of AKS The Istio control plane in EKS needs to access the API server of AKS to perform its core activities, such as service discovery, patching webhooks, etc. We can achieve that by generating a remote secret and applying the remote secret in the primary cluster AWS EKS. Step 3.1: Create Remote Cluster Secrets in AKS Cluster Use the following command to generate the remote secret of the remote cluster (AKS) and store it in a secret yaml file: Shell istioctl x create-remote-secret -name-cluster-aks > apiserver-creds-aks.yaml %AKS% The output file apiserver-creds-aks.yaml will look something like below: Step 3.2: Apply the Remote Cluster Secrets in the Primary Cluster (EKS) Use the following command to implement the secrets in EKS so that it can access the API server of AKS: Shell kubectl apply -f .\apiserver-creds-aks.yaml %EKS% Note: Apply the remote credentials first to connect both clusters and then expose the clusters’ services. Otherwise, there will be errors. Step 3.3: Install East-West Ingress Gateway in Remote Cluster AKS Use the command to install east-west ingress gateway controllers in AKS. Shell istioctl install -f east-west-gateway-cluster-aks.yaml %AKS% After the controller is installed, we will create a gateway of an east-west kind in the remote cluster by applying the following commands: Shell kubectl apply -f .\expose-istiod.yaml %AKS% kubectl apply -f .\expose-services.yaml %AKS% All the configurations for the multicluster Istio setup are done. Now we will deploy applications and check the multicluster communications. Step 4: Deploy a Multicloud Application and Verify How Istio Handles the Traffic You can refer to the Git source URL to download and deploy the demo-app to your cluster. The demo-app has 4 microservices: accounts, dashboard, profile, and sleep. The idea is that when we send a request to the dashboard service with an ID (say 4), it will communicate with the profile services to identify the person’s name and communicate with the account service to find the respective account balance. The deployment and service yaml files are in the Git repo. We have created an Istio-enabled namespace called multicluster and deployed all the applications there. Deploy all four services in the EKS cluster. Deploy just the profile service in the AKS cluster We will realize that though we have mentioned the count of replicas as one in all the deployment files, there will be two pods for each service: one application pod and one envoy proxy. After the deployment, if you want to access the profile services from one of the pods of sleep service in EKS, then you will see the east-west gateway is load-balancing the communication between the pods of profile services in AKS and EKS. You can also log in to any Envoy proxy pods to see how these proxies carry out the message. That’s the end of conjuring Istio in EKS for managing the network of multi-cloud and multicluster applications. Conclusion We have seen how Istio can abstract the network layer out of the core business logic or the cloud-native applications. It becomes effortless to manage the network and apply advanced strategies such as timeouts, retries, and failovers to ensure the high availability of applications. With the network getting abstracted, ensuring 100% security (or zero trust) becomes extremely easy. Stakeholders can make a global decision to encrypt all the east-west traffic with mTLS or granular authorization (RBAC) for chosen workloads. Platform and cloud engineers can quickly implement the security policies with the Istio control plane.

The Competing Consumers pattern is a powerful approach that enables multiple consumer applications to process messages from a shared queue in a parallel and distributed manner. The implementation of this messaging pattern requires the utilization of a point-to-point channel. Senders deliver messages (or work items) to this channel and consumers compete with each other to be the receiver and process the message (i.e., perform some work based on the contents of the message) in an asynchronous and non-blocking manner. How Does It Work? Usually, a message broker (e.g., RabbitMQ, ActiveMQ) or a distributed streaming platform (e.g., Apache Kafka) is needed in order to provide the necessary infrastructure to implement this pattern. Producer applications connect to the broker and push messages to a queue (or a Kafka Topic with multiple partitions implemented as point-to-point channels). Then, consumer applications that are also connected to the broker, pull items from the queue (or belong to the same Kafka Consumer Group and pick up items from a designated partition). Each message is effectively delivered initially to only one available consumer or remains in the queue till a consumer becomes available. Error Handling In case of error during message processing, there are many things to consider that depending on the use case, may complicate things. Many brokers support acknowledgments and transactions while others rely on each microservice or application to implement its logic with its own choice of technology, communication paradigm, and error handling. If a consumer fails to send back an ACK after receiving and processing the message in a timely manner or rolls back the transaction when an exception occurs, the message may automatically (or based on application logic) be requeued or end up in a dead letter queue (DLQ). However, there are always corner cases and scenarios that need attention. For example, if the consumer process has performed some work, and for some reason, "hangs" or there is a network error, then reprocessing the message may not be 100% safe. It depends on how idempotent the consumer processing logic is, and, depending on the use case, a manual inspection may be needed in order to decide if requeueing the message or processing it from a DLQ is ok. This is why we often run into delivery guarantee semantics like "at most once," "at least once," or "exactly once." The latter is the ideal, but in practice is technically impossible. Therefore aiming to achieve effectively once processing (i.e., as close as possible to "exactly once") should be our realistic goal. In case of error messaging on the producer side, or if the message broker is unavailable, things are theoretically a bit less complicated. A recommended pattern to use on this side is the transactional outbox pattern. It involves storing messages in an "outbox" table or collection within the system's database and using a background process to periodically look up and deliver these messages. Only after the broker confirms it received the message, it is considered as sent. Alternatives Based on all the above, the existence of a central server that will act as the messaging broker seems essential. Actually, for production systems, we will need to set up a cluster consisting of multiple broker instances, running on separate VMs to ensure high availability, failover, replication, and so on. If for some reason this is not possible (operational costs, timelines, learning curve, etc.), other solutions exist that depending on the use case may cover our needs. For simple cases, one could utilize a database table as a "queue" and have multiple consumer instances, polling the database periodically. If they find a record in a status appropriate for processing, then they have to apply a locking mechanism in order to make sure no other instance will duplicate the work. Upon finishing the task they need a release to update the status of the record and release the lock. Another solution is the one that we will demonstrate in the following sections. It relies on using Hazelcast in embedded mode and taking advantage of the distributed data structures it provides, namely the distributed queue. Solution Description For our demonstration, we will use 2 microservices implemented with Spring Boot: the Producer and the Consumer. Multiple instances of them can be executed. Hazelcast is included as a dependency on both of them, along with some basic Java configuration. As mentioned, Hazelcast will run in embedded mode, meaning that there will be no Hazelcast server. The running instances, upon startup, will use a discovery mechanism and form a decentralized cluster or Hazelcast IMDG (in-memory data grid). Hazelcast can also be used in the classic client-server mode. Each approach has its advantages and disadvantages. You may refer to Choosing an Application Topology for more details. We have selected to have the Producers configured as Hazelcast "Members" and the consumers as Hazelcast "Clients." Let's have a look at the characteristics of each: Hazelcast Client: A Hazelcast client is an application that connects to a Hazelcast cluster to interact with it. It acts as a lightweight gateway or proxy to the Hazelcast cluster, allowing clients to access and utilize the distributed data structures and services provided by Hazelcast. The client does not store or own data; it simply communicates with the Hazelcast cluster to perform operations on distributed data. Clients can be deployed independently of the Hazelcast cluster and can connect to multiple Hazelcast clusters simultaneously. Clients are typically used for read and write operations, as well as for executing distributed tasks on the Hazelcast cluster. Clients can be implemented in various programming languages (Java, .NET, C++, Python, etc.) using the Hazelcast client libraries provided. Hazelcast Member: A Hazelcast member refers to an instance of Hazelcast that is part of the Hazelcast cluster and participates in the distributed data grid. Members store data, participate in the cluster's data distribution and replication, and provide distributed processing capabilities. Members join together to form a Hazelcast cluster, where they communicate and collaborate to ensure data consistency and fault tolerance. Members can be deployed on different machines or containers, forming a distributed system. Each member holds a subset of the data in the cluster, and data is distributed across members using Hazelcast's partitioning strategy. Members can also execute distributed tasks and provide distributed caching and event-driven capabilities. Deciding which microservice will be a Member vs a simple Client again depends on multiple factors such as data volume, processing requirements, network communication, resource utilization, and the possible restrictions of your system. For our example, here's the diagram with the selected architecture: Demo Architecture Diagram On the Producer side, which is a Hazelcast Member, we have enabled Queueing with Persistent Datastore. The datastore of choice is a PostgreSQL database. In the following sections, we dive into the implementation details of the Producer and Consumer. You can find the complete source code over on the GitHub repository. Note that it includes a docker-compose file for starting the PostgreSQL database and the Hazelcast Management Center, which we will also describe later on. The Producer The Producer's Hazelcast configuration is the following: Java @Configuration public class HazelcastConfig { @Value("${demo.queue.name}") private String queueName; @Value("${demo.dlq.name}") private String dlqName; @Bean public HazelcastInstance hazelcastInstance(QueueStore<Event> queueStore) { Config config = new Config(); QueueConfig queueConfig = config.getQueueConfig("main"); queueConfig.setName(queueName) .setBackupCount(1) .setMaxSize(0) .setStatisticsEnabled(true); queueConfig.setQueueStoreConfig(new QueueStoreConfig() .setEnabled(true) .setStoreImplementation(queueStore) .setProperty("binary", "false")); QueueConfig dlqConfig = config.getQueueConfig("main-dlq"); dlqConfig.setName(dlqName) .setBackupCount(1) .setMaxSize(0) .setStatisticsEnabled(true); config.addQueueConfig(queueConfig); config.addQueueConfig(dlqConfig); return Hazelcast.newHazelcastInstance(config); } } We define 2 queues: The "main" queue, which will hold the messages (Events) to be processed and is backed by the PostgreSQL QueueStore A "main-dlq," which will be used as a DLQ, and consumers will push items to it if they encounter errors To keep it shorter, the DLQ is not backed by a persistent store. For a full explanation of configuration options and recommendations, refer to Hazelcast's Configuring Queue documentation. Our PostgreSQL QueueStore implements the QueueStore<T> interface and we inject a Spring Data JPA Repository for this purpose: Java @Component @Slf4j public class PostgresQueueStore implements QueueStore<Event> { private final EventRepository eventRepository; public PostgresQueueStore(@Lazy EventRepository eventRepository) { this.eventRepository = eventRepository; } @Override public void store(Long key, Event value) { log.info("store() called for {}" , value); EventEntity entity = new EventEntity(key, value.getRequestId().toString(), value.getStatus().name(), value.getDateSent(), LocalDateTime.now()); eventRepository.save(entity); } @Override public void delete(Long key) { log.info("delete() was called for {}", key); eventRepository.deleteById(key); } // rest methods ommited for brevity... } For testing purposes, the Producer includes a REST Controller with a single endpoint. Once invoked, it creates an Event object with a random UUID and offers it to the main queue: Java @RestController public class ProducerController { @Value("${demo.queue.name}") private String queueName; private final HazelcastInstance hazelcastInstance; public ProducerController(HazelcastInstance hazelcastInstance) { this.hazelcastInstance = hazelcastInstance; } private BlockingQueue<Event> retrieveQueue() { return hazelcastInstance.getQueue(queueName); } @PostMapping("/") public Boolean send() { return retrieveQueue().offer(Event.builder() .requestId(UUID.randomUUID()) .status(Status.APPROVED) .dateSent(LocalDateTime.now()) .build()); } } You can create a script or use a load-generator tool to perform more extensive testing. You may also start multiple instances of the Producer microservice and monitor the cluster's behavior via the LOGs and the Management Console. The Consumer Let's start again with the Consumer's Hazelcast configuration. This time we are dealing with a Hazelcast client: Java @Configuration public class HazelcastConfig { @Value("${demo.queue.name}") private String queueName; @Value("${demo.dlq.name}") private String dlqName; @Bean public HazelcastInstance hazelcastInstance() { ClientConfig config = new ClientConfig(); // add more settings per use-case return HazelcastClient.newHazelcastClient(config); } @Bean public IQueue<Event> eventQueue(HazelcastInstance hazelcastInstance) { return hazelcastInstance.getQueue(queueName); } @Bean public IQueue<Event> dlq(HazelcastInstance hazelcastInstance) { return hazelcastInstance.getQueue(dlqName); } } We are calling the HazelcastClient.newHazelcastClient() method this time in contrast to Hazelcast.newHazelcastInstance() on the Producer side. We are also creating 2 beans corresponding to the queues we will need access to; i.e., the main queue and the DLQ. Now, let's have a look at the message consumption part. Hazelcast library does not offer a pure event-based listener approach (see related GitHub issue). The only way of consuming a new queue entry is through the queue.take() blocking method (usually contained in a do...while loop). To make this more "resource-friendly" and to avoid blocking the main thread of our Spring-based Java application, we will perform this functionality in background virtual threads: Java @Component @Slf4j public class EventConsumer { @Value("${demo.virtual.threads}") private int numberOfThreads = 1; @Value("${demo.queue.name}") private String queueName; private final IQueue<Event> queue; private final DemoService demoService; private final FeatureManager manager; public static final Feature CONSUMER_ENABLED = new NamedFeature("CONSUMER_ENABLED"); public EventConsumer(@Qualifier("eventQueue") IQueue<Event> queue, DemoService demoService, FeatureManager manager) { this.queue = queue; this.demoService = demoService; this.manager = manager; } @PostConstruct public void init() { startConsuming(); } public void startConsuming() { for (var i = 0; i < numberOfThreads; i++) { Thread.ofVirtual() .name(queueName + "_" + "consumer-" + i) .start(this::consumeMessages); } } private void consumeMessages() { while (manager.isActive(CONSUMER_ENABLED)) { try { Event event = queue.take(); // Will block until an event is available or interrupted // Process the event log.info("EventConsumer::: Processing {} ", event); demoService.doWork(event); } catch (InterruptedException e) { // Handle InterruptedException as per your application's requirements log.error("Encountered thread interruption: ", e); Thread.currentThread().interrupt(); } } } } Depending on the nature of the processing we need and the desired throughput, we can have a multi-threaded EventConsumer by specifying the desired number of virtual threads in the configuration file. The important thing to note here is that because Hazecast Queue is like a distributed version of java.util.concurrent.BlockingQueue, we are assured that each item will be taken from the queue by a single consumer even if multiple instances are running and on different JVMs. The actual "work" is delegated to a DemoService, which has single @Retryable method: Java @Retryable(retryFor = Exception.class, maxAttempts = 2) @SneakyThrows // @Async: enable this if you want to delegate processing to a ThreadPoolTaskExecutor public void doWork(Event event) { // do the actual processing... EventProcessedEntity processed = new EventProcessedEntity(); processed.setRequestId(event.getRequestId().toString()); processed.setStatus(event.getStatus().name()); processed.setDateSent(event.getDateSent()); processed.setDateProcessed(LocalDateTime.now()); processed.setProcessedBy(InetAddress.getLocalHost().getHostAddress() + ":" + webServerAppCtxt.getWebServer().getPort()); eventProcessedRepository.save(processed); } @Recover void recover(Exception e, Event event) { log.error("Error processing {} ", event, e); var result = dlq.offer(event); log.info("Adding {} in DQL result {}", event, result); } If the method fails for a configurable number of retries, the @Recover annotated method is fired and the event is delegated to the DLQ. The demo does not include implementation for the manipulation of DLQ messages. One more thing to add here is that we could possibly annotate this method with @Async and configure a ThreadPoolTaskExecutor(or a SimpleAsyncTaskExecutor for which Spring will soon add first-class configuration options for virtual threads - check here for more info). This way the consumption of messages would be less blocking if the processing task (i.e., the actual work) is more "heavy." At this point and regarding error handling, we need to mention that Hazecast also supports Transactions. For more information about this capability, you may check the official documentation here. You could easily modify the Consumer's code in order to try a transactional approach instead of the existing error handling. Testing and Monitoring As mentioned above, once executed our docker-compose file will start the PostgreSQL database (with the necessary tables in place) and the Hazelcast Management console which you can access at http://localhost:8080. Hazelcast Management Console You can start multiple instances of Producers and Consumers and monitor them via the console. At the same time, you may start producing some traffic and confirm the load-balancing and parallelization achieved with our competing consumers. Moreover, the Consumer application includes the Chaos Monkey dependency as well, which you can use to introduce various exceptions, delays, restarts, etc., and observe the system's behavior. Wrap Up To wrap it up, competing consumers with Hazelcast and Spring Boot is like a friendly race among hungry microservices for delicious messages. It's a recipe for efficient and scalable message processing, so let the testing begin, and may the fastest consumer prevail!

Apache Camel is everything but a new arrival in the area of the Java enterprise stacks. Created by James Strachan in 2007, it aimed at being the implementation of the famous "EIP book" (Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf, published by Addison Wesley in October 2003). After having become one of the most popular Java integration frameworks in early 2010, Apache Camel was on the point of getting lost in the folds of history in favor of a new architecture model known as Enterprise Service Bus (ESB) and perceived as a panacea of the Service Oriented Architecture (SOA). But after the SOA fiasco, Apache Camel (which, meanwhile, has been adopted and distributed by several editors including but not limited to Progress Software and Red Hat under commercial names like Mediation Router or Fuse) is making a powerful comeback and is still here, even stronger for the next decade of integration. This comeback is also made easier by Quarkus, the new supersonic and subatomic Java platform. This article aims at proposing a very convenient microservices implementation approach using Apache Camel as a Java development tool, Quarkus as a runtime, and different Kubernetes (K8s) clusters - from local ones like Minikube to PaaS like EKS (Elastic Kubernetes Service), OpenShift, or Heroku - as the infrastructure. The Project The project used here in order to illustrate the point is a simplified money transfer application consisting of four microservices, as follows: aws-camelk-file: This microservice is polling a local folder and, as soon as an XML file is coming in, it stores it in a newly created AWS S3 bucket, which name starts with mys3 followed by a random suffix. aws-camelk-s3: This microservice is listening on the first found AWS S3 bucket, which name starts with mys3. As soon as an XML file comes in, it splits, tokenized, and streams it, before sending each message to an AWS SQS (Simple Queue Service) queue, which name is myQueue. aws-camelk-sqs: This microservice subscribes for messages to the AWS SQS queue named myQueue and, for each incoming message, unmarshals it from XML to Java objects, then marshals it to JSON format, before sending it to the REST service below. aws-camelk-jaxrs: This microservice exposes a REST API having endpoints for CRUD-ing money transfer orders. It consumes/produces JSON input/output data. It uses a service that exposes an interface defined by the aws-camelk-api module. Several implementations of this interface might be present but, for simplicity's sake, in the current case, we're using the one defined by theaws-camelk-provider module named DefaultMoneyTransferProvider, which only CRUDs the money transfer order requests in an in-memory hash map. The project's source code may be found here. It's a multi-module Maven project and the modules are explained below. The most important Maven dependencies are shown below: XML <dependencyManagement> <dependencies> <dependency> <groupId>io.quarkus.platform</groupId> <artifactId>quarkus-bom</artifactId> <version>${quarkus.platform.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>io.quarkus.platform</groupId> <artifactId>quarkus-camel-bom</artifactId> <version>${quarkus.platform.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>com.amazonaws</groupId> <artifactId>aws-java-sdk-bom</artifactId> <version>1.12.454</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> The Module aws-camelk-model This module defines the application's domain which consists of business objects like MoneyTransfer, Bank, BankAddress, etc. One of the particularities of the integration applications is the fact that the business domain is legacy and, generally, designed decades ago by business analysts and experts ignoring everything about the tool-set that you, as a software developer, are using currently. This legacy takes various forms, like Excel sheets and CSV or XML files. Hence we consider here the classical scenario according to which our domain model is defined as an XML grammar, defined by a couple of XSD files. These XSD files are in the src/main/resources/xsd directory and are processed by the jaxb2-maven-plugin in order to generate the associated Java objects. The listing below shows the plugin's configuration: XML <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>jaxb2-maven-plugin</artifactId> <dependencies> <dependency> <groupId>org.jvnet.jaxb2_commons</groupId> <artifactId>jaxb2-value-constructor</artifactId> <version>3.0</version> </dependency> </dependencies> <executions> <execution> <goals> <goal>xjc</goal> </goals> </execution> </executions> <configuration> <packageName>fr.simplex_software.quarkus.camel.integrations.jaxb</packageName> <sources> <source>${basedir}/src/main/resources/xsd</source> </sources> <arguments> <argument>-Xvalue-constructor</argument> </arguments> <extension>true</extension> </configuration> </plugin> Here, we're running the xjc schema compiler tool to generate Java classes in the target package fr.simplex_software.quarkus.camel.integrations.jaxb based on the XSD schema present in the project's src/main/resources/xsd directory. By default, these automatically generated Java objects having JAXB (Java Architecture for XML Binding) annotations don't have constructors, which makes them a bit hard to use, especially for classes with lots of properties that must be instantiated via setters. Accordingly, in the listing above, we configure the jaxb2-maven-plugin with a dependency to the jaxb3-value-constructor artifact. In doing that, we ask the xjc compiler to generate full argument constructors for every subsequent JAXB processed class. The final result of this module is a JAR file containing our domain model in the form of a Java class hierarchy that will be used as a dependency by all the other application's modules. This method is much more practical than the one consisting of manual implementation (again, in Java) of the domain object that is already defined by the XML grammar. The Module aws-camelk-api This module is very simple as it only consists of an interface. This interface, named MoneyTransferFacade, is the one exposed by the money transfer service. This service has to implement the exposed interface. In practice, such a service might have many different implementations, depending on the nature of the money transfer, the bank, the customer type, and many other possible criteria. In our example, we only consider a simple implementation of this interface, as shown in the next section. The Module aws-camelk-provider This module defines the service provider for the MoneyTransferFacade interface. The SPI (Software Provider Interface) pattern used here is a very powerful one, allowing to decouple the service interface from its implementation. Our implementation of the MoneyTransferFacade interface is the class DefaultMoneyTransferProvider and it's also very simple as it only CRUDing the money transfer orders in an in-memory hash map. The Module aws-camelk-jaxrs As opposed to the previous modules which are only common class libraries, this module and the next ones are Quarkus runnable services. This means that they use the quarkus-maven-plugin in order to create an executable JAR. This module, as its name implies, exposes a JAX-RS (Java API for RESTfull Web Services) API to handle money transfer orders. Quarkus comes with RESTeasy, a full implementation by Red Hat of the JAX-RS specifications, and this is what we're using here. There is nothing special to mention as far as the class MoneyTransferResource is concerned, which implements the REST API. It offers endpoints to create, read, update, and delete money transfer orders and, additionally, two endpoints that aim at checking the application's aliveness and readiness. The Module aws-camelk-file This module is the first one in the Camel pipeline, consisting of conveying XML files containing money transfer orders from their initial landing directory to the REST API, which processes them on behalf of the service provider. It uses Camel Java DSL (Domain Specific Language) for doing that, as shown in the listing below: Java fromF("file://%s?include=.*.xml&delete=true&idempotent=true&bridgeErrorHandler=true", inBox) .doTry() .to("validator:xsd/money-transfers.xsd") .setHeader(AWS2S3Constants.KEY, header(FileConstants.FILE_NAME)) .to(aws2S3(s3Name + RANDOM).autoCreateBucket(true).useDefaultCredentialsProvider(true)) .doCatch(ValidationException.class) .log(LoggingLevel.ERROR, failureMsg + " ${exception.message}") .doFinally() .end(); This code polls an input directory, defined as an external property, for the presence of any XML file (files having the .xml extension). Once such a file lands in the given directory, it is validated against the schema defined in the src/main/resources/xsd/money-transfers.xsd file. Should it be valid, it is stored in an AWS S3 bucket whose name is computed as being equal to an externally defined constant followed by a random suffix. Everything is encapsulated in a try...catch structure to consistently process the exception cases. Here, in order to define external properties, we use the Eclipse MP Configuration specs (among others) implemented by Quarkus, as shown in the listing below: Java private static final String RANDOM = new Random().ints('a', 'z') .limit(5) .collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append) .toString(); @ConfigProperty(name="inBox") String inBox; @ConfigProperty(name="s3Name") String s3Name; The RANDOM suffix is generated on behalf of the java.util.Random class and the properties inBox and s3Name are injected from the src/resource/application.properties file. The reason for using an S3 bucket name composed from a constant and a random suffix is that AWS S3 buckets need to have cross-region unique names and, accordingly, we need such a random suffix in order to guarantee unicity. The Module aws-camelk-s3 This module implements a Camel route which is triggered by the AWS infrastructure whenever a file lands in the dedicated S3 bucket. Here is the code: Java from(aws2S3(s3BucketName).useDefaultCredentialsProvider(true)) .split().tokenizeXML("moneyTransfer").streaming() .to(aws2Sqs(queueName).autoCreateQueue(true).useDefaultCredentialsProvider(true)); Once triggered, the Camel route splits the input XML file after having tokenized it, order by order. The idea is that an input file may contain several money transfer orders and these orders are to be processed separately. Hence, each single money transfer order issued from this tokenizing and splitting process is sent to the AWS SQS queue, which name is given by the value of the queueName property, injected from the application.properties file. The Module aws-camelk-sqs This is the last module of our Camel pipeline. Java from(aws2Sqs(queueName).useDefaultCredentialsProvider(true)) .unmarshal(jaxbDataFormat) .marshal().json(JsonLibrary.Jsonb) .setHeader(Exchange.HTTP_METHOD, constant("POST")) .to(http(uri)); This Camel route subscribes to the AWS SQS queue whose name is given by the queueName property and it unmarshals each XML message it receives to Java objects. Given that each XML message contains a money transfer order, it is unmarshaled in the correspondent MoneyTransfer Java class instance. Then, once unmarshalled, each MoneyTransfer Java class instance is marshaled again into a JSON payload. This is required because our REST interface consumes JSON payloads, and, as opposed to the standard JAX-RS client, which is able to automatically perform conversions from Java objects to JSON, the http() Camel component used here isn't. Hence, we need to do it manually. By setting the exchange's header to the POST constant, we set the type of HTTP request that will be sent to the REST API. Last but not least, the endpoint URI is, as usual, injected as an externally defined property, from the application.properties file. Unit Testing Before deploying and running our microservices, we need to unit test them. The project includes a couple of unit tests for almost all its modules - from aws-camelk-model, where the domain model is tested, as well as its various conversion from/to XML/Java, to aws-camelk-jaxrs, which is our terminus microservice. In order to run the unit test, it's simple. Just execute: Shell $ cd aws-camelk $ ./delete-all-buckets.sh #Delete all the buckets named "mys3*" if any $ ./purge-sqs-queue.sh #Purge the SQS queue named myQueue if it exists and isn't empty $ mvn clean package #Clean-up, run unit tests and create JARs A full unit test report will be displayed by the maven-surefile-plugin. In order that the unit tests run as expected, an AWS account is required and the AWS CLI should be installed and configured on the local box. This means that, among others, the file ~/.aws/credentials contains your aws_access_key_id and aws_secret_access_key properties with their associated values. The reason is that the unit tests use the AWS SDK (Software Development Kit) to handle S3 buckets and SQS queues, which makes them not quite unit tests but, rather, a combination of unit and integration tests. Deploying and Running Now, to deploy and run our microservices, there are many different scenarios that we have to consider - from the simple local standalone execution to PaaS deployments like OpenShift or EKS passing through local K8s clusters like Minikube. Accordingly, in order to avoid some confusion here, we have preferred to dedicate a separate post to each such deployment scenario. So stay close to your browser to see where the story takes us next.

With various access control models and implementation methods available, constructing an authorization system for backend service APIs can still be challenging. However, the ultimate goal is to ensure that the correct individual has appropriate access to the relevant resource. In this article, we will discuss how to enable the Role-based access control (RBAC) authorization model for your API with open-source API Gateway Apache APISIX and Open Policy Agent (OPA). What Is RBAC? Role-based access control (RBAC)and attribute-based access control (ABAC) are two commonly used access control models used to manage permissions and control access to resources in computer systems. RBAC assigns permissions to users based on their role within an organization. In RBAC, roles are defined based on the functions or responsibilities of users, and permissions are assigned to those roles. Users are then assigned to one or more roles, and they inherit the permissions associated with those roles. In the API context, for example, a developer role might have permission to create and update API resources, while an end-user role might only have permission to read or execute API resources. Basically, RBAC assigns permissions based on user roles, while ABAC assigns permissions based on attributes associated with users and resources. In RBAC, a policy is defined by the combination of a user’s assigned role, the actions they are authorized to perform, and the resources on which they can perform those actions. What Is OPA? OPA (Open Policy Agent) is a policy engine and a set of tools that provide a unified approach to policy enforcement across an entire distributed system. It allows you to define, manage, and enforce policies centrally from a single point. By defining policies as code, OPA enables easy review, editing, and roll-back of policies, facilitating efficient policy management. OPA provides a declarative language called Rego, which allows you to create and enforce policies throughout your stack. When you request a policy decision from OPA, it uses the rules and data that you have provided in a .rego file to evaluate the query and produce a response. The query result is then sent back to you as the policy decision. OPA stores all the policies and the data it needs in its in-memory cache. As a result, OPA returns results quickly. Here is an example of a simple OPA Rego file: package example default allow = false allow { input.method == "GET" input.path =="/api/resource" input.user.role == "admin" } In this example, we have a package called “example” that defines a rule called “allow”. The “allow” rule specifies that the request is allowed if the input method is “GET”, the requested path is /api/resource, and the user's role is "admin". If these conditions are met, then the "allow" rule will evaluate as "true", allowing the request to proceed. Why Use OPA and API Gateway for RBAC? API Gateway provides a centralized location to configure and manage API, and API consumers. It can be used as a centralized authentication gateway by avoiding having each individual service implement authentication logic inside the service itself. On the other hand, OPA adds an authorization layer and decouples the policy from the code by creating a distinct benefit for authorization. With this combination, you can add permissions for an API resource to a role. Users might be associated with one or more user roles. Each user role defines a set of permissions (GET, PUT, DELETE) on RBAC resources (defined by URI paths). In the next section, let’s learn how to achieve RBAC using these two. How to Implement RBAC With OPA and Apache APISIX In Apache APISIX, you can configure routes and plugins to define the behavior of your API. You can use the APISIX opa plugin to enforce RBAC policies by forwarding requests to OPA for decision-making. Then OPA makes an authorization decision based on users’ roles and permissions in real-time. Assume that we have Conference API where you can retrieve/edit event sessions, topics, and speaker information. A speaker can only read their own sessions and topics while the admin can add/edit more sessions and topics. Or attendees can leave their feedback about the speaker’s session via a POST request to /speaker/speakerId/session/feedback and the speaker can only see by requesting the GET method of the same URI. The below diagram illustrates the whole scenario: API consumer requests a route on the API Gateway with its credential such as a JWT token in the authorization header. API Gateway sends consumer data with a JWT header to the OPA engine. OPA evaluates if the consumer has a right to access the resource by using policies (roles and permissions) we specify in the .rego file. If the OPA decision is allowed, then the request will be forwarded to the upstream Conference service. Next, we install, configure APISIX, and define policies in OPA. Prerequisites Docker is used to installing the containerized etcd and APISIX. curl is used to send requests to APISIX Admin API. You can also use tools such as Postman to interact with the API. Step 1: Install Apache APISIX APISIX can be easily installed and started with the following quickstart script: curl -sL https://run.api7.ai/apisix/quickstart | sh Step 2: Configure the Backend Service (Upstream) To route requests to the backend service for the Conference API, you’ll need to configure it by adding an upstream server in Apache APISIX via the Admin API. curl http://127.0.0.1:9180/apisix/admin/upstreams/1 -X PUT -d ' { "name":"Conferences API upstream", "desc":"Register Conferences API as the upstream", "type":"roundrobin", "scheme":"https", "nodes":{ "conferenceapi.azurewebsites.net:443":1 } }' Step 3: Create an API Consumer Next, we create a consumer (a new speaker) with the username jack in Apache APISIX. It sets up the jwt-auth plugin for the consumer with the specified key and secret. This will allow the consumer to authenticate using a JSON Web Token (JWT). curl http://127.0.0.1:9180/apisix/admin/consumers -X PUT -d ' { "username": "jack", "plugins": { "jwt-auth": { "key": "user-key", "secret": "my-secret-key" } } }' Step 4: Create a Public Endpoint to Generate a JWT Token You also need to set up a new Route that generates and signs the token using the public-api plugin. In this scenario, API Gateway acts as an identity provider server to create and verify the token with our consumer jack’s key. The identity provider can be also any other third-party services such as Google, Okta, Keycloak, and Ory Hydra. curl http://127.0.0.1:9180/apisix/admin/routes/jas -X PUT -d ' { "uri": "/apisix/plugin/jwt/sign", "plugins": { "public-api": {} } }' Step 5: Claim a New JWT Token for the API Consumer Now we can get a new token for our speaker Jack from the API Gateway using the public endpoint we created. The following curl command generates a new token with Jack’s credentials and assigns role, and permission in the payload. curl -G --data-urlencode 'payload={"role":"speaker","permission":"read"}' http://127.0.0.1:9080/apisix/plugin/jwt/sign?key=user-key -i After you run the above command, you will receive a token as a response. Save this token somewhere — later we are going to use this token to access our new API Gateway endpoint. Step 6: Create a New Plugin Config This step involves configuring APISIX’s 3 plugins: proxy-rewrite, jwt-auth, and opa plugins. curl "http://127.0.0.1:9180/apisix/admin/plugin_configs/1" -X PUT -d ' { "plugins":{ "jwt-auth":{ }, "proxy-rewrite":{ "host":"conferenceapi.azurewebsites.net" } } }' The proxy-rewrite plugin is configured to proxy requests to the conferenceapi.azurewebsites.net host. OPA authentication plugin is configured to use the OPA policy engine running at http://localhost:8181/v1/data/rbacExample. Also, APISIX sends all consumer-related information to OPA. We will add this policy .rego file in the Opa configuration section. Step 7: Create a Route for Conference Sessions The final step is to create a new route for Conferences API speaker sessions: curl "http://127.0.0.1:9180/apisix/admin/routes/1" -X PUT -d ' { "name":"Conferences API speaker sessions route", "desc":"Create a new route in APISIX for the Conferences API speaker sessions", "methods": ["GET", "POST"], "uris": ["/speaker/*/topics","/speaker/*/sessions"], "upstream_id":"1", "plugin_config_id":1 }' The payload contains information about the route, such as its name, description, methods, URIs, upstream ID, and plugin configuration ID. In this case, the route is configured to handle GET and POST requests for two different URIs, /speaker/topics and /speaker/sessions. The "upstream_id" field specifies the ID of the upstream service that will handle incoming requests for this route, while the "plugin_config_id" field specifies the ID of the plugin configuration to be used for this route. Step 8: Test the Setup Without OPA So far, we have set up all the necessary configurations for APISIX to direct incoming requests to Conference API endpoints, only allowing authorized API consumers. Now, each time an API consumer wants to access an endpoint, they must provide a JWT token to retrieve data from the Conference backend service. You can verify this by hitting the endpoint and the domain address we are requesting now is our custom API Gateway but not an actual Conference service: curl -i http://127.0.0.1:9080/speaker/1/topics -H 'Authorization: {API_CONSUMER_TOKEN}' Step 9: Run OPA Service The other two steps are we run the OPA service using Docker and upload our policy definition using its API which can be used to evaluate authorization policies for incoming requests. docker run -d --network=apisix-quickstart-net --name opa -p 8181:8181 openpolicyagent/opa:latest run -s This Docker command runs a container of the OPA image with the latest version. It creates a new container on the existing APISIX network apisix-quickstart-netwith the name opaand exposes port 8181. So, APISIX can send policy check requests to OPA directly using the address [http://opa:8181](http://opa:8181) Note that OPA and APISIX should run in the same docker network. Step 10: Define and Register the Policy The second step on the OPA side is you need to define the policies that will be used to control access to API resources. These policies should define the attributes required for access (which users have which roles) and the permission (which roles have which permissions) that are allowed or denied based on those attributes. For example, in the below configuration, we are saying to OPA, check the user_roles table to find the role for jack. This information is sent by APISIX inside input.consumer.username. Also, we are verifying the consumer’s permission by reading the JWT payload and extracting token.payload.permission from there. The comments describe the steps clearly. curl -X PUT '127.0.0.1:8181/v1/policies/rbacExample' \ -H 'Content-Type: text/plain' \ -d 'package rbacExample # Assigning user rolesuser_roles := { "jack": ["speaker"], "bobur":["admin"] } # Role permission assignments role_permissions := { "speaker": [{"permission": "read"}], "admin": [{"permission": "read"}, {"permission": "write"}] } # Helper JWT Functions bearer_token := t { t := input.request.headers.authorization } # Decode the authorization token to get a role and permission token = {"payload": payload} { [_, payload, _] := io.jwt.decode(bearer_token) } # Logic that implements RBAC default allow = falseallow { # Lookup the list of roles for the user roles := user_roles[input.consumer.username] # For each role in that list r := roles[_] # Lookup the permissions list for role r permissions := role_permissions[r] # For each permission p := permissions[_] # Check if the permission granted to r matches the users request p == {"permission": token.payload.permission} }' Step 11: Update the Existing Plugin Config With the OPA Plugin Once we defined policies on the OPA service, we need to update the existing plugin config for the route to use the OPA plugin. We specify in the policy attribute of the OPA plugin. curl "http://127.0.0.1:9180/apisix/admin/plugin_configs/1" -X PATCH -d ' { "plugins":{ "opa":{ "host":"http://opa:8181", "policy":"rbacExample", "with_consumer":true } } }' Step 12: Test the Setup With OPA Now we can test all setups we did with OPA policies. If you try to run the same curl command to access the API Gateway endpoint, it first checks the JWT token as the authentication process and sends consumer and JWT token data to OPA to verify the role and permission as the authorization process. Any request without a JWT token in place or allowed roles will be denied. curl -i http://127.0.0.1:9080/speaker/1/topics -H 'Authorization: {API_CONSUMER_TOKEN}' Conclusion In this article, we learned how to implement RBAC with OPA and Apache APISIX. We defined a simple custom policy logic to allow/disallow API resource access based on our API consumer’s role and permissions. Also, this tutorial demonstrated how we can extract API consumer-related information in the policy file from the JWT token payload or consumer object sent by APISIX.

In many places, you can read that Podman is a drop-in replacement for Docker. But is it as easy as it sounds? In this blog, you will start with a production-ready Dockerfile and execute the Podman commands just like you would do when using Docker. Let’s investigate whether this works without any problems! Introduction Podman is a container engine, just as Docker is. Podman, however, is a daemonless container engine, and it runs containers by default as rootless containers. This is more secure than running containers as root. The Docker daemon can also run as a non-root user nowadays. Podman advertises on its website that Podman is a drop-in replacement for Docker. Just add alias docker=podman , and you will be fine. Let’s investigate whether it is that simple. In the remainder of this blog, you will try to build a production-ready Dockerfile for running a Spring Boot application. You will run it as a single container, and you will try to run two containers and have some inter-container communication. In the end, you will verify how volumes can be mounted. One of the prerequisites for this blog is using a Linux operating system. Podman is not available for Windows. The sources used in this blog can be found at GitHub. The Dockerfile you will be using runs a Spring Boot application. It is a basic Spring Boot application containing one controller which returns a hello message. Build the jar: Shell $ mvn clean verify Run the jar: Shell $ java -jar target/mypodmanplanet-0.0.1-SNAPSHOT.jar Check the endpoint: Shell $ curl http://localhost:8080/hello Hello Podman! The Dockerfile is based on a previous blog about Docker best practices. The file 1-Dockerfile-starter can be found in the Dockerfiles directory. Dockerfile FROM eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f AS builder WORKDIR application ARG JAR_FILE COPY target/${JAR_FILE} app.jar RUN java -Djarmode=layertools -jar app.jar extract FROM eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f WORKDIR /opt/app RUN addgroup --system javauser && adduser -S -s /usr/sbin/nologin -G javauser javauser COPY --from=builder application/dependencies/ ./ COPY --from=builder application/spring-boot-loader/ ./ COPY --from=builder application/snapshot-dependencies/ ./ COPY --from=builder application/application/ ./ RUN chown -R javauser:javauser . USER javauser ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"] Prerequisites Prerequisites for this blog are: Basic Linux knowledge, Ubuntu 22.04 is used during this post; Basic Java and Spring Boot knowledge; Basic Docker knowledge; Installation Installing Podman is quite easy. Just run the following command. Shell $ sudo apt-get install podman Verify the correct installation. Shell $ podman --version podman version 3.4.4 \You can also install podman-docker, which will create an alias when you use docker in your commands. It is advised to wait for the conclusion of this post before you install this one. Build Dockerfile The first thing to do is to build the container image. Execute from the root of the repository the following command. Shell $ podman build . --tag mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT -f Dockerfiles/1-Dockerfile-starter --build-arg JAR_FILE=mypodmanplanet-0.0.1-SNAPSHOT.jar [1/2] STEP 1/5: FROM eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f AS builder [2/2] STEP 1/10: FROM eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f Error: error creating build container: short-name "eclipse-temurin@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf" This returns an error while retrieving the base image. The error message refers to /etc/containers/registries.conf. The following is stated in this file. Plain Text # For more information on this configuration file, see containers-registries.conf(5). # # NOTE: RISK OF USING UNQUALIFIED IMAGE NAMES # We recommend always using fully qualified image names including the registry # server (full dns name), namespace, image name, and tag # (e.g., registry.redhat.io/ubi8/ubi:latest). Pulling by digest (i.e., # quay.io/repository/name@digest) further eliminates the ambiguity of tags. # When using short names, there is always an inherent risk that the image being # pulled could be spoofed. For example, a user wants to pull an image named # `foobar` from a registry and expects it to come from myregistry.com. If # myregistry.com is not first in the search list, an attacker could place a # different `foobar` image at a registry earlier in the search list. The user # would accidentally pull and run the attacker's image and code rather than the # intended content. We recommend only adding registries which are completely # trusted (i.e., registries which don't allow unknown or anonymous users to # create accounts with arbitrary names). This will prevent an image from being # spoofed, squatted or otherwise made insecure. If it is necessary to use one # of these registries, it should be added at the end of the list. To conclude, it is suggested to use a fully qualified image name. This means that you need to change the lines containing: Dockerfile eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f Into: Dockerfile docker.io/eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f You just add docker.io/ to the image name. A minor change, but already one difference compared to Docker. The image name is fixed in file 2-Dockerfile-fix-shortname, so let’s try building the image again. Shell $ podman build . --tag mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT -f Dockerfiles/2-Dockerfile-fix-shortname --build-arg JAR_FILE=mypodmanplanet-0.0.1-SNAPSHOT.jar [1/2] STEP 1/5: FROM docker.io/eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f AS builder Trying to pull docker.io/library/eclipse-temurin@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f... Getting image source signatures Copying blob 72ac8a0a29d6 done Copying blob f56be85fc22e done Copying blob f8ed194273be done Copying blob e5daea9ee890 done [2/2] STEP 1/10: FROM docker.io/eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f Error: error creating build container: writing blob: adding layer with blob "sha256:f56be85fc22e46face30e2c3de3f7fe7c15f8fd7c4e5add29d7f64b87abdaa09": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid: lchown /etc/shadow: invalid argument Now there is an error about potentially insufficient UIDs or GIDs available in the user namespace. More information about this error can be found here. It is very well explained in that post, and it is too much to repeat all of this in this post. The summary is that the image which is trying to be pulled, has files owned by UIDs over 65.536. Due to that issue, the image would not fit into rootless Podman’s default UID mapping, which limits the number of UIDs and GIDs available. So, how to solve this? First, check the contents of /etc/subuid and /etc/subgid. In my case, the following is the output. For you, it will probably be different. Shell $ cat /etc/subuid admin:100000:65536 $ cat /etc/subgid admin:100000:65536 The admin user listed in the output has 100.000 as the first UID or GID available, and it has a size of 65.536. The format is user:start:size. This means that the admin user has access to UIDs or GIDs 100.000 up to and including 165.535. My current user is not listed here, and that means that my user can only allocate 1 UID en 1 GID for the container. That 1 UID/GID is already taken for the root user in the container. If a container image needs an extra user, there will be a problem, as you can see above. This can be solved by adding UIDs en GIDs for your user. Let’s add values 200.000 up to and including 265.535 to your user. Shell $ sudo usermod --add-subuids 200000-265535 --add-subgids 200000-265535 <replace with your user> Verify the contents of both files again. The user is added to both files. Shell $ cat /etc/subgid admin:100000:65536 <your user>:200000:65536 $ cat /etc/subuid admin:100000:65536 <your user>:200000:65536 Secondly, you need to run the following command. Shell $ podman system migrate Try to build the image again, and now it works. Shell $ podman build . --tag mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT -f Dockerfiles/2-Dockerfile-fix-shortname --build-arg JAR_FILE=mypodmanplanet-0.0.1-SNAPSHOT.jar [1/2] STEP 1/5: FROM docker.io/eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f AS builder Trying to pull docker.io/library/eclipse-temurin@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f... Getting image source signatures Copying blob f56be85fc22e done Copying blob f8ed194273be done Copying blob 72ac8a0a29d6 done Copying blob e5daea9ee890 done Copying config c74d412c3d done Writing manifest to image destination Storing signatures [1/2] STEP 2/5: WORKDIR application --> d4f0e970dc1 [1/2] STEP 3/5: ARG JAR_FILE --> ca97dcd6f2a [1/2] STEP 4/5: COPY target/${JAR_FILE} app.jar --> 58d88cfa511 [1/2] STEP 5/5: RUN java -Djarmode=layertools -jar app.jar extract --> 348cae813a4 [2/2] STEP 1/10: FROM docker.io/eclipse-temurin:17.0.6_10-jre-alpine@sha256:c26a727c4883eb73d32351be8bacb3e70f390c2c94f078dc493495ed93c60c2f [2/2] STEP 2/10: WORKDIR /opt/app --> 4118cdf90b5 [2/2] STEP 3/10: RUN addgroup --system javauser && adduser -S -s /usr/sbin/nologin -G javauser javauser --> cd11f346381 [2/2] STEP 4/10: COPY --from=builder application/dependencies/ ./ --> 829bffcb6c7 [2/2] STEP 5/10: COPY --from=builder application/spring-boot-loader/ ./ --> 2a93f97d424 [2/2] STEP 6/10: COPY --from=builder application/snapshot-dependencies/ ./ --> 3e292cb0456 [2/2] STEP 7/10: COPY --from=builder application/application/ ./ --> 5dd231c5b51 [2/2] STEP 8/10: RUN chown -R javauser:javauser . --> 4d736e8c3bb [2/2] STEP 9/10: USER javauser --> d7a96ca6f36 [2/2] STEP 10/10: ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"] [2/2] COMMIT mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT --> 567fd123071 Successfully tagged localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT 567fd1230713f151950de7151da82a19d34f80af0384916b13bf49ed72fd2fa1 Verify the list of images with Podman just like you would do with Docker: Shell $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE localhost/mydeveloperplanet/mypodmanplanet 0.0.1-SNAPSHOT 567fd1230713 2 minutes ago 209 MB Is Podman a drop-in replacement for Docker for building a Dockerfile? No, it is not a drop-in replacement because you needed to use the fully qualified image name for the base image in the Dockerfile, and you needed to make changes to the user namespace in order to be able to pull the image. Besides these two changes, building the container image just worked. Start Container Now that you have built the image, it is time to start a container. Shell $ podman run --name mypodmanplanet -d localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT The container has started successfully. Shell $ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 27639dabb573 localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT 18 seconds ago Up 18 seconds ago mypodmanplanet You can also inspect the container logs. Shell $ podman logs mypodmanplanet . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v3.0.5) 2023-04-22T14:38:05.896Z INFO 1 --- [ main] c.m.m.MyPodmanPlanetApplication : Starting MyPodmanPlanetApplication v0.0.1-SNAPSHOT using Java 17.0.6 with PID 1 (/opt/app/BOOT-INF/classes started by javauser in /opt/app) 2023-04-22T14:38:05.898Z INFO 1 --- [ main] c.m.m.MyPodmanPlanetApplication : No active profile set, falling back to 1 default profile: "default" 2023-04-22T14:38:06.803Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2023-04-22T14:38:06.815Z INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2023-04-22T14:38:06.816Z INFO 1 --- [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.7] 2023-04-22T14:38:06.907Z INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2023-04-22T14:38:06.910Z INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 968 ms 2023-04-22T14:38:07.279Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2023-04-22T14:38:07.293Z INFO 1 --- [ main] c.m.m.MyPodmanPlanetApplication : Started MyPodmanPlanetApplication in 1.689 seconds (process running for 1.911) Verify whether the endpoint can be accessed. Shell $ curl http://localhost:8080/hello curl: (7) Failed to connect to localhost port 8080 after 0 ms: Connection refused That’s not the case. With Docker, you can inspect the container to see which IP address is allocated to the container. Shell $ podman inspect mypodmanplanet | grep IPAddress "IPAddress": "", It seems that the container does not have a specific IP address. The endpoint is also not accessible at localhost. The solution is to add a port mapping when creating the container. Stop the container and remove it. Shell $ podman stop mypodmanplanet mypodmanplanet $ podman rm mypodmanplanet 27639dabb5730d3244d205200a409dbc3a1f350196ba238e762438a4b318ef73 Start the container again, but this time with a port mapping of internal port 8080 to an external port 8080. Shell $ podman run -p 8080:8080 --name mypodmanplanet -d localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT Verify again whether the endpoint can be accessed. This time it works. Shell $ curl http://localhost:8080/hello Hello Podman! Stop and remove the container before continuing this blog. Is Podman a drop-in replacement for Docker for running a container image? No, it is not a drop-in replacement. Although it was possible to use exactly the same commands as with Docker, you needed to explicitly add a port mapping. Without the port mapping, it was not possible to access the endpoint. Volume Mounts Volume mounts and access to directories and files outside the container and inside a container often lead to Permission Denied errors. In a previous blog, this behavior is extensively described for the Docker engine. It is interesting to see how this works when using Podman. You will map an application.properties file in the container next to the jar file. The Spring Boot application will pick up this application.properties file. The file configures the server port to port 8082, and the file is located in the directory properties in the root of the repository. Properties files server.port=8082 Run the container with a port mapping from internal port 8082 to external port 8083 and mount the application.properties file into the container directory /opt/app where also the jar file is located. The volume mount has the property ro in order to indicate that it is a read-only file. Shell $ podman run -p 8083:8082 --volume ./properties/application.properties:/opt/app/application.properties:ro --name mypodmanplanet localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT Verify whether the endpoint can be accessed and whether it works. Shell $ curl http://localhost:8083/hello Hello Podman! Open a shell in the container and list the directory contents in order to view the ownership of the file. Shell $ podman exec -it mypodmanplanet sh /opt/app $ ls -la total 24 drwxr-xr-x 1 javauser javauser 4096 Apr 15 10:33 . drwxr-xr-x 1 root root 4096 Apr 9 12:57 .. drwxr-xr-x 1 javauser javauser 4096 Apr 9 12:57 BOOT-INF drwxr-xr-x 1 javauser javauser 4096 Apr 9 12:57 META-INF -rw-r--r-- 1 root root 16 Apr 15 10:24 application.properties drwxr-xr-x 1 javauser javauser 4096 Apr 9 12:57 org With Docker, the file would have been owned by your local system user, but with Podman, the file is owned by root. Let’s check the permissions of the file on the local system. Shell $ ls -la total 12 drwxr-xr-x 2 <myuser> domain users 4096 apr 15 12:24 . drwxr-xr-x 8 <myuser> domain users 4096 apr 15 12:24 .. -rw-r--r-- 1 <myuser> domain users 16 apr 15 12:24 application.properties As you can see, the file on the local system is owned by <myuser>. This means that your host user, who is running the container, is seen as a user root inside of the container. Open a shell in the container and try to change the contents of the file application.properties. You will notice that this is not allowed because you are a user javauser. Shell $ podman exec -it mypodmanplanet sh /opt/app $ vi application.properties /opt/app $ whoami javauser Stop and remove the container. Run the container, but this time with property U instead of ro. The U suffix tells Podman to use the correct host UID and GID based on the UID and GID within the container to change the owner and group of the source volume recursively. Shell $ podman run -p 8083:8082 --volume ./properties/application.properties:/opt/app/application.properties:U --name mypodmanplanet localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT Open a shell in the container, and now the user javauser is the owner of the file. Shell $ podman exec -it mypodmanplanet sh /opt/app $ ls -la total 24 drwxr-xr-x 1 javauser javauser 4096 Apr 15 10:41 . drwxr-xr-x 1 root root 4096 Apr 9 12:57 .. drwxr-xr-x 1 javauser javauser 4096 Apr 9 12:57 BOOT-INF drwxr-xr-x 1 javauser javauser 4096 Apr 9 12:57 META-INF -rw-r--r-- 1 javauser javauser 16 Apr 15 10:24 application.properties drwxr-xr-x 1 javauser javauser 4096 Apr 9 12:57 org On the local system, a different UID and GID than my local user have taken ownership. Shell $ ls -la properties/ total 12 drwxr-xr-x 2 <myuser> domain users 4096 apr 15 12:24 . drwxr-xr-x 8 <myuser> domain users 4096 apr 15 12:24 .. -rw-r--r-- 1 200099 200100 16 apr 15 12:24 application.properties This time, changing the file on the local system is not allowed, but it is allowed inside the container for user javauser. Is Podman a drop-in replacement for Docker for mounting volumes inside a container? No, it is not a drop-in replacement. The file permissions function is a bit different than with the Docker engine. You need to know the differences in order to be able to mount files and directories inside containers. Pod Podman knows the concept of a Pod, just like a Pod in Kubernetes. A Pod allows you to group containers. A Pod also has a shared network namespace, and this means that containers inside a Pod can connect to each other. More information about container networking can be found here. This means that Pods are the first choice for grouping containers. When using Docker, you will use Docker Compose for this. There exists something like Podman Compose, but this deserves a blog in itself. Let’s see how this works. You will set up a Pod running two containers with the Spring Boot application. First, you need to create a Pod. You also need to expose the ports you want to be accessible outside of the Pod. This can be done with the -p argument. And you give the Pod a name, hello-pod in this case. Shell $ podman pod create -p 8080-8081:8080-8081 --name hello-pod When you list the Pod, you notice that it already contains one container. This is the infra container. This infra container holds the namespace in order that containers can connect to each other, and it enables starting and stopping containers in the Pod. The infra container is based on the k8s.gcr.io/pause image. Shell $ podman pod ps POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS dab9029ad0c5 hello-pod Created 3 seconds ago aac3420b3672 1 $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES aac3420b3672 k8s.gcr.io/pause:3.5 4 minutes ago Created 0.0.0.0:8080-8081->8080-8081/tcp dab9029ad0c5-infra Create a container mypodmanplanet-1 and add it to the Pod. By means of the --env argument, you change the port of the Spring Boot application to port 8081. Shell $ podman create --pod hello-pod --name mypodmanplanet-1 --env 'SERVER_PORT=8081' localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT env Start the Pod. Shell $ podman pod start hello-pod Verify whether the endpoint can be reached at port 8081 and verify that the endpoint at port 8080 cannot be reached. Shell $ curl http://localhost:8081/hello Hello Podman! $ curl http://localhost:8080/hello curl: (56) Recv failure: Connection reset by peer Add a second container mypodmanplanet-2 to the Pod, this time running at the default port 8080. Shell $ podman create --pod hello-pod --name mypodmanplanet-2 localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT Verify the Pod status. It says that the status is Degraded. Shell $ podman pod ps POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS dab9029ad0c5 hello-pod Degraded 9 minutes ago aac3420b3672 3 Take a look at the containers. Two containers are running, and a new container is just created. That is the reason the Pod has the status Degraded. Shell $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES aac3420b3672 k8s.gcr.io/pause:3.5 11 minutes ago Up 2 minutes ago 0.0.0.0:8080-8081->8080-8081/tcp dab9029ad0c5-infra 321a62fbb4fc localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT env 3 minutes ago Up 2 minutes ago 0.0.0.0:8080-8081->8080-8081/tcp mypodmanplanet-1 7b95fb521544 localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT About a minute ago Created 0.0.0.0:8080-8081->8080-8081/tcp mypodmanplanet-2 Start the second container and verify the Pod status. The status is now Running. Shell $ podman start mypodmanplanet-2 $ podman pod ps POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS dab9029ad0c5 hello-pod Running 12 minutes ago aac3420b3672 3 Both endpoints can now be reached. Shell $ curl http://localhost:8080/hello Hello Podman! $ curl http://localhost:8081/hello Hello Podman! Verify whether you can access the endpoint of container mypodmanplanet-1 from within mypodmanplanet-2. This also works. Shell $ podman exec -it mypodmanplanet-2 sh /opt/app $ wget http://localhost:8081/hello Connecting to localhost:8081 (127.0.0.1:8081) saving to 'hello' hello 100% |***********************************************************************************************************************************| 13 0:00:00 ETA 'hello' saved Cleanup To conclude, you can do some cleanup. Stop the running Pod. Shell $ podman pod stop hello-pod The Pod has the status Exited now. Shell $ podman pod ps POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS dab9029ad0c5 hello-pod Exited 55 minutes ago aac3420b3672 3 All containers in the Pod are also exited. Shell $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES aac3420b3672 k8s.gcr.io/pause:3.5 56 minutes ago Exited (0) About a minute ago 0.0.0.0:8080-8081->8080-8081/tcp dab9029ad0c5-infra 321a62fbb4fc localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT env 48 minutes ago Exited (143) About a minute ago 0.0.0.0:8080-8081->8080-8081/tcp mypodmanplanet-1 7b95fb521544 localhost/mydeveloperplanet/mypodmanplanet:0.0.1-SNAPSHOT 46 minutes ago Exited (143) About a minute ago 0.0.0.0:8080-8081->8080-8081/tcp mypodmanplanet-2 Remove the Pod. Shell $ podman pod rm hello-pod The Pod and the containers are removed. Shell $ podman pod ps POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS $ podman ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Conclusion The bold statement that Podman is a drop-in replacement for Docker is not true. Podman differs from Docker on certain topics like building container images, starting containers, networking, volume mounts, inter-container communication, etc. However, Podman does support many Docker commands. The statement should be Podman is an alternative to Docker. This is certainly true. It is important for you to know and understand the differences before switching to Podman. After this, it is definitely a good alternative.

This post explains how to launch an Amazon EMR cluster and deploy a Kedro project to run a Spark job. Amazon EMR (previously called Amazon Elastic MapReduce) is a managed cluster platform for applications built using open-source big data frameworks, such as Apache Spark, that process and analyze vast amounts of data with AWS. 1. Set up the Amazon EMR Cluster One way to install Python libraries onto Amazon EMR is to package a virtual environment and deploy it. To do this, the cluster needs to have the same Amazon Linux 2 environment as used by Amazon EMR. We used this example Dockerfile to package our dependencies on an Amazon Linux 2 base. Our example Dockerfile is as below: Shell FROM --platform=linux/amd64 amazonlinux:2 AS base RUN yum install -y python3 ENV VIRTUAL_ENV=/opt/venv RUN python3 -m venv $VIRTUAL_ENV ENV PATH="$VIRTUAL_ENV/bin:$PATH" COPY requirements.txt /tmp/requirements.txt RUN python3 -m pip install --upgrade pip && \ python3 -m pip install venv-pack==0.2.0 && \ python3 -m pip install -r /tmp/requirements.txt RUN mkdir /output && venv-pack -o /output/pyspark_deps.tar.gz FROM scratch AS export COPY --from=base /output/pyspark_deps.tar.gz / Note: A DOCKER_BUILDKIT backend is necessary to run this Dockerfile (make sure you have it installed). Run the Dockerfile using the following command:DOCKER_BUILDKIT=1 docker build --output . <output-path> This will generate a pyspark_deps.tar.gz file at the <output-path> specified in the command above. Use this command if your Dockerfile has a different name:DOCKER_BUILDKIT=1 docker build -f Dockerfile-emr-venv --output . <output-path> 2. Set up CONF_ROOT The kedro package command only packs the source code and yet the conf directory is essential for running any Kedro project. To make it available to Kedro separately, its location can be controlled by setting CONF_ROOT. By default, Kedro looks at the root conf folder for all its configurations (catalog, parameters, globals, credentials, logging) to run the pipelines, but this can be customised by changing CONF_ROOT in settings.py.For Kedro versions < 0.18.5 For Kedro versions >= 0.18.5 Change CONF_ROOT in settings.py to the location where the conf directory will be deployed. It could be anything. e.g. ./conf or /mnt1/kedro/conf. For Kedro versions >= 0.18.5 Use the --conf-source CLI parameter directly with kedro run to specify the path. CONF_ROOT need not be changed in settings.py. 3. Package the Kedro Project Package the project using the kedro package command from the root of your project folder. This will create a .whl in the dist folder that will be used when doing spark-submit to the Amazon EMR cluster to specify the --py-files to refer to the source code. 4. Create .tar for conf As described, the kedro package command only packs the source code and yet the conf directory is essential for running any Kedro project. Therefore it needs to be deployed separately as a tar.gz file. It is important to note that the contents inside the folder needs to be zipped and not the conf folder entirely. Use the following command to zip the contents of the conf directory and generate a conf.tar.gz file containing catalog.yml, parameters.yml and other files needed to run the Kedro pipeline. It will be used with spark-submit for the --archives option to unpack the contents into a conf directory.tar -czvf conf.tar.gz --exclude="local" conf/* 5. Create an Entrypoint for the Spark Application Create an entrypoint.py file that the Spark application will use to start the job. This file can be modified to take arguments and can be run only using main(sys.argv) after removing the params array.python entrypoint.py --pipeline my_new_pipeline --params run_date:2023-02-05,runtime:cloud This would mimic the exact kedro run behaviour. Python import sys from proj_name.__main__ import main: if __name__ == "__main__": """ These params could be used as *args to test pipelines locally. The example below will run `my_new_pipeline` using `ThreadRunner` applying a set of params params = [ "--pipeline", "my_new_pipeline", "--runner", "ThreadRunner", "--params", "run_date:2023-02-05,runtime:cloud", ] main(params) """ main(sys.argv) 6. Upload Relevant Files to S3 Upload the relevant files to an S3 bucket (Amazon EMR should have access to this bucket), in order to run the Spark Job. The following artifacts should be uploaded to S3: .whl file created in step #3 Virtual Environment tar.gz created in step 1 (e.g. pyspark_deps.tar.gz) .tar file for conf folder created in step #4 (e.g. conf.tar.gz) entrypoint.py file created in step #5. 7.spark-submit to the Amazon EMR Cluster Use the following spark-submit command as a step on Amazon EMR running in cluster mode. A few points to note: pyspark_deps.tar.gz is unpacked into a folder named environment Environment variables are set referring to libraries unpacked in the environment directory above. e.g. PYSPARK_PYTHON=environment/bin/python conf directory is unpacked to a folder specified in the following after the # symbol ( s3://{S3_BUCKET}/conf.tar.gz#conf) Note the following: Kedro versions < 0.18.5. The folder location/name after the # symbol should match with CONF_ROOT in settings.py Kedro versions >= 0.18.5. You could follow the same approach as earlier. However, Kedro now provides flexibility to provide the CONF_ROOT through the CLI parameters using --conf-source instead of setting CONF_ROOT in settings.py. Therefore --conf-root configuration could be directly specified in the CLI parameters and step 2 can be skipped completely. Shell spark-submit --deploy-mode cluster --master yarn --conf spark.submit.pyFiles=s3://{S3_BUCKET}/<whl-file>.whl --archives=s3://{S3_BUCKET}/pyspark_deps.tar.gz#environment,s3://{S3_BUCKET}/conf.tar.gz#conf --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=environment/bin/python --conf spark.executorEnv.PYSPARK_PYTHON=environment/bin/python --conf spark.yarn.appMasterEnv.<env-var-here>={ENV} --conf spark.executorEnv.<env-var-here>={ENV} s3://{S3_BUCKET}/run.py --env base --pipeline my_new_pipeline --params run_date:2023-03-07,runtime:cloud Summary This post describes the sequence of steps needed to deploy a Kedro project to an Amazon EMR cluster. Set up the Amazon EMR cluster Set up CONF_ROOT (optional for Kedro versions >= 0.18.5) Package the Kedro project Create an entrypoint for the Spark application Upload relevant files to S3 spark-submit to the Amazon EMR cluster Kedro supports a range of deployment targets, including Amazon SageMaker, Databricks, Vertex AI and Azure ML, and our documentation additionally includes a range of approaches for single-machine deployment to a production server.

Kubernetes is an open-source container orchestration platform that has revolutionized the way applications are deployed and managed. With Kubernetes, developers can easily deploy and manage containerized applications at scale and in a consistent and predictable manner. However, managing Kubernetes environments can be challenging, and security risks are always a concern. Therefore, it's important to have the right auditing tools in place to ensure that the Kubernetes environment is secure, compliant, and free of vulnerabilities. In this article, we will discuss some of the top auditing tools that can be used to help secure Kubernetes and ensure compliance with best practices. 1. Kubernetes Audit Kubernetes Audit is a native Kubernetes tool that provides an audit log of all changes made to the Kubernetes API server. In addition, it captures events related to requests made to the Kubernetes API server and the responses generated by the server. This audit information can be used to troubleshoot issues and verify compliance with best practices. Kubernetes Audit can be enabled by adding a flag to the Kubernetes API server configuration file. Once enabled, Kubernetes Audit can capture a wide range of events, such as the creation and deletion of pods, services, and deployments, and changes to service accounts and role bindings. The audit log can be stored in various locations, including log files on the node, a container, or Syslog. By using Kubernetes Audit, administrators can quickly determine if there is any unauthorized access or activities within the Kubernetes environment. It also provides an auditable record of all changes made to the environment, making it easier to identify any issues that may arise. 2. Kube-bench Kube-bench is an open-source tool that is designed to check Kubernetes clusters against the Kubernetes Benchmarks - a collection of security configuration best practices developed by the Center for Internet Security (CIS). Kube-bench can be used to identify any misconfigurations or risks that may exist within the Kubernetes environment and ensure compliance with CIS Kubernetes Benchmark. Kube-bench checks Kubernetes clusters against the 120 available CIS Kubernetes Benchmark checks and produces a report of non-compliant configurations. It can be run manually on a one-time basis or in a continuous integration pipeline that can help ensure that new applications or changes do not affect the cluster's compliance. Kube-bench is capable of testing various aspects of Kubernetes security, including API server, etcd, nodes, pods, network policies, and others. Kube-bench provides detailed instructions on how to resolve each failed to check through remediation steps, making it easy for administrators to address any issues found during the audit process. Overall, kube-bench makes it easier for administrators to achieve a highly secure Kubernetes environment by providing an automated way of checking Kubernetes against the CIS Benchmarks. 3. Kube-hunter Kube-hunter is another open-source tool designed to identify Kubernetes security vulnerabilities by scanning a Kubernetes cluster for weaknesses. The tool uses a range of techniques to identify potential issues, including port scanning, service discovery, and scanning for known vulnerabilities. Kube-hunter can be used to perform various security checks on Kubernetes clusters, including checks for RBAC misconfigurations, exposed Kubernetes dashboards, and other security issues that could lead to unauthorized access. The tool is designed to be easy to use and requires no configuration - simply run kube-hunter from the command line and let it do its job. One unique feature of the kube-hunter is that it can be run as either an offensive or defensive tool. Offensive mode attempts to actively penetrate the Kubernetes cluster to identify vulnerabilities, while defensive mode simulates an attack by scanning for known vulnerabilities and misconfigurations. Both modes are great for identifying security vulnerabilities in a Kubernetes environment and improving overall security posture. Overall, kube-hunter is a powerful tool for identifying security risks in Kubernetes clusters and can be an essential part of any Kubernetes security strategy. The tool is actively developed by Aqua Security and has a large and active community backing it up. 4. Polaris Polaris is a free, open-source tool developed by Fairwinds that performs automated configuration validation for Kubernetes clusters. Polaris can be used to assess cluster compliance with Kubernetes configuration management best practices and ensures Kubernetes resources conform to defined policies. Polaris can detect and alert on various issues that might occur in a Kubernetes cluster, including inappropriate resource requests, non-compliant Pod security policies, misconfigured access control lists, and other common misconfigurations. One of Polaris' most valuable features is its integration with Prometheus Alert Manager, which automatically scans Kubernetes configurations and generates alerts when any of the predefined policies are violated. The tool can also be used to generate custom policies that meet specific cluster and workload requirements. Overall, Polaris is an essential tool for Kubernetes cluster configuration management and is well-suited to companies that require a more proactive approach to security. The automation of the tool significantly reduces the time it takes to perform cluster configurations and policy evaluations, ensuring that Kubernetes resources are continuously provisioned correctly and compliant with established policies. Conclusion Kubernetes provides a powerful platform for deploying and managing containerized applications, but it needs to be secured to protect sensitive data, prevent security breaches, and ensure compliance with industry regulations. Utilizing the right auditing tools is essential for maintaining the security and compliance of Kubernetes environments, detecting vulnerabilities, and verifying configurations. There are several Kubernetes auditing tools available, from native Kubernetes Audit to open-source tools like Kube-bench, kube-hunter, and Polaris. Each tool has its own unique features and capabilities, and finding the right one depends on your specific needs. By implementing and regularly using an auditing tool or combination of tools, organizations can minimize the risk of security breaches, mitigate vulnerabilities, and ensure compliance with regulatory requirements.

In the rapidly evolving landscape of technology, the role of cloud computing is more significant than ever. This revolutionary paradigm continues to reshape the way businesses operate, fostering an environment ripe for unprecedented innovation. In this in-depth exploration, we take a journey into the future of cloud computing, discussing emerging trends such as autonomous and distributed cloud, generative AI tools, multi-cloud strategies, and Kubernetes – the cloud’s operating system. We will also delve into the increasing integration of data, AI, and machine learning, which promises to unlock new levels of efficiency, insight, and functionality in the cloud. Let’s explore these fascinating developments and their implications for developer productivity and the broader industry. Autonomous Cloud: The Self-Managing Future One of the most anticipated trends is the autonomous cloud, where the management of cloud services is largely automated. Leveraging advanced AI and machine learning algorithms, autonomous clouds are capable of self-healing, self-configuring, and self-optimizing. They can predict and preemptively address potential issues, reducing the workload on IT teams and improving the reliability of services. As cloud infrastructure complexity grows, the value of such autonomous features will be increasingly critical in maintaining optimal performance and availability. Distributed Cloud: Cloud Computing at the Edge Distributed cloud is another compelling trend that can revolutionize how we consume cloud services. By extending cloud services closer to the source of data or users, distributed cloud reduces latency, enhances security, and provides better compliance with data sovereignty laws. Moreover, it opens a new horizon for applications that require real-time processing and decision-making, such as IoT devices, autonomous vehicles, and next-gen telecommunication technologies like 5G and beyond. Generative AI Tools: Reshaping Development The integration of generative AI tools into cloud platforms is set to redefine the software development lifecycle. These tools can generate code, perform testing, and even create UI designs, dramatically enhancing developer productivity. With AI-assisted development, software production will be faster and more efficient, enabling developers to focus on higher-level design and strategic tasks rather than getting bogged down in minutiae. Expect this technology to democratize software development and inspire a new generation of cloud-native applications. Developer Productivity: Elevation Through Cloud As cloud services become more sophisticated, they are streamlining processes and reducing the technical burdens on developers. Cloud platforms now offer an array of prebuilt services and tools, from databases to AI models, which developers can leverage without needing to build from scratch. Furthermore, the advent of serverless computing and Function-as-a-Service (FaaS) paradigms is freeing developers from infrastructure management, allowing them to focus solely on their application’s logic and functionality. Kubernetes: The OS Container of the Cloud Kubernetes, often regarded as the ‘OS of the cloud,’ is a crucial player in cloud evolution. As a leading platform for managing containerized workloads, Kubernetes offers a highly flexible and scalable solution for deploying, scaling, and managing applications in the cloud. Its open-source and platform-agnostic nature makes it a key enabler of hybrid and multi-cloud strategies. Kubernetes adoption is set to skyrocket further as more organizations realize the benefits of containerization and microservices architectures. Multi-Cloud Strategies: The Best of All Worlds Enterprises are increasingly adopting multi-cloud strategies, leveraging the strengths of different cloud service providers to meet specific needs. This approach ensures they have the flexibility to use the right tool for the right job. It also provides redundancy, protecting businesses from vendor lock-in and potential outages. However, it brings with it a new level of complexity in terms of management and integration. To address this, we can expect to see further development in multi-cloud management platforms and services. Cloud Security: New Approaches for New Threats With the rise in cyber threats and the increasing amount of sensitive data moving to the cloud, the focus on security is becoming more crucial than ever. As a result, we are likely to witness advancements in cloud security practices, with enhanced encryption, AI-driven threat detection, zero-trust architectures, and blockchain-based solutions. The idea is to create an environment where data is safe, no matter where it resides or how it is accessed. DataOps: The New DevOps Data Fabric DataOps, borrowing principles from the agile methodology and DevOps, is an emerging trend that aims to improve the speed, quality, and reliability of data analytics. It involves automated, process-oriented methodologies, tools, and techniques to improve the orchestration, management, and deployment of data transformations. Additionally, as Generative AI models become more complex and numerous, DataOps provides the necessary support for continuous model updates and refinements, automated deployment, and seamless integration of public and corporate data into production environments as per the data sovereignty requirements. Ultimately, DataOps is a critical component in harnessing the full power of Generative AI by managing the data it thrives on. Quantum Computing: The Frontier of the Next Technological Evolution Quantum computing, with its tremendous computational potential, is set to revolutionize technology by integrating with advanced systems like Generative AI. The emergence of quantum-specific hardware, tools, and programming languages will allow developers to harness quantum power effectively. However, accessibility is key to driving this evolution. Simplified APIs and cloud-based quantum computing services are crucial, enabling developers to create quantum algorithms and utilize quantum services without owning complex hardware. This blend of quantum computing, Generative AI, advanced tools, and improved accessibility is poised to ignite the next leap in technological innovation, solving complex problems and to accelerating progress across fields. Sustainability and Green Cloud Computing Lastly, as society becomes more environmentally conscious, the focus on energy-efficient, or ‘green,’ cloud computing will intensify. Green cloud computing is becoming paramount, aiming to minimize the environmental impact of data centers. This involves optimizing energy usage, leveraging renewable energy, and using AI to manage resources efficiently. Emerging tools allow companies to measure their sustainability metrics and aid in developing energy-efficient applications. Simultaneously, advancements in energy-efficient hardware and commitments from cloud providers towards carbon neutrality are bolstering this sustainable shift. As such, the evolution of cloud computing is not just about technological advancement but also about preserving our planet, reinforcing the industry’s drive toward a greener and more responsible future. Summary In this exploration of the future of cloud computing, we’ve delved into key trends such as autonomous and distributed cloud, generative AI tools, and Kubernetes. We’ve also examined the burgeoning role of DataOps in managing AI data and models, the transformative potential of quantum computing, and the increasing convergence of data, AI, and machine learning. As these trends shape the continuously evolving landscape of technology, they promise to drive innovation, enhance efficiency, and unlock unprecedented functionalities. While some trends may gain momentum, others may be reimagined under new visions, born from previous learning, continuously propelling the digital transformation journey forward.

Want to learn how to auto-scale your Kinesis Data Streams consumer applications on Kubernetes so you can save on costs and improve resource efficiency? This blog offers a step-by-step guide on how to do just that. By leveraging Kubernetes for auto-scaling Kinesis consumer applications, you can benefit from its built-in features such as the Horizontal Pod Autoscaler. What Are Amazon Kinesis and Kinesis Data Streams? Amazon Kinesis is a platform for real-time data processing, ingestion, and analysis. Kinesis Data Streams is a Serverless streaming data service (part of the Kinesis streaming data platform, along with Kinesis Data Firehose, Kinesis Video Streams, and Kinesis Data Analytics. Kinesis Data Streams can scale elastically and continuously adapt to changes in data ingestion rates and stream consumption rates. It can be used to build real-time data analytics applications, real-time dashboards, and real-time data pipelines. Let’s start off with an overview of some of the key concepts of Kinesis Data Streams. Kinesis Data Streams: High-Level Architecture A Kinesis data stream is a set of shards. Each shard has a sequence of data records. The producers continually push data to Kinesis Data Streams, and the consumers process the data in real-time. A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to. Consumers get records from Amazon Kinesis Data Streams, process them, and store their results in Amazon DynamoDB, Amazon Redshift, Amazon S3, etc. These consumers are also known as Amazon Kinesis Data Streams Application. One of the methods of developing custom consumer applications that can process data from KDS data streams is to use the Kinesis Client Library (KCL). How Do Kinesis Consumer Applications Scale Horizontally? The Kinesis Client Library ensures there is a record processor running for every shard and processing data from that shard. KCL helps you consume and process data from a Kinesis data stream by taking care of many of the complex tasks associated with distributed computing and scalability. It connects to the data stream, enumerates the shards within the data stream, and uses leases to coordinate shard associations with its consumer applications. A record processor is instantiated for every shard it manages. KCL pulls data records from the data stream, pushes the records to the corresponding record processor, and checkpoints processed records. More importantly, it balances shard-worker associations (leases) when the worker instance count changes or when the data stream is re-sharded (shards are split or merged). This means that you are able to scale your Kinesis Data Streams application by simply adding more instances since KCL will automatically balance the shards across the instances. But, you still need a way to scale your applications when the load increases. Of course, you could do it manually or build a custom solution to get this done. This is where Kubernetes Event-driven Autoscaling (KEDA) can help. KEDA is a Kubernetes-based event-driven autoscaling component that can monitor event sources like Kinesis and scale the underlying Deployments (and Pods) based on the number of events needing to be processed. To witness auto-scaling in action, you will work with a Java application that uses the Kinesis Client Library (KCL) 2.x to consume data from a Kinesis Data Stream. It will be deployed to a Kubernetes cluster on Amazon EKS and will be scaled automatically using KEDA. The application includes an implementation of the ShardRecordProcessor that processes data from the Kinesis stream and persists it to a DynamoDB table. We will use the AWS CLI to produce data to the Kinesis stream and observe the scaling of the application. Before, we dive in, here is a quick overview of KEDA. What Is KEDA? KEDA is an open-source CNCF project that's built on top of native Kubernetes primitives such as the Horizontal Pod Autoscaler and can be added to any Kubernetes cluster. Here is a high-level overview of its key components (you can refer to the KEDA documentation for a deep dive): The keda-operator-metrics-apiserver component in KEDA acts as a Kubernetes metrics server that exposes metrics for the Horizontal Pod Autoscaler A KEDA Scaler integrates with an external system (such as Redis) to fetch these metrics (e.g., length of a List) to drive auto-scaling of any container in Kubernetes based on the number of events needing to be processed. The role of the keda-operator component is to activate and deactivateDeployment; i.e., scale to and from zero. You will see the Kinesis Stream KEDA scaler in action that scales based on the shard count of AWS Kinesis Stream. Now let's move on to the practical part of this post. Prerequisites In addition to an AWS account, you will need to have the AWS CLI, kubectl, Docker, Java 11, and Maven installed. Setup an EKS Cluster, Create a DynamoDB Table and a Kinesis Data Stream There are a variety of ways in which you can create an Amazon EKS cluster. I prefer using the eksctl CLI because of the convenience it offers. Creating an EKS cluster using eksctl can be as easy as this: eksctl create cluster --name <cluster name> --region <region e.g. us-east-1> For details, refer to the Getting started with Amazon EKS – eksctl documentation. Create a DynamoDB table to persist application data. You can use the AWS CLI to create a table with the following command: aws dynamodb create-table \ --table-name users \ --attribute-definitions AttributeName=email,AttributeType=S \ --key-schema AttributeName=email,KeyType=HASH \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 Create a Kinesis stream with two shards using the AWS CLI: aws kinesis create-stream --stream-name kinesis-keda-demo --shard-count 2 Clone this GitHub repository and change it to the right directory: git clone https://github.com/abhirockzz/kinesis-keda-autoscaling cd kinesis-keda-autoscaling Ok, let's get started! Setup and Configure KEDA on EKS For the purposes of this tutorial, you will use YAML files to deploy KEDA. But you could also use Helm charts. Install KEDA: # update version 2.8.2 if required kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.2/keda-2.8.2.yaml Verify the installation: # check Custom Resource Definitions kubectl get crd # check KEDA Deployments kubectl get deployment -n keda # check KEDA operator logs kubectl logs -f $(kubectl get pod -l=app=keda-operator -o jsonpath='{.items[0].metadata.name}' -n keda) -n keda Configure IAM Roles The KEDA operator as well as the Kinesis consumer application need to invoke AWS APIs. Since both will run as Deployments in EKS, we will use IAM Roles for Service Accounts (IRSA) to provide the necessary permissions. In this particular scenario: KEDA operator needs to be able to get the shard count for a Kinesis stream: it does so by using DescribeStreamSummary API. The application (KCL library to be specific) needs to interact with Kinesis and DynamoDB: it needs a bunch of IAM permissions to do so. Configure IRSA for KEDA Operator Set your AWS Account ID and OIDC Identity provider as environment variables: ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) #update the cluster name and region as required export EKS_CLUSTER_NAME=demo-eks-cluster export AWS_REGION=us-east-1 OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") Create a JSON file with Trusted Entities for the role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:aud": "sts.amazonaws.com", "${OIDC_PROVIDER}:sub": "system:serviceaccount:keda:keda-operator" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust_keda.json Now, create the IAM role and attach the policy (take a look at policy_kinesis_keda.json file for details): export ROLE_NAME=keda-operator-kinesis-role aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust_keda.json --description "IRSA for kinesis KEDA scaler on EKS" aws iam create-policy --policy-name keda-kinesis-policy --policy-document file://policy_kinesis_keda.json aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/keda-kinesis-policy Associate the IAM role and Service Account: kubectl annotate serviceaccount -n keda keda-operator eks.amazonaws.com/role-arn=arn:aws:iam::${ACCOUNT_ID}:role/${ROLE_NAME} # verify the annotation kubectl describe serviceaccount/keda-operator -n keda You will need to restart KEDA operator Deployment for this to take effect: kubectl rollout restart deployment.apps/keda-operator -n keda # to verify, confirm that the KEDA operator has the right environment variables kubectl describe pod -n keda $(kubectl get po -l=app=keda-operator -n keda --output=jsonpath={.items..metadata.name}) | grep "^\s*AWS_" # expected output AWS_STS_REGIONAL_ENDPOINTS: regional AWS_DEFAULT_REGION: us-east-1 AWS_REGION: us-east-1 AWS_ROLE_ARN: arn:aws:iam::<AWS_ACCOUNT_ID>:role/keda-operator-kinesis-role AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token Configure IRSA for the KCL Consumer Application Start by creating a Kubernetes Service Account: kubectl apply -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: kcl-consumer-app-sa EOF Create a JSON file with Trusted Entities for the role: read -r -d '' TRUST_RELATIONSHIP <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:aud": "sts.amazonaws.com", "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:kcl-consumer-app-sa" } } } ] } EOF echo "${TRUST_RELATIONSHIP}" > trust.json Now, create the IAM role and attach the policy (take a look at policy.json file for details): export ROLE_NAME=kcl-consumer-app-role aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for KCL consumer app on EKS" aws iam create-policy --policy-name kcl-consumer-app-policy --policy-document file://policy.json aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/kcl-consumer-app-policy Associate the IAM role and Service Account: kubectl annotate serviceaccount -n default kcl-consumer-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::${ACCOUNT_ID}:role/${ROLE_NAME} # verify the annotation kubectl describe serviceaccount/kcl-consumer-app-sa The core infrastructure is now ready. Let's prepare and deploy the consumer application. Deploy KCL Consumer Application to EKS You first need to build the Docker image and push it to Amazon Elastic Container Registry (ECR) (refer to the Dockerfile for details). Build and Push the Docker Image to ECR # create runnable JAR file mvn clean compile assembly\:single # build docker image docker build -t kcl-consumer-app . AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) # create a private ECR repo aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com aws ecr create-repository --repository-name kcl-consumer-app --region us-east-1 # tag and push the image docker tag kcl-consumer-app:latest $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/kcl-consumer-app:latest docker push $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/kcl-consumer-app:latest Deploy the Consumer Application Update the consumer.yaml to include the Docker image you just pushed to ECR. The rest of the manifest remains the same: apiVersion: apps/v1 kind: Deployment metadata: name: kcl-consumer spec: replicas: 1 selector: matchLabels: app: kcl-consumer template: metadata: labels: app: kcl-consumer spec: serviceAccountName: kcl-consumer-app-sa containers: - name: kcl-consumer image: AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/kcl-consumer-app:latest imagePullPolicy: Always env: - name: STREAM_NAME value: kinesis-keda-demo - name: TABLE_NAME value: users - name: APPLICATION_NAME value: kinesis-keda-demo - name: AWS_REGION value: us-east-1 - name: INSTANCE_NAME valueFrom: fieldRef: fieldPath: metadata.name Create the Deployment: kubectl apply -f consumer.yaml # verify Pod transition to Running state kubectl get pods -w KCL App Autoscaling in Action With KEDA Now that you've deployed the consumer application, the KCL library should jump into action. The first thing it will do is create a "control table" in DynamoDB - this should be the same as the name of the KCL application (which in this case is kinesis-keda-demo). It might take a few minutes for the initial coordination to happen and the table to get created. You can check the logs of the consumer application to track the progress. kubectl logs -f $(kubectl get po -l=app=kcl-consumer --output=jsonpath={.items..metadata.name}) Once the lease allocation is complete, check the table and note the leaseOwner attribute: aws dynamodb describe-table --table-name kinesis-keda-demo aws dynamodb scan --table-name kinesis-keda-demo Now, let's send some data to the Kinesis stream using the AWS CLI. export KINESIS_STREAM=kinesis-keda-demo aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user1@foo.com --data $(echo -n '{"name":"user1", "city":"new york"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user2@foo.com --data $(echo -n '{"name":"user2", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user3@foo.com --data $(echo -n '{"name":"user3", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user4@foo.com --data $(echo -n '{"name":"user4", "city":"seattle"}' | base64) The KCL application persists each record to a target DynamoDB table (which is named users in this case). You can check the table to verify the records. aws dynamodb scan --table-name users Notice that the value for the processed_by attribute? It's the same as the KCL consumer Pod. This will make it easier for us to verify the end-to-end autoscaling process. Create the KEDA Scaler for Kinesis Here is the ScaledObject definition. Notice that it's targeting the kcl-consumer Deployment (the one we just created) and the shardCount is set to 1: apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: aws-kinesis-stream-scaledobject spec: scaleTargetRef: name: kcl-consumer triggers: - type: aws-kinesis-stream metadata: # Required streamName: kinesis-keda-demo # Required awsRegion: "us-east-1" shardCount: "1" identityOwner: "operator" Create the KEDA Kinesis scaler: kubectl apply -f keda-kinesis-scaler.yaml Verify KCL Application Auto-Scaling We started off with one Pod of our KCL application. But, thanks to KEDA, we should now see the second Pod coming up. kubectl get pods -l=app=kcl-consumer -w # check logs of the new pod kubectl logs -f <enter Pod name> Our application was able to auto-scale to two Pods because we had specified shardCount: "1" in the ScaledObject definition. This means that there will be one Pod for per shard in the Kinesis stream. Check kinesis-keda-demo control table in DynamoDB: you should see an update for the leaseOwner. Let's send some more data to the Kinesis stream. export KINESIS_STREAM=kinesis-keda-demo aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user5@foo.com --data $(echo -n '{"name":"user5", "city":"new york"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user6@foo.com --data $(echo -n '{"name":"user6", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user7@foo.com --data $(echo -n '{"name":"user7", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user8@foo.com --data $(echo -n '{"name":"user8", "city":"seattle"}' | base64) Verify the value for the processed_by attribute. Since we have scaled out to two Pods, the value should be different for each record since each Pod will process a subset of the records from the Kinesis stream. Increase Kinesis Stream Capacity Let's scale out the number of shards from two to three and continue to monitor KCL application auto-scaling. aws kinesis update-shard-count --stream-name kinesis-keda-demo --target-shard-count 3 --scaling-type UNIFORM_SCALING Once Kinesis re-sharding is complete, the KEDA scaler will spring into action and scale out the KCL application to three Pods. kubectl get pods -l=app=kcl-consumer -w Just like before, confirm that the Kinesis shard lease has been updated in the kinesis-keda-demo control table in DynamoDB. Check the leaseOwner attribute. Continue to send more data to the Kinesis stream. As expected, the Pods will share the record processing and this will reflect in the processed_by attribute in the users table. export KINESIS_STREAM=kinesis-keda-demo aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user9@foo.com --data $(echo -n '{"name":"user9", "city":"new york"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user10@foo.com --data $(echo -n '{"name":"user10", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user11@foo.com --data $(echo -n '{"name":"user11", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user12@foo.com --data $(echo -n '{"name":"user12", "city":"seattle"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user14@foo.com --data $(echo -n '{"name":"user14", "city":"tel aviv"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user15@foo.com --data $(echo -n '{"name":"user15", "city":"new delhi"}' | base64) aws kinesis put-record --stream-name $KINESIS_STREAM --partition-key user16@foo.com --data $(echo -n '{"name":"user16", "city":"seattle"}' | base64) Scale Down So far, we have only scaled in one direction. What happens when we reduce the shard capacity of the Kinesis stream? Try this out for yourself: reduce the shard count from three to two and see what happens to the KCL application. Once you have verified the end-to-end solution, you should clean up the resources to avoid incurring any additional charges. Delete Resources Delete the EKS cluster, Kinesis stream, and DynamoDB table. eksctl delete cluster --name keda-kinesis-demo aws kinesis delete-stream --stream-name kinesis-keda-demo aws dynamodb delete-table --table-name users Conclusion In this post, you learned how to use KEDA to auto-scale a KCL application that consumes data from a Kinesis stream. You can configure the KEDA scaler as per your application requirements. For example, you can set the shardCount to 3 and have one Pod for every three shards in your Kinesis stream. However, if you want to maintain a one-to-one mapping, you can set the shardCount to 1 and KCL will take care of distributed coordination and lease assignment, thereby ensuring that each Pod has one instance of the record processor. This is an effective approach that allows you to scale out your Kinesis stream processing pipeline to meet the demands of your applications.

File handling in AWS is essentially a cloud storage solution that enables corporations to store, manage, and access their data in the cloud. AWS provides a wide range of cloud storage solutions to handle files effectively. Among these, Amazon S3 is the most popular and widely used service for object storage in the cloud. It offers highly scalable, durable, and secure storage for any kind of data, such as images, videos, documents, and backups, at a low cost. Amazon EBS, on the other hand, is a block-level storage service that provides persistent storage volumes for use with Amazon EC2 instances. It is ideal for transactional workloads that require low latency and high throughput. Amazon EFS, a fully managed file system, is designed to be highly available, durable, and scalable. It provides a simple interface to manage file systems and supports multiple EC2 instances simultaneously. Finally, Amazon FSx for Windows File Server provides a fully managed native Microsoft Windows file system that can be accessed over the industry-standard SMB protocol. This service is ideal for customers who want to move their Windows-based applications to the AWS Cloud and need shared file storage. With these AWS file-handling services, firms can store, manage, and access their data efficiently and securely in the cloud. How EBS Can Help With High-Performance Computing Amazon Elastic Block Store (EBS) is a cloud-based block-level storage service that provides highly scalable and persistent storage volumes for use with Amazon EC2 instances. You can think of it as an external hard drive to your computer. You can attach it to a laptop, detach it and attach it to another laptop. However, you cannot attach it to two laptops at the same time. EBS volumes work in a similar fashion. EBS volumes are designed to deliver low-latency performance and high throughput, making them ideal for high-performance computing (HPC) workloads. EBS volumes can be easily attached to EC2 instances, allowing users to create customized HPC environments that meet their specific needs. Additionally, EBS volumes support a variety of data-intensive workloads, such as databases, data warehousing, and big data analytics, making them a versatile choice for enterprises looking to optimize their storage infrastructure. Since EBS is attached to a single ECS instance, it acts as a dedicated file store for that instance, thereby providing low-latency file access. How S3 Becomes the Cornerstone of Your Cloud Storage Needs and Capacities No discussion on cloud storage is complete without mentioning S3. EBS is great as a storage solution when the application accessing it is running on a single server. But when multiple servers need to access the same set of files, EBS will not work. Amazon S3 (Simple Storage Service) is an object storage service that offers industry-leading scalability, durability, and security for organizations of all sizes. With S3, users can store and retrieve any amount of data from anywhere in the world, making it an ideal solution for cloud storage needs. S3 provides a highly available and fault-tolerant architecture that ensures data is always accessible, with a durability of 99.999999999% (11 nines). This means that operations can rely on S3 for critical data storage, backup, and disaster recovery needs. Additionally, S3 supports a variety of use cases, such as data lakes, content distribution, and backup and archiving, making it a versatile choice for enterprises looking to optimize their cloud storage infrastructure. To connect to an S3 bucket, EC2 instances running within a VPC can either use a VPC endpoint or connect over the internet. Connecting through a VPC endpoint will provide much lower latency. When an application requires file interchange with a third-party application, S3 is one of the best choices. Files from external applications can be put into an S3 bucket using the SFTP protocol; S3 Transfer Family provides support for this. Applications running in EC2 instances can then access these files directly. Unlock the Power of EFS for Enterprise-Grade Scalable File Storage S3 can be accessed by many applications, even external ones running outside AWS. However, if the requirement is to store files to be accessed by multiple EC2 instances within a VPC, EFS provides a scalable low-latency solution. This is similar to accessing shared files within a corporate intranet. Amazon Elastic File System (EFS) is a fully managed file storage service designed to provide enterprise-grade scalable file storage for use with Amazon EC2 instances. EFS volumes are highly available, durable, and scalable, making them ideal for companies that require high-performance file storage solutions. With EFS, users can create file systems that can scale up or down automatically in response to changes in demand, without impacting performance or availability. This means that organizations can easily manage their file storage needs, without having to worry about capacity planning or performance issues. Additionally, EFS supports a variety of file-based workloads, such as content management, web serving, and home directories, making it a versatile choice for firms of all sizes. After setting up EFS within a VPC, mount targets need to be created. EFS supports one mount target per availability zone within a VPC. Once the mount targets are created the file system can be mounted onto compute resources like EC2, ECS and Lambda. Revolutionize Your Data Access and Analysis With FSx for Lustre and Windows File System Amazon FSx for Lustre and Windows file system is a fully managed file storage service that helps organizations revolutionize their data access and analysis capabilities. FSx for Lustre is a high-performance file system designed for compute-intensive workloads, such as machine learning, high-performance computing, and video processing. It delivers sub-millisecond latencies and high throughput, making it ideal for applications that require fast access to large datasets. FSx for Windows file system, on the other hand, provides fully managed Windows file shares backed by Windows Server and the Server Message Block (SMB) protocol. It supports a wide range of use cases, such as home directories, application data, and media editing, and provides built-in features for data protection and disaster recovery. Selecting a File-Handling Service When deciding whether to use Amazon EFS or Amazon FSx for Lustre and Windows file systems, it's important to consider the specific needs of your organization. EFS is a good choice for general-purpose file storage workloads that require high availability and scalability, such as content management, web serving, and home directories. On the other hand, FSx for Lustre is designed for compute-intensive workloads that require high-performance file storage, such as machine learning and high-performance computing. Furthermore, FSx for Windows file system is ideal for firms that require fully managed Windows file shares backed by Windows Server and the SMB protocol. Ultimately, the choice between EFS and FSx will depend on the specific requirements of your workload and the level of performance, scalability, and manageability that your application requires.