Tools

JetBrains IDEs based on the IntelliJ platform are probably one of the most common IDEs in existence nowadays. Their popularity is especially visible within the JVM languages community, where IntelliJ IDEA remains the right-away IDE pick for most developers. All of this is despite some new competitors showing up and old competitors overcoming their previous shortcomings and joining back the table. In this text, I would like to describe the plugins for IntelliJ IDEA that may be a great help for you in your daily fight with your tasks, and that will make your work easier. Some plugins will be language agonistic, while others can be language dependent. Downloadable Plugins Linter Static code check is a great tool and helps fight for our code quality. Additionally, it can give us an entry point about the overall system state when we start new work for an already existing system. Fortunately, there are also a greater number of plugins we can use to make such checks. SonarLint is probably the chief amongst them and is especially helpful when you are using SonarQube in your CI process – you can integrate your local SonarLint to use the same rules as CI SonarQube. As for overall UX, using SonarLint from IDE gives quite a good feeling, but Sonar is a relatively simple tool from the user's perspective, so it should be expected. Some checks on the overall project could be faster, but after a certain number of classes, it is understandable. It can also be a reasonable way to enforce some general practices among the team. If you want to use some other static check tools, I am aware that: PyCharm supports the Pylint plugin. WebStorm supports ESLint. Probably other IDEs support other more specialized linters, but I have no experience working with them. Kubernetes Most of us nowadays are using Kubernetes in one way or another via self-hosted or managed cloud services. The Kubernetes Plugin can help you interact with your K8s deployments, as it provides an extensive set of functionality for working with Kubernetes. Most notable of them are: Browsing cluster objects Extracting and editing configurations Describing deployments and pods Viewing and downloading pod logs Attaching pod console Running shell in a pod Forwarding ports to a pod Additionally, the plugin adds support for working with Kubernetes remotely (or locally) from your IDE. De facto adding the UI over the Kubectl to the functionalities of IDE. If you are bored or tired using your other Kubernetes tools like kube-dashboard or Lens, then give a try to the K8s IDE plugin, as it can be a way to go for you. As far as I know, the plugin is supported by all JetBrains IDEs. .ignore Probably 110% of us work with some version control systems (some with more than one) - either Git, Mercurial, or, god forbid, SVN or anything older. Additionally, we are working with software that sometimes requires a tremendous amount of configuration that we may not want to share with others. In such circumstances, the need to “hide” some files from others and not send them to remote repositories or not include them in our Docker containers is totally understandable. Of course, most of the tools offer their own type of ignore files - files that allow us to exclude certain other files from being sent to remote places - like ".gitignore" or ".dockerignore," but their default support for IDE is neither great nor terrible: it just is. The .ignore plugin aims to help us work with such files by syntax highlighting or rules inspection. Moreover, the plugin can mark excluded files in the IDE project view based on the configuration from a particular ".*ignore" file. Besides support for previously mentioned ".gitignore" and ".dockerignore" files, it supports other file types like ".npmignore" or ".helmignore". The full list of supported files is long and available on the plugin home page. Key Promoter X Using hotkeys and keyboard shortcuts for doing stuff inside the IDE is a great way to speed up your development process. Additionally, a good set of such keys can greatly improve your general UX of using the tool. However, remembering all the shortcuts, or even the bigger part - in fact, anything besides the ones that we are using every day - can be at least problematic, if not impossible. As in most modern-day IDEs, they number in dozens, and our brain-built-in RAM cannot contain them all, especially when we are using at most 5-10 shortcuts in our daily work (I do not have any hard data, it is just an educated guess based on some of my experience). Here comes the Key Promoter X plugin, all in white. The plugin knows all the hotkeys and keeps reminding us about them each time we use the feature by manually clicking instead of using a particular shortcut. It does so by prompting such a nice pop-up in the bottom right corner of IDE. Here you can see that I missed the shortcut for opening the terminal window 37 times and that Key Promoter reminds me of the correct one - in da face. I agree that in the long term, such pop-ups can be distracting and annoying. However, the Key Promoter allows us to disable certain alerts, just as you can see on the screen. Thus, the number of alerts can be greatly decreased. Moreover, you can just configure the promoter to work only for certain shortcuts from the start. Personally, I like to have this pop-up anyway. Maybe by accident, I will learn something new and useful. Using the Key Promoter can save you time reading IDE docs. Additionally, Key Promoter can be a good way to learn IDE hotkeys if you switch between systems - i.e., from Linux to Mac, or vice versa. Cloud Tools Probably a fair pair of us (software engineers) are using some kind of cloud. You name it AWS, GCP, Azure, or some other less commonly known provider. Fortunately for you and me (I am also using a cloud), JetBrains IDEs also have plugins for that, namely: AWS Toolkit Azure Toolkit for IntelliJ Google Cloud Code Alibaba Cloud Toolkit (Alibaba is probably the world's biggest cloud and chief player in Asian markets) In general, the plugins allow you to interact with your chosen cloud from IDE and manage your cloud services without changing the windows you work on. A deeper description of all of them in detail is worth an article itself, so I have just added links to each plugin homepage on JetBrains marketplace - one probably can not find better intro-level descriptions. AI Coding Buddy The importance of prompt engineering and overall machine learning base code helpers cannot be overseen nowadays. As the saying goes – you will not be replaced by AI but you will be replaced by a person using AI. No matter if you prefer Copilot or Chat GPT, JetBrains IDEs have plugins for all of that. Each tool has its own unique plugin – in the case of Chat GPT it is even a few plugins, so you can choose whichever suits you best. Of course, some problems may arise in case you are interested in less commonly known coding helpers, but maybe there is also a plugin for them. There are even plugins for quick Stack Overflow search (more than one) if you prefer a more “old-fashioned” approach to prompt-supported coding. .env Files Support This is a great plugin, especially when you are working a lot with all kinds of environmental variables. It promises and delivers better support for name completion, go-to usage and definition (same as normal go-to included in base IDE), and of course, syntax highlighting. Such a set of features can be very helpful while working with Docker or Docker Compose files, which in many cases have at least a couple of environmental variables inside. Additionally, if you are using PyCharm and .env files, the plugin also promises additional support over the one provided by IDE. Here is an example of .env files supported in PyCharm. Without plugin: With plugin: For me, the colored one looks better, and for you? Rainbow Brackets This is a very interesting and not-so-little plugin implementing a very simple yet extremely useful idea. It is just using different colors to mark openings and closings of brackets inside our code. Additionally, each variable has its unique color, which is constant through all of its usage in a particular scope; thus, we can easily catch which variable is used where. Unfortunately, the plugin is not open source, and only a part of the features is available in the free version. On the other hand, the cost is relatively low - a yearly subscription costs $15 USD (full pricing can be viewed here). Below is a simple presentation of what Rainbow is doing with your code. For me, it looks way more readable than plain old IntelliJ white, and you? Image from the official Rainbow Brackets plugin page gRPC Even if you are not a particular fan of Google and its doings, you must have heard about gRPC. If not, then my last piece of text may be interesting for you. Over recent years gRPC gained quite an audience. JetBrains also addressed the issue of its support through their IDEs. Thus, the gRPC plugin was created. It adds standard IDE support like syntax highlighting and go-to options for ".proto" files alongside some easily available documentation for gRPC building blocks. What is more, it allows us to create gRPC calls in IDEs' built-in HTTP client, effectively giving us a gRPC client we can use to call local and remote APIs. They also have decent documentation on how to do that - here is the link. Randomness This is quite a powerful utility plugin that specializes in generating all kinds of dummy data. The plugin is especially useful when writing tests – personally, I always have a problem with all the naming there, and in most cases, I end up with things like String testName = “test-{n}”. As for now, the plugin supports five basic types of data: Integers, such as 7,826,922, in any base from binary to hexatrigesimal (base 36) Decimals, such as 8,816,573.10, using customizable separators Strings, such as "PaQDQqSBEH," with custom symbol lists Words, such as "Bridge," with custom word lists UUIDs, such as 0caa7b28-fe58-4ba6-a25a-9e5beaaf8f4b, with or without dashes String Manipulation The plugin can make all kinds of magic with plain text for you. First of all, it gives you the possibility to easily switch cases of your text from kebab-case/snake_case or PascalCase/camelCase. Besides that, it allows for things like encoding text to HTML. Moreover, it can do all kinds of operations on plain text - swap words, reverse letters or multi-replace, and many others. I advise you to visit the plugin home page and check its complete feature list. You may find the one feature that you were missing until this point, and that will change your view. IdeaVim The plugin adds an extensive set of VIM features to IDE, from the simple inserts and removes to Vim hotkeys. It also supports VIM macros and plugins, effectively creating a fully functional VIM foreground over your IDE. Personally, I am not a fan; however, I can see certain benefits, especially if you are a VIM fan and have high proficiency in using it. In such cases, the plugin can boost your coding speed. On the other hand, if you are a VIM newcomer, the plugin can also be a decent way of learning how to use VIM – at least quitting the VIM is easier here than in the terminal. CPU Usage Indicator A “small” utility plugin that adds information about our current CPU usage in the bottom right corner of the IDE screen. Additionally, adding the information about the system CPU consumed by IDE itself. It also has some options that may be especially useful for troubleshooting potential IDE memory problems like taking the thread dump from the last IDE freeze. Just please keep in mind that constantly asking for CPU usage can be an “expensive” operation. Nyan Progress Bar Here comes the real champion of all the plugins for JetBrains IDEs, the plugin that will change your life and the way you are using your IDE, the Nyan Progress Bar.The plugin replaces the classic JetBrains progress bar with super extra Nyan Cat animation.There is nothing more I can say here but join me in the Nyan Progress Bar club – it is totally worth it. Themes Bundles The ability to customize the look of our IDE - probably the most viewed single window in our daily life - and express ourselves in some way may be quite an important thing for many people. Thus, JetBrains IDEs also have plugins for that - in fact, quite a few of them, starting from “simple” colors changes in the form of plugins like Material Theme UI through the additional icons pack in the form of plugins like Atom Material Icons. Everyone can pick something which suits their needs – just be careful. Choosing and customizing your perfect color design can take a very very long time (trust me - been there, done that, wasted a lot of time). JMH Plugin If you are a software engineer related to the JVM ecosystem, you probably have heard about JMH – the microbenchmark framework for JVM applications. This plugin adds full support of JMH to IDE. The level of support it provides is in pair with one IDE already has for libraries like JUnit or TestNG.We can run the singular, run the whole test suite from IntelliJ, or pass the configuration from the standard IDE window. Scala I would not be myself (Scala Software Engineer) if I would not mention this plugin. It adds full support for Scala syntax, build tools, and test libraries. Basically, providing a similar level of support as Java has from IntelliJ. Of course, there are some corner cases – like more complex implicit or Scala 3 support, but nevertheless, the level of support as a whole is pretty good. The plugin even has its own blog and Twitter profile, so if you want to know what is up in the plugin, both things may be worth following or checking from time to time. IntelliJ with this plugin is by far my favorite Scala IDE despite Metals rapidly growing in strength and fame. Built-In Plugins Docker The plugin was added to the default IDE plugin bundle in the November 2017 release. It focuses on extending an already existing set of IDE capabilities with Docker integration. It allows for connecting with local or remote Docker runtime and images. Besides, it adds standard (go-to, syntax highlight) support for Docker Compose and docker file, making it easier to work with. Additionally, it allows running and debugging Docker images from IDEs. Despite not being as advanced as Docker Desktop, it can be a reasonable replacement if, for some reason, you cannot use it. Lombok This plugin is a vital point of interest for Java software engineers as it adds standard IDE support for Lombok annotations. Lombok, on the other hand, tries to address a few mundane problems the Java language has. If the Lombok way is correct or not is another matter, and it is quite out of the scope of this article. The plugin is relatively “simple;” however, it is an interesting case to observe JetBrains’ reaction to feedback from their community. The plugin started as a community plugin and was then added by JetBrains to the standard IntelliJ plugins bundle based on the feedback from the users' community. Summary JetBrains IDEs are quite powerful beasts by themselves, but via the usage of plugins, we can bring their set of features to a whole new level. They are essentially ending up with an all-in-one machine for doing all kinds of things related to our daily work without even switching windows: that is the level of time-managed optimization. Thank you for your time.

Several tools/scripts are included in the bin directory of the Apache Kafka binary installation. Even if that directory has a number of scripts, through this article, I want to highlight the five scripts/tools that I believe will have the biggest influence on your development work, mostly related to real-time data stream processing. After setting up the development environment, followed by installation and configuration of either with single-node or multi-node Kafka cluster, the first built-in script or tool is kafka-topic.sh. Kafka-topics.sh Using kafka-topic.sh, we can create a topic on the cluster to publish and consume some test messages/data. Subsequently, alter the created topic, delete a topic, change/modify the replication factor, number of partitions for that topic, list of topics created on the cluster, etc, and so on. Kafka-console-producer.sh We may directly send records to a topic from the command line using this console producer script. Typically, this script is an excellent method to test quickly whether the dev or staging Kafka single/multi-node setup is working or not. Also very helpful to quickly test new consumer applications when we aren’t producing records to the topics yet by integrating messages/records senders apps or IoT devices etc. TypeScript kafka-console-producer --topic <<topic name>> / --broker-list <broker-host:port> Even though the above method of using the command line producer just sends values rather than any keys, using the key.separator, we could send full key/value pairs too. TypeScript kafka-console-producer –-topic <topic name>\ –-broker-list <broker-host:port> \ –-property "parse.key=true" \ –-property "key.separator=:" Kafka-console-consumer.sh Now let’s look at record consumption from the command line, which is the opposite side of the coin. We may directly consume records from a Kafka topic using the console consumer’s command-line interface. Quickly starting a consumer may be a very useful tool for experimenting or troubleshooting. Run the below command to rapidly verify that our producer application is delivering messages to the topic. To see all the records from the start, we can add a –from-beginning flag to the command, and we’ll see all records produced to that topic. TypeScript kafka-console-consumer -–topic <topic name> \ -–bootstrap-server <broker-host:port> \ -–from-beginning The plain consumers work with records of primitive Java types: String, Long, Double, Integer, etc. The default format expected for keys and values by the plain console consumer is the String type. We have to pass the fully qualified class names of the appropriate deserializers using the command line flags –key-deserializer and –value-deserializer if the keys or values are not strings. By default, the console consumer only prints the value component of the messages to the screen. If we want to see the keys as well, need to append the following flags: TypeScript kafka-console-consumer -–topic <topic name>\ -–bootstrap-server <broker-host:port> \ -–property "print.key=true" -–property "key.separator=:" --from-beginning Kafka-dump-log.sh The kafka-dump-log command is our buddy whether we just want to learn more about Kafka’s inner workings or we need to troubleshoot a problem and validate the content. By leveraging kafka-dump-log.sh, we can manually inspect the underlying logs of a topic. TypeScript kafka-dump-log \ -–print-data-log \ -–files ./usr/local/kafka-logs/FirstTopic-0/00000000000000000000.log For a full list of options and a description of what each option does, run kafka-dump-log with the –help flag. –print-data-log flag specifies to print the data in the log. –files flag is required. This could also be a comma-separated list of files. Each message’s key, payload (value), offset, and timestamp are easily visible in the dump log. Henceforth, lots of information that are available in the dump-log and can be extracted for troubleshooting, etc. Please note that in the captured screenshot, the keys and values for the topic FirstTopic are strings. To run the dump-log tool with key or value types other than strings, we’ll need to use either the –key-decoder-class or the –value-decoder-class flags. Kafka-delete-records.sh As we know, Kafka stores records for topics on disk and retains that data even once consumers have read it. Instead of being kept in a single large file, records are divided up into partition-specific segments where the offset order is continuous across segments for the same topic partition. As servers do not have infinite amounts of storage, using Kafka’s configuration settings, we can control how much data can be retained based on time and size. In the server.properties file using log.retention.hours, which defaults to 168 hours (one week), the time configuration for controlling data retention can be achieved. And another property log.retention.bytes control how large segments can grow before they are eligible for deletion. By default, log.retention.bytes is commented in the server.properties file. Using log.segment.bytes, the maximum size of a log segment file is defined. The default value is 1073741824, and when this size is reached, a new log segment will be created. Typically we never want to go into the filesystem and manually delete files. Typically we never want to go into the filesystem and manually delete files. In order to save space, we would prefer a controlled and supported method of deleting records from a topic. Using Kafka-delete-records.sh, we can delete data as desired. To execute Kafka-delete-records.sh on the terminal, two mandatory parameters are required. –bootstrap-server: the broker(s) to connect to for bootstrapping –offset-json-file: a JSON file containing the deletion settings Here’s an example of the JSON file: As we can see above, the format of the JSON is simple. It’s an array of JSON objects. Each JSON object has three properties: Topic: The topic to delete from Partition: The partition to delete from Offset: The offset we want the delete to start from, moving backward to lower offsets For this example, I’m reusing the same topic, FirstTopicfrom the dump-log tool, so it’s a very simple JSON file. If we had more partitions or topics, we would simply expand on the JSON config file above. We could simply determine the beginning offset to begin the deletion process because the example topic FirstTopic only has 17 records. But in actual use, we’ll probably be unsure what offset to use. If we provide -1, then the offset of the high watermark is used, which means we will be deleting all the data currently in the topic. The high watermark is the highest available offset for consumption. After executing the following command on the terminal, we can see the following TypeScript kafka-delete-records -–bootstrap-server <broker-host:port> \ --offset-json-file offsets.json The results of the command show that Kafka deleted all records from the topic partition FirstTopic-0. The low_watermark value of 17 indicates the lowest offset available to consumers. Because there were only 17 records in the FirstTopic topic. In a nutshell, this script is helpful when we want to reset the consumer without stopping and restarting the Kafka broker cluster to flush a Kafka topic if receives bad data. This script allows us to delete all the records from the beginning of a partition, until the specified offset. Hope you have enjoyed this read. Please like and share if you feel this composition is valuable.

GitOps is a relatively new addition to the growing list of "Ops" paradigms taking shape in our industry. It all started with DevOps, and while the term DevOps has been around for some years now, it seems we still can't agree whether it's a process, mindset, job title, set of tools, or some combination of them all. We captured our thoughts about DevOps in our introduction to DevOps post, and we dive even deeper in our DevOps engineer's handbook. The term GitOps suffers from the same ambiguity, so in this post we look at: The history of GitOps GitOps goals and ideals The limitations of GitOps The tools that support GitOps The practical implications of adopting GitOps in your own organization The Origins of GitOps The term GitOps was originally coined in a blog post by WeaveWorks called GitOps - Operations by Pull Request. The post described how WeaveWorks used Git as a source of truth, leading to the following benefits: Since that original blog post, initiatives like the GitOps Working Group have been organized to: This working group recently released version one of their principles, which states that: The contrast between low level implementations of GitOps found in most blog posts and the high level ideals of a GitOps system described by the working group is worth discussion, as the differences between them is a source of much confusion. GitOps Doesn't Imply the Use of Git Most discussions around GitOps center on how building processes on Git give rise to many of the benefits ascribed to the GitOps paradigm. Git naturally provides an (almost) immutable history of changes, with changes annotated and approved via pull requests, and where the current state of the Git repository naturally represents the desired state of a system, thus acting as a source of truth. The overlap between Git and GitOps is undeniable. However, you may have noticed that Git was never mentioned as a requirement of GitOps by the working group. So while Git is a convenient component of a GitOps solution, GitOps itself is concerned with the functional requirements of a system rather than checking your declarative templates into Git. This distinction is important, because many teams fixate on the "Git" part of GitOps. The term GitOps is an unfortunate name for the concept it's trying to convey, leading many to believe Git is the central aspect of GitOps. But GitOps has won the marketing battle and gained mind share in IT departments. While it may be a restrictive term to describe functional requirements unrelated to Git, GitOps is now the shorthand for describing processes that implement a set of high level concerns. GitOps Doesn't Imply the Use of Kubernetes Kubernetes was the first widely used platform to combine the ideas of declarative state and continuous reconciliation with an execution environment to implement the reconciliation and host running applications. It really is magic to watch a Kubernetes cluster reconfigure itself to match the latest templates applied to the system. So it's no surprise that Kubernetes is the foundation of GitOps tools like Flux and Argo CD, while posts like 30+ Tools List for GitOps mention Kubernetes 20 times. While continuous reconciliation is impressive, it's not really magic. Behind the scenes Kubernetes runs a number of operators that are notified of configuration changes and execute custom logic to bring the cluster back to the desired state. The key requirements of continuous reconciliation are: Access to the configuration or templates declaratively expressing the desired state The ability to execute a process capable of reconciling a system when configuration is changed An environment in which the process can run Kubernetes bakes these requirements into the platform, making it easy to achieve continuous reconciliation. But these requirements can also be met with some simple orchestration, Infrastructure as Code (IaC) tools like Terraform, Ansible, Puppet, Chef, CloudFormation, Arm Templates, and an execution environment like a CI server: IaC templates can be stored in Git, file hosting platforms like S3 or Azure Blob Storage, complete with immutable audit histories. CI/CD systems can poll the storage, are notified of changes via webhooks, or have builds or deployments triggered via platforms like GitHub Actions. The IaC tooling is then executed, bringing the system in line with the desired state. Indeed, a real world end-to-end GitOps system inevitably incorporates orchestration outside of Kubernetes. For example, Kubernetes is unlikely to manage your DNS records, centralized authentication platforms, or messaging systems like Slack. You'll also likely find at least one managed service for things like databases, message queues, scheduling, and reporting more compelling than attempting to replicate them in a Kubernetes cluster. Also, any established IT department is guaranteed to have non-Kubernetes systems that would benefit from GitOps. So while the initial selection of specialized GitOps tools tends to be tightly integrated into Kubernetes, achieving the functional requirements of GitOps across established infrastructure will inevitably require orchestrating one or more IaC tools. Continuous Reconciliation Is Half the Battle Continuous reconciliation, as described by the working group, describes responses to two types of system changes. The first is what you expect, where deliberate changes to the configuration held in Git or other versioned storage is detected and applied to the system. This is the logical flow of configuration change and represents the normal operation of a correctly configured GitOps workflow. The second is where an agent detects undesirable changes to the system that are not described in the source configuration. In this case, your system no longer reflects the desired state, and the agent is expected to reconcile the system back to the configuration maintained in Git. This ability to resolve the second situation is a neat technical capability, but represents an incomplete business process. Imagine the security guards from your front desk reporting they had evicted an intruder. As a once-off occurrence, this report would be mildly concerning, but the security team did their job and resolved the issue. But now imagine you were receiving these reports every week. Obviously there is a more significant problem forcing the security team to respond to weekly intrusions. In the same manner, a system that continually removes undesirable system states is an incomplete solution to a more fundamental root problem. The real question is who is making those changes, why are the changes being made, and why are they not being made through the correct process? The fact your system can respond to undesirable states is evidence of a robust process able to adapt to unpredictable events, and this ability should not be underestimated. It's a long established best practice that teams should exercise their recovery processes, so in the event of disaster, teams are able to run through a well-rehearsed restoration. Continuous reconciliation can be viewed as a kind of automated restoration process, allowing the process to be tested and verified with ease. But if your system has to respond to undesirable states, it's evidence of a flawed process where people have access that they shouldn't or are not following established processes. An over-reliance on a system that can undo undesirable changes after they've been made runs the risk of masking a more significant underlying problem. GitOps Is Not a Complete Solution While GitOps describes many desirable traits of well-managed infrastructure and deployment processes, it's not a complete solution. In addition to the 4 functional requirements described by GitOps, a robust system must be: Verifiable - infrastructure and applications must be testable once they are deployed. Recoverable - teams must be able to recover from an undesirable state. Visible - the state of the infrastructure and the applications deployed to it must be surfaced in an easily consumed summary. Secure - rules must exist around who can make what changes to which systems. Measurable - meaningful metrics must be collected and exposed in an easily consumed format. Standardized - applications and infrastructure must be described in a consistent manner. Maintainable - support teams must be able to query and interact with the system, often in non-declarative ways. Coordinated - changes to applications and infrastructure must be coordinated between teams. GitOps offers little advice or insight into what happens before configuration is committed to a Git repo or other versioned and immutable storage, but it is "left of the repo" where the bulk of your engineering process will be defined. If your Git repo is the authoritative representation of your system, then anyone who can edit a repo essentially has administrative rights. However, Git repos don't provide a natural security boundary for the kind of nuanced segregation of responsibility you find in established infrastructure. This means you end up creating one repo per app per environment per role. Gaining visibility over each of these repos and ensuring they have the correct permissions is no trivial undertaking. You also quickly find that just because you can save anything in Git doesn't mean you should. It's not hard to imagine a rule that says development teams must create Kubernetes deployment resources instead of individual pods, use ingress rules that respond to very specific hostnames, and always include a standard security policy. This kind of standardization is tedious to enforce through pull requests, so a much better solution is to give teams standard resource templates that they populate with their specific configuration. But this is not a feature inherent to Git or GitOps. We then have those processes "right of the cluster," where management and support tasks are defined. Reporting on the intent of a Git commit is almost impossible. If you looked at a diff between two commits and saw that a deployment image tag was increased, new secret values were added, and a config map was deleted, how would you describe the intent of that change? The easy answer is to read the commit message, but this isn't a viable option for reporting tools that must map high level events like "deployed a new app version" or "bug fix release" (which are critical if you want to measure yourself against standard metrics like those presented in the DORA report) to the diff between two commits. Even if you could divine an algorithm that understood the intent of a Git commit, a Git repo was never meant to be used as a time-series database. GitOps also provides no guidance on how to perform support tasks after the system is in its desired state. What would you commit to a Git repo to delete misbehaving pods so they can be recreated by their parent deployment? Maybe a job could do this, but you have to be careful that Kubernetes doesn't try to apply that job resource twice. But then what would you commit to the repo to view the pod logs of a service like an ingress controller that was preinstalled on your cluster? My mind boggles at the thought of all the asynchronous message handling you would need to implement to recreate kubectl logs mypod in a GitOps model. Adhoc reporting and management tasks like this don't have a natural solution in the GitOps model. This is not to say that GitOps is flawed or incomplete, but rather that it solves specific problems, and must be complemented with other processes and tools to satisfy basic operational requirements. Git Is the Least Interesting Part of GitOps I'd like to present you with a theory and a thought experiment to apply it to: In any sufficiently complex GitOps process, your Git repo is just another structured database. You start your GitOps journey using the common combination of Git and Kubernetes. All changes are reviewed by pull request, committed to a Git repo, consumed by a tool like Argo CD or Flux, and deployed to your cluster. You have satisfied all the functional requirements of GitOps, and enjoy the benefits of a single source of truth, immutable change history, and continuous reconciliation. But it becomes tedious to have a person open a pull request to bump the image property in a deployment resource every time a new image is published. So you instruct your build server to pull the Git repo, edit the deployment resource YAML file, and commit the changes. You now have GitOps and CI/CD. You now need to measure the performance of your engineering teams. How often are new releases deployed to production? You quickly realize that extracting this information from Git commits is inefficient at best, and that the Kubernetes API was not designed for frequent and complex queries, so you choose to populate a more appropriate database with deployment events. As the complexity of your cluster grows, you find you need to implement standards regarding what kind of resources can be deployed. Engineering teams can only create deployments, secrets, and configmaps. The deployment resources must include resource limits, a set of standard labels, and the pods must not be privileged. In fact, it turns out that of the hundreds of lines of YAML that make up the resources deployed to the cluster, only about 10 should be customized. As you did with the image tag updates, you lift the editing of resources from manual Git commits to an automated process where templates have a strictly controlled subset of properties updated with each deployment. Now that your CI/CD is doing most of the commits to Git, you realize that you no longer need to use Git repos as a means of enforcing security rules. You consolidate the dozens of repos that were created to represent individual applications and environments to a single repo that only the CI/CD system interacts with on a day-to-day basis. You find yourself having to roll back a failed deployment, only to find that the notion of reverting a Git commit is too simplistic. The changes to the one application you wanted to revert have been mixed in with a dozen other deployments. Not that anyone should be touching the Git repo directly anyway, because merge conflicts can have catastrophic consequences. But you can use your CI/CD server to redeploy an old version of the application, and because the CI/CD server has the context of what makes up a single application, the redeployment only changes the files relating to that application. At this point, you concede that your Git repo is another structured database reflecting a subset of "the source of truth:" Humans aren't to touch it. All changes are made by automated tools. The automated tools require known files of specific formats in specific locations. The Git history shows a list of changes made by bots rather than people. The Git history now reads "Deployment #X.Y.Z", and other commit information only makes sense in the context of the automated tooling. Pull requests are no longer used. The "source of truth" is now found in the Git repo (showing changes to files), the CI/CD platform's history (showing the people who initiated the changes, and the scripts that made them), and the metrics database. You consolidated your Git repos, meaning you have limited ability to segregate access to humans even if you want to. You also realize that the parts of your GitOps process that are adding unique business value are "left of the repo" with metrics collection, standardized templates, release orchestration, rollbacks, and deployment automation; and "right of the cluster" with reports, dashboards, and support scripts. The process between the Git repo and cluster is now so automated and reliable that it's not something you need to think about. Conclusion GitOps has come to encapsulate a subset of desirable functional requirements that are likely to provide a great deal of benefit for any teams that fulfill them. While neither Git nor Kubernetes are required to satisfy GitOps, they are the logical platforms on which to start your GitOps journey, as they're well supported by the more mature GitOps tools available today. But GitOps tooling tends to be heavily focused on what happens between a commit to a Git repo and the Kubernetes cluster. While this is no doubt a critical component of any deployment pipeline, there's much work to be done "left of the repo" and "right of the cluster" to implement a robust CI/CD pipeline and DevOps workflow. GitOps tools also tend to assume that because everything is in Git, the intent of every change is annotated with commit messages, associated with the author, put through a review process, and is available for future inspection. However, this is overly simplistic, as any team advanced enough to consider implementing GitOps will immediately begin iterating on the process by automating manual touch points, usually with respect to how configuration is added to the Git repo in the first place. As you project the natural evolution of a GitOps workflow, you're likely to conclude that so many automated processes rely on the declarative configuration being in a specific location and format, that Git commits must be treated in much the same way as a database migration. The inputs to a GitOps process must be managed and orchestrated, and the outputs must be tested, measured, and maintained. Meanwhile the processing between the Git repo and cluster should be automated, rendering much of what we talk about as GitOps today as simply an intermediate step in a specialized CI/CD pipeline or DevOps workflow. Perhaps the biggest source of confusion around GitOps is the misconception that it represents an end-to-end solution, and that you implement GitOps and GitOps-focused tooling to the exclusion of alternative processes and platforms. In practice, GitOps encapsulates one step in your infrastructure and deployment pipelines, and must be complemented with other processes and platforms to fulfill common business requirements. Happy deployments!

As we continue to delve deeper into the digital age, the importance of data continues to grow. Businesses, scientists, and governments alike are gathering increasingly vast amounts of information, and these datasets require sophisticated tools for processing and analysis. Two such tools that have gained significant attention recently are Amazon's AWS Glue ETL and AWS Batch services. Both offer robust functionalities for managing, transforming, and analyzing data, but how do you decide which is the best fit for your specific needs? In this article, we will take a detailed look at both AWS Glue ETL and AWS Batch, comparing their features, capabilities, and use cases to help you make an informed decision. Overview Data transformation tools play a crucial role in data analysis. They help in converting raw data into useful information that can be used for decision-making. The process involves cleaning, normalizing, and transforming raw data to prepare it for analysis. AWS Glue ETL and AWS Batch are among the top data transformation tools that are designed to handle these tasks, offering different capabilities and strengths based on the specific requirements of your workload. AWS Glue ETL is a fully managed service that provides a serverless Apache Spark environment to run your Extract, Transform, Load (ETL) jobs. It is designed to prepare and load your data for analytics. Following is a reference architecture for Glue ETL. On the other hand, AWS Batch is a service that makes it easy to run batch computing workloads on the AWS Cloud. It is designed to simplify batch operations, such as scientific simulations, financial modeling, image or video processing, and machine learning workloads. Following is a reference architecture for AWS Batch running transformation jobs. Resource Management AWS Glue ETL creates and manages the compute resources in your AWS account, giving you full control and visibility into the resources being used. It also allows you to scale up or down the resources based on the demand of your ETL jobs. AWS Glue ETL automatically provisions and manages the infrastructure required to create ETL jobs, allowing you to focus on writing and tuning your ETL code rather than managing resources. Like AWS Glue ETL, AWS Batch also creates and manages the compute resources in your AWS account. It supports multi-node parallel processes, allowing you to run single jobs across several EC2 instances. AWS Batch dynamically provisions the optimal quantity and type of compute resources based on the volume and specific resource requirements of the batch jobs submitted, providing efficient resource utilization. Both AWS Glue ETL and AWS Batch provide robust resource management features. However, the choice between these two tools depends on the nature of your workload. If your workloads involve ETL jobs and require a serverless Apache Spark environment, AWS Glue ETL would be more suitable. Conversely, if your workloads involve batch computing jobs that can be broken down into smaller, discrete units of work, AWS Batch would be more efficient. Data Processing and Transformation AWS Glue ETL is built to handle complex data transformations. It provides a visual interface to create, run, and monitor ETL jobs with ease. You can use AWS Glue ETL to catalog your data, clean it, enrich it, and move it reliably between various data stores. Additionally, AWS Glue ETL supports both batch and streaming ETL jobs, allowing you to process and transform data as it arrives or in batches, depending on your business needs. AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs. It provides support for multi-node parallel processes, which allow you to run single jobs across several EC2 instances. This feature is particularly useful for tightly-coupled HPC workloads. AWS Batch also offers priority-based job scheduling, enabling you to create several queues with various priority levels for your jobs. While both AWS Glue ETL and AWS Batch offer robust data processing and transformation features, their efficiency depends on the specific needs of your workloads. AWS Glue ETL shines when dealing with ETL operations, while AWS Batch is more suited for running batch computing jobs. Therefore, understanding the nature of your workload is key to choosing the right tool. Error Handling and Debugging AWS Glue ETL provides comprehensive error handling and debugging capabilities. It logs all the events related to your ETL jobs and maintains these logs in CloudWatch, making it easier to debug issues. In case of errors during ETL operations, AWS Glue ETL reruns the jobs from the point of failure, ensuring that your jobs complete successfully. AWS Batch provides detailed logging for your batch jobs using Amazon CloudWatch Logs. It also allows for the re-queueing of failed jobs, enabling them to be retried automatically. AWS Batch also provides an API operation to cancel jobs, giving you control over your batch workloads. Both AWS Glue ETL and AWS Batch provide strong error handling and debugging features. The choice between the two again depends on the nature of your workload. If your workloads involve ETL operations, AWS Glue ETL's features, such as automated rerunning of failed jobs, could be more beneficial. If your workloads involve batch processing, AWS Batch's ability to re-queue failed jobs and cancel jobs via API could prove to be handy. Scheduling and Automation AWS Glue ETL provides robust scheduling and automation capabilities. You can schedule ETL jobs to run at specific times or in response to specific events. AWS Glue ETL also supports workflow orchestration, allowing you to design complex ETL workflows and automate their execution. AWS Batch offers priority-based job scheduling, enabling you to create multiple queues with various priority levels. This feature allows you to prioritize the execution of your batch jobs based on their importance. AWS Batch also supports job dependencies, allowing you to specify that certain jobs depend on the successful completion of others. This feature enables you to automate complex workflows involving multiple interdependent jobs. Both AWS Glue ETL and AWS Batch offer strong scheduling and automation capabilities. AWS Glue ETL provides more sophisticated workflow orchestration features, which can be beneficial if your workloads involve complex ETL operations. On the other hand, AWS Batch's support for job dependencies offers greater flexibility when dealing with interconnected batch jobs. On top of that, both Glue and Batch jobs can be scheduled externally using either Amazon Managed Workflow for Apache Airflow or Step Function providing greater flexibility. Pricing and Plans AWS Glue ETL pricing is primarily based on the number of Data Processing Units (DPUs) used by your ETL jobs. A DPU is a measure of processing capacity that consists of 4 vCPUs of compute capacity and 16 GB of memory. AWS Glue ETL charges you an hourly rate for each DPU hour used by your ETL jobs. In addition to DPU usage, AWS Glue ETL may also incur additional costs for data transfer and storage. AWS Batch pricing is primarily based on the cost of AWS resources (like EC2 instances or AWS Fargate) used to run your batch jobs. There are no additional charges for using AWS Batch. You only pay for the AWS resources needed to store and execute your batch jobs. Both AWS Glue ETL and AWS Batch offer excellent value for money, considering their robust features and capabilities. The choice between the two would primarily depend on your workload requirements and budget constraints. AWS Glue ETL might be a better choice if your workloads require complex ETL operations and you are willing to pay for the convenience of a fully managed service. On the other hand, AWS Batch could be a more cost-effective option for running large-scale batch jobs, as you only pay for the AWS resources you use. Integration Options AWS Glue ETL integrates seamlessly with various AWS services. It integrates with Amazon S3 for data storage, Amazon Redshift for data warehousing, and Amazon Athena for interactive query services. AWS Glue ETL also integrates with AWS Lake Formation, enabling you to build, secure, and manage data lakes with ease. AWS Batch also offers robust integration options with other AWS services. It integrates with Amazon EC2 for compute resources, Amazon S3 for data storage, and Amazon CloudWatch for monitoring and logging. Additionally, AWS Batch integrates with AWS Step Functions, allowing you to orchestrate complex workflows involving multiple AWS Batch jobs and other AWS services. Both AWS Glue ETL and AWS Batch provide seamless integration with other AWS services, making them ideal for building comprehensive data workflows on the AWS platform. The key difference lies in the specific services they integrate with. AWS Glue ETL's integration with data warehousing and query services like Amazon Redshift and Amazon Athena makes it particularly suitable for ETL workflows. Conversely, AWS Batch's integration with AWS Step Functions makes it ideal for orchestrating complex workflows involving multiple batch jobs and other AWS services. Customer Support and Documentation AWS offers extensive support and documentation for AWS Glue ETL. The AWS Glue ETL documentation provides detailed guides and tutorials to help you get started, understand key concepts, and learn best practices. In case of technical issues, you can reach out to AWS support through multiple channels, including forums, email, phone, and chat. AWS also provides professional services and training programs to help you make the most of AWS Glue ETL. AWS offers similar support options and resources for AWS Batch. The AWS Batch documentation provides comprehensive guides and tutorials covering all aspects of the service. For technical assistance, you can contact AWS support via forums, email, phone, and chat. AWS also offers professional services and training programs specifically tailored for AWS Batch. Conclusion In conclusion, both AWS Glue ETL and AWS Batch are powerful tools for data transformation, each with its unique strengths. AWS Glue ETL excels in handling ETL operations and integrates well with data warehousing and query services. AWS Batch, on the other hand, shines in running batch computing jobs and orchestrating complex workflows. Choosing the best data transformation tool depends on your specific needs and preferences. If you primarily deal with ETL operations and require a fully managed service that is simple, easy to use and integrates well with data warehousing and query services, AWS Glue ETL might be the optimal choice for you. On the other hand, if your workloads involve batch computing jobs needing more control over resource management, priority-based job scheduling, flexibility to run tightly-coupled HPC workloads, and the ability to orchestrate complex workflows, AWS Batch could be the ideal tool.

This article outlines a solution for streaming events from Kafka, forwarding them to Redis using its Stream API, and reading individual streams from Redis via its streaming API. The added complexity in this scenario is the need to stream events from an HTTP endpoint using Server-Sent Events (SSE) while ensuring that only events relevant to a specific client ID are processed and sent. Problem Statement Many companies have an existing Kafka infrastructure where events are being produced. Our goal is to set up a system that subscribes to Kafka messages but only processes events relevant to a specific client ID. These filtered events should be forwarded to Redis using its Stream API. Additionally, we need to establish an HTTP endpoint for Server-Sent Events (SSE) that allows the specified client to receive real-time event updates. Solution Architecture Overview The architecture consists of the following components: Kafka: A distributed event streaming platform that allows you to publish and subscribe to streams of records (events). Spring Boot: A framework for building Java applications. We'll use it to create Kafka consumers and Redis Stream producers. Subscribes to Kafka messages, filters events based on the client ID, and forwards relevant events to Redis Streams. Redis: A high-performance, in-memory data store. We'll use its Streams feature to handle event streams. Stores the streamed events using its Streams API. Docker: A containerization platform. We'll use Docker and Docker-Compose to create containers for Kafka, Redis, and our Spring Boot application. We will utilize this for a local POT, POC. HTTP Server-Sent Events (SSE) Endpoint: Provides real-time event updates to the client, filtering events based on the client ID. Redis Streams Redis Streams is a feature in Redis that provides a way to handle real-time data streams with various use cases. Here are some scenarios where you might want to use Redis Streams: Real-Time Event Processing: Redis Streams are excellent for processing and storing real-time events. You can use it for things like logging, monitoring, tracking user activities, or any use case that involves handling a continuous stream of events. Task Queues: If you need a reliable and distributed task queue, Redis Streams can be a great choice. It allows you to push tasks into a stream and have multiple consumers process those tasks concurrently. Activity Feeds: If you're building a social network or any application that requires activity feeds, Redis Streams can efficiently handle the feed data, ensuring fast access and scalability. Message Brokering: Redis Streams can serve as a lightweight message broker for microservices or other distributed systems. It can handle message routing and ensure that messages are delivered to interested consumers. Real-Time Analytics: When you need to analyze data in real-time, Redis Streams can be useful for storing the incoming data and then processing and aggregating it using Redis capabilities. IoT Data Ingestion: If you're dealing with data from Internet of Things (IoT) devices, Redis Streams can handle the high-throughput and real-time nature of the data generated by these devices. Logging and Audit Trails: Redis Streams can be used to store logs or audit trails in real-time, making it easy to analyze and troubleshoot issues. Stream Processing: If you need to process a continuous stream of data in a specific order (for example, financial transactions or sensor readings), Redis Streams can help you manage the data in the order it was received. Prerequisites Docker and Docker Compose are installed. Basic understanding of Spring Boot, Redis Streams, and Kafka. Java 17 or higher An HTTP client of your choice. I used [httpie] Of course, you can get the code here. Steps 1. Set Up Docker Compose for the Backend Infrastructure and the Spring Boot App Create a `docker-compose.yml` file to define the services: YAML version: '3.8' services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ports: - 22181:2181 networks: node_net: ipv4_address: 172.28.1.81 kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper ports: - 29092:29092 - 9092:9092 - 9093:9093 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092,,EXTERNAL://172.28.1.93:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 networks: node_net: ipv4_address: 172.28.1.93 cache: image: redis:6.2-alpine #image: redis:5.0.3-alpine restart: always ports: - '6379:6379' #command: redis-server --save 20 1 --loglevel warning --requirepass eYVX7EwVmmxKPCDmwMtyKVge8oLd2t81 command: redis-server /usr/local/etc/redis/redis.conf --loglevel verbose --save 20 1 volumes: - cache:/data - ./redis.conf:/usr/local/etc/redis/redis.conf - $PWD/redis-data:/var/lib/redis #environment: # - REDIS_REPLICATION_MODE=master networks: node_net: ipv4_address: 172.28.1.79 volumes: cache: driver: local networks: node_net: ipam: driver: default config: - subnet: 172.28.0.0/16 Create the yaml for the application 'sse-demo.yml'. YAML version: "3.8" services: sse-demo: image: "sse/spring-sse-demo:latest" ports: - "8080:8080" #- "51000-52000:51000-52000" env_file: - local.env environment: - REDIS_REPLICATION_MODE=master - SPRING_PROFILES_ACTIVE=default - REDIS_HOST=172.28.1.79 - REDIS_PORT=6379 - KAFKA_BOOTSTRAP_SERVERS=172.28.1.93:9093 networks: node_net: ipv4_address: 172.28.1.12 networks: node_net: external: name: docker_node_net 2. Create Spring Boot Application Create a Spring Boot application with the required dependencies: Shell git checkout https://github.com/glawson6/spring-sse-demo.git In `pom.xml,` add the necessary dependencies for Kafka and Redis integration. 3. Implement Kafka Consumer and Redis Stream Producer Create a Kafka consumer that listens to Kafka events and sends them to Redis Streams. This component also consumes the Redis streams for the HTTP clients: Java package com.taptech.sse.event; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; import com.taptech.sse.utils.DurationSupplier; import com.taptech.sse.utils.ObjectMapperFactory; import com.taptech.sse.config.SSEProperties; import jakarta.annotation.PostConstruct; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.boot.context.event.ApplicationStartedEvent; import org.springframework.context.event.EventListener; import org.springframework.data.redis.connection.stream.*; import org.springframework.data.redis.core.ReactiveStringRedisTemplate; import reactor.core.Disposable; import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; import reactor.kafka.receiver.KafkaReceiver; import reactor.kafka.receiver.ReceiverRecord; import reactor.util.function.Tuple2; import reactor.util.function.Tuples; import reactor.util.retry.Retry; import java.nio.ByteBuffer; import java.time.Duration; import java.time.LocalDateTime; import java.time.temporal.ChronoUnit; import java.util.function.Function; public class DefaultEventReceiverService implements EventReceiverService { private static final Logger logger = LoggerFactory.getLogger(DefaultEventReceiverService.class); public final static String TEST_STREAM = "test.stream"; public final static String TEST_STREAM_KEY = "test.stream.key"; public final static String TEST_STREAM_VALUE = "test.stream.value"; private static final String EMPTY_STR = ""; public static final String CLIENT_STREAM_STARTED = "client.stream.started"; public static final String HASH_PREFIX = "hash."; private static ObjectMapper objectMapper = ObjectMapperFactory.createObjectMapper(ObjectMapperFactory.Scope.SINGLETON); ReactiveStringRedisTemplate redisTemplate; KafkaReceiver<String, String> kafkaReceiver; SSEProperties sseProperties; StreamReadOptions streamReadOptions; public DefaultEventReceiverService(ReactiveStringRedisTemplate redisTemplate, KafkaReceiver<String, String> kafkaReceiver, SSEProperties sseProperties) { this.redisTemplate = redisTemplate; this.kafkaReceiver = kafkaReceiver; this.sseProperties = sseProperties; this.streamReadOptions = StreamReadOptions.empty().autoAcknowledge() .block(Duration.of(sseProperties.getClientHoldSeconds(), ChronoUnit.SECONDS)); } static final Function<String,String> calculateHashKey = str -> new StringBuilder(HASH_PREFIX).append(str).toString(); @PostConstruct public void init() { this.redisTemplate.opsForValue().append(TEST_STREAM_KEY, TEST_STREAM_VALUE).subscribe(); } @EventListener(ApplicationStartedEvent.class) public Disposable startKafkaConsumer() { logger.info("############# Starting Kafka listener....."); return kafkaReceiver.receive() .doOnError(error -> logger.error("Error receiving event, will retry", error)) .retryWhen(Retry.fixedDelay(Long.MAX_VALUE, Duration.ofSeconds(sseProperties.getTopicRetryDelaySeconds()))) .doOnNext(record -> logger.info("Received event: key {}", record.key())) .filterWhen(record -> checkIfStreamBeingAccessed(record)) .concatMap(this::handleEvent) .subscribe(record -> record.receiverOffset().acknowledge()); } Mono<Boolean> checkIfStreamBeingAccessed(ReceiverRecord<String,String> record){ return this.redisTemplate.opsForHash().hasKey(calculateHashKey.apply(record.key()), CLIENT_STREAM_STARTED) .doOnNext(val -> logger.info("key => {}'s stream is being accessed {}",record.key(),val)); } public Mono<ReceiverRecord<String, String>> handleEvent(ReceiverRecord<String, String> record) { return Mono.just(record) .flatMap(this::produce) .doOnError(ex -> logger.warn("Error processing event: key {}", record.key(), ex)) .onErrorResume(ex -> Mono.empty()) .doOnNext(rec -> logger.debug("Successfully processed event: key {}", record.key())) .then(Mono.just(record)); } public Mono<Tuple2<RecordId, ReceiverRecord<String, String>>> produce(ReceiverRecord<String, String> recRecord) { ObjectRecord<String, String> record = StreamRecords.newRecord() .ofObject(recRecord.value()) .withStreamKey(recRecord.key()); return this.redisTemplate.opsForStream().add(record) .map(recId -> Tuples.of(recId, recRecord)); } Function<ObjectRecord<String, String>, NotificationEvent> convertToNotificationEvent() { return (record) -> { NotificationEvent event = null; try { event = objectMapper.readValue(record.getValue(), NotificationEvent.class); } catch (JsonProcessingException e) { e.printStackTrace(); event = new NotificationEvent(); } return event; }; } private Mono<String> createGroup(String workspaceId){ return redisTemplate.getConnectionFactory().getReactiveConnection().streamCommands() .xGroupCreate(ByteBuffer.wrap(workspaceId.getBytes()), workspaceId, ReadOffset.from("0-0"), true) .doOnError((error) -> { if (logger.isDebugEnabled()){ logger.debug("Could not create group.",error); } }) .map(okStr -> workspaceId) .onErrorResume((error) -> Mono.just(workspaceId)); } private Flux<NotificationEvent> findClientNotificationEvents(Consumer consumer, StreamOffset<String> streamOffset, DurationSupplier booleanSupplier){ return this.redisTemplate.opsForStream().read(String.class, consumer, streamReadOptions, streamOffset) .map(convertToNotificationEvent()) .repeat(booleanSupplier); } public Flux<NotificationEvent> consume(final String clientId){ return Flux.from(createGroup(clientId)) .flatMap(id -> addIdToStream(clientId)) .map(id -> Tuples.of(StreamOffset.create(clientId, ReadOffset.lastConsumed()), Consumer.from(clientId, clientId), new DurationSupplier(Duration.of(sseProperties.getClientHoldSeconds(), ChronoUnit.SECONDS), LocalDateTime.now()))) .flatMap(tuple3 -> findClientNotificationEvents(tuple3.getT2(), tuple3.getT1(), tuple3.getT3())); } private Mono<String> addIdToStream(String id) { return this.redisTemplate.opsForHash().put(calculateHashKey.apply(id), CLIENT_STREAM_STARTED, Boolean.TRUE.toString()).map(val -> id); } public Flux<Boolean> deleteWorkspaceStream(String workspaceId){ StreamOffset<String> streamOffset = StreamOffset.create(workspaceId, ReadOffset.lastConsumed()); StreamReadOptions streamReadOptions = StreamReadOptions.empty().noack(); Consumer consumer = Consumer.from(workspaceId, workspaceId); return this.redisTemplate.opsForStream().read(String.class, consumer, streamReadOptions, streamOffset) .flatMap(objRecord -> this.redisTemplate.opsForStream().delete(workspaceId,objRecord.getId()).map(val -> objRecord)) .flatMap(objRecord -> this.redisTemplate.opsForHash().delete(workspaceId)); } @Override public Flux<String> consumeString(String clientId) { return this.redisTemplate.opsForStream().read(String.class, StreamOffset.latest(clientId)).map(ObjectRecord::getValue); } } 4. Configure Services in Spring Boot Properties files cache.provider.name=redis cache.host=${REDIS_HOST:localhost} cache.port=${REDIS_PORT:6379} cache.password=password # Producer properties spring.kafka.producer.bootstrap-servers=127.0.0.1:9092 spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.group-id=group_id spring.kafka.boostrap.servers=${KAFKA_BOOTSTRAP_SERVERS:loclahost:9093} # Common Kafka Properties auto.create.topics.enable=true sse.client-hold-seconds=${SSE_CLIENT_HOLD_SECONDS:120} logging.level.root=INFO logging.level.com.taptech.sse=DEBUG 5. Build Docker Image for Spring Boot App Using the `kubernetes-maven-plugin` from jkube, create the image for your Spring Boot application: Shell ./mvnw clean package -Dmaven.test.skip=true k8s:build 6. Start the Services and Run the Application From the src/test/resources/docker directory Start the services: Shell ./startServices.sh Start the app: Shell ./start-sse-demo.sh 7. Connect to a Stream Using One of the IDS in the client-ids.json File Shell http --stream GET http://localhost:8080/sse clientId==dd07bd51-1ab0-4e69-a0ff-f625fa9e7fc0 8. Generate Some Events You can do an HTTP POST to http://localhost:8080/sse/generateNE Shell http POST http://localhost:8080/sse/generateNE After this, watch as your HTTP client receives an event for the clientId that it is subscribed to. Discussion Why would you use Kafka and Redis? Does not Kafka offer this alone? Many companies have invested in Kafka as a backend message provider between their systems. Kafka in itself does not handle message selection very easily. Message selection is not a typical feature provided natively by Kafka for a couple of reasons: Data Size and Latency: Kafka is designed for high-throughput, low-latency message processing. Its architecture focuses on distributing messages to a large number of consumers quickly. Introducing message selection based on arbitrary conditions can slow down the overall processing and introduce latency, which goes against Kafka's primary design goals. Idempotency: Kafka relies on the concept of idempotent producers and consumers. This means that if a consumer or producer retries a message due to a failure, it should not result in duplicate processing. Introducing selective message retrieval would complicate this idempotency guarantee, potentially leading to unintended duplicate processing. Consumer Offset Tracking: Kafka maintains consumer offsets, allowing consumers to keep track of the last processed message. If message selection is introduced, offsets become less straightforward, as some messages might be skipped based on selection criteria. Decoupled Architecture: Kafka is designed to decouple producers from consumers. Producers are unaware of consumer behavior, and consumers can independently decide what messages they want to consume. Message selection would break this decoupling, as producers would need to know which messages to produce based on specific consumer needs. Consumer Flexibility: Kafka consumers can be highly flexible in terms of message processing. They can be designed to filter, transform, and aggregate messages based on their own criteria. Introducing message selection at the Kafka level would limit this flexibility and make the system less adaptable to changing consumer requirements. Scaling and Parallelism: Kafka's scalability and parallelism benefits come from the ability to distribute messages across multiple partitions and allow multiple consumers to process messages in parallel. Selective message retrieval would complicate this parallelism, making it harder to distribute work efficiently. While Kafka itself doesn't provide native message selection features, it's essential to design the consumers to handle message filtering and selection if needed. Consumers can be designed to filter and process messages based on specific criteria, ensuring that only relevant messages are processed within the consumer application. This approach allows Kafka to maintain its core design principles while still providing the flexibility needed for various message-processing scenarios. Kafka could not essentially solve the problem in an easy way, which lead to pushing the messages to another persistent space that could easily select based on known criteria. This requirement leads to the decision to use Redis and allow pushing messages directly to Redis. A decision was made to limit the events being pushed into Redis based on whether there was a client actually expecting a message. If there were no clients, then Kafka messages were being filtered out. Java .filterWhen(record -> checkIfStreamBeingAccessed(record)) The client registers the id so that the Kafka listener will push the to the Redis stream. Java .flatMap(id -> addIdToStream(clientId)) Conclusion By following the steps outlined in this document, we have successfully implemented an event streaming architecture that takes events from Kafka, filters them based on a specific client ID, and forwards the relevant events to Redis using its Stream API. The SSE endpoint allows clients to receive real-time event updates tailored to their respective client IDs. This solution provides an efficient and scalable way to handle event streaming for targeted clients.

MQTT is a lightweight messaging protocol commonly used in IoT (Internet of Things) applications to enable communication between devices. As a popular open-source MQTT broker, EMQX provides high scalability, reliability, and security for MQTT messaging. By using Terraform, a widespread Infrastructure as Code (IaC) tool, you can automate the deployment of EMQX MQTT Broker on GCP, making it easy to set up and manage your MQTT infrastructure. This blog will provide a step-by-step guide on how to set up a GCP project, create a service account, and write a Terraform configuration file to deploy EMQX MQTT Broker. Prerequisites Before you start, prepare the following: A Google Cloud Platform account The Google Cloud SDK installed on your local machine Terraform installed on your local machine A basic understanding of GCP, Terraform, and MQTT Set Up the GCP Environment Follow the steps below to set up the GCP environment: Create a new GCP project or use an existing one. Enable the required APIs (Compute Engine API) for your project. Create a service account for Terraform with the required permissions. A Compute Engine Admin role is recommended. Download the JSON key file Deploy EMQX on GCP Using Terraform Configure Terraform Configure the GCP provider in your Terraform code and authenticate using the service account key file. provider "google" { credentials = file("<PATH-TO-KEY-FILE>") project = "<PROJECT-ID>" region = "<REGION>" zone = "<ZONE>" } Configure Network This step requires an understanding of three essential terms related to GCP: project, VPC, and subnets. These terms are defined as follows: A project is a top-level organizational unit in GCP that contains all the resources. A VPC is a private network defined within a GCP project, allowing you to create and manage your IP addresses, subnets, and routing tables. Subnets are a way to divide a VPC network into smaller, more manageable parts. They can allocate IP addresses to specific resources and define different network segments. The relationship between them can be illustrated as below: Create a VPC Network We need to create a VPC network to provide connectivity for your network-related resources, including： Compute Engine virtual machine (VM) instances Container Engine containers App Engine Flex services Other network-related resources resource "google_compute_network" "vnet" { project = "<PROJECT>" name = "<NAME>" auto_create_subnetworks = false } Create a Subnet in VPC Each VPC network is subdivided into subnets, and we need a subnet. resource "google_compute_subnetwork" "sn" { name = "<NAME>" ip_cidr_range = cidrsubnet(var.address_space, 8, 1) region = var.region network = google_compute_network.vnet.id } Create a Firewall Rule Each network has its firewall controlling access to and from the instances. All traffic to instances, even from other instances, is blocked by the firewall unless firewall rules are created to allow it. The ports define some ports related to MQTT, for example, "1883", "8883", "8083," and "8084”. resource "google_compute_firewall" "fw" { name = "<NAME>" network = google_compute_network.vnet.name source_ranges = ["0.0.0.0/0"] allow { protocol = "icmp" } allow { protocol = "tcp" ports = "<PORTS>" } } Configure EMQX Cluster Provide a VM Instance for Each EMQX Node Virtual machine instances can deploy applications, run services, or perform computing tasks. In the following example, We create a google_compute_instance resource named example-instance, specifying the name, machine_type, boot_disk, and network_interface attributes. resource "google_compute_instance" "example" { name = "example-instance" machine_type = "n1-standard-1" boot_disk { initialize_params { image = ""ubuntu-os-cloud/ubuntu-2004-lts"" } } network_interface { network = google_compute_network.example.name subnetwork = google_compute_subnetwork.example.name access_config { // Ephemeral external IP } } } Initiate EMQX Nodes Initialize each EMQX node after the VM instance is created. First, you must initialize and copy the init.sh to each one. Then download the EMQX package and execute the init.sh you’ve copied at each node. Finally, start EMQX separately. resource "null_resource" "init" { depends_on = [google_compute_instance.example] count = "<INSTANCE-COUNT>" connection { type = "ssh" host = "<HOST-LIST>" user = "ubuntu" private_key = "<YOUR-PRIVATE-KEY>" } # config init script provisioner "file" { content = templatefile("${path.module}/scripts/init.sh", { local_ip = <PRIVATE-IPS>[count.index], emqx_lic = <EMQX-LICENSE>, emqx_ca = <EMQX-CA> emqx_cert = <EMQX-CERT>, emqx_key = <PRIVATE-KEY> }) destination = "/tmp/init.sh" } # download EMQX package provisioner "remote-exec" { inline = [ "curl -L --max-redirs -1 -o /tmp/emqx.zip <EMQX-PACKAGE-URL>" ] } # init system provisioner "remote-exec" { inline = [ "chmod +x /tmp/init.sh", "/tmp/init.sh", "sudo mv /tmp/emqx <HOME>", ] } # start EMQX provisioner "remote-exec" { inline = [ "sudo <HOME>/bin/emqx start" ] } } Join the EMQX Node To Make a Cluster Randomly select a node from the EMQX cluster, and join the other nodes individually. resource "null_resource" "emqx_cluster" { depends_on = [null_resource.init] count = "<INSTANCE-COUNT>-1" connection { type = "ssh" host = <OTHERS>[count.index % <OTHERS>] user = "ubuntu" private_key = "<YOUR-PRIVATE-KEY>" } provisioner "remote-exec" { inline = [ "/home/ubuntu/emqx/bin/emqx_ctl cluster join emqx@${local.another_emqx}" ] } } Configure Load Balance In this example: We create a google_compute_http_health_check resource to configure the health check settings. We create a google_compute_target_pool resource, which refers to the instance group and the health check. We create a google_compute_forwarding_rule resource, which sets the forwarding rule for incoming traffic on port 1883 to be routed to the target pool. We could add more google_compute_forwarding_rule` for ports "8883", "8083", "8084" and "18083" resource "google_compute_http_health_check" "example" { name = "example-health-check" check_interval_sec = 30 timeout_sec = 5 port = 8081 request_path = "/status" } resource "google_compute_target_pool" "example" { name = "example-target-pool" instances = [ google_compute_instance_group.example.self_link ] health_checks = [ google_compute_http_health_check.example.name ] } resource "google_compute_forwarding_rule" "example-1883" { name = "example-forwarding-rule" target = google_compute_target_pool.example.self_link port_range = "1883" ip_protocol = "TCP" } resource "google_compute_forwarding_rule" "example-8883" { ... } Initialize and Apply Terraform terraform init terraform plan terraform apply After applying successfully, it will output the following: Outputs: loadbalancer_ip = ${loadbalancer_ip} tls_ca = <sensitive> tls_cert = <sensitive> tls_key = <sensitive> You can access different services over corresponding ports. Dashboard: ${loadbalancer_ip}:18083 MQTT: ${loadbalancer_ip}:1883 MQTTS: ${loadbalancer_ip}:8883 WS: ${loadbalancer_public_ip}:8083 WSS: ${loadbalancer_public_ip}:8084 Conclusion Deploying EMQX on GCP using Terraform streamlines the management of your IoT infrastructure, allowing you to focus on building applications that leverage the power of connected devices. Following the steps outlined in this blog post, you can easily set up a scalable and reliable MQTT broker on GCP to support your IoT projects. Reference GitHub Repo.

In this tutorial, we explore the powerful combination of Hazelcast and Redpanda to build high-performance, scalable, and fault-tolerant applications that react to real-time data. Redpanda is a streaming data platform designed to handle high-throughput, real-time data streams. Compatible with Kafka APIs, Redpanda provides a highly performant and scalable alternative to Apache Kafka. Redpanda's unique architecture enables it to handle millions of messages per second while ensuring low latency, fault tolerance, and seamless scalability. Hazelcast is a unified real-time stream data platform that enables instant action on data in motion by uniquely combining stream processing and a fast data store for low-latency querying, aggregation, and stateful computation against event streams and traditional data sources. It allows you to build resource-efficient, real-time applications quickly. You can deploy it at any scale from small edge devices to a large cluster of cloud instances. In this post, we will guide you through setting up and integrating these two technologies to enable real-time data ingestion, processing, and analysis for robust streaming analytics. By the end, you will have a solid understanding of how to leverage the combined capabilities of Hazelcast and Redpanda to unlock the potential of streaming analytics and instant action for your applications. So, let's dive in and get started! Pizza in Motion: The Solution Architecture for a Pizza Delivery Service First, let’s understand what we are going to build. Most of us love pizza, so let’s use a pizza delivery service as an example. Our pizza delivery service receives orders from multiple users in real time. These orders contain a timestamp, user_id, pizza_type, and quantity. We’ll generate orders using Python, ingest them into Redpanda, then use Hazelcast to process them. But what if you want to enrich pizza orders with contextual data? For example, recommending specific starters for specific types of pizzas. How can you do this in real-time? There are actually multiple options, but for this blog post, we’ll show you how to use Hazelcast to enrich pizza orders coming from Redpanda with starters stored in iMap in Hazelcast. Here’s a quick diagram of what this solution looks like. Tutorial: Real-Time Stream Processing With Redpanda and Hazelcast Before diving in, let's make sure we have all the necessary prerequisites in place. You can download the demo from this GitHub repository. Setting up Redpanda For the scope of this tutorial, we will set up a Redpanda cluster with Docker Compose. So, make sure you have Docker Compose installed locally. Create the docker-compose.yml file in a location of your choice and add the following content to it. XML version: "3.7" name: redpanda-quickstart networks: redpanda_network: driver: bridge volumes: redpanda-0: null services: redpanda-0: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 # Address the broker advertises to clients that connect to the Kafka API. # Use the internal addresses to connect to the Redpanda brokers' # from inside the same Docker network. # Use the external addresses to connect to the Redpanda brokers' # from outside the Docker network. - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082 # Address the broker advertises to clients that connect to the HTTP Proxy. - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 # Redpanda brokers use the RPC API to communicate with eachother internally. - --rpc-addr redpanda-0:33145 - --advertise-rpc-addr redpanda-0:33145 # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system. - --smp 1 # The amount of memory to make available to Redpanda. - --memory 1G # Mode dev-container uses well-known configuration properties for development in containers. - --mode dev-container # enable logs for debugging. - --default-log-level=debug image: docker.redpanda.com/redpandadata/redpanda:v23.1.11 container_name: redpanda-0 volumes: - redpanda-0:/var/lib/redpanda/data networks: - redpanda_network ports: - 18081:18081 - 18082:18082 - 19092:19092 - 19644:9644 console: container_name: redpanda-console image: docker.redpanda.com/redpandadata/console:v2.2.4 networks: - redpanda_network entrypoint: /bin/sh command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console' environment: CONFIG_FILEPATH: /tmp/config.yml CONSOLE_CONFIG_FILE: | kafka: brokers: ["redpanda-0:9092"] schemaRegistry: enabled: true urls: ["http://redpanda-0:8081"] redpanda: adminApi: enabled: true urls: ["http://redpanda-0:9644"] ports: - 8080:8080 depends_on: - redpanda-0 The above file contains the configuration necessary to spin up a Redpanda cluster with a single broker. If needed, you can use a three-broker cluster. But, a single broker would be more than enough for our use case. Please note that using Redpanda on Docker is only recommended for development and testing purposes. For other deployment options, consider Linux or Kubernetes. To generate the data, we use a Python script: import asyncio import json import os import random from datetime import datetime from kafka import KafkaProducer from kafka.admin import KafkaAdminClient, NewTopic BOOTSTRAP_SERVERS = ( "localhost:19092" if os.getenv("RUNTIME_ENVIRONMENT") == "DOCKER" else "localhost:19092" ) PIZZASTREAM_TOPIC = "pizzastream" PIZZASTREAM_TYPES = [ "Margherita", "Hawaiian", "Veggie", "Meat", "Pepperoni", "Buffalo", "Supreme", "Chicken", ] async def generate_pizza(user_id): producer = KafkaProducer(bootstrap_servers=BOOTSTRAP_SERVERS) while True: data = { "timestamp_": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "pizza": random.choice(PIZZASTREAM_TYPES), "user_id": user_id, "quantity": random.randint(1, 10), } producer.send( PIZZASTREAM_TOPIC, key=user_id.encode("utf-8"), value=json.dumps(data).encode("utf-8"), ) print( f"Sent a pizza stream event data to Redpanda: {data}" ) await asyncio.sleep(random.randint(1, 5)) async def main(): tasks = [ generate_pizza(user_id) for user_id in [f"user_{i}" for i in range(10)] ] await asyncio.gather(*tasks) if __name__ == "__main__": # Create kafka topics if running in Docker. if os.getenv("RUNTIME_ENVIRONMENT") == "DOCKER": admin_client = KafkaAdminClient( bootstrap_servers=BOOTSTRAP_SERVERS, client_id="pizzastream-producer" ) # Check if topics already exist first existing_topics = admin_client.list_topics() for topic in [PIZZASTREAM_TOPIC]: if topic not in existing_topics: admin_client.create_topics( [NewTopic(topic, num_partitions=1, replication_factor=1)] ) asyncio.run(main()) Setting up Hazelcast Start a Hazelcast local cluster. This will run a Hazelcast cluster in client/server mode and an instance of Management Center running on your local network. brew tap hazelcast/hz brew install hazelcast@5.3.1 hz -V Now that we understand what we are going to build, and have prerequisites set up, let’s jump right into the solution. Step 1: Start the Redpanda Cluster Let’s start the Redpanda cluster by running the following command in a terminal. Make sure you are in the same location where you saved the docker-compose.yml file. docker compose up -d An output similar to the following confirms that the Redpanda cluster is up and running. [+] Running 4/4 ⠿ Network redpanda_network Created 0.0s ⠿ Volume "redpanda-quickstart_redpanda-0" Created 0.0s ⠿ Container redpanda-0 Started 0.3s ⠿ Container redpanda-console Started 0.6s Step 2: Run Hazelcast You can run the following command to start a Hazelcast cluster with one node. hz start To add more members to your cluster, open another terminal window and rerun the start command. Step 3: Run SQL on Hazelcast We will use the SQL shell—the easiest way to run SQL queries on a cluster. You can use SQL to query data in maps and Kafka topics. The results can be sent directly to the client or inserted into maps or Kafka topics. You can also use Kafka Connector which allows you to stream, filter, and transform events between Hazelcast clusters and Kafka. You can do so by running the following command: bin/hz-cli sql Step 4: Ingest Into Hazelcast iMap (pizzastream) Using the SQL command, we create pizzastream Map: CREATE OR REPLACE MAPPING pizzastream( timestamp_ TIMESTAMP, pizza VARCHAR, user_id VARCHAR, quantity DOUBLE ) TYPE Kafka OPTIONS ( 'keyFormat' = 'varchar', 'valueFormat' = 'json-flat', 'auto.offset.reset' = 'earliest', 'bootstrap.servers' = 'localhost:19092'); Step 5: Enrich the Stream With Recommendations Data (recommender) For this step, we create another Map: CREATE or REPLACE MAPPING recommender ( __key BIGINT, user_id VARCHAR, extra1 VARCHAR, extra2 VARCHAR, extra3 VARCHAR ) TYPE IMap OPTIONS ( 'keyFormat'='bigint', 'valueFormat'='json-flat'); We add some values into the Map: INSERT INTO recommender VALUES (1, 'user_1', 'Soup','Onion_rings','Coleslaw'), (2, 'user_2', 'Salad', 'Coleslaw', 'Soup'), (3, 'user_3', 'Zucchini_fries','Salad', 'Coleslaw'), (4, 'user_4', 'Onion_rings','Soup', 'Jalapeno_poppers'), (5, 'user_5', 'Zucchini_fries', 'Salad', 'Coleslaw'), (6, 'user_6', 'Soup', 'Zucchini_fries', 'Coleslaw'), (7, 'user_7', 'Onion_rings', 'Soup', 'Jalapeno_poppers'), (8, 'user_8', 'Jalapeno_poppers', 'Coleslaw', 'Zucchini_fries'), (9, 'user_9', 'Onion_rings','Jalapeno_poppers','Soup'); Step 6: Combine Both Maps Using SQL Based on the above two Maps, we send the following SQL query: SELECT pizzastream.user_id AS user_id, recommender.extra1 as extra1, recommender.extra2 as extra2, recommender.extra3 as extra3, pizzastream.pizza AS pizza FROM pizzastream JOIN recommender ON recommender.user_id = recommender.user_id AND recommender.extra2 = 'Soup'; Step 7: Send the Combined Data Stream to Redpanda To send the results back to Redpanda, we create a Jet job in Hazelcast that stores the SQL query results into a new Map, then into Redpanda: CREATE OR REPLACE MAPPING recommender_pizzastream( timestamp_ TIMESTAMP, user_id VARCHAR, extra1 VARCHAR, extra2 VARCHAR, extra3 VARCHAR, pizza VARCHAR ) TYPE Kafka OPTIONS ( 'keyFormat' = 'int', 'valueFormat' = 'json-flat', 'auto.offset.rest' = 'earliest', 'bootstrap.servers' = 'localhost:19092' ); CREATE JOB recommender_job AS SINK INTO recommender_pizzastream SELECT pizzastream.timestamp_ as timestamp_, pizzastream.user_id AS user_id, recommender.extra1 as extra1, recommender.extra2 as extra2, recommender.extra3 as extra3, pizzastream.pizza AS pizza FROM pizzastream JOIN recommender ON recommender.user_id = recommender.user_id AND recommender.extra2 = 'Soup'; Conclusion In this post, we explained how to build a pizza delivery service with Redpanda and Hazelcast. Redpanda adds value by ingesting pizza orders as high-throughput streams, storing them reliably, and allowing Hazelcast to consume them in a scalable manner. Once consumed, Hazelcast enriches pizza orders with contextual data (recommending starters to users instantly) and sends enriched data back to Redpanda. Hazelcast allows you to quickly build resource-efficient, real-time applications. You can deploy it at any scale, from small-edge devices to a large cluster of cloud instances. A cluster of Hazelcast nodes shares the data storage and computational load, which can dynamically scale up and down. Referring back to the pizza example, that means that this solution is reliable, even when there are significantly higher volumes of users ordering pizzas, like after a Superbowl ad. We look forward to your feedback and comments about this blog post! Share your experience with us in the Hazelcast GitHub repository. Hazelcast also runs a weekly live stream on Twitch, so give us a follow to get notified when we go live. To start exploring Redpanda, download the Redpanda Community Edition on GitHub. See you there!

Ingress means the act of going in or entering. Also, a means or place of entry or entryway. That’s the job of a Kubernetes Ingress Resource. The primitive approach of exposing a service to the outside world involves a Kubernetes NodePort service to the outside world. For reference, a Node Port service is a special type of service in Kubernetes. For this service type, each cluster node opens a port on the node itself. Any incoming traffic received on that port is directed to the underlying service and the associated pods. But what makes the Kubernetes Ingress resource special? That’s because Ingress does so much heavy lifting in terms of features such as: Load balancing SSL termination Name-based hosting Operating at the application layer of the network Support for multiple services with a single IP address Here’s the big-picture view of the Kubernetes Ingress resource. As you can see, one Ingress resource can act as the gatekeeper for multiple services. Each of the services can be backed by multiple pods running on their own nodes. For a client, none of these details matters as it will only be communicating to the Ingress resource. Let’s look at setting one up as a demo and see it in action. 1. The Role of the Ingress Controller The Ingress resource doesn’t work on its own. You need an Ingress controller running within your cluster. Think of this controller as the brains behind the whole Ingress magic. Now, the Ingress controller isn’t a straightforward matter, either. Different Kubernetes environments provided by vendors use different implementations of the controller. Some don’t even provide a default controller at all. Anyways, that’s not the scope of this post. For our demo purpose, you can check whether an Ingress controller is already present by checking all the pods within the cluster. Shell $ kubectl get po --all-namespaces I’d be looking for something like this: Shell ingress-nginx ingress-nginx-controller-555596df87-4p46d 1/1 Running 1 (159m ago) 2d2h If nothing is there, don’t fret. You can always install it as an add-on. Here’s the command to install the ingress-nginx controller. It’s a particular implementation of the Ingress controller that works well in most cases. Shell $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.0/deploy/static/provider/cloud/deploy.yaml When you execute the above command, a bunch of resources get created. Shell namespace/ingress-nginx created serviceaccount/ingress-nginx created serviceaccount/ingress-nginx-admission created role.rbac.authorization.k8s.io/ingress-nginx created role.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrole.rbac.authorization.k8s.io/ingress-nginx created clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created rolebinding.rbac.authorization.k8s.io/ingress-nginx created rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created configmap/ingress-nginx-controller created service/ingress-nginx-controller created service/ingress-nginx-controller-admission created deployment.apps/ingress-nginx-controller created job.batch/ingress-nginx-admission-create created job.batch/ingress-nginx-admission-patch created ingressclass.networking.k8s.io/nginx created validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created Wow, that’s a lot of things to make Ingress work. But I promise it’ll be worth it. Check for the ingress-nginx pod again and give a moment for the necessary pod to start running. With the setup out of the way, it’s time to create the Ingress resource. 2. Creating the Kubernetes Ingress Resource Below is the YAML manifest for a brand-new Ingress resource: YAML apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-demo annotations: nginx.ingress.kubernetes.io/rewrite-target: / kubernetes.io/ingress.class: "nginx" spec: rules: - host: kubernetes.docker.internal http: paths: - path: /demo pathType: Prefix backend: service: name: nodeport-demo port: number: 80 I hope you want to understand what’s going on over here. Here’s the breakdown: The apiVersion and kind are pretty self-explanatory as we are trying to tell Kubernetes what type of resource we want to create. Next up, there is the metadata section. It has a name field for the Ingress resource. Then, we have the annotations section. The annotation nginx.ingress.kubernetes.io/rewrite-target specifies the Target URI where the incoming traffic must be redirected. It’s important property so don’t miss it. The next one i.e. kubernetes.io/ingress.class is meant to link the Ingress resource with the Ingress controller. Hence the value ‘nginx’. Moving on, we have the spec section. It’s the place where the Ingress magic happens as we specify the rules that will govern the routing. Within the rules section, you have the host. I have used kubernetes.docker.internal as this is something that’s available out of the box. Within the host, we have the http section that contains a list of paths. For each path, you need to specify the path value, its type, and the corresponding backend service name. In the above example, I’m pointing /demo path to a service named nodeport-demo available on port 80. And that’s basically all that is needed. Note that if you use any other host such as demo.example.com, you need to make appropriate changes to the DNS so that it resolves the domain name to the IP of the Ingress controller. If you are trying this out locally on something like Docker Desktop, you can directly use kubernetes.docker.internal as the hostname. An important point to remember is that we can also use the same Ingress to expose multiple services. Also, in case you are looking for the definition of the Node Port service, here’s the YAML for that as well. YAML apiVersion: v1 kind: Service metadata: name: nodeport-demo spec: type: NodePort ports: - port: 80 targetPort: 3000 nodePort: 30100 selector: app: hello-service Once you have applied the resources and made changes to the DNS if needed, you can actually see the Ingress in action. You can go to your browser and visit the URL http://kubernetes.docker.internal/demo and if there is a proper backing application, you’ll see the response. 3. How the Kubernetes Ingress Actually Works Though things may be working fine, it’s also important to understand how the wheel actually turns. And there are a few interesting things about how the Kubernetes Ingress actually works. Here’s an illustration showing the same. The below steps can help you figure it out: The client first performs a DNS lookup of the hostname from the DNS server and gets the IP address of the Ingress controller. Then, the client sends an HTTP request to the Ingress controller with the hostname in the Host header. The controller determines the correct service based on the hostname, checks the Kubernetes Endpoints object for the service, and forwards the client’s request to one of the pods. Note that the Ingress controller doesn’t forward the request to the service. It only uses the service to select a particular pod. Conclusion That’s all for this post. But don’t think that Kubernetes Ingress is done and dusted. There are a lot of other use cases such as exposing multiple services or enabling TLS support.

Kubernetes autoscaling quickly gets tricky, but developers can save time and effort thanks to all the ecosystem's tools that make configuration and management easier. One of them is Helm. Helm charts are there to help teams define, install, and upgrade complex Kubernetes applications. And the Cluster Autoscaler Helm Chart does some of the heavy lifting for you around autoscaling. But how exactly does this Helm chart work? And are there any alternatives you could use to make cluster autoscaling even easier? Let's Start With the Basics: What Is Cluster Autoscaler Anyway? Along with Kubernetes Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler, Cluster Autoscaler is one of the autoscaling mechanisms K8s provides. Its goal is pretty simple: Cluster Autoscaler changes the number of nodes (worker machines) in a cluster. Note that Cluster Autoscaler can only manage nodes on a handful of supported platforms. And every platform has its own specific requirements or limitations. The tricky part here is that the autoscaler controller operates at the infrastructure level. To do its job, it needs permission to add and delete virtual machines (if you're in the cloud). So before you set Cluster Autoscaler to work, ensure airtight security for these credentials. Following the principle of least privilege is definitely a good idea (I'll share some more Cluster Autoscaler best practices later on). When Should You Use Cluster Autoscaler? There's no denying that a well-configured Cluster Autoscaler can make a massive impact on your cloud bill. Three amazing things happen when you're able to dynamically scale the number of nodes to match the current level of utilization: Minimize cloud waste, Maximize your ROI from every dollar you spend on cloud services, And, at the same time, you make sure there is no downtime as your application scales. That isn't to say that Cluster Autoscaler doesn't have its limitations. How Does Cluster Autoscaler Work? Cluster Autoscaler simply loops through two tasks: It checks for unschedulable pods (pods that don't have a node to run on), And it calculates whether consolidating all the currently deployed pods on a smaller number of nodes is possible or not. Here's how Cluster Autoscaler works, step by step: 1. It Scans Clusters To Identify Any Pods That Can’t Be Scheduled on Any Existing Nodes Where do unschedulable pods come from? The issue might arise because of inadequate CPU or memory resources or due to the pod's node taint tolerations or affinity rules failing to match an existing node. In addition to that, it could be that you have just scaled your application, and the new pods haven't found a node to run on yet. Step 2: Cluster Autoscaler Detects a Cluster That Contains Unschedulable Pods Next, it checks the managed node pools of this cluster to understand whether adding a node would let the pod run. If this is true, the autoscaler adds a node to the node pool. Step 3: Cluster Autoscaler Also Scans Nodes Across the Node Pools It Manages If it detects a node with pods that could be rescheduled to other nodes in the cluster, the autoscaler evicts the pods, moves them to an existing node, and finally removes the spare node. Note: When deciding to move a pod, Cluster Autoscaler considers pod priority and PodDisruptionBudgets. If you’re looking for more guidance on how to configure and run Cluster Autoscaler, here’s a hands-on guide to EKS Cluster Autoscaler with code snippets. Why Use the Cluster Autoscaler Helm Chart? Let's start with Helm. Helm is a package manager that has the sole purpose of making Kubernetes application deployment easier. Helm does that via charts. There are two ways you can use it: you can create your own chart or reuse existing Helm charts. Once you start developing a lot of tools in your K8s cluster, you'll be thankful for Helm charts. But Helm doesn't only help you with deployment. You can also manage releases, including rollbacks, if something goes south in your deployment. Let's get back to the original topic of this article: The Cluster Autoscaler Helm Chart is a configuration template that allows you to deploy and manage the Cluster Autoscaler component in Kubernetes using Helm (here's the repo). But Cluster Autoscaler Comes With Limitations We've already started talking about maximizing utilization and minimizing cloud costs. The problem with Cluster Autoscaler is that it doesn’t consider CPU or memory utilization in its decision-making process. All it does is check the pod’s limits and requests. What does this mean in dollar terms? That the Autoscaler isn't able to see all the unutilized capacity requested by pods. As a result, your cluster will end up being wasteful, and your utilization efficiency will be low. Another issue with Cluster Autoscaler is that spinning up a new node takes more time than the autoscaler gives you, as it issues a request for scaling up within a minute. This delay might easily cause performance issues while your pods are waiting for capacity. Finally, Cluster Autoscaler doesn't take into account the changing costs of instance types, making the decisions less cost-optimized. Is There an Alternative to the Cluster Autoscaler Helm Chart? Certain cloud cost management tools feature autoscalers that select the best matching instance types autonomously or according to your preferences, which can be simply configured. However, it's important to note that not all tools necessarily come with this feature. These specific platforms continuously track cloud provider inventory pricing and availability in supported cloud provider regions and zones. This information is utilized to select instance families that provide the most value. Implementing the Cluster Autoscaler Helm Charts, when available, is more straightforward than doing all the work independently. Since these particular services are fully managed, you won't have to devote any time thinking about upgrades, scalability, and availability. Furthermore, they often come with Helm charts. Here's an example that illustrates how closely these select autoscaling tools follow the actual resource requests in the cluster. Such cloud management tools are compatible with Amazon EKS, Kops, and OpenShift, as well as GKE and AKS. If you're in the market for dependable autoscalers that save you money without causing any performance issues, be sure to delve into the documentation of the specific tools that offer these capabilities.

Infrastructure as Code (IaC) has emerged as a pivotal practice in modern software development, enabling teams to manage infrastructure resources efficiently and consistently through code. This analysis provides an overview of Infrastructure as Code and its significance in cloud computing and DevOps. In recent years, Terraform has dominated the Infrastructure as Code domain, driven by its multi-cloud support, declarative syntax, robust resource providers, and active community and state management capabilities. Organizations are encouraged to leverage the strengths of Terraform while remaining aware of emerging IaC solutions tailored to their specific requirements and cloud preferences. Overview of Infrastructure as Code In the traditional approach to managing IT infrastructure, manual processes, and configuration management tools were used to provision, configure, and manage infrastructure components. This manual approach often led to inefficiencies, human errors, and inconsistencies across different environments. Infrastructure as Code (IaC) emerged as a solution to address these challenges and bring automation, scalability, and consistency to infrastructure management. IaC refers to defining and managing infrastructure resources through machine-readable configuration files rather than manual processes. It treats infrastructure components such as servers, networks, and storage as code, applying software development principles to infrastructure management. With IaC, infrastructure can be provisioned, configured, and managed programmatically, leveraging the benefits of version control, automation, and collaboration. IaC enables rapid provisioning and deployment of infrastructure resources. Infrastructure configurations can be defined and deployed in minutes or even seconds, compared to manual provisioning processes that could take days or weeks. This agility allows organizations to respond quickly to changing business requirements and deliver new services and features faster. IaC promotes consistency and standardization in infrastructure deployments. Infrastructure configurations are defined in code, eliminating human errors and ensuring that environments are replicated accurately. Consistent infrastructure setups across development, testing, and production environments reduce configuration drift and improve stability. IaC facilitates scaling infrastructure resources based on demand. With programmable infrastructure, it becomes easier to dynamically provision or de-provision resources, ensuring optimal utilization and cost-efficiency. Automated scaling based on defined policies and triggers enables organizations to handle varying workloads efficiently. Infrastructure configurations written as code can be versioned, providing the ability to track changes, roll back to previous versions, and audit configurations. This reproducibility helps troubleshoot issues, ensure compliance, and maintain an auditable history of infrastructure changes. Infrastructure configurations stored as code promote collaboration and knowledge sharing among teams. Code repositories and version control systems allow multiple team members to work on infrastructure configurations concurrently, enabling better collaboration, peer review, and knowledge transfer. Infrastructure code also serves as a form of documentation, providing insights into the architecture and design choices. IaC plays a crucial role in implementing DevOps practices and automating infrastructure management. Organizations can integrate infrastructure provisioning, configuration, and testing into their continuous integration and continuous deployment (CI/CD) pipelines. Infrastructure changes can be tested, validated, and deployed automatically, reducing manual effort and ensuring reliability. IaC abstracts infrastructure resources from underlying providers and platforms. Abstraction allows organizations to adopt a multi-cloud or hybrid cloud strategy, managing infrastructure consistently across different providers. IaC also provides flexibility in transitioning between cloud providers or migrating on-premises infrastructure to the cloud, making infrastructure more portable. Infrastructure as Code revolutionizes IT infrastructure management by bringing automation, consistency, scalability, and agility to provision and manage infrastructure resources. It enables organizations to adopt modern software development practices, improve operational efficiency, and accelerate the delivery of applications and services. Terraform: The Frontrunner of IaC Tools Terraform is a widely adopted and widespread Infrastructure as Code (IaC) tool developed by HashiCorp. It provides a declarative way to provision, manage, and version infrastructure resources across various cloud providers, on-premises environments, and other service providers. With Terraform, infrastructure is defined using a simple and human-readable configuration language called HashiCorp Configuration Language (HCL). The configuration files describe the desired state of the infrastructure, including resources such as virtual machines, networks, databases, load balancers, and more. Terraform operates on the infrastructure concept as a graph, analyzing resource dependency and relationships. It intelligently plans and applies infrastructure changes, considering the desired state and existing resources. One of the significant advantages of Terraform is its ability to support multiple cloud providers and services. It offers a comprehensive collection of provider plugins allowing users to interact with various cloud platforms, including Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). Multi-cloud support makes Terraform a flexible choice for managing infrastructure in diverse environments. Terraform also emphasizes the principle of idempotency, ensuring that applying the same configuration multiple times results in a consistent state. It lets users preview changes before applying them through the "plan" command, enabling better control and validation of infrastructure modifications. The tool maintains a state file that tracks the current state of infrastructure, serving as a source of truth for Terraform. By leveraging remote state backends such as Terraform Cloud, AWS S3, or HashiCorp Consul, teams can securely store and access the state file, enabling collaboration and versioning. Terraform benefits from a vibrant and active community that contributes to its ecosystem. Users can leverage existing, reusable configurations for common infrastructure patterns or develop custom modules to encapsulate and share infrastructure configurations. Key Features of Terraform Terraform, developed by HashiCorp, is a robust Infrastructure as Code (IaC) tool that allows users to define, manage, and provision infrastructure resources declaratively through code. Declarative configuration: Terraform uses a declarative approach, where users define the desired state of their infrastructure in configuration files. Instead of specifying the sequence of steps to achieve the desired state, users describe what they want the infrastructure to look like. Terraform then provides, updates, and destroys resources to reach that state. Infrastructure graph and dependency management: Terraform builds a dependency graph based on the declared configuration to understand and manage the relationships between infrastructure resources. The graph allows Terraform to determine the correct order for provisioning resources and ensures that dependencies are correctly resolved. Multi-cloud support and resource abstraction: Terraform supports multiple cloud providers, including AWS, Azure, Google Cloud, and others. It provides a consistent way to manage resources across different cloud platforms using the same Terraform configuration. Terraform abstracts the underlying cloud provider APIs, enabling users to work with resources in a cloud-agnostic manner. Resource providers: Terraform offers an extensive set of resource providers that allow users to manage various types of resources offered by cloud providers. Each provider includes resource types and associated attributes that users can configure in their Terraform configuration files. Plan and preview: Terraform provides a "plan" command that allows users to preview the changes before applying them. This feature shows Terraform's actions to achieve the desired state, such as creating, updating, or deleting resources. The plan helps users understand the potential impact of their changes and identify any issues before making modifications. State management: Terraform maintains a state file that records the current state of the deployed infrastructure. The state file serves as a source of truth for Terraform to understand the existing resources and track changes over time. It helps Terraform perform updates and manage resources efficiently without affecting unrelated components. Modular configuration: Terraform supports modularization, allowing users to break down their configuration into reusable modules. It enables code reusability and helps organize and abstract complex infrastructure setups, making configurations more maintainable and scalable. Provisioners and external data: Terraform provides provisioners which allow users to run scripts or commands on newly created resources after provisioning. Additionally, Terraform can fetch data from external sources, like APIs or other systems, and use that data in the configuration. Extensibility: Terraform's architecture allows developers to create custom plugins and extensions, providing the flexibility to integrate with other tools and extend its functionality as needed. All these essential features make Terraform a preferred choice for automating infrastructure management, as it provides a robust and user-friendly way to define and manage infrastructure as code across various cloud platforms. Terraform's Community and Ecosystem Terraform's community and ecosystem are crucial in its widespread adoption and success as an Infrastructure as Code (IaC) tool. The strong community support and thriving ecosystem contribute to the tool's continuous improvement, knowledge sharing, and availability of pre-built solutions. Terraform boasts an active and engaged community of developers, operators, and cloud enthusiasts worldwide. This community actively participates in forums, mailing lists, and social media platforms, discussing best practices, troubleshooting issues, and sharing knowledge and experiences related to Terraform. Its open-source nature encourages community collaboration and contribution. Users frequently contribute to the project by submitting bug reports, feature requests, and pull requests. Collaborative effort drives regular updates and improvements to the tool. The Terraform Registry is a central repository that hosts reusable Terraform modules, providers, and other extensions contributed by the community. It allows users to find and leverage pre-built modules for standard infrastructure components, saving time and effort during the configuration process. In addition to the official modules maintained by HashiCorp, numerous community-maintained modules are available on the Terraform Registry. These modules cover many use cases, cloud providers, and configurations, providing users with many options. The Terraform community creates and shares various learning resources, including tutorials, blog posts, videos, and online courses. Resources cater to users of all skill levels, helping newcomers get started and experienced users explore advanced concepts. Terraform users often organize local meetups and attend conferences dedicated to cloud computing, DevOps, and IaC. Such events offer networking opportunities, sharing insights, and learning from industry experts and practitioners. The Terraform ecosystem integrates with many other tools and platforms, such as continuous integration/continuous deployment (CI/CD) systems, version control systems, cloud management platforms, and monitoring tools. Integration enhances the tool's capabilities and fits seamlessly into existing workflows. Terraform's provider model allows users to extend its functionality to support custom resources or integrate with niche cloud providers not officially supported. The community has developed custom providers for various specialized use cases. Terraform Enterprise (formerly known as Terraform Cloud) is an enterprise-grade platform that provides collaboration, governance, and security features for organizations using Terraform at scale. It offers additional benefits for teams working on larger infrastructure deployments. Terraform's vibrant community and ecosystem foster collaboration, knowledge sharing, and innovation. The availability of pre-built modules, learning resources, and integrations empowers users to maximize the benefits of Infrastructure as Code while benefiting from the collective expertise of the Terraform community. Terraform Security Terraform provides various security features and best practices to ensure the safety and integrity of your infrastructure code and the resources it provisions. Some critical security considerations and features provided by Terraform: Authentication and authorization: Terraform integrates with cloud providers' authentication mechanisms, such as AWS IAM, Azure AD, and Google Cloud IAM, to ensure that only authorized users and services can access and manage resources. State encryption: Terraform can encrypt the state file to protect sensitive information stored in the state, such as resource IDs and secrets, which prevents unauthorized access to critical data. Secure communication: Terraform uses secure communication channels like HTTPS when interacting with cloud providers' APIs and services. Provider security: Terraform's providers (e.g., AWS, Azure, etc.) adhere to industry best practices and security standards to protect resources and data. Secrets management: Terraform provides data sources and integration with secrets management tools like HashiCorp Vault, enabling secure storage and retrieval of sensitive data. Principle of least privilege: Terraform follows the principle of least privilege by allowing users to define precise permissions for service accounts and IAM roles, limiting access to only necessary actions and resources. Input validation: Terraform performs input validation to ensure that resources are provisioned correctly and securely. It helps prevent misconfigurations that might lead to security vulnerabilities. Plan Review and Approval: Terraform's "plan" feature allows users to review proposed changes before applying them, providing additional validation and control over infrastructure modifications. Terraform's code is continuously reviewed by a large community of developers and security experts, which helps identify and address potential security issues. Terraform can be configured to generate audit logs for various actions, enabling security teams to monitor and review changes made to the infrastructure. Users can follow secure coding practices while writing Terraform configurations to avoid introducing security vulnerabilities into the code. It's important to note that while Terraform provides these security features, securing your infrastructure goes beyond the tool itself. Organizations must also implement security best practices at the cloud provider level, network security, and access controls and follow other security measures to maintain a robust security posture. Regularly updating Terraform to the latest version and adhering to security best practices are essential for keeping your infrastructure secure. Continuous monitoring, vulnerability assessments, and penetration testing can help proactively identify and address security risks. Other IaC Tools In addition to Terraform, several other Infrastructure as Code (IaC) tools are available, each with unique features and capabilities. AWS CloudFormation: AWS CloudFormation is a native IaC tool that Amazon Web Services (AWS) provides. It allows users to define infrastructure resources using JSON or YAML templates, called CloudFormation templates. Users can provision and manage AWS resources, including EC2 instances, S3 buckets, IAM roles, and more, in a declarative manner. Azure Resource Manager (ARM) Templates: Microsoft's Azure Resource Manager (ARM) Templates serve as the IaC solution for Microsoft Azure. ARM templates are JSON files that describe the desired state of Azure resources. Like Terraform and CloudFormation, ARM templates enable infrastructure provisioning and management on Azure cloud. Google Cloud Deployment Manager: Google Cloud Deployment Manager is Google Cloud Platform's (GCP) IaC tool. It uses YAML or Python templates to describe and deploy GCP resources. Like other IaC tools, Deployment Manager enables consistent and repeatable infrastructure deployments on Google Cloud. Ansible: Ansible is a powerful automation tool that goes beyond IaC and can handle configuration management tasks. It uses simple YAML-based playbooks to describe desired configurations and automate tasks across various environments, including cloud, on-premises, and network devices. Chef: Chef is a configuration management and automation tool that can also be used for IaC. It allows users to define infrastructure configurations using Ruby-based Domain-Specific Language (DSL) code. Chef is beneficial for configuring and managing complex server environments. Puppet: Puppet is another configuration management and automation tool that can be used for IaC. It uses its declarative language to define infrastructure configurations and automate resource management across various platforms and operating systems. SaltStack: SaltStack is an open-source automation and configuration management tool that can be used for IaC. It uses YAML or Jinja templates to define infrastructure configurations and manage resources in a scalable and efficient manner. Pulumi: Pulumi is an IaC tool that supports multiple cloud providers and allows users to define infrastructure using familiar programming languages like Python, JavaScript, TypeScript, and Go. This approach makes it easy for developers to leverage their existing coding skills to describe infrastructure. Cloudify: Cloudify is an open-source IaC and orchestration tool that enables users to model, deploy, and manage applications and infrastructure resources across multiple clouds and environments using YAML or DSL-based blueprints. Each IaC tool has its strengths and may be better suited to specific use cases, cloud provider preferences, and team preferences. Organizations should evaluate their requirements, cloud environment, and existing tooling to choose the IaC solution that best fits their needs. Conclusion Infrastructure as Code (IaC) has emerged as a fundamental practice for efficiently managing and provisioning infrastructure resources using code. Among the various IaC tools available, Terraform, developed by HashiCorp, stands out as the dominant choice in the industry. Terraform's success can be attributed to its unique features and capabilities. Its multi-cloud support empowers organizations to manage infrastructure across various cloud providers seamlessly. While Terraform dominates the IaC landscape, organizations should continuously assess their specific requirements and cloud preferences to make informed decisions regarding IaC tooling. Alternatives like AWS CloudFormation, Azure Resource Manager, and Google Cloud Deployment Manager may better suit specific scenarios. As the field of IaC continues to evolve, organizations need to remain vigilant, stay updated with the latest developments, and adopt best practices to maximize the benefits of IaC while ensuring secure, scalable, and well-managed infrastructure deployments. By embracing Terraform's strengths and keeping an eye on emerging technologies, organizations can build a robust foundation for successful, agile, and efficient infrastructure management in the ever-changing landscape of modern cloud computing and DevOps.