Tools

The availability of various tools in the market has often kept you thinking about which tool is appropriate for testing the web application. It is important to test the web application to ensure that it functions as per the user’s requirement and gives a high-end user experience. End-to-end testing is an approach that is designed to ensure the functionality of the applications by automating the browsers to run the scenario of particular actions made by end users. To accomplish this, Cypress and Puppeteer have commonly used tools, and their detailed comparison is the main focus of the blog. The use of Cypress has increased in the recent year for web automation testing addressing issues faced by modern web applications. Now, Puppeteer is also widely accepted for web automation testing. This triggered debate on Cypress vs. Puppeteer. To have a piece of good information on the testing tools and Cypress vs. Puppeteer’s detailed comparison is crucial. Let’s get started with discussing the overview of Cypress and Puppeteer What Is Cypress? Cypress is an open-source automation testing tool based on JavaScript solution, mainly used for modern web automation. The front-end testing framework helps us write the test cases in de-factor web language for the web application. It gives the option for testing with respect to unit tests and integration tests, including significance like easy reporting, test configuration, and many more. It also supports the Mocha test framework. The working of Cypress is different from other testing tools. For example, when you need to run a script inside the browser, it is mainly executed in the same loop as that of your application. However, when its execution needs to be done outside the browser for the same scripts, it leverages the Node.js server to support it. Features of Cypress Some of the exciting features of Cypress that you should know are as follows: · It takes snips of snapshots during the running of the tests. · It allows real-time and fast debugging by using tools like Developer Tools. · It has automated waiting, so you do not have to add waits or sleep to the running tests. · You can verify and manage the behavior of the functions, timers, and server response. · It effortlessly controls, test, and stub the edge cases without involving the servers. What Is Puppeteer? It is an open-source node js library-based framework used for automation testing and web scraping tools. It gives high-level API to control Chromium and Chrome, which runs headless by default. Puppeteer is very easy to use by the testers as it is based on the DevTools Protocol, which is similar to the one used by the Chrome Developer Tools. You need to be aware of the Chrome Developer Tools to ensure running with Puppeteer quickly. Cypress vs. Puppeteer The comparison between the Cypress vs. Puppeteer is made below based on highlighting aspects that will help you get a clear picture. Brief A puppeteer is a tool which is developed by Google which works for automating Chrome with the use of DevTool protocol. However, Cypress is developed by Semiconductors which is a test runner-open source. The main difference between Puppeteer and Cypress is based on their work. Puppeteer is basically allowing browser automation based on node library, whereas Cypress is purely a test automation framework allowing End-to-End Testing, Integration Testing, and Unit Testing. It could better be understood that Puppeteer is not a framework but just a chromium version of the node version, which provides browser automation for Chrome and Chromium. The running is executed headless by default which can be further configured to run full Chromium or Chrome. In addition, Puppeteer is such a tool that provides a high level of API for controlling Chrome and Chromium over the DevTool protocol. Relating this to Cypress, it is mainly a front-end testing tool that is built for the modern web. Lastly, Puppeteer is free to use, whereas Cypress comes with both free and paid versions. Language With respect to the testing in the programming language, both Cypress and Puppeteer are based on JavaScript language. This gives you an easy option to work on both tools. Types of Testing Comparing the testing done by Cypress and Puppeteer, Cypress gives you wider options. For example, if you are looking the testing an entire application, Puppeteer cannot be the best option. It is basically great for web scrapping and crawling SPA. However, Cypress is a tool through which you can do End-to-end tests, Unit tests, and integration tests, and it can test anything that runs in a browser. Uses Puppeteer is mainly used for automating UI testing, mouse and keyboard movement, and others. It basically tests the application developed in Angularjs and Angular. Like Cypress, it is not considered an automation tool but rather manages the internal aspects of the chromium browser. It is a development tool that is able to perform tasks by developers, like locating elements and handling requests and responses. Architecture Cypress and Puppeteer differ in their architecture. Generally, most of the testing tools work by running outside of the browser, which is executed remote commands across the network. Cypress-testing tools operate inside the browsers which execute the test codes. It allows Cypress to listen and verify the browser performance at run time by modifying DOM and altering network requests and responses on the fly. It does not require any of the driver binaries. It runs on a NodeJS server which associates with the test runner manipulated by Cypress to operate the application and test code which is another iframe in a similar event loop. The supported browser of Cypress includes Canary, Chromium, Microsoft Edge, Mozilla Firefox browsers, and electron. Relating to the Puppeteer architecture follows the DevTools protocol as mentioned above. It manages the Chromium and chrome browser with the aid of high-quality API given by the Node library. The browser platform executes the action on the browser engine with and without headless mode. Followed to this, all the test execution is done in Chromium which is a real place. Other browsers, like Microsoft edge, make use of Chromium as a browser engine. It is regarded as the package which is based on a node module and hence known as Nodejs level. With the use of JavaScript, the development of automation code is done by the end user. Testing Speed Comparing the testing speed of the Puppeteer and Cypress, Puppeteer is regarded to be much faster than Cypress. In using Cypress, the test scripts are executed in the browser, where you need to click on a particular button. This will not send the command to involve a specific driver to the browser but rather utilizes DOM events to send the click command to the button. However, Puppeteer has great control over the browser due to high-level API control over Chrome and Chromium. Further, it works with minimal settings, eliminates extras, and uses less space than Cypress, making them consume less memory and start faster. Cypress is slower when executing the run test in a larger application. The main reason is that it tends to take snips of the application state at a different point in time of the tests, which makes it take more time. However, such cases are not evident in the Puppeteer, which makes it faster than Cypress. Reliability and Flexibility Relating to the testing of the web application, Cypress can be more user-friendly and reliable in doing JavaScript framework for performing end-to-end testing as compared with Puppeteer. It is because Puppeteer is not a framework but just a chromium version of the node module. Nevertheless, Puppeteer can be a great option for quick testing; however, when we want to test the entire performance and functionality of application testing, it is better suggested to use a stronger tool like Cypress. The main reason is that Cypress has its individual assertion, but Puppeteer does not, and rather it is based on Mocha, Jasmine, or Jest frameworks. Further, Cypress has its individual IDE, and Puppeteer is dependent on the VS Code and Webstorm. In a nutshell, Puppeteer only supports Chromium engine-based browsers, whereas Cypress supports many different browsers, thus, making it more reliable and flexible. Testing Code Execution on the Client Side, Like the Web Browser Puppeteer and Cypress have aspects of the client side where they allow testing code execution on the client-like web browser. In Puppeteer, manual operation can be done in the browser, and it is easy to create a testing environment for the test to run directly. You have the option to test the front-end function and UI testing with the use of Puppeteer. Further, Cypress aims to test like anything that could be run in the browser and executed to build a high user experience. It tests the flow of the application from start to end according to the view of the user. It also works equally well on older servers for the pages and applications. Testing Behavior on the Server Side The major difference between Puppeteer and Cypress is related to the allowance of the testing behavior of the server-side code, whereas Puppeteer does not have such aspects. However, Cypress has the ability to test the back-end behavior, say, for example, with the use of cy.task() command. It gives the way to run the Node code. Through this, users can take actions crucial for the tests beyond the scope of Cypress. Test Recording Cypress comes with dashboards where you can be able to see the recorded tests and provide details on the events which happen during the execution. However, Puppeteer does not have such a dashboard, making it unable to record the test. Hence, transparency in the execution of the test is not maintained in Puppeteer. Fixtures Fixtures are the specific and fixed states of data that are test locals. This helps confirms a particular environment for a single test. Comparing the two, Cypress has the inbuilt fixtures abilities. With the use of the command cy.fixture(filePath), you can easily load a fixed set of data that is located in a file. However, Puppeteer does not have any such fixtures. Group Fixtures Group fixtures let to define particular and fixed states of data for a group of tests which helps ensures the environment for a given group of tests. For this also, Puppeteer does not have any such fixtures. At the same time, Cypress has the ability to create group fixtures with the use of the cy.fixture command. Conclusion The blog has presented detailed comparisons of the Puppeteer and Cypress, which gives you enough information for you to decide which tools will be best according to your test requirement. LambdaTest is a cloud-based automation testing platform with an online Cypress automation tool that writes simple Cypress automation testing and sees its actions. Using LambdaTest, you can also test your Puppeteer test scripts online. Both Cypress and Puppeteer come with their own advantages and limitations. It would be best if you decided which suits best to your tests.

Developers have many options to build a cloud environment using available tools, but today’s complex infrastructure involves numerous interconnected services and processes. To ensure smooth operation amidst daily changes in your production environment, it is crucial to use advanced tools to design and build an elastic cloud environment. These tools can simplify coding and prevent repetitive tasks for you and your team. Here are some tips that will simplify the code for you and prevent repetition from you and the rest of the team: Get to Know the Terraform Tool Terraform is a tool that enables developers to define the architecture of their infrastructure through a clear and readable file format known as HCL. This file format describes the elastic components of the infrastructure, such as VPCs, Security Groups, Load Balancers, and more. It provides a simple and concise way to define the topology of the development environment. Two Major Challenges When Designing Automated Cloud Environments The first challenge arises during the initial run, when there is little or no interdependence between the different resources. This is relatively easy to handle. The second challenge is more complex and occurs when new resources need to be developed, updated, or deleted in an existing cloud environment that is constantly changing. In a cloud environment, changes should be carefully planned and executed to minimize errors between teams and ensure a smooth and efficient implementation. To achieve this, simplification of the cloud environment is crucial to prevent code duplication among developers working on the same source code. How Terragrunt Solves This Problem: An In-Depth Look Terragrunt is a tool that enhances Terraform’s functionality by providing additional infrastructure management tools that help maintain a DRY (Don’t Repeat Yourself) code base, dressing up existing Terraform modules and managing the remote state of the cloud environment. An example where Terragrunt can be particularly useful is in a distributed cloud environment where multiple resources share the same values, such as subnets and security groups. Without proper management, developers may inadvertently duplicate these values, leading to errors and inconsistencies. Terragrunt helps prevent code duplication by allowing the parameters to be defined only once, ensuring there is no confusion about the shared parameters in the environment. To optimize performance, Terragrunt enforces specific development principles and restrictions on the organization of Terraform code: Terragrunt mandates the use of a hierarchical folder structure to maintain consistency and prevent errors in the Terraform code. Terragrunt promotes centralized file management for shared common variables, enabling the organization of code and changes in a single location. Optimizing Cloud Environments With Logical Organization Using Terragrunt Before using Terragrunt, it is essential to organize the logical environment by dividing it into smaller scopes through folders. This approach enables the reuse of resources and modules without the need for rewriting --> promoting efficiency and reducing the risk of errors. By organizing the workspace in this manner, Terragrunt allows for the import of entries from Terragrunt.hcl files located in other hierarchies within the cloud environment. This process avoids duplicating values required for different resources by using the “Include Block” or “Dependency Block” to import previously written values from other hierarchies in the environment. Efficient File Management in Shared Spaces for Improved Collaboration Terragrunt offers a powerful capability for sharing configuration files with ease. Much like Terraform, Terragrunt receives parameters to launch resources. However, unlike Terraform, Terragrunt allows to define these parameters at a higher level, so different resources within the environment can make use of the same parameters. For instance, if an environment is running in the us-east-1 region, defining the region value in the root directory allows any resource within the environment to inherit the value for its own use. This approach minimizes redundancy and ensures consistency throughout the environment. Terragrunt’s ability to define parameters at a higher level streamlines the configuration process and makes it easier to manage resources. For instance, consider this use case: YAML Infra VPC Terragrunt.hcl Backend Service-01 Terragrunt.hcl Service-02 Terragrunt.hcl Terragrunt.hcl Modules VPC Terragrunt.hcl Terragrunt.hcl Referring to the hierarchy, as shown above: Infra: This concept organizes our environment and its structure by arranging the infrastructure in a specific order. This order starts with the VPC and everything related to it, followed by the backend and its various service definitions, and so on. Modules: This concept connects us to a group of resources that we intend to utilize. For instance, if we have decided to use a VPC in our infrastructure, we would define its source artifactory and initialization parameters within the module scope. Similarly, if our backend includes a service like Kubernetes-dashboard, we would also define its source artifactory within the module scope, and so on. Terragrunt.hcl: This file serves as the Terragrunt configuration file, as previously explained. However, we are also using it to define common values for the environment. For instance, if Service-01 and Service-02 share some common parameters, then we would define these parameters at a higher level in the Terragrunt.hcl file, under the backend scope, which is located in the root folder of both services. Furthermore, we have created a Terragrunt.hcl file within the root directory. By doing so, we are consolidating common values that pertain to the all environment, and that other parts of the hierarchy do not need to be aware of. This approach allows us to propagate shared parameters downward in the hierarchy, enabling us to customize our environment without duplicating values. Conclusion Numerous organizations prioritize the aspect of quality as a fundamental objective that underpins the entire process. As such, it is crucial to ask the right questions in order to achieve this goal: How do we envision the design of our cloud environment, taking into account both the production and development environments? How can we model the environment in a way that minimizes the risk of errors in shared variables? How flexible is the environment when it comes to accommodating rapid changes? Based on our experience, we follow the following approach: Begin by outlining the problem on paper. Break down the problem into smaller, manageable modules. Develop each module separately, using elastic thinking that enables the module to be reused in additional use cases, either by ourselves or other developers. Abstract the implementation of the solution while avoiding code duplication, i.e., propagating shared variables for resources that have common values. Sharing our acquired knowledge with the community is a priority for us, and we are excited to offer our solutions to anyone interested in learning from them. You can easily access and gain insights into our methods, and even implement them yourself. As an additional resource, we have a Github repository that contains a multitude of examples for creating efficient and effective applications in a cloud environment using Terragrunt. You’re welcome to use this as a reference to help you develop your own solutions.

Data streaming emerged as a new software category. It complements traditional middleware, data warehouse, and data lakes. Apache Kafka became the de facto standard. New players enter the market because of Kafka’s success. One of those is Redpanda, a lightweight Kafka-compatible C++ implementation. This article explores the differences between Apache Kafka and Redpanda, when to choose which framework, and how the Kafka ecosystem, licensing, and community adoption impact a proper evaluation. Data Streaming: A New Software Category Data-driven applications are the new black. As part of this, data streaming is a new software category. If you don’t understand yet how it differs from other data management platforms like a data warehouse or data lake, check out the following article series: Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies? Data Streaming for Data Ingestion into the Data Warehouse and Data Lake Data Warehouse Modernization: From Legacy On-Premise to Cloud-Native Infrastructure Case Studies: Cloud-native Data Streaming for Data Warehouse Modernization Apache Kafka: The De Facto Standard for Data Streaming Apache Kafka became the de facto standard for data streaming similar to Amazon S3 is the de facto standard for object storage. Kafka is used across industries for many use cases. The Adoption Curve of Apache Kafka The growth of the Apache Kafka community in the last years is impressive: >100,000 organizations using Apache Kafka >41,000 Kafka meetup attendees >32,000 Stack Overflow questions >12,000 Jiras for Apache Kafka >31,000 open job listings request Kafka skills Look at the increased number of active monthly unique users downloading the Kafka Java client library with Maven: Source: Sonatype The numbers grow exponentially. That’s no surprise to me as the adoption pattern and maturity curve for Kafka are similar in most companies: Start with one or few use cases (that prove the business value quickly). Deploy the first applications to production and operate them 24/7. Tap into the data streams from many domains, business units, and technologies. Move to a strategic central nervous system with a decentralized data hub. Kafka Use Cases by Business Value Across Industries The main reason for the incredible growth of Kafka’s adoption curve is the variety of potential use cases for data streaming. The potential is almost endless. Kafka’s characteristics of combing low latency, scalability, reliability, and true decoupling, establish benefits across all industries and use cases: The Emergence of Many Kafka Vendors The market for data streaming is enormous. With so many potential use cases, it is no surprise that more and more software vendors add Kafka support to their products. Most vendors use Kafka or implement its protocol because Kafka has become the de facto standard for data streaming. To be clear: An increasing number of Kafka vendors is a great thing! It proves the creation of a new software category. Competition pushes innovation. The market share is big enough for many vendors. And, I am 100% convinced that we are still in a very early stage of the data streaming hype cycle... After a lengthy introduction to set the context, let’s review a new entrant into the Kafka market: Redpanda. Introducing Redpanda: Kafka-Compatible Data Streaming Redpanda is a data streaming platform. Its website explains its positioning in the market and product strategy as follows (to differentiate it from Apache Kafka): No Java: A JVM-free and ZooKeeper-free infrastructure. Designed in C++: Designed for a better performance than Apache Kafka. A single-binary architecture: No dependencies to other libraries or nodes. Self-managing and self-healing: A simple but scalable architecture for on-premise and cloud deployments. Kafka-compatible: Out-of-the-box support for the Kafka protocol with existing applications, tools, and integrations. This sounds great. You need to evaluate whether Redpanda is the right choice for your next project or if you should stick with “real Apache Kafka.” How To Choose the Proper “Kafka” Implementation for Your Project A recommendation that some people find surprising: Qualify out first! That’s much easier. Similarly, like I explained when NOT to use Apache Kafka. As part of the evaluation, the question is if Kafka is the proper protocol for you. And for Kafka, pick different offerings and begin with the comparison. Start your evaluation with the business case requirements and define your most critical needs like uptime SLAs, disaster recovery strategy, enterprise support, operations tooling, self-managed vs. fully-managed cloud service, capabilities like messaging vs. data ingestion vs. data integration vs. applications, and so on. Based on your use cases and requirements, you can start qualifying out vendors like Confluent, Redpanda, Cloudera, Red Hat / IBM, Amazon MSK, Amazon Kinesis, Google Pub Sub, and others to create a shortlist. The following sections compare the open-source project Apache Kafka versus the re-implementation of the Kafka protocol of Redpanda. You can use these criteria (and information from other blogs, articles, videos, and so on) to evaluate your options. Similarities Between Redpanda and Apache Kafka The high-level value propositions are the same in Redpanda and Apache Kafka: Data streaming to process data in real-time at scale continuously. Decouple applications and domains with a distributed storage layer. Integrate with various data sources and data sinks. Leverage stream processing to correlate data and take action in real-time. Self-managed operations or consuming a fully-managed cloud offering. However, the devil is in the details and facts. Don’t trust marketing, but look deeper into the various products and cloud services. Deployment Options: Self-Managed vs. Cloud Service Data streaming is required everywhere. While most companies across industries have a cloud-first strategy, some workloads must stay at the edge for different reasons: Cost, latency, or security requirements. Besides operating Redpanda by yourself, you can buy Redpanda as a product and deploy it in your environment. Instead of self-hosting Redpanda, you can deploy it as a data plane in your environment using Kubernetes (supported by the vendor’s external control plane) or leverage a cloud service (fully managed by the vendor). The different deployment options for Redpanda are great. Pick what you need. This is very similar to Confluent’s deployment options for Apache Kafka. Some other Kafka vendors only provide either self-managed (e.g., Cloudera) or fully managed (e.g., Amazon MSK Serverless) deployment options. What I miss from Redpanda: No official documentation about SLAs of the cloud service and enterprise support. I hope they do better than Amazon MSK (excluding Kafka support from their cloud offerings). I am sure you will get that information if you reach out to the Redpanda team, who will probably soon incorporate some information into their website. Bring Your Own Cluster (BYOC) There is a third option besides self-managing a data streaming cluster and leveraging a fully managed cloud service: Bring your own Cluster (BYOC). This alternative allows end users to deploy a solution partially managed by the vendor in your own infrastructure (like your data center or your cloud VPC). Here is Redpanda’s marketing slogan: “Redpanda clusters hosted on your cloud, fully managed by Redpanda, so that your data never leaves your environment.” This sounds very appealing in theory. Unfortunately, it creates more questions and problems than it solves: How does the vendor access your data center or VPC? Who decides how and when to scale a cluster? When to act on issues? How and when do you roll a cluster to incorporate bug fixes or version upgrades? What about cost management? What is the total cost of ownership? How much value does the vendor solution bring? How do you guarantee SLAs? Who has to guarantee them, you or the vendor? For regulated industries, how are security controls and compliance supported? How are you sure about what the vendor does in an environment you ostensibly control? How much harder will a bespoke third-party risk assessment be if you aren’t using pure SaaS? For these reasons, cloud vendors only host managed services in the cloud vendor's environment. Look at Amazon MSK, Azure Event Hubs, Google Pub Sub, Confluent Cloud, etc. All fully managed cloud services are only in the VPC of the vendor for the above reasons. There are only two options: Either you hand over the responsibility to a SaaS offering or control it yourself. Everything in the middle is still your responsibility in the end. Community vs. Commercial Offerings The sales approach of Redpanda looks almost identical to how Confluent sells data streaming. A free community edition is available, even for production usage. The enterprise edition adds enterprise features like tiered storage, automatic data balancing, or 24/7 enterprise support. No surprise here. And a good strategy, as data streaming is required everywhere for different users and buyers. Technical Differences Between Apache Kafka and Redpanda There are plenty of technical and non-functional differences between Apache Kafka products and Redpanda. Keep in mind that Redpanda is NOT Kafka. Redpanda uses the Kafka protocol. This is a small but critical difference. Let’s explore these details in the following sections. Apache Kafka vs. Kafka Protocol Compatibility Redpanda is NOT an Apache Kafka distribution like Confluent Platform, Cloudera, or Red Hat. Instead, Redpanda re-implements the Kafka protocol to provide API compatibility. Being Kafka-compatible is not the same as using Apache Kafka under the hood, even if it sounds great in theory. Two other examples of Kafka-compatible offerings: Azure Event Hubs: A Kafka-compatible SaaS cloud service offering from Microsoft Azure. The service itself works and performs well. However, its Kafka compatibility has many limitations. Microsoft lists a lot of them on its website. Some limitations of the cloud service are the consequence of a different implementation under the hood, like limited retention time and message sizes. Apache Pulsar: An open-source framework competing with Kafka. The feature set overlaps a lot. Unfortunately, Pulsar often only has good marketing for advanced features to compete with Kafka or to differentiate. One example is its Kafka mapper to be compatible with the Kafka protocol. Contrary to Azure Event Hubs, as a serious implementation (with some limitations), Pulsar’s compatibility wrapper provides a basic implementation that is compatible with only minor parts of the Kafka protocol. So, while the alleged “Kafka compatibility” sounds nice on paper, one shouldn’t seriously consider this for migrating your running Kafka infrastructure to Pulsar. We have seen compatible products for open-source frameworks in the past. Re-implementations are usually far away from being complete and perfect. For instance, MongoDB compared the official open source protocol to its competitor Amazon DocumentDB to pinpoint the fact that DocumentDB only passes ~33% of the MongoDB integration test chain. In summary, it is totally fine to use these non-Kafka solutions like Azure Event Hubs, Apache Pulsar, or Redpanda for a new project if they fulfill your requirements better than Apache Kafka. But, keep in mind that it is not Kafka. There is no guarantee that additional components from the Kafka ecosystem (like Kafka Connect, Kafka Streams, REST Proxy, and Schema Registry) behave the same when integrated with a non-Kafka solution that only uses the Kafka protocol with its own implementation. How Good Is Redpanda’s Kafka Protocol Compatibility? Frankly, I don’t know. Probably and hopefully, Redpanda has better Kafka compatibility than Pulsar. The whole product is based on this value proposition. Hence, we can assume the Redpanda team spends plenty of time on compatibility. Redpanda has NOT achieved 100% API compatibility yet. Time will tell when we see more case studies from enterprises across industries that migrated some Apache Kafka projects to Redpanda and successfully operated the infrastructure for a few years. Why wait a few years to see? Well, I compare it to what I see from people starting with Amazon MSK. It is pretty easy to get started. However, after a few months, the first issues happen. Users find out that Amazon MSK is not a fully-managed product and does not provide serious Kafka SLAs. Hence, I see too many teams starting with Amazon MSK and then migrating to Confluent Cloud after some months. Let’s be clear: If you run an application against Apache Kafka and migrate to a re-implementation supporting the Kafka protocol, you should NOT expect 100% the same behavior as with Kafka. Some underlying behavior will differ even if the API is 100% compatible. This is sometimes a benefit. For instance, Redpanda focuses on performance optimization with C++. This is only possible in some workloads because of the re-implementation. C++ is superior compared to Java and the JVM for some performance and memory scenarios. Redpanda = Apache Kafka—Kafka Connect—Kafka Streams Apache Kafka includes Kafka Connect for data integration and Kafka Streams for stream processing. Like most Kafka-compatible projects, Redpanda does exclude these critical pieces from its offering. Hence, even 100 percent protocol compatibility would not mean a product re-implements everything in the Apache Kafka project. Lower Latency vs. Benchmarketing Always think about your performance requirements before starting a project. If necessary, do a proof of concept (POC) with Apache Kafka, Apache Pulsar, and Redpanda. I bet, that in 99% of scenarios, all three of them will show a good enough performance for your use case. Don’t trust opinionated benchmarks from others. Your use case will have different requirements and characteristics. Performance is typically just one of many evaluation dimensions. I am not a fan of most “benchmarks” of performance and throughput. Benchmarks are almost always opinionated and configured for a specific problem (whether a vendor, independent consultant, or researcher conducts them). My colleague Jack Vanlightly explained this concept of benchmarketing with excellent diagrams: Source: Jack Vanlightly Here is one concrete example you will find in one of Redpanda’s benchmarks: Kafka was not built for very high throughput producers, and this is what Redpanda is exploiting when they claim that Kafka’s throughput is inferior to Redpanda. Ask yourself this question: Of 1GB/s use cases, who would create that throughput with just 4 producers? Benchmarketing at its finest. Hence, once again, start with your business requirements. Then choose the right tool for the job. Benchmarks are always built for winning against others. Nobody will publish a benchmark where the competition wins. Soft Real-Time vs. Hard Real-Time When we speak about real-time in the IT world, we mean end-to-end data processing pipelines that need at least a few milliseconds. This is called soft real-time. And this is where Apache Kafka, Apache Pulsar, Redpanda, Azure Event Hubs, Apache Flink, Amazon Kinesis, and similar platforms fit into. None of these can do hard real time. Hard real-time requires a deterministic network with zero latency and no spikes. Typical scenarios include embedded systems, field buses, and PLCs in manufacturing, cars, robots, securities trading, etc. Time-Sensitive Networking (TSN) is the right keyword if you want more research. I wrote a dedicated blog post about why data streaming is NOT hard real-time. Hence, don’t try to use Kafka or Redpanda for these use cases. That’s OT (operational technology), not IT (information technology). OT is plain C or Rust on embedded software. No ZooKeeper With Redpanda vs. No ZooKeeper With Kafka Besides being implemented in C++ instead of using the JVM, the second big differentiator of Redpanda is no need for ZooKeeper and two complex distributed systems. Well, with Apache Kafka 3.3, this differentiator is gone. Kafka is now production-ready without ZooKeeper. KIP-500 was a multi-year journey and an operation at Kafka’s heart: To be fair, it will still take some time until the new ZooKeeper-less architecture goes into production. Also, today, it is only supported by new Kafka clusters. However, migration scenarios with zero downtime and without data loss will be supported in 2023, too. But that’s how a severe release cycle works for a mature software product: Step-by-step implementation and battle-testing instead of starting with marketing and selling of alpha and beta features. ZooKeeper-less data streaming with Kafka is not just a massive benefit for the scalability and reliability of Kafka but also makes operations much more straightforward, similar to ZooKeeper-less Redpanda. By the way, this was one of the major arguments why I did not see the value of Apache Pulsar. The latter requires not just two but three distributed systems: Pulsar broker, ZooKeeper, and BookKeeper. That’s nonsense and unnecessary complexity for virtually all projects and use cases. Lightweight Redpanda + Heavyweight Ecosystem = Middleweight Data Streaming? Redpanda is very lightweight and efficient because of its C++ implementation. This can help in limited compute environments like edge hardware. As an additional consequence, Redpanda has fewer latency spikes than Apache Kafka. That are significant arguments for Redpanda for some use cases. However, you need to look at the complete end-to-end data pipeline. If you use Redpanda as a message queue, you get these benefits compared to the JVM-based Kafka engine. You might then pick a message queue like RabbitMQ or NATs instead. I don’t start this discussion here as I focus on the much more powerful and advanced data streaming use cases. Even in edge use cases where you deploy a single Kafka broker, the hardware, like an industrial computer (IPC), usually provides at least 4GB or 8GB of memory. That is sufficient for deploying the whole data streaming platform around Kafka and other technologies. Data Streaming Is More Than Messaging or Data Ingestion My fundamental question is: what is the benefit of a C++ implementation of the data hub if all the surrounding systems are built with JVM technology or even worse and slow technologies like Python? Kafka-compatible tools, like Redpanda, integrate well with the Kafka ecosystem, as they use the same protocol. Hence, tools like Kafka Connect, Kafka Streams, KSQL, Apache Flink, Faust, and all other components from the Kafka ecosystem work with Redpanda. You will find such an example for almost every existing Kafka tool on the Redpanda blog. However, these combinations kill almost all the benefits of having a C++ layer in the middle. All integration and processing components would also need to be as efficient as Redpanda and use C++ (or Go or Rust) under the hood. These tools do not exist today (likely, as they are not needed by many people). And here is an additional drawback: The debugging, testing, and monitoring infrastructure must combine C++, Python, and JVM platforms if you combine tools like Java-based Kafka Connect and Python-based Faust with C++-based Redpanda. So, I don’t get the value proposition here. Data Replication Across Clusters Having more than one Kafka cluster is the norm, not an exception. Use cases like disaster recovery, aggregation, data sovereignty in different countries, or migration from on-premise to the cloud require multiple data streaming clusters. Replication across clusters is part of open-source Apache Kafka. MirrorMaker 2 (based on Kafka Connect) supports these use cases. More advanced (proprietary) tools from vendors like Confluent Replicator or Cluster Linking make these use cases more effortless and reliable. Data streaming with the Kafka ecosystem is perfect as the foundation of a decentralized data mesh: How do you build these use cases with Redpanda? It is the same story as for data integration and stream processing: How much does it help to have a very lightweight and performant core if all other components rely on “3rd party” code bases and infrastructure? In the case of data replication, Redpanda uses Kafka’s Mirrormaker. Make sure to compare MirrorMaker to Confluent Cluster Linking—the latter uses the Kafka protocol for replications and does not need additional infrastructure, operations, offset sync, etc. Non-Functional Differences Between Apache Kafka and Redpanda Technical evaluations are dominant when talking about Redpanda vs. Apache Kafka. However, the non-functional differences are as crucial before making the strategic decision to choose the data streaming platform for your next project. Licensing, adoption curve, and the total cost of ownership (TCO) are critical for the success of establishing a data streaming platform. Open Source (Kafka) vs. Source Available (Redpanda) As the name says, Apache Kafka is under the very permissive Apache license 2.0. Everyone, including cloud providers, can use the framework for building internal applications, commercial products, and cloud services. Committers and contributions are spread across various companies and individuals. Redpanda is released under the more restrictive Source Available License (BSL). The intention is to deter cloud providers from offering Redpanda’s work as a service. For most companies, this is fine, but it limits broader adoption across different communities and vendors. The likelihood of external contributors, committers, or even other vendors picking the technology is much smaller than in Apache projects like Kafka. This has a significant impact on the (future) adoption curve. Maturity, Community, and Ecosystem The introduction of this article showed the impressive adoption of Kafka. Just keep in mind: Redpanda is NOT Apache Kafka. It just supports the Kafka protocol. Redpanda is a brand-new product and implementation. Operations are different. The behavior of the engine is different. Experts are not available. Job offerings do not exist. And so on. Kafka is significantly better documented, has a tremendously larger community of experts, and has a vast array of supporting tooling that makes operations more straightforward. There are many local and online Kafka training options, including online courses, books, meetups, and conferences. You won’t find much for Redpanda beyond the content of the vendor behind it. And don’t trust marketing. That’s true for every vendor, of course. If you read a great feature list on the Redpanda website, double-check if the feature truly exists and in what shape it is. Example: RBAC (role-based access control) is available for Redpanda. The devil lies in the details. Quote from the Redpanda RBAC documentation: “This page describes RBAC in Redpanda Console and, therefore, manages access only for console users but not clients that interact via the Kafka API. To restrict Kafka API access, you need to use Kafka ACLs.” There are plenty of similar examples today. Just try to use the Redpanda cloud service. You will find many things that are more alpha than beta today. Make sure not to fall into the same myths around the marketing of product features as some users did with Apache Pulsar a few years ago. The Total Cost of Ownership and Business Value When you define your project’s business requirements and SLAs, ask yourself how much downtime or data loss is acceptable. The RTO (recovery time objective) and RPO (recovery point objective) impact a data streaming platform’s architecture and overall process to ensure business continuity, even in the case of a disaster. The TCO is not just about the cost of a product or cloud service. Full-time engineers need to operate and integrate the data streaming platform. Expensive project leads, architects, and developers build applications. Project risk includes the maturity of the product and the expertise you can bring in for consulting and 24/7 support. Similar to benchmarketing regarding latency, vendors use the same strategy for TCO calculations. Here is one concrete example you always hear from Redpanda: “C++ does enable more efficient use of CPU resources.” This statement is correct. However, the problem with that statement is that Kafka is rarely CPU-bound and much more IO-bound. Redpanda has the same network and disk requirements as Kafka, which means Redpanda has limited differences from Kafka in terms of TCO regarding infrastructure. When to Choose Redpanda Instead of Apache Kafka? You need to evaluate whether Redpanda is the right choice for your next project or if you should stick with the “real Apache Kafka” and related products or cloud offerings. Read articles and blogs, watch videos, search for case studies in your industry, talk to different competitive vendors, and build your proof of concept or pilot project. Qualifying out products is much easier than evaluating plenty of offerings. When to Seriously Consider Redpanda? You need C++ infrastructure because your ops team cannot handle and analyze JVM logs—but be aware that this is only the messaging core, not the data integration, data processing, or other capabilities of the Kafka ecosystem. The slight performance differences matter to you—and you still don’t need hard real-time. Simple, lightweight development on your laptop and in automated test environments—but you should then also run Redpanda in production (using different implementations of an API for TEST and PROD is a risky anti-pattern). You should evaluate Redpanda against Apache Kafka distributions and cloud services in these cases. This article explored the trade-offs Redpanda has from a technical and non-functional perspective. If you need an enterprise-grade solution or fully-managed cloud service, a broad ecosystem (connectors, data processing capabilities, etc.), and if 10ms latency is good enough and a few p99 spikes are okay, then I don’t see many reasons why you would take the risk of adopting Redpanda instead of an actual Apache Kafka product or cloud service. The Future Will Tell Us if Redpanda Is a Severe Competitor I didn’t even cover the fact that a startup always has challenges finding great case studies, especially with big enterprises like fortune 500 companies. The first great logos are always the hardest to find. Sometimes, startups never get there. In other cases, a truly competitive technology and product are created. Such a journey takes years. Let’s revisit this article in one, two, and five years to see the evolution of Redpanda (and Apache Kafka). What are your thoughts? When do you consider using Redpanda instead of Apache Kafka? Are you using Redpanda already? Why and for what use cases?

As DevOps has become mainstream now, it is time for businesses to stop thinking about what DevOps is and start leveraging the amazing benefits offered by DevOps automation tools. DevOps is a methodology that integrates development and operations teams to seamlessly collaborate throughout the product development lifecycle. While it started with Dev and Ops, the functionality is not limited to these two departments anymore. Based on your project requirements and niche, you can create cross-functional teams composed of members from development, design, testing, QA, security, business, etc. DevOps not only transforms business operations but also brings a culture change across the organization. Benefits of DevOps Automation Automation is an important component of today’s IT infrastructure. By incorporating automation across the infrastructure, administrators can run various processes without human intervention. It enables you to build CI/CD pipeline workflows to accelerate software development. DevOps and automation go hand in hand. Developers can combine API-centric code with automation and deliver software products faster and better. Similarly, administrators can apply Infrastructure as Code (IaC) methods to automate provisioning and deployment with self-service capabilities without human intervention. DevOps automation tools are, at times, confused with the Infrastructure as Code (IaC) tools. However, IaC is a part of DevOps automation. IaC enables administrators to provision and manage the infrastructure using code while DevOps automation talks about a broader picture of managing the entire software development product lifecycle using DevOps tools for automation that include IaC tools as well. The key to fully leveraging the DevOps revolution is choosing the right DevOps automation tools for the infrastructure. DevOps Automation Tools for Infrastructure Management AWS CloudFormation AWS CloudFormation is a DevOps automation tool from Amazon that enables administrators to model and provision AWS resources by simply describing the target state of the infrastructure as a template. With CloudFormation, you don’t have to configure the remote state. CloudFormation manages the state of the infrastructure out-of-the-box. You can use “Change Sets” to verify changes before applying them. It supports rolling updates for autoscaling groups. You can perform a rollback if needed. CloudFormation works across all the services offered by AWS and is good at handling secrets. This managed service is free and AWS support includes support for CloudFormation. Benefits of CloudFormation CloudFormation supports cross-country and cross-region management, which means you can use a single CloudFormation template to manage AWS resources across multiple regions and accounts. You can use YAML or JSON to declare the configuration. AWS CloudFormation Designer is a visual designer that lets you visually configure CloudFormation templates. Cloud environments can be defined using familiar languages such as Python, Java, TypeScript, and .NET. It offers a preview to check how the changes impact the environment. AWS Serverless Application Model (SAM) allows you to build serverless applications faster by writing a few lines of code per resource. Dependency management is available. Good safety controls in place. Good document and community support. Things To Improve Modularization is not straightforward in CloudFormation. You have to take different approaches, such as using nested stacks or import/export output values, between modules. There is no central place for sharing templates. Terraform Terraform is a popular Infrastructure as Code (IaC) tool offered by Hashicorp Inc. This open-source tool was developed by Mitchell Hashimoto using the Go programming language and was released in 2014. It supports all major platforms, including Windows, Linux, FreeBSD, macOS, Solaris and OpenBSD. In addition to supporting the AWS platform, Terraform also supports other cloud providers such as Google Cloud Platform, Azure, IBM Cloud, OpenStack, Oracle Cloud, Digital Ocean, VMware vSphere, and other third party services. Terraform uses a declarative model to define the state of the target infrastructure. Hashicorp Configuration Language (HCL) is the domain-specific language you should use to write your configuration files. Optionally, JSON can be used as well. Terraform enables you to efficiently manage high-level components, such as SaaS features and DNS entries, and low-level components such as networking, storage and compute instances. Benefits of Terraform Terraform offers execution plans that let you preview changes before applying them to the infrastructure. Terraform proactively monitors infrastructure changes and incrementally creates execution plans to ensure the infrastructure is always maintained in the desired state. Efficiently manages multi-cloud environments. Integrates well with your existing version control systems. HCL code can be translated into JSON format. You can lock modules so only one person can apply changes to the infrastructure at one time. Things To Improve Terraform uses a domain-specific language HCL. It means you need to learn HCL just to manage the infrastructure. Secondly, the infrastructure should always be in sync with the states. There is no error handling and automatic rollback feature. Refactoring or renaming resources is not easy. Ansible Ansible is an open-source provisioning and configuration management tool that was released in 2012 by Ansible Inc. It was written in PowerShell, Python, and Ruby by Michael DeHaan and supports Linux, macOS, and Windows operating systems. Ansible was acquired by RedHat in 2015, and is now included as a part of the Fedora distribution of Linux. Ansible uses an agentless architecture, which means there is no code running on the controlled nodes. During an orchestration task, the module process communicates with the controlled node via a JSON-based protocol. When Ansible is not managing nodes, there is no resource consumption on the node machine. It allows you to describe the configuration in its own declarative language. You can create consistent environments using Ansible. Tasks are stored in YAML files called Playbooks. Benefits of Ansible Free and open-source. Secure as there are no agents deployed on node machines. Allows you to create consistent environments. With minimal dependencies, Ansible is easy to configure and manage. There is a minimal learning curve as the descriptive language is YAML-based. Without the need to install extra software or daemon on controlled nodes, Ansible optimizes resource usage and improves operational efficiencies. Ansible is robust enough to support complex IT workflows and flexible enough to customize and orchestrate the entire app environment. Supports a wide range of environments, including bare metals, virtual machines, AWS, Azure, GCP, Oracle Cloud, and other cloud environments as well as VMware and XenServer virtual environments. Agentless architecture simplifies IT management. Good community support. Things To Improve Ansible was initially a CLI tool. Later, AWX GUI was introduced with a GUI capability. The improved version of AWX GUI is now called the Ansible Tower. However, the UI Interface is not visually appealing and can still be improved. You might experience conflicting query results owing to synchronization issues between the GUI and CLI. Windows support is limited. Ansible doesn’t track dependencies, which means there is no notion of state. Pulumi Pulumi is an Infrastructure as Code (IaC) tool released in 2017. The best thing about Pulumi is that it allows you to manage code using real programming languages such as Python, TypeScript, JavaScript, Go, and .NET languages. It means you don’t have to learn a domain-specific language just to manage your infrastructure. Pulumi helps you build reusable cloud infrastructure platforms. It offers SDK that comes with a consistent interface to work with 50+ cloud service providers. Pulumi allows you to define cloud resources more expressively and efficiently using variables and loops instead of just copy-pasting code. You can also use any node library. It automatically checks for errors. Being designed with cloud-native computing in mind, the tool allows you to easily manage containers and serverless architecture to quickly build and deploy apps on a variety of IT environments. Benefits of Pulumi Supports real programming languages such as Go, .NET Core, Node.js, and Python. Allows dynamic resource creation at runtime. Easy standard utility functions. Policy compliance is good. Pulumi creates a preview and checks if it complies with policies before resources are created or modified. State management through Pulumi comes by default. Self-management is also available. Allows you to work alongside Terraform Business-friendly Apache License 2.0. Things To Improve When it comes to structuring large projects, Pulumi structures them as multiple micro-projects or a single monolithic project. As such, it becomes difficult to deserialize stack references when trying to map multiple resources via higher-level Pulumi extensions. Compared to big names, like Terraform or CloudFormation, Pulumi documentation and community support is limited. Bash Scripting Bash scripting allows developers to write commands as inputs and execute a specific task. You can write the script once and reuse it again. The length of a Bash script may vary from a few lines to thousands of lines. Using Bash shell scripts, you can automatically provision and manage environments, deploy resources, perform test suites, etc. You can use them in CI/CD pipelines and inside virtual machines as well. Bash is a popular tool for writing shell scripts and is included in all Unix-like operating systems, enabling you to write portable POSIX scripts. Bash scripting is one of the top ten most popular technologies and offers the highest salaries as well, as reported by Stack Overflow. There are several reasons to use Bash as a part of your DevOps tools in automation instead of PHP or JSP. It allows developers to derive maximum output from a bare-minimum machine. When you don’t have access to PHP, you can use Bash to retrieve the system info in JSON format and display it as a webpage as well. It works as a time series based data exporter. Bash allows you to write commands using simple grammar, making it easy to learn and use. It also supports basic concepts of programming. It natively supports process execution. Benefits of Bash Easy and readable syntax. Available with all Unix-like operating systems. Most popular shell script tool. Reusable scripts for repetitive tasks. Natively supports process execution. Supports loops, if-else, functions, variables, arithmetic operations, etc. Easy to write portable POSIX scripts. Things To Improve Bash scripts perform slower when compared with other programming languages owing to the lack of a standard API wherein you have to spawn processes to process data. Being a command language, Bash considers everything you write as a command. When the automation logic grows bigger, it becomes a challenge to manage Bash scripts for automation tasks. Python Scripting Python scripting is a good alternative to Bash scripting. Python is one of the most popular programming languages used by developers across the globe. One of the main reasons for this popularity is that it is developer-friendly. There are hundreds of built-in libraries that help developers quickly and easily write code and deploy products faster. Most of the Unix-like operating systems come preinstalled with a Python interpreter, allowing developers to write portable automation scripts. For instance, Boto is a Python package that allows you to write scripts for automating AWS tasks such as starting/stopping EC2 instances and managing resources via APIs. Boto3 is the latest Boto version that now supports more than fifty Amazon services spanning storage, compute, network, billing, application, database, etc. Benefits of Python Scripting Python is a popular programming language used by several developers. Developer-friendly language. Offers a large portfolio of built-in libraries. Python interpreter is available with all Unix-like OS’. Working with commands and processes is made easy with Plumbum and Shellpy APIs. Allows you to connect to AWS resources and manage them with scripts using third party tools such as Boto. Things To Improve Python is not a command language, which means you cannot directly execute shell scripts as commands. It does not natively support process execution as Bash does. However, you can use the subprocess module to spawn processes and connect to the input/output and error pipelines to receive the return codes. Python scripts are not readable and the syntax is slightly complex when compared to Bash scripts. However, you can simplify Python scripts using tools like Shellpy. Python execution is slow compared to other scripting languages. When dependent libraries are involved, it gets much slower. Best Toolset for DevOps Infrastructure Automation Here are the best toolsets for DevOps infrastructure automation: Docker Kubernetes CloudFormation Terraform Python Docker for Containerization Docker is the most popular containerization solution available in the market. It offers a robust and comprehensive containerization ecosystem that lets you manage the entire application deployment lifecycle with ease. Docker is flexible, modular, collaborative, highly-portable, scalable, and enables you to create consistent and isolated environments. With automatic rollbacks and repeatability, it offers a cost-effective way of deploying apps faster and better. While finalizing DevOps tools for automation, Docker should always be an automatic inclusion. Kubernetes for Container Orchestration Kubernetes is the leader in the container orchestration segment. It works with almost every container runtime, and is flexible and highly portable. It is battle-tested and proven technology. Lastly, it supports multi-cloud deployments while increasing developer productivity. CloudFormation for Infrastructure Automation CloudFormation is a powerful infrastructure automation tool offered by AWS. It seamlessly integrates with AWS services, enabling you to easily connect and collaborate all your development and infrastructure management tasks. Terraform for Non-AWS Environments If you use non-AWS environments, such as Microsoft Azure, IBM Cloud, Oracle Cloud, or Google Cloud Platform (GCP) cloud environments, Terraform is a good option to automate infrastructure management tasks. It best suits multi-cloud deployments as well. Python for Automation Scripts Python is a popular programming language and is best suited for creating portable automation scripts. You can write shell scripts and use APIs to connect to infrastructure resources and manage them with ease. It is open-source and cost-effective. DevOps Automation Tools for CI/CD Pipelines The advent of DevOps has revolutionized the software development segment. Gone are the days when the software was developed through a waterfall approach, wherein the code traveled through different stages of development linearly. DevOps brings cross-functional teams into the picture wherein development and operations teams, along with the testing, security, and business professionals come together to collaborate throughout the application life cycle. Continuous Integration (CI) and Continuous Deployment (CD) is an important component of the DevOps continuous delivery model. CI/CD is an innovative approach to delivering applications by incorporating automation into each stage of the application development lifecycle. Here are some DevOps automation Tools: Jenkins CircleCI AWS CodePipeline Gitlab Why CI/CD? Continuous integration allows developers to frequently merge the changes onto the main branch while the testing team automatically performs tests to ensure the software is of high quality and is not broken. Similarly, continuous delivery allows automatic deployment of code to the test environment and then to the production environment. Continuous integration enables developers to deploy software faster and better as you continuously deliver code to production while automatically fixing bugs and errors. CI servers can automatically run thousands of builds in quick time. Developers are automatically alerted when the code is broken so they can immediately fix it. It enables the quality team to spend less time on fixing bugs and more time on improving the quality. Continuous delivery enables the operations teams to automatically deploy software without spending several days preparing the environment. Releases are frequently done and the feedback loop is accelerated, which means organizations don’t have to worry about making changes to the code. While improvements could be done faster, the quality can be improved without breaking the environment. With continuous deployment, the software is automatically deployed. When there is a change in the code, the deployment pipeline gets triggered, meaning fewer risks when making a release. Similarly, customers can enjoy quality software owing to this continuous effort of improvement. While there is a cost factor in maintaining the CI server, there are other cost-effective options available in the form of open-source tools. When it comes to business value, organizations can shorten the release cycle, gain faster time to market, and deliver what the customers want. They can also monitor changes and proactively make changes to the product. It improves operational efficiency and increases revenues. The real-time visibility into the entire development process provides a better control over the infrastructure. Here are some of the popular DevOps automation tools for CI/CD tasks: Jenkins Jenkins is a popular continuous integration tool that is offered as an open-source. It was called Hudson earlier. Hudson was written by Kosuke Kawaguchi in the Java programming language and was released in 2005 when he was working in Sun Microsystems. Oracle acquired Hudson in 2011, and decided to trademark the name Hudson to release it as a commercial version. This decision did not go down well with the Hudson community. Eventually, the tool was forked as Oracle Hudson. The original tool was renamed as Jenkins. Both versions continued for some time. The Oracle Hudson version became absolute in 2017. Jenkins enables organizations to automate the build, test, and deployment process. It uses a server-based system and Apache Tomcat runs the servlet containers. Jenkins supports all version control tools such as Git, Mercurial, Subversion (SVN), etc. It also allows you to execute Apache Maven, shell scripts, and Windows commands. Benefits of Jenkins: It is an open-source tool and free. It is easy to use and comes with a good community support. Jenkins offers a rich plugin ecosystem comparison 1000+ plugins. It is easy to install because you don’t have to install additional components. At the same time, configuring the server requires knowledge of the tool. Being a Java-based server, it is highly portable and works with all major platforms. It can automate all tasks, schedule builds, and automatically test errors and repair them. Things To Improve: While there are thousands of Jenkins plugins available, they are contributed by third parties, which means the quality might not be consistent. Some of them are not maintained now. So, you should be careful while integrating and customizing plugins to ensure consistent performance is delivered. Moreover, there are several redundant plugins. As such, it requires your time and effort in choosing the right plugin for your CI/CD automation needs. Secondly, the Jenkins interface is not user-friendly and intuitive. As it runs on a server, you should monitor activities. You need expertise and knowledge to integrate it with other DevOps tools for automation. CircleCI CircleCI is a cloud-based continuous integration tool that was developed in 2011 by CircleCI, a San Francisco-based IT company. The tool offers power, flexibility, and greater control to manage CI/CD pipelines. It integrates with GitHub and Bitbucket, and every time you make a code change, it automatically triggers the CircleCI pipeline and automatic testing is done in a virtual machine or a container, meaning you can test every commit. When the build fails, the concerned team is automatically alerted. It comes with Slack integration for alert notifications. Builds that successfully pass the test are automatically deployed to the corresponding environments. CircleCI is language-agnostic and platform-agnostic. It supports all programming languages, including JavaScript, PHP, Python, Ruby, .NET, C++, Linux, Windows, and Mac OS environments. You can create tasks using a YAML configuration file. All tasks can be written in a single YAML file. It offers expanded caching, which brings speeds to the pipeline processes. It enables faster code deployment and allows complicated pipelines to be easily managed. Using SSH, you can quickly identify issues and solve problems. It uses LDAP user management and VM isolation to provide better security. It also offers an insights dashboard to monitor and manage all the tasks that are happening in the environment. CircleCI is a popular CI server used by 30,000 organizations, where 1 million tasks are managed by the tool per day. Benefits of CircleCI: Easy and quick set up. Easy to get started as it is cloud-based and there is no need to install and configure dedicated servers. You can customize it to suit different platforms such as Kubernetes Azure, and AWS EC2 instances. It supports parallelism to deploy multiple executions for speed and increased performance. All tasks are created in a single file called circle.yaml, so it is easy and simple to backup and share tasks. With SSH, debugging becomes easy. Built-in Docker support. Job configuration is simple and easy. You just have to add a YAML file in the Git root repository. Things To Improve: Compute costs are low but the cost per user is a bit high. The search feature can be improved. Customization options are limited. Documentation and tutorials are not adequate compared to other tools. The configuration is easy, but gets complicated when the files become larger as you write all code in a single file. AWS CodePipeline AWS CodePipeline is a fully managed continuous delivery service that is quickly gaining popularity in the DevOps automation tools segment. This tool helps organizations automate all stages of the CI/CD pipeline and deliver quality applications while easily managing infrastructure updates. By automating the build, test, and deployment phases, you can rapidly deliver features and updates to your customers. AWS CodePipeline easily integrates with third party version control systems such as GitHub, Bitbucket, GitHub Enterprise, and other AWS services, such as Amazon S3 AWS, Elastic Beanstalk, AWS CloudFormation, etc. Or AWS CodePipeline architecture comprises three important components: AWS CodeBuild: is a fully managed build service that handles build operations such as compiling the source code, running unit tests, and creating the artifacts for deployment. As such, you don’t have to worry about provisioning or managing your build servers. It comes pre-packaged with build environments for popular build tools such as Gradle, Apache Maven, and Apache Ant. AWS CodeCommit: is a fully managed source control service that enables organizations to host and manage private Git repositories. It means you don’t need to manage the source control service, nor worry about scaling the infrastructure environment. Right from code to binaries, it allows you to store anything related to source control. It supports standard Git functionality, which means you can integrate all your existing Git-based tools. With automatic encryption of files in rest, transit, and integration with IAM, AWS CodeCommit offers a highly secure environment to manage your repositories. AWS CodeDeploy: is a fully managed deployment service offered by AWS. It enables organizations to automate deployments across a range of AWS services such as AWS Lambda, AWS Fargate, AWS EC2 instances, etc. In addition, you can deploy apps and services to your on-premise environments. Using AWS CodeDeploy, you can rapidly release new features, eliminate downtimes, and easily manage error-free infrastructure operations. Benefits of AWS CodePipeline: Easy to set up and manage. AWS provisions the infrastructure, so you don’t have to manage any servers. Allows you to define the pipeline structure using a decorative JSON document. Parallel execution is supported that increases your workflow speeds Easily integrates with all AWS services, third party, and custom plugins. Highly secure with IAM user management, IAM roles, and automatic encryption. You can build Docker images. Things To Improve: AWS CodePipeline comes with a pay-as-you-go subscription model. For instance, the AWS CodeBuild tool charges you for every minute of build time. On the contrary, Jenkins charges you for the usage of underlying hardware comprising the Jenkins master and EC2 clusters. So, it is important to monitor resource usage carefully. Secondly, while Jenkins offers a rich plugin ecosystem, AWS CodePipeline is still in the nascent stage. However, it allows you to integrate third party plugins to customize UI and build processes. Docker image caching is not available. Gitlab Gitlab is a powerful DevOps platform that brings product development, product security, and operations onto a single platform. It is a web-based Git repository that allows organizations to easily create and manage open and private repositories while facilitating seamless collaboration between teams at every stage of the product lifecycle to deliver high-quality software faster. In addition, it enables you to set role-based user permissions once and manage permissions automatically. You can self-host your projects. Gitlab Inc. is the company behind Gitlab. The project was developed by Sytse Sijbrandij and Dmytro Zaporozhets using Ruby and Go programming languages. The current technology stack involves Ruby on Rails, Go, and Vue.js programming languages. Gitlab is quite popular in the DevOps automation tools segment now with around 30 million users and 1 million active license users. Benefits of Gitlab: Easy to set up and use. Powerful and robust CI/CD Pipelines. Tight integration with Kubernetes. The availability of a built-in registry facilitates instant deployments without the need for any configurational settings. Easily import and export enormous projects and manage them with ease. Allows free and unlimited private repositories. Delivers reliable performance and 99.9% uptime. Easy integration with multiple third party plugins and APIs. Intuitive and user-friendly UI. Good documentation and great community support. Things To Improve: There are a few bugs that dampen the performance of the tool. While the UI is good, it gets complicated when you try to monitor and review certain processes. Customization of issue tracking can be improved. Best Toolset for DevOps CI/CD Automation Choosing the right technology stack is the key to fully leveraging CI/CD and DevOps automation methodologies. However, it depends on several factors such as the company’s business objectives, the technology stack in use, IT budgets, expertise levels, organization culture, existing infrastructure, etc. If you love AWS and your infrastructure is AWS-heavy, the best option is to choose the AWS CodePipeline. It will enable you to manage the entire DevOps product lifecycle while being able to easily integrate other services, seamlessly collaborate with teams, and optimize resources to the core. While IT management is simplified, costs are optimized. Gitlab combined with Jenkins best suits the CI/CD needs of large enterprises. Gitlab offers a robust and comprehensive platform that helps them manage the entire CI/CD pipeline from a single platform. It acts as a single source of truth for all CI/CD operations and efficiently handles complex enterprise development projects. At the same time, Jenkins brings a rich plugin ecosystem that helps them customize every CI/CD operation to suit their organizational needs. For instance, the IaC plugin allows administrators to manage the infrastructure with readable APIs without having deep knowledge of the tool. Jenkins X Cloud system is another plugin that lets you automate deployments of large IT environments. For startups, CircleCI is a good choice as it allows them to cost-effectively manage CI/CD tasks. They can take advantage of the free tier offerings and easily manage development operations. It easily integrates with other DevOps tools in automation. DevOps Strategy When rightly implemented, DevOps enables organizations to build and deliver products faster, reduce time to market, improve product quality and productivity while facilitating seamless collaboration and teamwork across the infrastructure. Here are the key components of a good DevOps Strategy: Infrastructure as Code (IaC): and automation enables organizations to provision, configure, and manage the infrastructure using code while always maintaining it in the desired state. Amazon CloudFormation, Terraform, Ansible, Puppet, Chef are a few popular IaC and configuration management tools. Continuous Integration/Continuous Delivery (CI/CD): pipelines enable organizations to automate the integration of code changes and code build, reduce the time of release cycles, automatically fix bugs, and improve the quality of code, ensuring software is released as per predefined schedules. Jenkins, TeamCity, CircleCI, and Gitlab are a few popular CI/CD tools. Continuous feedback: loops enable organizations to receive instant notifications about bugs, performance issues, customer requirements etc., and resolve them on the go. Microservices architecture: enables organizations to break the code into smaller deployable services that are modular, independent, and communicate via APIs. Container management systems: enables organizations to deploy, manage, and scale clusters of containers with ease. Kubernetes is the most popular container orchestration tool. Here are eight steps to implement the DevOps strategy: Identify the right cloud service provider. Design the software architecture. Incorporate the 12-factor methodology. Design a container orchestration system. Design CI/CD pipelines. Incorporate automation wherever possible using Infrastructure as Code. Ensure security and compliance is implemented. Set up a system for support, maintenance, and incident response DevOps ToolChain used by SaaS Companies Conclusion At this point, you should have a better understanding of DevOps tools for SaaS companies, infrastructure management, infrastructure automation, CI/CD pipelines, CI/CD automation, and more! I hope you take away something valuable from this article. Feel free to comment below and share.

CI/CD (Continuous Integration and Continuous Delivery) is an essential part of modern software development. CI/CD tools help developers automate the process of building, testing, and deploying software, which saves time and improves code quality. GitLab and Jenkins are two popular CI/CD tools that have gained widespread adoption in the software development industry. In this article, we will compare GitLab and Jenkins and help you decide which one is the best CI/CD tool for your organization. GitLab vs Jenkins 1. Ease of Use GitLab is an all-in-one platform that provides a comprehensive solution for CI/CD, version control, project management, and collaboration. It has a simple and intuitive user interface that makes it easy for developers to set up and configure their pipelines. On the other hand, Jenkins is a highly customizable tool that requires some technical expertise to set up and configure. It has a steep learning curve, and new users may find it challenging to get started. 2. Integration GitLab and Jenkins support integration with a wide range of tools and services. However, GitLab offers more native integrations with third-party services, including cloud providers, deployment platforms, and monitoring tools. This makes it easier for developers to set up their pipelines and automate their workflows. Jenkins also has a vast library of plugins that support integration with various tools and services. 3. Performance GitLab is known for its fast and reliable performance. It has built-in caching and parallel processing capabilities that allow developers to run their pipelines quickly and efficiently. Jenkins, on the other hand, can suffer from performance issues when running large and complex pipelines. It requires manual optimization to ensure it can handle the load. 4. Security GitLab has built-in security features that ensure code is secure at every pipeline stage. It provides features, like code scanning, vulnerability management, and container scanning, that help developers identify and fix security issues before they make it into production. Jenkins relies heavily on plugins for security features. This can make it challenging to ensure your pipeline is secure, especially if you are using third-party plugins. 5. Cost GitLab offers free and paid plans. The free plan includes most features a small team would need for CI/CD. The paid plans include additional features like deployment monitoring, auditing, and compliance. Jenkins is an open-source tool that is free to use. However, it requires significant resources to set up and maintain, which can add to the overall cost of using the tool. The Technical Difference Between GitLab and Jenkins Feature GitLab Jenkins Version Control Git N/A (requires integration with a separate VCS tool). Continuous Integration Yes, built-in. Yes, built-in. Continuous Delivery Yes, built-in. Requires plugins or scripting. Security Built-in security features. Requires plugins or scripting. Code Review Built-in code review features. Requires plugins or scripting. Performance Generally faster due to built-in Git repository May require additional resources for performance Scalability Scales well for small to medium-sized teams. Scales well for large teams. Cost Free for self-hosted and cloud-hosted versions. Free for self-hosted and has a cost for cloud-hosted. Community Active open-source community and enterprise support. Active open-source community and enterprise support. GitLab vs Jenkins: Which One Is Best? GitLab and Jenkins are two popular tools used in the software development process. However, it’s difficult to say which one is better as it depends on the specific needs of your project and organization. GitLab is a complete DevOps platform that includes source code management, continuous integration/continuous delivery (CI/CD), and more. It offers features such as Git repository management, issue tracking, code review, and continuous integration/continuous delivery (CI/CD) pipelines. GitLab also has a built-in container registry and Kubernetes integration, making it easy to deploy applications to container environments. On the other hand, Jenkins is a popular open-source automation server widely used for continuous integration and continuous delivery (CI/CD) pipelines. It offers several plugins for various functionalities, such as code analysis, testing, deployment, and monitoring. Jenkins can be easily integrated with other tools in the software development process, such as Git, GitHub, and Bitbucket. Ultimately, the choice between GitLab and Jenkins will depend on your specific needs and preferences. GitLab is an all-in-one solution, while Jenkins is more flexible and can be customized with plugins. GitLab is a better choice if you want an integrated solution with an intuitive interface and built-in features. Jenkins is better if you want a customizable and extensible automation server that can be easily integrated with other tools in your workflow. Conclusion GitLab and Jenkins are excellent CI/CD tools that offer a range of features and integrations. However, GitLab has the edge when it comes to ease of use, integration, performance, security, and cost. GitLab’s all-in-one platform makes it easy for developers to set up and configure their pipelines, while its native integrations and built-in features make it more efficient and secure than Jenkins. Therefore, if you are looking for a CI/CD tool that is easy to use, cost-effective, and reliable, GitLab is the best option for your organization.

One of the most useful features of real-time stream processing is to combine the strengths and advantages of various technologies to provide a unique developer experience and an efficient way of processing data in real-time at scale. Hazelcast is a real-time distributed computation and storage platform for consistently low latency queries, aggregation, and stateful computation against real-time event streams and traditional data sources. Apache Pulsar is a real-time multitenant geo-replicated distributed pub-sub messaging and streaming platform for real-time workloads, handling millions of events per hour. However, real-time stream processing is not an easy task, especially when combining multiple live streams with large volumes of data stored in external data storages to provide context and instant results. When it comes to usage, Hazelcast can be used for the following things: Stateful data processing over real-time streaming data. Data at rest. A combination of data at rest and stateful data processing over real-time streaming data. Querying streaming. Batch data sources directly using SQL. Distributed coordination for microservices. Replicating data from one region to another. Replicating data between data centers in the same region. While Apache Pulsar can be used for messaging and streaming, use cases take the place of multiple products and provide a superset of their features. Apache Pulsar is a cloud-native multitenant unified messaging platform to replace Apache Kafka, RabbitMQ, MQTT, and legacy messaging platforms. Apache Pulsar provides an infinite message bus for Hazelcast to act as an instant source and sink for any and all data sources. Prerequisites We’re building an application where we ingest data from Apache Pulsar into Hazelcast and then process it in real time. To run this application, make sure your system has the following components: Hazelcast installed on your system: we’re using CLI. Pulsar installed on your system: we’re using Docker. If you have macOS and Homebrew, you can install Hazelcast using the following command: Shell brew tap hazelcast/hz brew install hazelcast@5.2.1 Check if Hazelcast is installed: hz -V Then start a local cluster: hz start You should see the following in the console: Shell INFO: [192.168.1.164]:5701 [dev] [5.2.1] Members {size:1, ver:1} [ Member [192.168.1.164]:5701 - 4221d540-e34e-4ff2-8ad3-41e060b895ce this ] You can start Pulsar in Docker using the following command: Shell docker run -it -p 6650:6650 -p 8080:8080 \ --mount source=pulsardata,target=/pulsar/data \ --mount source=pulsarconf,target=/pulsar/conf \ apachepulsar/pulsar:2.11.0 bin/pulsar standalone To install Management Center, use one of the following methods, depending on your operating system: Shell brew tap hazelcast/hz brew install hazelcast-management-center@5.2.1 Check that Management Center is installed: hz-mc -V Data Collection For our application, we wish to ingest air quality readings from around the United States via the AirNow data provider. Source: AirNow API With a simple Java application, we make REST calls to the AirNow API that provides air quality reading for major zip codes around the United States. The Java application sends the JSON-encoded AirNow data to the “airquality” Pulsar topic. From this point, a Hazelcast application can read it. Source: GitHub We also have a Java Pulsar function receiving each event from the “airquality” topic and parsing it into different topics based on which type of air quality reading it is. This includes PM2.5, PM10, and Ozone. Source: GitHub Example AirQuality Data JSON {"dateObserved":"2023-01-19 ","hourObserved":12,"localTimeZone":"EST","reportingArea":"Philadelphia","stateCode":"PA","latitude":39.95,"longitude":-75.151,"parameterName":"PM10","aqi":19,"category":{"number":1,"name":"Good","additionalProperties":{},"additionalProperties":{} Example Ozone Data JSON {"dateObserved":"2023-01-19 ","hourObserved":12,"localTimeZone":"EST","reportingArea":"Philadelphia","stateCode":"PA","parameterName":"O3","latitude":39.95,"longitude":-75.151,"aqi":8} Example PM10 Data JSON {"dateObserved":"2023-01-19 ","hourObserved":12,"localTimeZone":"EST","reportingArea":"Philadelphia","stateCode":"PA","parameterName":"PM10","latitude":39.95,"longitude":-75.151,"aqi":19} Example PM2.5 Data JSON {"dateObserved":"2023-01-19 ","hourObserved":12,"localTimeZone":"EST","reportingArea":"Philadelphia","stateCode":"PA","parameterName":"PM2.5","latitude":39.95,"longitude":-75.151,"aqi":54} Data Processing In order to process the data collected, we used the Hazelcast Pulsar connector module to ingest data from Pulsar topics. Note: you can use the same connector to write to Pulsar topics. Using Hazelcast allows us to compute various aggregation functions (sum, avg, etc.) in real-time on a specified window of stream items. The Pulsar connector uses the Pulsar client library, which has two different ways of reading messages from a Pulsar topic. These are Consumer API and Reader API; both use the builder pattern (for more information, click here). In your pom file, import the following dependencies: XML <dependency> <groupId>com.hazelcast</groupId> <artifactId>hazelcast</artifactId> <version>5.1.4</version> </dependency> <dependency> <groupId>com.hazelcast.jet.contrib</groupId> <artifactId>pulsar</artifactId> <version>0.1</version> </dependency> <dependency> <groupId>org.apache.pulsar</groupId> <artifactId>pulsar-client</artifactId> <version>2.10.1</version> </dependency> We create a PulsarSources.pulsarReaderBuilder instance to connect to the previously started Pulsar cluster located at pulsar://localhost:6650. Java StreamSource<Event>source = PulsarSources.pulsarReaderBuilder( topicName, () -> PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build(), () -> Schema.JSON(Event.class), Message::getValue).build(); We then create a pipeline to read from the source with a sliding window and aggregate count before we write to logger: Java Pipeline p = Pipeline.create(); p.readFrom(source) .withNativeTimestamps(0) .groupingKey(Event::getUser) .window(sliding(SECONDS.toMillis(60), SECONDS.toMillis(30))) .aggregate(counting()) .writeTo(Sinks.logger(wr -> String.format( "At %s Pulsar got %,d messages in the previous minute from %s.", TIME_FORMATTER.format(LocalDateTime.ofInstant( Instant.ofEpochMilli(wr.end()), ZoneId.systemDefault())), wr.result(), wr.key()))); JobConfig cfg = new JobConfig() .setProcessingGuarantee(ProcessingGuarantee.EXACTLY_ONCE) .setSnapshotIntervalMillis(SECONDS.toMillis(1)) .setName("pulsar-airquality-counter"); HazelcastInstance hz = Hazelcast.bootstrappedInstance(); hz.getJet().newJob(p, cfg); You can run the previous code from your IDE (in this case, it will create its own Hazelcast member and run the job on it), or you can run this on the previously started Hazelcast member (in this case, you need to create a runnable JAR including all dependencies required to run it): Shell mvn package bin/hz-cli submit target/pulsar-example-1.0-SNAPSHOT.jar To cancel the job and shut down the Hazelcast cluster: Shell bin/hz-cli cancel pulsar-message-counter hz-stop Conclusion In this article, we demonstrated how you can combine the strengths and advantages of various technologies to provide a unique developer experience and an efficient way of processing data in real-time at scale. We streamed air quality data from Apache Pulsar into Hazelcast, where we processed data in real-time. The rising trend in cloud technologies, the need for real-time intelligent applications, and the urgency to process data at scale have brought us to a new chapter of real-time stream processing, where latencies are measured, not in minutes but in milliseconds and submillisecond’s. Hazelcast allows you to quickly build resource-efficient, real-time applications. You can deploy it at any scale, from small-edge devices to a large cluster of cloud instances. A cluster of Hazelcast nodes shares the data storage and computational load, which can dynamically scale up and down. When you add new nodes to the cluster, the data is automatically rebalanced across the cluster, and currently running computational tasks (known as jobs) snapshot their state and scale with processing guarantees. Pulsar allows you to use your choice of messaging protocols to quickly distribute events between multiple types of consumers and producers and act as a universal message hub. Pulsar separates compute from storage, allowing for dynamic scaling and efficient handling of fast data. StreamNative is the company made up of the original creators of Apache Pulsar and Apache BookKeeper. StreamNative provides a full enterprise experience for Apache Pulsar in the cloud and on-premise.

We will go over Apache Kafka basics, installation, and operation, as well as a step-by-step implementation using a .NET Core 6 web application. Prerequisites Visual Studio 2022 .NET Core 6 SDK SQL Server Java JDK 11 Apache Kafka Agenda Overview of Event Streaming Introduction to Apache Kafka. Main concepts and foundation of Kafka. Different Kafka APIs. Use cases of Apache Kafka. Installation of Kafka on Windows 10. Step-by-step implementation Overview of Event Streaming Events are the things that happen within our application when we navigate something. For example, we sign up on any website and order something, so, these are the events. The event streaming platform records different types of data like transaction, historical, and real-time data. This platform is also used to process events and allow different consumers to process results immediately and in a timely manner. An event-driven platform allows us to monitor our business and real-time data from different types of devices like IoT and many more. After analyzing, it provides a good customer experience based on different types of events and needs. Introduction to Apache Kafka Below, are a few bullet points that describe Apache Kafka: Kafka is a distributed event store and stream-processing platform. Kafka is open source and is written in Java and Scala. The primary purpose to designed Kafka by Apache foundation is to handle real-time data feeds and provide high throughput and low latency platforms. Kafka is an event streaming platform that has many capabilities to publish (write) and subscribe to (read) streams of events from a different system. Also, to store and process events durably as long as we want, by default, Kafka stores events from seven days of the time period, but we can increase that as per need and requirement. Kafka has distributed system, which has servers and clients that can communicate via TCP protocol. It can be deployed on different virtual machines and containers in on-premise and cloud environments as per requirements. In the Kafka world, a producer sends messages to the Kafka broker. The messages will get stored inside the topics and the consumer subscribes to that topic to consume messages sent by the producer. ZooKeeper is used to manage the metadata of Kafka-related things, it tracks which brokers are part of the Kafka cluster and partitions of different topics. Lastly, it manages the status of Kafka nodes and maintains a list of Kafka topics and messages. Main Concepts and Foundation of Kafka 1. Event An event or record is the message that we read and write to the Kafka server; we do this in the form of events in our business world, and it contains a key, a value, a timestamp, and other metadata headers. The key, value, and time stamp, in this case, are as follows: Key: “Jaydeep” Value: “Booked BMW” Event Timestamp: “Dec. 11, 2022, at 12:00 p.m.” 2. Producer The producer is a client application that sends messages to the Kafka node or broker. 3. Consumer The consumer is an application that receives data from Kafka. 4. Kafka Cluster The Kafka cluster is the set of computers that share the workload with each other with varying purposes. 5. Broker The broker is a Kafka server that acts as an agent between the producer and consumer, who communicate via the broker. 6. Topic The events are stored inside the “topic,” it’s similar to our folder in which we store multiple files. Each topic has one or more producers and consumers, which write and reads data from the topic. Events in “topic” can be read as often as needed because it persists events and it’s not like another messaging system that removes messages after consuming. 7. Partitions Topics are partitions, meaning the topic is spread over multiple partitions that we created inside the topic. When the producer sends some event to the topic, it will store it inside the particular partitions, and then, the consumer can read the event from the corresponding topic partition in sequence. 8. Offset Kafka assigns one unique ID to the message stored inside the topic partition when the message arrives from the producer. 9. Consumer Groups In the Kafka world, the consumer group acts as a single logical unit. 10. Replica In Kafka, to make data fault-tolerant and highly available, we can replicate topics in different regions and brokers. So, in case something wrong happens with data in one topic, we can easily get that from another to replicate the same. Different Kafka APIs Kafka has five core APIs that serve different purposes: Admin API: This API manages different topics, brokers, and Kafka objects. Producer API: This API is used to write/publish events to different Kafka topics. Consumer API: This API is used to receive the different messages corresponding to the topics that are subscribed by the consumer. Kafka Stream API: This API is used to perform different types of operations like windowing, joins, aggregation, and many others. Basically, its use is to transform objects. Kafka Connect API: This API works as a connector to Kafka, which helps different systems connect with Kafka easily. It has different types of ready-to-use connectors related to Kafka. Use Cases of Apache Kafka Messaging User activity tracking Log aggregation Stream processing Realtime data analytics Installation of Kafka on Windows 10 Step 1 Download and install the Java SDK of version 8 or more. Note: I have Java 11, that’s why I put the same path in all commands that I used here. Step 2 Open and install EXE. Step 3 Set the environment variable for Java using the command prompt as admin. Command: setx -m JAVA_HOME “C:\Program Files\Java\jdk-11.0.16.1” setx -m PATH “%JAVA_HOME%\bin;%PATH%” Step 4 After that, download and install Apache Kafka. Step 5 Extract the downloaded Kafka file and rename it “Kafka.” Step 6 Open D:\Kafka\config\ and create a “zookeeper-data” and “kafka-logs” folder inside that. Step 7 Next, open D:\Kafka\config\zookeeper.properties file and add the folder path inside that: D:\Kafka\config\zookeeper.properties dataDir=D:/Kafka/zookeeper-data Step 8 After that, open D:\Kafka\config\server.properties file and change the log path over there: D:\Kafka\config\server.properties log.dirs=D:/Kafka/kafka-logs Step 9 Saves and close both files. Step 10 Run ZooKeeper: D:\Kafka> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties Step 11 Start Kafka: D:\Kafka> .\bin\windows\kafka-server-start.bat .\config\server.properties Step 12 Create Kafka topic: D:\Kafka\bin\windows>kafka-topics.bat — create — bootstrap-server localhost:9092 — replication-factor 1 — partitions 1 — topic testdata Step 13 Create a producer and send some messages after you’ve started a producer and consumer: D:\Kafka\bin\windows>kafka-console-producer.bat — broker-list localhost:9092 — topic testdata Step 14 Next, create a consumer. After, you will see the message the producer sent: D:\Kafka\bin\windows>kafka-console-consumer.bat — bootstrap-server localhost:9092 — topic testdata Step-by-Step Implementation Let’s start with practical implementation. Step 1 Create a new .NET Core Producer Web API: Step 2 Configure your application: Step 3 Provide additional details: Step 4 Install the following two NuGet packages: Step 5 Add configuration details inside the appsettings.json file: JSON { "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } }, "AllowedHosts": "*", "producerconfiguration": { "bootstrapservers": "localhost:9092" }, "TopicName": "testdata" } Step 6 Register a few services inside the “Program” class: C# using Confluent.Kafka; var builder = WebApplication.CreateBuilder(args); // Add services to the container. var producerConfiguration = new ProducerConfig(); builder.Configuration.Bind("producerconfiguration", producerConfiguration); builder.Services.AddSingleton<ProducerConfig>(producerConfiguration); builder.Services.AddControllers(); // Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); var app = builder.Build(); // Configure the HTTP request pipeline. if (app.Environment.IsDevelopment()) { app.UseSwagger(); app.UseSwaggerUI(); } app.UseHttpsRedirection(); app.UseAuthorization(); app.MapControllers(); app.Run(); Step 7 Next, create the CarDetails model class: C# using Microsoft.AspNetCore.Authentication; namespace ProducerApplication.Models { public class CarDetails { public int CarId { get; set; } public string CarName { get; set; } public string BookingStatus { get; set; } } } Step 8 Now, create the CarsController class: C# using Confluent.Kafka; using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.Configuration; using Newtonsoft.Json; using ProducerApplication.Models; namespace ProducerApplication.Controllers { [Route("api/[controller]")] [ApiController] public class CarsController : ControllerBase { private ProducerConfig _configuration; private readonly IConfiguration _config; public CarsController(ProducerConfig configuration, IConfiguration config) { _configuration = configuration; _config = config; } [HttpPost("sendBookingDetails")] public async Task<ActionResult> Get([FromBody] CarDetails employee) { string serializedData = JsonConvert.SerializeObject(employee); var topic = _config.GetSection("TopicName").Value; using (var producer = new ProducerBuilder<Null, string>(_configuration).Build()) { await producer.ProduceAsync(topic, new Message<Null, string> { Value = serializedData }); producer.Flush(TimeSpan.FromSeconds(10)); return Ok(true); } } } } Step 9 Finally, run the application and send a message: Step 10 Now, create a “consumer” application: For that, create a new .NET Core console application: Step 11 Configure your application: Step 12 Provide additional information: Step 13 Install the NuGet below: Step 14 Add the following code, which consumes messages sent by the consumer: C# using Confluent.Kafka; var config = new ConsumerConfig { GroupId = "gid-consumers", BootstrapServers = "localhost:9092" }; using (var consumer = new ConsumerBuilder<Null, string>(config).Build()) { consumer.Subscribe("testdata"); while (true) { var bookingDetails = consumer.Consume(); Console.WriteLine(bookingDetails.Message.Value); } } Step 15 Finally, run the producer and consumer, send a message using the producer app, and you will see the message immediately inside the consumer console sent by the producer: Here is the GitHub URL I used in this article. Conclusion Here, we discussed Apache Kafka introduction, working, benefits, and step-by-step implementation using .NET Core 6. Happy Coding!

Trino is an open-source distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources. Trino was designed to handle data warehousing, ETL, and interactive analytics by large amounts of data and producing reports. Alluxio is an open-source data orchestration platform for large-scale analytics and AI. Alluxio sits between compute frameworks such as Trino and Apache Spark and various storage systems like Amazon S3, Google Cloud Storage, HDFS, and MinIO. This is a tutorial for deploying Alluxio as the caching layer for Trino using the Iceberg connector. Why Do We Need Caching for Trino? A small fraction of the petabytes of data you store is generating business value at any given time. Repeatedly scanning the same data and transferring it over the network consumes time, compute cycles, and resources. This issue is compounded when pulling data from disparate Trino clusters across regions or clouds. In these circumstances, caching solutions can significantly reduce the latency and cost of your queries. Trino has a built-in caching engine, Rubix, in its Hive connector. While this system is convenient as it comes with Trino, it is limited to the Hive connector and has not been maintained since 2020. It also lacks security features and support for additional compute engines. Trino on Alluxio Alluxio connects Trino to various storage systems, providing APIs and a unified namespace for data-driven applications. Alluxio allows Trino to access data regardless of the data source and transparently cache frequently accessed data (e.g., tables commonly used) into Alluxio distributed storage. Using Alluxio Caching via the Iceberg Connector Over MinIO File Storage We’ve created a demo that demonstrates how to configure Alluxio to use write-through caching with MinIO. This is achieved by using the Iceberg connector and making a single change to the location property on the table from the Trino perspective. In this demo, Alluxio is run on separate servers; however, it’s recommended to run it on the same nodes as Trino. This means that all the configurations for Alluxio will be located on the servers where Alluxio runs, while Trino’s configuration remains unaffected. The advantage of running Alluxio externally is that it won’t compete for resources with Trino, but the disadvantage is that data will need to be transferred over the network when reading from Alluxio. It is crucial for performance that Trino and Alluxio are on the same network. To follow this demo, copy the code located here. Trino Configuration Trino is configured identically to a standard Iceberg configuration. Since Alluxio is running external to Trino, the only configuration needed is at query time and not at startup. Alluxio Configuration The configuration for Alluxio can all be set using the alluxio-site.properties file. To keep all configurations colocated on the docker-compose.yml, we are setting them using Java properties via the ALLUXIO_JAVA_OPTS environment variable. This tutorial also refers to the master node as the leader and the workers as followers. Master Configurations alluxio.master.mount.table.root.ufs=s3://alluxio/ The leader exposes ports 19998 and 19999, the latter being the port for the web UI. Worker Configurations alluxio.worker.ramdisk.size=1G alluxio.worker.hostname=alluxio-follower The follower exposes ports 29999 and 30000, and sets up a shared memory used by Alluxio to store data. This is set to 1G via the shm_size property and is referenced from the alluxio.worker.ramdisk.size property. Shared Configurations Between Leader and Follower alluxio.master.hostname=alluxio-leader # Minio configs alluxio.underfs.s3.endpoint=http://minio:9000 alluxio.underfs.s3.disable.dns.buckets=true alluxio.underfs.s3.inherit.acl=false aws.accessKeyId=minio aws.secretKey=minio123 # Demo-only configs alluxio.security.authorization.permission.enabled=false The alluxio.master.hostname needs to be on all nodes, leaders and followers. The majority of shared configs points Alluxio to the underfs, which is MinIO in this case. alluxio.security.authorization.permission.enabled is set to “false” to keep the Docker setup simple. Note: This is not recommended to do in a production or CI/CD environment. Running Services First, you want to start the services. Make sure you are in the trino-getting-started/iceberg/trino-alluxio-iceberg-minio directory. Now, run the following command: docker-compose up -d You should expect to see the following output. Docker may also have to download the Docker images before you see the “Created/Started” messages, so there could be extra output: [+] Running 10/10 ⠿ Network trino-alluxio-iceberg-minio_trino-network Created 0.0s ⠿ Volume "trino-alluxio-iceberg-minio_minio-data" Created 0.0s ⠿ Container trino-alluxio-iceberg-minio-mariadb-1 Started 0.6s ⠿ Container trino-alluxio-iceberg-minio-trino-coordinator-1 Started 0.7s ⠿ Container trino-alluxio-iceberg-minio-alluxio-leader-1 Started 0.9s ⠿ Container minio Started 0.8s ⠿ Container trino-alluxio-iceberg-minio-alluxio-follower-1 Started 1.5s ⠿ Container mc Started 1.4s ⠿ Container trino-alluxio-iceberg-minio-hive-metastore-1 Started Open Trino CLI Once this is complete, you can log into the Trino coordinator node. We will do this by using the exec command and run the trino CLI executable as the command we run on that container. Notice the container id is trino-alluxio-iceberg-minio-trino-coordinator-1, so the command you will run is: <<<<<<< HEAD docker container exec -it trino-alluxio-iceberg-minio-trino-coordinator-1 trino ======= docker container exec -it trino-minio_trino-coordinator_1 trino >>>>>>> alluxio When you start this step, you should see the trino cursor once the startup is complete. It should look like this when it is done: trino> To best understand how this configuration works, let’s create an Iceberg table using a CTAS (CREATE TABLE AS) query that pushes data from one of the TPC connectors into Iceberg that points to MinIO. The TPC connectors generate data on the fly so we can run simple tests like this. First, run a command to show the catalogs to see the tpch and iceberg catalogs since these are what we will use in the CTAS query: SHOW CATALOGS; You should see that the Iceberg catalog is registered. MinIO Buckets and Trino Schemas Upon startup, the following command is executed on an intiailization container that includes the mc CLI for MinIO. This creates a bucket in MinIO called /alluxio, which gives us a location to write our data to and we can tell Trino where to find it: /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 minio minio123) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/alluxio; /usr/bin/mc mb minio/alluxio; /usr/bin/mc policy set public minio/alluxio; exit 0; " Note: This bucket will act as the mount point for Alluxio, so the schema directory alluxio://lakehouse/ in Alluxio will map to s3://alluxio/lakehouse/. Querying Trino Let’s move to creating our SCHEMA that points us to the bucket in MinIO and then run our CTAS query. Back in the terminal, create the iceberg.lakehouse SCHEMA. This will be the first call to the metastore to save the location of the schema location in the Alluxio namespace. Notice, we will need to specify the hostname alluxio-leader and port 19998 since we did not set Alluxio as the default file system. Take this into consideration if you want Alluxio caching to be the default usage and transparent to users managing DDL statements: CREATE SCHEMA iceberg.lakehouse WITH (location = 'alluxio://alluxio-leader:19998/lakehouse/'); Now that we have a SCHEMA that references the bucket where we store our tables in Alluxio, which syncs to MinIO, we can create our first table. Optional: To view your queries run, log into the Trino UI and log in using any username (it doesn’t matter since no security is set up). Move the customer data from the tiny generated TPCH data into MinIO using a CTAS query. Run the following query, and if you like, watch it running on the Trino UI: CREATE TABLE iceberg.lakehouse.customer WITH ( format = 'ORC', location = 'alluxio://alluxio-leader:19998/lakehouse/customer/' ) AS SELECT * FROM tpch.tiny.customer; Go to the Alluxio UI and the MinIO UI, and browse the Alluxio and MinIO files. You will now see a lakehouse directory that contains a customer directory that contains the data written by Trino to Alluxio and Alluxio writing it to MinIO. Now, there is a table under Alluxio and MinIO, you can query this data by checking the following: SELECT * FROM iceberg.lakehouse.customer LIMIT 10; How are we sure that Trino is actually reading from Alluxio and not MinIO? Let’s delete the data in MinIO and run the query again just to be sure. Once you delete this data, you should still see data return. Stopping Services Once you complete this tutorial, the resources used for this excercise can be released by runnning the following command: docker-compose down Conclusion At this point, you should have a better understanding of Trino and Alluxio, how to get started with deploying Trino and Alluxio, and how to use Alluxio caching with an Iceberg connector and MinIO file storage. I hope you enjoyed this article. Be sure to like this article and comment if you have any questions!

Say you are building a cloud app. Clearly, you will unit test the app during development to ensure it behaves the way you expect. And no matter how you develop your app, your favorite unit testing tool will make it easy to author, execute, and obtain results from your tests. What about testing your app when you deploy it in a Kubernetes cluster (test/dev/staging/prod)? Does the app handle realistic load conditions with acceptable performance? Does the new version of the app improve business metrics relative to the earlier version? Is it resilient? In this article, I will introduce Iter8 and one of its new features, AutoX. Iter8 is an open-source Kubernetes release optimizer that can help you get started with testing of Kubernetes apps (experiments) in seconds. With Iter8, you can perform various kinds of experiments, such as performance testing, A/B/n test, chaos injection test, and more. Iter8 recently introduced a new feature: AutoX. AutoX, short for automatic experiments, allows you to perform the above experiments automatically by leveraging Argo CD, a popular continuous delivery tool. We will explore automatically launching performance testing experiments for an HTTP service deployed in Kubernetes. You can familiarize yourself with Iter8 on Iter8’s official website. AutoX Releasing a new version of an application typically involves the creation of new Kubernetes resource objects and/or updates to existing ones. AutoX can be configured to watch for such changes and automatically launch new experiments. You can configure AutoX with multiple experiment groups and, for each group, specify the Kubernetes resource object that AutoX will watch and one or more experiments to be performed in response to new versions of this object. Let us now see this in action using a Kubernetes HTTP service and configuring AutoX, so whenever a new version of the service is released, AutoX will start a new HTTP performance test that will validate if the service meets latency and error-related requirements. Download Iter8 CLI Shell brew tap iter8-tools/iter8 brew install iter8@0.13 The Iter8 CLI provides the commands needed to see experiment reports. Setup the Kubernetes Cluster With ArgoCD As mentioned previously, AutoX is built on top of ArgoCD, so we will also need to install it. A basic install of Argo CD can be done as follows: Shell kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml Deploy Application Now, we will create an httpbin deployment and service: Shell kubectl create deployment httpbin --image=kennethreitz/httpbin --port=80 kubectl expose deployment httpbin --port=80 Apply the Version Label Next, we will assign httpbin deployment the app.kubernetes.io/version label (version label). AutoX will launch experiments only when this label is present on your trigger object. It will relaunch experiments whenever this version label is modified. Shell kubectl label deployment httpbin app.kubernetes.io/version=1.0.0 Setup Kubernetes Cluster With Iter8 AutoX Next, we will configure and install the AutoX controller: Shell helm install autox autox --repo https://iter8-tools.github.io/hub/ --version 0.1.6 \ --set 'groups.httpbin.trigger.name=httpbin' \ --set 'groups.httpbin.trigger.namespace=default' \ --set 'groups.httpbin.trigger.group=apps' \ --set 'groups.httpbin.trigger.version=v1' \ --set 'groups.httpbin.trigger.resource=deployments' \ --set 'groups.httpbin.specs.iter8.name=iter8' \ --set 'groups.httpbin.specs.iter8.values.tasks={ready,http,assess}' \ --set 'groups.httpbin.specs.iter8.values.ready.deploy=httpbin' \ --set 'groups.httpbin.specs.iter8.values.ready.service=httpbin' \ --set 'groups.httpbin.specs.iter8.values.ready.timeout=60s' \ --set 'groups.httpbin.specs.iter8.values.http.url=http://httpbin.default/get' \ --set 'groups.httpbin.specs.iter8.values.assess.SLOs.upper.http/error-count=0' \ --set 'groups.httpbin.specs.iter8.values.assess.SLOs.upper.http/latency-mean=50' \ --set 'groups.httpbin.specs.iter8.version=0.13.0' \ --set 'groups.httpbin.specs.iter8.values.runner=job' The configuration of the AutoX controller is composed of a trigger object definition and a set of experiment specifications. In this case, the trigger object is the httpbin deployment and there is only one experiment, an HTTP performance test with SLO validation associated with this trigger. To go into more detail, the configuration is a set of groups, and each group is composed of a trigger object definition and a set of experiment specifications. This enables AutoX to manage one or more trigger objects, each associated with one or more experiments. In this tutorial, there is only one group named httpbin (groups.httpbin...), and within that group, there is the trigger object definition (groups.httpbin.trigger...), and a single experiment spec named iter8 (groups.httpbin.specs.iter8...). The trigger object definition is a combination of the name, namespace, and the group-version-resource (GVR) metadata of the trigger object. In this case, httpbin, default, and GVR apps, deployments, and v1, respectively. The experiment is an HTTP SLO validation test on the httpbin service. This Iter8 experiment is composed of three tasks, ready, http, and assess. The ready task will ensure that the httpbin deployment and service are running. The http task will make requests to the specified URL and will collect latency and error-related metrics. Lastly, the assess task will ensure that the mean latency is less than 50 milliseconds and the error count is “0.” In addition, the runner is set to “job” as this will be a single-loop experiment. Observe Experiment After starting AutoX, the HTTP SLO validation test should quickly follow. You can now use the Iter8 CLI to check the status and see the results of the test. The following command allows you to check the status of the test. Note: You need to specify an experiment group via the -g option. The experiment group for experiments started by AutoX is in the form autox-<group name>-<experiment spec name>, so, in this case, it would be autox-httpbin-iter8: Shell iter8 k assert -c completed -c nofailure -c slos -g autox-httpbin-iter8 We can see in the sample output that the test has been completed, there were no failures, and all SLOs and conditions were satisfied. Plain Text INFO[2023-01-11 14:43:45] inited Helm config INFO[2023-01-11 14:43:45] experiment completed INFO[2023-01-11 14:43:45] experiment has no failure INFO[2023-01-11 14:43:45] SLOs are satisfied INFO[2023-01-11 14:43:45] all conditions were satisfied The following command allows you to see the results as a text report: Shell iter8 k report -g autox-httpbin-iter8 You can also produce an HTML report that you can view in the browser: Shell iter8 k report -g autox-httpbin-iter8 -o html > report.html The HTML report will look similar to the following: Continuous and Automated Experimentation Now that AutoX is watching the httpbin deployment, the new version will relaunch the HTTP SLO validation test. The version update must be accompanied by a change to the deployment’s app.kubernetes.io/version label (version label); otherwise, AutoX will not do anything. For simplicity, we will change the version label to the deployment to relaunch the HTTP SLO validation test. In the real world, a new version would typically involve a change to the deployment spec (e.g., the container image) and this change should be accompanied by a change to the version label: Shell kubectl label deployment httpbin app.kubernetes.io/version=2.0.0 --overwrite Observe the New Experiment Check if a new experiment has been launched. Refer to the previous “Observe Experiment” section for the necessary commands. If we were to continue to update the deployment (and change its version label), then AutoX would relaunch the experiment for each such change. More Things You Can Do Firstly, the HTTP SLO validation test is flexible, and you can augment it in a number of ways, such as adding headers, providing a payload, or modulating the query rate. To learn more, see the documentation for the http task.AutoX is designed to use any Kubernetes resource object (including those with a custom resource type) as a trigger object in AutoX. For example, the trigger object can be a Knative service, a KServe inference service, or a Seldon deployment.AutoX is designed to automate a variety of experiments. For example, instead of using the http task, you can use grpc task in order to run an gRPC SLO validation test. Here is the documentation for the grpc task as well as a tutorial for gRPC SLO Validation.Furthermore, you can add additional tasks that ship out-of-the-box with Iter8, in order to enrich the experiments. For example, you can add a slack task so that your experiment results will be posted on Slack. That way, you can automatically have the latest performance statistics after every update. Here is the documentation for the slack task as well as a tutorial for using the Slack task.You can also automate experiments that are not from Iter8. For example, a Litmus Chaos chaos experiment is available on Iter8 hub, which can also be configured with AutoX.Lastly, recall that you can provide multiple groups and experiment specs so AutoX can launch and manage a whole suite of experiments for multiple Kubernetes applications and namespaces. Takeaways AutoX is a powerful new feature of Iter8 that lets you automatically launch performance experiments on your Kubernetes applications as soon as you release a new version. Configuring AutoX is straightforward, and requires specifying a trigger Kubernetes resource object and the experiments you want to associate with this trigger object. After trying out the tutorial, consider trying it out on your own Kubernetes apps.

In earlier days, it was easy to deduct and debug a problem in monolithic applications because there was only one service running in the back end and front end. Now, we are moving toward microservices architecture, where applications are divided into multiple independently deployable services. These services have their own goal and logic to serve. In this kind of application architecture, it becomes difficult to observe how one service depends on or affects other services. To make the system observable, some logs, metrics, or traces must be emitted from the code, and this data must be sent to an observability back end. This is where OpenTelemetry and Jaeger come into the picture. In this article, we will see how to monitor application trace data (Traces and Spans) with the help of OpenTelemetry and Jaeger. A trace is used to observe the requests as they propagate through the services in a distributed system. Spans are a basic unit of the trace; they represent a single event within the trace, and a trace can have one or multiple spans. A span consists of log messages, time-related data, and other attributes to provide information about the operation it tracks. We will use the distributed tracing method to observe requests moving across microservices, generating data about the request and making it available for analysis. The produced data will have a record of the flow of requests in our microservices, and it will help us understand our application's performance. OpenTelemetry Telemetry is the collection and transmission of data using agents and protocols from the source in observability. The telemetry data includes logs, metrics, and traces, which help us understand what is happening in our application. OpenTelemetry (also known as OTel) is an open source framework comprising a collection of tools, APIs, and SDKs. OpenTelemetry makes generating, instrumenting, collecting, and exporting telemetry data easy. The data collected from OpenTelemetry is vendor-agnostic and can be exported in many formats. OpenTelemetry is formed after merging two projects OpenCensus and OpenTracing. Instrumenting The process of adding observability code to your application is known as instrumentation. Instrumentation helps make our application observable, meaning the code must produce some metrics, traces, and logs. OpenTelemetry provides two ways to instrument our code: Manual instrumentation Auto instrumentation 1. Manual Instrumentation The user needs to add an OpenTelemetry code to the application. The manual instrumentation provides more options for customization in spans and traces. Languages supported for manual instrumentations are C++, .NET, Go, Java, Python, and so on. 2. Automatic Instrumentation This is the easiest way of instrumentation as it requires no code changes and no need to recompile the application. It uses an intelligent agent that gets attached to an application, reads its activity, and extracts the traces. Automatic instrumentation supports Java, NodeJS, Python, and so on. Difference Between Manual and Automatic Instrumentation Both manual and automatic instrumentation have advantages and disadvantages that you might consider while writing your code. A few of them are listed below: Manual Instrumentation Automatic Instrumentation Code changes are required. Code changes are not required. It supports maximum programming languages. Currently, .Net, Java, NodeJS, and Python are supported. It consumes a lot of time as code changes are required. Easy to implement as we do not need to touch the code. Provide more options for the customization of spans and traces. As you have more control over the telemetry data generated by your application. Fewer options for customization. Possibilities of error are high as manual changes are required. No error possibilities. As we don't have to touch our application code. To make the instrumentation process hassle-free, use automatic instrumentation, as it does not require any modification in the code and reduces the possibility of errors. Automatic instrumentation is done by an agent which reads your application's telemetry data, so no manual changes are required. For the scope of this post, we will see how you can use automatic instrumentation in a Kubernetes-based microservices environment. Jaeger Jaeger is a distributed tracing tool initially built by Uber and released as open-source in 2015. Jaeger is also a Cloud Native Computing Foundation graduate project and was influenced by Dapper and OpenZipkin. It is used for monitoring and troubleshooting microservices-based distributed systems. The Jaeger components which we have used for this blog are: Jaeger Collector Jaeger Query Jaeger UI / Console Storage Backend Jaeger Collector: The Jaeger distributed tracing system includes the Jaeger collector. It is in charge of gathering and keeping the information. After receiving spans, the collector adds them to a processing queue. Collectors need a persistent storage backend, hence Jaeger also provides a pluggable span storage mechanism. Jaeger Query: This is a service used to get traces out of storage. The web-based user interface for the Jaeger distributed tracing system is called Jaeger Query. It provides various features and tools to help you understand the performance and behavior of your distributed application and enables you to search, filter, and visualise the data gathered by Jaeger. Jaeger UI/Console: Jaeger UI lets you view and analyze traces generated by your application. Storage Back End: This is used to store the traces generated by an application for the long term. In this article, we are going to use Elasticsearch to store the traces. What Is the Need for Integrating OpenTelemetry With Jaeger? OpenTelemetry and Jaeger are the tools that help us in setting the observability in microservices-based distributed systems, but they are intended to address different issues. OpenTelemetry provides an instrumentation layer for the application, which helps us generate, collect and export the telemetry data for analysis. In contrast, Jaeger is used to store and visualize telemetry data. OpenTelemetry can only generate and collect the data. It does not have a UI for the visualization. So we need to integrate Jaeger with OpenTelemetry as it has a storage backend and a web UI for the visualization of the telemetry data. With the help of Jaeger UI, we can quickly troubleshoot microservices-based distributed systems. Note: OpenTelemetry can generate logs, metrics, and traces. Jaeger does not support logs and metrics. Now you have an idea about OpenTelemetry and Jaeger. Let's see how we can integrate them with each other to visualize the traces and spans generated by our application. Implementing OpenTelemetry Auto-Instrumentation We will integrate OpenTelemetry with Jaeger, where OpenTelemetry will act as an instrumentation layer for our application, and Jaeger will act as the back-end analysis tool to visualize the trace data. Jaeger will get the telemetry data from the OpenTelemetry agent. It will store the data in the storage backend, from where we will query the stored data and visualize it in the Jaeger UI. Prerequisites for this article are: The target Kubernetes cluster is up and running. You have access to run the kubectl command against the Kubernetes cluster to deploy resources. Cert manager is installed and running. You can install it from the website cert-manager.io if it is not installed. We assume that you have all the prerequisites and now you are ready for the installation. The files we have used for this post are available in this GitHub repo. Installation The installation part contains three steps: Elasticsearch installation Jaeger installation OpenTelemetry installation Elasticsearch By default, Jaeger uses in-memory storage to store spans, which is not a recommended approach for the production environment. There are various tools available to use as a storage back end in Jaeger; you can read about them in the official documentation of Jaeger span storage back end. In this article, we will use Elasticsearch as a storage back end. You can deploy Elasticsearch in your Kubernetes cluster using the Elasticsearch Helm chart. While deploying Elasticsearch, ensure you have enabled the password-based authentication and deploy that Elasticsearch in observability namespaces. Elasticsearch is deployed in our Kubernetes cluster, and you can see the output by running the following command. Shell $ kubectl get all -n observability NAME READY STATUS RESTARTS AGE pod/elasticsearch-0 1/1 Running 0 17m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 17m NAME READY AGE statefulset.apps/elasticsearch 1/1 17m Jaeger Installation We are going to use Jaeger to visualize the trace data. Let's deploy the Jaeger Operator on our cluster. Before proceeding with the installation, we will deploy a ConfigMap in the observability namespace. In this ConfigMap, we will pass the username and password of the Elasticsearch which we have deployed in the previous step. Replace the credentials based on your setup. YAML kubectl -n observability apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: jaeger-configuration labels: app: jaeger app.kubernetes.io/name: jaeger data: span-storage-type: elasticsearch collector: | es: server-urls: http://elasticsearch:9200 username: elastic password: changeme collector: zipkin: http-port: 9411 query: | es: server-urls: http://elasticsearch:9200 username: elastic password: changeme agent: | collector: host-port: "jaeger-collector:14267" EOF If you are going to deploy Jaeger in another namespace and you have changed the Jaeger collector service name, then you need to change the values of the host-port value under the agent collector. Jaeger Operator The Jaeger Operator is a Kubernetes operator for deploying and managing Jaeger, an open source, distributed tracing system. It works by automating the deployment, scaling, and management of Jaeger components on a Kubernetes cluster. The Jaeger Operator uses custom resources and custom controllers to extend the Kubernetes API with Jaeger-specific functionality. It manages the creation, update, and deletion of Jaeger components, such as the Jaeger collector, query, and agent components. When a Jaeger instance is created, the Jaeger Operator deploys the necessary components and sets up the required services and configurations. We are going to deploy the Jaeger Operator in the observability namespace. Use the below-mentioned command to deploy the operator. Shell $ kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.38.0/jaeger-operator.yaml -n observability We are using the latest version of Jaeger, which is 1.38.0 at the time of writing this article. By default, the Jaeger script is provided for cluster-wide mode. Suppose you want to watch only a particular namespace. In that case, you need to change the ClusterRole to Role and ClusterBindingRole to RoleBinding in the operator manifest and set the WATCH_NAMESPACE env variable on the Jaeger Operator deployment. To verify whether Jaeger is deployed successfully or not, run the following command: Shell $ kubectl get all -n observability NAME READY STATUS RESTARTS AGE pod/elasticsearch-0 1/1 Running 0 17m pod/jaeger-operator-5597f99c79-hd9pw 2/2 Running 0 11m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 17m service/jaeger-operator-metrics ClusterIP 172.20.220.212 <none> 8443/TCP 11m service/jaeger-operator-webhook-service ClusterIP 172.20.224.23 <none> 443/TCP 11m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/jaeger-operator 1/1 1 1 11m NAME DESIRED CURRENT READY AGE replicaset.apps/jaeger-operator-5597f99c79 1 1 1 11m NAME READY AGE statefulset.apps/elasticsearch 1/1 17m As we can see in the above output, our Jaeger Operator is deployed successfully, and all of its pods are up and running; this means Jaeger Operator is ready to install the Jaeger instances (CRs). The Jaeger instance will contain Jaeger components (Query, Collector, Agent); later, we will use these components to query OpenTelemetry metrics. Jaeger Instance A Jaeger Instance is a deployment of the Jaeger distributed tracing system. It is used to collect and store trace data from microservices or distributed applications, and provide a UI to visualize and analyze the trace data. To deploy the Jaeger instance, use the following command. Shell $ kubectl apply -f https://raw.githubusercontent.com/infracloudio/Opentelemertrywithjaeger/master/jaeger-production-template.yaml To verify the status of the Jaeger instance, run the following command: Shell $ kubectl get all -n observability NAME READY STATUS RESTARTS AGE pod/elasticsearch-0 1/1 Running 0 17m pod/jaeger-agent-27fcp 1/1 Running 0 14s pod/jaeger-agent-6lvp2 1/1 Running 0 15s pod/jaeger-collector-69d7cd5df9-t6nz9 1/1 Running 0 19s pod/jaeger-operator-5597f99c79-hd9pw 2/2 Running 0 11m pod/jaeger-query-6c975459b6-8xlwc 1/1 Running 0 16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 17m service/jaeger-collector ClusterIP 172.20.24.132 <none> 14267/TCP,14268/TCP,9411/TCP,14250/TCP 19s service/jaeger-operator-metrics ClusterIP 172.20.220.212 <none> 8443/TCP 11m service/jaeger-operator-webhook-service ClusterIP 172.20.224.23 <none> 443/TCP 11m service/jaeger-query LoadBalancer 172.20.74.114 a567a8de8fd5149409c7edeb54bd39ef-365075103.us-west-2.elb.amazonaws.com 80:32406/TCP 16s service/zipkin ClusterIP 172.20.61.72 <none> 9411/TCP 18s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/jaeger-agent 2 2 2 2 2 <none> 16s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/jaeger-collector 1/1 1 1 21s deployment.apps/jaeger-operator 1/1 1 1 11m deployment.apps/jaeger-query 1/1 1 1 18s NAME DESIRED CURRENT READY AGE replicaset.apps/jaeger-collector-69d7cd5df9 1 1 1 21s replicaset.apps/jaeger-operator-5597f99c79 1 1 1 11m replicaset.apps/jaeger-query-6c975459b6 1 1 1 18s NAME READY AGE statefulset.apps/elasticsearch 1/1 17m As we can see in the above screenshot, our Jaeger instance is up and running. OpenTelemetry To install the OpenTelemetry, we need to install the OpenTelemetry Operator. The OpenTelemetry Operator uses custom resources and custom controllers to extend the Kubernetes API with OpenTelemetry-specific functionality, making it easier to deploy and manage the OpenTelemetry observability stack in a Kubernetes environment. The operator manages two things: Collectors: It offers a vendor-agnostic implementation of how to receive, process, and export telemetry data. Auto-instrumentation of the workload using OpenTelemetry instrumentation libraries. It does not require the end-user to modify the application source code. OpenTelemetry Operator To implement the auto-instrumentation, we need to deploy the OpenTelemetry operator on our Kubernetes cluster. To deploy the k8s operator for OpenTelemetry, follow the K8s operator documentation. You can verify the deployment of the OpenTelemetry operator by running the below-mentioned command: Shell $ kubectl get all -n opentelemetry-operator-system NAME READY STATUS RESTARTS AGE pod/opentelemetry-operator-controller-manager-7f479c786d-zzfd8 2/2 Running 0 30s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/opentelemetry-operator-controller-manager-metrics-service ClusterIP 172.20.70.244 <none> 8443/TCP 32s service/opentelemetry-operator-webhook-service ClusterIP 172.20.150.120 <none> 443/TCP 31s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/opentelemetry-operator-controller-manager 1/1 1 1 31s NAME DESIRED CURRENT READY AGE replicaset.apps/opentelemetry-operator-controller-manager-7f479c786d 1 1 1 31s As we can see in the above output, the opentelemetry-operator-controller-manager deployment is running in the opentelemetry-operator-system namespace. OpenTelemetry Collector The OpenTelemetry facilitates the collection of telemetry data via the OpenTelemetry Collector. Collector offers a vendor-agnostic implementation on how to receive, process, and export the telemetry data. The collector is made up of the following components: Receivers: It manages how to get data into the collector. Processors: It manages the processing of data. Exporters: Responsible for sending the received data. We also need to export the telemetry data to the Jaeger instance. Use the following manifest to deploy the collector. YAML kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: config: | receivers: otlp: protocols: grpc: http: processors: exporters: logging: jaeger: endpoint: "jaeger-collector.observability.svc.cluster.local:14250" tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [] exporters: [logging, jaeger] EOF In the above code, the Jaeger endpoint is the address of the Jaeger service which is running inside the observability namespace. We need to deploy this manifest in the same namespace where our application is deployed, so that it can fetch the traces from the application and export them to Jaeger. To verify the deployment of the collector, run the following command. Shell $ kubectl get deploy otel-collector NAME READY UP-TO-DATE AVAILABLE AGE otel-collector 1/1 1 1 41s OpenTelemetry Auto-Instrumentation Injection The above-deployed operator can inject and configure the auto-instrumentation libraries of OpenTelemetry into an application's codebase as it runs. To enable the auto-instrumentation on our cluster, we need to configure an instrumentation resource with the configuration for the SDK and instrumentation. Use the below-given manifest to create the auto-instrumentation. YAML kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: exporter: endpoint: http://otel-collector:4317 propagators: - tracecontext - baggage - b3 sampler: type: parentbased_traceidratio argument: "0.25" EOF In the above manifest, we have used three things: exporter, propagator, and sampler. Exporter: Used to send data to OpenTelemetry collector at the specified endpoint. In our scenario, it is "http://otel-collector:4317". Propagators: Carry traces, context, and baggage data between distributed tracing systems. They have three propagation mechanisms: tracecontext: This refers to the W3C Trace Context specification, which defines a standard way to propagate trace context information between services. baggage: This refers to the OpenTelemetry baggage mechanism, which allows for the propagation of arbitrary key-value pairs along with the trace context information. b3: This refers to the B3 header format, which is a popular trace context propagation format used by the Zipkin tracing system. Sampler: Uses a "parent-based trace ID ratio" strategy with a sample rate of 0.25 (25%). This means that when tracing a request, if any of its parent requests has already been sampled (with a probability of 0.25), then this request will also be sampled, otherwise it will not be traced. To verify that our custom resource is created or not, we can use the below-mentioned command. Shell $ kubectl get otelinst NAME AGE ENDPOINT SAMPLER SAMPLER ARG my-instrumentation 6s http://otel-collector:4317 parentbased_traceidratio 0.25 This means our custom resource is created successfully. We are using the OpenTelemetry auto-instrumented method, so we don’t need to write instrumentation code in our application. All we need to do is, add an annotation in the pod of our application for auto-instrumentation. As we are going to demo a Java application, the annotation that we will have to use here is: Shell instrumentation.opentelemetry.io/inject-java: "true" Note: The annotation can be added to a namespace as well so that all pods within that namespace will get instrumentation, or by adding the annotation to individual PodSpec objects, available as part of Deployment, Statefulset, and other resources. Below is an example of how your manifest will look after adding the annotations. In the below example, we are using annotation for a Java application. YAML apiVersion: apps/v1 kind: Deployment metadata: name: demo-sagar spec: replicas: 1 selector: matchLabels: app: demo-sagar template: metadata: labels: app: demo-sagar annotations: instrumentation.opentelemetry.io/inject-java: "true" instrumentation.opentelemetry.io/container-names: "spring" spec: containers: - name: spring image: sagar27/petclinic-demo ports: - containerPort: 8080 We have added instrumentation “inject-java” and “container-name” under annotations. If you have multiple container pods, you can add them in the same “container-names” annotation, separated by a comma. For example, “container-name1,container-name-2,container-name-3” etc. After adding the annotations, deploy your application and access it on the browser. Here in our scenario, we are using port-forward to access the application. Shell $ kubectl port-forward service/demo-sagar 8080:8080 To generate traces, either you can navigate through all the pages of this website or you can use the following Bash script: Shell while true; do curl http://localhost:8080/ curl http://localhost:8080/owners/find curl http://localhost:8080/owners?lastName= curl http://localhost:8080/vets.html curl http://localhost:8080/oups curl http://localhost:8080/oups sleep 0.01 done The above-given script will make a curl request to all the pages of the website, and we will see the traces of the request on the Jaeger UI. We are making curl requests to https://localhost:8080 because we use the port-forwarding technique to access the application. You can make changes in the Bash script according to your scenario. Now let’s access the Jaeger UI, as our service jaeger-query uses service type LoadBalancer, we can access the Jaeger UI on the browser by using the load balancer domain/IP. Paste the load balancer domain/IP on the browser and you will see the Jaeger UI there. We have to select our app from the service list and it will show us the traces it generates. In the above screenshot, we have selected our app name “demo-sagar” under the services option and its traces are visible on Jaeger UI. We can further click on the traces to get more details about it. Summary In this article, we have gone through how you can easily instrument your application using the OpenTelemetry auto-instrumentation method. We also learned how this telemetric data could be exported to the Elasticsearch backend and visualized it using Jaeger. Integrating OpenTelemetry with Jaeger will help you in monitoring and troubleshooting. It also helps perform root cause analysis of any bug/issues in your microservice-based distributed systems, performance/latency optimization, service dependency analysis, and so on. We hope you found this post informative and engaging. References OpenTelemetry Jaeger Tracing